Reading Time: 18 minutes

Introduction

Today I want to write about what makes a good vulnerability management program. I see a lot of confusion out there where folks are thinking that they have a Qualys or Tenable scanning product therefor they have a vulnerability management program. Or they are performing penetration testing once or twice a year, or they are ISO 27001 certified, and the policies mandated under that constitute vulnerability management. This is very misguided thinking that does nothing to secure your company.

There are all sort of buzzwords flying around the industry where the focus is on one or two aspects that are sexy or exciting. Unfortunately, the work to secure your company is neither sexy nor exciting. It is a lot of arduous work. Because of how multifaceted the security problem is, there is no single solution. The more complex your environment is the more complex it is to secure it. You can either pretend nothing is wrong and have faith you will not experience a security incident, or you can put in the work to secure your environment and thus significantly reduce your risk of a security incident.

Unfortunately believing or declaring that you are secure does not make it so. If you do not want to burst your bubble that you are perfectly safe, you may want to stop reading. If you are ready to take on the arduous work to make your company as secure as possible then this article is for you.

If you want to get a sense for how you really stand, vs what you believe, you should check out my security quiz.

This is a large topic and could easily become a huge book, so to keep this blog sized, I will keep this article somewhat high-level. If you have any questions, need clarifications, or specific actionable steps, please reach out to me. Multiple contact options are listed on the contact us page.

So, what is it

Let us start with some definitions, setting expectation and such. First thing to point out, is that there is no tool you can just buy and be done with it and there is no easy button. Setting up a vulnerability management program and securing your system is a sizable undertaking, possible one of the largest undertaking your organization has ever undertaken. I say this not to scare you but to help you not underestimate the effort involved. Also, if you are a hosting provider it is unlikely you will be able to charge for this work. From a customer’s perspective this would be like charging for installing fire prevention system, or physical security system. You still want to do this because it is bad business to have your hosting customers involved in a security incident.

Also, just as with installation of physical security system and fire prevention system, calculating ROI is very challenging. To continue with this analogy, just like physical security system does not guarantee you will not be burglarized, vulnerability management does not guarantee you will be safe. Both offer you increased safety but there is no such thing as perfect safety.

Next thing to point out, is that compliance and security are two completely different things. There is a bit of a symbiotic relationship between the two, but they are separate entities. Work done under one can help the other, but thinking you are secure because you are compliant with some standard is a dangerous thinking. Similarly, just because you have all your security ducks in a row does not mean you are going to pass any specific audit. It is wise to think of these two as separate, distinct and independent.

As I mentioned in the introduction, vulnerability management is not just a one thing. It is combination of many things, such as:

• Policies, Procedures, culture, and organizational structure
• Asset and configuration Management
• Patch Management
• Event Log Management
• Malware defenses
• Risk and threat modeling
• Vulnerability cataloging and remediating.

Want to point out again that none of this calls for buying any systems. This can all be done with what you already have. Depending on your staffing and expertise, there might be need to hire external help. There might be some systems that could make some of this easier depending on the size of your organization. The smaller you are, the more likely you can do this all without buying anything. To put another way this needs a lot of man hours but little in terms of purchasing budget. If you do not have enough in-house expertise, or your existing staff simply does not have the cycles to take this on you would need to bring in additional help, whether that be consultants or staff augmentation. Expense from setting this up should be 80% staffing expenses.

It is also worth pointing out that Disaster Recovery Planning (DRP) as well as Business Continuity Planning (BCP) play a pivotal part in securing the business and they are likely intertwined. The team that manages DRP and BCP need to work very closely with the security team, in many cases it might even make sense to have them be the same team. DRP/BCP is however outside the scope of this article.

Now let us dive into each of those aspects and explain in more details.

Policies, Procedures, culture, structure

It surprises many just how important all these pieces are to security of an organization. Sometime policies and procedures are included but it is exceedingly rare to see any thought going into culture and organizational structure, when in fact it should be other way around. If you have a strong security-oriented culture and everyone focuses on security first, polices and procedures are less important. However, policies and procedures that no one pays attention to, or is an afterthought is fairly useless.

Polices

These are documents that outline what needs to be done and when. For example, dictates password length, what should be logged, system access rules, etc. These are high level guides if you will, check lists of sorts of what needs to be paid attention to.

Procedures

Procedures take a policy and specifies how it should be implemented. Often known a Standard Operating Procedure (SOP) or Method of Procedure (MOP). Exactly how this is implemented can be culturally dependent. In some organizations SOP give generic, or templated instructions, where are MOP are more system specific changes.

Culture

As I mentioned before, culture plays a significant role in the overall security of the organization. Sometime this is also called the mentality of the employees. This is something that senior leadership cultivates and is a byproduct of their directions, directives and priorities and is more product of actions than words. Even if senior leaders talk about security being important, if they never allow any time or budget for it and are constantly pressuring to get the latest product or version out the door it is a ship-it culture not a security culture regardless of how much leadership talks about security. It is impossible to maintain a secure posture in a hard-core ship-it culture. If, on the other hand, every employee up and down the food chain thinks about security first and when a security issue is discovered everything stops until it is fixed, that is a security-oriented culture.

Senior leadership builds a security minded culture by insisting that all project plans have a security builtin from the beginning and throughout. Every project pitch should have a “how are we securing this.” Every employee and every manager are scored on how they are securing their project, their environment, etc. Security bugs have priority over functional bugs. Time and money are spent on security training applicable to each specific job as well as documentation. Every employee think “security is my job,” instead “security is someone else’s job.”

Organizational Structure

If you have a strong security culture this is less important. It is still useful to have a person or a team in charge of centralized overview. This person/team would make sure that the organization has what they need to support best security posture, oversee all security related policies, help coordinate risk & threat modeling, catalog vulnerabilities and coordinate remediations. Where this team is placed in the organizational structure is critical to its success.

For best success and effectiveness this person or team should be scoped with securing the whole company. If we return to the analogy to physical security, I have never heard of physical security be scoped to only parts of the company, so why would you only secure parts of your electronic assets.

Since it is scoped for the whole company, it should either be a standalone entity or be placed with other departments that work for the whole company, such as payroll, HR, finance, etc. Placing the security person or team at the bottom of the IT or hosting group is a recipe for disaster where failure is all but certain. This is because of human nature and possibility for conflict of interest. It is quite common for folks to ignore those seen as below them in the pecking order. Also this team can be seen as having an auditor function, and it is not possible for an auditor to be impartial when auditing their own boss. Again, if there is a strong security culture this is not as important, otherwise this is critical. Either way the success of the team is based on where it is placed in the org. If it is successful as just another IT team, it will be ten times more successful outside the IT function.

Most successful security teams are standalone groups that report directly to the CEO and have a strong influence over the head of IT/hosting. In corporate America this is often referred to as a dotted line reporting, where a team has more than one reporting responsibility. The head of a security team that reports directly to the CEO is often called Chef Information Security Officer (CISO) or Chief Security Officer (CSO). Title of this person is not as important as the function and the job of this person. What is most critical is that they are outside the divisions that have ownership over the systems and products being secured. So, if it does not make sense for your org to have this be a standalone function that reports directly to the CEO, then make it an administrative function that is in the same division as HR, accounting, and the like.

One other thing we need to touch on here, which I alluded to before and that is to keep compliance away from security. Here I am going to say that titles matter. The person in charge of all the compliance stuff, ISO27001, PCI-DSS, etc., should be called Compliance officer, and the person in charge of security should be called security officer. Even if you are small enough to have one person serve both roles, make it clear that they are two distinct roles, maybe even go as far as saying they have two 50% roles. In this case it might make sense to title this person Compliance and security officer. This is to reinforce that security and compliance are two separate things and reduce the risk of confusing the too. As stated before, confusing compliance and security risks reducing your overall security.

Asset and Configuration Management

This is underpinning and foundation for all security efforts. Simply put if you do not know what you have, you have no chance of securing it. This involves having detailed record of everything in the environment. Your finance department will hopefully have a part of this for account reasons (depreciation and stuff like that). This can be a good base for these purposes, but you need to go a lot deeper than what they need. For example, finance record will typically contain information about what we call financial owner, which is who purchased the system and who bears fiscal responsibility for the system. For security purposes we also need something we call operational owner. This tells us who is responsible for day-to-day operation of the system, specifically who is responsible for fixing it if it breaks, who patches it, etc. Fiscal owner and operational owner could be the same group, but usually it is not, so it should be tracked as two owners even if it is the same. You might also find make and model information in the financial records, but I would not expect to find much beyond that.

A good asset management system has the following pieces of information about each system.

• Asset Name
• Make and Model
• Physical location of the device
• Owner information (both financial and operational)
• Operating system and version
• Every IP address of the system, whether that is one or hundreds.

For a really great system you need to go a lot deeper and upgrade to what is often referred to as Configuration Management Database (CMDB). This is simply an asset management system on steroids. In a great CMDB you capture every little detail about each system. No detail is too small for a great CMDB. This is also one of the tie ins to a strong BCP/DRP. Here are just some of the things that need to go into a CMDB.

• Serial number of the asset
• Asset Type (server, network gear, ADC, firewall, etc)
• Installed hardware components. The more detailed the better
  • What hard drives are installed (Size, make, model, Serial number)
  • What NIC’s are installed (make, model, interface type, MAC address, etc) which IPs live on which NIC.
  • Video cards make, model and capabilities
  • Processor and memory
• Installed software components
  • Any software components installed on top of the OS?
  • For Servers think about:
    • Database software such as MySQL, Microsoft SQL, etc.
    • Web server software like TomCat, Apache httpd, IIS, etc
    • Middleware software like WebLogic
    • Other components or framework like dotNet, PHP, python, log4J, etc.
  • For desktops think about:
    • Office
    • Adobe
    • Browsers
    • Collaboration tools such as Slack and Teams
    • Developer environments
• System criticality indicator on a scale, either numeric or (1-10, 1-5, etc) or word list (High, medium, Low, etc)
• Dependency map. What assets depends on this asset and what assets does it depend on. It would be very unusual for an enterprise server to be truly standalone.
• Lifecycle state (pre-prod, production, decom, etc). This is an informational only and should not play into scoping, etc.

When I say each system, I am talking anything in your network (if it generates traffic or accepts connections it needs to be tracked), this should include cloud assets as well. This could prove tricky considering the ephemeral nature of the cloud and might require integration into the cloud providers control panels.

If you are small enough you should be able to track this in a spreadsheet. If you are bigger you might need to invest in a commercial or open-source asset management system suitable to your organization. Either way you will find that some sort of discovery engine and other automation can greatly help with this effort. These are widely available both as commercial products and free open-source products. There are also number of frameworks and packets available to enable you write your own discovery tool to suite your needs. The only way to obtain 100% accuracy is to use some sort of a discovery, as that can catch the system that Joe user installed without documenting anything and other cases like that. Many years ago I wrote a quick and dirty discovery system in Perl because my employer didn’t have any asset management and was going to fail an audit unless they produced something. So I know it is possible to do this with nothing but your existing staff.

You can not count on automation to do the work for you as I have never seen any automation that can provide dependency map, system criticality, lifecycle state, physical location, ownership, etc. As stated, before spending money on commercial product is optional, a lot of time implementing, collecting, and recording is required either way.

In the end you should be able to answer the following questions at a moment’s notice with 100% accuracy:

• For any given IP address, specify who owns it, what it is (make/model/type), what OS and what software is installed.
• Produce a list of all system running particular OS, particular application, or a particular framework/package.
• Produce a list of all system of particular type, make or model.
• Produce a list of all system with a specified NIC or other hardware component.

Another part of this that is often overlooked is change management. Part of this is always knowing what changed when, both for security and incident management purposes as well keeping the asset management up to date and correct. If you spend hundreds of man hours building the perfect CMDB but you never update it, then it will very quickly become obsolete and useless, and all that time is wasted. The CMDB should be updated with each change to the environment and event logs (see further section) should be leveraged to ensure change management policies are being followed, as well periodic discovery scans.

Patch Management

It could be argued that this is really just a subsection of Policy and Procedure, but it so critical that I felt it was needed to call out separately.

This is about having a solid policy and procedure around who is responsible for keeping all the system patched and up to date on a regular basis. How frequently things are patched and by whom. I would advocate for patching things at least monthly if not more frequently. There are a lot of security incidents that can be traced back to outdated systems. Experian US Credit reporting agency experience a huge breach in September of 2017 which is largely attributed to negligence in keeping their system patched.

Like all aspect of vulnerability management this depends on excellent asset management, because if there is a system in your network that is not being maintained properly because that one guy that installed it and did not document it has left the company, it could be leveraged by cyber criminals (also known as threat actors) to compromise your whole company.

Proper prioritizing is a key aspect of any good patch management program, this prioritization is often built on top of the critically field in the CMDB. It is critical that your patch management program can get through patching all your system in a timely manner. If there is a sense of falling behind in your patching, the patch management program needs to be re-evaluated.

Also note that there are often cases where system cannot be patched, in these cases you need to have compensating control that limit the exposure and risk of the system. See the section on Risk and threat model for more details.

Event Log Management

This is your monitoring system. You can equate it to video monitoring system in a physical security system. If you are not monitoring your event log you are completely blind to what is going on in your environment and will not have a clue if a security incident is taking place. Some call this blissful ignorance; others call this the Ostrich method. I call this willful stupidity and what you do not know will hurt you.

This is a complex topic and there is no way to do it justice in a blog article. Here are some of the things to make sure you are capturing:

• All authentication event (both successful and failures, all types such as local, interactive and network)
• Who talks to who
• Where the traffic egressing to
• What applications are being used, how and when.

Here again is where strong CMDB is critical. It is critically important to know what applications are in use in your environment as well as how they are used and when. Knowing this will allow you know when something is a miss, which is usually an indicator of compromise or a threat actor in your network.

This will likely generate a large flow of data, and rather than trying to watch each event, establish a baseline and trends. Then when things deviate from norm, that is when you investigate. Also look for weird patterns and absence of data. System suddenly stopped sending logs, or is only sending 10% of normal volume, investigate. System suddenly went months out of sync, which is weird so investigate.

These can all be accomplished with free open-source system and some in house development. This is one area where spending more on system can save you on man hours. However, do not be fooled to think that some fancy SIEM system will eliminate the need to spend a bunch of man hours. Yes, it will reduce it, maybe even significantly, but you still need to spend a lot of time and effort tuning the off the shelf system to your environment and to your needs. If you develop your own, you create it based on your needs, off the shelve products need to be customized to your needs. Either case there is significant effort required for tuning.

Malware defenses

This one is a bit contested as this has become a lot less of an issue than it once was and there is heavy debate about how useful it is vs how easy it is circumvent. Most security experts are still advocating heavily for strong malware defenses, and I am one of them. Malware is a term that includes a lot of malicious software, such as viruses. I have a separate article that explains the concept of malware in more details. Suffice it say this is something you want to keep out of your environment for multiple reasons. Having strong anti-malware software, such as F-Secure, Crowdstrike, Sentinel One, etc., installed and configured properly is a critical aspect of your vulnerability management program. Investing in the EDR/XDR functionality of your chosen anti-malware vendor is usually a worthwhile investment for most organizations as it aids with the visibility issue. One just needs to be careful not to become overly reliant on any one tool as that can give one tunnel vision.

Risk and threat modeling

I have a separate blog on this as well, but I will cover it briefly here as well for sake of completeness. First, we should cover the relationship between Risk, threats, and vulnerabilities. This is a simple multiplication formula where risk is vulnerability times threat. Let us do another analogy for this. We are all vulnerable to gunshots, so if someone wanted to shoot us, we would have a big problem. So, if someone were to threaten to shoot us, we would have a substantial risk. How big of risk would depend on how credible and big the threat is. So, let us talk a bit about how to evaluate your risks

This diagram illustrates the connection between opportunity, means and motive to threat. Crime fiction aficionados may recognize some of these words. This is because crime and threat are the same thing, simply different time point. Threat is a crime that has not happened yet but might be happening. As any crime fiction aficionados knows you need three things for crime or threat to be present. The criminal or threat actor needs opportunity, intent (aka motive) and capabilities (aka means). If we return to our shooting scenario, if someone wants to shoot you, they have intent but if they do not have access to a gun (and know how to use it) and they cannot get close enough to you to shoot you (this will depend on how good of a marksman they are) there really is no threat. On the other side, someone that is an excellent expert shooter could be right next to you all day carrying a gun but if they have no desire to shoot you then there really is no threat either. These could all change at any time, so this is a very fluid thing and not a fixed or static thing.

Let us look at a different scenario from the physical world. Let us say someone threatens to extort you over something. They send you their proofs to demonstrate their point and threaten to release it to both social media and traditional media unless you do as they say. You validate that the threat is real, they have the means, motive, and opportunity to do as they threat. However, the material they are threatening to release is insignificant to you and you really could not care less if they sent it to the entire world. Therefor you are not vulnerable to their threat. Even thought the threat when represented numerically might be a high number, the vulnerability is approaching zero so the multiplication of the two is near zero. So, your risk is minimal.

One thing to keep in mind here is that you can never say that either risk or vulnerability is absolutely zero, therefor risk can never be totally zero. Best case scenario is to have minimal risk, there is no such as no risk anymore than there is no absolute security.

The concept of risk and threat modeling is all about going through the exercise of cataloging all the possible threats your business could face, scoring them and lining them up with your vulnerability catalogue (more on this in next section) to find out what your overall risk profile is. This is typically well above the risk appetite set by senior leadership, so you set out to remediate either the vulnerabilities or the risks. This is best done in close coordination with your risk management team or whoever is responsible for you BCP/DRP. Also, as with the other sections having a solid asset management is crucial, because if you have unknown system in your environment, you have unknown risk which can be fatal.

The resulting risk catalogue is a very fluid thing and should be refreshed frequently.

As you work to remediate various issue there a lot of ways to accomplish this. You can reduce the vulnerability, this is often accomplished via patching, or you can minimize the threat by reducing the opportunity or capability to exploit the vulnerability. This is known as compensating controls. Two of the most common means for this is to turn off services that are not in use or use some sort of access control to limit the access to the service that is vulnerable, usually through the use of firewall. In the interest of keeping this blog from getting even longer I will leave the discussion of compensating control to another time.

Vulnerability cataloging and remediating

This is part that many folks think is all they need. I put this last in the article on purpose as this should be the last thing you do in your journey to a solid vulnerability management program. This is because it is always best to build the house after you finish the foundation and all the utilities, power, plumbing, etc. is ready to go. Ever see a house builder start on the house before they finish the foundation and try to retrofit plumbing and utilities midway through or after the house is up? Trying to build your vulnerability management program by starting with this step will work equally badly.

This part involves having some sort of catalogue of all the vulnerabilities across all your systems. The most popular way to accomplish this is to purchase a vulnerability scanner from either the Qualys or the Tenable company. However, you can also accomplish this via the home-grown way. Either way you need to have a powerful asset management as the foundation, especially if you try to go the home-grown route. Both Qualys and Tenable Vulnerability scanners have a powerful discovery engine as the core of their products. Qualys will even sell you an Asset Management solution on top of their Vulnerability Scanner. I have seen way too many organizations try to use these products as a shortcut to overcome lack of asset management and it has always ended in a big struggle. Their scanners will tell you what you have in your environment, but they cannot tell you where it is, who owns it, how critical it is, etc., which are all critical in remediating them. Spinning your wheels for hours trying to figure out owner info, etc., is maddening, especially when those hours could be used remediating more issues. Their scanners can log into your asset and perform a brief inventory on them, assuming you know what credentials are required on what systems. I have seen many instances of this being insurmountable challenge.

If you have a strong CMDB already built, that you have validated with some sort of discovery engine is valid, you can save money on Qualys or Tenable products assuming you are willing to invest in some internal dev efforts. All you need to do is create automation that compares your CMDB to the NVD and to the guidelines published by the like of CIS, OWASP and more. This will give you a rudimentary list of all the things you need to address or remediate. This will not be as detailed as the data you get out of Tenable or Qualys vulnerability scanners, but should be sufficient. On the plus side you might get heads up quicker from your home-grown tool of new issues that are important to you as it takes time for the big vendors to write detection logic for new vulnerabilities.

All vulnerabilities in the world are collated by an institution in USA funded by the US Government. This institution is called National Institute of Standard and Technology (NIST) and it deals with all standards, processes, and procedures for the US Government. For more accurate description check out https://www.nist.gov/about-nist. They compile all the vulnerabilities discovered into something called the National Vulnerability Database (NVD) where each vulnerability gets assigned an identifier called CVE. The whole cybersecurity industry operates on the NVD, and the CVEs are what we use to identify what we are talking about. By integrating your CMDB into the NVD you can get notification the moment a new CVE is published for something you are using. I recommend doing this even if you have a vulnerability scanner from Tenable or Qualys as this will give you heads up days or weeks before those products can scan for it. I have a python script that interacts with the NVD API for this purpose, and I would be happy to share it with anyone who wants to use it as the bases for these efforts.

Once you have your vulnerability catalog you need to work with your management and the business risk team to prioritize the remediation of these vulnerabilities. Depending on the product used you may have multiple datapoints indicating severity of the issue. These are very generic and may not be applicable to your environment. You need to work through the list and establish your own prioritization, decide what you are going to patch, where you have compensating controls, etc. Having a CMDB that tells you how critical a system is, who owns it, where it is, etc., will make this process much easier.

One last point on this: Goals, and metric around eliminating all (or even most) vulnerabilities are very unrealistic goals especially for larger orgs. If you examined the pace of new CVEs being released, you will understand why.

As I said in the beginning, if you have any question, need clarification, or help scoping out the problem please do not hesitate to reach out. My contact info is on the contact us page.

Penetration testing

One final note of caution, until you have your vulnerability management program in place and have remediated all your most critical issues (however you choose to define that) do not waste your money on penetration test, unless of course it is required for compliance reasons. I know it is the cool thing to do these days because all the cool kids are doing it, but it is unlikely to give you any added value until you have cleaned up your house a little. They will end up costing you several tens of thousand dollars (or euros), money better spent on other things I have called out here.