In the post, “Understanding Vulnerabilities,” I introduced you to my dear friends at MITRE, the NVD, and OWASP. You’ll want to get friendly with those fine organizations, if you want to manage your cybersecurity on your own. You’re going to need them as we move into the next step of this exercise: namely, vulnerability remediation.
Now, in a perfect world, every system vulnerability would be patched up immediately after its discovery. But you already know what I am about to say: we don’t live in that world.
In reality, you may not be able to plug every known vulnerability in your system. Why? Often it happens because you’ve got an application or workflow may actually depend on it. For instance, maybe all your accounting data requires obsolete applications that will not run on modern operating systems. You’re stuck running them on old, vulnerable operating systems that are past their end-of-life and no longer supported. Therefore, their security vulnerabilities are not patched, and you cannot upgrade to the current operating system version because the “jalopy app” will not run on it.
Now you have a real problem: You know you’re vulnerable, and you also know you can’t fix it. What do you do? The ideal fix is of course to stop using out-dated apps and transfer your data to something born in the 21st century. But as we just noted above, what’s “ideal” and what’s feasible are not always the same. In which case, you’ll need to apply controls around the known vulnerability to mitigate the risk, until such time as the genius who wrote the application issues a fix.
After you have remediated all the vulnerabilities you can, you’ll need to confirm them as fixed. Your next step, therefore, is to perform a vulnerability test.
This is not to be confused with a penetration test, or “pen test” for short. A pen test, simply put, is a simulated cyberattack against your systems or company. Pen tests are used to assure that a cybersecurity program is working as intended. A vulnerability test, on the other hand, is a diagnostic test that will help in the development of your program.
Best practice, and my strong recommendation, is to retain a reputable third-party vendor to run vulnerability tests on your behalf. They would get the list of your assets and create sets of tests to check against all known vulnerabilities against those assets. Once done, the vendor will present you with a list of holes you need to plug.
A typical vulnerability test follows these steps:
STEP 1: The vulnerability-test vendor will use your asset classification work as input.
This is a critical step, and it’s where the quality of your earlier work gets to shine. The more detailed and prioritized it is, the easier this step becomes. For example, if your list of systems and assets is long, you may be forced to prioritize and segment the test. You will do this based on the criticality of systems and assets that you established earlier. I strongly recommend that you test all systems, but I do understand that time and budgets may force a tiered-testing approach. That’s fine, insofar as you don’t make the mistake of thinking that the least important system doesn’t need the test.
In their attempts to be frugal, too many firms approach cybersecurity as if it were the Maginot Line—France’s “impenetrable” defense against Germany that didn’t extend along the Brussels border, enabling the Nazis to simply go around it. That is exactly what hackers hope you will do with your security. So even if you cannot test everything at once, make sure you do test everything and sooner rather than later. Otherwise, attackers will simply go around your hardened systems and compromise the least important one, using it as a staging point.
STEP 2: Compare the list of assets against known vulnerabilities.
With the assets prioritized, the tester will use external resources to match each asset to its known vulnerabilities. For example, one question might be: “What are the known vulnerabilities for the asset Windows Server 2008?” There are automated applications that do these comparisons, making life easier for all. Nevertheless, that does not absolve you, or the tester, from double-checking that work, especially on critical assets. You want to make sure that any automated system is picking up the most up-to-date vulnerability list and does so from multiple sources and feeds.
STEP 3: Test each asset against these identified vulnerabilities.
This can get tricky. In some cases, testing for the vulnerabilities is as easy as making sure that a security patch has been applied, or the system is running the most current version of software. In other cases, you may need to take the system off-line and check the underlying hardware for one specific vulnerability or another. Either way, you will need the help of at least two technical experts: the vendor performing the test and your in-house (or outsourced) IT team. Both must be in alignment about the goals, the methodology, and the desired end result.
A critical thing to remember here is to avoid pointing any fingers at the asset’s custodians (e.g., the IT team). You need to look at this exercise dispassionately—you simply need to know the truth of the matter—but also with empathy toward the people charged with supporting the asset. Think of a patient going for a CAT scan: the radiologists are not there to judge whether you smoked or not. They want to perform a test and get a result. Even in an adverse diagnosis, one should never berate the patient. What is done is done. The goal after the diagnosis is only to apply the cure and the lessons learned.
STEP 4: Recommend specific steps to either eliminate or mitigate vulnerabilities.
The results of the vulnerability tests tend to be fairly black-and-white: either the asset is vulnerable or it is not. If it is, then you hope that there is a clear path to eliminating the vulnerability. As discussed earlier, that may not always be the case, and the reasons may surprise you. There may not be a “fix” available for the vulnerability—yet! The manufacturer or developer may be currently working on it. A worse situation is if your system is no longer supported and there are no plans to fix the vulnerability (but in this case you might well wonder what you’re doing running an antique system!).
Another common problem is when the vulnerability does in fact have a fix, but if you apply it, your systems downstream may not work properly. The list goes on and on, but the good news here is that you have an answer: you either fix the vulnerability or you take mitigating action. Be aware, however, that the answer may be different for each vulnerable asset.
Prioritizing Vulnerability Remediation
Managing your vulnerabilities can feel like shoveling snow in the middle of a blizzard: Get the list of assets, research the vulnerabilities, plug the vulnerabilities, test the fix, start all over again. No matter how diligent you are, there will always be new ones that are discovered designed to make your life miserable.
The failure to keep up with vulnerability assessments is one of the most common problems I see as a consultant. It is boring work. It is overwhelming. It is constantly changing. And, for many firms, the job is just not as big a priority as it should be. I know many good technologists who may well be aware of a vulnerability on a server but hesitate to apply the vendor patch. Oftentimes they are worried that something else may break downstream, and all of a sudden, what is a half-hour job turns into a two-day crisis.
I understand the hesitation, but I can’t excuse it. According to the Info Sec Institute, the average time it takes companies to fix vulnerabilities is between 60 and 150 days. That’s as much as five months from the moment someone knew about a vulnerability to the moment they addressed it. And as Info Sec notes, “failure to patch vulnerable systems in a timely manner can potentially result in loss of revenue, damage to an organization’s brand, and in some cases complete shutdown of the business.”
The excuses are many: overworked staff, underfunded departments, competing priorities—I’ve heard them all, and I believe them. Most cybersecurity professionals and most IT shops truly are overworked, underfunded, and drowning in competing priorities. Unfortunately, that is not likely to change anytime soon. We therefore need to re-prioritize vulnerability assessment and remediation. It truly represents relatively low-hanging fruit, and we are leaving it to rot on the branch because we are too busy to go get the step stool?
My recommendation is that you engage with your IT department (be it in-house or outsourced) and make sure that vulnerability identification and remediation, along with frequent testing, is someone’s explicit responsibility. You should further insist on a monthly report that shows you the vulnerability status per system, and the remediation timeline. It is silly for us to keep trying to secure the fort if someone leaves the side entrance unlocked!
Speaking of “locks,” in the cybersecurity biz we call those controls! You can learn the basics here in our post here, where we discuss how to select them, how to deploy them, and how to manage them.