Patching, patching, and patching again

Some time ago I listed some of the critical security patches which became available in February. It was indeed a very busy month, but nothing exceptional. Patching is a very important activity whcih can become overwhelming of it is not done properly. It is second (application patching) and third (OS patching) on the Strategies to Mitigate Targeted Cyber Intrusions published by the Australian's DoD Intelligence and Security team. The first item on the list is application whitelisting.

Patching is not only about the OS. Secunia found that 86% of vulnerabilities in Windows are coming from non-Microsoft products. So Windows Updates are not sufficient.

Steps

Patching is something which should be done as soon as possible. But it does not mean it should be rushed. The typical steps are:

  • Evaluate the criticality of the patch: Critical security vulnerabilities exploitable remotely without user intervention and giving full admin control should not be treated the same as a patch which corrects a typo in the help document.  
  • Evaluate whether the vulnerability could be exploited in your environment: If it's a flaw in the JVM but you (wisely) removed the Java plugin from your browser then this should give you some time. If it's a critical security problem in the very specific version of Apache you're running on public facing servers then you should probably have a serious look into it, and rapidly.
  • Evaluate the risk this vulnerability represents to the assets your want to protect: Would give access to confidential information? Could critical services be impacted, destroyed? Or is it a less important secondary service which would be impacted?
  • Evaluate the mitigation measures you may have or could put in place: If the attacker tries to infect workstations via a known set of servers maybe you can block these IPs at the border. Maybe your IPS already has the signatures to prevent a specific attack to succeed.
  • Evaluate the impact of applying the patch: Some patches are easy to test and install, some require a restart of the service, a reboot of the server, or have an impact on functionalities. Your environment will also determine how easy and rapidly you can test and deploy a patch: Do you have a test environment which functionally mirrors your production environment? What could break with this patch? Do you have high availability in place so you can easily deploy a patch on one node, test, and rollback if necessary? How easy is it to rollback?
  • Deploy the patch: Once you know you have evaluated that the risk of not-patching is higher than the risk of patching then you should deploy the patch as rapidly as you can. Do you have the appropriate tools to deploy patches? Do you have a patching window in place? Do you have the change control process in place?
  • Monitor the environment: Control your monitoring tools to see the impact on your environment (you have monitoring tools and a good baseline, right?) Do you see an impact on the load? Do you see more errors in the logs? More crashes?

Advice

  • Make sure you have a well documented process in place: Checklists and sign-offs.
  • Try to have a test environment which mirror functionally your production environment so you can test patches effectively.
  • Have redundancy wherever you can afford, it makes patching safer and much easier.
  • Have the right tools to ensure you are fully patched, it's easy to miss a critical patch and stay exposed for weeks or months. Secunia or others are doing a pretty good job (their Personal Software Inspector is free for home use).
  • Use auto-update functions where possible: Firefox and Chrome both come with very good and transparent auto-update features, with very limited impact and risk for the environment.
  • Patch regularly, even non-critical patches: you may end up to have to install a whole lot of non-critical patches as a prerequisite to install a critical one. And you don't want to do this overnight...
  • Always use security-in-depth: It is not always possible to patch immediately or a patch may not even be available for a known vulnerability (Secunia estimates that on average 20% of vulnerabilities still don't have a patch available after 30 days). If you have done your job right you should often have ways to mitigate the risk a vulnerability creates: Firewall, IPS, disabled unused services, etc.
How fast can you deploy a critical patch in your environment? On key systems applying a patch overnight is excellent, 48 hours is very good, a few days becomes risky, and more is often not good enough. As always measure the time to deploy a patch and try to improve the slower parts of the process.

Happy patching!

What gets measured gets done

The original saying may have been "If you can't measure it, you can't manage it" or "If you can measure it, you can manage it", and its origin in not clear. But the core idea stays:
  1. Measure where you are today
  2. Decide where you want to be tomorrow
  3. Work towards that goal
  4. Got back to 1
Measuring helps you understand:
  • where you are
  • which direction you're going
  • how far you are from your destination
Not measuring is a bit like finding yourself in the middle of the desert without water, knowing there is an oasis somewhere, but not knowing where, with no map or compass (no GPS either, sorry). Your chances of survival are pretty limited...

In security measuring and managing is called risk management:
  1. Measure your current risk profile (threats, vulnerabilities, assets)
  2. Decide the acceptable risk level (assets value)
  3. Implement mitigations based on priorities (risk reduction, cost)
  4. Go back to 1
Trying to understand where you are today is a very important first step. Many organisation think they are safe because they believe they have not been compromised so far. Security experts have probably all heard something in the lines of: 
nothing happened so far, so why would it be a problem now?
If you don't have antivirus, if you don't look at your logs, if you don't have an IDS, how would you know you have been infected in the first place? You may think you are safe when in fact you've been compromised long time ago. HP estimates that it takes 416 days on average to detect a breach, and that 94% of breaches are found by a third party! The potential loss of money (IP, trust of clients...) is very high.

The second part of the question is about the "why is it a problem now?" and it quite easy: Because the internet is becoming more dangerous. Attackers are smarter, better equipped, better organized and more motivated: The majority of attacks now are done for profit, political or financial. In its latest quarterly report F-Secure found that almost 60% of mobile threats are motivated by money. A clear trend can be seen over the years.
Source: F-Secure
So measure, audit, control!

On your next day to work go to see you cyber security team, and ask them to see what is measured in your company:
  • Do you have the right tools to do the job?
    • Firewall between the different network zones, on all servers and workstations
    • Antivirus on the gateways, all servers and all workstations
    • Hardened devices, whitelisting of applications
    • IDS/IPS on the network and on the hosts
    • Configuration management to monitor changes in the environment
    • Data Loss Prevention mechanisms
    • Central collection of all logs (OS, AV, apps, network...) to avoid losing traces of attack
  • Do you have the process to find and correct problems?
    • Regular review of the collected logs, possibly automatic (SIEM)
    • Rapid deployment of security patches
    • Regular pentests from inside and outside the network
    • Strong SDLC which help deal with security aspects from idea to launch to decommission of a solution
  • How well is this working?
    • How many security patches have not been applied? For what reason?
    • What are the trends of detection and mitigation of attacks (IPS/IDS/Firewall) and other nasties (antivirus, whitelisting)?
    • How many data loss have you been able to detect/prevent?
Don't be afraid to ask for evidence, numbers, independent audit reports. Whether you are the CxO, the business owner , the Security Manager, the Ops Manager or the System Owner "I did not know" or "nobody told me" are not acceptable answers.

 Measure it and manage it!

Busy month for Ops teams!

This month has seen Ops teams needing to install long list of critical updates coming from Microsoft, Adobe and Java on a very regular basis:

And a Polish security firm found yet other vulnerabilities in Java which have not been patched yet!

My recommendations about this are:

  • Limit the execution of Java applets to a limited list trusted websites only, or better to disable Java applets altogether
  • Auto-update as much as you can on the client side: Java, Flash, Chrome, Firefox, etc.
  • Push Microsoft update very rapidly on workstations, preferably immediatly: The risk (likelihood and impact) of something breaking is lower than the cost of cleaning up the environment/reputation after a breach

Cloud services, hosting providers and in-house data centres are all the same

Cloud computing comes to NERSC
Cloud computing comes to NERSC (Photo credit: Lawrence Berkeley National Laboratory)
What have Amazon Web Services, Google Gmail, and IBM data centre in Auckland in common? They all failed and became inaccessible at one point or another. Just as many others.

Can we really rely on cloud services? Are they solid enough?
Yes, they are, provided you have done a good risk analysis, understand what your company is getting and mitigated the risks accordingly. Cloud services or hosting providers are just like in-house hosting:

  • Both can fail and become inaccessible
  • Both can get attacked and broken into
In all cases you need to ask yourself the same questions:
  • Confidentiality
    • How can I prevent unauthorized access to the information?
    • Attackers will try to get access to your data regardless of where it is hosted.
  • Integrity
    • How can I ensure the data is not being tampered with?
    • Voluntary or involuntary modification of your data can occur in your data centre just as it could in the cloud
  • Availability
    • How can I guarantee a level of availability which is in line with the business needs?
    • Your physical or virtual servers can go down, the application can fail, the network can crash, the SAN can fail, your ISP can go down, just like your power or you cloud provider.
Regardless of the option you prefer you'll have choices to make, risks to take or to mitigate, and your environment may suffer. Just make sure you evaluate the CIA of your solution carefully, that the business understands the current risk position and that your have a Business Continuity Plan in place.
Subscribe to: Posts (Atom)