Eight Year Old Paradigm
Back in 2005 I was working for the InfoWorld Test Center. A strange trend started at that time: every intrusion detection system (IDS) began to morph into intrusion prevention systems (IPS.) It seemed that the mandate from analysts and the market to IDS vendors was to either start automatic mitigation (and become an IPS) or get out of the security business. The business problem was that while organizations might be willing to pay top dollar for surveillance solutions, they weren't willing to staff the expensive responders necessary to make the derived intelligence actionable. The prevailing thought of the day was that the cost of staffing a bone fide incident response team was much higher than the value delivered. The end of that logic was that investing in a security operations center (SOC) and incident response teams (IRT) gave a negative return (net loss) and should be avoided at all costs.
Heyday for Hackers
With enterprise security relegated to automated enforcement mechanisms (firewalls, antivirus, etc.) hackers entered into a period that underground history will undoubtedly dub with a moniker akin to the "Golden Period" or some other grandiose and nostalgic title. Fortune 500 companies, were perfectly happy to be oblivious to regular breaches in exchange for the savings that a vacant SOC and IRT provides. If attackers had a problem it was not giggling loud enough to draw attention to the easy money they were hauling out of organizations that weren't staffing smart people to look for them.
A One Legged Stool
In a previous blog entry, Before There Was a Great Wall, I laid out the ignored parallels between physical security and the recent cousin of network security.
Physical security (and a little sense) teaches us that effective security not only requires enforcement (mechanisms that stop bad behavior) but also surveillance and subsequent response. A covert attacker that has no one watching the effectiveness of the enforcement mechanisms has an infinite amount of time to breach them. If there is noone to respond and investigate detected breaches, the infrastructure will continue to be exploited.
Hiring the Team
Below is a maturity path of an organization's security staffing.
Security Administrators
Security Administrators focus on the deployment and management of security solutions designed to stop bad behaviors. Examples of these technologies include firewalls, NAC, antivirus, proxies, advanced mitigation, data loss prevention and IPS. These personnel are critical to keeping the routine threats out yet they lack the skill sets and cycles to look for and respond to attacks that are attempting to circumvent the security infrastructure.
I've been in organizations that will call these administrators security operators or analysts but an examination of both their available tools and operational workflows reveal that they are not effectively looking to detect or respond to sophisticated attacks.
Governance
An investment in governance focuses on meeting fiduciary responsibilities of doing business. PCI, HIPAA and other governing legislation or regulations drive these efforts. Governance employs tools including vulnerability scanners, log collection/archival and infrastructure to deploy segmentation (firewalls, etc.)
While job descriptions often contain declarations of monitoring, these personnel normally are more concerned with appeasing auditors than looking for smart people trying to rob their respective organizations.
Security Operation Center (SOC)
Security Operations Centers (SOC) monitor the network for breaches. SOC may assist Governance in detecting policy violations. SOC utilizes analytical tools largely leaning on security event management solutions (SEIM.) SEIM will be fed by the log data of the Security Admin and Governance tools as well as being supplemented with Global Threat Intelligence, conventional (signature based) intrusion detection systems (IDS) and network based anomaly detection (NBAD.)
This is where things starting getting expensive. Monitoring a global enterprise for intrusions may take anywhere from 5 to 40 people parsing logs and looking for suspicious activity. Each member of the SOC will add more than $120k of payroll and benefit cost to the company with connected yearly training costs (to deal with changing threats) not to mention the expensive tools listed above and their connected maintenance costs.
Incident Response Team (IRT)
Incident Response Team (IRT) will investigate events detected by SOC. IRT will perform forensics on the endpoints and the network logs to determine how a breach was successful and what the impact of the breach was. IRT will coordinate with law enforcement when appropriate. In addition to using data contained in SEIM, IRT will utilize forensic network logging from NetFlow, computer imaging solutions (i.e. EnCase), strategic packet capture and reverse engineering tool sets (IDA Pro, etc.)
For organizations that thought that manning a SOC was expensive, incident response teams will seem to break the bank. Because enterprise has not been staffing these positions, there are very few trained incident responders in the workforce. Staffing a skilled IRT manager may cost more than $300k a year in payroll alone. Training the rest of the team may be necessary to develop the requisite skills. An upfront cost of training may exceed $100k per hire and a global organization may need a IRT of 5-20 people.
Making things even more expensive is the logistic costs of getting physical access to computers and network equipment to the IRT. It will require extensive travel and/or shipping costs.
Expensive But Valuable
After a couple of decades of not having to shell out eight figures worth of payroll to man a SOC and IRT, it is a hard pill for organizations to swallow. It was hard in 2005 and it is hard (or harder) now. The difference between 2005 and now is we are increasingly aware that the value these teams are delivering is not negligible. Customer data and trade secrets are being aggressively targeted by state sponsored attackers (see APT Number One) and organized crime. A decade ago, organizations looked at rumors of mysterious hackers breaking into their networks as urban legend. In 2013, it has become very clear in reports from responders ranging from Mandiant to Symantec Research that hackers are regularly stealing trade secrets, customer data and financial records. Gartner recently reported that an estimated 85% of breaches go completely undetected and 92% of the detected breaches are reported by third parties (law enforcement, hacker extortion demands, etc.) In the 3rd Annual Cost of Cyber Crime Study by Ponemon Institute released in October 2012, the survey of 56 organizations showed an average (detected) loss of $8.9M with the most effected organization logging $46M in loss.
Run The Math Again
These statistics are the most conservative available and are based on quantifiable, detected loss. The actual impact to organizations will actually come in much higher. In a larger organization that can post cyber crime losses of more than $30M a year, why not invest in a $10M SOC and IRT? In 2005, the perceived loss was less than $1M and a cost of $10M would be ridiculous value proposition. The data available in 2013 indicates that a $10M advanced security team can save tens of millions each year.
Get A Little Angry
It may take a while before loss of trade secrets become detected. When trade secrets are covertly stolen, they are plugged into R&D by the organization funding the theft. They will cover their tracks well and you may never be able to make the connection between their "new innovation" and theft of your intellectual property. This theft will undoubtedly steal marketshare from your organization and result in losses that you can't connect to cybercrime. You may not be able to quantify how much a SOC and IRT saves you until after they start catching breaches. They may be the only way you will ever know with certainity that you are being robbed. Are you really ok letting your scumbag competition slowly kill you?
Go Wake Up HR
Companies deciding to accept these attacks without fighting back are going to go the way of the do-do. The only hope of companies not wanting to be robbed blind by smart hackers is to staff smart responders. No amount of technology can ever replace the ingenuity of the human brain whether it be for good or evil. Companies that are paying attention to the writing on the wall are beginning to staff groups that can protect what pays their employees and shareholders: corporate knowledge. The sad state of affairs is that many (if not most) organizations won't know that they were robbed in 2013 until 2016. It's not 2005 anymore; build a security team that can detect and combat advanced attacks.
