Security and Risk Management Strategies Blog

Static Analysis: How Important is Accuracy?

Software security - or application security, if you prefer - is (to no surprise, I'm sure) a significant ongoing research topic for us. Most recently I completed two documents on static software security analysis, which should publish in the coming 90 days. Talking to users and vendors many important aspects of the static analysis practice came to light, and it is clear that it provides a tremendous value in reducing software security flaws. Most important, of course, is that tools find those "holes you can drive a truck through." I.e. the ones that would be particularly embarrassing to make it into production. And although one would expect these to be easy to find, they may not always be.

There have been various studies of static analysis accuracy, both in the academic and commercial world. Not all are public, and with some of the public ones the detailed results are unfortunately not publicly available. We also spoke with customers about their product evaluations, and even have one customer who performed an incredibly detailed test of several types of security testing tools and services. The results show non-trivial variability in accuracy, inconsistency in the relative performance between products, and potential impact of code structure factors (such as complexity). One study pegs best-case tool performance around 60% for critical flaws. And while this by no means diminish the usefulness of static analysis, it may certainly complicate a customer's evaluation process. Many only perform a comparative analysis of tools run on enterprise code, and thus do not have a baseline to compare against - a method which may or may not actually provide the right decision-making information (but there is often no cost-effective alternative).

A big question of course is how much accuracy really matters. After all, many who use the static analysis and dynamic testing tools are finding large numbers of vulnerabilities already. Being able to work with the results to efficiently identify root causes and put in place effective fixes may for them well be more important than finding even more. Still, the results of the various studies should serve as a reminder that accuracy in the field may be less predictable than one would think, and that ongoing improvements (and more studies) are desirable.

On a related note, the Catalyst workshop Application Security: Process, Tools, and Architecture is now available online at the Burton Group Institute web site. In this workshop I provide an overview of common application security failings and describe at a mid-level detail the components of a software security program.

bioLock: Shameless use of “DLP”

As an analyst, I get a truckload of vendor email trying to convince me how great various security products are. The DELETE key often serves me well, but today I got a message that caught my interest:

“Ultimate Data Loss Prevention for SAP: www.DLP4SAP.com”

Having recently done a fair amount of data leakage prevention research, I thought to myself, “There’s a DLP tool for SAP?”

Looking more closely, it turns out the technology in question is a biometric authenticator. Basically, they’re claiming that better authentication will thwart leakage.

NO, NO, NO!

Talking with clients, the number one concern with leakage is employees inadvertently (or intentionally, in some cases) causing data to go to the wrong place. This is information the employees work with each and every day. In other words, these are authorized users either doing bad things or just making mistakes. But in either case, they’ve already been vetted, authenticated, authorized, etc. The problem isn’t to do a better job of figuring out who they are, but controlling how they use the information. Authentication has nothing to do with it.

I refuse to believe that bioLock doesn’t understand this. Which means they are blatantly (mis)using the DLP tag to get attention for their solution.

That really annoys me!

(By the way, I’ll freely admit that lots of vendors do this – bioLock just happened to catch my attention today.)

The Open Source Path to Security Products

Blogger: Eric Maiwald

Yesterday, Rapid7 announced the acquisition of Metasploit. Two other examples of an open source project being transformed into a commercial enterprise in the area of vulnerabilities, exploits, and detection signatures (all of them are closely related) come to mind – the other two that I’m thinking of Nessus/Tenable and Snort/Sourcefire. The model seems makes sense for technologies around security vulnerabilities.

In all three cases, a technology was created and a community was formed (worldwide in all three cases). The community contributed their knowledge to the project. While some of this knowledge went to the development of the software, much more went into the development of the vulnerability (or signature or exploit) libraries. Beyond the initial creation of the libraries, the community continued to contribute. Thousands of people (perhaps even larger but I don’t think anyone really knows) were continually on the lookout for security vulnerabilities or new exploits. These same people then either notified the community about the issue or they created a vulnerability check, signature, or exploit and contributed it to the project. For the commercialized products, there still must be a QA/QC testing process to make sure the check/exploit/signatures works and does not cause problems when it is deployed.

Now compare that model to a really smart person (or team of people) that starts a commercial company in this same area. In order to get things going, the team must build the software and do all of the research needed to create the vulnerability checks, signatures, or exploits. That is a lot of work and given the number of products already on the market, there is a lot of catch up to do. Once the initial work has been done, the company will need to maintain an expert research team to keep creating the vulnerability checks, signatures, or exploits. The really good teams are not small – they will need people with expertise in a number of different areas – and they are not cheap. An interesting point is that many teams maintain contact with the open source communities and trade information. There are certainly examples of companies that have successfully created a product and maintained a world class team. However, there are also examples of companies trying to get started and failing.

It is hard to beat a large community when it comes to identifying potential problems (even if additional help is needed to commercialize what has been identified) so perhaps the area of vulnerabilities, exploits, and signatures is well served by the open source model. What do you think?

Clouds, Malware, and Quibbling about Free Stuff

“Internet public health crisis.” That’s a phrase we security analysts came up with a couple of years ago to describe the dramatic upswing in damaging malware across the net. When we think about increased reliance on the public Internet for all manner of business, malware is not merely an annoying occasional productivity drain: it’s an epidemiological disaster! A move to cloud computing models is contingent on sound Internet infrastructure in order to work. But with continuous targeted attacks, massive botnets, denial-of-service strikes, and general mayhem so common, there’s considerable risk in relying on the Internet to deliver cloud services.

Ubiquitous home broadband plus dodgy consumer security practices are part of the problem. That’s what makes malware a public health issue. Bruce Schneier talked about this years ago when Microsoft was deliberating limiting Windows XP updates to verified copies. If millions of home users are unprotected from emerging badware, their infected machines affect us all. It’s comparable to human vaccination: every person walking around without inoculation is a potential infection vector for others. Not only home computing users are culpable, either. A recent Network World article, “Cheapskate SMBs dodge buying security software,” points out that small businesses may also be part of the problem.

So I personally welcomed Microsoft’s free entrant to the software security wars, Security Essentials (MSE). By all accounts, it’s a solid if modest way for home users to enhance their protection. Most importantly, it’s cost-effective (namely, free). Perhaps it’s also a lower level of effort compared to other free—and admittedly good—third party options, which also spurs adoption. And by vaccinating themselves, consumers help us all.

Predictably, both Trend Micro and Symantec (and probably others that I don’t know about) have called MSE basic or poor or otherwise unfit for duty. Clearly, they aren’t in this game for charity, and MSE could take a bite out of revenues.  Also, from the perspective of identity fraud and personal security threats, these vendors have a point: consumers should seek security tools that best protect them against life-derailing harm—like a drained bank account due to password theft. However, when it comes to Internet epidemiology, MSE may be just what the doctor ordered. A basic level of protection, offered to consumers for free, which keeps the lid on the most egregious attacks, can only be a good thing for network health. And in an era of increased Internet reliance, free is better than nothing. Let’s stop quibbling about it.

Examining the Data Leakage Prevention (DLP) Market

Data leakage prevention (DLP) continues to be a big topic of interest with Burton Group clients. We just released a “Market Insight” study on the DLP market that describes how vendors are answering the four major use cases:

• Protection of data in motion
• Discovery and protection of data at rest
• Discovery and protection of data in use at user endpoints
• Improving user awareness and training of usage policies for sensitive data

Eric Maiwald and Trent Henry recorded a free BrightTalk webinar to summarize our findings. Here it is:

Open Question from McAfee Focus 09 Security Conference

Blogger: Dan Blum

At the McAfee Focus 09 Security Conference last week in Las Vegas, CEO Dave DeWalt stood under the bright lights on a theater-in-the-round stage, and kicked off proceedings with the pro forma scary characterization of a dangerous threat landscape and McAfee’s strategy on the defense.

Dave highlighted the need for “multilayer defense,” “multilayer correlation,” and “situation awareness.” He said that despite the breadth of McAfee’s product line and its ongoing strategy to build its Security Innovation Alliance through partnerships that enhance the strong, integrated management capabilities of ePolicy Orchestrator (ePO), “we can’t do it alone.” DeWalt appealed to his audience of end users and partners to share their ideas of how the community can be more effectively coordinate multi-vendor products on the defense.

Later that morning, Dave arrived to answer questions from the press and analyst community. At some point as he introduced the session, or perhaps it had been during the keynote itself, DeWalt mentioned that the numbers of McAfee’s alliance partners were growing, Symantec was taking notice and might start something similar, but McAfee had better tools and application program interfaces (APIs) for effective partner coordination. That’s when I decided to ask a question.

After a brief delay while paparazzi snapping innumerable photos blocked my view, the paparazzi sat down, I raised my hand and was acknowledged. My question: “Mr. DeWalt, I’m glad to hear you emphasize the need to coordinate multiple security vendors and products. We hear that from our customers all the time. It’s great to see your partner alliance growing in numbers. But I think it has to be more than a vendor ecosystem and some APIs. If McAfee has an alliance, Symantec has one, and Microsoft has one who knows at the end of the day if all the products will talk to each other? I think there’s a strong need for standards. You spoke about situation awareness; I’d like to call your attention to the Common Event Expression (CEE) work on log standards which would relate to your security information management product line. But more generally, what standards does your Security Innovation Alliance (SIA) plan to promote?”

Paraphrasing Dave’s reply from memory: “We have all kinds of IT standards, but no standards for security. We’ll promote standards but we’ll put our own stamp on it. We’re looking in the areas of a common service bus and a common policy language. But it’s not just technical standards that are needed; it’s also standards for governance.”

A reasonable general sort of answer. But while writing this I looked at the web pages promoting the Security Innovation Alliance, and there’s nothing about standards there now. If standards will be emphasized in the future, the site needs set forth a vision, define objectives, milestones, and progress.

I didn’t really expect a CEO to respond on the spot to something as technical as CEE log standards. Although these standards could in time greatly enhance both real time situation awareness and deeper event log correlation among multiple vendors, they are fairly obscure. That’s why I worded my question to let Dave address standards in a general way. But I did hope to plant a CEE seed in McAfee land. More likely my seed blew away in the wind of a busy week, but perhaps McAfee will see this blog entry and consider what I say. Their comments are welcome here.

Software security: numbers needed!

Two quarters ago, when I did research on general SDLC security, and again this quarter while doing research on the static software analysis space, I find the lack of numbers disturbing. With all the organizations starting or running their software security programs, and with the tools having been available for quite some time now, one would expect more than anecdotal evidence on their effectiveness. It’s not that the numbers haven’t been run by some, but few – if any – are both publicly available and sufficient to base decisions on. Part of the problem is that it’s just such a huge undertaking to evaluate process and tools, another that aggregated and averaged results only fit the average software portfolio and threat environment (if such a thing exists).

The good news however, is that people are working on this. We are continuing to talk to customers about their programs, pain points, and future plans - more research will likely follow. And in the meantime, I will be following the next round of the NIST Static Analysis Tool Exposition, the ongoing work at OWASP and WASC, and I very much look forward to seeing results from Cigital’s BSIMM Begin survey project that just opened for participation. With support from customers and vendors we’ll surely get closer to figuring out how to create better software in better ways. If you know of, or are planning other such evaluation projects and surveys I would be very interested to hear about them.

The issue of SaaS Security

Blogger: Eric Maiwald

Over the last two days I have had a few conversations with clients around security when using software-as-a-service (SaaS). Since the conversations went along the same lines, I thought I would take a few minutes and provide a glimpse of the issue from the vendor’s perspective. For this discussion, I’m talking about SaaS vendors who offer an application that requires the customer to input some type of sensitive information (sales data, customer data, employee data, etc.).

SaaS vendors make money by having a lot of customers use their applications. They make a profit by offering the service as cheaply as possible. For most SaaS vendors this means that they create a single infrastructure (network, servers, databases, etc.) to serve all of their customers (this is called multi-tenancy). What this means is that the vendors build an infrastructure to serve the needs of the majority of customers they seek to attract and they mix customer information within the infrastructure. It also means that they will do their best to make a one-size-fits-all product. In fact, earlier this year, I moderated a panel of vendors and I asked about this issue (the idea of one size fits all). The response I received from one of the vendors was this, “We cannot afford to customize our offerings or contracts for every customer. If we need to build in extra security protections to attract a large customer, we will provide those same protections to every client.”

When a vendor is very young and still learning the business, customers may be able to influence how contracts are written and how the vendor conducts business but as the vendor matures it will become increasingly difficult to cause a change in how the vendor does business. In some cases the vendors will allow customization of the application but if you look carefully at what can be customized, you can see that the vendor’s infrastructure and basic way of doing business does not change.

Another aspect that the vendor has to deal with is risk. The SaaS vendor wants to control his risk (just like you want to control the risk to your business). The vendor will build into the infrastructure the necessary controls that the vendor feels are necessary. However, when it comes to guarantees or service level agreements, most of the vendors become very conservative. If the vendor offers a guarantee, he is balancing his costs against the possibility of failing to meet the guaranteed level of service and having to pay some type of penalty. Some things can be controlled (or at least planned for) pretty well. Availability is a good example of this. The vendor may promise a certain level of availability of the application. He provides this by the extra cost of redundant infrastructure (which may include backup and recovery sites). If the vendor fails to meet the required availability, he promises to refund customers for the time the application is not available. In this case, the vendor knows the cost of implementing the necessary redundancy and the cost of a failure to live up to the agreement.

Generally, you will not find the same types of agreements around the confidentiality of customer information. The vendor may know the cost of implementing various security controls but the vendor faces the same dilemma that all of us in the security business face – it is very hard to determine the probability of a breach depending on the controls within an environment. The second issue is that a breach of confidentiality could occur in many different places. It could be that the vendor was breached or it could be that an authorized user made a mistake and disclosed the information. The vendor might end up in a situation where he has to prove that it was not his fault (which could be nearly impossible). The third issue is the penalty for a failure of confidentiality. Does the vendor have to pay for breach notification, damage to the customer’s brand, damage to the customer’s customers? The costs can be large and variable so how does the vendor determine his risk?

In the end, vendors will provide the controls they feel are necessary to get and retain customers. They will provide proof to the customers of the controls if asked and as long as the proof does not increase their costs too much. The risk to the customer still resides with the customer and therefore the customer still must do the appropriate investigation of the vendor.

Measuring Security Performance, Part II

As a follow-up to the “Measuring Security Performance” blog post I thought I’d spend some more time talking about my thoughts on performance metrics now that our Catalyst conference is over, my “Security KPIs” document has made it to our production group, and the Metricon 4.0 conference happened last week.

I had some very good audience questions during my Catalyst talk – one of which was from a business line manager who wondered how we should measure the “failings” of the security team that complicated the work day of his team. To me the answer to this doesn’t lie in the security metrics space (although we could certainly use them to look for the cause and effect once we know there’s an issue) but rather in general operational metrics such as help desk calls and bug tracking systems.  And this has to be one of our considerations when trying to measure the program: what are the neighboring processes, technologies and their metrics, and how can we leverage them to their best effect?

The question about what is already happening around us is of course part of the context in which we measure, and this is something I heavily focus on in the upcoming security KPIs document. Coincidentally, or perhaps in a stroke of luck, the theme of Metricon 4.0 was actually “The Importance of Context.” I consider contextualized measurement to be of great value for the program-level measurements: by defining units of measurement that are abstracted from the underlying implementation, we can more easily define the goals we’re looking to achieve. Measuring virus counts and patching speed is certainly interesting at the lower level, but at the mid-level we really need to think in terms of such concepts as risks, audit findings, and incidents, and how these relate in terms of performance.

Because enterprises have varying definitions of what constitutes an incident or a material risk this doesn’t necessarily provide us with an easy way to compare between organizations (benchmarking), but then again neither do measurements such as “security budget as percentage of IT budget” or “average time to patch.” But at least by creating an abstraction layer we should be able to better understand and communicate the things that really matter in terms that are understandable.

Pesky Virtual Environments

Blogger: Trent Henry

Burton Group’s Catalyst Conference is in the rear-view mirror now, but we continue to get questions from clients about some of the talks we gave. One that I delivered was called “Security, Audit, and Compliance in Virtual Environments.” (If you’re dying to see me actually wander the stage in full-motion video, check it out at Catalyst Replay: www.catalystreplay.com [pay site].)

Generally speaking, auditors are in the early stages of figuring out the implications of virtual environments, whether VMware, Citrix XenServer, Solaris Containers, or even mainframe LPARs (although they tend not to worry about mainframes—IT auditors have history and comfort with them). PCI DSS is causing the first set of issues, because of two requirements. First, “Implement only one primary function per server.” Virtual machines muddy the interpretation of this:

 

Some QSAs take a strict interpretation, and don’t feel that any application processes should coexist on a physical host. Others believe the hypervisors offer a meaningful level of isolation, and therefore applications running in separate guest OSes are not problematic. The PCI Security Standards Council has created a special interest group to clarify guidance here, and virtualization vendors have joined the group to weigh in. The second troubling requirement is “install and maintain a firewall,” but more on this in a moment.

As I spoke with auditors and clients about virtualization security, a number of questions arose. None of these has been consistently addressed by the industry, but we could be at loggerheads soon:

How do we separate systems and limit audit scope with perimeters and zones?

How are servers hardened from attack and kept up-to-date?

How is storage protected in easily-replicated virtual machines?

How are privileged user access and activities controlled (especially with hypervisor admin users who arguably have uber-root control)?

How do we monitor virtual systems?

System separation is a particularly galling concern, because the notion of “firewall” can be quite different in a virtual machine than on a copper-connected network. Is a virtual security appliance sufficient? Can host-based firewalls serve in lieu of network ones, especially to protect peered hosts that share intra-VM traffic on the same physical server (that is, only the virtual switch is used for communication)? Must we employ software that taps hypervisor privileged APIs like VMsafe or Xen Introspection? Or should we backhaul traffic from virtual switches to traditional network firewalls? (Yikes.)

A final consideration is something I call “controls on the move.”

 

The beauty of virtualization is the flexibility it provides, especially in moving workloads from underutilized machines to help with power management (green computing, and all that…) But notice in the graphic that when we rely on a virtual security appliance to protect guests, in-motion workloads can cause some problems of control inconsistency. Namely, if not every machine in a cluster is configured with the same security controls (e.g., IDS, firewall, monitoring), a guest OS could find itself dynamically moved and effectively stripped of some of its protection: That makes both auditors and security teams unhappy. The answer is to ensure good cluster configuration management—that all machines have a common set of protections. But not every vendor or every enterprise has managed that. An alternative is to focus on host-based controls, so each guest OS has its own security detail no matter where it travels. But this has issues, too, not the least of which is licensing cost.

There’s more to discuss, but this is a starting point. The bottom line is that virtualization is a wonderful technology but has to be approached with balance: audit and security, not only flexibility, are important.