Tools – Yvan Boily

Why

At Mozilla I work with a team of really bright people who are working hard on some of the challenges of building secure software and managing the IT security related risks associated with a high-profile open source project. One of the biggest challenges we face is that Mozilla is a public benefit organization; this means that we don’t tend to drive the same scale of insane profits that some of our competitors do (and profits aren’t bad thing!), which means that we have to find new and interesting ways to scale; I spoke about some of these things at AppSecUSA at the end of October, and the video should be available online soon.

Adding to this challenge is the state of the Application Security workforce. For skilled AppSec people, the unemployment rate is essentially 0%. For under-skilled[1] AppSec people, it is not much higher; this provides the challenge of trying to recruit skilled, knowledgeable people to help with some of our activities. We have had a huge amount of success over the last two years in growing and building an effective team, but like every other security team, we are increasingly competing for a small number of high value individuals. (Hey, interested in applying for jobs? Look here!)

To even further complicate matters the amount of competition in the security space, and the history of the security community telling everyone that security sucks, security is hard, and they aren’t smart enough to get security right, have all contributed to the proliferation of Security As a Service vendors. Unless you have a huge amount of financial resources, compelling and interesting problems to work on, or some other incentive, it is very difficult to attract security talent when vendors are offering top dollar for people to work in their bug mills (again, not a bad thing; good quality teams are expensive, and vendors can help offer access to good talent).

What

For the last year I have been preaching automation to our team, and last spring we started growing our Security Automation Engineering team. Following that, we recently started to build out a new team to focus on developer oriented tools. We wanted an “security tool automation system”. We envision a tool that will provide basic automation for a range of security tools, with intelligently selected configurations for ease of use, and well thought out configuration options to allow the user to configure a range of tools by modifying a small set of values. This tool is called Minion.

Minion is intended to be disruptive.

First, it will help us break out of the pattern of running and using security tools within our team; we want all of the developers in our organization to use them. We want our developers to do horrible things to the applications and services they write, and we want it to be as easy as the push of a button. If my team is asked to review an application[2] for release to production, having passed through Minion should be one of the first steps.

Second, we want Minion to become a Security As A Service platform. Automation is not the answer to everything; In the long term, Minion should provide a solid toolkit that any security team can use to organize around. Once we finish the core feature set of Minion, we will start to focus on the team oriented aspects of the platform. It should be easy to extend, easy to adopt, and support hooking into each step of the SDLC. In short, we want to create a platform that allows any good security team to compete with the established Security As A Service players in terms of service delivery, documentation, and workflow, without constraining people to a specific philosophy or process.

Finally, we want Minion to be a framework. As an information gathering tool, it is effectively useless if that information feeds into a write-only repository. To that end we are working to define clear, extensible formats that can be used to integrate with and consume data from Minion. Want to create a fancy dashboard for C-Level executives? Minion should allow that. Want to write a module to automate generating a report worthy of a typical audit firm? Minion should allow you to do that! Want to perform data mining to find out vulnerabilities that are most common in the applications you test? Minion should allow that! By building Minion around REST APIs and modern web standards, it is possible to build any kind of mashup you can think of with the data that Minion collects and the results it tracks.

How

I pushed for Minion to be created, but it would still be a poorly written set of PoC scripts if it wasn’t for the hard work of Matthew Fuller, Simon Bennetts, and Stefan Arentz. These guys have done an amazing job, and I look forward to seeing how they continue to push the project forward! If you are interested in Minion, you can check out the repo, or join the mailing list using the links below!

Thanks for reading!

Yvan

[1] “Underskilled”: Securing web applications is hard. Securing applications is hard. Securing applications that people use to build web applications and access web applications is really hard. Not everyone is cut out for the job, and there is nothing wrong with that; there are a lot of opportunities for people that don’t demand staying on the bleeding edge of technology.

[2] Note that this is different from the consultative security work we do over the course of a project; Minion is not a substitute for a security program, it simply offloads some of the testing burden to automated tools and lays a foundation that the security team can look at when starting a final review before a product or service is released.

This post is cross-posted from the Mozilla Web Application Security blog.

Garmr is deprecated. This post is informational.

Earlier this year I wrote about some of the challenges of scaling security efforts in an organization, and I mentioned that we are working to adopt better tooling to assist us in this. We have been working towards improving security in the development lifecycle by making security tests a part of the quality assurance process. In order to accomplish this we worked with the QA team at Mozilla to create a simple tool called Garmr to integrate automated security test cases as part of our continuous integration (CI) processes.

When we started the discussion about which tools to use, our requirements were pretty straightforward; it needed to be fast, simple, and accurate. When we looked at Selenium and Mozmill the feedback we got from other teams is that although powerful, these frameworks were complex to configure properly, and had significant overhead to maintain as a project progresses. Since the focus was on testing web applications in a repeatable fashion, we identified a set of required functionality and David Burns from the very awesome Mozilla QA team put together the Garmr prototype and initial test cases.

Garmr has been public since its initial version, but the tool has now reached the point where we will start to leverage it in our testing and CI processes. The tool works by running a series of tests against target URLs and reporting back the results. The results are currently formatted as a JUnit style XML report that can be consumed by other tools such as Jenkins.

Garmr Test Cases

Garmr currently supports two types of test cases that can be authored, an Active Test and a Passive Test. ActiveTests are expected to make at least one HTTP request, and return the most relevant HTTP response after the check is completed. Here is an example of an ActiveTest that makes a simple HTTP GET request:

class WebTouch(ActiveTest):
    run_passives = True
    description = "Make a GET request to the specified URL, and check for a 200 response after resolving redirects."
    def do_test(self, url):
        response = requests.get(url)
        if response.status_code == 200:
            result = self.result("Pass", "The request returned an HTTP 200 response.", None)
        else:
            result = self.result("Fail", "The response code was %s" % response.status_code, None)
        return (result, response)

The test case makes an HTTP Get request, and passes or fails dependent on the response code. This very basic check does serve a purpose; it grabs an HTTP response to run passive tests against. For an example of a more complex check, look at the StsUpgradeCheck implemented in the corechecks module.

PassiveTest instances receive a copy of an HTTP response and analyze it to determine if the test passes or fails. In most circumstances, Garmr will run all of the configured PassiveTests against the result of each ActiveTest. The goal of a passive test is to inspect each the response of a query for properties that are expected from a secure web application.

This is an example of a passive test that checks an HTTP response for the presence of X-Frame-Options headers:

<code data-enlighter-language="python" class="EnlighterJSRAW">class XFrameOptionsPresent(PassiveTest):
    description = "Check if X-Frame-Options header is present."
    def analyze(self, response):
        xfoheader = "X-Frame-Options"
        xfo = xfoheader in response.headers
        if xfo == False:
            result = self.result("Fail", "X-Frame-Options header not found.", None)
        else:
            result = self.result("Pass", "X-Frame-Options header present.", response.headers[xfoheader])
        return result</code>

The goal is to allow developers to create test cases that are specific to their applications and gain the benefit of running pedantic checks for security related attributes such as cookies and headers each step of the way. Note that there is no practical limitation on what a passive check does, as long as the result object is returned properly.

Using Garmr

The current version of Garmr can be downloaded from the Garmr Github repository.

Getting, installing, and using Garmr is very easy:

git clone https://github.com/mozilla/Garmr.git
cd Garmr
sudo python setup.py install
garmr -u http://my.target.app

Limitations

Garmr is an alpha tool. The only part of it that we expect to remain stable at this point is the Xml Report format that it produces, since this is specified by the Jenkins tool chain that we are integrating with.

This tool is not intended to replace a typical dynamic or static analysis tool any more than the existence of Unit Tests replaces end to end functional testing, but it does provide a simple facility for QA, development, and security teams to collaborate on web application testing.

Currently the tool can be used to run a number of simple tests, and supports the ability to load modules; two examples of how a module would be written are included in djangochecks.py and webchecks.py

There is a list of features still to come:

less noisy CLI
proxy support (already supported in requests)
sessions (controlled; sequence for active tests, with a cookie jar that is propagated through the session)
detailed reporting, including the ability to record all HTTP requests and responses generated
the ability to filter which passive checks are run by check name or by check type (i.e. cookies, headers, content-type, etc)
support for additional protocols (websockets, spdy)
Implement instances of each test case for each target scanned to allow them to retain state as a set of tests progresses.

I am very excited to see how the tool will develop over the next few months as we have an intern who will be working to implement these and other features, so please test it out, give us feedback (patches and feature requests are welcome too!).

Tag: Tools

Minion – Why, What, and How

Why

What

How

Automating Test Cases

Garmr Test Cases

Using Garmr

Limitations