Imitation Game – The New Frontline of Security at QCon San Francisco

This week over 1400 software developers are gathering in San Francisco for QCon to share the latest innovations in the developers’ community. The conference highlights best practices in a wide range of emerging technology trends such as microservices, design thinking, and next generation security.

Below are three sessions that will inspire your thinking in next-gen web security and technology.

Wednesday Keynote: The Imitation Game – The New Frontline of Security, 9:00 am, Grand Ballroom, Shuman Ghosemajumder
As one of the four keynote speakers, Shuman Ghosemajumder, Shape’s VP of product management, will discuss the next wave of security challenges: telling the difference between humans and bots. From Blade Runner to Ex Machina, robots in sci-fi have become increasingly sophisticated and hard to distinguish from humans. How about in real life? How are bots taking advantage of user interfaces designed for humans? In his keynote on Wednesday, Shuman will explain how a complex bot ecosystem is now being used to breach applications thought to be secure.

Wednesday Track: The Dark Side of Security, 10:10 am, Bayview A/B, Nwokedi Idika
As Sun Tzu noted in The Art of War, “If you know the enemy and know yourself, you need not fear the result of a hundred battles.” To win the battle against rising cyber criminals, you must know your enemy. How do they think? What do they do before and after the compromise? How do they monetize? In this track, Dr. Nwokedi Idika, Senior Research Scientist at Shape, will guide you on a journey into the minds of the cyber criminals.

Wednesday Track: Javascript Everywhere, 10:10 am, Pacific LMNO, Jarrod Overson
JavaScript usage has been expanding past the browser for years. It’s now used in server applications at companies like Paypal and Walmart, native apps like Slack and Atom, mobile apps like Untappd, and even compilers for game engines like Unreal and Unity. Come to this track led by Jarrod Overson, Director of Software Engineering at Shape and JavaScript super fan, to learn why and how JavaScript is used everywhere.

Want more QCon inspirations? Follow #ShapeSecurity and #QConSF on twitter now.

Salvation is Coming (to CSP)

CSP (Content Security Policy) is a W3C candidate recommendation for a policy language that can be used to declare content restrictions for web resources, commonly delivered through the Content-Security-Policy header. Serving a CSP policy helps to prevent exploitation of cross-site scripting (XSS) and other related vulnerabilities. CSP has wide browser support according to

Content Security Policy 1.0 Implementation Status

Content Security Policy Level 2 Implementation Status

There’s no downside to starting to use CSP today. Older browsers that do not recognise the header or future additions to the specification will safely ignore them, retaining the current website behaviour. Policies that use deprecated features will also continue to work, as the standard is being developed in a backward compatible way. Unfortunately, our results of scanning the Alexa top 50K websites for CSP headers align with other reports which show that only major web properties like Twitter, Dropbox, and Github have adopted CSP. Smaller properties are not as quick to do so, despite how relatively little effort is needed for a potentially significant security benefit. We would be happy to see CSP adoption grow among smaller websites.

Writing correct content security policies is not always straightforward, and mistakes make it into production. Browsers will not always tell you that you’ve made a typo in your policy. This can provide a false sense of security.

Announcing Salvation

Today, Shape Security is releasing Salvation, a FOSS general purpose Java library for working with Content Security Policy. Salvation can help with:

  • parsing CSP policies into an easy-to-use representation
  • answering questions about what a CSP policy allows or restricts
  • warning about nonsensical CSP policies and deprecated or nonstandard features
  • safely creating, manipulating, and merging CSP policies
  • rendering and optimising CSP policies

We created Salvation with the goal of being the easiest and most reliable standalone tool available for managing CSP policies. Using this library, you will not have to worry about tricky cases you might encounter when manipulating CSP policies. Working on this project helped us to identify several bugs in both the CSP specification and its implementation in browsers.

Try It Out In Your Browser

We have also released, which exposes a subset of Salvation’s features through a web interface. You can validate and inspect policies found on a public web page or given through text input. Additionally, you can try merging CSP policies using one of the two following strategies:

  • Intersection combines policies in such a way that the result will behave similar to how browsers enforce each policy individually. To better understand how it works, try to intersect default-src a b with default-src; script-src *; style-src c.
  • Union, which is useful when crafting a policy, starting with a restrictive policy and allowing each resource that is needed. See how union merging is not simply concatenation by merging script-src * with script-src a in the validator.


You can check out the source code for Salvation on Github or start using it today by adding a dependency from Maven Central. We welcome contributions to this open source project.


Web Security Guide to Black Hat 2015

An important web security concept around “A Breach Anywhere is Breach Everywhere,” will be highlighted at Shape’s booth during Black Hat conference this week. Prominent attacks such as Uber account hijackings highlight how spilled credentials obtained from previous breaches can lead to account hijackings on another B2C site.

Make sure to check out Black Hat sessions relevant to escalating web security threats such as password cracking (Cracklord) as well as expanding web attack surface on technologies like EdgeHTML and Node.JS. You can also engage with web security anti-automation experts at the Shape Security booth #558. On Wednesday at 2:30 pm Shape will be hosting Ted Schlein, Partner at Kleiner Perkins (investor in ArcSight, Fortify, Mandiant), former CEO of Fortify and executive at Symantec.

Cracklord – A Friend of Credential Stuffers
If credential stuffing allows criminals to turn lead into gold, hash cracking is the act of digging lead from the Earth. Cracklord, a system designed to crack password hashes, will be explained by researchers from Crowe Horwath. As password cracking tools increase the pool of available credentials, B2C companies need to strengthen their web security defenses to defeat credential stuffing and account hijacking attacks.

New web attack surfaces revealed

Web attack surfaces are constantly expanding as new web technology frameworks and browser technologies continue to be developed and popularized. Those web frameworks offer both the opportunity for built-in security, as well as the risk of a vulnerability affecting the entire user base. In this year’s BlackHat, two briefings on EdgeHTML and Node.JS are particularly relevant.

Researchers from IBM will talk about new attack surfaces within Microsoft’s next generation rendering engine EdgeHTML (codename Project Spartan). Researchers from Checkmarx will talk about different attack methods on Node.JS as well. It’s important for B2C companies to be aware of these new vulnerabilities as attackers are likely to exploit them.

Stop by Shape’s booth #558
Stop by to engage with Shape’s anti-automation specialists to evaluate risks to your website and learn how to protect your web application and mobile API services. On Wednesday, you will get a chance to meet with Ted Schlein, Veteran VC at KPCB (investor in ArcSight, Fortify, Mandiant) and former CEO of Fortify and exec at Symantec.

Have fun and hope you enjoy your week at Black Hat!

Links for relevant sessions on web security

Please follow Shape Security on Twitter – #ShapeSecurity

3 Infosec Notes From Our Time At the MIT Sloan CIO Symposium

Last week, Shape Security attended the MIT Sloan CIO Symposium. Hundreds of CEOs, CIOs, and senior IT professionals from all over the world met to discuss the issues that keep them up at night.
Here we have distilled for you the three most captivating points discussed during the cybersecurity panel.
3. “We are approaching a cybersecurity perfect storm,” said George Wrenn, CSO of European electricity distribution leader Schneider Electric.
Wrenn believes the convergence of  “aging infrastructure, the interconnection of everything, the increasing sophistication of cybercriminals, and the unfixed security weaknesses of the early Internet age” leaves consumers and enterprises vulnerable to attack for the foreseeable future. Not only will it be difficult to address these issues individually, but it will be near impossible to survive a severe, multi-platform attack.
2. “No IT leader wants to stand in the way of innovation or customer satisfaction,” said Roland Cloutier, CSO of payroll services leader ADP
To prevent and survive future attacks, enterprises must shift their focus to mitigating risk over short-term rewards. Customer growth and user retention will only get a company so far if the danger of a breach is always looming. To combat this attitude, product and security leaders must lower risk tolerance across all departments and work together to establish a realistic baseline – for example, a threshold of affected users or records lost.
1. “Adversaries have better technology capabilities than security professionals do sometimes,” said Roland Cloutier, CSO of payroll services leader ADP
Today’s attackers are well-funded entities armed with thousand-node botnets, sophisticated malware, and an entire darknet economy willing to do anything for the right price. This leaves enterprises stuck implementing reactive security measures. The eventual worst-case scenario would be a major national attack that would spur enterprises, governments, and regulatory bodies to produce and enact new security standards. Although the situation would be devastating, the outcome could lead to better protections for consumers.
Take a look at the other events where Shape is attending, exhibiting, and presenting on our website: 

5 Things We Learned At RSA

1. KBA and NSA are shaping tech startups
General Keith B. Alexander, who retired as NSA Director in 2014, has become the founder and CEO of a new startup, Ironnet. During his RSA session this year, he talked about how to heal the wounds to the tech community and what gift he’d send Snowden if he were given the opportunity. For the tech community, he recommended classified briefings to get technology companies the facts. For Snowden, he said he would send him the oath, which was met with loud applause from the audience. Take a look at the FCW article here.

2. Breaches are happening, even during RSA
On the 2nd day of RSA, a major hotel chain notified their 18 million members via email that their accounts had been reset out of an abundance of caution. According to us at Shape, it seems possible, even likely, that account checkers had been used to hijack 200 accounts at the hotel chain. Take a look at the Shape blog post on account checkers.

3. Taking security up one level – to the Board
Everyone seemed to like and agree with what was said at the presentation, “A CISO’s Perspective on Talking to the Board about Cybersecurity”. See what WSJ wrote about it here.

4. Password management is hard
Shape’s own Zhiwei Li spoke about password managers, exposing several vulnerabilities (now plugged) and discussing which manager would be the best manager in various cases. Take a look at his presentation slides.

5. Botnets are alive and well despite takedowns
Botnets are alive and well, despite takedowns. The federal agencies behind the takeover of a major Zeus botnet (12 governments, 13 companies, 4 non-profits and 3 USG federal agencies) said the criminal enterprises have learned and adapted to build more sophisticated and evasive botnets. Check out the list of agencies involved on the RSA session summary page.

It was a great show for Shape Security. If you go to a lot of conferences, like we do, then we’ll be seeing you at Blackhat in Vegas, and again at RSA in San Francisco in 2016.

Two-Phase Parsing in the Shift JavaScript Parser

Today, we merged the two-phase-parsing branch of the Shift Parser. This branch implemented an experiment with a significant change to the parser’s architecture. To understand it, we first need to understand ECMAScript early errors, and how they are typically handled.

Early Errors

The ECMAScript language is defined using a formal grammar, specifically a context-free grammar. A context-free grammar consists of a number of productions, each associating a symbol called a nonterminal with a (possibly empty) sequence of nonterminal and terminal symbols. Section 5.1 of the ECMAScript 6 specification explains the meaning of its grammars in great detail, but I will summarise it below.

There are two ECMAScript grammars: the lexical grammar, given in section 11, and the syntactic grammar, given in sections 11 through 15. The terminal symbols for the lexical grammar are Unicode code points (the text of your program), while the terminal symbols for the syntactic grammar are tokens, sequences of Unicode code points in the language defined by the lexical grammar.

When a program is parsed as a Script or Module (the two goal symbols of the syntactic grammar), it is first converted to a stream of tokens by repeatedly applying the lexical grammar to the remainder of the program. The token stream only represents an ECMAScript program if the token stream can be parsed by a single application of the syntactic grammar with no tokens left over.

An early error is an additional condition that must hold when a grammar production is matched. From section 5.3 of the ECMAScript 6 specification,

A conforming implementation must, prior to the first evaluation of a Script, validate all of the early error rules of the productions used to parse that Script. If any of the early error rules are violated the Script is invalid and cannot be evaluated.

Let’s take a look at an early error definition from the specification.

12.5.1 Static Semantics: Early Errors

UnaryExpression :
  ++ UnaryExpression
  -- UnaryExpression

It is an early Reference Error if IsValidSimpleAssignmentTarget of UnaryExpression is false.

This early error prevents constructions like ++0 because the named static semantic rule IsValidSimpleAssignmentTarget is true only for identifiers (a), static member access (a.b), computed member access (a[b]), and any of the previous productions enclosed in parentheses ((a)). Notably absent is Literal, the production that matches 0.

The final version of the ECMAScript 6 specification has over one hundred early errors. I know this because I had to write tests for every single one of them. It took a while.

Typical Early Error Handling

Not all early errors are as simple as the one above, where you can immediately know of the existence of an early error. For example, an object literal must not have more than one __proto__ data property. A label cannot be contained within another label with the same name (without crossing function boundaries). As you can imagine, tracking all of the information required to make these decisions can require a lot of complexity. Typical parsers like esprima and acorn use the following strategies:

Lookup Tables

Esprima has an object called state with a field named labelSet. Label names are added to this object when encountered, but if the label already exists, an early error is thrown. Because this state is unique only per parser instance, a stack of labelSet objects must be maintained (in this case, they are using the call stack) so that a fresh labelSet may be generated as each function is entered.

Wrapped Return Values

When parsing function parameters, some early errors must be collected and saved for later, in case it is determined that the parameters are in strict mode and violate a strict-mode-only early error. Remember that a function with a "use strict" directive in its body is strict mode code – including its parameters. Because of this, the function that parses a parameter must wrap all of its return values in a wrapper object that contains both the parameter node and any early errors that would be produced in a strict mode context.

Out Parameters

To avoid wrapper objects in some esprima parsing functions, an object like { value: false } will be passed to the function as an argument. The parsing function will then be able to mutate its parameter as well as return a value, essentially returning more than one value. This strategy is used when the same object can be re-used multiple times, such as the indicator that a data property with the name __proto__ has been parsed by the object property parsing function (the mutation hasProto.value = true is idempotent).

Two-Phase Parsing

We initially implemented our Shift parser in the same way as these others, but it ended up just as unruly and difficult to maintain. So we decided to remove all of the early error handling from the parser, and add an optional second pass that would detect the early errors. This means that the first pass would produce an AST for any input that passed the syntactic grammar. There were some risks with this: we needed to make sure we preserved enough information in the AST to determine if there should be an early error, and we needed to make sure the AST could represent all grammatically correct programs that would otherwise be invalid due to early errors.

The first step was to separate the failure tests that failed due to early errors into their own test suite. We then ensured that we had failure tests for each and every early error in the specification. We then used our new Shift reducer 2.0 to reduce the AST in a single pass into a list of early errors. The reduction state object needed fewer states than we had initially thought it would. It tracks label names, break/continue statements, labeled break/continue statements, expressions, names in binding position, lexical declarations, vardeclarations, var declarations in for-of heads, function declarations, exported names, exported bindings, super calls, and super member accesses. All early errors relate to one or more of those pieces of information and some context.

There weren’t really many difficulties with this approach. There were a few cases that we labeled as “early grammar errors”, which were grammatically correct productions that our AST could not represent: 0 = 0because AssignmentExpression requires a Binding on its left-hand side, (...a) because ArrowExpression must have a body, etc. Additionally, we labeled some early errors as “early tokenisation errors”, including a\u0000, an identifier with a unicode escape sequence whose value is disallowed by an early error.

What We’ve Gained

So was all this trouble worth it? We certainly think it was! First off, the parser is much cleaner, easier to read, and easier to maintain. But if we wanted to, we wouldn’t even have to maintain the parser. Now that there’s no special grammar exceptions, we can replace the entire hand-written parser with one generated by a parser generator. This should be even more easily maintainable and more performant.

Speaking of performance, the parser with early errors disabled is over twice as fast as it was before. In fact, it is faster than any other parser we tested. For our business use, we will almost always want to run in this “loose” mode – we don’t have to worry about being very strict with the inputs we accept. Now one can choose between extremely correct or extremely fast parsing (but even in fast mode, you will never fail to parse a valid program).

Error reporting has also improved with the two-phase parser, since the separate early error checker will collect and report all problems due to early errors in a syntactically valid program instead of just reporting the first one. Finally, this opens up the possibility for the first phase to be extensible, allowing one to add support for JSX, macros, etc. through plugins instead of forks!


We took a pretty big risk with an experimental new parsing strategy, and it has paid off. We’re pretty excited here at Shape about the new possibilities this has opened for us, and will be working with the maintainers of other popular parsers to popularise this approach.

Come see Shape at FluentConf!

Shape will be at O’Reilly’s Fluent Conference in a big way next week and we’re hoping to meet a huge round of new faces in the web community. Several of us will be speaking, doing a book signing, and hanging around our booth in the sponsor’s hall. Make sure you stop by and say hello to:

We’ll probably be chatting about ES2015 (and beyond), JavaScript parsing, the Shift tools, esprima, phantomjs, speedgun, plato, as well as security and performance, of course.

Definitely stop by to see us if you’re still curious as to what Shape is all about. We’re a quiet company doing some amazing things and we’ll take any opportunity to go into detail in person.

Don’t miss the talks below!

PhantomJS for Web Automation by Ariya Hidayat

PhantomJS, the scriptable headless WebKit-based automation tool, has gained a lot of traction in its first 4 years of existence. With >11,000 GitHub stars and ~10M downloads, it becomes the de-facto tool for web application continuous integration, whether to run basic tests or to catch rendering regressions. Many JavaScript test frameworks work well with PhantomJS. In addition, because PhantomJS permits the inspection of network traffic, it is suitable to run various analysis on the network behavior and performance. This talk will highlight the basic usages of PhantomJS and explore various PhantomJS-tools for web applications testing, screen capture, performance analysis, and other page automation tasks.

High Performance Web Sockets by Wesley Hales

Adding a WebSocket service to an application is often misunderstood to be high performance by default, however there are many more considerations that must be made, both on the client and server, before the best performance can be achieved. Real-time technologies like SPDY, WebSocket, and soon HTTP 2.0 have their own sets of hurdles and anti-patterns to overcome and this talk will provide the checklist you need to fine tune your application’s real-time performance.

Debunking Front-end Performance Myths by Ben Vinegar

High Performance Websites, by Steve Souders, was first released in 2007. The follow-up – Even Faster Web Sites – was published in 2009. These books have served as web optimization canon for a generation of web developers. The problem is: it’s now 2015. Browsers, browser features, internet connectivity – they’ve all changed dramatically. A lot of the best practices from 2007 and 2009 no longer apply. And yet, many developers are still holding on to those practices – advocating for performance tweaks that are no longer relevant.

See you there!