5 Things We Learned At RSA

1. KBA and NSA are shaping tech startups
General Keith B. Alexander, who retired as NSA Director in 2014, has become the founder and CEO of a new startup, Ironnet. During his RSA session this year, he talked about how to heal the wounds to the tech community and what gift he’d send Snowden if he were given the opportunity. For the tech community, he recommended classified briefings to get technology companies the facts. For Snowden, he said he would send him the oath, which was met with loud applause from the audience. Take a look at the FCW article here.

2. Breaches are happening, even during RSA
On the 2nd day of RSA, a major hotel chain notified their 18 million members via email that their accounts had been reset out of an abundance of caution. According to us at Shape, it seems possible, even likely, that account checkers had been used to hijack 200 accounts at the hotel chain. Take a look at the Shape blog post on account checkers.

3. Taking security up one level – to the Board
Everyone seemed to like and agree with what was said at the presentation, “A CISO’s Perspective on Talking to the Board about Cybersecurity”. See what WSJ wrote about it here.

4. Password management is hard
Shape’s own Zhiwei Li spoke about password managers, exposing several vulnerabilities (now plugged) and discussing which manager would be the best manager in various cases. Take a look at his presentation slides.

5. Botnets are alive and well despite takedowns
Botnets are alive and well, despite takedowns. The federal agencies behind the takeover of a major Zeus botnet (12 governments, 13 companies, 4 non-profits and 3 USG federal agencies) said the criminal enterprises have learned and adapted to build more sophisticated and evasive botnets. Check out the list of agencies involved on the RSA session summary page.

It was a great show for Shape Security. If you go to a lot of conferences, like we do, then we’ll be seeing you at Blackhat in Vegas, and again at RSA in San Francisco in 2016.

Two-Phase Parsing in the Shift JavaScript Parser

Today, we merged the two-phase-parsing branch of the Shift Parser. This branch implemented an experiment with a significant change to the parser’s architecture. To understand it, we first need to understand ECMAScript early errors, and how they are typically handled.

Early Errors

The ECMAScript language is defined using a formal grammar, specifically a context-free grammar. A context-free grammar consists of a number of productions, each associating a symbol called a nonterminal with a (possibly empty) sequence of nonterminal and terminal symbols. Section 5.1 of the ECMAScript 6 specification explains the meaning of its grammars in great detail, but I will summarise it below.

There are two ECMAScript grammars: the lexical grammar, given in section 11, and the syntactic grammar, given in sections 11 through 15. The terminal symbols for the lexical grammar are Unicode code points (the text of your program), while the terminal symbols for the syntactic grammar are tokens, sequences of Unicode code points in the language defined by the lexical grammar.

When a program is parsed as a Script or Module (the two goal symbols of the syntactic grammar), it is first converted to a stream of tokens by repeatedly applying the lexical grammar to the remainder of the program. The token stream only represents an ECMAScript program if the token stream can be parsed by a single application of the syntactic grammar with no tokens left over.

An early error is an additional condition that must hold when a grammar production is matched. From section 5.3 of the ECMAScript 6 specification,

A conforming implementation must, prior to the first evaluation of a Script, validate all of the early error rules of the productions used to parse that Script. If any of the early error rules are violated the Script is invalid and cannot be evaluated.

Let’s take a look at an early error definition from the specification.

12.5.1 Static Semantics: Early Errors

UnaryExpression :
  ++ UnaryExpression
  -- UnaryExpression

It is an early Reference Error if IsValidSimpleAssignmentTarget of UnaryExpression is false.

This early error prevents constructions like ++0 because the named static semantic rule IsValidSimpleAssignmentTarget is true only for identifiers (a), static member access (a.b), computed member access (a[b]), and any of the previous productions enclosed in parentheses ((a)). Notably absent is Literal, the production that matches 0.

The final version of the ECMAScript 6 specification has over one hundred early errors. I know this because I had to write tests for every single one of them. It took a while.

Typical Early Error Handling

Not all early errors are as simple as the one above, where you can immediately know of the existence of an early error. For example, an object literal must not have more than one __proto__ data property. A label cannot be contained within another label with the same name (without crossing function boundaries). As you can imagine, tracking all of the information required to make these decisions can require a lot of complexity. Typical parsers like esprima and acorn use the following strategies:

Lookup Tables

Esprima has an object called state with a field named labelSet. Label names are added to this object when encountered, but if the label already exists, an early error is thrown. Because this state is unique only per parser instance, a stack of labelSet objects must be maintained (in this case, they are using the call stack) so that a fresh labelSet may be generated as each function is entered.

Wrapped Return Values

When parsing function parameters, some early errors must be collected and saved for later, in case it is determined that the parameters are in strict mode and violate a strict-mode-only early error. Remember that a function with a "use strict" directive in its body is strict mode code – including its parameters. Because of this, the function that parses a parameter must wrap all of its return values in a wrapper object that contains both the parameter node and any early errors that would be produced in a strict mode context.

Out Parameters

To avoid wrapper objects in some esprima parsing functions, an object like { value: false } will be passed to the function as an argument. The parsing function will then be able to mutate its parameter as well as return a value, essentially returning more than one value. This strategy is used when the same object can be re-used multiple times, such as the indicator that a data property with the name __proto__ has been parsed by the object property parsing function (the mutation hasProto.value = true is idempotent).

Two-Phase Parsing

We initially implemented our Shift parser in the same way as these others, but it ended up just as unruly and difficult to maintain. So we decided to remove all of the early error handling from the parser, and add an optional second pass that would detect the early errors. This means that the first pass would produce an AST for any input that passed the syntactic grammar. There were some risks with this: we needed to make sure we preserved enough information in the AST to determine if there should be an early error, and we needed to make sure the AST could represent all grammatically correct programs that would otherwise be invalid due to early errors.

The first step was to separate the failure tests that failed due to early errors into their own test suite. We then ensured that we had failure tests for each and every early error in the specification. We then used our new Shift reducer 2.0 to reduce the AST in a single pass into a list of early errors. The reduction state object needed fewer states than we had initially thought it would. It tracks label names, break/continue statements, labeled break/continue statements, new.target expressions, names in binding position, lexical declarations, vardeclarations, var declarations in for-of heads, function declarations, exported names, exported bindings, super calls, and super member accesses. All early errors relate to one or more of those pieces of information and some context.

There weren’t really many difficulties with this approach. There were a few cases that we labeled as “early grammar errors”, which were grammatically correct productions that our AST could not represent: 0 = 0because AssignmentExpression requires a Binding on its left-hand side, (...a) because ArrowExpression must have a body, etc. Additionally, we labeled some early errors as “early tokenisation errors”, including a\u0000, an identifier with a unicode escape sequence whose value is disallowed by an early error.

What We’ve Gained

So was all this trouble worth it? We certainly think it was! First off, the parser is much cleaner, easier to read, and easier to maintain. But if we wanted to, we wouldn’t even have to maintain the parser. Now that there’s no special grammar exceptions, we can replace the entire hand-written parser with one generated by a parser generator. This should be even more easily maintainable and more performant.

Speaking of performance, the parser with early errors disabled is over twice as fast as it was before. In fact, it is faster than any other parser we tested. For our business use, we will almost always want to run in this “loose” mode – we don’t have to worry about being very strict with the inputs we accept. Now one can choose between extremely correct or extremely fast parsing (but even in fast mode, you will never fail to parse a valid program).

Error reporting has also improved with the two-phase parser, since the separate early error checker will collect and report all problems due to early errors in a syntactically valid program instead of just reporting the first one. Finally, this opens up the possibility for the first phase to be extensible, allowing one to add support for JSX, macros, etc. through plugins instead of forks!


We took a pretty big risk with an experimental new parsing strategy, and it has paid off. We’re pretty excited here at Shape about the new possibilities this has opened for us, and will be working with the maintainers of other popular parsers to popularise this approach.

Come see Shape at FluentConf!

Shape will be at O’Reilly’s Fluent Conference in a big way next week and we’re hoping to meet a huge round of new faces in the web community. Several of us will be speaking, doing a book signing, and hanging around our booth in the sponsor’s hall. Make sure you stop by and say hello to:

We’ll probably be chatting about ES2015 (and beyond), JavaScript parsing, the Shift tools, esprima, phantomjs, speedgun, plato, as well as security and performance, of course.

Definitely stop by to see us if you’re still curious as to what Shape is all about. We’re a quiet company doing some amazing things and we’ll take any opportunity to go into detail in person.

Don’t miss the talks below!

PhantomJS for Web Automation by Ariya Hidayat

PhantomJS, the scriptable headless WebKit-based automation tool, has gained a lot of traction in its first 4 years of existence. With >11,000 GitHub stars and ~10M downloads, it becomes the de-facto tool for web application continuous integration, whether to run basic tests or to catch rendering regressions. Many JavaScript test frameworks work well with PhantomJS. In addition, because PhantomJS permits the inspection of network traffic, it is suitable to run various analysis on the network behavior and performance. This talk will highlight the basic usages of PhantomJS and explore various PhantomJS-tools for web applications testing, screen capture, performance analysis, and other page automation tasks.

High Performance Web Sockets by Wesley Hales

Adding a WebSocket service to an application is often misunderstood to be high performance by default, however there are many more considerations that must be made, both on the client and server, before the best performance can be achieved. Real-time technologies like SPDY, WebSocket, and soon HTTP 2.0 have their own sets of hurdles and anti-patterns to overcome and this talk will provide the checklist you need to fine tune your application’s real-time performance.

Debunking Front-end Performance Myths by Ben Vinegar

High Performance Websites, by Steve Souders, was first released in 2007. The follow-up – Even Faster Web Sites – was published in 2009. These books have served as web optimization canon for a generation of web developers. The problem is: it’s now 2015. Browsers, browser features, internet connectivity – they’ve all changed dramatically. A lot of the best practices from 2007 and 2009 no longer apply. And yet, many developers are still holding on to those practices – advocating for performance tweaks that are no longer relevant.

See you there!

Join our RSA session

The Emperor’s New Password Manager: Security Analysis of Password Managers

Friday, April 24, 2015
9:00 AM – 9:50 AM
Room: 3009

Session abstract: We conducted a security analysis of popular web-based password managers. Unlike local password managers, web-based password managers run in browsers. We identify four key security concerns and representative vulnerabilities. Our attacks are severe: in four out of the five password managers we studied, attackers can learn credentials for arbitrary websites. This work is a wake-up call for developers.

Speaker: Zhiwei Li, Research Scientist @ Shape

More information

Meet Shape at RSA 2015

April 20-24 • Moscone Center, San Francisco, CA

RSA 2015 is around the corner. Will you be attending? Come meet with Shape Security and learn more about our technology. We have a booth, are hosting a private meeting suite at the St. Regis hotel, and offering free expo passes to everyone with our discount code. Read more details below.

Shape Booth

Private Meeting Suite

Our Co-Founders sponsored a Suite in the St. Regis. The suite has food, wifi, and a relaxing place to put up your feet. Reserve a time to come unwind and learn about Shape.


Free RSA Expo Passes

Interested in coming to RSA? Enjoy a free pass on Shape.

Register: https://ae.rsaconference.com/US14/portal/login.ww
Coupon code X5ESHPSEC

Browser Compatibility Testing with BrowserStack and JSBin

There are some great services out there that go a long way to outline browser compatibility of JavaScript, HTML, and CSS features. Can I Use? is one of the most thorough, and kangax’s ES5 and ES6 compatibility table are extremely valuable and regularly updated. The Mozilla Developer Network also has a browser support matrix for almost every feature that is documented, but it is a public wiki and a lot of the information comes from the previously mentioned sources. Outside of those resources, the reliability starts dropping quickly.

What if the feature you’re looking for isn’t popular enough to be well documented? Maybe you’re looking for confirmation of some combination of esoteric behaviors that isn’t documented anywhere?

BrowserStack & JSBin

BrowserStack has a service that allows you to take screenshots across a massive number of browsers. Services like these are often used to quickly verify that sites look the same or similar across a wide array of browsers and devices.

While that may not sound wildly exciting for our use case, we can easily leverage a service like jsbin to whip up a test case that exposes the pass or failure in a visually verifiable way.

Using a background color on the body (red for unsupported, green for supported) we can default an html body to the ‘unsupported’ class and, upon a successful test for our feature, simply change the class of the body to ‘supported’. For example, we recently wanted to test the browser support for document.currentScript, a feature that MDN says is not well supported but according to actual, real world chrome and IE/spartan developers, is widely supported and has been for years!

Well we had to know, so we put together a jsbin to quickly test this feature.

<!DOCTYPE html>
  <meta charset="utf-8">
  <title>JS Bin</title>
<body class=unsupported>
  <script id=foo></script>
  <script id=target>
    if (document.currentScript && document.currentScript.id === 'target')
      document.body.className = 'supported';
  <script id=bar></script>

After the bin is saved and public, we select a wide range of browser we wanted to test in browserstack and pass it the output view of the JSBin snippet. In this case we wanted to test way back to IE6, a range of ios and android browsers, opera, safari, and a random sampling of chrome and firefox.

After a couple minutes, we get our results back from BrowserStack and we can immediately see what the support matrix looks like.

This binary pass/fail approach may not scale too well but you could get creative and produce large text, a number, or even an image that is easily parsable at varying resolutions, like a QR code, to convey test results. The technique is certainly valuable for anyone that doesn’t have an in-house cluster of devices, computers, and browsers at their disposal simply for browser testing. Credit goes to Michael Ficarra (@jspedant) for the original idea.

Oh, and by the way, document.currentScript is definitely not widely supported.