Intercepting and Modifying responses with Chrome via the Devtools Protocol

At Shape we come across many sketchy pieces of JavaScript. As part of our everyday routine, we dive into them head first to understand what they’re doing and how. The scripts might be maliciously injected into pages, they might be sent by a customer for advice, or our security teams might find a resource on the web that seems to specifically reference some aspects of our service. They are usually minified, often obfuscated, and always require multiple levels of modification before they are really ready for deep analysis.

Until now, the easiest way to do this analysis was either with locally cached setups that enable manual editing or by using proxies to rewrite content on the fly. The local solution is the most convenient, but websites do not always translate perfectly to other environments and it often led people down a rabbit hole of troubleshooting just to get productive. Proxies are extremely flexible, but are usually cumbersome and not very portable – everyone has their own custom setup for their environment and some people are more familiar with one proxy vs another. I’ve started to use Chrome and its devtools protocol in order to hook into requests and responses as they are happening and modify them on the fly. This is portable to any platform that has Chrome, bypasses a whole slew of issues, and integrates well with common JavaScript tooling. In this post, I’ll go over how to intercept and modify JavaScript on the fly using Chrome’s devtools protocol.

We’ll use, node but a lot of the content is portable to your language of choice provided you have the devtools hooks easily accessible.

First off, if you’ve never explored scripting chrome, Eric Bidelman wrote up an excellent Getting Started guide for headless Chrome. The tips there apply to both headless and GUI Chrome (with one quirk I’ll address in the next section).

Launching Chrome

We’ll use the chrome-launcher library from npm to make this easy.

npm install chrome-launcher

chrome-launcher does precisely what you think it would do and you can pass the same command line switches you’re used to from the terminal unchanged (a great list is maintained here). We’ll pass the following options:

  • –window-size=1200,800
    • Automatically set the window size to something that works for us.
  • –auto-open-devtools-for-tabs
    • Automatically open up the devtools because we use them almost every time.
  • –user-data-dir=/tmp/chrome-testing
    • Set a constant user data directory. Ideally we wouldn’t need this but, for some reason, non-headless mode on Mac OSX won’t allow you to intercept requests without this flag. I couldn’t figure out why and as soon as I found a configuration that worked around it I moved on. If you find a better way, please let me know via twitter!
const chromeLauncher = require('chrome-launcher');

async function main() {
  const chrome = await chromeLauncher.launch({
    chromeFlags: [
      '--window-size=1200,800',
      '--user-data-dir=/tmp/chrome-testing',
      '--auto-open-devtools-for-tabs'
    ]
  });
}

main()

Try running your script to make sure you’re able to open chrome. You should see something like this:

Screen Shot 2018-09-13 at 10.46.26 AM

Using the Chrome Devtools Protocol

This is also referred to as the “Chrome debugger protocol” and it seems to be used interchangeably in some of Google’s docs 🤷‍♂️. First, install the package chrome-remote-interface via npm which gives us convenient methods to interact with the devtools protocol. Make sure to have the protocol docs handy if you want to dive in more deeply.

npm install chrome-remote-interface

To use the CDP, you need to connect to the debugger port and, because we’re using the chrome-launcher library, this is conveniently accessible via chrome.port.

const protocol = await CDP({ port: chrome.port });

Many of the domains in the protocol need to be enabled first and we’re going to start with the Runtime domain so that we can hook into the console API and deliver any console calls in the browser to the command line.

const { Runtime } = protocol;
await Promise.all([Runtime.enable()]);

Runtime.consoleAPICalled(
   ({ args, type }) =>
   console[type].apply(console, args.map(a => a.value))
);

Now when you run your script you get a fully functional chrome window that also outputs all of its console messages to your terminal. That’s kind of awesome on its own, especially for testing purposes!

Intercepting requests

First, we’ll need to register what we want to intercept by submitting a list of RequestPatterns to setRequestInterception. You can intercept at either the “Request” stage or the “HeadersReceived” stage and, to actually modify a response, we’ll need to wait for “HeadersReceived”. The resource type maps to the types that you’d commonly see on the network pane of the devtools.

Don’t forget to enable the Network domain as you did with Runtime, above, by adding Network.enable() to the same array.

await Network.setRequestInterception(
  { patterns: [
    {
      urlPattern: '*.js*',
      resourceType: 'Script',
      interceptionStage: 'HeadersReceived'
    }
  ] }
);

Registering the event handler is relatively straightforward and each intercepted request comes with an ​interceptionId that can be used to query information about the request or eventually issue a continuation. Here we’re just stepping in and logging every request we intercept to the terminal.

Network.requestIntercepted(async ({ interceptionId, request}) => console.log(
    `Intercepted ${request.url} {interception id: ${interceptionId}}`
  );

  Network.continueInterceptedRequest({
    interceptionId,
  });
});

Modifying requests

To modify requests we’ll need to install some helper libraries that will encode and decode base64 strings. There are loads of libraries available; feel free to pick your own. We’ll use atob and btoa. Make sure you ​require them to make them available in your script.

npm install btoa atob

The API to deal with responses is a little awkward. To deal with responses you need to do everything on the request interception (as opposed to simply intercepting a response, for example) and then you have to query for the body by the interception ID. This is because the body might not be available at the time your handler is called and this allows you to explicitly wait for just what you’re looking for. The body can also come base64 encoded so you’ll want to check and decode it before blindly passing it along.

const response = await Network.getResponseBodyForInterception({ interceptionId });

const bodyData = response.base64Encoded ? atob(response.body) : response.body;

At this point you’re free to go wild on the JavaScript – you’re in the middle of a response, you have access to the complete JavaScript that was requested, and you can send back whatever you like. Awesome! We’ll just tweak the JS a little by appending a console.log at the end of it so our terminal will get a message when our modified code is executed in the browser.

const newBody = bodyData + `\nconsole.log('Executed modified resource for ${request.url}');`;

We can’t simply pass along a modified body alone because the content might conflict with the headers that were sent with the original resource. Since you’re actively testing and tweaking, you’ll probably want to start with the basics before worrying too much about any other header information you need to convey. You can access the response headers via ​responseHeaders passed to the event handler if necessary, but for now we’ll just craft our own minimal set in an array for easy manipulation and editing later.

const newHeaders = [
  'Date: ' + (new Date()).toUTCString(),
  'Connection: closed',
  'Content-Length: ' + newBody.length,
  'Content-Type: text/javascript'
];

Sending the new response down requires crafting a full, base64 encoded HTTP response (including the HTTP status line) and sending it through a rawResponse property in the object passed to continueInterceptedRequest.

Network.continueInterceptedRequest({
  interceptionId,
  rawResponse: btoa(
    'HTTP/1.1 200 OK\r\n' +
    newHeaders.join('\r\n') +
    '\r\n\r\n' +
    newBody
  )
});

Now, when you execute your script and navigate around the internet, you’ll see something like the following in your terminal as your script intercepts JavaScript and also as your modified JavaScript executes in the browser and the ​console.log()s bubble up through the hook we made at the start of the tutorial.

Screen Shot 2018-09-13 at 11.16.22 AM

The complete working code for the basic example is here:

const chromeLauncher = require('chrome-launcher');
const CDP = require('chrome-remote-interface');
const atob = require('atob');
const btoa = require('btoa');

async function main() {
  const chrome = await chromeLauncher.launch({
    chromeFlags: [
      '--window-size=1200,800',
      '--user-data-dir=/tmp/chrome-testing',
      '--auto-open-devtools-for-tabs'
    ]
  });

  const protocol = await CDP({ port: chrome.port });

  const { Runtime, Network } = protocol;
  await Promise.all([Runtime.enable(), Network.enable()]);

  Runtime.consoleAPICalled(({ args, type }) => console[type].apply(console, args.map(a => a.value)));

  await Network.setRequestInterception({ patterns: [{ urlPattern: '*.js*', resourceType: 'Script', interceptionStage: 'HeadersReceived' }] });

  Network.requestIntercepted(async ({ interceptionId, request}) => {
    console.log(`Intercepted ${request.url} {interception id: ${interceptionId}}`);

    const response = await Network.getResponseBodyForInterception({ interceptionId });
    const bodyData = response.base64Encoded ? atob(response.body) : response.body;

    const newBody = bodyData + `\nconsole.log('Executed modified resource for ${request.url}');`;

    const newHeaders = [
      'Date: ' + (new Date()).toUTCString(),
      'Connection: closed',
      'Content-Length: ' + newBody.length,
      'Content-Type: text/javascript'
    ];

    Network.continueInterceptedRequest({
      interceptionId,
      rawResponse: btoa('HTTP/1.1 200 OK' + '\r\n' + newHeaders.join('\r\n') + '\r\n\r\n' + newBody)
    });
  });

}

main();

Where to go from here

You can start by pretty printing the source code, which is always a useful way to start reverse engineering something. Yes, of course, you can do this in most modern browsers but you’ll want to control each step of modification yourself in order to keep things consistent across browsers and browser versions, and to be able to connect the dots as you analyze the source. When I’m digging into foreign, obfuscated code I like to rename variables and functions as I start to understand their purpose. Modifying JavaScript safely is no trivial exercise and that’s a blog post on its own, but for now you could use something like ​unminify to undo common minification and obfuscation techniques.

You can install it via npm and wrap our new JavaScript body with a call to ​unminify in order to see it in action:

const unminify = require('unminify');

[...]

const newBody = unminify(bodyData + `\nconsole.log('Intercepted and modified ${request.url}');`);

We’ll dive more into the transformations at a later time. I hope this was useful to you and if you have any questions, comments, or other neat tricks please reach out to me via twitter!

Key Findings from the 2018 Credential Spill Report

In 2016 we saw the world come to grips with the fact that data breaches are almost a matter of when, not if, as some of the world’s largest companies announced spills of incredible magnitude. In 2017 and 2018, we started to see regulatory agencies make it clear that companies need to proactively protect users from attacks fueled by these breaches as they show little sign of slowing.

In the time between Shape’s inaugural 2017 Credential Spill Report and now, we’ve seen a vast number of new industries roll up under the Shape umbrella and, with that, troves of new data on how different verticals are exploited by attacker—from Retail and Airlines to Consumer Banking and Hotels. Shape’s 2018 Credential Spill Report is nearly 50% larger and includes deep dives on how these spills are used by criminals and how their attacks play out. We hope that the report helps companies and individuals understand the downstream impact these breaches have. Credential stuffing is the vehicle that enables endless iterations of fraud and it is critical to have eyes on the problem as soon as possible. This is a problem that is only getting worse and attackers are becoming more advanced at a rate that is devaluing modern mitigation techniques rapidly.

Last year, over 2.3 billion credentials from 51 different organizations were reported compromised. We saw roughly the same number of spills reported each of the past 2 years, though the average size of the spill decreased slightly despite having a new record breaking announcement reported by Yahoo. Even after excluding Yahoo’s update from the measurements in 2017, we saw an average of 1 million credentials spilled every single day.

These credential spills will affect us for years and, with an average time of 15 months between a breach and the report, attackers are already well ahead of the game before companies can even react to being compromised. This window of opportunity creates strong motives for criminals, as evidenced by the e-commerce sector where 90% of login traffic comes from credential stuffing attacks. The result is that attacks are successful as often as 3% of the time and the costs can quickly add up for businesses. Online retail loses about $6 billion per year while the consumer banking industry faces over $50 million per day in potential losses from attacks.

2017 also gave us many credential spills from smaller communities – 25% of the spills recorded were from online web forums. These spills did not contribute the largest number of credentials but their presence underlines a significant and important role in how data breaches occur in the first place. Web forums frequently run on similar software stacks and often do not have IT teams dedicated to keeping that software up-to-date as a top priority. This makes it possible for one vulnerability to affect many different properties with minimal to no retooling effort. Simply keeping your software up to date is the easiest way to protect your company and services from being exploited.

As a consumer, the advice is always the same: never reuse your passwords. This may seem like an oversimplification but it is the 100% foolproof way to ensure that any credential spill doesn’t leave you open to a future credential stuffing attack. Data breaches can still affect you in different ways depending on the details of the data that was exfiltrated, but credential stuffing is the trillion dollar threat and you can sidestep it completely by ensuring every password is unique.

As a company, protecting your users against the repercussions of these breaches is becoming a greater priority. You can get a pretty good idea of whether or not you may already have a problem by monitoring the patterns of your login success rate compared to daily traffic patterns. Most companies and websites have a fairly constant percentage of login success and failures, if you see deviations that coincide with unusual traffic spikes you are likely already under attack. Of course, Shape can help you identify this traffic with greater detail but it’s important to get a handle on this problem regardless of the vendor – we all win if we disrupt criminal behavior that puts us all at risk. As part of our commitment to do this ourselves, Shape also released its first version of Blackfish, a collective defense system aimed at sharing alerts of credential stuffing attacks within Shape’s defense network for its customers. This enables companies to preemptively devalue a credential spill well before it has even been reported.

You can download Shape’s 2018 Credential Spill report here.

Please feel free to reach out to us over twitter at @shapesecurity if you have any feedback or questions about the report.