What Your Login Success Rate Says About Your Credential Stuffing Threat

One of the problems with imitation attacks such as sophisticated credential stuffing is that they are designed to blend in with legitimate traffic. How can you measure something that you can’t detect? Fear-mongering marketing compounds this problem and makes everything sound like a snake-oil solution for a problem people don’t think they have.

Imitation attacks against your services and APIs leverage inherent functionality in your system. In other words, they can be successful even when you have patched, firewalled, and done everything perfectly from an application security standpoint. Blocking basic credential stuffing attacks generated from naive bots is straightforward but, as attackers evolve, they develop more sophisticated tools to launch attacks that blend in with legitimate traffic better. They use machine learning to emulate user behavior like mouse movements and keystrokes. They generate or harvest digital fingerprints to distribute across a botnet to make each node appear more “real.” They proxy requests through residential IP addresses to give the traffic the appearance of originating from known-good home networks. Googles themselves make tools like Puppeteer & headless Chrome to automate and script the world’s most common browser which is exactly what you would use if you were trying to blend in with legitimate users. This is a problem that is getting harder, not easier.

Imitation attacks like advanced credential stuffing do have one thing in common, though – they send millions of requests hoping that a fraction of a percentage end up successful and result in an account takeover, a valid credit card, an account number with a loyalty balance, anything. This success/failure ratio is observable with data you have now. What we’ve found at Shape, is that similar companies have similar success ratios for similar user experience flows.

If you’re debating if you have a credential stuffing problem, then take a long look at your login success ratio.

What is your average login success ratio?

The average login success ratio drops dramatically during periods of credential stuffing attacks. These attacks use combolists with millions of usernames and passwords and of course the majority of these credentials aren’t valid on your site. Shape sees credential stuffing success rates between .2 and 2%, typically – attackers don’t need a very high success rate as long as the attack is cheap to perform. These attacks push the login success rate for your site down well below normal numbers. Some Shape customers have seen login success ratios lower than 5% before enabling countermeasures. Success ratios that low are abnormal and should be immediately investigated. Below are average login success ratios for one month of traffic across three major industries:

  • Financial institutions: 79%
  • Travel industry: 73%
  • Retailers: 62%

Individual companies deviate from this average as much as 10% – the sites where customers log in more frequently tend to have a higher login success ratio. The users at these sites are more likely to remember their passwords and are also more likely to have stored their credentials in their devices or web browsers. Banks and financial institutions only keep users logged in for 15 minutes leading to more successful logins than retailers or social media sites that keep users logged in for longer periods of time. This results in much higher login success rates for banks than for retailers.

Users also have access to few bank accounts and do not change them often, as a result they are more likely to remember their login credentials. Users however regularly shop at multiple retailers and it is easy to create a retail account. This results in lower login success rates for such sites, reflecting a higher rate of users who may be visiting for the first time in months or even years. Infrequent visitors naturally forget their passwords more regularly.

Companies should expect to see 60-85% login success rates. Anything higher or lower is suspect.

No matter the industry, companies should expect to see 60-85% login success rates. Anything higher or lower is suspect. Spikes in traffic can temporarily affect the login success ratio but those should be explainable by commonly understood events like promotions or viral marketing. If there are spikes that have nothing in common then you should look deeper, that traffic is probably a credential stuffing attack that you need to stop as soon as possible.

Graph of a customer who experienced a credential stuffing attack during a steady state of normal login successes.

One caveat

Some industries like banks and other financial institutions are frequently targets for aggregators, services like Mint and Plaid that act as delegates with user permission to log in and gather data across many companies and present it in one unified interface. Aggregators use legitimate credentials and log in multiple times a day, unnaturally inflating the login success rate. You can look for evidence of aggregators by querying for successful logins across multiple users from the same IP addresses, especially if the IP addresses are from cloud or hosting providers. This is not a foolproof method of detection but you will see traces that will help you get a better understanding of your true login success ratio. If you see login success rates in the high 80s or 90s, that is abnormally high and indicative of low credential stuffing threat but high aggregator traffic. Whether or not to consider aggregators a threat is different for every business.

Where to go from here?

What do you do if you find a login success ratio that is concerning? Like with any threat, you need visibility into the attack before you can think about mitigation. Start with free options before committing to a vendor. Tying yourself up with a vendor too early can spin you in the wrong direction and end up wasting months of time. I’ve written an article on 10 things you can do to stop credential stuffing attacks which goes over some free detection methods as well as some mitigation strategies. This should be enough to get you started understanding your problem and, once you understand the scope of your issue, then you can have better conversations with security vendors. Of course we at Shape Security are available to answer questions any time of day and you can feel free to reach out to me personally on twitter.

5 Rando Stats from Watching eCrime All Day Every Day

David Holmes here, cub reporter for Shape Security. While I’m luxuriating in United Airlines’ steerage class, our crack SOC team is back at HQ slaving away over their dashboards as tidal waves of automated traffic crash against the Shape breakers. At least they have Nespresso and those convenient eggs-in-a-bag from the kitch. The day shift of SOC team #1 actually sits pretty close to the corporate marketing brigade, so we kind of know each other and exchange awkward greetings in the hallway.

Breakfast of SOC Champions

ANYWAY, I thought it would be cool to share some statistics from SOC’s recent cases that highlight the shape of the anti-automation industry today.

1. 750 Million in a Week for One Site

Since the release of the Collection #1 credential corpus, some of our customers are experiencing insane levels of login events. One customer saw over 1.5 billion automation attempts in a two-week period. That’s pretty high even for them, one of the largest banks in the solar system. If, for some tragic reason, the Collection #1 campaign persists at its current level, you could extrapolate 39 billion automation attempts in a year (assuming no cracker vacation). Against a single site. That’s sick, brah. Sick.

2. IP Address Re-use: 2.2

This stat is actually sadder than last week’s Grammys. During a credential-stuffing campaign, the attacker throws millions of credentials (gathered from breaches or the “dark web”). If he tried them all from a single IP address, then, of course, you’d just block that IP address, right? So he uses multiple IP addresses. In extreme cases, the most sophisticated cracker will only try a single login from each IP address (no re-use). Lately, the average number of times an IP address will get reused during a campaign is a paltry 2.2.

Basically, blocking by IP address is useless. By the time you add an IP address to your blacklist, it’s too late—it’s not going to be reused again during the campaign. If you see a vendor touting address-blocking, or CAPTCHAs, as a solution, please put your hands on your hips, throw back your head, and issue forth the biggest belly laugh you can. Bwahahaha!

Sadly, some of the technical people we talk to just don’t get it. We tell them: “Blacklists are useless,” and they say “Sure, but you block by IP address, right?” Then we explain it again, and they still don’t get it. Someone should write a paper! Oh, wait, that’s us.

3. Credential Stuffing Succeeds 2% of the Time

2% is funny. It’s our favorite milk. It’s the conversion from US dollars to Philippine pesos. It’s our reader-retention rate when we let Holmes write. Two percent may not sound like much, but consider an attacker testing a million stolen credentials against your web property. That’s 20,000 valid usernames and passwords he’s going to confirm. Actually, the success rate varies between 0.1 and three percent, but two percent is good enough for government work. And speaking of government…

You might be thinking: Actually, guys, 0.1 to 3.0 is a huge range. That’s a multiple of 30. An order of magnitude and then some.  True enough, but when dealing with a million—or even a billion—credentials, the difference is really just “bad outcome” versus “really bad outcome.”

Yesterday Shape looked at a small campaign where a single, lonely attacker in Vietnam had 1,500,000 credentials. Even a 0.1-percent success rate, for him, would have translated to the confirmation, and possible account takeover (ATO), of 1,500 accounts. We say “would have” because we foiled all of his posts. He didn’t even seem to notice, which makes us think maybe he’s TOO automated, or that he suffers from some kind of “educational gap” (that’s the new euphemism for stupidity).

4. 15 Months to an Ugly Baby

The number of months between when some dood stole all your credentials and when you read about it in The Register while eating your precious Honey Smacks is: 15. A lot can happen in 15 months; French words, mostly. Organization penetration, exfiltration, hacker celebration, hacker inebriation, and stock depreciation. Of course 15 months is just an average, and individual cases vary widely, but the point is that it’s an eternity in Internet time.

“Well, dang!” you sputter around your Honey Smacks. “What’s being done about this???”

We’ve got a solution we call Blackfish. We’re already seeing all the waves of credential stuffing against the busiest commercial sites in the world. So we can tell when someone stuffs, say, the creds from your entire customer login database against HoneySmacks.com. Now you don’t have to wait 15 months; if you had Blackfish, you’d know the minute someone tried your logins. How cool is that? If you’re interested, a single chat with our trusty sales chatbot can get the ball rolling for you.

And if you want to read a much more coherent explanation of the 15-month effect, print out our award-winning Credential Spill report, and read it over your Honey Smacks tomorrow.

Disclaimer: Shape Security in no way endorses Honey Smacks; in fact, they have been voted the number #2 worst breakfast you can possibly eat. But dang, they are yums.

5. 99.5% of POSTs are against “forgot-password.js”

Our SOC team dealt with an ATO campaign last month. We remember it well because against that website, we detected that 99.5 percent of requests headed for their “forgot-password” page were automated. Yes, that’s 199/200 for the fractionally-minded (aren’t numbers fun)!

Sure, that’s a single campaign, but in our experience, it’s not an uncommon one. Check your own weblogs and see how the access requests to your forgot-password page compare to, well, anything else (and then call us).

We have many customers for whom forgot-password is their most-frequented page by far. By far! And if our customers weren’t the paragons of morality that they are, they’d put ads on that page and fund themselves a couple of truckloads of egg-in-a-bags. Or is it eggs-in-a-bags? The Oxford dictionary is strangely silent on this topic.


Well, there you have it: five random statistics about fighting anti-automation we slapped together compiled from the last month. Stay tuned, friends, and we at Shape Security’s marketing brigade will bring you more pseudo-cogent security-related statistics, probably from RSA 2019, in a couple of weeks.

Lessons Learned from 2018 Holiday Attacks: No Rest for the Wicked

Scrooge would approve—attackers work on Christmas Eve, and now on New Year’s Eve, too

We at Shape Security defend the world’s top banking, retail, and travel websites. And while you might be just getting back to work this first full week of January, our attack forensics teams are finally getting a break, because this holiday season was a busy one. Now that the dust has settled, we’ve analyzed our data to determine how 2018’s online holiday-season shenanigans differ from 2017’s.

During this festive Holiday season, attackers worked through Christmas Eve and Christmas Day. But in a striking change from the previous year, the most sophisticated attackers no longer took a New Year’s Eve (NYE) off. In fact, this year, we saw several intense campaigns that started or peaked on NYE.

The Best Time to Rob a Bank is Christmas Day

No matter what institution they use, most online banking customers have one thing in common: they stop checking their online balances during the December holidays. Turning a blind eye to one’s finances is optimistic human nature; our customers report that legitimate online banking activity often drops as much as 30 to 40 percent during this period.

Financial institutions may not observe the full extent of this change, however, because the drop in legitimate banking activity is overshadowed by an increase in malicious activity. According to our data, in both 2017 and 2018, malicious actors took advantage of the holiday, launching new attacks on or right around Christmas.


Figure 1: A malicious actor waited to launch their attack until Christmas Day itself.

Shape’s Christmas present to the Top 5 US bank, the target in the above graph, was the fact that we didn’t take Christmas Day off, either.

New Year’s Eve is Cancelled (for Professional Criminals)

With some notable exceptions, nearly all attackers took New Year’s Eve off. On that night, attacks aimed at Shape’s customers dropped over 65% overall – and in one case over 99%, We observed this trend across all industries, including retail, travel, financial services, and tech. Perhaps tired from their exertions over Christmas, nearly all attackers put their keyboards away and joined the poor furloughed federal workers on a break for the New Year’s holiday.

“The holiday season now separates the hobbyists from the dedicated professional cybercriminals.”


Figure 2: Reductions in both legitimate consumer traffic and automated attack traffic.

But the sophisticated attackers, the ones who do this for a living, actually used the global holiday for surgical strikes, particularly against banks .

The attack graph below illustrates the trend. The tiny, tiny red bars on the left (they look like a dotted line) show the normal level of traffic on a financial institution’s website.

Figure 3: Attacker launches failed campaign, retools on NYE, gives up on Jan 1

On December 29, malicious actors launched a large attack against the site. Even by spoofing dozens of signals at all levels – network, client and behavioral, they still couldn’t penetrate Shape’s defenses. On New Year’s Eve they retooled, doubling the number of signals that they were spoofing, but that too, failed, and they gave up towards the end of the day.

Why Launch Attacks During the Holidays?

Sophisticated attackers, the ones for whom crime is their day job, know they are playing a chess game that requires human intervention. So they plan their moves according to when organizations are most vulnerable, i.e., when a security team is most likely to be distracted or short-staffed. What are the days that a security operations team is most likely to be away from their desks? Christmas and New Year’s.

Furthermore, because professional criminals are relying on their ill-gotten gains, they are loath to waste resources. Everyone knows that the top banks are the most lucrative targets, yet hardest to crack. So we suspect that’s why FSIs in particular are targeted during the holidays.

The clearest example of this theory comes from the most sophisticated attack group Shape saw in 2018—a bot that mimicked iOS clients (see our 2018 Credential Spill Report, in which we talk about this attack group). They’d previously targeted a top Canadian retailer, a top global food and beverage company, and a Top 10 North American bank, and we had successfully held them off across our entire customer network.

Figure 4: Sophisticated attacker activity on NYE

This group had been lying low for a couple of months, but on NYE they came back with a sneaky, retooled attack when they thought we weren’t watching. But Shape detected the new attack and quickly blocked it. The attacker gave up on New Year’s Day.

It is not clear why only sophisticated attackers worked on New Year’s Eve this year. We suspect they are getting desperate as more and more organizations harden their application defenses against automated fraud and are looking for any type of vulnerability to exploit. In that case, it’s possible we will see this behavioral trend extend to other major holidays in which companies effectively shut down, such as Chinese New Year and Labor Day.

About Shape Security

Shape Security is defining a new future in which excellent cybersecurity not only stops attackers, but also reduces friction for good customers. Shape disrupts the economics of cybercrime by making it too expensive for attackers to commit online fraud, while also enabling enterprises to more easily transact with genuine customers. The Shape platform, covered by 55 patents, was designed to stop the most dangerous application attacks enabled by bots and cybercriminal tools, including credential stuffing (account takeover), fake account creation, and unauthorized aggregation. The world’s leading organizations rely on Shape as their primary line of defense against attacks on their web and mobile applications, including three of the Top 5 US banks, five of the Top 10 global airlines, two of the Top 5 global hotels, and two of the Top 5 US government agencies. Today, the Shape Network defends 1.7 billion user accounts from account takeover and protects 40% of the consumer banking industry. Shape was recognized by the Deloitte Technology Fast 500 as the fastest-growing company in Silicon Valley and was recently inducted into J.P. Morgan Chase’s Hall of Innovation.


Key Findings from the 2018 Credential Spill Report

In 2016 we saw the world come to grips with the fact that data breaches are almost a matter of when, not if, as some of the world’s largest companies announced spills of incredible magnitude. In 2017 and 2018, we started to see regulatory agencies make it clear that companies need to proactively protect users from attacks fueled by these breaches as they show little sign of slowing.

In the time between Shape’s inaugural 2017 Credential Spill Report and now, we’ve seen a vast number of new industries roll up under the Shape umbrella and, with that, troves of new data on how different verticals are exploited by attacker—from Retail and Airlines to Consumer Banking and Hotels. Shape’s 2018 Credential Spill Report is nearly 50% larger and includes deep dives on how these spills are used by criminals and how their attacks play out. We hope that the report helps companies and individuals understand the downstream impact these breaches have. Credential stuffing is the vehicle that enables endless iterations of fraud and it is critical to have eyes on the problem as soon as possible. This is a problem that is only getting worse and attackers are becoming more advanced at a rate that is devaluing modern mitigation techniques rapidly.

Last year, over 2.3 billion credentials from 51 different organizations were reported compromised. We saw roughly the same number of spills reported each of the past 2 years, though the average size of the spill decreased slightly despite having a new record breaking announcement reported by Yahoo. Even after excluding Yahoo’s update from the measurements in 2017, we saw an average of 1 million credentials spilled every single day.

These credential spills will affect us for years and, with an average time of 15 months between a breach and the report, attackers are already well ahead of the game before companies can even react to being compromised. This window of opportunity creates strong motives for criminals, as evidenced by the e-commerce sector where 90% of login traffic comes from credential stuffing attacks. The result is that attacks are successful as often as 3% of the time and the costs can quickly add up for businesses. Online retail loses about $6 billion per year while the consumer banking industry faces over $50 million per day in potential losses from attacks.

2017 also gave us many credential spills from smaller communities – 25% of the spills recorded were from online web forums. These spills did not contribute the largest number of credentials but their presence underlines a significant and important role in how data breaches occur in the first place. Web forums frequently run on similar software stacks and often do not have IT teams dedicated to keeping that software up-to-date as a top priority. This makes it possible for one vulnerability to affect many different properties with minimal to no retooling effort. Simply keeping your software up to date is the easiest way to protect your company and services from being exploited.

As a consumer, the advice is always the same: never reuse your passwords. This may seem like an oversimplification but it is the 100% foolproof way to ensure that any credential spill doesn’t leave you open to a future credential stuffing attack. Data breaches can still affect you in different ways depending on the details of the data that was exfiltrated, but credential stuffing is the trillion dollar threat and you can sidestep it completely by ensuring every password is unique.

As a company, protecting your users against the repercussions of these breaches is becoming a greater priority. You can get a pretty good idea of whether or not you may already have a problem by monitoring the patterns of your login success rate compared to daily traffic patterns. Most companies and websites have a fairly constant percentage of login success and failures, if you see deviations that coincide with unusual traffic spikes you are likely already under attack. Of course, Shape can help you identify this traffic with greater detail but it’s important to get a handle on this problem regardless of the vendor – we all win if we disrupt criminal behavior that puts us all at risk. As part of our commitment to do this ourselves, Shape also released its first version of Blackfish, a collective defense system aimed at sharing alerts of credential stuffing attacks within Shape’s defense network for its customers. This enables companies to preemptively devalue a credential spill well before it has even been reported.

You can download Shape’s 2018 Credential Spill report here.

Please feel free to reach out to us over twitter at @shapesecurity if you have any feedback or questions about the report.

2017 Credential Spill Report

social_media_10largest_spillsOver the past 12 months, we have seen dozens of the world’s largest online services report that they had been breached by attackers who were able to gain access to their customers’ login credential data. By the end of 2016, over three billion credentials in total were reported stolen, at an average pace of one new credential spill reported every week.

These numbers are a record and include the two largest reported credential spills of all time, both by Yahoo. Near the end of the year, the National Institute of Standards and Technology published the Draft NIST Special Publication 800-63B Digital Identity Guidelines, recommending that online account systems check their users’ passwords against known spilled credential lists.

As the size and frequency of credential spills appears to be increasing, today we are publishing the 2017 Credential Spill Report. This report includes key findings from the credential spills reported in the past year and data from the Shape network to provide insight into the scale of credential theft and how stolen credentials are used.

In particular, stolen credentials are now used every day in credential stuffing attacks on all major online services. In these attacks, cybercriminals test for the reuse of passwords across websites and mobile applications. In the past, announcements of credential spills would focus on the security of accounts at the organization which reported the data breach, but now people are realizing that the widespread reuse of passwords by users across websites means that a breach on one account system endangers all other account systems.

At Shape, we have a unique view into this activity because our technology protects the world’s most attacked web and mobile applications—those run by the largest corporations in financial services, retail, travel, and other industries, as well as the largest government agencies—on a 24/7 basis.

Key statistics from spills reported in the past year include:

Over 3 billion credentials were reported stolen in 2016.

  • 51 companies reported suffering a breach where user credentials were stolen.
  • Yahoo in 2016 reported the two largest credential spills of all time. The next largest credential spills in 2016 were reported by Friend Finder, MySpace, Badoo and LinkedIn.
  • Tech companies had the largest total number of spilled credentials (1.75 billion).
  • The gaming industry had the largest number of companies with spills (11).

From Shape’s network data, we also observed:

  • 90% of login requests on many of the world’s largest web and mobile applications is attributable to traffic from credential stuffing attacks.
  • There is up to a 2% success rate for account takeover from credential stuffing attacks, meaning that cybercriminals are taking over millions of accounts across the Internet on a daily basis as a result of credential spills.
  • Credential stuffing attacks are now the single largest source of account takeover on most major websites and mobile applications.
  • One Fortune 100 retailer experienced a credential stuffing attack with over 10,000 login attempts in one day coming from the cybercriminal attack tool Sentry MBA, which is the most popular credential stuffing software and appears to be used to attack nearly every company in every industry.
  • Analyzing 15.5M account login attempts for one customer during a four month period, over 500K accounts were confirmed to be on publicly spilled credential lists.

Dealing with credential spills and the credential stuffing attacks that they fuel is a complex topic. Here are some basic recommended actions for consumers and enterprises:

The most important takeaway for consumers is that you should never reuse passwords across online accounts. Selecting a strong password is not enough; if you have reused that same password on multiple sites, and one of those sites is breached, your accounts on all of the other sites where you have used the same password are now at risk.

For companies, a lot of public attention is focused on any organization that experiences a data breach and loses control of their users’ credentials. However, the real issue other companies should focus on is protecting themselves against those passwords being used to attack them and their own users. Credential stuffing attacks easily bypass simple security controls like CAPTCHA and Web Application Firewalls, so relying on those mechanisms does not offer any protection. Controls like two-factor authentication can help, but of course come with other drawbacks.

In any case, getting educated is the best course of action. The Open Web Application Security Project (OWASP) provides a starting point for learning about credential stuffing and other automated attacks in their list of OWASP Automated Threats To Web Applications.

To learn more, download the full 2017 Credential Spill Report.

Dan Woods,

Director, Shape Intelligence Center