Automation enables bot operators to launch attacks at scale and quickly pivot to evade detection. In a race to defend web, mobile, and API channels against these evolving bot attacks, several anti-bot technologies have risen to the occasion. However, motivated bot operators have become highly sophisticated and use multiple techniques to mimic human behavior and fly beneath the radar of traditional bot mitigation solutions.
Modern anti-bot solutions leverage creative tactics to accurately identify these advanced techniques and ultimately disrupt bot operators. The first step in disrupting the never-ending game of cat and mouse is to understand the players.
Understanding the Cat
Such client-side data collection is designed to detect the trace elements of browser automation frameworks that are indicative of a bot. Successfully passing these tests generates a “human” token that provides access to the protected website.
These criticisms largely come from anti-bot vendors who only use server-side detection, arguing that JS client-side detection is the old way of identifying advanced persistent bots. The truth is that there is no silver bullet when it comes to bot mitigation. In order to effectively identify and stop sophisticated bot attacks, accurate real-time defensible client-side detection and server-side detection are both required — and need to work together to disrupt attackers.
Example data flow of a modern anti-bot provider: (1) Client-side scripts execute inside the browser, (2) The browser sends signal data to the telemetry endpoint of the bot mitigation solution, and (3) Server-side detection logic leverages telemetry and threat intelligence to determine whether the request is from a human or bot.
Most anti-bot providers also include “dynamic” challenges to continually revalidate that the requests are being sent from within a browser. For example, a commonly used dynamic challenge is to monitor mouse clicks, movements, and screen interactions. These signals provide validation of human activity by looking for randomisation and variations in movements.
Example of bot-generated mouse movement vs. human mouse movement.
Understanding the Mouse
Bot operators need to obtain a human token and continually imitate human behaviours while conducting their operations. To accomplish this, they attempt to exploit and bypass the anti-bot code that is executed client-side on public devices.
They have two main options to bypass the challenges that the bot mitigation provider uses to differentiate between humans and bots at scale:
- Browser-based bots: use heavily customised versions of the leading automation frameworks, such as Playwright, Puppeteer (Stealth), or Electron. Alternatively, a bot operator may use a fully customised browser. We’ve previously explained why bot builders love these frameworks to fly beneath the radar of anti-bot systems.
- Request bots: use non-browser based scripts to generate and maintain valid “human” tokens by replaying the expected human telemetry to an anti-bot provider. With request bots, the bot builder is applying the benefits of automation towards bypassing the anti-bot system itself.
What Are Request Bots? And How Do They Work?
Unlike browser-based bots, request bots bypass bot mitigation solutions without using browsers or automation frameworks. In its most simple form, a request bot is built by replaying telemetry outside a browser to generate tokens that mimic human behavior to evade detection of bot mitigation solutions.
Step 1 – Generate tokens:
Bot operators will first deobfuscate an anti-bot provider’s detection script and reverse engineer the detection logic and application flow. Once the logic and structure of the anti-bot model are understood, the bot operator can craft and replay requests that contain the required data to ensure that a human classification + token is provided, and in doing so also pollutes the anti-bot provider’s data source used for analytical modeling.
By reverse engineering the logic of the anti-bot providers model, the bot operator can avoid using a browser-based bot. The replayed requests can be written and executed in the coding language of choice. There are many benefits in avoiding the need to execute within a browser, including throughput, cost, complexity, and skill.
Step 2 – Pass dynamic validations:
Once the bot operator can view the deobfuscated version of the anti-bot script and view the unencrypted data being sent back to the provider, they can reverse engineer the dynamic validations that are occurring.
When presented with a mouse movement obstacle, a bot operator generates (or buys) fake mouse movements. This can be easily accomplished using a self-generating randomised human activity script, recording legitimate activity, or paying for a subscription. One example of a resource to generate human-like mouse movement is Ghost Cursor.
Example of a mouse movement generator that only requires 20 lines of code from the sneakerdev crew.
Once the bot operator has reverse engineered the script and request logic, this fake data can be added to the request bot’s fake replayed requests to defeat the anti-bot provider’s validation process. Check out our infographic to see a visual representation of how request bots work.
Why Can’t My Anti-Bot Provider Detect These Request Bots?
Many bot management providers struggle to detect request bots, as the bot operators supply the anti-bot providers’ system with the seemingly legitimate data that tricks the system to validate the request as human. But this is only possible because the design of the anti-bot system allows for the data to be reliably faked and replayed. So there’s a reasonable chance that your anti-bot provider isn’t detecting these request bots, and unfortunately, you wouldn’t know any better.
We see two primary reasons bot mitigation providers fail to accurately detect request bots:
- Use of static detection logic that allows bot operators to hard-code a request bot and replay data. This model is perfect for request bots to succeed, as the bot operator can predict exactly what is needed and hard-code or statically script their requests accordingly. Most providers will provide the same structure and detection logic in every script for every customer.
What Does Kasada Do Differently?
Kasada’s innovative take on bot detection is purpose-built to prevent bot operators from successfully using request bots to evade detection, in addition to detecting stealthy browser-based bots. Our platform uses both client-side detection and server-side data analysis, which share accurate information and secure data with one another.
There are multiple layers to our model that disrupt the bot operators’ modus operandi:
- Dynamic and flexible architecture with reliable data – We further our protection from replay requests with dynamic detection logic. This allows us to present a moving target defence – the detection logic can change at any point, which is a highly efficient way to break a request bot.
- Superior protection against advanced automation – By randomising the structure of our scripts, Kasada ensures that bots cannot automate against our scripts. This removes the possibility of building a request bot as each request needs to honour the structure of the individual instance of our script.
Don’t just take our word for it – see what our customers have to say.
“When evaluating technology providers, we selected Kasada’s solution for its innovative architecture and its immersive 24/7 customer service that is best characterized as ‘embedded’. We find Kasada to be one of our most valuable controls within our ecosystem.” – Benjamin Vaughn, VP & CISO, Hyatt Hotels Corp
“Kasada’s bot defense platform helps us prepare for whatever the latest tactics and attacks are, stop them before they can cause any damage, and deter future attempts to infiltrate our operations. Put simply, Kasada works.” – Howard Blumenthal, Sr. Director eCommerce, Kaman Distribution Group
“From the moment we switched on the platform, there was immediate feedback on the number of page requests that were bot-driven, and I can tell you Kasada neutralized them from the very first page load request. This has enabled us to provide a secure customer experience without the added friction normally found in other solutions.” – Phil Hawkins, Chief Operating Officer, Flybuys
Join our upcoming Stealth Mode Virtual Meetup on this topic, where we will show you how bot mitigation solutions are being worked around by request bots. Register here!
Want to see Kasada in action? Request a demo here. You can also run an instant test to see if your website can detect modern bots, including those leveraging open source Puppeteer Stealth and Playwright automation frameworks.