Bot Mitigation: The Complete Guide to Mitigating Bots

The majority (83%) of companies say that bad bots are becoming more sophisticated and difficult for their security tools to detect, according to our recent State of Bot Mitigation survey. At the same time, only 15% report that their anti-bot solution retained its effectiveness a year after initial deployment.

Casual Internet users can think of it as a relatively simple process: point your browser to a website, hit enter, and hey presto, you’ve got what you came for. However, things are rarely so straightforward, and especially not when we’re talking about something as complex and sprawling as the internet. You won’t find all the answers in the Dummies Guide to the Internet.

Much of the Internet traffic takes place behind closed doors, invisible to the public, invisible, even, to the people who run the websites. Many of these elements are logistical in nature and of no interest to anyone but the die-hard internet aficionados.

But there are some invisible elements that should be of interest to website and web application owners (and everyone else). They’re called bots. They’re not all bad (in fact, many are good), but they do pose a threat that has to be managed. In this blog, we’re going to look at everything related to bots, including the different types, the threats they pose, and the importance of bot mitigation. (And you can expect to learn much more than you would in the Dummies Guide.)

So to begin, let’s take a look at what bots are. If you’re a regular internet user, then you’ll likely have heard of the name before, but do you know what they are? Essentially, they’re just software applications that perform automated tasks over the internet. Well, we say that they’re “just” that — an internet bot is pretty complicated by nature. They perform functions at a much higher speed and at a higher rate than humans can. They are robots in the shape of software, which is where the name ‘internet bot’ comes from.

As with any capable tool, it’s not the existence of bots that causes problems. It’s what the bot is ordered to do that causes issues. You could have a good bot that crawls the internet to find the most relevant web pages related to a search (as in the case of Google and other search engines). But similarly, you could have a bad bot that’s solely interested in being bad — these are the ones that are responsible for hacking, spamming, and all the other dangerous things we associate with nefarious internet use. Those bots are good for nobody (except for the people that are behind them).

So, what distinguishes a good bot from a bad bot? We’re about to share more on that topic so you know how to identify a good bot when you see one.

As we said above, that bots exist isn’t an issue. In fact, the good parts of the internet wouldn’t function as it does without bots. So, we shouldn’t be too critical of these pieces of software, at least not across the board. But, alas, much of the bad aspects — even just the annoying aspects — of the internet can be traced back to bots. To get a clearer idea of an internet bot’s influence, let’s dive deeper into the good and bad kinds. We’ll start with the good, because you should always try to build someone up (before you knock them down, as we will).

Not all bots have a bad reputation. Good bots can actually help your business. Here are just a few of the good bots out there:

One of the joys of using an internet application to purchase items is that you can usually get whatever it is you’re looking for at a lower price than you’d find in the real world. But sometimes, finding those deals can be difficult. Shopping comparison websites use bots to crawl the internet and feed back the best price for whatever item is being sought. Some online retailers also use bots to show the best price to the user based on their use of the website.

Websites don’t necessarily stick around forever. If the owner of the site has decided to move onto new projects, then the website — along with all the content — will die. But sometimes, that content is worth preserving. Web scraping crawlers, or web spiders, go around the web and record the data, which can then be accessed if the website goes offline. An example of this use would be Wayback Machine, the internet archive website. It should be noted that web scraping crawlers can be used for more sinister motives, too.

If you’ve ever used the ‘live chat’ feature of a website, then you’ll likely have been talking with a chatbot. These bots are designed to stimulate the conversation of a real human, giving responses based on the wording of the customer. Many of these bots identify themselves as non-human. They can be basic (gathering basic information before handing it over to a live agent) or sophisticated, providing a full service of information in much the same way a “real human” would.

Bots are increasingly used to get important information out into the world. They’ll push out breaking news stories via notifications, and in some instances — such as TechCrunch — bots are used to send out personalized stories to their users. Many automated Twitter accounts publish relevant news as and when it’s available, without any human oversight. These user accounts are dangerous.

Much of the chat related to bots is pretty serious, or it can feel that way. If it’s not serious, then it’s boring. But bots are also designed with just fun in mind. There are many games where you can play “against” the bot, while some bots just sit around and make art. A nice life!

And perhaps the most useful aspect of a bot software application: the search engine. It’s impossible to imagine what the internet would be like without a solid search engine by our sides (really: what would you do? Ask your friends if they know a site?). A big part of Google’s success has been the sophisticated nature of its crawler bots. What these bots look for is in the hands of the search engine, and they do change from time to time. Ultimately, they exist to deliver the best possible experience to the user.

OK, now we’ve got the good bots out the way; let’s focus on their less-than-stellar cousins. Invariably, bad bots are used for criminal purposes as a way to extract money from unsuspecting individuals and companies. But the uses of bad bots go beyond cash — indeed, it’s not that much of an exaggeration to say that bad bots are among the most damaging aspects of modern life, especially since they can have significant sway in elections.

Bot IP block lists are a good start to managing bad bot behavior. You can use bot IP block lists to prevent malicious IP addresses from accessing your website. Bot IP block lists prevent millions of fraudulent IP addresses daily. Bot IP block lists may include IP addresses that have been used for phishing scams, malware, spam email, and domain hijacking. These lists can be used to manage bot activity. Bot IP block lists are not the only solution you should look into, as they are not 100% effective on their own.

Blocking IP addresses with block lists is not enough to stop bad bots. Instead of using easily identifiable data center IP addresses, bad bots often use residential IP addresses that mimic real people. Therefore, security tools that rely on IP reputation are not enough to stop malicious automation. Bots exploit residential IP addresses worldwide to mask themselves as human users.

Hacker bots do exactly what they’re called: they hack. They’re there to deceive people, install malware, attack websites, and all-around get up to no good. They find their way into people’s computers or networks by exposing security vulnerabilities and inserting a line of code into the engine. There’s nearly always a financial motive behind these bots, with the owners of compromised computers and networks having to pay to get the malicious code off.

We mentioned scraper bots in the ‘good bots’ list — but they can also feature on this list. As we said, nearly all tools can be used for good and bad! Scraper bots are designed to steal information from a website, including the content, images, design, prices, things like that. What’s the advantage of stealing this information? So that the person who has the data can create their own website, using the information they’ve stolen. This might not sound like a big deal — who would visit a copy of a site when the real thing is there? — but it has major impacts. There are some “stolen” websites that rank higher on the search engine results list than the original. Scraping prices can also be used to undercut you and destroy your margins. If you’re a business, then that can have devastating consequences.

You’ll likely have seen the work of spam bots around the internet. A spam bot inserts links into places where they don’t belong — think of, say, a response to a Facebook post. They’re usually just links to a pretty poor website that needs all the help it can get. They’re becoming less effective because they’re so easy to spot, and also, the number of spam comments is down. The spam bot was one of the more annoying aspects of the internet, and because of that, the powers to be cleaned it up — which is to say, put measures in place to limit their number. However, spam bots are still a problem within emails. Most email services have separate folders where they track and send suspicious spam emails. Businesses should be aware of all the places spam bots can lurk and impact your processes.

These types of bots don’t target specific individuals or companies. They aim to influence public discussion. They’re the ones that are responsible for posting politically motivated posts on social media channels.They can create fake accounts and cause havoc on social media networks.

When bots exploit social media networks via fake accounts, it’s not the same as a human user creating a fan page for their favorite celebrity. Impersonation bots on social media networks are designed to disclose false information. They’re put into action by political figures who wish to consolidate their power or otherwise sway public opinion.

We don’t need studies to highlight the problems that this can cause, but the studies that have been carried out show what we all know: that these bots spread disinformation on social media via users accounts and severely impinge on the democratic process.

OK, so now we’ve got an idea of the problems that bots can cause. But is this really such a problem? Just because bots can be used for nefarious purposes, that doesn’t necessarily mean that they are. So the question is: are they? The numbers suggest that, yes, bots are a serious problem that needs to be addressed. In fact, the numbers are staggering. It’s estimated that around 30% of bot traffic comes from bad bots, who are responsible for 40% of login attempts. That’s 40% of login attempts being made by users that aren’t real. And to get a sense of just how impactful these bad bots can be — they can be responsible for much of $25.6 billion in online fraud losses each year. That’s no small change.

So, what is bot traffic anyway?

Any non-human traffic to a website or web server is considered bot traffic. The main issue with bot traffic is that it can negatively impact user experience if not managed properly. Not only can users be blocked from accessing your website, they can also have sensitive information leaked if bad bots gain access. Monitoring bot traffic is crucial to prevent malicious bots from gaining control.

Bot traffic can be identified through unusually high pageviews and a high bounce rate. If a user’s session duration is extremely low or high, it just might be bot traffic masquerading as a user. Bot traffic could be detected if there is a spike in internet traffic from an unexpected location. It is important to monitor all of these to catch malicious bot traffic. Identifying bot traffic can be time-consuming for businesses. Implementing a bot management solution can help alleviate some of the work and time needed to stop bots and reduce bad bot traffic.

With so many bots contributing to bot traffic, these numbers can impact your business analytics. If bot traffic is counted as actual users, you will not know how much bot traffic you are actually receiving. It can be difficult to manage your business and track user behavior if bot traffic gets out of hand. Bot traffic can also impact your performance when large amounts of bot traffic come in the form of a DDoS attack. A DDoS attack can cause servers to be overloaded and slow for users.

Businesses should also be aware of bot traffic in the form of click fraud. While bots commiting click fraud can temporarily boost ad sales, if an advertising network detects the fraud, they can ban your website completely. If your site hosts any ads for revenue purposes, be aware of bot traffic contributing to click fraud.

So, how can you manage malicious bot traffic? There are many tools and solutions available. You will want to get a comprehensive solution that combines many tools to identify bot traffic. By doing this, you can skip through smaller steps that might not correctly identify bot traffic. One of Kasada’s integrations, Cloudflare Bot Management, uses intelligence and applies machine learning to identify bot traffic and stop bot abuse. We highly recommend our integration if you’ve noticed an uptick in bot traffic on your site.

Of course, bots — both good and bad — don’t just appear out of thin air. They’re brought to life by people and organizations that have an end goal. There’s no single type of person or group behind bots, in large part because the goals of each differ. For some bots, the goal is to extract hard-earned cash from unsuspecting victims. Others are trying to shortcut their way to success (or at least some sweet ad revenue) by stealing the content of popular websites. And in the case of political bots, the goal is more straightforward — power. Sometimes, the use of bad bots isn’t necessarily illegal, just greatly frowned upon — an example of this would be the people who use bots to snag sneakers for in-demand items and hype sales. They’re able to snap up the inventory within seconds of them going on sale and can then resell them for significantly more money.

Bad bots should be avoided at all costs, in large part because — true to their name — they’re up to no good. While many, such as all those spam comments that you’ll find on Facebook groups, are more annoying than anything, others are much more dangerous. And for businesses, things can be catastrophic. We already mentioned the ridiculously high figure from early about how much bad bots cost the business world each year. That’s money that’s flowing away from businesses and into the hands of the people behind the bad bots. Below, we’ll look at various ways in which cybercriminals use bots to do this.

ATO is another way of saying identity theft. In this scam, a cybercriminal will use bots for credential stuffing in order to gain access to the victim’s bank account, eCommerce account, or other sort of digital online account. Many users use the same credentials to log into websites, so this method allows cybercriminals to use credential stuffing across multiple websites to access sensitive data. Credential stuffing can be extremely detrimental to customers and businesses. Cybercriminals use bots to automate the credential stuffing process and increase the scale of the attack.

If a bot gets access to one of your customers’ accounts, then they’ll be able to make fraudulent purchases. Once discovered, this money may be returned to the customer, leaving the company out of pocket. Even if that doesn’t happen, the reputation of the business will usually take a hit. To prevent ATO and credential stuffing, your business should implement a bot management solution. A bot management solution can identify this form of attack and stop successful credential stuffing.

Carding involves bots trying to use the details of stolen credit cards on a website. This money from the stolen credit cards is usually returned to the original owner in the event of discovery. Worse, if it happens too regularly, then the business will suffer poor merchant history, which means they may be unable to accept credit card payments in the future. This can be devastating for merchants that process primarily credit card payments.

SQL injection is a type of web vulnerability that allows an attacker to manipulate the SQL query of a web application’s database by inserting malicious SQL code. This can lead to unauthorized access, data breaches, and other security issues.

Bots can be used to perform SQL injection attacks by exploiting vulnerabilities in the web application’s input validation process. One use case for a SQL injection bot attack would be to gain unauthorized access to sensitive data or modify database records for malicious purposes.

It takes a lot of time, effort, and money to develop an eCommerce website that’s set up for success. And so, of course, it can be more than a little disappointing to see your hard work appearing on another website. In some use cases, it can result in a significant reduction of visitors, and thus revenue. Some bots are set up to scrape the prices, product descriptions, and other information from a site, which they then place on their own site. This gives a massive boost to the new website, which gets the high-quality intellectual property for free and has a negative impact on the targeted website, since duplicated content can negatively impact SEO.

Sites that are not equipped to protect their intellectual property may suddenly find their search engine rankings tanking overnight. Although intellectual property law coincides with SEO, it cannot prevent the initial damage from being done. You must stop bots from stealing your intellectual property (and scraping your product descriptions).

E-Commerce sites only have a finite amount of stock for their products. When someone puts a product in their basket, the quantity goes down. If all the available numbers of the product were in people’s baskets, then the item would show as unavailable on the site. There are bots that add products to baskets with no intention of completing the purchase. This means that other people cannot make the purchase, resulting in loss of income for the business.

Sometimes, the bot attacks on websites are more subtle.  Bot attacks can be so subtle that they can go undetected since there’s no obvious problem, at least from the owner’s side. Application Distributed Denial of Service (DDoS) causes a surge of web traffic, which results in a slowed-down response or knocks a website offline. In addition to lost revenue due to downtime, an overload of web traffic can impact the user experience, reduce website conversion, and increase the bounce rate. Bot attacks can be sophisticated and may require advanced solutions. To combat denial of service attacks and increased web traffic, you will want to acquire a bot management solution.

The techniques we outlined above can impact businesses of all industries and sizes. But there are some industries where the challenges presented by bot attacks are more targeted. In particular, bots are more likely to target advertising businesses, financial services companies, marketplace websites, and the travel industry.

Let’s take a look at how each of these industries is targeted by bot attacks.

For advertising to work, it has to be trusted. There has to be an underlying understanding of the usefulness of the ad space between the seller of the advertisement space and the advertiser. Bots can negatively impact that agreement by generating artificial clicks, which inflates the cost of advertising. That’s a bad practice that takes money from an innocent party and hands it over to a malicious party. If an advertising company was the victim of a bot attack, then they’d need to engage in some reputation damage control to build themselves back up.

A Distributed Denial of Service (DDoS) attack occurs when a web server has so much bot traffic that legitimate users cannot access sites and services.

DDoS attacks are devastating for financial institutions and can contribute to financial losses. In fact, multi-vector DDoS attacks are on the rise and the number of successful attacks increases each year.

The financial services industry is an obvious target for bots purely because they represent the biggest cash cow. They’re a high-value target that, if it pays off, can pay off in a big way in the form of financial losses. People behind bots deploy a variety of tactics to defraud these institutions. For example, they’ll use DDoS attacks and scrape content. The sophistication of these attacks allows them to move slowly and undetected over a period of time until they’re eventually successful (if they’re not stopped in time, that is).

A DDoS attack utilizes thousands, if not millions, of bots to achieve its goal. Due to its size, businesses have a harder time working against DDoS attacks. With a bot mitigation solution in place, your business can prevent these bot attacks and reduce financial losses.

These types of websites are targeted for their content, which is their entire business model. They rely on a steady influx of fresh and unique postings in order to build up a following. Cybercriminals will scrape the website for data and publish it elsewhere, or in some cases, sell it on to another website. They can also respond to adverts with fake leads, which slow down operations and ultimately don’t lead anywhere at all.

Online travel websites get business by offering dynamic pricing — and also by keeping their prices a secret, apart from those who show a genuine interest in making a booking. Competitors can deploy bots to scrape the data of the site and, in the process, see how much they’re charging. They also make fake queries to websites, which takes up the company’s time and resources, all for queries that won’t lead anywhere.

Bot attacks happen every day and can seriously harm a company’s competitiveness and bottom line. Every now and again, bot attacks will make the news because of the scale of the attack or because of who it was targeted at.

Most of these bot attacks happen from cybercriminals (or at least, people without morals) with the aim of profit. For instance, over the past few years, famous artists such as Coldplay, Ed Sheeran, and BTS all had shows that sold out within seconds, only for the tickets to reappear on reselling websites minutes later — at a price that was up to 3000% more than the face value of the ticket.

Some bot attacks are less instantaneous. In 2018, Panera Bread found that they had inadvertently been leaking their customers’ data to hackers, all because a simple .txt line of code had been inserted into their website. Sometimes, the battle is between two companies — the low-cost airline Ryanair took Expedia to court for scraping the prices of their tickets from the Ryanair website.

The most high-profile political bot attack came from Cambridge Analytica, who were found to have scraped the data of nearly 90 million US citizens from Facebook, with the aim of ultimately influencing their behavior.

There’s more than one way to deploy a bot. Bot attacks occur over many channels. Here are the main channels cybercriminals target: APIs, mobile apps,  websites, and web apps. However, these networks rarely exist in isolation, which means if you gain access to one, you usually have access to others. Bots will typically target all three, since the more you target, the more likely it is that you’ll get access. Below, we’ll look in more detail at each of these channels and the common vulnerabilities of each.

APIs are the backbone that allows systems to talk to one another. The use of web APIs has been steadily rising over the past ten years and looks to be the future, especially since they’re such a major component behind mobile apps and IoT products.

The vulnerability lies in that APIs rely on machine-to-machine communication, and it can be difficult to determine whether a machine is the real deal or a bad actor. On a website, a bot may have to mimic a real user (say, during the signup process). With APIs, all they have to do is speak the machine’s language, and they could be in. The vulnerabilities are furthered by the use of third party companies, which many businesses use for their API needs. At that stage, the security risk becomes more spread and harder to track. A bot management solution can protect not only your business but also any third party companies you use.

Mobile apps are a recent phenomenon, and that makes them more vulnerable to bot attacks. This is based on two things. First, the conventional security measures that a website would have typically aren’t as effective on mobile apps. Second, anyone can make a mobile app, and if they aren’t as thorough with the security aspect, then there’ll be a risk. Mobile apps are designed for use, of course, on mobiles — but if a hacker can connect the app to a virtual machine (that isn’t Android or iOS), then they’ll be able to run scripts at a much greater rate and ultimately gain access to customers’ personal information, payment details, and sensitive business information.

The most common deployment of bots is on websites. They’re everywhere: in comments sections, in login attempts, in scraping. We mentioned earlier the sheer scale of bot activity on the web; the vast majority of bot attacks come through the web. They can be simple or sophisticated, but they’re everywhere — and that means that it’s something that all businesses need to be aware of.

Cross site scripting is a security vulnerability bots can use to impact web applications. User accounts can be fully taken over with cross site scripting, as it lets an attacker pretend to be the user account and perform many actions on their account. This approach can be catastrophic for users accounts.

If bots make their way into a web application, a web application firewall can be useful to stop them. It is a web traffic management tool between user accounts and web applications. A web application firewall works to filter content of a web application to prevent bot attacks. If a web application firewall detects an attack request, they will intervene. A set of rules or policies are applied to HTTP conversations. If anything falls outside those rules, a web application firewall can step in. For an extra layer of protection, you can invest in a bot mitigation solution. These solutions provide more protection than a web application firewall can.

OK, now we’ve run through all the scary scenarios that can happen as a result of bots. The question is — can you do anything about them? Thankfully, the answer is yes, you can use bot detection. The cybercriminals don’t have it all their own way. In fact, they don’t have much of anything going their own way. Every time a new bot approach is developed, it’s not long before a remedy to combat that threat is created.

However, bot detection is an attacker’s nightmare. The threats are only kept at bay if the company/website owner actively works to keep them at bay through bot detection. The first step towards bot management is bot detection. You can’t find something if you don’t know what the enemy is.

Bot detection is a tricky customer, though. For one, it’s not as if there’s just one bot that you should be looking for. Second, identifying that you might have a problem doesn’t tell you what your problem is. It just tells you that something isn’t quite right. And also, having a bot detection software application that identifies bots can be useful — but it can also be harmful. For instance, if your bot detection software applications were too defensive, then it might just block everything, including real-world, potential customers, from accessing your website.

Bot detection determines whether a request is coming from a bot or a human. A bot detection solution determines if the request is from a good or bad bot. Bot detection can prevent credential stuffing and list validation bot attacks. Complications arise with bot detection when bots replicate human behavior. This makes it more difficult for your bot detection solution to detect them. However, bot detection uses many tools to identify bots and stop them. Bot detection utilizes device fingerprinting, device hashes, firewalls, block lists, CAPTCHAs, and more.

Rather than trying to block everything with bot detection, the better, more advanced approach is to think about bot mitigation.

Bot Mitigation

So, what is bot mitigation, and how can it help in the battle against bots? Essentially, you can think about bot mitigation as a way to manage bots. And you can manage them so that they don’t have a negative impact on your online business. Below, we’ll run through some of the most effective bot mitigation approaches. Not all of them will be relevant to your online business, but you’ll likely find that you can deploy one or two.

If you can’t beat them, then you can at least call it a draw. If you identify a bot on your system, then one solution is to feed it fake data. For example, the incorrect price for your products. We call this a draw, but it’s more of a win for you since you can make the bot believe whatever it is you want it to believe. The downside is that this will take up some of your time, but not enough to make it unsatisfying!

Machine learning uses data and algorithms to imitate the human experience online. This method can uncover key insights and collect information to better serve your online business. Machine learning is good for bot mitigation because it can identify which bots are not behaving like normal users. Machine learning is often used in chatbots with users. The system can remember specific conversations and spot malicious behavior. Machine learning can also work to improve responses to customers and provide them with excellent service.

As an internet user yourself, you’ll have some experience with CAPTCHAs. It’s the box you tick that tells the website you’re a human (in some use cases, you have to pass a small test). This has been touted as an effective way to ensure all visitors to your site are human by other bot mitigation vendors. However, sophisticated bots may be able to easily and affordably bypass CAPTCHAs. If that happens, then they’ll have access to your site (unless you have other safety measures).

The nice thing about humans is that their interactions with websites are all more or less the same. They move the mouse, and they click. You can set up a system whereby you expect these actions to occur — and if they don’t, then you may have a bot on your hands, which you can then kick out. This is effective in the sense it doesn’t impact the user experience – but sophisticated bots may be able to bypass this method as well.

If bots are overusing a website or web application, rate limiting can come into play. Rate limiting blocks suspicious users and stops some bot attacks. It limits how often a user can repeat a certain action within a certain timeframe. Rate limiting can stop a bot from trying to log in to the same server over and over again. These quick, repeated actions can be prevented by rate limiting and mitigate bot activity. Rate limiting cannot manage all bot activity, so you will want another solution in place to put an end to all bad bots.

Sometimes, rate limiting falls short because some sophisticated bots can mimic human behavior, use distributed networks, or use multiple IP addresses to evade detection and bypass rate limiting mechanisms. Even so, we recommend using rate limiting in tandem with other bot mitigation tactics.

If you know what type of bot you’re trying to protect your website against, then you can simply disallow that type of bot from accessing your website. The only downside with this is that it’s not a permanent solution — a bot that gets blocked may reappear as a more sophisticated bot.

Advanced problems call for advanced solutions. It’s not realistic to expect that a company will have the in-house resources or expertise to keep all bots at bay. That’s why working with a bot management company, such as Kasada, can be highly beneficial.

Using the latest technology, a bot management company can help prevent and stop malicious automation before it has had a chance to infiltrate your system. The very best anti-bot companies will offer a full level service against all types of bots, including system takeover, ATO, denial of service, and website content scraping.

Bots are not going away anytime soon. To protect the long-term interests of your company, it’s important that you’re taking the threat that they pose seriously. It’s the companies that do that have the best chance of keeping their data and systems intact while having their online business thrive.