Bot Mitigation

UPDATE:

The majority (80%) of companies say that bad bots are becoming more sophisticated and difficult for their security tools to detect, according to our recent survey. At the same time, only 15% report that their anti-bot solution retained its effectiveness a year after initial deployment. To learn more, check out our 2021 State of Bot Mitigation to review the key findings from this first-of-its-kind survey that is exclusively from the perspective of organizations already using an anti-bot solution.

Download your copy of the 2021 State of Bot Mitigation to find out how organizations are leveraging bot mitigation solutions.

Casual users of the internet can think of it as a relatively simple process: point your browser to a website, hit enter, and hey presto, you’ve got what you came for. However, things are rarely so straightforward, and especially not when we’re talking about something as complex and sprawling as the internet. Much of the internet takes place behind closed doors, invisible to the public, invisible, even, to the people that run the websites. Many of these elements are logistical in nature and of no interest to anyone but the die-hard internet aficionados.

But there are some invisible elements that should be of interest to website owners (and everyone else). They’re called bots. They’re not all bad (in fact, many are good), but they do pose a threat that has to be managed. In this blog, we’re going to look at everything related to bots, including the different types, the threats they pose, and the importance of bot mitigation.

So to begin, let’s take a look at what bots are. If you’re a regular internet user, then you’ll likely have heard of the name before, but do you know what they are? Essentially, they’re just software applications that perform automated tasks over the internet. Well, we say that they’re “just” that — they’re pretty complicated by nature. They perform functions at a much higher speed and at a higher rate than humans can. They are robots in the shape of software, which is where the name ‘bots’ comes from.

As with any capable tool, it’s not the existence of bots that causes problems. It’s what the bot is ordered to do that causes issues. You could have a good bot that crawls the internet to find the most relevant web pages related to a search (as in the case of Google and other search engines). But similarly, you could have a bad bot that’s solely interested in being bad — these are the ones that are responsible for hacking, spamming, and all the other dangerous things we associate with nefarious internet use. Those bots are good for nobody (except for the people that are behind them).

As we said above, that bots exist isn’t an issue. In fact, the good parts of the internet wouldn’t function as it does without bots. So we shouldn’t be too critical of these pieces of software, at least not across the board. But, alas, much of the bad aspects — even just the annoying aspects — of the internet can be traced back to bots. To get a clearer idea of the extent of bots’ influence on the internet, let’s dive deeper into the good and bad kinds. We’ll start with the good, because you should always try to build someone up (before you knock them down, as we will).

One of the joys of using the internet to purchase items is that you can usually get whatever it is you’re looking for at a lower price than you’d find in the real world. But sometimes, finding those deals can be difficult. Shopping comparison websites use bots to crawl the internet and feed back the best price for whatever item is being sought. Some online retailers also use bots to show the best price to the user based on their use of the website.

Websites don’t necessarily stick around forever. If the owner of the site has decided to move onto new projects, then the website — along with all the content — will die. But sometimes, that content is worth preserving. Web scraping crawlers go around the web and record the data, which can then be accessed if the website goes offline. An example of this use would be Wayback Machine, the internet archive website. It should be noted that web scraping crawlers can be used for more sinister motives, too.

If you’ve ever used the ‘live chat’ feature of a website, then you’ll likely have been talking with a chatbot. These bots are designed to stimulate the conversation of a real human, giving responses based on the wording of the customer. Many of these bots identify themselves as non-human. They can be basic (gathering basic information before handing it over to a live agent) or sophisticated, providing a full service of information in much the same way a “real human” would.

Bots are increasingly used to get important information out into the world. They’ll push out breaking news stories via notifications, and in some instances — such as TechCrunch — bots are used to send out personalized stories to their users. Many automated Twitter handles publish relevant news as and when it’s available, without any human oversight.

Much of the chat related to bots is pretty serious, or it can feel that way. If it’s not serious, then it’s boring. But bots are also designed with just fun in mind. There are many games where you can play “against” the bot, while some bots just sit around and make art. A nice life!

And perhaps the most useful aspect of bot software: the search engine. It’s impossible to imagine what the internet would be like without a solid search engine by our sides (really: what would you do? Ask your friends if they know a site?). A big part of Google’s success has been the sophisticated nature of its crawler bots. What these bots look for is in the hands of the search engine, and they do change from time to time. Ultimately, they exist to deliver the best possible experience to the user.

OK, now we’ve got the good bots out the way; let’s focus on their less-than-stellar cousins. Invariably, bad bots are used for criminal purposes as a way to extract money from unsuspecting individuals and companies. But the uses of bad bots go beyond cash — indeed, it’s not that much of an exaggeration to say that bad bots are among the most damaging aspects of modern life, especially since they can have significant sway in elections.

Blocking IP addresses is not enough to stop bad bots. Instead of using easily identifiable data center IP addresses, bad bots often use residential IP addresses that mimic real people. Therefore, security solutions that rely on IP reputation are not enough to stop malicious automation. Bots exploit residential IP addresses worldwide to mask themselves as human users.

Hacker bots do exactly what they’re called: they hack. They’re there to deceive people, install malware, attack websites, and all-around get up to no good. They find their way into people’s computers or networks by exposing security vulnerabilities and inserting a line of code into the engine. There’s nearly always a financial motive behind these bots, with the owners of compromised computers and networks having to pay to get the malicious code off.

We mentioned scraper bots in the ‘good bots’ list — but they can also feature on this list. As we said, nearly all tools can be used for good and bad! Scraper bots are designed to steal information from a website, including the content, images, design, prices, things like that. What’s the advantage of stealing this information? So that the person who has the data can create their own website, using the information they’ve stolen. This might not sound like a big deal — who would visit a copy of a site when the real thing is there? — but it has major impacts. There are some “stolen” websites that rank higher on the search engine results list than the original. Scraping prices can also be used to undercut you and destroy your margins. If you’re a business, then that can have devastating consequences.

You’ll likely have seen the work of spambots around the internet. These insert links into places where they don’t belong — think of, say, a response to a Facebook post. They’re usually just links to a pretty poor website that needs all the help it can get. They’re becoming less effective because they’re so easy to spot, and also, the number of spam comments is down. It was one of the more annoying aspects of the internet, and because of that, the powers to be cleaned it up — which is to say, put measures in place to limit their number.

These types of bots don’t target specific individuals or companies. They aim to influence public discussion. They’re the ones that are responsible for posting politically motivated posts on social media channels.

When bots exploit social media networks, it’s not the same as a human user creating a fan page for their favorite celebrity. Impersonation bots on social media are designed to disclose false information. They’re put into action by political figures who wish to consolidate their power or otherwise sway public opinion.

We don’t need studies to highlight the problems that this can cause, but the studies that have been carried out show what we all know: that these bots spread disinformation on social media and severely impinge on the democratic process.

OK, so now we’ve got an idea of the problems that bots can cause. But is this really such a problem? Just because bots can be used for nefarious purposes, that doesn’t necessarily mean that they are. So the question is: are they? The numbers suggest that, yes, bots are a serious problem that needs to be addressed. In fact, the numbers are staggering. It’s estimated that around 30% of internet traffic comes from bad bots, who are responsible for 40% of login attempts. That’s 40% of login attempts being made by users that aren’t real. And to get a sense of just how impactful these bad bots can be — they can be responsible for much of $25.6 billion in online fraud losses each year. That’s no small change.

Of course, bots — both good and bad — don’t just appear out of thin air. They’re brought to life by people and organizations that have an end goal. There’s no single type of person or group behind bots, in large part because the goals of each differ. For some bots, the goal is to extract hard-earned cash from unsuspecting victims. Others are trying to shortcut their way to success (or at least some sweet ad revenue) by stealing the content of other websites. And in the case of political bots, the goal is more straightforward — power. Sometimes, the use of bad bots isn’t necessarily illegal, just greatly frowned upon — an example of this would be the people who use bots to snag sneakers for in-demand items and hype sales. They’re able to snap up the inventory within seconds of them going on sale and can then resell them for significantly more money.

Bad bots should be avoided at all costs, in large part because — true to their name — they’re up to no good. While many, such as all those spam comments that you’ll find on Facebook groups, are more annoying than anything, others are much more dangerous. And for businesses, things can be catastrophic. We already mentioned the ridiculously high figure from early about how much bad bots cost the business world each year. That’s money that’s flowing away from businesses and into the hands of the people behind the bad bots. Below, we’ll look at various ways in which cybercriminals use bots to do this.

Account Takeover is another way of saying identity theft. In this scam, a cybercriminal will use bots to stuff stolen credentials in order to gain access to the victim’s bank account, eCommerce account, or other sort of digital online account. If a bot gets access to one of your customers’ accounts, then they’ll be able to make fraudulent purchases. Once discovered, this money may be returned to the customer, leaving the company out of pocket. Even if that doesn’t happen, the reputation of the business will usually take a hit.

Similar to account takeover, carding involves bots trying to use the details of stolen credit cards on a website. As with an account takeover, this money is usually returned to the original owner in the event of discovery. Worse, if it happens too regularly, then the business will suffer poor merchant history, which means they may be unable to accept credit card payments in the future. This can be devastating for merchants that process primarily credit card payments.

It takes a lot of time, effort, and money to develop an eCommerce website that’s set up for success. And so, of course, it can be more than a little disappointing to see your hard work appearing on another website. In some cases, it can result in a significant reduction of visitors, and thus revenue. Some bots are set up to scrape the prices, product descriptions, and other information from a site, which they then place on their own site. This gives a massive boost to the new website, which gets the high-quality intellectual property for free and has a negative impact on the targeted website, since duplicated content can negatively impact SEO.

Sites that are not equipped to protect their intellectual property may suddenly find their search engine rankings tanking overnight. Although intellectual property law coincides with SEO, it cannot prevent the initial damage from being done.

E-Commerce sites only have a finite amount of stock for their products. When someone puts a product in their basket, the quantity goes down. If all the available numbers of the product were in people’s baskets, then the item would show as unavailable on the site. There are bots that add products to baskets with no intention of completing the purchase. This means that other people cannot make the purchase, resulting in loss of income for the business.

Sometimes, the bot attacks on websites are more subtle. Indeed, they can be so subtle that they can go undetected since there’s no obvious problem, at least from the owner’s side. Application DDoS causes a surge of traffic to a website, which results in a slowed-down response or knocks a website offline. In addition to lost revenue due to downtime, this can, impact the user experience and reduce website conversion rates.

The techniques we outlined above can impact businesses of all industries and sizes. But there are some industries where the challenges presented by bots are more targeted. In particular, bots are more likely to target advertising businesses, financial services companies, marketplace websites, and the travel industry.

Let’s take a look at how each of these industries is targeted.

For advertising to work, it has to be trusted. There has to be an underlying understanding of the usefulness of the ad space between the seller of the advertisement space and the advertiser. Bots can negatively impact that agreement by generating artificial clicks, which inflates the cost of advertising. That’s a bad practice that takes money from an innocent party and hands it over to a malicious party. If an advertising company was the victim of a bot attack, then they’d need to engage in some reputation damage control to build themselves back up.

DDoS attacks are devastating for financial institutions. In fact, multi-vector DDoS attacks are on the rise and the number of successful attacks increases each year.

The financial services industry is an obvious target for bots purely because they represent the biggest cash cow. They’re a high-value target that, if it pays off, can pay off in a big way. People behind bots deploy a variety of tactics to defraud these institutions. For example, they’ll attempt account takeover, use DDoS attacks, and scrape content. The sophistication of these attacks allows them to move slowly and undetected over a period of time until they’re eventually successful (if they’re not stopped in time, that is).

These types of websites are targeted for their content, which is their entire business model. They rely on a steady influx of fresh and unique postings in order to build up a following. Cybercriminals will scrape the website for data and publish it elsewhere, or in some cases, sell it on to another website. They can also respond to adverts with fake leads, which slow down operations and ultimately don’t lead anywhere at all.

Online travel websites get business by offering dynamic pricing — and also by keeping their prices a secret, apart from those who show a genuine interest in making a booking. Competitors can deploy bots to scrape the data of the site and, in the process, see how much they’re charging. They also make fake queries to websites, which takes up the company’s time and resources, all for queries that won’t lead anywhere.

Bot attacks happen every day and can seriously harm a company’s competitiveness and bottom line. Every now and again, a bot attack will make the news because of the scale of the attack or because of who it was targeted at.

Most of these attacks happen from cybercriminals (or at least, people without morals) with the aim of profit. For instance, over the past few years, famous artists such as Coldplay, Ed Sheeran, and BTS all had shows that sold out within seconds, only for the tickets to reappear on reselling websites minutes later — at a price that was up to 3000% more than the face value of the ticket.

Some attacks are less instantaneous. In 2018, Panera Bread found that they had inadvertently been leaking their customers’ data to hackers, all because a simple .txt line of code had been inserted into their website. Sometimes, the battle is between two companies — the low-cost airline Ryanair took Expedia to court for scraping the prices of their tickets from the Ryanair website.

The most high-profile political bot attack came from Cambridge Analytica, who were found to have scraped the data of nearly 90 million US citizens from Facebook, with the aim of ultimately influencing their behavior.

There’s more than one way to deploy a bot. There are main channels that cybercriminals target: APIs, Mobile Apps, and Websites. However, these networks rarely exist in isolation, which means if you gain access to one, you usually have access to others. Bots will typically target all three, since the more you target, the more likely it is that you’ll get access. Below, we’ll look in more detail at each of these channels and the common vulnerabilities of each.

APIs are the backbone that allows systems to talk to one another. The use of web APIs has been steadily rising over the past ten years and looks to be the future, especially since they’re such a major component behind mobile apps and IoT products.

The vulnerability lies in that APIs rely on machine-to-machine communication, and it can be difficult to determine whether a machine is the real deal or a bad actor. On a website, a bot may have to mimic a real user (say, during the signup process). With APIs, all they have to do is speak the machine’s language, and they could be in. The vulnerabilities are furthered by the use of third-party companies, which many businesses use for their API needs. At that stage, the security risk becomes more spread and harder to track.

Mobile apps are a recent phenomenon, and that makes them more vulnerable to attacks. This is based on two things. First, the conventional security measures that a website would have typically aren’t as effective on mobile apps. Second, anyone can make a mobile app, and if they aren’t as thorough with the security aspect, then there’ll be a risk. Mobile apps are designed for use, of course, on mobiles — but if a hacker can connect the app to a virtual machine (that isn’t Android or iOS), then they’ll be able to run scripts at a much greater rate and ultimately gain access to customers’ personal information, payment details, and sensitive business information.

The most common deployment of bots is on websites. They’re everywhere: in comments sections, in login attempts, in scraping. We mentioned earlier the sheer scale of bot activity on the web; the vast majority of those attacks come through the web. They can be simple or sophisticated, but they’re everywhere — and that means that it’s something that all businesses need to be aware of.

OK, now we’ve run through all the scary scenarios that can happen as a result of bots. The question is — can you do anything about them? Thankfully, the answer is yes, you can. The cybercriminals don’t have it all their own way. In fact, they don’t have much of anything going their own way. Every time a new bot approach is developed, it’s not long before a remedy to combat that threat is created.

However — and this is crucial — those threats are only kept at bay if the company/website owner actively works to keep them at bay. The first step towards bot management is bot detection. You can’t find something if you don’t know what the enemy is.

Bot detection is a tricky customer, though. For one, it’s not as if there’s just one bot that you should be looking for. Second, identifying that you might have a problem doesn’t tell you what your problem is. It just tells you that something isn’t quite right. And also, having software that identifies bots can be useful — but it can also be harmful. For instance, if your bot software was too defensive, then it might just block everything, including real-world, potential customers, from accessing your website. Rather than trying to block everything, the better, more advanced approach is to think about bot mitigation.

Bot Mitigation

So what is bot mitigation, and how can it help in the battle against bots? Essentially, you can think about bot mitigation as a way to manage bots. And you can manage them so that they don’t have a negative impact on your business. Below, we’ll run through some of the most effective bot mitigation approaches. Not all of them will be relevant to your business, but you’ll likely find that you can deploy one or two.

If you can’t beat them, then you can at least call it a draw. If you identify a bot on your system, then one solution is to feed it fake data. For example, the incorrect price for your products. We call this a draw, but it’s more of a win for you since you can make the bot believe whatever it is you want it to believe. The downside is that this will take up some of your time, but not enough to make it unsatisfying!

As an internet user yourself, you’ll have some experience with CAPTCHAs. It’s the box you tick that tells the website you’re a human (in some cases, you have to pass a small test). This has been touted as an effective way to ensure all visitors to your site are human by other bot mitigation vendors. However, sophisticated bots may be able to easily and affordably bypass CAPTCHAs. If that happens, then they’ll have access to your site (unless you have other safety measures).

The nice thing about humans is that their interactions with websites are all more or less the same. They move the mouse, and they click. You can set up a system whereby you expect these actions to occur — and if they don’t, then you may have a bot on your hands, which you can then kick out. This is effective in the sense it doesn’t impact the user experience – but sophisticated bots may be able to bypass this method as well.

If you know what type of bot you’re trying to protect your website against, then you can simply disallow that type of bot from accessing your website. The only downside with this is that it’s not a permanent solution — a bot that gets blocked may reappear as a more sophisticated bot.

Advanced problems call for advanced solutions. It’s not realistic to expect that a company will have the in-house resources or expertise to keep all bots at bay. That’s why working with a bot management company, such as Kasada, can be highly beneficial. Using the latest technology, a bot management company can help prevent and stop malicious automation before it has had a chance to infiltrate your system. The very best anti-bot companies will offer a full level service against all types of bots, including system takeover, account takeover, denial of service, and content scraping.

Bots are not going away anytime soon. To protect the long-term interests of your company, it’s important that you’re taking the threat that they pose seriously. It’s the companies that do that have the best chance of keeping their data and systems intact while having their online business thrive.