+48 501 250 220

A small shop, a server at 100 percent for months, and tens of thousands of visitors an hour who were not customers

A WooCommerce shop with 52 products loaded slowly or not at all. The cause was a paid network of bots sending tens of thousands of requests an hour, each from a different home address, and every single bot behaved perfectly. Our engineers stopped the traffic at the edge of the network, and when the bots changed tactics a month later, the second rule took the load off within two minutes.

Book an intro call The web application security service
  • Shop CPU from 926 % to 16 %
  • 57 622 different addresses in one hour
  • No customer noticed anything
A field of many small dots stands for the addresses; three lanes from it end at an orange wall labelled WAF, while one lane from a laptop passes through a gap in the wall to a shop front

A small online shop for workwear on WooCommerce, 52 products, taken over from a previous provider

Challenge

According to the owner the problem had lasted for months before the shop came to us: the server permanently at 100 percent, pages that loaded very slowly, and intermittent errors because the server did not answer in time. Standard measures had been tried and had changed nothing.

One hour of logs told the story. Of 172 057 requests, 68 110 went to a single category address with filter parameters, in 68 238 different combinations of colour and size. They came from 57 622 different addresses, most of which sent between one and five requests: home internet connections in many countries, routed through a commercial network of residential proxies. The browser signature was forged, and the estimated transfer of about 90 GB a day meant someone was paying for the operation.

Solution

First the measurement: what one request of each kind costs the server. A static file 0.65 milliseconds, any request to WordPress 90 to 120 milliseconds, a filtered product list 257 milliseconds. The static files were the bulk of the requests and 0.4 percent of the cost; the filter requests were the whole load.

Then a rule at the edge of the network: the firewall in front of the shop rejects the bot traffic in its earliest phase, before a request reaches WordPress. Rejecting a request costs about a millisecond; rendering a list costs 257. We recognise the traffic by how it moves through the shop, not by where it comes from. When the bots adapted a month later, a second rule based on the navigation behaviour of a real browser was in place minutes after the diagnosis. Every change was checked against real customers: filtering, search, cart, checkout and visits from search engines work as before.

Results

Shop server
From 926 % to 16 % of CPU after the first rule
Shop server after the bots adapted
From 905 % to 1 % within two minutes of the second rule
Database server
From 123 % to 0.01 % of CPU
Filter requests reaching the shop
From 68 110 an hour to a few hundred, all from people
Requests rejected at the edge
About 90 a second
Changes customers noticed
None: filters, search, cart and search-engine visits unchanged

How we worked

  1. Measure instead of guess

    Production logs for one hour, split by request type and address, and the processor cost of every kind of request measured separately.

  2. Find the real cause

    Not the images, not the code, not the customers: 68 110 filter requests an hour from 57 622 addresses, each of them polite on its own.

  3. A rule at the edge of the network

    The firewall in front of the shop rejects the bot traffic in its earliest phase, before it reaches WordPress. The shop's CPU fell from 926 percent to 16.

  4. Thresholds from the data

    Customers and bots differ clearly in how they use the filters; the rule is set so that no customer can be hit by it.

  5. A second layer when the bots adapted

    A month later the traffic came back just under the first rule's threshold and moved to other pages. A rule based on the navigation behaviour of a real browser took the load off within two minutes.

  6. Side effects checked

    Filtering, search, cart, checkout and visits from search engines were tested after every change, and the operator keeps a written account of what the rules do.

Challenges we overcame

Most of what looked like a defence or a solution did not work in this case.

The images were not the problem, although they were most of the traffic

Most requests were for static files, so the images were the obvious suspect. They cost 2 percent of one processor core. The whole load came from requests to WordPress, and every one of them, even a 404 or a request for robots.txt, starts the whole shop with its plugins.

A cache would not have helped, and it was not working anyway

A cache plugin was installed and its configuration file claimed that everything was on. In practice it was not working: files and a setting were missing without which WordPress ignores it. More important: even a working cache would have changed nothing. A cache speeds up repeated addresses, and here almost every address was different.

Rate limits and automatic bans could not work

The firewall limits the request rate per address and bans addresses that collect error responses. Both mechanisms look at the sender. A bot sent about three requests an hour from one address, against a limit of two a second, and got a correct response every time. In the firewall's eyes it was a model visitor. No defence based on the sender's address could work here.

The address blocklist did not see the parameters

The firewall can block by address pattern. It turned out to match only the path, not the parameters after the question mark, and the bot traffic differed exactly in the parameters. Such a rule looks applied and does nothing. We found that in the firewall's source code before anyone relied on it.

The bots adapted within a month

The first rule rejected the traffic by the shape of the address. A month later the traffic was back: the same volume, almost 30 000 addresses in ten minutes, exactly below the rule's threshold, and moved from the categories to the product tag pages. Someone was watching the responses and tuning the attack. One feature is not enough for long; rules have to be hard to work around, not just catch today's traffic.

The best feature had been in the data from the start

The second rule relies on a behaviour that a real browser does not skip when a customer clicks a filter. That feature was already visible in the first hour of logs, in 67 113 of the 68 110 requests. The first time we picked the most conspicuous feature instead of the hardest one to forge. Today both layers run together.

The website banned itself

WordPress runs its scheduled tasks by calling its own public address. During the flood every bot visit triggered those tasks, more than 1 300 calls in ten minutes from the server's own address. The firewall treated that as an attack and banned the website's own server for a day. Mail and shop tasks stopped, and our tests from the server were refused, which for a moment looked like a false alarm from the new rule.

Blocking bots must not ban customers

A rejected request counts towards the automatic ban. Had the rule been too strict, a customer clicking "select all" in a filter would have been banned for a day. The thresholds came from the data: customers selected a few values, bots many more, and the two groups did not overlap.

At the top one house with three short arrows and a gauge whose needle barely moves; below a field of twelve houses, a wide arrow and a gauge whose orange needle stands at the stop

Before and after

Measurement Before After
Shop CPU, first incident 926 % (9.3 of 12 cores) 16 %
Filter requests reaching the shop 68 110 an hour About 360 an hour, from people
Shop CPU, after the bots adapted 905 % 1.1 %
Database CPU 123 % 0.01 %
Requests rejected at the edge 0 About 90 a second
Time from the second rule to the effect About 2 minutes

What we take from it

  • Measure before you touch anything. The images were most of the traffic and none of the load; the cache was "on" and did nothing; the limits were working and irrelevant.
  • Filters are an almost infinite number of addresses. With 36 colours and 11 sizes, layered navigation exposes more addresses than any cache can hold. Mark them as not to be indexed and expect them to be abused.
  • Address-based limits do not protect against residential proxy networks. Each address is polite; only the sum is an attack. The traffic has to be recognised by how it moves through the shop.
  • Reject at the edge. A request rejected by the firewall costs about a millisecond, a rendered list 257. That is why the rule runs before WordPress is started at all.
  • Plan for the adaptation. The operators of the bots read the responses. A rule has to be hard to work around, and a second layer should be ready before it is needed.

For shop owners the simple check is this: if the server is always busy and sales have not changed, find out who is really visiting the shop.

Two lanes whose length is the cost: the upper request ends at an orange wall labelled WAF after 1 ms, with an orange cross under the lane; the lower one passes a dashed wall and runs on to a shop front, labelled 257 ms

Related situations

  • Website hacked?

    Google warns visitors about your website, visitors end up on unknown sites, or the hosting provider has suspended the account? We isolate the website, close the vulnerability and clean the website within 24 hours of access, weekends included. Afterwards you receive a written report.

  • Take over a legacy system without documentation

    Your system runs, but nobody knows it in detail. We record application, database, server and email, assess the risks, recommend the next step in writing and, if you wish, take over operations.

More projects

  • A score scale with a dashed threshold: the phishing envelope moves in an orange arc across the threshold

    A spam filter that ran but did not protect

    Services up, configuration valid, no errors in the system status. Yet phishing landed in the inboxes while partners' newsletters went to spam. Our engineers measured which detection mechanisms actually fired, repaired the causes, and three months later found and fixed a regression that only showed up hour by hour.

  • Four envelopes pass the gates SPF, DKIM and DMARC, each ticked in orange, on the way into the inbox

    Email deliverability for four company domains

    A mail server for four company domains signed nothing, published no DMARC policy and was rejected by Microsoft. Our engineers audited it, piloted the change on one domain and rolled it out in batches. The result: all three checks pass at Gmail, the Microsoft block was lifted, and the users noticed nothing.

In 20 minutes we work out which first step fits your system

You tell us the goal and the current state, and we say openly whether and how we can help. Reply within one working day, no obligation.

Book an intro call How we work
Phone (German, Polish, English) +48 501 250 220
Email Email

Reachable: Monday to Friday, 9:00 to 17:00 Central European Time; support and emergencies around the clock (24/7)