1 Billion Requests: The Attack on Pxl

Last weekend, Pxl was attacked through a series of high-traffic requests. We are still not entirely sure whether this was a targeted DDoS attack on some of the links of a specific Pxl team or whether it was actual traffic.

In any case, what we do know is that the mind-blowing number of 1 billion requests was managed by Pxl’s infrastructure in just a couple of hours, with downstream effects only on (i) our server bill and (ii) my ability to sleep.

The attack

Of course, it happened in summer when we were least expecting it.

Of course, it happened over the weekend.

Of course, it happened when I was offline for just two days, because I attended a small event in the Hungarian woods.

(Spoiler: I’m never offline.)

Luckily – or should I say expectedly? – the majority of incoming requests were blocked by measures we put in place earlier, but by the nature of a link shortener like Pxl, we need to process requests we think are real visits.

In total, over 8M visits were generated in about 5 minutes. For context, Pxl would typically process about 1,000~2,000 requests across all teams in such a timeframe.

The cherry on top was the monthly server bill of 2,466 USD. And not only that, most other services started sending alerts for increased billings, too.

While still out in the woods, I grabbed my phone and started realizing what was happening. We immediately connected over team chat and began investigating to find the cause of this unusual behavior.

The most severe attack occurred on Sunday, with some minor incidents in the following days.

By then, however, we had already implemented all measures to mitigate this and any future attacks.

First thing Monday morning, I contacted support to explain the situation and ask (= beg) a bill reduction.

Just one day later, they lowered the bill back to Pxl's usual monthly amount after we confirmed that we had taken precautions to be prepared if such an attack happens again.

What we learned

However, what I'm most happy about is something else: Pxl's infrastructure handled everything extremely well.

Some parts were only briefly offline when we had to refine the firewall rules, but other than that, the hwole infrastructure held steady like an oak tree in a storm... of requests.

There is just no way to be fully prepared for this when you are running a service whose very job is to process and redirect requests, but for the next time, we have measures in place. 🛡️