Nearly 20% of all websites you’ll want to scrape use Cloudflare’s anti-bot protection. While it’s highly efficient and challenging to bypass, there are still techniques to get around it. We got our best developers on the job to show you the five best methods to bypass Cloudflare.
First, let’s see more about how this bot detection works!
What Is Cloudflare and How It Works
Cloudflare is a content delivery network and internet security company that protects websites against unwanted bot traffic. It uses advanced machine learning algorithms to distinguish between human and automated requests and block the latter to avoid spamming, DDoS attacks, and similar threats. Unfortunately, web scrapers are casualties in this war.
Cloudflare detects bots through a combination of active and passive techniques. Here are some examples:
- Botnet detection: Cloudflare collects information about devices, IPs, and behavioral patterns associated with bot activity and keeps it in a catalog for real-time reference.
- IP address reputation analysis: Your IP reputation is based on several factors, including ISP, online behavior history, and geolocation. Cloudflare uses that to determine the trustworthiness of your IP.
- HTTP request headers analysis: The lack of a User Agent or the use of a non-browser one will quickly raise Cloudflare’s suspicion.
- CAPTCHAs: These challenges aim to distinguish human from bot traffic. They’re becoming increasingly difficult to bypass, so it’s best to avoid triggering them in the first place.
- Canvas fingerprinting: Every device has a web client class based on its browser, operating system, and graphics hardware. Cloudflare has a large database of canvas fingerprints to distinguish actual users from bots.
- Event tracking: Humans interact with a site much differently than bots. Cloudflare uses event listeners to track actions like mouse movements and keystrokes to detect deviations from the expected behavioral pattern.
- JavaScript challenges: Botnet’s requests lack the required JavaScript stack to pass the challenge, thus resulting in Cloudflare blocking them.
Overall, Cloudflare uses these and other methods to collect sensor data and detect inconsistencies on the server side to block bots like your scraper. Now, let’s see what you can do about that!
How to Bypass Cloudflare
Let’s dive into the five best methods to bypass Cloudflare.
Use an API to Bypass Cloudflare
Developing and maintaining your own solver is a lot of work. But fortunately, there’s an effective solution: ZenRows.
ZenRows is a web scraping API capable of bypassing Cloudflare’s protective measures. It can take care of all that stands in your way, so you don’t need to worry about detection techniques, dynamic obfuscation, or challenge solving.
It comes with premium features like rotating residential proxies, geo-targeting, and WAF bypass, and it integrates seamlessly with any programming language.
Use Cloudflare Solvers
You’ve probably seen libraries that claim they can bypass Cloudflare’s challenge. In reality, most of them won’t do much good, as they’re out-of-date or not actively maintained.
However, there are still some relatively reliable options like FlareSolverr that use headless Selenium with Undetected ChromeDriver to avoid detection. The downsides are that this tool uses a lot of memory, is difficult to scale, and may fail against advanced anti-bot techniques.
Avoid CAPTCHAs
When dealing with CAPTCHAs, you have two options, solve or avoid them. If you choose the former, you can use services like 2Captcha, which employs real people to solve the tests manually. However, that will end up being quite expensive.
Alternatively, avoiding CAPTCHAs is easier and cheaper, but some of the best-protected sites present these challenges to every visitor. In that case, you’ll need to build a Cloudflare CAPTCHA bypass using the mentioned solver services or save time and resources by preventing it from appearing.
However, if you analyze your target carefully, you may find out it only uses maximum security measures at certain times or days. So only give up when you’ve exhausted your options.
Get Around Cloudflare CDN
In a nutshell, Cloudflare can’t block you if your request doesn’t go to its server but directly to the origin server. Pretty neat! Unfortunately, it’s only possible in some instances, so you’ll have to go through a trial-and-error process here.
First, you’ll need to find the origin IP. That won’t be easy, as Cloudflare hides the DNS records of its protected websites. That’s why you should check unprotected subdomains, mailing, or old services. Alternatively, you can visit databases like Shodan or use tools like CloudFlair.
Once you have the IP, you’ll need to find a way to request the data. Pasting it on your browser’s URL bar won’t always work, so instead, you’ll need programmatic tools like cURL or Python Requests. It’s a good solution, but it won’t work every time, so let’s see what else is on the table!
Bypass the Waiting Room and Reverse Engineer the Challenge
Every time you visit a Cloudflare-protected website, you’re placed in the waiting room, so your browser can solve challenges to prove you’re human. Depending on its success, you’ll either be redirected to the page you want to visit or get the “Access denied” screen and the option to solve a CAPTCHA challenge.
The way to bypass this is to analyze the JavaScript challenge to understand the algorithm behind it. That will allow you to reverse-engineer the script. Definitely, not easy to do but worth the effort. Let’s see if there’s a more straightforward technique.
Conclusion
As you can see, Cloudflare lives up to its name. Bypassing its bot detection measures requires much time, effort, and other resources. And yet, it’s possible. We discussed the best methods to go about it, as well as their downsides and limitations.
Overall, using a web scraping API like ZenRows is the safest option, as it handles most of the work on its own with advanced features to avoid Cloudflare’s suspicion.