To enable our crawlers to access the website you want to monitor, you may need to whitelist the User Agent (UA) depending on your Cloudflare settings.
We recommend whitelisting Hexometer User Agent to prevent blocking your IP or any other unwanted situation while monitoring websites.
This will allow the crawlers to access the site and prevent false incident alerts from being generated by Cloudflare. It is important to take this step to ensure proper archiving of the site.
Below, we will demonstrate the steps to whitelist the Hexowatch User Agent in Cloudflare.
Step 1: Log in to the Cloudflare account and select the Security tab from the left menu.
Step 2: Next, select WAF. It will show the Web Application Firewall security settings.
Step 3: Then, navigate to the Firewall rules tab.
Step 4: Click on the Create firewall Rule.
Step 5: The next step is giving a name to the Firewall rule.
Step 6: Set Field to User Agent, Contains as the Operator, and Value to Hexometer
Or
Set Field to User Agent, Equals as the Operator, and Value to:
Mozilla/5.0 (X11; Linux i686; rv:78.0) Gecko/20100101 Firefox/84.0 | Hexometer Then, turn on the Allow.
Step 7: As a final step, click Deploy in the bottom-right corner after completing the form.
Marketing Specialist | Content Writer
Experienced in SaaS content writing, helps customers to automate time-consuming tasks and solve complex scraping cases with step-by-step tutorials and in depth-articles.
Follow me on Linkedin for more SaaS content