Why does your protection from bots prevent the Labrika crawler from scanning the site? How to solve this problem?
“Bad” bots represent one of the most serious threats to websites today. Malicious bot traffic can lead to a decrease in website performance, distortion of information about the availability of goods and services on online platforms, violation of the confidentiality of personal data and, as a result, to an outflow of traffic and loss of profit.
Bots attack websites, causing a number of problems for businesses, such as account hijacking, DDoS attacks, API misuse, web scraping, spam creation, analytics distortion, and fraud.
The ability to distinguish between good and bad bots is a priority for modern companies. However, according to the Radware Application Security Report—Web Application Security Report—79% of organizations cannot accurately distinguish between the traffic of useful and malicious bots.
Bot traffic and the threat of their destructive impact on business are rapidly increasing. This means that specialized solutions to protect against bot attacks are fundamentally important for businesses. The situation has become even more complicated with the emergence of a new generation of malicious bots that can mimic the behavior of a real user and easily bypass traditional full-time information security and protection systems against bots.
Using the Labrika http_info utility, you can check whether the error with the site scan is the result of the triggered protection of the site from bots. To do this, you need to put a check mark on the display of the content and see the response code of the page. Usually, if there are inscriptions like “protect” in the response code, we can conclude that the triggered protection of the site is the cause of the error.
If your site has installed protection against bots, then in this case you will need to do the following:
- You need to add Labrika's IP addresses (18.104.22.168, 22.214.171.124) to the white list.
- After that, in the site settings in Labrika, in the “Site crawl settings” section, check the “Use only basic IP addresses” item.
How add ip to white list:
Whitelist an IP Address
By default, Sucuri Firewall will restrict access to your admin pages (e.g.
/admin) so that only authorized IP’s can log in. That way, if your user accounts are ever compromised, your site will still be safe.
To authorize an IP to log in to your site, you will need to whitelist them in your dashboard. If you are getting the following error when trying to access your site, you need to whitelist your own IP.
Block ID: IPB17
Block reason: Your request was not allowed due to IP blocking (not white listed).
Whitelisting IPs using the dashboard
Whitelisting IPs is very simple and there are two ways of doing this.
- Click here and enter your current IP address under the Whitelist IP Addresses section.
- Click on Whitelist to whitelist your IP address once you have typed it into the box.
Whitelisting IPs using the API
- Click here to go to your API settings and then click on the Whitelist IP green button in the Quick Links section.
That will automatically whitelist your IP address. You can bookmark that link and share with other administrators of your site to allow them access. Any time they get blocked, all they need to do is visit that link to allow them back in.
Whitelisting IP Ranges
Unless it is absolutely necessary, this isn’t recommended due to the security risks involved with whitelisting too many IP addresses.
You can whitelist subnets by adding IP addresses using the CIDR (Classless Inter-Domain Routing) format.
Let’s say you want to whitelist all IP addresses from
126.96.36.199, then you just need to whitelist the
188.8.131.52/24 range. If you want to whitelist all IP addresses from
184.108.40.206, you would need to whitelist
At this time, you can only whitelist /24, /16 or /8 ranges.
Disabling admin panel restriction
- Click the link here.
- Under the Advanced Security Options uncheck the Admin panel restricted to only Whitelisted IP addresses option and click on Proceed.
This will turn off this option and allow anyone to access the administrator login pages.
Step 1 – Find your IP
The quickest and easiest way to discover your public IP address is to type into Google.com “what is my IP address“. You’ll see it at the top. Highlight and copy this address. It will look like 220.127.116.11 or similar.
Copy your IP address for later.
Step 2 – Open WordPress admin
Navigate to yourdomain.com/wp-admin – and login. If you are currently locked out, you may need to add your email address and go through the WordFence recovery process (open email, click link) first.
Step 3 – Navigate to WordFence > Firewall > All Firewall Options
Hover over WordFence in the left hand menu, click on Firewall from the screen which pops out.
Then on the screen which opens, click on “All Firewall Options” (there’s a gear icon next to this).
In the screen which follows, the top option should be a box to input “Whitelisted IP addresses that bypass all rules” – add your copied IP address (from above) on a new line in this box, and hit “Save Changes” in the blue box at the top right.
That should be it!
3. iThemes Security
From your WordPress dashboard, you need to go to your iThemes Security Settings > Global Settings > Configure Settings as demonstrated in the below image.
You then will be taken to the next page, where you will scroll down to "Lockout Whitelist" section to enter the IPs need to be whitelisted for ContentStudio.
Once done, click on "save settings" to complete the process.
Log in to your Cloudflare account
- Click the Firewall icon and then on the Tools tab
- List ContentKing’s crawl IP addresses under the IP Access Rules:
a) Enter the IP address
Whitelistas the action
c) Choose the website to which the whitelisting rules will apply
- Repeat with all of our crawl IP addresses
5. Brute-Force Firewall
Log into WHM as the 'root' user.
- Navigate to "Home » Security Center » cPHulk Brute Force Protection."
- Open the "Whitelist Management" tab.
- Enter the IP address or IP address range to be whitelisted in the "New Whitelist Records" textbox.
- Click the "Add" button.
To whitelist IP addresses with MalCare, follow these steps: Install MalCare and go to the Security and Firewall tab. Select the Traffic Requests section, and find the IP you want to whitelist. Hover over the edit icon, and click on 'Add to whitelist'.