Robots.txt Not Found
If the warning you received says that the robot.txt file cannot be found on your site and therefore the whole site will be scanned in an uncontrolled manner, you should prepare a file by paying attention to the following details:
- Your
robot.txt
file must be located in the root directory of your site. - The URL of the
robot.txt
file must be in the same format as your website URL. - While preparing the
robots.txt
file, you should complete your work by adhering to theUTF-8
character encoding discipline.
Checklist for solving robots.txt not found problem
Did you pay attention to the warnings above? After preparing your file, you can make a check according to the following items:
- Your
robots.txt
file should be plain text. - There should be no characters other than
UTF-8 encoding
system in the file. - You can add dynamic or variable content to the file. Also, the file itself is dynamic and can be changed at any time.
- Keep the file running at all times. Hiding or removing the file may result in unplanned results in the crawling process of your website.
How to determine the URL of the Robots.txt file?
We said that the URL of the robots.txt file
should be in the same format as your site address. So what exactly does this mean? Here is a small example of this:
- Site URL: https://www.yourwebpageaddress.com/
- Robots.txt URL: https://www.yourwebpageaddress.com/robots.txt
Test Your Website Issues
You can quickly analyze your site