If the warning you received says that the robot.txt file cannot be found on your site and therefore the whole site will be scanned in an uncontrolled manner, you should prepare a file by paying attention to the following details:
robot.txtfile must be located in the root directory of your site.
- The URL of the
robot.txtfile must be in the same format as your website URL.
- While preparing the
robots.txtfile, you should complete your work by adhering to the
UTF-8character encoding discipline.
Checklist for solving robots.txt not found problem
Did you pay attention to the warnings above? After preparing your file, you can make a check according to the following items:
robots.txtfile should be plain text.
- There should be no characters other than
UTF-8 encodingsystem in the file.
- You can add dynamic or variable content to the file. Also, the file itself is dynamic and can be changed at any time.
- Keep the file running at all times. Hiding or removing the file may result in unplanned results in the crawling process of your website.
How to determine the URL of the Robots.txt file?
We said that the URL of the
robots.txt file should be in the same format as your site address. So what exactly does this mean? Here is a small example of this:
- Site URL: https://www.yourwebpageaddress.com/
- Robots.txt URL: https://www.yourwebpageaddress.com/robots.txt
Test Your Website Issues
You can quickly analyze your site