Robots.txt Not Found

If the warning you received says that the robot.txt file cannot be found on your site and therefore the whole site will be scanned in an uncontrolled manner, you should prepare a file by paying attention to the following details:

1. Your robot.txt file must be located in the root directory of your site.
2. The URL of the robot.txt file must be in the same format as your website URL.
3. While preparing the robots.txt file, you should complete your work by adhering to the UTF-8 character encoding discipline.

Checklist for solving robots.txt not found problem

Did you pay attention to the warnings above? After preparing your file, you can make a check according to the following items:

1. Your robots.txt file should be plain text.
2. There should be no characters other than UTF-8 encoding system in the file.
3. You can add dynamic or variable content to the file. Also, the file itself is dynamic and can be changed at any time.
4. Keep the file running at all times. Hiding or removing the file may result in unplanned results in the crawling process of your website.

How to determine the URL of the Robots.txt file? 

We said that the URL of the robots.txt file should be in the same format as your site address. So what exactly does this mean? Here is a small example of this:

1. Site URL: https://www.yourwebpageaddress.com/
2. Robots.txt URL: https://www.yourwebpageaddress.com/robots.txt