Shifter automatically creates a robots.txt file in the root directory of your project. This file contains code that instructs search engine crawlers.
The Role of robots.txt
The robots.txt file is used to instruct search engine crawlers on which parts of a website they should not crawl.
Features of Shifter’s robots.txt
The robots.txt file automatically generated by Shifter provides the following features:
- Denying access to /wp-admin/ (e.g., Disallow: /wp-admin/)
- Specifying the path to the sitemap file (Supports XML sitemap generators such as WordPress’s default, Yoast SEO, and the XML Sitemaps plugin)
About Sitemaps
For more details, please refer to the support documentation below or the official websites of each plugin.
- Yoast SEO
- XML Sitemaps
Conditions for Generating the robots.txt File
- If a custom domain name is not in use: It instructs search engine crawlers to deny access to all content.
- When a subdirectory is enabled: tells search engine crawlers to deny access to all content (e.g., Disallow: /).
Example robots.txt
Example robots.txt for Shifter’s default artifact:
User-agent: *
Disallow: /wp-admin/
Sitemap: https://{RANDOM-WORDS}.on.getshifter.io/wp-sitemap.xml
Disallow: /Example robots.txt for an artifact with a custom domain applied
User-agent: *
Disallow: /wp-admin/
Sitemap: https://{YOUR-CUSTOM-DOMAIN-NAME}/wp-sitemap.xml