Shifter automatically creates robots.txt on the root of the artefacts. This file contains some codes to tell engines crawlers.

  • Tell search engines crawlers that this site disallows access to /wp-admin/ (Disallow: /wp-admin/ ).

  • Tell the path to the sitemap file. (supported XML Sitemap generators are WordPress default, Yoast SEO or XML Sitemaps plugins). Refer to our support docs (Yoast SEO, XML Sitemaps) or plugins' official sites.

  • Tell search engines crawlers to disallow access to all contents if the custom domain name is not applied to the website.

  • Tell search engines crawlers to disallow access to all contents if a subdirectory is enabled (Disallow: /).

Example robots.txt for Shifter default artiafcts

User-agent: * 
Disallow: /wp-admin/
Sitemap: https://{RANDOM-WORDS}.on.getshifter.io/wp-sitemap.xml
Disallow: /

Example robots.txt placed for default artifact with subdirectory applied

User-agent: * 
Disallow: /{SUBDIR}/wp-admin/
Allow: /{SUBDIR}/wp-admin/admin-ajax.php
Disallow: /

Example robots.txt for custom domain name attached artiafcts

User-agent: * 
Disallow: /wp-admin/
Sitemap: https://{YOUR-CUSTOM-DOMAIN-NAME}/wp-sitemap.xml

Example robots.txt for custom domain name (www.example.com) applied artifact with subdirectory applied

User-agent: * 
Disallow: /{SUBDIR}/wp-admin/
Allow: /{SUBDIR}/wp-admin/admin-ajax.php
Did this answer your question?