Shifter automatically creates robots.txt on the root of the artefacts. This file contains some codes to tell engines crawlers.
Tell search engines crawlers that this site disallows access to
/wp-admin/(Disallow: /wp-admin/).Tell the path to the sitemap file. (supported XML Sitemap generators are WordPress default, Yoast SEO or XML Sitemaps plugins). Refer to our support docs (Yoast SEO, XML Sitemaps) or plugins' official sites.
Tell search engines crawlers to disallow access to all contents if the custom domain name is not applied to the website.
Tell search engines crawlers to disallow access to all contents if a subdirectory is enabled (
Disallow: /).
Example robots.txt for Shifter default artiafcts
User-agent: *
Disallow: /wp-admin/
Sitemap: https://{RANDOM-WORDS}.on.getshifter.io/wp-sitemap.xml
Disallow: /
Example robots.txt for custom domain name attached artiafcts
User-agent: *
Disallow: /wp-admin/
Sitemap: https://{YOUR-CUSTOM-DOMAIN-NAME}/wp-sitemap.xml