Always publish robots.txt and include Sitemap.xml reference

Currently, robots.txt is only published if the site owner configures a custom robots.txt value in site settings.

I think a robots.txt should always be published, even if a custom one is not specified. It should also include a reference to sitemap.xml.


User-agent: *
Allow: /

  • Michael Wells
  • Aug 31 2022