Sitemaps and Robots
Search engines need to understand which URLs exist and which areas should or should not be crawled. Next.js supports special conventions for generating sitemap and robots resources.
A sitemap should contain canonical URLs that you want indexed. Robots rules should not be treated as an access-control mechanism because they are instructions to crawlers rather than authentication.
Key Takeaways
- Sitemaps help search engines discover important URLs.
- Robots rules do not protect private resources.
- Authentication must protect private content.