Sitemaps, Robots.txt and Canonical Tags

An XML sitemap is a file that lists the important pages on a site, helping search engines find and prioritize them, especially useful for large sites or pages with few internal links pointing to them.

A robots.txt file tells search engine bots which parts of a site they should not crawl, such as admin pages or duplicate content used internally. It’s a set of instructions, not a security measure, since the file itself is publicly visible.

A canonical tag tells search engines which version of a page is the "main" one when multiple URLs show very similar or identical content, for example a product page accessible through several different URL parameters. Without it, search engines may split ranking signals across duplicates or pick the wrong version to show.