Robots.txt File, Sitemap.xml and the XML schema

What the hell are these bots or robots?   The robots exclusion protocol or simply robots.txt, is a standard used by most websites to communicate with web crawlers and other web…

read more