robotstxt

Robots.txt is a text file that tells search engine robots which parts of a website they can and cannot crawl and index. It is important to have a robots.txt file in place to prevent search engines from crawling and indexing pages that you do not want them to, such as pages that are under construction…