A robots.txt is a text file that has to be placed on your server to ask different search engine spiders not to index or crawl some pages or sections of your site. We can use it in order to prevent ind...