robots.txt file is used to provide instructions to web crawlers and bots on how to properly crawl the website.