To avoid unwanted content material while in the lookup indexes, website owners can instruct spiders to not crawl certain documents or directories through the typical robots.txt file in the root directory in the domain. In addition, a web site may be explicitly excluded from a online search engine's database through https://news.rafeeg.ae