To prevent undesirable information inside the look for indexes, webmasters can instruct spiders not to crawl specified data files or directories through the common robots.txt file in the foundation directory of the domain. On top of that, a site may be explicitly excluded from the internet search engine's database by https://news.rafeeg.ae