Folks use DA to view which Sites are dependable in their industry and to figure out where by for getting very good links. txt file is then parsed and may instruct the robotic as to which web pages usually are not to generally be crawled. Like a online search engine https://www.youtube.com/watch?v=MxDLv74CqK0