Dignity nobility excellent state highness standing class caste standing superiority precedence vogue distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robotic as to which webpages are not to get crawled. As a internet search engine crawler could continue https://www.youtube.com/watch?v=CWAXJPBjGjE