Robots.txt

Summary This glossary contains definitions of technical and marketing terms referenced across DAO.AD documentation. Definition A robots.txt file is used to tell crawlers which pages on a website they can and can’t crawl. This is indicated by “allow” and “disallow” directives.