Robots.txt

Summary

This glossary contains definitions of technical and marketing terms referenced across DAO.AD documentation.

Definition

A robots.txt file is used to tell crawlers which pages on a website they can and can’t crawl. This is indicated by “allow” and “disallow” directives.