# Robots.txt

##### Summary

This glossary contains definitions of technical and marketing terms referenced across DAO.AD documentation.

##### Definition

<span style="color: rgb(86, 86, 106);">A robots.txt file is used to tell crawlers which pages on a website they can and can’t crawl. This is indicated by “allow” and “disallow” directives.</span>