Aggregate robots.txt for Folder Mode projects

Create a robots.txt file in each folder mode project’s root, which will aggregate automatically in the default project’s public folder.

To enable this automatic aggregation, you need to set the GENERATE_AGGREGATED_ROBOTS_TXT environment variable to "true" for the container.

Setting this variable causes regeneration of the robots.txt on every container restart.

Prepend and Append the robots.txt file

It is possible to prepend and append the aggregated robots.txt file by creating a robots.txt.pre for prepend and for appending in the /projects/default/ folder.