Follow

I find it interesting how, when I describe what robots.txt with User-agent: *; Disallow: / does, most non-technical users on flounder.online want to set it

· · Web · 2 · 0 · 3

This is the exact language I use:

"You can add a special file named robots.txt to your page. This will inform web-crawlers, for example, search engines, how to treat your page. It will not block them from crawling your page, but well behaved crawlers will respect it. For example, if you don't want your web page archived by Google, Internet Archive, or anyone else, add the following to your robots.txt:

User-agent: *
Disallow: /

For more info, see:

robotstxt.org/"

@aw just nice to know that the option to turn off the all-seeing eyes is there
Sign in to participate in the conversation
Merveilles

Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them.