movq
Anyone got a link to a robots.txt that “blocks” all the “AI” stuff?
1 month ago
💬 Reply
tkanos https://twtxt.net/user/tkanos/twtxt.txt Remove
eaplme https://eapl.me/twtxt.txt Remove
eaplmx https://eapl.mx/twtxt.txt Remove
lyse https://lyse.isobeef.org/twtxt.txt Remove
prologic https://twtxt.net/user/prologic/twtxt.txt Remove
rrraksamam https://twtxt.net/user/rrraksamam/twtxt.txt Remove
darch https://neotxt.dk/user/darch/twtxt.txt Remove
shreyan https://twtxt.net/user/shreyan/twtxt.txt Remove
movq https://www.uninformativ.de/twtxt.txt Remove
bender https://twtxt.net/user/bender/twtxt.txt Remove
stigatle https://yarn.stigatle.no/user/stigatle/twtxt.txt Remove
darch http://darch.dk/twtxt.txt Remove
xuu https://txt.sour.is/user/xuu/twtxt.txt Remove
jason https://jasonsanta.xyz/twtxt.txt Remove
mckinley https://twtxt.net/user/mckinley/twtxt.txt Remove
eapl-mes-7-daily-links https://feeds.twtxt.net/eapl-mes-7-daily-links/twtxt.txt Remove
movq
Anyone got a link to a robots.txt that “blocks” all the “AI” stuff?
1 month ago
💬 Reply
movq
Reply to #px274va
… or maybe I should do this based on allowlisting rather than blocklisting. 🤔 Only allow a couple of bots that I think are fine …
1 month ago
💬 Reply
movq
Reply to #px274va
@prologic Ahhh, I right, now I remember. That `ai.txt` boils down to this, I guess:
User-Agent: *
Disallow: /
1 month ago
💬 Reply
movq
Reply to #px274va
@aelaraji Yeah, there is no guarantee with any of these things, it can all be faked or ignored. 🫤 I’m still going to do it in the hopes that *some* of those bots respect it.
1 month ago
💬 Reply
prologic
Reply to #px274va
@aelaraji Hmmm looks like the core idea is to intercept requests, Inspect the `UserAgent` header and respond accordingly.
1 week ago
💬 Reply
prologic
Reply to #px274va
Can we trust the bots not to fake their identity? 🤔
1 week ago
💬 Reply
movq
Reply to #px274va
@aelaraji @prologic Hmm, yeah, looks a bit better than `ai.txt` / `robots.txt`, but I wouldn’t trust that they don’t spoof their user agent. 🤔
6 days ago
💬 Reply
prologic
Reply to #px274va
@movq me neither 🤦♂️
6 days ago
💬 Reply