AI Bots Threaten Open Source Developers

This is a Xe Iaso news story, published by TechCrunch, that relates primarily to Git news.

Xe Iaso news

For more Xe Iaso news, you can click here:

more Xe Iaso news

Git news

For more Git news, you can click here:

more Git news

software applications news

For more software applications news, you can click here:

more software applications news

TechCrunch news

For more news from TechCrunch, you can click here:

more news from TechCrunch

About the Otherweb

Otherweb, Inc is a public benefit corporation, dedicated to improving the quality of news people consume. We are non-partisan, junk-free, and ad-free. We use artificial intelligence (AI) to remove junk from your news feed, and allow you to select the best tech news, business news, entertainment news, and much more. If you like software applications news, you might also like this article about

AI crawler bots

. We are dedicated to bringing you the highest-quality news, junk-free and ad-free, about your favorite topics. Please come every day to read the latest search engine bots news, AI crawlers news, software applications news, and other high-quality news about any topic that interests you. We are working hard to create the best news aggregator on the web, and to put you in control of your news feed - whether you choose to read the latest news through our website, our news app, or our daily newsletter - all free!

AI scraper bots

TechCrunch

•

Technology

Open source devs are fighting AI crawlers with cleverness and vengeance | TechCrunch

Summary

Nutrition label

59% Informative

Free and open source sites hosting FOSS projects share more of their infrastructure publicly.

Many AI crawler bots don't honor the Robots Exclusion Protocol robot.txt file, the tool that tells bots what not to crawl, originally created for search engine bots.

Developer Xe Iaso built a tool called Anubis to block bots but let through browsers operated by humans.

Anubis is a reverse proxy proof-of-work check that must be passed before requests are allowed to hit a Git server.

SourceHut ’s DeVault told TechCrunch that “Nepenthes has a satisfying sense of justice to it, since it feeds nonsense to the crawlers and poisons their wells, but ultimately Anubis is the solution that worked” for his site.

Cloudflare , perhaps the biggest commercial player offering several tools to fend off AI crawlers, last week released a similar tool called AI Labyrinth .

VR Score

Informative language

Neutral language

Article tone

semi-formal

Language

English

Language complexity

Offensive language

possibly offensive

Hate speech

not hateful