Post by @dillo@fosstodon.org

Dillo browser
@dillo @fosstodon.org
Public en
The Dillo git resitory displayed by stagit on Dillo itself. The page lists the last 30 commits but asks the user to fetch the git repository to see the rest.
The Dillo git resitory displayed by stagit on Dillo itself. The page lists the last 30 commits but asks the user to fetch the git repository to see the rest.

Moved our git web server from cgit to stagit to reduce load on the server caused by crawlers, as it only uses static files.

git.dillo-browser.org/dillo/

Roughly 99.9% of requests are bots fetching commits and files over and over again at a rate of 50 req/s despite our robots.txt excluding any crawler. Stagit only exposes a small number of files and commits so they are not reachable to bots.

Sadly, many anti-bot protections rely on JS or fingerprinting which come with their own set of problems.

120 likes
Refresh interactions
85 boosts
Replying to @dillo@fosstodon.org
Paul Wilde
@paul
Public en edited

@dillo@fosstodon.org at some point somebody is going to have to tell these AI/bot scrapers that 50 req/s is verging on cyber terrorism and they need to stop or be put in jail, or both

Replying to @paul
Amin, minor deity of the legume realm
@amin @polymaths.social
Public en

@paul @dillo

Any number of requests from a crawler blocked by robots.txt is seriously problematic. But yeah, 50 per second is insane.