Tag: Crawling

A run of recent posts on aifoc.us that I did not plan as a series but which turned into one. The thread is that the web a model sees is staler and thinner than the one you ship, and I have mostly been trying to find ways to see it.

ua-tracer is a tiny site that attempts to show you what a user agent fetches, follows, and executes when it loads a page. I built it because we have almost no visibility into what bots and AI agents actually see when they crawl the web.

My site's bandwidth usage spiked mid-month, and I discovered the culprit was MSN Bot repeatedly accessing my Atom.xml feed. It was hitting my feed every minute, consuming a significant portion of my bandwidth. The bot is apparently part of a syndication search engine, constantly checking for new content. I added a robots.txt file to throttle the bot's access frequency, which solved the issue.