Shop

Bots, crawlers, scripts, and scrapers: at NICAR you can learn to build them, but can you defend your site from them? Whether they're built by data journalists, search engines, web archivists or malicious troublemakers, bots are substantial users of news websites. At The Texas Tribune, bots account for half of our site's traffic, and they're responsible for our search presence and our archival legacy as well as site attacks and performance problems. In this slideshow, we'll dive into the world of bot users, and share some tips for identifying and managing these crawlers and scrapers, helping the "good" bots do their work, and keeping the "bad" ones from wreaking havoc.