Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard - The current setup on linkblogs is that robots.txt only allows crawlers on custom domains, the idea is that it's easier to backup your linkblog to sites like archive.orgventurebeat.com #
If you like or find value in my writings, please make a donation. Several easy options & you choose the amount. Check out the donation page.
For enquiries about my consulting, development, training and writing services, aswell as sponsorship opportunities contact me directly via email. More details about me here.