Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard - The current setup on linkblogs is that robots.txt only allows crawlers on custom domains, the idea is that it's easier to backup your linkblog to sites like archive.orgventurebeat.com #
2019/07/02 #
-
-
For example here is the robots.txt for on my linkblog custom domainlinks.markjgsmith.com #
-
Domain overseer lifts all price caps on .Org domain namesdomainnamewire.com #
-
In the early hours this morning I deployed the latest version of linkblog to the production servers, so far today everything seems to be working - There are a lot of structural changes that make the underlying platform stronger, and the load balancing is setup in a way that makes SSL on custom domains possible, the other big change is that it's possible to signup for a 2 week trial before committing to a yearly subscription - This last part of the build has been really quite difficult, hopefully the vampires will stop sucking my blood during the night now, either way there will be linkblogen.wikipedia.org #