Google preparing to make robots.txt an Internet standard after 25 years
Google today announced that it has posted a Request for Comments to the Internet Engineering Task Force to formalize the Robots Exclusion Protocol specification after it being an informal 25-year-old…