It happened one summer in 2006.
I had just started my new job as SEO campaign manager for a well-known agency in a well known U.S. city. One of my tasks was to deal the the problem of a particular Web site’s lack of organic search performance. This site had done well for the agency because it had helped bring on board a very large paid search account which had resulted in a great deal of revenue. This site had about 100 articles and was completely search engine friendly in terms of on-site SEO and running a propriety CMS that seemed stable. Word around the office was that the site was becoming very important for the company’s organic search initiatives.
I was flummoxed. I’d done everything I knew how to make the site rank for the keywords. So, I set out to do some serious link building and link building I did. Still, no movement in the SERPS.
Then, one day as I plowed through the Google SERPs I began to notice that the actual title tags from the site were not appearing. Instead, I saw “Untitled” or simply the domain name.
Something spurred me to look in this new Google Webmaster Tools product that had been launched a while back. I did this and noticed a message stating the site files had been blocked from crawling via robots.txt.
So, immediately I checked the Robots.txt file and saw:
# go away User-agent: * Disallow: /
How could I be so dumb? Immediately I contacted IT to remove this. Apparenlty they had pushed the robots.txt file from the deve server to the live server and the robots.txt file had been included with it.