Hooray, Google says Goodbye to the Supplemental Index (or does it?)

On 19th December 2007 Google Webmaster Central announced "The Ultimate Fate of Supplemental Results". But before all you webmasters rush out and let all the church bells ring in relief, all may not be as it seems.

Lets put this announcement in the context of the recent history of the dreaded "Google Hell".

Many people are unaware that until very recently Google had two indexes: the main index, used to deliver the main search results, and a Supplemental Index containing pages that did not meet its “quality guidelines”.

Pages in the Supplemental Index were only delivered in the search results where no other pages were relevant, which meant, to all intents and purposes, never, except for virtually unique exact phrase matches.

Historically the Supplemental Index used to contain only a rag tag of pages which probably quite rightly did not belong in the main index, however in early 2006 Google introduced a major infrastructure update (“Big Daddy”) which dramatically lifted the quality threshold for the Supplemental Index, resulting in many millions of pages being consigned to Google Hell, as well as many others being liquidated altogether.

This caused a huge furore with site owners and SEO firms alike.  Google’s public explanations that these changes were to make “quality improvements” was somewhat undermined by their CEO having to disclose to a US Stock Market committee, whilst explaining the need to yet more investment in capacity, that “Google was full”.

Page Rank was one of the most influential factors in determining whether pages would be dropped into the SI, and two types of site were most affected by this update: e-commerce sites with large numbers of product pages where  Page Rank was spread too thinly over too many pages, and small company and personal sites without only few significant inbound links, and consequently little PageRank to be distributed in the first place.

As far as small “corporate sites” are concerned the position improved slightly through the end of 2006 and early 2007 as Google adjusted its algorithm to re-include many of these sites, but for many e-commerce sites and smaller sites, many pages have been continuing to languish in Hell.

In mid 2007 Google announced that it was implementing technical changes to “improve the freshness” of its main index and incorporate more results from the Supplemental Index in its main results, although this was not obviously reflected in changes in search engine rankings. At the same time, on the grounds that because they were incorporating more results from the SI into the main results, they dropped the "Supplemental Result" tag from search results which of course made it a lot more difficult for site owners to spot it they had a problem with pages still consigned to the SI. Which of course they did, its just that Google didn't want them to know that they did.

So now comes the Google announcement of the end of the Supplemental Index.  So all those pages formerly in the SI have now been restored to the main index? Well of course, no.

This announcement was made in conjunction with a major update of the main index which (coincidentally?) resulted in very many fewer pages being reported as returned for any given search, and checks that we have carried out on client sites for pages that were in the SI now shows them not to be in the Google index at all.  In other words, Google has disposed of much of the Supplemental Index by the simple expedient of dropping out many of the pages previously dumped into it. Naughty Google putting a positive spin on a dastardly trick.

OK, maybe not that dastardly, because this sort of thing has happened in the past, and over time, the dumped pages are recovered and restored to the index, but as one of Google's drivers in all of this is to restrict its data capacity requirements, maybe it might be starting to be a bit more choosy about the "quality" of pages it adds to the index.

Only time will tell, so hold off on those church bells just yet.