Re: Search points to ancient manuals

On Thu, Nov 1, 2012 at 12:35 PM, Marti Raudsepp <marti(at)juffo(dot)org> wrote:

> On Thu, Nov 1, 2012 at 1:13 PM, Craig Ringer <ringerc(at)ringerc(dot)id(dot)au>
> wrote:
> > I've often wished that the docs would redirect to the
> /current/interactive version when the referer is google. Not enough to
> write the code yet, though.
>
> That solution is backwards. You click on the Google link that says
> "Documentation: 8.1: EXPLAIN", but when you click on it, you magically
> end up in PostgreSQL 9.2 docs.
>
>
Yeah, I htink that's a really bad idea.

> Instead we should encourage or force Googlebot (and other search
> engines) to index/prefer the current version of docs. I hear the
> sitemaps file already gives higher priority to current, but clearly
> that's not having the effect it should. I suspect most of our
> referrers link to versioned docs, which boosts their rank, and the
> search engine has no way of telling that the individual versions are
> mostly equivalent.
>

Yeah, that's my impression as well. Once we put the sitemap into place
properly, after some time (a month or so I think) the *obsolete*
documentation links started to disappear from the google search hits
(they're not in the sitemap at all). But it seems to pay no attention
whatsoever to the priorities given in the sitemap.

> We might detect search engines by their User-Agent header and always
> serve them redirects to /current/ (if the page exists in current) or
> the latest version. But that would have the effect of de-indexing old
> versions of pages that have been revised -- which isn't an issue most
> of the time, but is not ideal either.
>

If we want to do that, we can remove the old versions from the sitemap,
which is a much easier way of doing it. However, that would remove any way
of searching for the old documentation there - and there are certainly many
usecases where doing that makes sense.

Also it would require us to serve docs pages with "Vary: User-Agent",
> which would make caching less efficient (but maybe there are
> workarounds?)
>

Not really, we could fairly easily work around that specifically for the
docs. No need to use the big gun of setting vary:user-agent on every page
(even every docs page).

However, I still think it's a non-starter since it would make it impossible
to search for old stuff.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/