Re: once more: documentation search indexing

Hi,

On 6/12/21 4:29 PM, Andres Freund wrote:
> Hi,
>
> in a recent twitter discussion [1] $subject again has been brought
> up. Unsurprisingly - it's still awful.
>
> It's been brought up many times before:
> - https://www.postgresql.org/message-id/CA%2BOCxoyVwmmZkWUJCez2hCqa89iGv%3Dvq58NF1yQkTg9gtpkn%3Dg%40mail.gmail.com
> - https://www.postgresql.org/message-id/CAHyXU0wu7w%3DOpeHtvpei4J9SAr7TTmdRJOyCWF6MRXpQcFNHGw%40mail.gmail.com
> - https://www.postgresql.org/message-id/CANNMO%2B%2BkxJmaaB7X6hq_8SqcEruySZrF%3DUkcPm-EG1JCKVascw%40mail.gmail.com
> - https://www.postgresql.org/message-id/38c68b83-30ae-c039-acd0-9e853997edc4@2ndquadrant.com
> - https://www.postgresql.org/message-id/560614CA.1080304@mail.com
> - ...
>
> One issue around the topic is that we seem to get bogged down in finding
> a perfect solution to how to present "versioned document" to google,
> preventing us from making small incremental adjustments. Since it seems
> unlikely that we'll get a perfect solution anytime soon (we'd have found
> it already), I'd like to try to see if we can find a way to agree on
> some incremental steps.

Thank you for bringing this up I applaud the suggestion of approach. I
will note that we have taken incremental steps however the years to
improve this, some of which I will site below.

However, part of the issue with the "incremental steps" is ensuring that
we're following things according to the best upstream guidance.

The one element that I know that is cited as "this should work" is
setting canonical references, which I'll touch on below.

> Suggested small steps:
>
> - add a docs/current link to https://www.postgresql.org/docs/. Often
> enough that's what a user wants anyway, and it's not useful to add
> additional steps for users and search engines to navigate to
> docs/current/.

We do that at the very top: that is the first link in the main body.
This was done back in Nov 2020[1]

> I can see us either making it a separate row in the versioned table,
> or to split the most recent released version's link into a /current/
> and $major link.

I'm not sure if that's any different than the above right now; if there
is something you could cite around that, I'm happy to be convinced
otherwise.

However, I'm also not opposed to putting a (Current) link next to the
current version in the table. I think that'd at least be helpful from a
user perspective, if they don't click the big button up top.

> - put version in page titles where it makes sense. E.g. change
> "PostgreSQL: Documentation: 10: 6.1. Inserting Data" to
> "PostgreSQL 10 Documentation: 6.1. Inserting Data"
>
> The current ordering doesn't seem like it has much going for it, and
> it can't help search engines to have the version number people might
> search for removed from the product name.
>
> Right now this seem to contribute to less than helpful titles in
> search engine results. Searching anonymously for "postgres alter
> table" I get the less than helpful "Documentation: 12: ALTER TABLE -
> PostgreSQL" on google.
>
> It might also be worth to go a bit further and put the documentation
> version *after* the page title, given that it's most likely already
> clear to the reader that this is about postgres. I.e. something like
> "ALTER TABLE - Documentation for PostgreSQL 14"

I think having "PostgreSQL $MAJOR_VERSION" together would help both for
some of the indexing issues + readability in the search engine. The
question is around how the content is ordered. in the title.

Doing "PostgreSQL $MAJOR_VERSION: Documentation: $page_title" might be
the way to go. The other thing I see done for SEO what you suggest, but
just hyphenated i.e. "ALTER TABLE - Documentation - PostgreSQL 14"

Anyway, I'm generally in favor for combining at least "PostgreSQL
$MAJOR_VERSION."

> - Consider removing chapter numbers from page titles. I'd argue that the
> particular chapter number for content isn't interesting as the title. E.g.
> https://www.postgresql.org/docs/12/plpgsql-declarations.html#PLPGSQL-DECLARATION-PARAMETERS
> has a title of "PostgreSQL: Documentation: 12: 42.3. Declarations"

We'd likely need to do something to indicate what chapter you're in
(e.g. PL/pgSQL) given there could be multiple "Declarations" sections in
the docs.

> (see also previous item). The 42.3 piece seems pointless in a title of
> a website - although the actual chapter name could be helpful, because
> it's not immediately obvious that the page refers to plpgsql.
>
>
> - Add a meta description - even just including what we have for the
> og:description thing seems like it would often be better what google
> is kind of forced to make up?

Not necessarily opposed, but I'd like to freshen up on some of the
modern SEO practices. I think that may help with search engine display,
but not move the needle on the indexing.

That all said, as stated and cited in some of those previous threads, I
think the biggest lift is around making our documentation URLs
canonical. After discussing with Magnus a bit, there are a few things
that we need to consider in it:

1. Whether or not the documentation page is in "current"
2. If it's not in "current", which is the last version the page is a
part of? We make that the canonical

I've attached a patch that does this. The one part I'm not sure I like
is how we treat something that is solely in "devel" -- knowing that
eventually something in devel could end up in current. Perhaps if
something is only in "devel", we exclude it from being part of the
canonical tree?

Thanks,

Jonathan

[1] https://git.postgresql.org/gitweb/?p=pgweb.git;a=commitdiff;h=24a48d2037