From: | Josh Kupershmidt <schmiddy(at)gmail(dot)com> |
---|---|
To: | "w^3" <pgsql-www(at)postgresql(dot)org> |
Subject: | robots.txt sometimes disallowing all? |
Date: | 2014-06-18 13:26:27 |
Message-ID: | CAK3UJRGeUk+hAiaMVLnWT4echkwzL=-17CpCjhpwNycT0r=7Yg@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-www |
I noticed an unusual search result shown as the top result by Google
(search query "POSTGRESQL DROP TRIGGER", first result for me leads to
www.postgresql.org/docs/8.3/static/sql-droptrigger.html ). The title
of the result is somehow "英語 - PostgreSQL", and below that title
reads: "A description for this result is not available because of this
site's robots.txt – learn more."
Sure enough, when I checked http://www.postgresql.org/robots.txt in
Chrome on OS X, I see:
User-agent: *
Disallow: /
though when I check in other browsers (Safari, wget), I see a more
reasonable robots.txt:
===
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /docs/devel/
Disallow: /list/
Disallow: /search/
Disallow: /message-id/raw/
Disallow: /message-id/flat/
Sitemap: http://www.postgresql.org/sitemap.xml
===
Is it intentional that we're serving up that first robots.txt to
(apparently) Googlebot and Chrome?
Josh
From | Date | Subject | |
---|---|---|---|
Next Message | Craig Ringer | 2014-06-19 09:16:20 | New mailing list? |
Previous Message | David Rowley | 2014-06-08 09:08:19 | Re: Link mistake on Wiki |