Simple repo browsers for the website (PoC)

From: Dave Page <dpage(at)pgadmin(dot)org>
To: PostgreSQL WWW <pgsql-www(at)lists(dot)postgresql(dot)org>
Subject: Simple repo browsers for the website (PoC)
Date: 2021-05-27 13:49:10
Message-ID: CA+OCxoxyXqwaVUbwtEJ_nHbRV3jPiujt1CFeTpG8rLF+PEvnTg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-www

I've been working on a simple browser for the apt/yum/zypp repos to add to
the website, for which a PoC is attached.

There has been past discussion of a similar patch from Christoph Berg (
https://www.postgresql.org/message-id/flat/CA%2BOCxoxK1KR7cyn_wn-VwVfYOvpUXhooNC0VAuOysPp2uoM8%3DQ%40mail.gmail.com#de0886d11ab262a5dddfcffb94eaa051)
however, I believe that patch is a) way more complex than we'd want on the
main website, both to maintain and for the user experience for the vast
majority of visitors, and b) it only supports apt, not yum and zypp.

This patch implements a pretty simple browser, allowing users to search,
list and view details of all the packages in any of the repos. A separate
script (which would run on ftpmaster) would scan the repositories
periodically, and load the data into the database behind the website.

Current limitations:

- Search is pretty basic, allowing substring matching on the package name
and selection of the repository (e.g. "fedora-31 testing"). That can be
enhanced to allow drill-down selections (distro, arch, repo etc), but that
would really require some JS scripting which whilst easy enough, we
typically try to avoid if there's no pure-HTML alternative.

- I can't figure out a sane way to get the licence for .deb packages. That
seems to be the one thing that's not in the repo metadata, and would
require reading the package itself (and there are thousands of them, so...).

- Loading the data is currently a transactional delete/reload process.
That's somewhat icky, but doing a better job requires figuring out what
packages have been deleted from the repos which I suspect may be more
expensive in the long run.

I've also attached a patch that will sync the required repo meta data from
the PostgreSQL servers for local testing.

The loader script (scan-repos.py) needs the psycopg2 and repomd packages.

I've also attached a couple of screenshots.

Comments/suggestions/objections please!

--
Dave Page
Blog: https://pgsnake.blogspot.com
Twitter: @pgsnake

EDB: https://www.enterprisedb.com

Attachment Content-Type Size
repo_browsers.patch application/octet-stream 22.7 KB
sync-repos.sh application/x-sh 483 bytes
scan-repos.py text/x-python-script 11.8 KB
Screenshot 2021-05-27 at 14.47.18.png image/png 419.1 KB
Screenshot 2021-05-27 at 14.46.20.png image/png 389.7 KB

Responses

Browse pgsql-www by date

  From Date Subject
Next Message Ray O'Donnell 2021-05-27 17:21:43 Re: Simple repo browsers for the website (PoC)
Previous Message Rocio Avila 2021-05-27 05:58:59 Wiki editor request