Re: Seeking Insights on Roll-out Process for Multiple PyPI Packages

Hello,

On Fri, 2 Jun 2023, 16:25 Jan-Hendrik Lorenz, <
jan-hendrik(dot)lorenz(at)tngtech(dot)com> wrote:

>
> 1) How did the Psycopg team handle deploying and distributing two
> PyPI packages from a single source repository?
>

With one line of sed:
https://github.com/psycopg/psycopg2/blob/8b17e218bea6dbaf43f27d6aa97bafb33e2fb66f/scripts/build/wheel_linux_before_all.sh#L48

The package name is replaced in the setup.py, and that's enough to create a
different distribution package from the same Python module.

2) Were there any challenges encountered during the roll-out
> process, and if so, how were they addressed?
>

The biggest problem is faced with dependency tracking, because pypi
metadata don't actually consider the possibility of different distribution
package providing the same python package. Both psycopg2 and
psycopg2-binary can be installed in the same environment, but they will
trample on each other's files, with undefined results. Libraries shouldn't
depend on -binary, but of course many either don't read the indication or
they consider themselves superior and do it otherwise, causing problems
when two libraries both depending on psycopg2 are installed alongside.

In Psycopg 3 we are doing something similar, but we solved large part of
the problems. There is a pure python package defining the interface, which
can be used as dependency target, and optional speedup. The psycopg_c
speedup requires to be built on the client, so it's similar to psycopg2;
the psycopg_binary speedup is precompiled and distributed as wheel together
with the library dependencies, so it's similar to psycopg2-binary. The two
distribution packages install two distinct python packages, so they can be
installed alongside. See
https://www.psycopg.org/psycopg3/docs/basic/install.html

In this case too, both packages are created from the same source code;
modifications are a bit more than sed'ing a single file but also not
difficult: see the script
https://github.com/psycopg/psycopg/blob/master/tools/build/copy_to_binary.py

3) Are there any recommended tools, scripts, or configuration
> adjustments for effectively managing the two PyPI packages?
>

Feel free to use the scripts and techniques above for your own use. I
strongly advise against distributing the same python package in different
distribution packages and suggest you to follow the approach of psycopg 3.

Best

-- Daniele