Re: Packaging automation, testing and work-reduction

On 08/20/2014 02:01 AM, Jeff Frost wrote:
> This all sounds exciting, but since Devrim is out, let's table the discussion till he gets back (or at least gets near a keyboard).

Devrim, any thoughts re the following?

I'm particularly interested in adopting a unified specfile for the
postgresql package (for starters), then adopting mock for builds. With
Koji and packages in git to follow.

> On Aug 19, 2014, at 1:45 AM, Craig Ringer <craig(at)2ndQuadrant(dot)com> wrote:
>
>> Hi all
>>
>> I'm interested in getting more involved in the RPM packaging process for
>> PostgreSQL. I know I'm new to the scene, so the following might be
>> jumping the gun a bit, but I'm also aware that it's hard to find time
>> for PGDG RPM work and more contribution might be valued.
>>
>> I've produced new spec files for PostgreSQL 9.4 that unify all of
>> EL-5/6/7 and Fedora 19/20/21 into a single spec that can be rebuilt for
>> each platform, and I think that's ready for wider testing now. Builds
>> for all platforms work correctly using 'mock' on a Fedora 20 build-host,
>> and they test-install into the target platform correctly, and
>> interoperate with existing PGDG RPMs.
>>
>> I've fixed a number of packaging bugs in the process. I'll attach the
>> spec and the rest of the RPM sources in a followup mail to the "unifying
>> the spec files" thread.
>>
>> I'm looking at future steps now, and I'd like to outline some ideas that
>> might make PGDG RPMs easier to maintain, more friendly toward
>> contribution from others, more reliable, easier to test, and more
>> inter-operable with other repositories.
>>
>> My personal motivation - just so I'm upfront about this - is that what
>> I'm talking about would make it a *lot* easier for me to produce
>> "hotfix" RPMs for 2ndQuadrant customers who want fixes backported before
>> a new minor release, produce backport RPMs where they want something
>> backported into 9.2 or whatever, produce RPMs for customers with
>> perf/bug fixes that haven't been accepted by upstream yet, and make
>> packaging BDR a lot easier too.
>>
>> From my understanding of how things currently work I think it'd also
>> make it easier to maintain PGDG, but of course I don't have much history
>> here or the experience with the current process to really back that.
>>
>>
>> I'd like to (short version):
>>
>> * Depend on EPEL
>> (some packages already do, it's just not declared)
>> * Drop packages from PGDG that're already in EPEL
>> * Unify spec files for different dists
>> * Build with Mock
>> * Use Koji
>> * Test with Mock
>> * Let Koji handle createrepo etc
>> * Move package management to git
>>
>>
>> Details of what I'm suggesting:
>>
>> - Depend on EPEL for RHEL/CentOS/SciLinux and remove all packages
>> provided by EPEL from PGDG to reduce duplication and potential for
>> conflicts. (This might involve applying for EPEL dev status
>> eventually, but shouldn't to start with).
>>
>> As many packages would work w/o EPEL I doubt it'd have to be a hard
>> dependency. Just say that EPEL is recommended and will be required
>> for some packages.
>>
>>
>> - Progressively unify spec files for other packages to remove the need
>> for individual specs for each target OS, like I've already done
>> for postgresql94. (Done for Pg 9.4, not prior, but see next item).
>>
>>
>> - For PostgreSQL packages also make the major versions supported in the
>> same define, just by changing the macros. So packaging bug fixes are
>> easier. (I haven't done this, but it doesn't look too hard).
>>
>>
>> - Always build packages using mock, so issues with build-depends and
>> undeclared runtime dependencies like those I've recently reported are
>> identified at build time. (This works with the packages I've
>> rewritten).
>>
>> Use mock target definition files that include EPEL repositories for
>> EL targets.
>>
>>
>> - Test-install packages in another mock chroot after building, and
>> run some basic tests - initdb, etc, at least. Trivial to automate,
>> just run a script within the mock chroot. (I've done part of this
>> and it's not hard to finish off).
>>
>>
>> - Automate package building with mock via Koji by setting up a Koji
>> server for PGDG. This is fairly well documented, and really just
>> needs a host with a pile of disk space, preferably running a new
>> Fedora release. Builds can then be submitted to Koji, either as SRPMs
>> or via pushes to git. The latter involves tarballs in git, but
>> that's what Fedora does.
>>
>> Koji can pull the packages from Fedora and EPEL to use as a base
>> for builds, but (using Mock) will only install and enable those
>> declared for dependencies or build-depends in any individual build.
>>
>> I haven't set a Koji up, but I'd very much like to. If PGDG
>> wants to go this way I'll be happy to do it for PGDG, rather than
>> creating a private Koji.
>>
>> Check out:
>>
>> http://koji.fedoraproject.org/koji/
>> https://fedorahosted.org/koji/
>>
>> 'mash' is used to manage koji -> yum repo work.
>>
>>
>> - Let Koji take care of running createrepo, repoview, etc as updates
>> are pushed, rather than running the manually;
>>
>>
>> - Test repositories with repoclosure and complain if we find
>> dependencies that can't be satisfied, e.g.
>>
>> repoclosure -l fedora -l pgdg94
>>
>> or
>>
>> repoclosure -l el5 -l epel5 -l pgdg91
>>
>> (with a corresponding yum configuration).
>>
>> This can be done within 'mock', or with individual yum config files
>> rather than using the system ones.
>>
>>
>>
>> Thoughts?
>>
>> It probably makes sense to fire up a Koji instance first. Then start
>> progressively getting packages building with corrected build-depends and
>> depends, using mock, and where possible unified to build the same spec
>> on all dists, moving them into Koji-friendly git management as that happens.
>>
>> As packages are updated, the copy in svn would want to be updated to
>> match so there's no drift, marking the spec in git as the authoritative one.
>>
>> Once everything's building happily in Koji its repos could become the
>> authorative ones.
>>
>> Crazy talk?
>> --
>> Craig Ringer http://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Training & Services
>>
>>
>> --
>> Sent via pgsql-pkg-yum mailing list (pgsql-pkg-yum(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-pkg-yum
>
> ---
> Jeff Frost <jeff(at)pgexperts(dot)com>
> CTO, PostgreSQL Experts, Inc.
> Phone: 1-888-PG-EXPRT x506
> FAX: 415-762-5122
> http://www.pgexperts.com/
>
>
>
>
>

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services