Re: Starting build-system work (Windows/Mac users please speak up)

On Sat, May 23, 2020 at 1:18 AM Chapman Flack <chap(at)anastigmatix(dot)net> wrote:

> On 05/22/20 14:24, Kartik Ohri wrote:
>
> > #77.8 and #77.17 illustrate this issue. In #77.8, macports is unable to
> > install openssl which is installed properly in all of the remaining
> builds.
> > #77.17 fails with some include path error due to nar-maven.
>
> I have seen that in reported issues too. Inconvenient that the
> nar-maven-plugin developers wrote an IncludePath object that doesn't
> override toString(), so when it is printed in an error message it
> just comes out IncludePath(at)hex-address(dot)
>
> Don't do that in your plugin. :)

:+1:

>
>
I see Maven gives a hint there that if you run with -e it might
> give a full stack trace of the errors. I haven't tried it, but
> in a case like this I wonder if the full stack trace would include
> a lower-level exception like FileNotFoundException that would
> actually show the name that wasn't found!
>
> My suspicion is that it also is related to some dependency like
> openssl that intermittently isn't getting downloaded/installed properly.
>
>
>
> Another thing I notice that could be annoying when going through
> these logs is that the vast majority of the log is just download
> progress of all the dependencies for the build.
>
> It looks like that bulk can be reduced a lot by adding these Maven options:
>
> --batch-mode
>
> -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
>
> The first one just tones down some of the message bloat like color changes,
> and setting the TransferListener logging level to 'warn' means we should
> see
> if something goes wrong, but not fill the whole log with all the stuff
> that went right.
>
> Found here: https://stackoverflow.com/a/35653426

I will make these changes and try to reduce any other unnecessary logs as
well.

> As another thing to think about, if every build is starting from a
> clean slate, all of that stuff has to get re-downloaded every time.
> If I start from a clean slate, build PL/Java once, then look at the
> size of my local Maven repository:
>
> $ du -sh ~/.m2/repository
> 72M /var/tmp/testbuilds/.m2/repository
>
> that's 72 megs of stuff getting downloaded ... each time.
>
> If we had a way of doing multiple PG version / multiple Java version
> test builds without completely resetting the environment every time,
> that could save the Travis guys more than half a gig of bandwidth.
>
> For every set of test runs.
>
> What I do here is I have several different PG versions built and
> installed in testbuilds/pg12 testbuilds/pg11 and so on, and likewise
> for Java versions.
>
> Then if I run Maven like so:
>
> JAVA_HOME=/var/tmp/jdk-14+36/ mvn \
> -Dpljava.libjvmdefault=/var/tmp/jdk-14+36/lib/server/libjvm.so \
> -Dpgsql.pgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \
> clean install
>
> it builds for that Java version and that PG version:
>
> - the JAVA_HOME in front controls what Java version will run Maven itself
> - the -Dpljava.libjvmdefault builds in a default value of
> pljava.libjvm_location so it doesn't need to be specified later in PG
> - the -Dpgsql.pgconfig controls which PG version to build against.
>
>
> Then when it has created the pljava-packaging/target/pl*jar file, I can
> run it like this:
>
> /var/tmp/jdk-14+36/bin/java \
> -Dpgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \
> -jar pljava-packaging/target/pl*jar
>
> which makes sure to use the tested Java version to run the jar, and
> again gives the pg_config location, which determines where the files
> are installed.
>
> This doesn't all have to be in place at once, if you are focused on
> getting the basic functionality in place. But it would be a significant
> optimization later.
>
This would be worthwhile but as mentioned at
https://docs.travis-ci.com/user/build-stages/#data-persistence-between-stages-and-jobs
, state is not persisted between jobs. Every build job compulsorily starts
as a clean state. We can though cache dependencies to speed up the build
and save bandwidth though to some extent using
https://docs.travis-ci.com/user/caching/ .

> Also, I suggest (as in the build docs) using these Maven options:
>
> -Dnar.cores=1 -Pwnosign
>
> The -Pwnosign only works on platforms using gcc, so it would have to be
> left off for others. It silences a whole category of sign-conversion
> warnings that are only distracting.
>
> Setting nar.cores to 1 makes it a lot easier to read error messages if
> there are errors; otherwise, if the compilation uses a bunch of threads,
> the messages all come out on top of each other and it is hard to see
> where an error came from.
>
> Also -Psaxon-examples is needed to build a set of XML examples that are
> not built by default.
>
> I will incorporate these changes into the builds.

> Regards,
> -Chap
>
Thanks for the detailed feedback. I will work on this and get back to you
soon.