Re: optimize file transfer in pg_upgrade

Hello Nathan,

20.03.2025 04:02, Nathan Bossart wrote:
> On Wed, Mar 19, 2025 at 04:28:23PM -0500, Nathan Bossart wrote:
> And here is yet another new version of the full patch set. I'm planning to
> commit 0001 (the new pg_upgrade transfer mode test) tomorrow so that I can
> deal with any buildfarm indigestion before committing swap mode. I did run
> the test locally for upgrades from v9.6, v13, and v17, but who knows what
> unique configurations I've failed to anticipate...

I found a couple of the 006_transfer_modes failures during the past month:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2025-04-08%2004%3A18%3A15
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2025-04-21%2008%3A03%3A06

Both happened on Windows, but what's worse is that the failure logs
contain no information on the exact reason. We can see:
#   Failed test 'pg_upgrade with transfer mode --swap: stdout matches'
#   at C:/tools/xmsys64/home/pgrunner/bf/root/HEAD/pgsql/src/bin/pg_upgrade/t/006_transfer_modes.pl line 61.
...
# Restoring database schemas in the new cluster
# *failure*
#
# Consult the last few lines of
"C:/tools/xmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/006_transfer_modes/data/t_006_transfer_modes_new_data/pgdata/pg_upgrade_output.d/20250421T081115.575/log/pg_upgrade_dump_1.log"
for
# the probable cause of the failure.
# Failure, exiting
# '
#     doesn't match '(?^:.* not supported on this platform|could not .* between old and new data directories: .*)'

there is a reference to pg_upgrade_dump_x.log, but no such files saved.

I tried to reproduce this failure locally, but failed. Still I discovered
that when the test fails, the target directory containing pgdata/ gets
removed, because of this coding:
    my $result = command_ok_or_fails_like(
...
    # If pg_upgrade was successful, check that all of our test objects reached
    # the new version.
    if ($result)
    {
...
    }

    $old->clean_node();
    $new->clean_node();

Moreover, even when pg_upgrade succeeds, IPC::Run::run inside
command_ok_or_fails_like() returns false, as we can see from a
successful test run:
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fairywren&dt=2025-04-27%2001%3A03%3A06&stg=misc-check

pgsql.build/testrun/pg_upgrade/006_transfer_modes/log/regress_log_006_transfer_modes
[01:18:38.210](21.036s) ok 1 - pg_upgrade with transfer mode --clone: stdout matches
[01:18:38.211](0.001s) ok 2 - pg_upgrade with transfer mode --clone: stderr matches

The corresponding code is:
    print("# Running: " . join(" ", @{$cmd}) . "\n");
    my $result = IPC::Run::run $cmd, '>' => \$stdout, '2>' => \$stderr;
    if (!$result)
    {
        like($stdout, $expected_stdout, "$test_name: stdout matches");
        like($stderr, $expected_stderr, "$test_name: stderr matches");
    }

So maybe it's worth to adjust the test somehow to have interesting logs
left after a failure?

Best regards,
Alexander Lakhin
Neon (https://neon.tech)