InitControlFile misbehaving on graviton

Bernd and I have been chasing a bug that happens when all of the
following conditions are fulfilled:

* PG 15..18 (older PGs are ok)
* gcc 14.2 on Debian unstable/testing (older Debians and Ubuntus are ok)
* arm64 running on graviton (AWS EC2 c8g.2xlarge, ok on different arm64 host)
* -O2 (ok with -O0)
* --with-openssl (ok without openssl)
* using no -m flag, or using -marm8.4-a (using `-march=armv9-a` fixes it)

The problem happens early during initdb:

$ ./configure --with-openssl --enable-debug
...
$ /usr/local/pgsql/bin/initdb -D broken --no-clean
...
running bootstrap script ... 2025-01-13 18:02:44.484 UTC [523300] FATAL: control file contains invalid database cluster state
child process exited with exit code 1
initdb: data directory "broken" not removed at user's request

Looking at the control file, we can see that the cluster state is
"starting up":

$ /usr/local/pgsql/bin/pg_controldata broken/
pg_control version number: 1700
Catalog version number: 202501101
Database system identifier: 7459462110308027428
Database cluster state: starting up
pg_control last modified: Mon 13 Jan 2025 06:02:44 PM UTC
Latest checkpoint location: 0/1000028
Latest checkpoint's REDO location: 0/1000028
Latest checkpoint's REDO WAL file: 000000010000000000000001
Latest checkpoint's TimeLineID: 1
Latest checkpoint's PrevTimeLineID: 1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID: 0:3

The relevant code is in BootStrapXLOG():

/* Now create pg_control */
InitControlFile(sysidentifier, data_checksum_version);
ControlFile->time = checkPoint.time;
ControlFile->checkPoint = checkPoint.redo;
ControlFile->checkPointCopy = checkPoint;

/* some additional ControlFile fields are set in WriteControlFile() */
WriteControlFile();

and InitControlFile():

if (!pg_strong_random(mock_auth_nonce, MOCK_AUTH_NONCE_LEN))
ereport(PANIC,
(errcode(ERRCODE_INTERNAL_ERROR),
errmsg("could not generate secret authorization token")));

memset(ControlFile, 0, sizeof(ControlFileData));
/* Initialize pg_control status fields */
ControlFile->system_identifier = sysidentifier;
memcpy(ControlFile->mock_authentication_nonce, mock_auth_nonce, MOCK_AUTH_NONCE_LEN);
ControlFile->state = DB_SHUTDOWNED;

So the state should actually be DB_SHUTDOWNED (1), but on this system,
the value is DB_STARTUP (0).

Stepping through InitControlFile we can see that ControlFile->state is
never written to. (The trace jumps back to InitControlFile a lot
because it seems inlined into BootStrapXLOG):

$ cd broken
$ rm -f global/pg_control; PGDATA=$PWD gdb /usr/lib/postgresql/17/bin/postgres
Reading symbols from /usr/local/pgsql/bin/postgres...
(gdb) b InitControlFile
Breakpoint 1 at 0x1819bc: file xlog.c, line 4214.
(gdb) r --boot -F -c log_checkpoints=false -X 16777216 -k
Starting program: /usr/local/pgsql/bin/postgres --boot -F -c log_checkpoints=false -X 16777216 -k

Breakpoint 1, 0x0000aaaaaac219bc in InitControlFile (sysidentifier=<optimized out>, data_checksum_version=<optimized out>) at xlog.c:4214
4214 if (!pg_strong_random(mock_auth_nonce, MOCK_AUTH_NONCE_LEN))
(gdb) s
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4214
4214 if (!pg_strong_random(mock_auth_nonce, MOCK_AUTH_NONCE_LEN))
(gdb)
pg_strong_random (buf=buf(at)entry=0xfffffffff670, len=len(at)entry=32) at pg_strong_random.c:79
79 for (i = 0; i < NUM_RAND_POLL_RETRIES; i++)
(gdb)
81 if (RAND_status() == 1)
(gdb)
87 RAND_poll();
(gdb)
90 if (RAND_bytes(buf, len) == 1)
(gdb)
InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4219
4219 memset(ControlFile, 0, sizeof(ControlFileData));
(gdb)
4221 ControlFile->system_identifier = sysidentifier;
(gdb)
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
0x0000aaaaaac21a08 in InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4221
4221 ControlFile->system_identifier = sysidentifier;
(gdb)
4222 memcpy(ControlFile->mock_authentication_nonce, mock_auth_nonce, MOCK_AUTH_NONCE_LEN);
(gdb)
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
0x0000aaaaaac21a3c in InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4234
4234 ControlFile->track_commit_timestamp = track_commit_timestamp;
(gdb)
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
0x0000aaaaaac21a4c in InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4232
4232 ControlFile->wal_level = wal_level;
(gdb)
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
0x0000aaaaaac21a6c in InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=1) at xlog.c:4233
4233 ControlFile->wal_log_hints = wal_log_hints;
(gdb)
5175 InitControlFile(sysidentifier, data_checksum_version);
(gdb)
0x0000aaaaaac21a8c in InitControlFile (sysidentifier=7459466832287685723, data_checksum_version=<optimized out>) at xlog.c:4233
4233 ControlFile->wal_log_hints = wal_log_hints;
(gdb)
BootStrapXLOG (data_checksum_version=data_checksum_version(at)entry=1) at xlog.c:5181
5181 WriteControlFile();

(gdb) p *ControlFile
$1 = {system_identifier = 7459466832287685723, pg_control_version = 1700, catalog_version_no = 202501101, state = DB_STARTUP, time = 1736792463,
checkPoint = 16777256, checkPointCopy = {redo = 16777256, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, wal_level = 1, nextXid = {
value = 3}, nextOid = 10000, nextMulti = 1, nextMultiOffset = 0, oldestXid = 3, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1,
time = 1736792463, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 0}, unloggedLSN = 1000, minRecoveryPoint = 0,
minRecoveryPointTLI = 0, backupStartPoint = 0, backupEndPoint = 0, backupEndRequired = false, wal_level = 1, wal_log_hints = false, MaxConnections = 100,
max_worker_processes = 8, max_wal_senders = 10, max_prepared_xacts = 0, max_locks_per_xact = 64, track_commit_timestamp = false, maxAlign = 8,
floatFormat = 1234567, blcksz = 8192, relseg_size = 131072, xlog_blcksz = 8192, xlog_seg_size = 16777216, nameDataLen = 64, indexMaxKeys = 32,
toast_max_chunk_size = 1996, loblksize = 2048, float8ByVal = true, data_checksum_version = 1,
mock_authentication_nonce = "*\307\177t\215\362\344 \326\307I\374\005f7v(at)\242ə\265\230\273#+\301\t\212\204\377\004A", crc = 4294967295}

Disassembling at the breakpoint:

(gdb) disassemble
Dump of assembler code for function BootStrapXLOG:
0x0000aaaaaac21708 <+0>: stp x29, x30, [sp, #-272]!
0x0000aaaaaac2170c <+4>: mov w1, #0x0 // #0
0x0000aaaaaac21710 <+8>: mov x29, sp
...
=> 0x0000aaaaaac219bc <+692>: add x19, sp, #0x90
0x0000aaaaaac219c0 <+696>: mov x0, x19
0x0000aaaaaac219c4 <+700>: mov x1, #0x20 // #32
0x0000aaaaaac219c8 <+704>: str w2, [x21, #28]
0x0000aaaaaac219cc <+708>: bl 0xaaaaab0ac824 <pg_strong_random>
0x0000aaaaaac219d0 <+712>: tbz w0, #0, 0xaaaaaac21b28 <BootStrapXLOG+1056>
0x0000aaaaaac219d4 <+716>: ldr x3, [x22, #32]
0x0000aaaaaac219d8 <+720>: mov x2, #0x128 // #296
0x0000aaaaaac219dc <+724>: mov w1, #0x0 // #0
0x0000aaaaaac219e0 <+728>: mov x0, x3
0x0000aaaaaac219e4 <+732>: bl 0xaaaaaab7f3b0 <memset(at)plt>
0x0000aaaaaac219e8 <+736>: mov x3, x0
0x0000aaaaaac219ec <+740>: mov x1, #0x3e8 // #1000
0x0000aaaaaac219f0 <+744>: ldr w9, [x21, #32]
0x0000aaaaaac219f4 <+748>: adrp x7, 0xaaaaab3ce000 <fmgr_builtins+72112>
0x0000aaaaaac219f8 <+752>: ldr x7, [x7, #3720]
0x0000aaaaaac219fc <+756>: str x1, [x3, #128]
0x0000aaaaaac21a00 <+760>: ldr w1, [sp, #120]
0x0000aaaaaac21a04 <+764>: add x0, x0, #0x28
0x0000aaaaaac21a08 <+768>: str x23, [x3]
0x0000aaaaaac21a0c <+772>: str w1, [x3, #252]
0x0000aaaaaac21a10 <+776>: adrp x6, 0xaaaaab3cf000
0x0000aaaaaac21a14 <+780>: ldr x6, [x6, #2392]
0x0000aaaaaac21a18 <+784>: adrp x5, 0xaaaaab3cf000
0x0000aaaaaac21a1c <+788>: ldr x5, [x5, #2960]
0x0000aaaaaac21a20 <+792>: adrp x4, 0xaaaaab3cf000
0x0000aaaaaac21a24 <+796>: ldr x4, [x4, #3352]
0x0000aaaaaac21a28 <+800>: ldp q26, q25, [x19]
0x0000aaaaaac21a2c <+804>: str s15, [x3, #16]
0x0000aaaaaac21a30 <+808>: adrp x2, 0xaaaaab3cf000
0x0000aaaaaac21a34 <+812>: ldr x2, [x2, #2552]
0x0000aaaaaac21a38 <+816>: str x26, [x3, #24]
0x0000aaaaaac21a3c <+820>: adrp x1, 0xaaaaab3ce000 <fmgr_builtins+72112>
0x0000aaaaaac21a40 <+824>: ldr x1, [x1, #3320]
0x0000aaaaaac21a44 <+828>: ldp q27, q29, [x20]
0x0000aaaaaac21a48 <+832>: str x25, [x3, #32]
0x0000aaaaaac21a4c <+836>: str w9, [x3, #172]
0x0000aaaaaac21a50 <+840>: ldr w4, [x4]
0x0000aaaaaac21a54 <+844>: ldr w5, [x5]
0x0000aaaaaac21a58 <+848>: ldr w6, [x6]
0x0000aaaaaac21a5c <+852>: ldr w7, [x7]
0x0000aaaaaac21a60 <+856>: ldr q30, [x20, #64]
0x0000aaaaaac21a64 <+860>: ldp q28, q31, [x20, #32]
0x0000aaaaaac21a68 <+864>: stur q27, [x3, #40]
0x0000aaaaaac21a6c <+868>: ldrb w8, [x22, #240]
0x0000aaaaaac21a70 <+872>: ldr w2, [x2]
0x0000aaaaaac21a74 <+876>: ldrb w1, [x1]
0x0000aaaaaac21a78 <+880>: stp w7, w6, [x3, #180]
0x0000aaaaaac21a7c <+884>: stp w5, w4, [x3, #188]
0x0000aaaaaac21a80 <+888>: strb w1, [x3, #200]
0x0000aaaaaac21a84 <+892>: ldr x1, [x20, #80]
0x0000aaaaaac21a88 <+896>: str x1, [x3, #120]
0x0000aaaaaac21a8c <+900>: strb w8, [x3, #176]
0x0000aaaaaac21a90 <+904>: str w2, [x3, #196]
0x0000aaaaaac21a94 <+908>: stp q26, q25, [x3, #256]
0x0000aaaaaac21a98 <+912>: stp q29, q28, [x0, #16]
0x0000aaaaaac21a9c <+916>: stp q31, q30, [x0, #48]
0x0000aaaaaac21aa0 <+920>: bl 0xaaaaaac1dcf0 <WriteControlFile>
...

The really weird thing is that the very same binaries work on a
different host (arm64 VM provided by Huawei) - the
postgresql_arm64.deb files compiled there and present on
apt.postgresql.org are fine, but when installed on that graviton VM,
they throw the above error.

It smells like graviton's arm9 isn't as backwards compatible to arm8
as it should be. (But then I don't understand why disabling openssl
fixes it.)

Christoph