BUG #18129: GiST index produces incorrect query results

The following bug has been logged on the website:

Bug reference: 18129
Logged by: Alexander Lakhin
Email address: exclusion(at)gmail(dot)com
PostgreSQL version: 16.0
Operating system: Ubuntu 22.04
Description:

The following test data manipulation:
sed '/{73,23,20}/!s/\([0-9]\+\)/\1,10\1,20\1,30\1/g' -i
contrib/intarray/data/test__int.data
(effectively multiplying contents of all arrays (except for one) in the
test by 3)

makes `make check contrib/intarray` fail, with random results, e.g.:
--- .../contrib/intarray/expected/_int.out 2023-09-21
21:44:46.967470822 +0300
+++ .../contrib/intarray/results/_int.out 2023-09-22 08:12:18.269739402
+0300
@@ -497,13 +497,13 @@
SELECT count(*) from test__int WHERE a && '{23,50}';
count
-------
- 403
+ 395
(1 row)

SELECT count(*) from test__int WHERE a @@ '23|50';
count
-------
- 403
+ 395
(1 row)

SELECT count(*) from test__int WHERE a @> '{23,50}';
@@ -557,13 +557,13 @@
SELECT count(*) from test__int WHERE a @@ '20 | !21';
count
-------
- 6566
+ 6478
(1 row)

SELECT count(*) from test__int WHERE a @@ '!20 & !21';
count
-------
- 6343
+ 6258
(1 row)

INSERT INTO test__int SELECT array(SELECT x FROM generate_series(1, 1001)
x); -- should fail
@@ -639,13 +639,13 @@
SELECT count(*) from test__int WHERE a @@ '20 | !21';
count
-------
- 6566
+ 6381
(1 row)

SELECT count(*) from test__int WHERE a @@ '!20 & !21';
count
-------
- 6343
+ 6165
(1 row)

DROP INDEX text_idx;

(Here we deal with intarray's gist__int_ops, but I suspect that other gist
opclasses might be affected as well.)

First bad commit is 9155580fd, on which three different outcomes are
possible: test passes/test fails/assertion fails.
Assertion failures, that we had observed after 9155580fd, were fixed by
a7ee7c851 and 741b88435, but apparently there is another defect induced
by that gist implementation change.