BUG #15503: Bytea type column select failed when the length of the value is more than 512M

The following bug has been logged on the website:

Bug reference: 15503
Logged by: li chuancheng
Email address: lichuancheng(at)highgo(dot)com
PostgreSQL version: 10.4
Operating system: centos7
Description:

Question1:
Bytea accecp almost 1G data length, but we select failed when data is more
than 512M-5byte with HEX type output
###################Here is the sample.###################
postgres=# insert into t1 values(1,pg_read_binary_file('testfile_512M'));
INSERT 0 1
postgres=# select j from t1 where i = 1;
2018-11-14 13:39:43.251 CST [19366] ERROR: invalid memory alloc request
size 1073741827
2018-11-14 13:39:43.251 CST [19366] STATEMENT: select j from t1 where i =
1;
ERROR: invalid memory alloc request size 1073741827
postgres=#
######################################################
It is due to the length expand when trans to HEX.

Question2:
When i investigation question1 i find some problem of code.
Problem1:
------------------
Datum
byteaout(PG_FUNCTION_ARGS)
{
...
else if (bytea_output == BYTEA_OUTPUT_ESCAPE)
{
...
int len;
...
rp = result = (char *) palloc(len);
...
}
}
------------------
The type of 'len' shoule be int64. Because i have find a bug below when
select with ESCAPE and 1G length of bytea value.
#########################################################
postgres=# insert into t1 values(3,pg_read_binary_file('testfile_1G'));
INSERT 0 1
postgres=# select j from t1 where i = 3;
2018-11-14 14:08:55.688 CST [20832] ERROR: invalid memory alloc request
size 18446744072506499617
2018-11-14 14:08:55.688 CST [20832] STATEMENT: select j from t1 where i =
3;
ERROR: invalid memory alloc request size 18446744072506499617
postgres=#
#########################################################

Question3:
Here is a bug of function get_bit()
#########################################################
postgres=# select get_bit(j,10) from t1 where i = 1;
2018-11-14 14:15:33.158 CST [20832] ERROR: index 10 out of valid range,
0..-1
2018-11-14 14:15:33.158 CST [20832] STATEMENT: select get_bit(j,10) from t1
where i = 1;
ERROR: index 10 out of valid range, 0..-1
postgres=# select get_bit(j,10) from t1 where i = 3;
2018-11-14 14:15:38.212 CST [20832] ERROR: index 10 out of valid range,
0..-8388609
2018-11-14 14:15:38.212 CST [20832] STATEMENT: select get_bit(j,10) from t1
where i = 3;
ERROR: index 10 out of valid range, 0..-8388609
postgres=#
#########################################################
It is similar to question2. In function byteaGetByte, the type of 'len'
should be int64.

is it a bug?
thanks,
best regards.
lchch