[PATCH] Add error handling to byteaout.

Hi,

when dealing with bytea values that are below the 1GB limit, but too
large to be escaped, the error messages emitted are not very helpful for
users. Especially if they appear in an unrelated context such as during
pg_dump. I've attached a patch that adds ereport()ing that would have
prevented some confusion.

Example:

,----[ master ]
| ase=# select mkbytea(29);
| ERROR: invalid memory alloc request size 1073741827
| ase=# set bytea_output to escape;
| ase=# select mkbytea(29);
| ERROR: invalid memory alloc request size 18446744071562067969
`----

The scary one is due to an integer overflow the attached patch also
fixes. I don't see any security implications though as it's only the
sign bit that is affected.

,----[ with patch applied ]
| ase=# set bytea_output to 'escape';
| ase=# select mkbytea(29);
| ERROR: escaped bytea value would be too big
| DETAIL: Value would require 2147483649 bytes.
| HINT: Use a different bytea_output setting or binary methods such as COPY BINARY.
`----

regards,
Andreas

create function mkbytea(power int, part bytea default '\x00')
returns bytea as
$$select case when power>0 then mkbytea(power-1,part||part) else part end;$$
language sql;

From f62a101a690fc9251c4c2de9c87323cedd0e9a54 Mon Sep 17 00:00:00 2001
From: Andreas Seltenreich <andreas(dot)seltenreich(at)credativ(dot)de>
Date: Mon, 1 Jun 2015 16:17:21 +0200
Subject: [PATCH] Add error handling to byteaout.

Emit a comprehensible error message when escaping fails due to
MaxAllocSize instead of waiting for palloc to fail. Also use size_t
for size computations to prevent integer overflow in
BYTEA_OUTPUT_ESCAPE branch.
---
src/backend/utils/adt/varlena.c | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 779729d..ec2594e 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -347,8 +347,21 @@ byteaout(PG_FUNCTION_ARGS)

if (bytea_output == BYTEA_OUTPUT_HEX)
{
+ Size len;
/* Print hex format */
- rp = result = palloc(VARSIZE_ANY_EXHDR(vlena) * 2 + 2 + 1);
+ len = (Size)VARSIZE_ANY_EXHDR(vlena) * 2 + 2 + 1;
+
+ if (!AllocSizeIsValid(len)) {
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("escaped bytea value would be too big"),
+ errdetail("Value would require %zu bytes.",
+ len),
+ errhint("Use a different bytea_output setting or"
+ " binary methods such as COPY BINARY.")));
+ }
+
+ rp = result = palloc(len);
*rp++ = '\\';
*rp++ = 'x';
rp += hex_encode(VARDATA_ANY(vlena), VARSIZE_ANY_EXHDR(vlena), rp);
@@ -357,8 +370,8 @@ byteaout(PG_FUNCTION_ARGS)
{
/* Print traditional escaped format */
char *vp;
- int len;
- int i;
+ Size len;
+ Size i;

len = 1; /* empty string has 1 char */
vp = VARDATA_ANY(vlena);
@@ -371,6 +384,15 @@ byteaout(PG_FUNCTION_ARGS)
else
len++;
}
+ if (!AllocSizeIsValid(len)) {
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("escaped bytea value would be too big"),
+ errdetail("Value would require %zu bytes.",
+ len),
+ errhint("Use a different bytea_output setting or"
+ " binary methods such as COPY BINARY.")));
+ }
rp = result = (char *) palloc(len);
vp = VARDATA_ANY(vlena);
for (i = VARSIZE_ANY_EXHDR(vlena); i != 0; i--, vp++)
--
2.1.4