Re: Transparent column encryption

Hi,

On 2023-03-16 11:26:46 +0100, Peter Eisentraut wrote:
> On 13.03.23 22:11, Andres Freund wrote:
> > > It adds branches, and it makes tupledescs wider. In tight spots, such as
> > > printtup, that can hurt, even if the branches aren't ever entered.
> > In fact, I do see a noticable, but not huge, regression:
>
> I tried to reproduce your measurements, but I don't have the CPU affinity
> tools like that on macOS, so the differences were lost in the noise. (The
> absolute numbers looked very similar to yours.)
>
> I can reproduce a regression if I keep adding more columns to pg_attribute,
> like 8 OID columns does start to show.
>
> [...]
> How do we proceed here?

Maybe a daft question, but why do we need a separate type and typmod for
encrypted columns? Why isn't the fact that the column is encrypted exactly one
new field, and we use the existing type/typmod fields?

> A lot of user-facing table management stuff like compression, statistics,
> collation, dropped columns, and probably future features like column
> reordering or periods, have to be represented in pg_attribute somehow, at
> least in the current architecture. Do we hope that hardware keeps up with
> the pace at which we add new features?

Yea, it's not great as is. I think it's also OK to decide that the slowdown is
worth it in some cases - it just has to be a conscious decision, in the open.

> Do we need to decouple tuple descriptors from pg_attribute altogether?

Yes. Very clearly. The amount of memory and runtime we spent on tupledescs is
disproportionate. A second angle is that we build tupledescs way way too
frequently. The executor infers them everywhere, so not even prepared
statements protect against that.

> How do we decide what goes into the tuple descriptor and what does not? I'm
> interested in addressing this, because obviously we do want the ability to
> add more features in the future, but I don't know what the direction should
> be.

We've had some prior discussion around this, see e.g.
https://postgr.es/m/20210819114435.6r532qbadcsyfscp%40alap3.anarazel.de

Greetings,

Andres Freund