Implementing an expanded object in C

Hello,

Apologies in advance for the long question. I've made a lot of progress on
my GraphBLAS extension and getting close to having most of the API usefully
exposed to postgres, but I'm been struggling with an issue related to when
i switched to using an expanded representation of matrix types.

I've tried to follow closely how arrays work, but the answer still eludes
me. The situation is slightly different in that where an array's flat
representation is useful, a sparse matrix's flat form is an edge-list, so
it's not useful unexpanded so I have a macro, PGGRB_GETARG_MATRIX(n) always
returns the expanded form by checking VARATT_IS_EXTERNAL_EXPANDED_RW and
then DatumGetEOHP if true, otherwise expanding from the flat representation:

https://github.com/michelp/pggraphblas/blob/bug/eohsegfault/src/matrix.c#L203

I also have a PGGRB_RETURN_MATRIX(m) macro that calls `return
EOHPGetRWDatum(&(A)->hdr)`

https://github.com/michelp/pggraphblas/blob/bug/eohsegfault/src/pggraphblas.h#L140

This chain of calls works for me in some cases, for example an operator
function, 'matrix_mxm' which overloads the '*' operator, can be used to
multiply two matrices:

postgres=# select '{{0,1,2},{1,2,0},{4,5,6}}'::matrix *
'{{0,1,2},{1,2,0},{4,5,6}}'::matrix;
?column?
------------------------------
{{0,1,2},{2,0,1},{20,30,24}}
(1 row)

Works great! Internally this was `matrix_out(matrix_mxm(matrix_in(),
matrix_in()))` where the data flow fine both in and out of the functions.
But I have another function, 'matrix_agg', that aggregates edges from a
query into a matrix. It builds and returns the result matrix similarly to
matrix_mxm does and returns it using the same macro, but matrix_out's call
to get the agregates final value segfaults.

select matrix_agg(i, j, v) from edges; -- segfaults in matrix_out at
PG_GETARG_MATRIX(0)

at

https://github.com/michelp/pggraphblas/blob/bug/eohsegfault/src/matrix.c#L207

Afaict matrix_agg and matrix_mxm are both creating and returning matrices
the same way, using the same function to build them and the same macro that
`return EOHPGetRWDatum(&(A)->hdr)`, but when matrix_out fetches the
argument to print the result it bombs on the aggregate's final value. The
only salient different I can see if the agg's final function calls:

if (!AggCheckCallContext(fcinfo, &resultcxt)) {
resultcxt = CurrentMemoryContext;
}

oldcxt = MemoryContextSwitchTo(resultcxt);
// do matrix creation stuff
MemoryContextSwitchTo(oldcxt);

But even if I remove that and do not switch contexts, it still crashes the
same way.

It must be possible to return expanded objects from aggregates so I'm
clearly doing something wrong. The final function actually worked before I
was using expanded representation and just using PG_RETURN_POINTER, but
despite having all these clues I've been staring at this segfault in gdb
for a couple of days now.

Any pointers on this subject would be greatly appreciated! I know someone
else out there recently was working on an expanded object posted on the
list, if you don't see this, I may reach out to you. :)

-Michel