Re: Use streaming read API in ANALYZE

On Sun, Sep 15, 2024 at 12:14 AM Mats Kindahl <mats(at)timescale(dot)com> wrote:
> I used the combination of your patch and making the computation of vacattrstats for a relation available through the API and managed to implement something that I think does the right thing. (I just sampled a few different statistics to check if they seem reasonable, like most common vals and most common freqs.) See attached patch.

Cool. I went ahead and committed that small new function and will
mark the open item closed.

> I need the vacattrstats to set up the two streams for the internal relations. I can just re-implement them in the same way as is already done, but this seems like a small change that avoids unnecessary code duplication.

Unfortunately we're not in a phase where we can make non-essential
changes, we're right about to release and we're only committing fixes,
and it seems like you have a way forward (albeit with some
duplication). We can keep talking about that for v18.

From your earlier email:
> I'll take a look at the thread. I really think the ReadStream abstraction is a good step in the right direction.

Here's something you or your colleagues might be interested in: I was
looking around for a fun extension to streamify as a demo of the
technology, and I finished up writing a quick patch to streamify
pgvector's HNSW index scan, which worked well enough to share[1] (I
think it should in principle be able to scale with the number of graph
connections, at least 16x), but then people told me that it's of
limited interest because everybody knows that HNSW indexes have to fit
in memory (I think there may also be memory prefetch streaming
opportunities, unexamined for now). But that made me wonder what the
people with the REALLY big indexes do for hyperdimensional graph
search on a scale required to build Skynet, and that led me back to
Timescale pgvectorscale[2]. I see two obvious signs that this thing
is eminently and profitably streamifiable: (1) The stated aim is
optimising for indexes that don't fit in memory, hence "Disk" in the
name of the research project it is inspired by, (2) I see that
DIskANN[3] is aggressively using libaio (Linux) and overlapped/IOCP
(Windows). So now I am waiting patiently for a Rustacean to show up
with patches for pgvectorscale to use ReadStream, which would already
get read-ahead advice and vectored I/O (Linux, macOS, FreeBSD soon
hopefully), and hopefully also provide a nice test case for the AIO
patch set which redirects buffer reads through io_uring (Linux,
basically the newer better libaio) or background I/O workers (other
OSes, which works surprisingly competitively). Just BTW for
comparison with DiskANN we have also had early POC-quality patches
that drive AIO with overlapped/IOCP (Windows) which will eventually be
rebased and proposed (Windows isn't really a primary target but we
wanted to validate that the stuff we're working on has abstractions
that will map to the obvious system APIs found in the systems
PostgreSQL targets). For completeness, I've also had it mostly
working on the POSIX AIO of FreeBSD, HP-UX and AIX (though we dropped
support for those last two so that was a bit of a dead end).

[1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJ_7NKd46nx1wbyXWriuZSNzsTfm%2BrhEuvU6nxZi3-KVw%40mail.gmail.com
[2] https://github.com/timescale/pgvectorscale
[3] https://github.com/microsoft/DiskANN