Re: Disk buffering of resultsets

From: Vitalii Tymchyshyn <vit(at)tym(dot)im>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: John R Pierce <pierce(at)hogranch(dot)com>, PG-JDBC Mailing List <pgsql-jdbc(at)postgresql(dot)org>
Subject: Re: Disk buffering of resultsets
Date: 2014-09-23 01:39:42
Message-ID: CABWW-d3QW0ywPge0-mVZ1d1WhsGwB5cWy4wOo3CK-sPOdqEKiA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-jdbc

Some more thoughts:
1) is it really not possible to stream results in autocommit mode? The only
problem I can see is that connection can't be used (e.g. by other statement
objects) until after all the results are fetched. So what? In many cases
it's OK.
2) another alternative to temp files are direct buffers. They are not
exactly the same, but are not counted towards heap and can be swapped out
by system if needed.
3) For any disk saving I'd
- postpone it as long as possible (e.g. until after connection is used by
another statement)
- do as little as possible (may be by saving frames from network as is with
little decoding)
- do it in background (a little conflicting to postponing, but more
thinking is needed). Return first fetchSize rows and start copying network
to disk in background thread.
Anyway one needs to go through everything to issue next command, does not
he? The question is if all of this needs to be saved or simply skipped (as
soon as result set is closed).

Best regards, Vitalii Tymchyshyn
22 вер. 2014 20:46, користувач "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> написав:

> John R Pierce <pierce(at)hogranch(dot)com> writes:
> > this still won't address the issue that the postgresql server itself
> > ALSO marshals the entire result set into ITS memory before sending it to
> > the client.
>
> If it actually did that, then there would be an issue ... but it never
> has, and very likely never will. The server sends rows on-the-fly as
> they're computed. That is indeed the very reason that client libraries
> tend to want to accumulate full resultsets: they're hiding that behavior
> from applications, so as to make it look like you get either an error or
> a full resultset, not some rows and then an error.
>
> regards, tom lane
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc
>

In response to

Responses

Browse pgsql-jdbc by date

  From Date Subject
Next Message Lussier, Denis 2014-09-23 01:55:27 Re: Disk buffering of resultsets
Previous Message Tom Lane 2014-09-23 00:46:15 Re: Disk buffering of resultsets