From: | Merlin Moncure <mmoncure(at)gmail(dot)com> |
---|---|
To: | Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> |
Cc: | Radosław Smogura <rsmogura(at)softperience(dot)eu>, PG Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: Streaming solution and v3.1 protocol |
Date: | 2011-06-03 18:19:04 |
Message-ID: | BANLkTimVNRnmpyUg3B+qXUiohJYE_58VLw@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Fri, Jun 3, 2011 at 12:04 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 03.06.2011 19:19, Radosław Smogura wrote:
>>
>> Hello,
>>
>> Sorry for short introduction about this, and plese as far as possible,
>> disconnet it from LOBs, as it on top of LOB.
>>
>> Idea of streaming is to reduce memory copy mainly during receiving and
>> sending
>> tuples. Currently receive works as follows
>> 1. Read bytes of tuple (allocate x memory).
>> 2. Eventually convert it to database encoding.
>> 3. Use this data and create datum (which is little changed copy of 1 or 2)
>> 4. Streaming will be allowed only in binary mode, and actually stream
>> in/out
>> will return binary data.
>
> Hmm, I was thinking that streaming would be a whole new mode, alongside the
> current text and binary mode.
>
>> Look for example at execution chain from textrecv.
>>
>> Idea is to add stream_in, stream_out columns pg_type.
>>
>> When value is to be serialized the sin/sout is called. Struct must pass
>> len of
>> data, and struct of stream (simillar to C FILE*).
>>
>> Caller should validate if all bytes has been consumed (expose simple
>> methods
>> for this)
>>
>> To implement(code API requirements):
>> First stream is buffered socekt reader.
>>
>> Fast substreams - create fast stream limited to x bytes basing on other
>> stream
>>
>> Skipping bytes + skipAll()
>>
>> Stream filtering - do fast (faster will be if conversion will occure in
>> buffered chunks) encoding conversion.
>>
>> Support for direct PG printf() version. Linux has ability to create cookie
>> streams and use it with fprintf(), so its greate advantage to format huge
>> strings. Other system should buffer output. Problem is if Linux cookie
>> will
>> fail will it write something to output? Windows proxy will push value to
>> temp
>> buffer.
>
> This is pretty low-level stuff, I think we should focus on the protocol
> changes and user-visible libpq API first.
+1. in particular, I'd like to see the libpq api changes.
merlin
From | Date | Subject | |
---|---|---|---|
Next Message | Merlin Moncure | 2011-06-03 18:22:35 | Re: Remove support for 'userlocks'? |
Previous Message | Noah Misch | 2011-06-03 18:10:03 | Re: reducing the overhead of frequent table locks - now, with WIP patch |