| From: | Rob Sargent <robjsargent(at)gmail(dot)com> |
|---|---|
| To: | Joe Carlson <jwcarlson(at)lbl(dot)gov> |
| Cc: | pgsql-general(at)lists(dot)postgresql(dot)org |
| Subject: | Re: TEXT column > 1Gb |
| Date: | 2023-04-12 16:36:25 |
| Message-ID: | d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-general |
On 4/12/23 08:59, Joe Carlson wrote:
> I’m curious what you learned. I’ve been tripping over the buffer
> allocation issue when either splitting input text into chunks or
> aggregating chunks in selects. I’ve decided that I need to move this
> to client side.
>
> The use case is genomics. Extracting substrings is common. So going to
> chunked storage makes sense.
>
> And we have a menagerie of code to deal with. Legacy Perl for loading.
> Clients in Perl, Java, node, python. A whole zoo.
>
> Thanks
My first stab was a java implementation based Selector class which
worked ok using a fixed buffer size but finding the end of last
transmission was hokey. I move to an embedded Tomcat implementation and
tomcat does the heavy lifting PERFECTLY. Easily slipped in to both
client and backend.
rjs
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Mark Dilger | 2023-04-12 17:04:32 | Re: TEXT column > 1Gb |
| Previous Message | Adrian Klaver | 2023-04-12 15:27:17 | Re: FW: Error! |