| From: | Aleksander Alekseev <aleksander(at)timescale(dot)com> |
|---|---|
| To: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
| Cc: | Andres Freund <andres(at)anarazel(dot)de>, PostgreSQL Development <pgsql-hackers(at)postgresql(dot)org>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> |
| Subject: | Re: Make COPY extendable in order to support Parquet and other formats |
| Date: | 2022-06-24 15:04:26 |
| Message-ID: | CAJ7c6TNACaX1aVh-b9Z8n7aWoxAg2vK22TduZOP-9vkEk9_3Pw@mail.gmail.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi Andrew,
> > IMO decent COPY FROM / TO STDIN support is crucial, because otherwise you
> > can't do COPY from/to a client. Which would make the feature unusable for
> > anybody not superuser, including just about all users of hosted PG.
> >
>
> +1
>
> Note that Parquet puts the metadata at the end of each file, which makes
> it nice to write but somewhat unfriendly for streaming readers, which
> would have to accumulate the whole file in order to process it.
It's not necessarily that bad since data is divided into pages, each
page can be processed separately. However personally I have limited
experience with Parquet at this point. Some experimentation is
required. I will keep in mind the requirement regarding COPY FROM / TO
STDIN.
--
Best regards,
Aleksander Alekseev
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Imseih (AWS), Sami | 2022-06-24 16:17:34 | Re: [BUG] Panic due to incorrect missingContrecPtr after promotion |
| Previous Message | Robert Haas | 2022-06-24 14:59:25 | Re: making relfilenodes 56 bits |