| From: | Josef Šimánek <josef(dot)simanek(at)gmail(dot)com> |
|---|---|
| To: | sud <suds1434(at)gmail(dot)com> |
| Cc: | pgsql-general <pgsql-general(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: Load a csv or a avro? |
| Date: | 2024-07-05 10:02:50 |
| Message-ID: | CAFp7QwoNgw_3rGsJF1nK5Mzceg2_+UL0-9pJ2WRCw04O71S4zA@mail.gmail.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-general |
pá 5. 7. 2024 v 11:08 odesílatel sud <suds1434(at)gmail(dot)com> napsal:
>
> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in avro format messages from another system to load it into our postgres database. The volume will be 300million messages per day across many files in batches.
>
> My question was, which format should we chose in regards to faster data loading performance ? and if any other aspects to it also should be considered apart from just loading performance?
We are able to load ~300 million rows per one day using CSV and COPY
functions (https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND)
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Muhammad Ikram | 2024-07-05 10:15:28 | Re: Load a csv or a avro? |
| Previous Message | Kashif Zeeshan | 2024-07-05 09:57:37 | Re: Load a csv or a avro? |