From: | Ron Johnson <ronljohnsonjr(at)gmail(dot)com> |
---|---|
To: | pgsql-general <pgsql-general(at)postgresql(dot)org> |
Subject: | Re: Load a csv or a avro? |
Date: | 2024-07-05 13:03:15 |
Message-ID: | CANzqJaC419Dfe3xQHdOtQBkQPPWwPAUbNUvdfo=MWxf9eNWb3g@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
On Fri, Jul 5, 2024 at 5:08 AM sud <suds1434(at)gmail(dot)com> wrote:
> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in
> avro format messages from another system to load it into our postgres
> database. The volume will be 300million messages per day across many files
> in batches.
>
> My question was, which format should we chose in regards to faster data
> loading performance ?
>
What application will be loading the data? If psql, then go with CSV;
COPY is *really* efficient.
If the PG tables are already mapped to the avro format, then maybe avro
will be faster.
> and if any other aspects to it also should be considered apart from just
> loading performance?
>
If all the data comes in at night, drop as many indices as possible before
loading.
Load each file in as few DB connections as possible: the most efficient
binary format won't do you any good if you open and close a connection for
each and every row.
From | Date | Subject | |
---|---|---|---|
Next Message | Krishnakant Mane | 2024-07-05 13:28:21 | confused about material view locks please explain |
Previous Message | Murthy Nunna | 2024-07-05 12:37:11 | RE: psql help |