Insert into on conflict, data size upto 3 billion records

From: Karthik Kumar Kondamudi <karthikkumar09(at)gmail(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Insert into on conflict, data size upto 3 billion records
Date: 2021-02-12 18:46:05
Message-ID: CAD-twtSfABMBH3ODxJiKdh6FHBtB0UuXn4mN-xwnC7tb=Cphjg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi,

I'm looking for suggestions on how I can improve the performance of the
below merge statement, we have a batch process that batch load the data
into the _batch tables using Postgres and the task is to update the main
target tables if the record exists else into it, sometime these batch table
could go up to 5 billion records. Here is the current scenario

target_table_main has 700,070,247 records and is hash partitioned into 50
chunks, it has an index on logical_ts and the batch table has 2,715,020,546
close to 3 billion records, so I'm dealing with a huge set of data so
looking of doing this in the most efficient way.

Thank you

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Christophe Pettus 2021-02-12 19:11:30 MultiXactMemberControlLock contention on a replica
Previous Message Noah Bergbauer 2021-02-12 17:38:06 Re: Preventing free space from being reused