| From: | Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> |
|---|---|
| To: | yudhi s <learnerdatabase99(at)gmail(dot)com> |
| Cc: | pgsql-general <pgsql-general(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: Moving delta data faster |
| Date: | 2024-04-05 00:39:54 |
| Message-ID: | f4853f55-5b16-4e9c-bb59-682f2b0bdefc@aklaver.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-general |
On 4/4/24 13:42, yudhi s wrote:
>
> On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com
> <mailto:adrian(dot)klaver(at)aklaver(dot)com>> wrote:
>
> On 4/3/24 22:24, yudhi s wrote:
> >
> > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver
> <adrian(dot)klaver(at)aklaver(dot)com <mailto:adrian(dot)klaver(at)aklaver(dot)com>
> > <mailto:adrian(dot)klaver(at)aklaver(dot)com
> <mailto:adrian(dot)klaver(at)aklaver(dot)com>>> wrote:
>
> S3 is not a database. You will need to be more specific about '... then
> from the S3 it will be picked and gets merged to the target postgres
> database.'
>
>
> The data from S3 will be dumped into the stage table and then the
> upsert/merge from that table to the actual table.
The S3 --> staging table would be helped by having the data as CSV and
then using COPY. The staging --> final table step could be done as
either ON CONFLICT or MERGE, you would need to test in your situation to
verify which works better.
--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Rama Krishnan | 2024-04-05 04:47:53 | About postgres pg_upgrade |
| Previous Message | Greg Sabino Mullane | 2024-04-04 21:08:28 | Re: Moving delta data faster |