| From: | Andrey Borodin <x4mmm(at)yandex-team(dot)ru> |
|---|---|
| To: | pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
| Cc: | boris(dot)novikov(at)acm(dot)org |
| Subject: | Logical archiving |
| Date: | 2020-12-04 07:33:44 |
| Message-ID: | 20D9328B-A189-43D1-80E2-EB25B9284AD6@yandex-team.ru |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi all
I was discussing problems of CDC with scientific community and they asked this simple question: "So you have efficient WAL archive on a very cheap storage, why don't you have a logical archive too?"
This seems like a wild idea. But really, we have a super expensive NVMe drives for OLTP workload. And use this devices to store buffer for data to be dumped into MapReduce\YT analytical system.
If OLAP cannot consume data fast enough - we are out of space due to repl slot.
If we have a WAL HA switchover - OLAP has a hole in the stream and have to resync data from the scratch.
If we could just run archive command ```archive-tool wal-push 0000000900000F2C000000E1.logical``` with contents of logical replication - this would be super cool for OLAP. I'd prefer even avoid writing 0000000900000F2C000000E1.logical to disk, i.e. push data on stdio or something like that.
What do you think?
Best regards, Andrey Borodin.
| From | Date | Subject | |
|---|---|---|---|
| Next Message | tsunakawa.takay@fujitsu.com | 2020-12-04 07:49:22 | RE: In-placre persistance change of a relation |
| Previous Message | k.jamison@fujitsu.com | 2020-12-04 07:05:50 | RE: [Patch] Optimize dropping of relation buffers using dlist |