Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3

From: Raj kumar <rajkumar820999(at)gmail(dot)com>
To: Pgsql-admin <pgsql-admin(at)lists(dot)postgresql(dot)org>
Subject: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3
Date: 2021-12-12 10:48:55
Message-ID: CACxU--XrvwT6Ju5=WhNtoWE=fUYTLjnuFFr=NDEc_NwzAdv7KQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

AWS DMS supports change data capture (CDC) using logical replication. I'm
using PostgreSQL V12 as the DMS Source and migrating data to S3 DataLake
through DMS.

*After creating replication slot for pglogical and enabling CDC from
Postgresql RDS to S3, the Storage space consumed heavily and increased by
1TB in a day and filled up the storage completely. What could be the reason
for this? How can I avoid this? Please advice.*

To enable logical replication of a self-managed PostgreSQL source database,
I followed the below steps.

1. Set wal_level = logical.
2. Set max_replication_slots to a value greater than 1.
3. Set the max_replication_slots value to 10
4. Set max_wal_senders to a value greater than 10.
5. Set wal_sender_timeout =0.

GRANT USAGE on SCHEMA pglogical to DATALAKE_USER;

GRANT ALL on ALL TABLES in SCHEMA pglogical to DATALAKE_USER;

SELECT pglogical.create_node(node_name := 'postgresdb',dsn := 'host=
postgresinstancedb.ap-south-1.rds.amazonaws.com port=5432
dbname=masterpoc');

select pglogical.create_replication_set('pgdms_replication_slot_datalake',
true, false, false, true);

Thanks & Regards,
Raj Kumar Narendiran.

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Raj kumar 2021-12-12 10:51:21 Re: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3
Previous Message Mladen Gogala 2021-12-12 06:53:32 Re: package conversion error from oracle to postgresql database !!!!!!!!!