Re: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3

Raj
You don’t need to create the slot, this is the reason. When you create the
slot replication PostgreSQL start to hold the logs and consequently the
storage grows until you use this slot for replicate the data.

Just set the parameter for CDC and create the task for migrate the data to
S3. DMS will create the slot automatic.

Monitor the slot status and transaction log size.
L

On Sun, 12 Dec 2021 at 07:51 Raj kumar <rajkumar820999(at)gmail(dot)com> wrote:

> *Correction ** - To enable logical replication of a AWS-RDS PostgreSQL
> source database,* *I followed the below steps.*
>
> On Sun, Dec 12, 2021 at 4:18 PM Raj kumar <rajkumar820999(at)gmail(dot)com>
> wrote:
>
>>
>> AWS DMS supports change data capture (CDC) using logical replication. I'm
>> using PostgreSQL V12 as the DMS Source and migrating data to S3 DataLake
>> through DMS.
>>
>> *After creating replication slot for pglogical and enabling CDC from
>> Postgresql RDS to S3, the Storage space consumed heavily and increased by
>> 1TB in a day and filled up the storage completely. What could be the reason
>> for this? How can I avoid this? Please advice.*
>>
>> To enable logical replication of a self-managed PostgreSQL source
>> database, I followed the below steps.
>>
>> 1. Set wal_level = logical.
>> 2. Set max_replication_slots to a value greater than 1.
>> 3. Set the max_replication_slots value to 10
>> 4. Set max_wal_senders to a value greater than 10.
>> 5. Set wal_sender_timeout =0.
>>
>> GRANT USAGE on SCHEMA pglogical to DATALAKE_USER;
>>
>> GRANT ALL on ALL TABLES in SCHEMA pglogical to DATALAKE_USER;
>>
>> SELECT pglogical.create_node(node_name := 'postgresdb',dsn := 'host=
>> postgresinstancedb.ap-south-1.rds.amazonaws.com port=5432
>> dbname=masterpoc');
>>
>> select
>> pglogical.create_replication_set('pgdms_replication_slot_datalake', true,
>> false, false, true);
>>
>> Thanks & Regards,
>> Raj Kumar Narendiran.
>>
> --
Licio Matos