[ERROR] Copy from CSV fails due to memory error.

From: Kostas Chasialis <koschasialis(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: [ERROR] Copy from CSV fails due to memory error.
Date: 2022-01-19 13:01:25
Message-ID: 21D3CCA3-328F-46AC-93CB-CDEBF81BD4F8@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hey.

I am facing an issue when I try to run the following command

COPY <table_name> FROM <file> WITH DELIMITER E',’;

This file, is rather large, it's around 178GBs.

When I try to run this COPY command I get the following error:

ERROR: out of memory
DETAIL: Failed on request of size 2048 in memory context "AfterTriggerEvents".
CONTEXT: COPY ssbm300_lineorder, line 50796791

Clearly a memory allocation function is failing but I have no clue how to fix it.

I have tried experimenting with shared_buffers value in postgresql.conf file but after searching a bit I quickly realized that I do not know what I am doing there so I left it with default value. Same with work_mem value.

Did you face this issue before? Can you help me resolve it?

Thanks in advance!

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Kapila 2022-01-19 13:04:53 Re: row filtering for logical replication
Previous Message Juan José Santamaría Flecha 2022-01-19 12:24:40 Re: WIN32 pg_import_system_collations