From: | Lian Jiang <jiangok2006(at)gmail(dot)com> |
---|---|
To: | Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> |
Cc: | pgsql-general(at)lists(dot)postgresql(dot)org |
Subject: | Re: speed up full table scan using psql |
Date: | 2023-06-01 05:51:19 |
Message-ID: | CA+aY8X6+ENwVh0megxbHczoX-W-Yd6BmHKRUYS3MJNLGLJDz_w@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
The whole command is:
psql %(pg_uri)s -c %(sql)s | %(sed)s | %(pv)s | %(split)s) 2>&1 | %(tr)s
where:
sql is "copy (select row_to_json(x_tmp_uniq) from public.mytable
x_tmp_uniq) to stdout"
sed, pv, split, tr together format and split the stdout into jsonl files.
Hope this helps.
On Wed, May 31, 2023 at 9:16 PM Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
wrote:
> On 5/31/23 13:57, Lian Jiang wrote:
> > The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq)
> > from public.mytable x_tmp_uniq) to stdout"
>
> What is taking the stdout and what it is it doing?
>
> > postgres version: 14.7
> > Does this mean COPY and java CopyManager may not help since my psql
> > command already uses copy?
> >
> > Regarding pg_dump, it does not support json format which means extra
> > work is needed to convert the supported format to jsonl (or parquet) so
> > that they can be imported into snowflake. Still exploring but want to
> > call it out early. Maybe 'custom' format can be parquet?
> >
> >
> > Thanks
> > Lian
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com
>
>
--
Create your own email signature
<https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
From | Date | Subject | |
---|---|---|---|
Next Message | Yu Shi (Fujitsu) | 2023-06-01 07:39:51 | RE: Support logical replication of DDLs |
Previous Message | Adrian Klaver | 2023-06-01 04:16:03 | Re: speed up full table scan using psql |