| From: | George Woodring <george(dot)woodring(at)iglass(dot)net> |
|---|---|
| To: | Scott Ribe <scott_ribe(at)elevated-dev(dot)com> |
| Cc: | PostgreSQL General <pgsql-general(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: export to parquet |
| Date: | 2020-08-26 19:39:10 |
| Message-ID: | CACi+J=QB8g3SLXHSob2eh1BBqyxYNvf07qvP+gqA-knw4PcHwQ@mail.gmail.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-general |
I don't know how many hoops you want to jump through, we use AWS and Athena
to create them.
- Export table as JSON
- Put on AWS S3
- Create JSON table in Athena
- Use the JSON table to create a parquet table
The parquet files will be in S3 as well after the parquet table is
created. If you are interested I can share the AWS CLI commands we use.
George Woodring
iGLASS Networks
www.iglass.net
On Wed, Aug 26, 2020 at 3:00 PM Scott Ribe <scott_ribe(at)elevated-dev(dot)com>
wrote:
> I have no Hadoop, no HDFS. Just looking for the easiest way to export some
> PG tables into Parquet format for testing--need to determine what kind of
> space reduction we can get before deciding whether to look into it more.
>
> Any suggestions on particular tools? (PG 12, Linux)
>
>
> --
> Scott Ribe
> scott_ribe(at)elevated-dev(dot)com
> https://www.linkedin.com/in/scottribe/
>
>
>
>
>
>
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Tom Lane | 2020-08-26 20:40:58 | Re: Finding description pg_description |
| Previous Message | Scott Ribe | 2020-08-26 19:29:48 | Re: export to parquet |