Re: Postgres bulk insert/ETL performance on high speed servers - test results

On Thu, Sep 1, 2016 at 11:30 PM, Mike Sofen <msofen(at)runbox(dot)com> wrote:
> PASS 2:
> Process: Transform/Load (all work local to the server - read,
> transform, write as a single batch)
> Num Source Rows: 10,554,800 (one batch from just a single source table
> going to a single target table)
> Avg Rowcount Compression: 31.5 (jsonb row compression resulting in
> 31.5x fewer rows)
> AWS Time in Secs: 2,493 (41.5 minutes)
> Cisco Time in Secs: 661 (10 minutes)
> Difference: 3.8x
> Comment:AWS: 4.2k rows/sec Cisco: 16k rows/sec
>
> It's obvious the size of the batch exceeded the AWS server memory, resulting
> in a profoundly slower processing time. This was a true, apples to apples
> comparison between Pass 1 and Pass 2: average row lengths were within 7% of
> each other (1121 vs 1203) using identical table structures and processing
> code, the only difference was the target server.
>
> I'm happy to answer questions about these results.

Are you sure it's a memory thing and not an EBS bandwidth thing?

EBS has significantly less bandwidth than direct-attached flash.