Running PostgreSQL with ZFS ZIL

We recently built a new server for our Production database. The machine is
top of the line with 128GB of RAM, dual E5-2650. We also included NVME
drives for ZIL and L2ARC. Currently we have 3 zpools. First one holds the
FreeBSD install. Second holds the jails, and third holds all of the
database data. Needless to say it's fast.

Here is the problem. I am running benchmarks using pgbench-tools. I have
been running the tpc-b test. When I have the ZIL drive attached to the
zpool, the transactions per second average out at 220. When I disable the
ZIL drive, transactions per second average out at 700.

Anybody ever experience this? I understand the use of the external ZIL
drive for data protection, but with PostgreSQL is it really needed? The
performance hit is pretty big. My next test is loading up the machine with
a copy of our Production data and seeing how some transactions behave with
it enabled and disabled.

I've included a copy of one of the tests done.

Thanks,
-Joseph Kregloh