Re: Postgres Spark connector

Hi Zhihong,

On Wed, 23 Dec 2020, 17:55 Zhihong Yu, <zyu(at)yugabyte(dot)com> wrote:

> Hi,
> I searched for Postgres support in Apache Spark.
> I found Spark doc related to JDBC.
>
> I wonder if the community is aware of Spark connector for Postgres
> (hopefully open source) where predicate involving jsonb columns can be
> pushed down.
>

JDBC driver is indeed the best driver which can be used if you have to
persist your Spark dataframes in PostgreSQL, IMO.

It's a connector which supports just pure SQL (no mapping between your
Scala/Java classes and the DB schema for instance, despite ORM frameworks
like Hibernate), so it works at a lower level allowing you to use directly
the queries you would use to handle jsonb data.

Maybe you need to use pure json's between Spark objects and the DB, but the
communication using JDBC driver can be completely based on jsonb.

Giuseppe.

>