Re: PostgreSQL suitable?

On Tue, Dec 19, 2017 at 3:07 PM, Kellner Thiemo <thiemo(dot)kellner(at)usb(dot)ch>
wrote:

> Hi
>
> We are developing a data warehouse of which the integration layer will
> start with over 100 TB of data. There are not many entities though we
> probably can partition and foremost we should use inheritance for the lab
> results. I just was wondering if PostgreSQL was able to cope with. In case
> it depends on the modelling kind, we have not yet decided between classic
> erd, anchor modelling and data vault.
>
> Does someone have experience with such a set up?
>

There are a significant number of issues that happen when you try to put
that much data in PostgrSQL. Remember there is a hard limit of 32TB per
table.

I currently help administer an analytics environment where 400TB of data is
pre-aggregated into 32TB of ready-to-serve metrics. We generally try to
keep our initial point of entry databases to under 20TB of possible.

Nonetheless it is quite possible either using distributed add-ons like
Citus, forks like Postgres-XL, or (as we did at Adjust) a mixture of data
consolidation and application-level sharding.

As a plug, I expect to deliver talks about this in various places. Stay
tuned ;-)

>
> Kind regards
>
> Thiemo
>
>

--
Best Wishes,
Chris Travers

Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor
lock-in.
http://www.efficito.com/learn_more