Re: Loading 500m json files to database

On 3/24/20 11:53 AM, Kevin Brannen wrote:
> From: Rob Sargent <robjsargent(at)gmail(dot)com>
>
>> Sorry if I missed it, but have we seen the size range of these json files?
> Not that I've seen, but that's an implementation detail for whoever is doing the work. As someone else pointed out, pick the value as needed, whether that's 10, 100, 1000, or whatever. But issuing 1000 lines of "\copy file" sort of commands at a time isn't a big deal by itself. OTOH, you have a good point that 1000 could be too much work for the server to handle, especially if the "-1" flag is also used. As always: test, test, test... 😊
>

My fear is this:  the community helps OP load 500M "records" in a
reasonable timeframe then OP's organization complains postgres is slow
once they start using it... because the transition from file system to
rdbms was ill conceived.  Are the json files large documents or
arbitrary content or are they small data structures of recognizable
types. And so on