Re: COPY enhancements

Greg Smith wrote:
>> After some thought, I think that Andrew's feature *is* generally
>> applicable, if done as IGNORE COLUMN COUNT (or, more likely,
>> column_count=ignore). I can think of a lot of data sets where column
>> count is jagged and you want to do ELT instead of ETL.
>
> Exactly, the ELT approach gives you so many more options for cleaning
> up the data that I think it would be used more if it weren't so hard
> to do in Postgres right now.
>
>

+1. That's exactly what my client wants to do. They know perfectly well
that they get junk data. They want to get it into the database with a
minimum of fuss where they will have the right tools for checking and
cleaning it. If they have to spend effort whacking it into shape just to
get it into the database, then their cleanup effort essentially has to
be done in two pieces, part inside and part outside the database.

>
> While complicated, COPY is a pretty walled off command of around 3500
> lines of code, and the hackery required here is pretty small. For
> example, it turns out we do already have the code to get it to ignore
> column overruns here, and it's all of 50 new lines--much of which is
> shared with code that does other error ignoring bits too. It's easy to
> make a case for a grand future extensibility cleanup here, but it's
> really not necessary to provide a significant benefit here for the
> cases I mentioned. And I would guess the maintenance burden of a more
> general solution has to be higher than a simple implementation of the
> feature list I gave in my last message.
>
> In short: there's a presumption that adding any error-ignoring code
> would require significant contortions. I don't think that's really
> true though, and would like to keep open the possibilty of accepting
> some simple but useful ad-hoc features in this area, even if they
> don't solve every possible problem in this space just yet.
>
>

Right. What I proposed would not have been terribly invasive or
difficult, certainly less so than what seems to be our direction by an
order of magnitude at least. I don't for a moment accept the assertion
that we can get a general solution for the same effort.

cheers

andrew