what would tar file FDW look like?

From: Bear Giles <bgiles(at)coyotesong(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: what would tar file FDW look like?
Date: 2015-08-17 14:14:18
Message-ID: CALBNtw6C8XGWmO2g6V2pwkPyF=z=-toXCWeZBTj6P4WmRy2C6Q@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I'm starting to work on a tar FDW as a proxy for a much more specific FDW.
(It's the 'faster to build two and toss the first away' approach - tar lets
me get the FDW stuff nailed down before attacking the more complex
container.) It could also be useful in its own right, or as the basis for a
zip file FDW.

I have figured out that in one mode the FDW mapping that would take the
name of the tarball as an option and produce a relation that has all of the
metadata for the contained files - filename, size, owner, timestamp, etc. I
can use the same approach I used for the /etc/passwd FDW for that.

(BTW the current version is at https://github.com/beargiles/passwd-fdw.
It's skimpy on automated tests until I can figure out how to handle the
user mapping but it works.)

The problem is the second mode where I pull a single file out of the FDW.
I've identified three approachs so far:

1. A FDW mapping specific to each file. It would take the name of the
tarfile and the embedded file. Cleanest in some ways but it would be a real
pain if you're reading a tarball dynamically.

2. A user-defined function that takes the name of the tarball and file and
returns a blob. This is the traditional approach but why bother with a FDW
then? It also brings up access control issues since it requires disclosure
of the tarball name to the user. A FDW could hide that.

3. A user-defined function that takes a tar FDW and the name of a file and
returns a blob. I think this is the best approach but I don't know if I can
specify a FDW as a parameter or how to access it.

I've skimmed the existing list of FDW but didn't find anything that can
serve as a model. The foreign DB are closest but, again, they aren't
designed for dynamic use where you want to do something with every file in
an archive / table in a foreign DB.

Is there an obvious approach? Or is it simply a bad match for FDW and
should be two standard UDF? (One returns the metadata, the second returns
the specific file.)

Thanks,

Bear

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Greg Stark 2015-08-17 14:29:58 Re: what would tar file FDW look like?
Previous Message Kouhei Kaigai 2015-08-17 13:40:31 Re: Our trial to TPC-DS but optimizer made unreasonable plan