Re: [SQL] Dilema.

> I have a problem that I'm hoping someone can help me with. I've built a
> web-pased application that is back-ended by PGsql.. It works great, I
> couldn't ask for better preformance.

And Bruce just said (on the hackers list) that big tuples
aren't an item of big pressure. Reading this less than 24
hours past tells another story.

> However, I just recently coded another module for it and this module needed
> to accept text files (or just large text blocks lets say). I ran into the 8k
> query limit barrier. I need these blocks of text to be searchable and am
> having to do this with a flat-file based solution right now, it's slow and
> cumbersome. I would much rather have the text stored in a PG field so I
> don't have to stray from the design of the search engine (which is designed
> to search only the pg database, but has been hacked to search flat files
> too).. I could break the texts up but they really all need to be in a single
> field, not broken up. I can't think up a way to do it with the 8k query
> limit.

If you really need to store big texts into a table, a
combination of a view, update rules and PL functions can
definitely do it. It might look a little complex first, but
after understanding the trick, it's pretty neat.

Look at this:

pgsql=> \d bttest
View "bttest"
Attribute | Type | Extra
-----------+------+-------
key | int4 |
content | text |
View definition: SELECT bttest_shadow."key",
bigtext_fetch(bttest_shadow.btid) AS content FROM
bttest_shadow;

pgsql=> select key, length(content) from bttest;
key | length
-----+--------
1 | 3
2 | 9
4 | 135840
(3 rows)

Yes, the row with key=4 really contains a text of that size.
And I can get it back exactly as it was inserted. And yes, I
can do INSERT/UPDATE/DELETE on bttest. Anything behaves as if
it where a regular table.

But it needs the unlimited querysize, supported only in
CURRENT development tree, so you'll have to wait for 7.0
anyway.

> Also feel free to suggest some kind of indexing system for the flat files,
> the only problem with that will be integration into my existing application
> but I've left some hooks open so I can deal with it..

That's another story, and Tom's comment on it says it all.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck(at)debis(dot)com (Jan Wieck) #