From: | dg(at)illustra(dot)com (David Gould) |
---|---|
To: | mimo(at)interdata(dot)com(dot)pl (Michal Mosiewicz) |
Cc: | maillist(at)candle(dot)pha(dot)pa(dot)us, hackers(at)postgreSQL(dot)org |
Subject: | Re: [HACKERS] Re: [QUESTIONS] Does Storage Manager support >2GB tables? |
Date: | 1998-03-13 01:53:02 |
Message-ID: | 9803130153.AA28898@hawk.illustra.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Michal Mosiewicz writes:
> As I was always biased to threading I would note that in many cases it
> is a big win. First of all, today it's the IO which is usually the
> slowest part of the database. ...
...
> However, if you do some IO, then some processing, then some IO.... you
> loose the capability of optimising your requests. ...
> ... But if your system is not loaded too heavily, it's
> good to parallelize IO tasks. And the easiest way to accomplish it is to
> use threads for parallel execution of tasks.
Agreed, but what you are talking about here is decomposing a query into
it's parallel components and executing them in parallel. This is a win
of course, but the optimizer and executor have to support it. Also, you
start to want things like table fragmentation across devices to make this
work. A big job. As a shortcut, you can just do some lookahead on index scans
and do prefetch. Doesn't buy as much, but could probably be done very
quickly.
> But I notice that many people still think of threads as a replacement
> for fork. Of course, in such case it's pretty useless since fork is fast
> enough. But the key to the success is to parallelize single queries not
> only to leverage processor usage, but also to push IO to it's maximum.
This is indeed what I was thinking about. The process per connection scheme
of Postgres is often criticised vs a thread per connection scheme as in
Sybase for example. I was responding to that criticism.
> > That is a very easy win for us. I hadn't considered the synchonization
> > problems with palloc/pfree, and that could be a real problem.
>
> Few months ago I was thinking about it. Actually I don't see much
> problems with things like palloc/pfree. I don't see any problems with
If you have multiple threads each allocing memory at the same time, the
allocator data structures have to be protected.
> any heap data that is used locally. But it is a big problem when you
> take a look at global variables and global data that is accessed and
> modified in many places. This is potential source of troubles.
Too right.
-dg
David Gould dg(at)illustra(dot)com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
- I realize now that irony has no place in business communications.
From | Date | Subject | |
---|---|---|---|
Next Message | Bruce Momjian | 1998-03-13 02:27:12 | Re: [HACKERS] Re: [QUESTIONS] Does Storage Manager support >2GB tables?t |
Previous Message | Michal Mosiewicz | 1998-03-13 00:40:23 | Re: [HACKERS] Re: [QUESTIONS] Does Storage Manager support >2GB tables? |