Re: ice-broker scan thread

Qingqing Zhou wrote:

>[also with reply to Gavin] look up dictionary for "gut-rot", got it ... Uh,
>this behavior is intended - I try to push enough requests shortly to kernel
>so that it understands that I am doing sequential scan, so it would pull the
>data from disk to file system cache more efficiently. Some file systems may
>have "free-behind" mechanism, but our main thread (who really process the
>query) should be fast enough before the data vanished.
>
>
I guess I was concerned that very large numbers of concurrent operations
on the same file handle
in flight at the same time might lead to poor performance or even
instability. e.g. the kernel may
make long linked lists, it might create lock contention with itself,
that kind of bad stuff. My thinking
being that the kernel wasn't designed with applications that fire off
10,000 concurrent reads against
the same file.

>I guess this is also Gavin's point - I understand that will be two different
>methodologies to handle "read-ahead". If no other thread/process involved,
>then the main thread will be responsible to grab a free buffer page from
>bufferpool and ask the kernel to put the data there by sync IO (current
>PostgreSQL does) or async IOs. And that's what I want to avoid. I'd like to
>use a dedicated thread/process to "break the ice" only, i.e., pull data from
>disk to file system cache, so that the main thread will only issue *logical*
>read.
>
>
Right, understood. My point was that a thread with sync I/O and the
query thread with
async I/O are in fact logically identical. They're just two different
implementation techniques
for the same fundemental functionality. In some cases the non-thread
implementation might
be prefered (for example on a platform with no threads).