Re: parallel restore vs. windows

ITAGAKI Takahiro wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>
>> I did this, but it turned out that the problem was a logic error that I
>> found once I had managed to get a working debugger. However, the Windows
>> thread code should now be more robust, so thanks to Andrew and Magnus
>> for the suggestions.
>>
>
> Hello, I tested parallel restore on Windows.
> I have some random comments about it:
>

Thanks for this
> * Two compiler warnings.
> pg_backup_custom.c: In function `_PrintTocData':
> pg_backup_custom.c:437: warning: unused variable `ctx'
> pg_backup_custom.c: In function `_ReopenArchive':
> pg_backup_custom.c:849: warning: unused variable `ctx'
>

Will be fixed in code cleanup
> * No description about new options in pg_restore --help.
> There are no help messages about multi-thread (-m) and
> truncate-before-load options.
>

Will fix
> * multi-thread option is ignored if --data-only is on.
> Is it an intended behavior? Even if so, we'd better to have
> warning messages here.
>

Not intended, unless my memory is fading. I will check.
> * Threads, forked processes and connections are disposed per entry.
> I think it's a designed behavior, but there might be room for
> improvement. The present implementation is slower when there
> are many small objects. If we can specialize in thread-based
> implementation, thread pooling and connections pooling are
> typically used in the context. -- it might be a ToDo item in 8.5.
>

Yes. I only got threading working at all just a few days ago. I think
your suggestion is a good one, and we should probably converge on a
threaded implementation and then look at using pooling. However, as you
say that would be work for the 8.5 timeframe.

> ----
> I have no idea about performance because I don't have multi-core
> machine for windows. Parallel restore seems to be slower than
> serial restore on single-cpu machine.
>

Not surprising. There is extra connection, worker setup/breakdown,
dependency housekeeping and context switching involved. However, I'd be
surprised if the overhead were huge.

cheers

andrew