Re: Iterating generator from C (PostgreSQL's pl/python RETUN

Sorry for cross-posting, but this IS a cross-platform issue.

Christian Tismer tismer at stackless.com wrote:
> Sven Suursoho wrote:
>
> >>> Is there any way to rewrite following program to handle returned
> >>> generator without hitting this bug?
>
> The only way I can think of getting around this is to make
> sure that there is always a running python frame.
> This would be possible if you can control how the
> extension is called.

What would be the easiest way to hold a "always running" python frame ?

The actual calls to iterator come through a pl/python framework, which can hold some
state - like some inter-call dictionaries - so keeping also a simple outer frame
there may be possible.

What danger (apart from general uglyness) may lurk there in keeping this frame ?

> > Unfortunately, this is not an option because I can't control used
> > environment: I'm trying to improve PostgreSQL's stored procedure
> > language PL/Python and this software can be used everywhere.
>
> Then there is no other way than to report the bug, get the
> fix back-ported and nagging the PL/Python maintainers
> to update things after the fix.

Unfortunately there is no 'after the fix', as the fix must happen outside
postgresql/plpython development and should also run on oldish systems.

The reason we want to get some workaround for that bug is the need
to overcome resistance from core postgreSQL developers to inclusion of our
plpython enchancements to postgreSQLs bundled plpython due to one specific use
of our generic enchancement (using a built-in generator, usually a function with yield)
on buggy RedHat's bundled plpython.so causing crashes.

Also, PL/Python has been in minimal maintenance mode for many years, with
basically only immediate bugfixing going on.

We at Skype (that is currently Sven Suursoho) are the first ones doing
active development on it after Andrew Bosma dropped development many years
ago once he got just the very basic stuff working.

> Also a test should be added which is probably missing since a while.

Test where ? In python.so build process, so that RedHat will spot it and
fix their RPM builds ?

As for testing in actual pl/python build environment, we had objections from
leading postgresql Tom Lane that even if we do test it at build time,
a determined DBA may substitute a buggy python.so later and still crash her DB instance.

Do you have any ideas, how to test for buggy asserts in python runtime
environment without actually triggering the assert ?

Then we could introduce some (uglish) special handling for generators in
pl/pythons iterator.

> I'd put a warning somewhere that generators are broken in
> debug mode, file an issue as well, but avoid trying to hack
> around this. It would make the bug even more resistent :-)

We have been trying to advocate such approach, but so far with modest results :(

--
----------------
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me: callto:hkrosing
Get Skype for free: http://www.skype.com