Re: [COMMITTERS] pgsql: Add transforms feature

On 04/28/2015 04:10 PM, Andrew Dunstan wrote:
>
> On 04/28/2015 12:03 PM, Andrew Dunstan wrote:
>>
>> [switching to -hackers]
>>
>> On 04/28/2015 11:51 AM, Andrew Dunstan wrote:
>>>
>>> On 04/28/2015 09:04 AM, Michael Paquier wrote:
>>>> On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
>>>> <michael(dot)paquier(at)gmail(dot)com> wrote:
>>>>> On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:
>>>>>> w.r.t MSVC builds, it looks like we need entries in
>>>>>> $contrib_extraincludes
>>>>>> in src/tools/msvc/Mkvcbuild.pm at the very least.
>>>>> If our goal is to put back to green the Windows nodes as quick as
>>>>> possible, we could bypass their build this way , and we'll
>>>>> additionally need to update the install script and
>>>>> vcregress.pl/contribcheck to bypass those modules accordingly. Now I
>>>>> don't think that we should make the things produced inconsistent.
>>>> OK, attached are two patches to put back the buildfarm nodes using
>>>> MSVC to green
>>>> - 0001 adds support for build and installation of the new transform
>>>> modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
>>>> this patch is enough to put back the buildfarm to a green state for
>>>> MSVC, but it disables the regression tests for those modules,
>>>> something addressed in the next patch...
>>>> - 0002 adds support for regression tests for the new modules. The
>>>> thing is that we need to check the major version version of Python
>>>> available in configuration and choose what are the extensions to
>>>> preload before the tests run. It is a little bit hacky... But it has
>>>> the merit to work, and I am not sure we could have a cleaner solution
>>>> by looking at the Makefiles of the transform modules that use
>>>> currently conditions based on $(python_majorversion).
>>>> Regards,
>>>
>>>
>>> I have pushed the first of these with some tweaks.
>>>
>>> I'm looking at the second.
>>>
>>>
>>
>>
>> Regarding this second patch - apart from the buggy python version
>> test which I just fixed in the first patch, I see this:
>>
>> + if ($pyver eq "2")
>> + {
>> + push @opts, "--load-extension=plpythonu";
>> + push @opts, '--load-extension=' . $module . 'u';
>> + }
>>
>>
>> But what do we do for Python3?
>>
>> Do we actually have a Windows machine building with Python3?
>
>
> The answer seems to be "probably not". When I tried enabling this with
> bowerbird I got a ton of errors like these:
>
> plpy_cursorobject.obj : error LNK2001: unresolved external symbol
> PyObject_SelfIter [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
> plpy_cursorobject.obj : error LNK2019: unresolved external symbol
> __imp_PyType_Ready referenced in function PLy_cursor_init_type
> [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
>
>
> Something else to fix I guess.
>
>

OK, I fixed this as Michael suggested by installing the 64 bit python3
(it's a pity that python.org didn't offer that to me as a download on
their front page - that's a bit ugly).

However, when it came to running these tests that was a miserable
failure. And indeed, we don't run any regression tests for plpython3 on
MSVC.

So I committed this with just python2 tests enabled. All the buildfarm
MSVC hosts seem to be using python2 anyway.

Meanwhile, we have some work to do on the mingw/gcc side.

These changes help make some progress - they let compilation succeed for
hstore_plperl but I still get linkage failures. I suspect we need some
things marked for export that we haven't to be marked needed before.

diff --git a/contrib/hstore_plperl/Makefile b/contrib/hstore_plperl/Makefile
index ddf6036..81fe1af 100644
--- a/contrib/hstore_plperl/Makefile
+++ b/contrib/hstore_plperl/Makefile
@@ -3,7 +3,7 @@
MODULE_big = hstore_plperl
OBJS = hstore_plperl.o

-PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(perl_archlibexp)/CORE
-I$(top_srcdir)/contrib/hstore
+PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib/hstore

EXTENSION = hstore_plperl hstore_plperlu
DATA = hstore_plperl--1.0.sql hstore_plperlu--1.0.sql
@@ -21,3 +21,6 @@ top_builddir = ../..
include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
+
+override CPPFLAGS := $(CPPFLAGS) -DPLPERL_HAVE_UID_GID
-I$(perl_archlibexp)/CORE
+override CFLAGS += -Wno-comment

The linkage failures look like:

x86_64-w64-mingw32-gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -g -O2 -Wno-comment -I../../src/pl/plperl
-I../../contrib/hstore -I. -I. -I../../src/include
-I./src/include/port/win32 -DEXEC_BACKEND
-I/c/prog/3p64/include/libxml2 -I/c/prog/3p64/include
-I/c/prog/3p64/openssl/include "-I../../src/include/port/win32"
-DPLPERL_HAVE_UID_GID -IC:/Perl64/lib/CORE -c -o hstore_plperl.o
hstore_plperl.c
x86_64-w64-mingw32-gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -g -O2 -Wno-comment -shared -static-libgcc
-o hstore_plperl.dll hstore_plperl.o -L../../src/port
-L../../src/common -Wl,--allow-multiple-definition
-Wl,--disable-auto-import -L/c/prog/3p64/lib -L/c/prog/3p64/lib
-L/c/prog/3p64/openssl/lib/VC -Wl,--as-needed -L../../src/backend
-lpostgres -lpgcommon -lpgport -lintl -lxslt -lxml2 -lz -lws2_32 -lm
-lws2_32 -Wl,--export-all-symbols -Wl,--out-implib=libhstore_plperl.a
hstore_plperl.o: In function `hstore_to_plperl':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:17:
undefined reference to `hstoreUpgrade'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:24:
undefined reference to `__imp_Perl_get_context'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:24:
undefined reference to `__imp_Perl_newSV_type'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:26:
undefined reference to `__imp_Perl_hv_common_key_len'
hstore_plperl.o: In function `cstr2sv':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:115:
undefined reference to `__imp_Perl_newSVpv'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:119:
undefined reference to `__imp_Perl_newSVpv'
hstore_plperl.o: In function `hstore_to_plperl':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:32:
undefined reference to `__imp_Perl_newSV'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:37:
undefined reference to `__imp_Perl_newRV'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:57:
undefined reference to `__imp_Perl_get_context'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:57:
undefined reference to `__imp_Perl_hv_iterinit'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:59:
undefined reference to `__imp_Perl_hv_iternext_flags'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:68:
undefined reference to `hstoreCheckKeyLen'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:71:
undefined reference to `__imp_Perl_newSVsv'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:89:
undefined reference to `__imp_Perl_sv_2pvutf8'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:64:
undefined reference to `__imp_Perl_newSVpvn_flags'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:71:
undefined reference to `__imp_Perl_newSVsv'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:80:
undefined reference to `hstoreCheckValLen'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98:
undefined reference to `__imp_Perl_sv_free'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98:
undefined reference to `__imp_Perl_sv_free2'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:89:
undefined reference to `__imp_Perl_sv_2pvutf8'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:87:
undefined reference to `__imp_Perl_sv_2pv_flags'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:87:
undefined reference to `hstoreUniquePairs'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:88:
undefined reference to `hstorePairs'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98:
undefined reference to `__imp_Perl_sv_free'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98:
undefined reference to `__imp_Perl_sv_free2'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:87:
undefined reference to `__imp_Perl_sv_2pv_flags'
collect2.exe: error: ld returned 1 exit status

cheers

andrew