Re: JDBC CTS 1.2.1

On Wednesday 27 October 2004 14:15, Vadim Nasardinov wrote:
> Before attempting to tackle the 1.3.1 suite, I tried to get 1.2.1 up
> and running. My results are as follows:
>
> PASSED: 1778
> FAILED: 510

I took a closer look at the failures and discovered that the situation
was a lot better than the above figures suggest. It turns out that
that the testsuite only has 575 tests or so -- not 1700+, as one might
imagine by looking at the above.

(If you use the slightly newer exclusion list available from
http://java.sun.com/products/jdbc/exclude.1.2.1.txt?button=Download
then the number of tests goes down to 548.)

Each test is run in four different "vehicles": appclient, ejb, jsp,
and servlet. If you sort passed and failed tests by vehicle, the
following picture emerges:

-------+-----------+-----+-----+--------
| appclient | ejb | jsp | servlet
-------+-----------+-----+-----+--------
PASSED | 551 | 129 | 549 | 549
-------+-----------+-----+-----+--------
FAILED | 24 | 442 | 22 | 22
-------+-----------+-----+-----+--------
TOTAL | 575 | 571 | 571 | 571
-------+-----------+-----+-----+--------

Note that the "ejb" vehicle is an outlier. I haven't taken a close
look at it yet, but I suspect that the vast majority of tests that
fail in the ejb vehicle do so for reasons unrelated to the JDBC
driver.

A more correct breakdown is as follows:

PASSED: 549
FAILED: 22
TOTAL: 571

Quite respectable, if you ask me. Furthermore, I suspect that most
(if not all) of the remaining 22 failures are due to lack of support
for in/out parameters in stored procedures in the current backend, as
Dave Cramer pointed out in
http://archives.postgresql.org/pgsql-jdbc/2004-10/msg00290.php

The updated version of the script that counts passed and failed tests
is available here:
http://people.redhat.com/~vadimn/scratch/pgsql-jdbc/cts/cts-logs.py

Vadim