Same query - Slow in production

I'm trying to determine why an identical query is running
approximately 500 to 1000 times slower on our production database
compared to our backup database server.

Both database servers are dual 2.3 GHz G5 Xserves running PostgreSQL
8.1.3; both are configured with 8GB of RAM with identical shared
memory settings; both postgresql.conf files are identical; both
databases have identical indexes defined.

The three relevant tables are all clustered the same, although I'm
not sure when clustering was last performed on either server. All
three tables have recently been analyzed on both servers.

The different explain plans for this query seem to be consistent on
both servers regardless of category and the production server is
consistently and drastically slower than the backup server.

If anyone has any ideas on how to have the production server generate
the same explain plan as the backup server, or can suggest anything I
might want to try, I would greatly appreciate it.

Brian Wipf
ClickSpace Interactive Inc.
<brian(at)clickspace(dot)com>

Here's the query:

SELECT ac.attribute_id
FROM attribute_category ac
WHERE is_browsable = 'true' AND
category_id = 1000962 AND
EXISTS ( SELECT 'X'
FROM product_attribute_value pav,
category_product cp
WHERE pav.attribute_id = ac.attribute_id AND
pav.status_code is null AND
pav.product_id = cp.product_id AND
cp.category_id = ac.category_id AND
cp.product_is_active = 'true' AND
cp.product_status_code = 'complete'
)

Explain plans:

Fast (backup server):
Index Scan using attribute_category__category_id_fk_idx on
attribute_category ac (cost=0.00..47943.34 rows=7 width=4) (actual
time=0.110..0.263 rows=5 loops=1)
Index Cond: (category_id = 1000962)
Filter: (((is_browsable)::text = 'true'::text) AND (subplan))
SubPlan
-> Nested Loop (cost=0.00..7983.94 rows=3 width=0) (actual
time=0.043..0.043 rows=1 loops=5)
-> Index Scan using
category_product__category_id_is_active_and_status_idx on
category_product cp (cost=0.00..4362.64 rows=1103 width=4) (actual
time=0.013..0.015 rows=2 loops=5)
Index Cond: ((category_id = $1) AND
((product_is_active)::text = 'true'::text) AND
((product_status_code)::text = 'complete'::text))
-> Index Scan using
product_attribute_value__product_id_fk_idx on product_attribute_value
pav (cost=0.00..3.27 rows=1 width=4) (actual time=0.016..0.016
rows=1 loops=8)
Index Cond: (pav.product_id = "outer".product_id)
Filter: ((attribute_id = $0) AND (status_code IS
NULL))
Total runtime: 0.449 ms
(11 rows)

Slow (production server):
Index Scan using attribute_category__category_id_fk_idx on
attribute_category ac (cost=0.00..107115.90 rows=7 width=4) (actual
time=1.472..464.437 rows=5 loops=1)
Index Cond: (category_id = 1000962)
Filter: (((is_browsable)::text = 'true'::text) AND (subplan))
SubPlan
-> Nested Loop (cost=18.33..23739.70 rows=4 width=0) (actual
time=92.870..92.870 rows=1 loops=5)
-> Bitmap Heap Scan on product_attribute_value pav
(cost=18.33..8764.71 rows=2549 width=4) (actual time=10.191..45.672
rows=5869 loops=5)
Recheck Cond: (attribute_id = $0)
Filter: (status_code IS NULL)
-> Bitmap Index Scan on
product_attribute_value__attribute_id_fk_idx (cost=0.00..18.33
rows=2952 width=0) (actual time=9.160..9.160 rows=33330 loops=5)
Index Cond: (attribute_id = $0)
-> Index Scan using x_category_product_pk on
category_product cp (cost=0.00..5.86 rows=1 width=4) (actual
time=0.007..0.007 rows=0 loops=29345)
Index Cond: ((cp.category_id = $1) AND
("outer".product_id = cp.product_id))
Filter: (((product_is_active)::text = 'true'::text)
AND ((product_status_code)::text = 'complete'::text))
Total runtime: 464.667 ms
(14 rows)

Table Descriptions:

\d attribute_category;
Table "public.attribute_category"
Column | Type | Modifiers
-----------------+----------------------+-----------
attribute_id | integer | not null
category_id | integer | not null
is_browsable | character varying(5) |
is_required | character varying(5) |
sort_order | integer |
default_unit_id | integer |
Indexes:
"attribute_category_pk" PRIMARY KEY, btree (attribute_id,
category_id)
"attribute_category__attribute_id_fk_idx" btree (attribute_id)
"attribute_category__category_id_fk_idx" btree (category_id)
CLUSTER
Foreign-key constraints:
"attribute_category_attribute_fk" FOREIGN KEY (attribute_id)
REFERENCES attribute(attribute_id) DEFERRABLE INITIALLY DEFERRED
"attribute_category_category_fk" FOREIGN KEY (category_id)
REFERENCES category(category_id) DEFERRABLE INITIALLY DEFERRED

\d product_attribute_value;
Table "public.product_attribute_value"
Column | Type | Modifiers
----------------------------+-----------------------+-----------
attribute_id | integer | not null
attribute_unit_id | integer |
attribute_value_id | integer |
boolean_value | character varying(5) |
decimal_value | numeric(30,10) |
product_attribute_value_id | integer | not null
product_id | integer | not null
product_reference_id | integer |
status_code | character varying(32) |
Indexes:
"product_attribute_value_pk" PRIMARY KEY, btree
(product_attribute_value_id)
"product_attribute_value__attribute_id_fk_idx" btree (attribute_id)
"product_attribute_value__attribute_unit_id_fk_idx" btree
(attribute_unit_id)
"product_attribute_value__attribute_value_id_fk_idx" btree
(attribute_value_id)
"product_attribute_value__decimal_value_idx" btree (decimal_value)
"product_attribute_value__product_id_fk_idx" btree (product_id)
CLUSTER
"product_attribute_value__product_reference_id_fk_idx" btree
(product_reference_id)
Foreign-key constraints:
"product_attribute_value_attribute_fk" FOREIGN KEY
(attribute_id) REFERENCES attribute(attribute_id) DEFERRABLE
INITIALLY DEFERRED
"product_attribute_value_attributeunit_fk" FOREIGN KEY
(attribute_unit_id) REFERENCES attribute_unit(attribute_unit_id)
DEFERRABLE INITIALLY DEFERRED
"product_attribute_value_attributevalue_fk" FOREIGN KEY
(attribute_value_id) REFERENCES attribute_value(attribute_value_id)
DEFERRABLE INITIALLY DEFERRED
"product_attribute_value_product_fk" FOREIGN KEY (product_id)
REFERENCES product(product_id) DEFERRABLE INITIALLY DEFERRED
"product_attribute_value_productreference_fk" FOREIGN KEY
(product_reference_id) REFERENCES product(product_id) DEFERRABLE
INITIALLY DEFERRED

\d category_product;
Table "public.category_product"
Column | Type | Modifiers
---------------------+------------------------+-----------
category_id | integer | not null
product_id | integer | not null
en_name_sort_order | integer |
fr_name_sort_order | integer |
merchant_sort_order | integer |
price_sort_order | integer |
merchant_count | integer |
is_active | character varying(5) |
product_is_active | character varying(5) |
product_status_code | character varying(32) |
product_name_en | character varying(512) |
product_name_fr | character varying(512) |
product_click_count | integer |
Indexes:
"x_category_product_pk" PRIMARY KEY, btree (category_id,
product_id)
"category_product__category_id_is_active_and_status_idx" btree
(category_id, product_is_active, product_status_code)
"category_product__is_active_idx" btree (is_active)
"category_product__merchant_sort_order_idx" btree
(merchant_sort_order)
"x_category_product__category_id_fk_idx" btree (category_id)
CLUSTER
"x_category_product__product_id_fk_idx" btree (product_id)
Foreign-key constraints:
"x_category_product_category_fk" FOREIGN KEY (category_id)
REFERENCES category(category_id) DEFERRABLE INITIALLY DEFERRED
"x_category_product_product_fk" FOREIGN KEY (product_id)
REFERENCES product(product_id) DEFERRABLE INITIALLY DEFERRED