slow index scan performance

Consider these 2 index scan produced by a query

-> Index Scan using response_log_by_activity on public.response_log rl2
(cost=0.00..51.53 rows=21 width=8) (actual time=9.017..9.056 rows=0
loops=34098)
Output: rl2.activity_id, rl2.feed_id
Index Cond: (rl2.activity_id =
rl.activity_id)
Filter: rl2.success
Buffers: shared hit=3357159
read=153313
-> Index Scan using activity_pkey on
public.activity a (cost=0.00..51.10 rows=1 width=12) (actual
time=0.126..0.127 rows=1 loops=34088)
Output: a.status_id, a.activity_id,
a.visit_id
Index Cond: (a.activity_id =
rl.activity_id)
Buffers: shared hit=137925 read=32728

And it's size

conscopy=# select
pg_size_pretty(pg_relation_size('response_log_by_activity'::regclass));
pg_size_pretty
----------------
7345 MB
(1 row)

conscopy=# select
pg_size_pretty(pg_relation_size('activity_pkey'::regclass));
pg_size_pretty
----------------
8110 MB
(1 row)

Index scan on response_log_by_activity is far slower. The table has just
been repacked, and index rebuilt, but still slow.

Is there any other way to make it faster ?

Why Buffers: shared hit=3,357,159 read=153,313 on response_log_by_activity
is much bigger than Buffers: shared hit=137925 read=32728 on activity_pkey
while activity_pkey size is bigger ?

--
Regards,

Soni Maula Harriz