index speed and failed expectations?

sol=> \d stats;
Table "public.stats"
Column | Type | Modifiers
--------------+--------------------------------+-----------
id | integer | not null
start_time | timestamp(0) without time zone | not null
...
Indexes:
"stats_start_time_idx" btree (start_time)
"stats_id_key" btree (id)

There are roughly half a million rows.

This query from the console:

select * from stats order by start_time;

takes 8 seconds before starting its output. Am I wrong in assuming that
the index on start_time should make ORDER BY orders of magnitude faster?
Or is this already fast enough? Or should I max up some memory (buffer)
setting to achieve greater speeds? Not that the speed is crucial, just
curious.

TIA.