order of nested loop

I have two queries that return identical results. One is a SELECT DISTINCT and the other
is the same query without the DISTINCT. The explain for the second one makes it seem as
if it would be faster:

Sort (cost=73560.75..73560.75 rows=3 width=604)
vs.
Sort (cost=67246.81..67246.82 rows=3 width=604)

However in reality the first query runs much faster. The problem is this nested loop:
not distinct:
-> Subquery Scan "*SELECT* 2" (cost=0.00..30602.38 rows=25 width=604)
-> Limit (cost=0.00..30602.38 rows=25 width=604)
-> Nested Loop (cost=0.00..5499145.64 rows=4492 width=604)
================ vs. =================================
distinct:
-> Sort (cost=36903.81..36915.04 rows=4492
width=604)
Sort Key: <snip>
-> Nested Loop (cost=0.00..36631.27
rows=4492 width=604)

In the query with the distinct one table is done first, in the other the order is
reversed. This makes all the difference in the query, because in my test case there is
only one matching entry in one of the tables and that is always the table that determines
the number of rows in the result (and except in pathalogical cases will always be much
lower than the number returned from the first table). So how can I tell postgres which
table to scan in the loop first?