About b-tree usage

Please let me know, if there is any option in postgresql to achieve the
following usage of a b-tree index:

For a relation R(att0, att1) and a btree index on attribute att0

In each insertion of a tuple on table:
- look on index if the value of att0 of new entry does already exist in
index, and
- if no, allow the aprorpiate entry on b-tree
- if yes, do not allow an entry.

In my aplication i have always my relation clustered according to att0.
And the only information needed for a query with a range condition over
att0 in WHERE clause, is the place on disc where the first tuple with a
given value on att0 is placed.

The hint, is that beacause of too many raws of table, the index size is
too big. But the number of discrete values of att0 is orders of
magnitudes smaller than the number of tuples.

I try to investigate, if there is a way to use an alternative of b-tree
index, to decrease the blocks of indexed that are fetched into memory.

Thanks.