Remove Modifiers on Table

I created a modifier for auto incrementing my primary key as follows:

records=# \d users
Table "public.users"
Column | Type | Modifiers
--------+-----------------------+----------------------------------------------------
id | integer | not null default
nextval('users_seq_id'::regclass)
fname | character varying(40) | not null
lname | character varying(40) | not null
email | character varying(40) | not null
office | character varying(5) | not null
dob | date | not null
title | character varying(40) | not null
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_email_key" UNIQUE, btree (email)

I recently tried to remove the modifier and it failed because it was
associated with the 'id' column so my question is how do I remove /
delete the modifier so I can delete the sequence I created to auto
increment my 'id' value? I don't want to drop the id column / loss my
column data, I just want to remove the associated modifier so I can
drop the sequence.

Thanks for any assistance.