EXCLUDE constraint with not equals

Hello,

Given the following table, I would like to ensure that all the rows for an
email that have a user defined map to the same user.

CREATE TABLE person (
id INTEGER PRIMARY KEY,
user TEXT,
email TEXT NOT NULL);

What I think I'm looking for is something like this:

CREATE TABLE person (
id INTEGER PRIMARY KEY,
user TEXT,
email TEXT NOT NULL,
EXCLUDE (email WITH =, user WITH <>)
WHERE (user IS NOT NULL));

The not equals comparison isn't supported, but it would be useful here.

Is there another way to do this, short of creating a separate table that
associates email and user?