Try this as a user with NOBYPASSRLS:
CREATE TABLE rlsbug (deleted boolean);
INSERT INTO rlsbug VALUES (FALSE);
CREATE POLICY p_sel ON rlsbug FOR SELECT TO laurenz USING (NOT deleted);
CREATE POLICY p_upd ON rlsbug FOR UPDATE TO laurenz USING (TRUE);
ALTER TABLE rlsbug ENABLE ROW LEVEL SECURITY;
ALTER TABLE rlsbug FORCE ROW LEVEL SECURITY;
UPDATE rlsbug SET deleted = TRUE WHERE NOT deleted;
ERROR: new row violates row-level security policy for table "rlsbug"
I'd say that this error is wrong. The FOR SELECT policy should be applied
to the WHERE condition, but certainly not to check new rows.
Yours,
Laurenz Albe