Help with simple query

I have a simple table called notes which contains notes for users.
The table has 4 columns: note_id (auto-incrementing primary key),
user_id (foreign key to a users table), note (varchar), and
modified_date (timestamp).

Is there a nice simple query I can run that will return me a list of
all the *latest* notes for all users (users can have many notes in the
table)? I'm trying to figure out a simple way of doing it but seem to
be having some mental block or there is no easy way to do it.

The following query will return me all the latest dates, but I can't
return the note_id or subject with it.
SELECT n.user_id, max(n.modified_date)
FROM notes n
GROUP by n.user_id
ORDER BY n.user_id

Is this simpler than I am making it?

Regards,
Collin