Hello Viktor,
There was a known slowness issue in this view.
It was fixed in PG 10:
https://www.postgresql.org/message-id/flat/2d533e5b-687a-09fa-a772-dac9e6cf9dd4(at)imap(dot)cc#2d533e5b-687a-09fa-a772-dac9e6cf9dd4@imap.cc
You can try to use a solution from there to create a faster view that
returns the same data as information_schema.constraint_column_usage.
I wouldn't be surprised if it was safe to replace
information_schema.constraint_column_usage itself, but I cannot
guarantee it.
Best,
Alex