pgsql: Fix edge-case crashes and misestimation in range containment sel

Fix edge-case crashes and misestimation in range containment selectivity.

When estimating the selectivity of "range_var <@ range_constant" or
"range_var @> range_constant", if the upper (or respectively lower)
bound of the range_constant was above the last bin of the range_var's
histogram, the code would access uninitialized memory and potentially
crash (though it seems the probability of a crash is quite low).
Handle the endpoint cases explicitly to fix that.

While at it, be more paranoid about the possibility of getting NaN
or other silly results from the range type's subdiff function.
And improve some comments.

Ordinarily we'd probably add a regression test case demonstrating
the bug in unpatched code. But it's too hard to get it to crash
reliably because of the uninitialized-memory dependence, so skip that.

Per bug #16122 from Adam Scott. It's been broken from the beginning,
apparently, so backpatch to all supported branches.

Diagnosis by Michael Paquier, patch by Andrey Borodin and Tom Lane.

Discussion: https://postgr.es/m/16122-eb35bc248c806c15@postgresql.org

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/5832be6ca4d7d408dcda3746db378c524aef7b87

Modified Files
--------------
src/backend/utils/adt/rangetypes_selfuncs.c | 100 ++++++++++++++++++++--------
1 file changed, 72 insertions(+), 28 deletions(-)