Used memory calculation in containers - docker stats and file cache

Hey folks,

I noticed some behaviour which I was not expecting at all and after hours
of debugging I am getting somewhere, but still any help understanding what
is happening would be appreciated.

I was running a PostgreSQL server within a docker container, and after
starting to generate some moderate load (I/O heavy), I noticed the memory
usage was going up (reaching the container limit) and down, which reminded
me of a Heap GC event, but it was not.

The container limit was 16GB of RAM, and as soon as this limit was reached,
there was no restart or OOM errors, rather than a huge drop in memory
(image `ContainerRelativeAbsolute`).

After digging into this issue I found that this is file cache (image `Below
CLI`), and by default this measurement is included to docker stats CLI, and
is the default way of measuring in some observability tools as far as I can
tell.

I have two questions:
1. Should the file cache be indeed part of the reported memory that is
allocated to the container? The issue I see is that if we have an alerting
system, for example, on 90% of memory used in a container, this definitely
triggers many false positives. If we ignore the file cache, could this lead
to any reliability issues?
2. What is happening on the OS level when suddenly 15GB of file cache is
getting erased (image `ContainerRelativeAbsolute`)? I would expect for
incremental deletes rather than so many GB of cache being evicted.

Cheers,
Costa