Re: silent data loss with ext4 / all current versions

Hi,

On 11/27/2015 02:28 PM, Greg Stark wrote:
> On Fri, Nov 27, 2015 at 11:17 AM, Tomas Vondra
> <tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>> I plan to do more power failure testing soon, with more complex
>> test scenarios. I suspect there might be other similar issues (e.g.
>> when we rename a file before a checkpoint and don't fsync the
>> directory -then the rename won't be replayed and will be lost).
>
> I'm curious how you're doing this testing. The easiest way I can
> think of would be to run a database on an LVM volume and take a large
> number of LVM snapshots very rapidly and then see if the database can
> start up from each snapshot. Bonus points for keeping track of the
> committed transactions before each snaphsot and ensuring they're
> still there I guess.

I do have reliable storage (Intel SSD with power-loss protection), and
I've connected the system to a sophisticated power-loss-making device
called "the power switch" (image attached).

In other words, in the last ~7 days the system got rebooted more times
than in the previous ~5 years.

> That always seemed unsatisfactory because in the past we were mainly
> concerned with whether fsync was actually getting propagated to the
> physical media. But for testing whether we're fsyncing enough for
> the filesystem that would be good enough.

Yeah. I considered some form of virtualized setup initially, but my
original intent was to verify whether disabling write barriers really is
safe (because I've heard numerous complaints that it's stupid). And as
write barriers are more tightly coupled to the hardware, I opted for the
more brutal approach.

But I agree some form of virtualized setup might be more flexible,
although I'm not sure LVM snapshots are good approach as snapshots may
wait for I/O requests to complete and such. I think something qemu might
work better when combined with "kill -9" and I plan to try reproducing
the data loss issue on such setup.

regards

--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services