Re: storage of sensor data with Fourier transforms

On May 4, 2007, at 10:13 PM, Nathan Buchanan wrote:

> Hello!
>
> I have a potential situation where I will have a lot of sensor data
> coming in very often. (every second or so) The sensor data is from
> physics type measurements, and will normally follow a slowly
> changing pattern with sinusoidal disturbances. The overall shape of
> the data is more important than catching high frequency disturbances.
>
> I had the idea of taking the Fourier transform of the waveform and
> storing the waveform internally that way to reduce storage
> requirements. When I get some free cycles, I may try doing this. I
> would want it to be done in the database in such a way that the
> user can still join to a table using this internal storage scheme.
>
> Why am I mailing this list? I'd like to ask if anyone has heard of
> someone doing anything like this. I did a small search of the lists
> and internet but didn't come up with anything. I just don't want to
> re-invent the wheel.

It's an interesting idea in theory (sounds like a simple low-pass
filter would do most of what you want) but it doesn't sound like that
great an idea in practice.

Sampling at 1Hz isn't going to generate very much data at all, unless
you're capturing ridiculous amounts of data at each sample point (I
have boxes averaging hundreds of pieces of data a second totaling 10s
of k, along with a bunch of index updates without breaking much of an
I/O sweat).

Unless you have some very tight storage constraints (embedded
controllers, flash storage, ...) you're not going to see a problem
with storing the actual data, and you're likely to regret both
discarding the original data and expending much effort developing
software low-pass pre-processing for the input signal.

(Processing it for analysis is a whole other thing, but even that
will be much easier to do if you can apply variants on your algorithm
to the untouched original data).

Cheers,
Steve