Isolating semi-periodic waveforms in a signal

Hello,

I have collected several years worth of samples of a few housholds' power usage
with a samplerate of 2 seconds. An example snippet of the signal :

        http://tinyurl.com/pmgraph

As can be seen in the above graph, there are a few perodic signals in the
waveform, caused by a fridge an a freezer which are switched on and off by
their thermostats.

I'm trying to find a way to automatically isolate and remove these periodic
signals to get more insight in the (for my particular case) more interesting
parts of the waveform, which are non-periodic. For example, the events in the
last few thousand samples of the above graph.

I am able to find and locate common patterns in the waveform using
autocorrelation, but my problem is that these signals are actually
just-not-really-periodic. For example, the time and interval of a fridge
switching on and off depends on the outside temperature and is disturbed by
opening the door, etc. This makes it kind of hard to simply find the timestamps
of the fridges switching on, and subtracting the known-fridge-pattern from the
wave.

Can anybody suggest a algorithm or method for - preferrably fully automatic -
finding and isolating the various 'almost periodic' parts of a waveform like
this ?

Ideally, I would like to be able to split up the signal in several parts, one
for each periodic waveform, which when all added up, result in the original
signal.

Thanks,

On 18 Sep, 21:06, Zevv <use...@zeev.nl> wrote:
> Hello,
>
> I have collected several years worth of samples of a few housholds' power=
 usage
> with a samplerate of 2 seconds. An example snippet of the signal :
>
> =A0 =A0 =A0 =A0http://tinyurl.com/pmgraph
>
> As can be seen in the above graph, there are a few perodic signals in the
> waveform, caused by a fridge an a freezer which are switched on and off b=
y
> their thermostats.
>
> I'm trying to find a way to automatically isolate and remove these period=
ic
> signals to get more insight in the (for my particular case) more interest=
ing
> parts of the waveform, which are non-periodic. For example, the events in=
 the
> last few thousand samples of the above graph.
>
> I am able to find and locate common patterns in the waveform using
> autocorrelation, but my problem is that these signals are actually
> just-not-really-periodic. For example, the time and interval of a fridge
> switching on and off depends on the outside temperature and is disturbed =
by
> opening the door, etc. This makes it kind of hard to simply find the time=
stamps
> of the fridges switching on, and subtracting the known-fridge-pattern fro=
m the
> wave.
>
> Can anybody suggest a algorithm or method for - preferrably fully automat=
ic -
> finding and isolating the various 'almost periodic' parts of a waveform l=
ike
> this ?
>
> Ideally, I would like to be able to split up the signal in several parts,=
 one
> for each periodic waveform, which when all added up, result in the origin=
al
> signal.

I would forget about 'almost periodic' an instead treat
each event separately. If you are able to come up with a
unique template signal for each device, use the correlation
trick to identify the events that are similar to the template.

In the case of several devices being active at the same time,
this might become awkward depending on how similar the
signatures for the different devices are.

In the case of similar signatures of different events, I
would use the 'quiet' periods to establish a statistic for
the mean intervals and mean durations of the uninteresting
but regularly occuring events. That way you can establish
a likelihood for an observed event being an instance of a
regular (not periodic) sequence of events, or being a stray
event.

In the ideal case I would log power status on the individual
devices directly.

Rune

Rune Allnor <allnor@tele.ntnu.no> wrote:
> On 18 Sep, 21:06, Zevv <use...@zeev.nl> wrote:
>
>> I have collected several years worth of samples of a few housholds' power usage
>> with a samplerate of 2 seconds. An example snippet of the signal :
>>
>> &#4294967295; &#4294967295; &#4294967295; &#4294967295;http://tinyurl.com/pmgraph
>>
>> As can be seen in the above graph, there are a few perodic signals in the
>> waveform, caused by a fridge an a freezer which are switched on and off by
>> their thermostats.
>>
>> I'm trying to find a way to automatically isolate and remove these periodic
>> signals to get more insight in the (for my particular case) more interesting
>> parts of the waveform, which are non-periodic. For example, the events in the
>> last few thousand samples of the above graph.
>>
>> I am able to find and locate common patterns in the waveform using
>> autocorrelation, but my problem is that these signals are actually
>> just-not-really-periodic. For example, the time and interval of a fridge
>> switching on and off depends on the outside temperature and is disturbed by
>> opening the door, etc. This makes it kind of hard to simply find the timestamps
>> of the fridges switching on, and subtracting the known-fridge-pattern from the
>> wave.
>>
>> Can anybody suggest a algorithm or method for - preferrably fully automatic -
>> finding and isolating the various 'almost periodic' parts of a waveform like
>> this ?
>>
>> Ideally, I would like to be able to split up the signal in several parts, one
>> for each periodic waveform, which when all added up, result in the original
>> signal.
> 
> I would forget about 'almost periodic' an instead treat
> each event separately. If you are able to come up with a
> unique template signal for each device, use the correlation
> trick to identify the events that are similar to the template.
> 
> In the case of several devices being active at the same time,
> this might become awkward depending on how similar the
> signatures for the different devices are.

Well, this is one of the methods I tried, but unfortunately this does
not work out quite right yet because the 'signature' of the devices are
not always exactly the same: a fridge might run a minute shorter or
longer every now and then, which leaves nasty peaks in the signal if I
simply find and substract the template. My latest try is to find the
events, and then try to adjust the template length to match the event.
I'm not there yet, but this might work.

> In the case of similar signatures of different events, I would use the
> 'quiet' periods to establish a statistic for the mean intervals and
> mean durations of the uninteresting but regularly occuring events.
> That way you can establish a likelihood for an observed event being an
> instance of a regular (not periodic) sequence of events, or being a
> stray event.

Sounds like a good plan, although it seems that things get a lot more
complicated then I hoped for...

> In the ideal case I would log power status on the individual devices
> directly.

Of course, but like most things in life, this is just not a ideal case :)

Thanks for your input,

Zev

On 19 Sep, 08:01, Zev <use...@zeev.nl> wrote:

> > I would forget about 'almost periodic' an instead treat
> > each event separately. If you are able to come up with a
> > unique template signal for each device, use the correlation
> > trick to identify the events that are similar to the template.
>
> > In the case of several devices being active at the same time,
> > this might become awkward depending on how similar the
> > signatures for the different devices are.
>
> Well, this is one of the methods I tried, but unfortunately this does
> not work out quite right yet because the 'signature' of the devices are
> not always exactly the same: a fridge might run a minute shorter or
> longer every now and then, which leaves nasty peaks in the signal if I
> simply find and substract the template. My latest try is to find the
> events, and then try to adjust the template length to match the event.
> I'm not there yet, but this might work.

Look for changes in state - starts and ends of events.
Then try and 'pair up' the starts and ends for a given
event. This should be simple in a 'clean' signal, but
use the statsistics you fond, to match up starts and ends
for the tangled-up events.

> > In the case of similar signatures of different events, I would use the
> > 'quiet' periods to establish a statistic for the mean intervals and
> > mean durations of the uninteresting but regularly occuring events.
> > That way you can establish a likelihood for an observed event being an
> > instance of a regular (not periodic) sequence of events, or being a
> > stray event.
>
> Sounds like a good plan, although it seems that things get a lot more
> complicated then I hoped for...

They always are.

> > In the ideal case I would log power status on the individual devices
> > directly.
>
> Of course, but like most things in life, this is just not a ideal case :)

Maybe.

Keep in mind that an experiment that does not produce useful
data, is no more than an excercise in the installation and
operation of instruments and measurement devices. Which
might be a useful experience in its own right. However,
instruments are expensive to use and operate, so once you
get to play with them, make sure you get data that are useful.

In the case that you spend a lot of time, effort and $$$ on
an experiment or measurement, do take the extra two minutes
to think thorugh

1) Are the data to be measured actually useful to me?
2) What can be done to increase the usefulness of the data?

Often, very simple improvements, like keeping a log of
what you do or adding a filter or noise shield to the sensor,
makes or brakes the measurement.

Rune

Rune Allnor <allnor@tele.ntnu.no> wrote:
> On 19 Sep, 08:01, Zev <use...@zeev.nl> wrote:
> 
>> > In the case of several devices being active at the same time,
>> > this might become awkward depending on how similar the
>> > signatures for the different devices are.
>>
>> Well, this is one of the methods I tried, but unfortunately this does
>> not work out quite right yet because the 'signature' of the devices are
>> not always exactly the same: a fridge might run a minute shorter or
>> longer every now and then, which leaves nasty peaks in the signal if I
>> simply find and substract the template. My latest try is to find the
>> events, and then try to adjust the template length to match the event.
>> I'm not there yet, but this might work.
> 
> Look for changes in state - starts and ends of events.

Yes, I just tried something similar. I take the first derivative of the
signal, which also gives much better results on the correlation.
Overlapping events are now easier to find since they do not affect each
others shape, they merely interleave.

>> Of course, but like most things in life, this is just not a ideal case :)
> 
> Keep in mind that an experiment that does not produce useful
> data, is no more than an excercise in the installation and
> operation of instruments and measurement devices. Which
> might be a useful experience in its own right. However,
> instruments are expensive to use and operate, so once you
> get to play with them, make sure you get data that are useful.

Well, this whole thing got started the other way around. I built a
simple power logger a few years ago to get insight in my own power
usage, and this has now been running for a few years. My current attempt
to do other things with the logged data is just a exercise to see if it
can be down and how to do this. 

I think the results can be useful though, if I would be able to
postprocess these measurements and calculate the total electricity cost
for each device.

> Often, very simple improvements, like keeping a log of what you do or
> adding a filter or noise shield to the sensor, makes or brakes the
> measurement.

Very true.

On 19 Sep, 11:43, Zev <use...@zeev.nl> wrote:

> >> Of course, but like most things in life, this is just not a ideal case :)
>
> > Keep in mind that an experiment that does not produce useful
> > data, is no more than an excercise in the installation and
> > operation of instruments and measurement devices. Which
> > might be a useful experience in its own right. However,
> > instruments are expensive to use and operate, so once you
> > get to play with them, make sure you get data that are useful.
>
> Well, this whole thing got started the other way around. I built a
> simple power logger a few years ago to get insight in my own power
> usage, and this has now been running for a few years. My current attempt
> to do other things with the logged data is just a exercise to see if it
> can be down and how to do this.

Ah. That approach is more common than one would like to
believe - even in the professional communities: "Now that
we have all these data, let's see if they can be useful..."

For some reason, problems with hardware and doing the
measurements gets far more attention than ensuring that
the measured data are useful. Or even related to the
questions at hand.

> I think the results can be useful though, if I would be able to
> postprocess these measurements and calculate the total electricity cost
> for each device.

Then keep logs of when you switch the various devices on
and off. Correlate the measuremets with your log. If you can,
relocate the meter and measure as many devices as possible
indepenently.

Rune

Rune Allnor <allnor@tele.ntnu.no> wrote:
> On 19 Sep, 11:43, Zev <use...@zeev.nl> wrote:
> 
>> Well, this whole thing got started the other way around. I built a
>> simple power logger a few years ago to get insight in my own power
>> usage, and this has now been running for a few years. My current attempt
>> to do other things with the logged data is just a exercise to see if it
>> can be down and how to do this.
> 
> Ah. That approach is more common than one would like to
> believe - even in the professional communities: "Now that
> we have all these data, let's see if they can be useful..."

And fun!

> Then keep logs of when you switch the various devices on
> and off. Correlate the measuremets with your log.

Indeed, that's what I'm doing now.

> If you can, relocate the meter and measure as many devices as possible
> indepenently.

Not possible: the meter uses a IR led + sensor to watch the dots on the
spinning wheel of the central electricity meter :)

-- 
:wq
^X^Cy^K^X^C^C^C^C

On 19 Sep, 13:05, Ico <use...@zeev.nl> wrote:
> Rune Allnor <all...@tele.ntnu.no> wrote:
> > On 19 Sep, 11:43, Zev <use...@zeev.nl> wrote:
>
> >> Well, this whole thing got started the other way around. I built a
> >> simple power logger a few years ago to get insight in my own power
> >> usage, and this has now been running for a few years. My current attempt
> >> to do other things with the logged data is just a exercise to see if it
> >> can be down and how to do this.
>
> > Ah. That approach is more common than one would like to
> > believe - even in the professional communities: "Now that
> > we have all these data, let's see if they can be useful..."
>
> And fun!

Not at all. I tend to prefer to work project that has
a chance of succeeding.

Rune

On 18 Sep 2009 19:06:01 GMT, Zevv <usenet@zeev.nl> wrote:

>Hello,
>
>I have collected several years worth of samples of a few housholds' power usage
>with a samplerate of 2 seconds. An example snippet of the signal :
>
>        http://tinyurl.com/pmgraph
>
>As can be seen in the above graph, there are a few perodic signals in the
>waveform, caused by a fridge an a freezer which are switched on and off by
>their thermostats.
>
>I'm trying to find a way to automatically isolate and remove these periodic
>signals to get more insight in the (for my particular case) more interesting
>parts of the waveform, which are non-periodic. For example, the events in the
>last few thousand samples of the above graph.
>
>I am able to find and locate common patterns in the waveform using
>autocorrelation, but my problem is that these signals are actually
>just-not-really-periodic. For example, the time and interval of a fridge
>switching on and off depends on the outside temperature and is disturbed by
>opening the door, etc. This makes it kind of hard to simply find the timestamps
>of the fridges switching on, and subtracting the known-fridge-pattern from the
>wave.
>
>Can anybody suggest a algorithm or method for - preferrably fully automatic -
>finding and isolating the various 'almost periodic' parts of a waveform like
>this ?
>
>Ideally, I would like to be able to split up the signal in several parts, one
>for each periodic waveform, which when all added up, result in the original
>signal.
>
>Thanks,

Hi Zevv,
   I'm sorry to say that I don't have quick ideas, 
off the top of my head, on how you can isolate what 
you call "almost periodic" waveforms.   But I must say, 
your data plot is very interesting.  (Although it would 
be nice tohave the time axis labeled in either hours 
or days.)

The reason I'm replying to your post is to let you 
know, in case you don't know, about a portable power 
measurement device at:

http://www.amazon.com/P3-International-P4400-Electricity-Monitor/dp/B00009MDBU

I bought the P4400.  I like it a lot.  I have a large 
fan and wanted to know its (along with other appliances') 
power consumption.  The P4400 will measure the current 
through the fan, but also measures the fan's 
"power factor" so as to tell me what is the "actual" 
power consumption in watts.

Good Luck,
[-Rick-]

Rick Lyons <R.Lyons@_bogus_ieee.org> wrote:
> On 18 Sep 2009 19:06:01 GMT, Zevv <usenet@zeev.nl> wrote:
> 
>>Hello,
>>
>>I have collected several years worth of samples of a few housholds' power usage
>>with a samplerate of 2 seconds. An example snippet of the signal :
>>
>>        http://tinyurl.com/pmgraph
>
> I'm sorry to say that I don't have quick ideas, off the top of my
> head, on how you can isolate what you call "almost periodic"
> waveforms.   But I must say, your data plot is very interesting.
> (Although it would be nice tohave the time axis labeled in either
> hours or days.)

The samplerate is 0.5 Hz, so the graph has a span of approx 13.8 hours.

Here is the data of the last 24 hours, in the form I usually look at it:

  http://tinyurl.com/pmgraph2

Below the graph you can find the current power usage (146.2 watt), the
average over the day, the total use in KWH, and the total price for
that. The 'q' is the quality of the measurement in arbitrary units, this
indicates the strength of the signal received by the IR diode looking at
the dots of the wheel of the electricty meter.

> The reason I'm replying to your post is to let you know, in case you
> don't know, about a portable power measurement device at:
> 
> http://www.amazon.com/P3-International-P4400-Electricity-Monitor/dp/B00009MDBU

Yes, I know of these devices. Part of the fun of making my own solution
was the direct connection to a server for storing the measurements for
making the graphs. I can easily look at a years average, compare a year
with the previous year, etc.http://tinyurl.com/pmgraph2.