I am very new to dsp, can anyone show me some hints?

Hi all,

I can sample some data (about 700 data points in 6ms).

The data shows the following characteristics:

no event:
http://flic.kr/p/8XNrq7

small event:
http://flic.kr/p/8XKodB

big event:
http://flic.kr/p/8XKodZ

All these events can be detected by human eye, so I think some dsp
algorithm should be able to  detect it too. I only want to detect that
the effect occurs (at which point it occurs doesn't matter to me).

But I am completely new to dsp and therefore I have no idea how I
shall do it. Some simple moving average or averaging algorithms don't
help.

Can you give me some hints on which filter may potentially be used? Or
at least how shall I analyze on such problem to determine how to
design a good DSP algorithm? I don't have the time to read a whole
book right now but at least I can check the details of some specific
types of DSP filters.

Thank you very much!

Rgds,
MZ

ZMZ wrote:
> Hi all,
>
> I can sample some data (about 700 data points in 6ms).
>
> The data shows the following characteristics:
>
> no event:
> http://flic.kr/p/8XNrq7
>
> small event:
> http://flic.kr/p/8XKodB
>
> big event:
> http://flic.kr/p/8XKodZ
>
> All these events can be detected by human eye, so I think some dsp
> algorithm should be able to  detect it too. I only want to detect that
> the effect occurs (at which point it occurs doesn't matter to me).
>
> But I am completely new to dsp and therefore I have no idea how I
> shall do it. Some simple moving average or averaging algorithms don't
> help.
>
> Can you give me some hints on which filter may potentially be used? Or
> at least how shall I analyze on such problem to determine how to
> design a good DSP algorithm? I don't have the time to read a whole
> book right now but at least I can check the details of some specific
> types of DSP filters.
>
> Thank you very much!
>
> Rgds,
> MZ
>

Don't be too enamored with the term "dsp".
You have almost solved your problem.
The key is in your last paragraph - "how shall I *analyze* on 
such problem" (emphasis added).

Think about your statement that "All these events can be detected 
by human eye". Why is it obvious to the eye.

Suggestion - superimpose all three plots.
Sketch an idealized response that would cover all three cases.
Hint: 6 or 7 line segments should do it.
Do you see now what your eye saw?

HTH

On Dec 2, 2:46&#4294967295;am, ZMZ <zhangmin...@gmail.com> wrote:
> Hi all,
>
> I can sample some data (about 700 data points in 6ms).
>
> The data shows the following characteristics:
>
> no event:http://flic.kr/p/8XNrq7
>
> small event:http://flic.kr/p/8XKodB
>
> big event:http://flic.kr/p/8XKodZ
>
> All these events can be detected by human eye, so I think some dsp
> algorithm should be able to &#4294967295;detect it too. I only want to detect that
> the effect occurs (at which point it occurs doesn't matter to me).
>
> But I am completely new to dsp and therefore I have no idea how I
> shall do it. Some simple moving average or averaging algorithms don't
> help.
>
> Can you give me some hints on which filter may potentially be used? Or
> at least how shall I analyze on such problem to determine how to
> design a good DSP algorithm? I don't have the time to read a whole
> book right now but at least I can check the details of some specific
> types of DSP filters.
>
> Thank you very much!
>
> Rgds,
> MZ

In reviewing the data you may be better off to just write up some code
that measures a downslope on a pulse and then looks for what the pulse
does after it achieves the bottom value for some period of time.  You
may be trying to use fancy DSP theory when it may not be needed.

On Dec 2, 8:46&#4294967295;am, ZMZ <zhangmin...@gmail.com> wrote:
> Hi all,
>
> I can sample some data (about 700 data points in 6ms).
>
> The data shows the following characteristics:
>
> no event:http://flic.kr/p/8XNrq7
>
> small event:http://flic.kr/p/8XKodB
>
> big event:http://flic.kr/p/8XKodZ
>
> All these events can be detected by human eye, so I think some dsp
> algorithm should be able to &#4294967295;detect it too.

That's a very common argument. Usually wrong, though.
That the human eye can detect something doesn't always
mean that DSP or related techniques can.

I think this link shows an image that demonstrates
a very hard case to replicate:

 www.zazzle.com/cards?cg=103337974720533275

If the link doesn't work, google for images with keywords
"dalmatian cognitive"

> I only want to detect that
> the effect occurs (at which point it occurs doesn't matter to me).
>
> But I am completely new to dsp and therefore I have no idea how I
> shall do it. Some simple moving average or averaging algorithms don't
> help.

Nope. Exactly the opposite: Differentiate the sequences first.

Rune

On Dec 2, 2:46&#4294967295;am, ZMZ <zhangmin...@gmail.com> wrote:
> Hi all,
>
> I can sample some data (about 700 data points in 6ms).
>
> The data shows the following characteristics:
>
> no event:http://flic.kr/p/8XNrq7
>
> small event:http://flic.kr/p/8XKodB
>
> big event:http://flic.kr/p/8XKodZ
>
> All these events can be detected by human eye, so I think some dsp
> algorithm should be able to &#4294967295;detect it too. I only want to detect that
> the effect occurs (at which point it occurs doesn't matter to me).
>
> But I am completely new to dsp and therefore I have no idea how I
> shall do it.

ask yourself what is common to all three functions (a step from about
800 to about 500) and what is different (the "event" seems to ring
with some resonance, "no event" has little resonance).

maybe take a look at the difference between these in the frequency
domain.  that might give you an idea about what is different.

r b-j

Rune Allnor <allnor@tele.ntnu.no> wrote:
(snip)

>> All these events can be detected by human eye, so I think some dsp
>> algorithm should be able to &#4294967295;detect it too.
 
> That's a very common argument. Usually wrong, though.
> That the human eye can detect something doesn't always
> mean that DSP or related techniques can.

Some years ago at a talk by someone from a well known
biotechnology company, there was a story about being asked
by the computer people in the company if they (biologists)
had any problems that needed to be solved.  One being told
the problems, the usual answer was that it was too easy,
and not worth working on.

So, as the story goes, one time after that answer, the person
actually did try to work on the problem.  The problem was,
given a microscope picture of a strand of DNA, figure out how
long it is.  (Imagine a rope on the floor, crossing over itself
some number of times, and, from a picture trying to figure out
how long it is.)  After not so long, it was found to actually
be pretty hard!  The eye can usually follow where one crosses
over, and continues on, but it is much more difficult for a
computer to do that.  It sounds so easy, though!

-- glen

Thank you for all the replies. I think I know a little more about DSP
but I still didn't get the method.

It seems that I miscommunicated a little bit.

The absolute values in the figure with no event changes through time/
temperature and etc. What's more, due to energy constraint, I can't
sample the event continuously.

What I was doing is, sample once, I have a figure, I have to detect
whether the event occurs only based on one figure, but not two
(because the previously saved figure may not compare with current one
due to temperature/time change).

What I can see through human eyes is that, when the event occurs,
there will be some jitters. What I want to do is to differentiate the
jitters from a graph without jitters. The jitter can be small.

Thanks anyway, I think I got to investigate further on this problem by
myself.

>Thank you for all the replies. I think I know a little more about DSP
>but I still didn't get the method.
>
>It seems that I miscommunicated a little bit.
>
>The absolute values in the figure with no event changes through time/
>temperature and etc. What's more, due to energy constraint, I can't
>sample the event continuously.
>
>What I was doing is, sample once, I have a figure, I have to detect
>whether the event occurs only based on one figure, but not two
>(because the previously saved figure may not compare with current one
>due to temperature/time change).
>
>What I can see through human eyes is that, when the event occurs,
>there will be some jitters. What I want to do is to differentiate the
>jitters from a graph without jitters. The jitter can be small.
>
>Thanks anyway, I think I got to investigate further on this problem by
>myself.
>

Seems like the definition of your "event" is very different from what
everybody else; including me got from your first post. Can you please
clearly explain what is it that you are specifically interested in those
graphs. Are you looking for the huge change in the signal that is clearly
visible or are you referring to the tiny fluctuations in amplitude all
along the signal?

On Dec 3, 4:18&#4294967295;pm, "mir_aculous" <gnu.fanz@n_o_s_p_a_m.gmail.com>
wrote:
> >Thank you for all the replies. I think I know a little more about DSP
> >but I still didn't get the method.
>
> >It seems that I miscommunicated a little bit.
>
> >The absolute values in the figure with no event changes through time/
> >temperature and etc. What's more, due to energy constraint, I can't
> >sample the event continuously.
>
> >What I was doing is, sample once, I have a figure, I have to detect
> >whether the event occurs only based on one figure, but not two
> >(because the previously saved figure may not compare with current one
> >due to temperature/time change).
>
> >What I can see through human eyes is that, when the event occurs,
> >there will be some jitters. What I want to do is to differentiate the
> >jitters from a graph without jitters. The jitter can be small.
>
> >Thanks anyway, I think I got to investigate further on this problem by
> >myself.
>
> Seems like the definition of your "event" is very different from what
> everybody else; including me got from your first post. Can you please
> clearly explain what is it that you are specifically interested in those
> graphs. Are you looking for the huge change in the signal that is clearly
> visible or are you referring to the tiny fluctuations in amplitude all
> along the signal?

Hi,

Thanks for reply. The small variations can be viewed as noise. That's
the reason that I put the "no event" figure. Any variation larger than
"no event" type of variation can be considered as event.

I tried for some hours and I found that the following algorithm can
differentiate my events, I derived this algorithm not based on theory,
but based on my observations.  I think there should be some better way
in theory to support it and I want to learn the standard approach in
industry for this type of DSP design.

My algorithm is as follow:

1) cut all the first 160 points (including the big fall, which doesn't
detect my event at all)
2) average_data = average(raw_data, 10); // get the average of raw
data to remove noise
3) falling_differential_square_data =
falling_differential_square(average_data);
//this step is to get the differential square on falling edge of
variation, i.e.,
// return (sample[n+1] - sample[n])^2 if sample[n+1] < sample[n], else
return 0
//I only measure the falling edge because there's a large rising after
the big falling
//  and I can't simply cut the large rising because my event occurs
here too
4) ma_data = moving_average(falling_differential_square_data, 2);
//I observe that the falling edge caused by random noise
// will not tend to be consecutive, but the falling edge caused by
event tend to be consecutive,
// therefore a moving average of 2 neighbors will help me identify the
event

after step 4, the max data will not be exceeding 10 when no event
occurs, and will be ranged from 30-1000+ when event occurs

I hope I am at least on the track in dealing with the data. Please do
give me some suggestions if I am not right at any place.

Thank you again.

On Dec 2, 8:46&#4294967295;pm, Rune Allnor <all...@tele.ntnu.no> wrote:
> On Dec 2, 8:46&#4294967295;am, ZMZ <zhangmin...@gmail.com> wrote:
>
> > Hi all,
>
> > I can sample some data (about 700 data points in 6ms).
>
> > The data shows the following characteristics:
>
> > no event:http://flic.kr/p/8XNrq7
>
> > small event:http://flic.kr/p/8XKodB
>
> > big event:http://flic.kr/p/8XKodZ
>
> > All these events can be detected by human eye, so I think some dsp
> > algorithm should be able to &#4294967295;detect it too.
>
> That's a very common argument. Usually wrong, though.
> That the human eye can detect something doesn't always
> mean that DSP or related techniques can.
>
> I think this link shows an image that demonstrates
> a very hard case to replicate:
>
> &#4294967295;www.zazzle.com/cards?cg=103337974720533275
>
> If the link doesn't work, google for images with keywords
> "dalmatian cognitive"
>
> > I only want to detect that
> > the effect occurs (at which point it occurs doesn't matter to me).
>
> > But I am completely new to dsp and therefore I have no idea how I
> > shall do it. Some simple moving average or averaging algorithms don't
> > help.
>
> Nope. Exactly the opposite: Differentiate the sequences first.
>
> Rune

For people that think computers can do everything, I like to go the
example of an upside-down picture. Toddlers (~18-20 months) can tell
if a picture is inverted, while it's extremely difficult for a
computer to do so.

Bryan