ROC curves

Hi guys

I am doing my phd related to sonar signal processing. I have implemented
active sonar signal processing chain involving complex demodulation,
beamforming, match filter , normalization etc in MATLAB. I have tested sp
chain with simulated data also.

The problem i am facing now is how to set detection threshold for my active
system?I have gone through some literature describing ROC curves to find DT
but these ROC curves are related to some typical sonar system.

How should i calculate ROC curves for my active system?From my
understanding, these ROC curves should not only related to SP chain
implemented but also to environmental conditions in which this active
system will operate. 

Please help me as i am lost here.

On Mar 20, 7:15&#4294967295;am, "abraham007" <abrahamnick73@n_o_s_p_a_m.yahoo.com>
wrote:
> Hi guys
>
> ...
>
> The problem i am facing now is how to set detection threshold for my active
> system?I have gone through some literature describing ROC curves to find DT
> but these ROC curves are related to some typical sonar system.
>
> ...

You will have to determine the signal and noise/interference
probability density functions for your model. There are numerous
examples of this process to examine. This is a topic area developed
and published by a variety of national defense labs:

AUSTRALIA:

Detection Threshold Modelling Explained
DSTO-TR-0586
http://traktoria.org/files/sonar/signal_processing/detection_threshold_modelling_explained.pdf
also available from DTIC as ADA335337
Handle / proxy Url:   http://handle.dtic.mil/100.2/ADA335337

CANADA:

"Active Sonar Modelling
With Emphasis on Sonar Stimulators"
http://pubs.drdc.gc.ca/PDFS/unc35/p522681.pdf

The classic passive sonar reference:
DREA Technical Memoramdum 82/A
"Detection Performance of FFT Processors for Narrowband Signals"
R. S. Walker
http://cradpdf.drdc.gc.ca/PDFS/unc41/p36264.pdf

USA:

ESTIMATION OF MULTI-PING, ACTIVE SONAR DETECTION ROC CURVES
USING EMPIRICAL TARGET DATA
ftp://ftp.arlut.utexas.edu/ATL/JASA/multiping_roc.pdf
or from DTIC:
Handle / proxy Url:   http://handle.dtic.mil/100.2/ADA421690

These sources have many other papers as well.

Good luck!
Dale B. Dalrymple

On 03/20/2011 10:15 AM, abraham007 wrote:
> Hi guys
>
> I am doing my phd related to sonar signal processing. I have implemented
> active sonar signal processing chain involving complex demodulation,
> beamforming, match filter , normalization etc in MATLAB. I have tested sp
> chain with simulated data also.
>
> The problem i am facing now is how to set detection threshold for my active
> system?I have gone through some literature describing ROC curves to find DT
> but these ROC curves are related to some typical sonar system.
>
> How should i calculate ROC curves for my active system?From my
> understanding, these ROC curves should not only related to SP chain
> implemented but also to environmental conditions in which this active
> system will operate.
>
> Please help me as i am lost here.

[vantrees] has some textbook examples of how to compute the ROC.

--Randy

@book{vantrees,
   title = "Detection, Estimation, and Modulation Theory, Part I",
   author = "Harry L. Van Trees",
   publisher = "Wiley",
   year = "2001"}

-- 
Randy Yates                             Digital Signal Labs
919-577-9882                            http://www.digitalsignallabs.com
yates@digitalsignallabs.com

>On Mar 20, 7:15=A0am, "abraham007" <abrahamnick73@n_o_s_p_a_m.yahoo.com>
>wrote:
>> Hi guys
>>
>> ...
>>
>> The problem i am facing now is how to set detection threshold for my
acti=
>ve
>> system?I have gone through some literature describing ROC curves to find
=
>DT
>> but these ROC curves are related to some typical sonar system.
>>
>> ...
>
>You will have to determine the signal and noise/interference
>probability density functions for your model. There are numerous
>examples of this process to examine. This is a topic area developed
>and published by a variety of national defense labs:
>
>AUSTRALIA:
>
>Detection Threshold Modelling Explained
>DSTO-TR-0586
>http://traktoria.org/files/sonar/signal_processing/detection_threshold_mode=
>lling_explained.pdf
>also available from DTIC as ADA335337
>Handle / proxy Url:   http://handle.dtic.mil/100.2/ADA335337
>
>CANADA:
>
>"Active Sonar Modelling
>With Emphasis on Sonar Stimulators"
>http://pubs.drdc.gc.ca/PDFS/unc35/p522681.pdf
>
>The classic passive sonar reference:
>DREA Technical Memoramdum 82/A
>"Detection Performance of FFT Processors for Narrowband Signals"
>R. S. Walker
>http://cradpdf.drdc.gc.ca/PDFS/unc41/p36264.pdf
>
>USA:
>
>ESTIMATION OF MULTI-PING, ACTIVE SONAR DETECTION ROC CURVES
>USING EMPIRICAL TARGET DATA
>ftp://ftp.arlut.utexas.edu/ATL/JASA/multiping_roc.pdf
>or from DTIC:
>Handle / proxy Url:   http://handle.dtic.mil/100.2/ADA421690
>
>These sources have many other papers as well.
>
>Good luck!
>Dale B. Dalrymple

Thanks for all help. With the help of references mentioned and other
literatures i tried to plot ROC curves for different SNRs for my processing
system.First of all i generated gaussian noise signals (single ping data) 
with no target in it as a linear array hydrophones outputs. I performed
complex demodulation, beamforming, match filter with normalization on this
data.I got beams X range matrix at the output of this process. From
histogram of one beam of data, i selected a suitable detection threshold
(DT) and count the range samples which were above this DT in whole beam X
range matrix and get probability of false alarm (Pfa) by dividing this
count with total no of samples in matrix.

Then in the beamformer output, i inserted synthetic target (an LFM pulse)
with same SNR in each beam and perform replica correlation. I searched only
those ranges in each beam at which synthetic target was inserted and count
the samples which exceede the DT set above.Probability of detection Pd was
calculated by dividing this count with total_beams*no of ranges
(corresponding to a pulse length only).

I repeated the same procedure by varying the DT and keeping SNR constant
and got a ROC curve for one SNR. Same procedure then repeated for different
SNRs.

Please give me a suggestion whether i am doing correct or not and how can i
improve ?

THANKS

Thanks for all help. With the help of references mentioned and other
literatures i tried to plot ROC curves for different SNRs for my processing
system.First of all i generated gaussian noise signals (single ping data) 
with no target in it as a linear array hydrophones outputs. I performed
complex demodulation, beamforming, match filter with normalization on this
data.I got beams X range matrix at the output of this process. From
histogram of one beam of data, i selected a suitable detection threshold
(DT) and count the range samples which were above this DT in whole beam X
range matrix and get probability of false alarm (Pfa) by dividing this
count with total no of samples in matrix.

Then in the beamformer output, i inserted synthetic target (an LFM pulse)
with same SNR in each beam and perform replica correlation. I searched only
those ranges in each beam at which synthetic target was inserted and count
the samples which exceede the DT set above.Probability of detection Pd was
calculated by dividing this count with total_beams*no of ranges
(corresponding to a pulse length only).

I repeated the same procedure by varying the DT and keeping SNR constant
and got a ROC curve for one SNR. Same procedure then repeated for different
SNRs.

Please give me a suggestion whether i am doing correct or not and how can i
improve ?

THANKS

On Apr 5, 3:36 am, "abraham007" <abrahamnick73@n_o_s_p_a_m.yahoo.com>
wrote:
> ...
>
> Please give me a suggestion whether i am doing correct or not and how can i
> improve ?
> ...

One of the reasons for the use of ROCs is that they are an overall
"system" characteristic. It is important to be sure that what you
simulate corresponds to the "system" you are interested in.

 You have collected false alarm statistics for a single beam. Is a
single beam the correct output for your "system"? If there are
multiple beams, how are their statistics combined to get the "system"
statistics? Are they independent? (Really?).

It is much easier to insert the signal after the beamformer than to
correctly simulate the return in a realistic environment at the
individual sensor level. But if you do insert after the beamformer,
you only generate the ROC for your assumptions about the system rather
than the ROC of your implementation of the system. What is your goal?

Finally, no matter how thoroughly you simulate your implementation, it
is always good to be wary of how well the assumptions you made in
generating noise at the inputs actually represent the noise in the
environment the system will be used in.

Dale B. Dalrymple

Thanks for your help.

> You have collected false alarm statistics for a single beam. Is a
>single beam the correct output for your "system"? If there are
>multiple beams, how are their statistics combined to get the "system"
>statistics? Are they independent? (Really?).

I have simply calculated total no of samples from full matrix (beams*range)
and divide result with matrix size to get probability of false alarm.what
you suggest i should do to get more accurate results?

>Finally, no matter how thoroughly you simulate your implementation, it
>is always good to be wary of how well the assumptions you made in
>generating noise at the inputs actually represent the noise in the
>environment the system will be used in.

Yes you are right.I think i have simulated the array's hydrophones data
without considering many environmental effects like reverberation, array
flow noise, tow ship noise etc.I think it is another topic on which whole
Phd can be done.
Is there any sonar array simulater available (preferably free) which can
simulate array's sensor data so that i can verify SP chain?

Best Regards
nick





>On Apr 5, 3:36 am, "abraham007" <abrahamnick73@n_o_s_p_a_m.yahoo.com>
>wrote:
>> ...
>>
>> Please give me a suggestion whether i am doing correct or not and how
can i
>> improve ?
>> ...
>
>One of the reasons for the use of ROCs is that they are an overall
>"system" characteristic. It is important to be sure that what you
>simulate corresponds to the "system" you are interested in.
>
> You have collected false alarm statistics for a single beam. Is a
>single beam the correct output for your "system"? If there are
>multiple beams, how are their statistics combined to get the "system"
>statistics? Are they independent? (Really?).
>
>It is much easier to insert the signal after the beamformer than to
>correctly simulate the return in a realistic environment at the
>individual sensor level. But if you do insert after the beamformer,
>you only generate the ROC for your assumptions about the system rather
>than the ROC of your implementation of the system. What is your goal?
>
>Finally, no matter how thoroughly you simulate your implementation, it
>is always good to be wary of how well the assumptions you made in
>generating noise at the inputs actually represent the noise in the
>environment the system will be used in.
>
>Dale B. Dalrymple
>

On Apr 29, 10:54�am, "abraham007"
<abrahamnick73@n_o_s_p_a_m.yahoo.com> wrote:
> Thanks for your help.
>
> > You have collected false alarm statistics for a single beam. Is a
> >single beam the correct output for your "system"? If there are
> >multiple beams, how are their statistics combined to get the "system"
> >statistics? Are they independent? (Really?).
>
> I have simply calculated total no of samples from full matrix (beams*range)
> and divide result with matrix size to get probability of false alarm.what
> you suggest i should do to get more accurate results?
>
> >Finally, no matter how thoroughly you simulate your implementation, it
> >is always good to be wary of how well the assumptions you made in
> >generating noise at the inputs actually represent the noise in the
> >environment the system will be used in.
>
> Yes you are right.I think i have simulated the array's hydrophones data
> without considering many environmental effects like reverberation, array
> flow noise, tow ship noise etc.I think it is another topic on which whole
> Phd can be done.

Wrong. Dozens of PhD these have been written on
this already - daceades ago. Do a litterature search.

> Is there any sonar array simulater available (preferably free) which can
> simulate array's sensor data so that i can verify SP chain?

Not free, no. The people who do these kinds of things
don't publish (much of) what they do.

Rune

On Apr 29, 1:54 am, "abraham007" <abrahamnick73@n_o_s_p_a_m.yahoo.com>
wrote:
> Thanks for your help.
> ...
>
> I have simply calculated total no of samples from full matrix (beams*range)
> and divide result with matrix size to get probability of false alarm.what
> you suggest i should do to get more accurate results?
>
That's OK for a start. The real fun begins when you have multiple
channels with signals, interference and noise that appear on multiple
channels and are not independent. Modern PhD level work usually deals
with solutions to these kinds of less tractable problems.

>...
> Is there any sonar array simulater available (preferably free) which can
> simulate array's sensor data so that i can verify SP chain?
>
> Best Regards
> nick

There are a lot of array beamforming implementations on the net.
Whether they are applicable or useful is something that you will need
to decide. Don't be one of the comp.dsp clowns who thinks that the
answer will fall on the first page or for complex topics even in the
first ten pages. Many sources use different vocabulary. You may need
to evaluate tens of pages of results to just learn the right search
terms.

Good luck!

Dale B. Dalrymple