On Aug 18, 4:19 am, Richard Owlett <rowl...@atlascomm.net> wrote:
> ...
> What is "split window"?

A split window averager is a rectangular filter with one or more
center values set to zero so that the bin being  normalized is not
included in the background estimate and so, can not bias the
background estimate.

>
>
> > This assumes that the signals of interest are narrower in spectral
> > width than the background features to be removed. The window might be
> > made the full passband.
>
> I'm not sure if we use "feature" in the same sense.
> For each time slice, I FFT the input.
> A particular bin _may_ be in a feature if above a threshold value.
> To be a feature bin(s) in adjacent slice(s) must also be above
> threshold. [e.g. artic loon's call goes up in frequency with time]
>

By "feature" I mean any signal that you don't want normalized away.

Dale B. Dalrymple
http://dbdimages.com

Richard Owlett wrote:

> >>HOWEVER it suggests an idea. For an area whose amplitude is less than
> >>some value, give higher gain to a region with a peak whose amplitude is
> >>a absolute amount different that the average of a larger surrounding
> >>area. [Compare to using a magnifying glass to examine the immediate area
> >>of a blip on a topographic map.]
> >
> >
> > That description sounds like an unsharp mask (AKA unsharp filter).
> > Basically you would subtract some weighted amount of the mean in the
> > neighborhood surrounding each point. It's a image processing
> > technique, but what you have is equivalent to a grayscale bitmap. I
> > assume your 3d surface is a uniform 2d grid with each point having its
> > own z height.
> >
> 
> Googled. YES!
> http://tavmjong.free.fr/INKSCAPE/MANUAL/html/Filters-Pixel.html has a
> tantalizing description. The hits are either for canned programs of
> research discussions. Any suggestions for introductory material that's
> not tied to a specific program. Meanwhile I'll wander Scilab toolboxes
> for image manipulation routines.


	As I said I'm assuming your 3d representation is in the form of a  2d
array that contains the height of the 3d surface  at any point. This
is equivalent to a grayscale bitmap. I don't know Scilab but I would
expect it has the tools for doing many of the image processing
functions shown on the website you found.  A grayscale image array
contains values that are integers in the range 0-255, but Scilab may
also have the ability to accept arrays composed of floats with a range
of 0-1.0.

	The unsharp filter is an old photo developer's dark room technique
that consists of subtracting from the image a blurred version of the
original. This is done in a way so that the mean or DC level of the
result remains unchanged. Obviously there are lots of ways to blur an
image so there isn't one single right way to do this. 
	In the page you posted the unsharp filter they show is equivalent to
taking the image with a gain of 18 and subtracting from that a blurred
version that has been blurred by a 3x3 boxcar filter. The subtracted
blurred image has a gain of 9. So subtraction results in an image with
a DC gain of 9 so then they divide by 9 (divisor=9) to produce a
result with a DC gain of one. To produce the same result, those steps
could be done as a series of individual operations or all as one
filter operation.

-jim


----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.pronews.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= - Total Privacy via Encryption =---

jim wrote:

> 
> Richard Owlett wrote:
> 
>>Richard Owlett wrote:
>>
>>>jim wrote:
>>>
>>>
>>>>Richard Owlett wrote:
>>>>
>>>>
>>>>>I'm plotting a time series of fft's, similar to spectrograms but in 3D.
>>>>>If I use a linear scale I clearly see details of primary signal.
>>>>>But If I use a log scale I can now see the existence of much smaller
>>>>>features but at the expense of losing detail of the larger features.
>>>>>
>>>>>Are there any common methods for dealing with this problem?
>>>>
>>>>
>>>>
>>>>It isn't clear what "this problem" is. Statements like "primary
>>>>signal" and "smaller features" sound like value judgments.
>>>
>>>
>>>*Definitely*
>>>
>>> > My guess is your criteria for those judgments is time.
>>>
>>>No ;) It's more like "hmm there's something going on in the "background"
>>>that I didn't expect but may be important". Its amplitude is much
>>>smaller than the primary feature but it's also displaced in both time
>>>and frequency.
>>>
>>>Rune had pointed out that he had used spectrograms of the call of an
>>>artic loon to illustrate (among other things) overtones. I didn't see it
>>>with my presentation method and source recording. While waiting for a
>>>better recording I attempted to improve my plot. The strong portion of
>>>the loon's call is from 1000-1500 Hz. But there is also something
>>>starting near 2200 Hz but offset in time by ~.1 sec. I'm trying to
>>>investigate that.
>>>
>>
>>My ride came so I had to end my last post. While I was out your comment
>>about a "running average" got me thinking on different paths.
>>
>>
>>>>     Is the primary signal something you perceive as lasting a while and
>>>>the smaller features are more transient and appear only for a short
>>>>while?.
>>
>>The projection onto the time-frequency plane are comparable. The only
>>thing definite is that the amplitude of the "smaller" features are smaller.
> 
> 
> I thought you were looking at human voice recordings - I missed that
> you were looking at a recording of a loon. 
> 
> 
> 
>>>>If so perhaps a running average of the log scale
>>>>representation would be the way achieve a happier median between
>>>>smaller and larger features. Of course the ideal length for the
>>>>running average would need to be determined. The  longer your running
>>>>average the more the long term features would be enhanced at the
>>>>expense of short term features.
>>
>>Don't think so.
>>HOWEVER it suggests an idea. For an area whose amplitude is less than
>>some value, give higher gain to a region with a peak whose amplitude is
>>a absolute amount different that the average of a larger surrounding
>>area. [Compare to using a magnifying glass to examine the immediate area
>>of a blip on a topographic map.]
> 
> 
> That description sounds like an unsharp mask (AKA unsharp filter).
> Basically you would subtract some weighted amount of the mean in the
> neighborhood surrounding each point. It's a image processing
> technique, but what you have is equivalent to a grayscale bitmap. I
> assume your 3d surface is a uniform 2d grid with each point having its
> own z height.
> 

Googled. YES!
http://tavmjong.free.fr/INKSCAPE/MANUAL/html/Filters-Pixel.html has a 
tantalizing description. The hits are either for canned programs of 
research discussions. Any suggestions for introductory material that's 
not tied to a specific program. Meanwhile I'll wander Scilab toolboxes 
for image manipulation routines.

dbd wrote:
> On Aug 17, 5:29 am, Richard Owlett <rowl...@atlascomm.net> wrote:
> 
>>I'm plotting a time series of fft's, similar to spectrograms but in 3D.
>>If I use a linear scale I clearly see details of primary signal.
>>But If I use a log scale I can now see the existence of much smaller
>>features but at the expense of losing detail of the larger features.
>>
>>Are there any common methods for dealing with this problem?
>>
>>As I'm looking at only the magnitude of the fft, my data is _strictly_
>>positive. I've experimented with simplistic compression.
>>
>>For an input "x" I've experimented with a gain of form
>>
>>gain = k*((max(x)-x)/max(x))^n
>>
>>The results weren't great, but they were encouraging.
>>It did highlight a potential complication.
>>I will want something similar to the attack and release(???) settings of
>>  AGC. But it will have to be dependent on time and frequency. To really
>>mix metaphors, it will be dependent on radial distance in the time vs
>>frequency plane. (as I have stored data, looking to the future is allowed ;)
>>
>>Comments?
> 
> 
> Richard
> 
> The common passive sonar approach is to estimate the 'background' of
> the (possibly time averaged) power spectrum. This has been done across
> frequency with:
> 1) boxcar window
> 2) split window
> 3) thresholded split window
> 4) windowed thresholded split window
> 5) thresholded windowed thresholded split window
> ...
> N) median

What is "split window"?


> 
> This assumes that the signals of interest are narrower in spectral
> width than the background features to be removed. The window might be
> made the full passband.

I'm not sure if we use "feature" in the same sense.
For each time slice, I FFT the input.
A particular bin _may_ be in a feature if above a threshold value.
To be a feature bin(s) in adjacent slice(s) must also be above 
threshold. [e.g. artic loon's call goes up in frequency with time]

> 
> The background estimate can also be averaged in the time domain. For
> reasons of efficiency, the most common method may have been the
> exponential averager because of it's controlled variability while
> requiring minimal storage of history data. Others no doubt work better
> at greater processing and storage cost.

I'm working with stored data so time is not a tight constraint.
I can hold the equivalent of ~8-10 copies of data in memory.

> 
> The power spectrum is divided by the background estimate to give an
> estimate of signal to noise ratio plus 1.0 as:
> (Sinal + Noise) / Noise-est. => S/N + 1

That gives me an idea. If the value in a bin is below a threshold, I can 
set it to an "out of range" value. Scilab's plot command can be set to 
ignore those points.

> 
> This assumes the data is still in the linear domain. The plus one is a
> convenience to assure that the data can then be converted to dB. Plot
> as you wish.
> 
> The US and NATO navies have used the above methods.
> 
> YMMV
> 
> Dale B. Dalrymple
> http://dbdimages.com
> 

jim wrote:
> 
> Richard Owlett wrote:
> 
>>Richard Owlett wrote:
>>
>>>jim wrote:
>>>
>>>
>>>>Richard Owlett wrote:
>>>>
>>>>
>>>>>I'm plotting a time series of fft's, similar to spectrograms but in 3D.
>>>>>If I use a linear scale I clearly see details of primary signal.
>>>>>But If I use a log scale I can now see the existence of much smaller
>>>>>features but at the expense of losing detail of the larger features.
>>>>>
>>>>>Are there any common methods for dealing with this problem?
>>>>
>>>>
>>>>
>>>>It isn't clear what "this problem" is. Statements like "primary
>>>>signal" and "smaller features" sound like value judgments.
>>>
>>>
>>>*Definitely*
>>>
>>> > My guess is your criteria for those judgments is time.
>>>
>>>No ;) It's more like "hmm there's something going on in the "background"
>>>that I didn't expect but may be important". Its amplitude is much
>>>smaller than the primary feature but it's also displaced in both time
>>>and frequency.
>>>
>>>Rune had pointed out that he had used spectrograms of the call of an
>>>artic loon to illustrate (among other things) overtones. I didn't see it
>>>with my presentation method and source recording. While waiting for a
>>>better recording I attempted to improve my plot. The strong portion of
>>>the loon's call is from 1000-1500 Hz. But there is also something
>>>starting near 2200 Hz but offset in time by ~.1 sec. I'm trying to
>>>investigate that.
>>>
>>
>>My ride came so I had to end my last post. While I was out your comment
>>about a "running average" got me thinking on different paths.
>>
>>
>>>>     Is the primary signal something you perceive as lasting a while and
>>>>the smaller features are more transient and appear only for a short
>>>>while?.
>>
>>The projection onto the time-frequency plane are comparable. The only
>>thing definite is that the amplitude of the "smaller" features are smaller.
> 
> 
> I thought you were looking at human voice recordings - I missed that
> you were looking at a recording of a loon. 

This is a rabbit trail did start with an interest in speech recognition.

> 
> 
> 
>>>>If so perhaps a running average of the log scale
>>>>representation would be the way achieve a happier median between
>>>>smaller and larger features. Of course the ideal length for the
>>>>running average would need to be determined. The  longer your running
>>>>average the more the long term features would be enhanced at the
>>>>expense of short term features.
>>
>>Don't think so.
>>HOWEVER it suggests an idea. For an area whose amplitude is less than
>>some value, give higher gain to a region with a peak whose amplitude is
>>a absolute amount different that the average of a larger surrounding
>>area. [Compare to using a magnifying glass to examine the immediate area
>>of a blip on a topographic map.]
> 
> 
> That description sounds like an unsharp mask (AKA unsharp filter).
> Basically you would subtract some weighted amount of the mean in the
> neighborhood surrounding each point. It's a image processing
> technique, but what you have is equivalent to a grayscale bitmap. I
> assume your 3d surface is a uniform 2d grid with each point having its
> own z height.
> 
> -jim
> 

That gives me some key words to Google with. Thanks.

Richard Owlett wrote:
> 
> Richard Owlett wrote:
> > jim wrote:
> >
> >>
> >> Richard Owlett wrote:
> >>
> >>> I'm plotting a time series of fft's, similar to spectrograms but in 3D.
> >>> If I use a linear scale I clearly see details of primary signal.
> >>> But If I use a log scale I can now see the existence of much smaller
> >>> features but at the expense of losing detail of the larger features.
> >>>
> >>> Are there any common methods for dealing with this problem?
> >>
> >>
> >>
> >> It isn't clear what "this problem" is. Statements like "primary
> >> signal" and "smaller features" sound like value judgments.
> >
> >
> > *Definitely*
> >
> >  > My guess is your criteria for those judgments is time.
> >
> > No ;) It's more like "hmm there's something going on in the "background"
> > that I didn't expect but may be important". Its amplitude is much
> > smaller than the primary feature but it's also displaced in both time
> > and frequency.
> >
> > Rune had pointed out that he had used spectrograms of the call of an
> > artic loon to illustrate (among other things) overtones. I didn't see it
> > with my presentation method and source recording. While waiting for a
> > better recording I attempted to improve my plot. The strong portion of
> > the loon's call is from 1000-1500 Hz. But there is also something
> > starting near 2200 Hz but offset in time by ~.1 sec. I'm trying to
> > investigate that.
> >
> 
> My ride came so I had to end my last post. While I was out your comment
> about a "running average" got me thinking on different paths.
> 
> >>
> >>      Is the primary signal something you perceive as lasting a while and
> >> the smaller features are more transient and appear only for a short
> >> while?.
> 
> The projection onto the time-frequency plane are comparable. The only
> thing definite is that the amplitude of the "smaller" features are smaller.

I thought you were looking at human voice recordings - I missed that
you were looking at a recording of a loon. 


> 
> >> If so perhaps a running average of the log scale
> >> representation would be the way achieve a happier median between
> >> smaller and larger features. Of course the ideal length for the
> >> running average would need to be determined. The  longer your running
> >> average the more the long term features would be enhanced at the
> >> expense of short term features.
> 
> Don't think so.
> HOWEVER it suggests an idea. For an area whose amplitude is less than
> some value, give higher gain to a region with a peak whose amplitude is
> a absolute amount different that the average of a larger surrounding
> area. [Compare to using a magnifying glass to examine the immediate area
> of a blip on a topographic map.]

That description sounds like an unsharp mask (AKA unsharp filter).
Basically you would subtract some weighted amount of the mean in the
neighborhood surrounding each point. It's a image processing
technique, but what you have is equivalent to a grayscale bitmap. I
assume your 3d surface is a uniform 2d grid with each point having its
own z height.

-jim


----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.pronews.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= - Total Privacy via Encryption =---

On Aug 17, 5:29 am, Richard Owlett <rowl...@atlascomm.net> wrote:
> I'm plotting a time series of fft's, similar to spectrograms but in 3D.
> If I use a linear scale I clearly see details of primary signal.
> But If I use a log scale I can now see the existence of much smaller
> features but at the expense of losing detail of the larger features.
>
> Are there any common methods for dealing with this problem?
>
> As I'm looking at only the magnitude of the fft, my data is _strictly_
> positive. I've experimented with simplistic compression.
>
> For an input "x" I've experimented with a gain of form
>
> gain = k*((max(x)-x)/max(x))^n
>
> The results weren't great, but they were encouraging.
> It did highlight a potential complication.
> I will want something similar to the attack and release(???) settings of
>   AGC. But it will have to be dependent on time and frequency. To really
> mix metaphors, it will be dependent on radial distance in the time vs
> frequency plane. (as I have stored data, looking to the future is allowed ;)
>
> Comments?

Richard

The common passive sonar approach is to estimate the 'background' of
the (possibly time averaged) power spectrum. This has been done across
frequency with:
1) boxcar window
2) split window
3) thresholded split window
4) windowed thresholded split window
5) thresholded windowed thresholded split window
...
N) median

This assumes that the signals of interest are narrower in spectral
width than the background features to be removed. The window might be
made the full passband.

The background estimate can also be averaged in the time domain. For
reasons of efficiency, the most common method may have been the
exponential averager because of it's controlled variability while
requiring minimal storage of history data. Others no doubt work better
at greater processing and storage cost.

The power spectrum is divided by the background estimate to give an
estimate of signal to noise ratio plus 1.0 as:
(Sinal + Noise) / Noise-est. => S/N + 1

This assumes the data is still in the linear domain. The plus one is a
convenience to assure that the data can then be converted to dB. Plot
as you wish.

The US and NATO navies have used the above methods.

YMMV

Dale B. Dalrymple
http://dbdimages.com

Richard Owlett wrote:
> jim wrote:
> 
>>
>> Richard Owlett wrote:
>>
>>> I'm plotting a time series of fft's, similar to spectrograms but in 3D.
>>> If I use a linear scale I clearly see details of primary signal.
>>> But If I use a log scale I can now see the existence of much smaller
>>> features but at the expense of losing detail of the larger features.
>>>
>>> Are there any common methods for dealing with this problem?
>>
>>
>>
>> It isn't clear what "this problem" is. Statements like "primary
>> signal" and "smaller features" sound like value judgments. 
> 
> 
> *Definitely*
> 
>  > My guess is your criteria for those judgments is time.
> 
> No ;) It's more like "hmm there's something going on in the "background" 
> that I didn't expect but may be important". Its amplitude is much 
> smaller than the primary feature but it's also displaced in both time 
> and frequency.
> 
> Rune had pointed out that he had used spectrograms of the call of an 
> artic loon to illustrate (among other things) overtones. I didn't see it 
> with my presentation method and source recording. While waiting for a 
> better recording I attempted to improve my plot. The strong portion of 
> the loon's call is from 1000-1500 Hz. But there is also something 
> starting near 2200 Hz but offset in time by ~.1 sec. I'm trying to 
> investigate that.
> 

My ride came so I had to end my last post. While I was out your comment
about a "running average" got me thinking on different paths.

>>
>>      Is the primary signal something you perceive as lasting a while and
>> the smaller features are more transient and appear only for a short
>> while?.

The projection onto the time-frequency plane are comparable. The only 
thing definite is that the amplitude of the "smaller" features are smaller.

>> If so perhaps a running average of the log scale
>> representation would be the way achieve a happier median between
>> smaller and larger features. Of course the ideal length for the
>> running average would need to be determined. The  longer your running
>> average the more the long term features would be enhanced at the
>> expense of short term features.

Don't think so.
HOWEVER it suggests an idea. For an area whose amplitude is less than 
some value, give higher gain to a region with a peak whose amplitude is 
a absolute amount different that the average of a larger surrounding 
area. [Compare to using a magnifying glass to examine the immediate area 
of a blip on a topographic map.]

>>
>> -jim
>>
>>
>>
>>> As I'm looking at only the magnitude of the fft, my data is _strictly_
>>> positive. I've experimented with simplistic compression.
>>>
>>> For an input "x" I've experimented with a gain of form
>>>
>>> gain = k*((max(x)-x)/max(x))^n
>>>
>>> The results weren't great, but they were encouraging.
>>> It did highlight a potential complication.
>>> I will want something similar to the attack and release(???) settings of
>>>  AGC. But it will have to be dependent on time and frequency. To really
>>> mix metaphors, it will be dependent on radial distance in the time vs
>>> frequency plane. (as I have stored data, looking to the future is 
>>> allowed ;)
>>>
>>> Comments?
>>
>>
>>
>>
>> ----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet 
>> News==----
>> http://www.pronews.com The #1 Newsgroup Service in the World! >100,000 
>> Newsgroups
>> ---= - Total Privacy via Encryption =---

On 17 Aug, 13:29, Richard Owlett <rowl...@atlascomm.net> wrote:
> I'm plotting a time series of fft's, similar to spectrograms but in 3D.
> If I use a linear scale I clearly see details of primary signal.
> But If I use a log scale I can now see the existence of much smaller
> features but at the expense of losing detail of the larger features.
>
> Are there any common methods for dealing with this problem?
>
> As I'm looking at only the magnitude of the fft, my data is _strictly_
> positive. I've experimented with simplistic compression.
>
> For an input "x" I've experimented with a gain of form
>
> gain = k*((max(x)-x)/max(x))^n
>
> The results weren't great, but they were encouraging.
> It did highlight a potential complication.
> I will want something similar to the attack and release(???) settings of
> &#4294967295; AGC. But it will have to be dependent on time and frequency. To really
> mix metaphors, it will be dependent on radial distance in the time vs
> frequency plane. (as I have stored data, looking to the future is allowed ;)
>
> Comments?

Not that it's common practice, but what about a monotonic increasing
function based on a sample amplitude histogram?

cheers
jacko

jim wrote:
> 
> Richard Owlett wrote:
> 
>>I'm plotting a time series of fft's, similar to spectrograms but in 3D.
>>If I use a linear scale I clearly see details of primary signal.
>>But If I use a log scale I can now see the existence of much smaller
>>features but at the expense of losing detail of the larger features.
>>
>>Are there any common methods for dealing with this problem?
> 
> 
> It isn't clear what "this problem" is. Statements like "primary
> signal" and "smaller features" sound like value judgments. 

*Definitely*

 > My guess is your criteria for those judgments is time.

No ;) It's more like "hmm there's something going on in the "background" 
that I didn't expect but may be important". Its amplitude is much 
smaller than the primary feature but it's also displaced in both time 
and frequency.

Rune had pointed out that he had used spectrograms of the call of an 
artic loon to illustrate (among other things) overtones. I didn't see it 
with my presentation method and source recording. While waiting for a 
better recording I attempted to improve my plot. The strong portion of 
the loon's call is from 1000-1500 Hz. But there is also something 
starting near 2200 Hz but offset in time by ~.1 sec. I'm trying to 
investigate that.

> 
> 	 Is the primary signal something you perceive as lasting a while and
> the smaller features are more transient and appear only for a short
> while?. If so perhaps a running average of the log scale
> representation would be the way achieve a happier median between
> smaller and larger features. Of course the ideal length for the
> running average would need to be determined. The  longer your running
> average the more the long term features would be enhanced at the
> expense of short term features.
> 
> -jim
> 
> 
> 
>>As I'm looking at only the magnitude of the fft, my data is _strictly_
>>positive. I've experimented with simplistic compression.
>>
>>For an input "x" I've experimented with a gain of form
>>
>>gain = k*((max(x)-x)/max(x))^n
>>
>>The results weren't great, but they were encouraging.
>>It did highlight a potential complication.
>>I will want something similar to the attack and release(???) settings of
>>  AGC. But it will have to be dependent on time and frequency. To really
>>mix metaphors, it will be dependent on radial distance in the time vs
>>frequency plane. (as I have stored data, looking to the future is allowed ;)
>>
>>Comments?
> 
> 
> 
> ----== Posted via Pronews.Com - Unlimited-Unrestricted-Secure Usenet News==----
> http://www.pronews.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
> ---= - Total Privacy via Encryption =---