fft bin magnitude normalization trouble

I have lurked here for some time. I must admit my profession is developer, 
not mathematician, however I am learning all I can in my efforts.



I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.



My problem is this, if i attain the magnitude of each bin, and graph it, i 
see nice peaks where I would expect, and very little spectral leakage or 
background noise.

When I attempt to normalize the bin magnitudes, what used to be one bin 
wide, mushrooms out to more than 10, and the background comes up 40-50 db.



C# source below:



How I am calculating magnitude :

double normBinMag = 2 * Math.Sqrt((re*re)+(im*im)) / (nFourierPoints);



How I am attempting to normalize it :

double amplitude = 20 * Math.Log10( normBinMag );



I have played with this for some time now, and although the normalization 
method above normalizes the output, it has the above mentioned flaw.



What am I doing wrong? I am assuming a window will not help this.

Thanks in advance.



Brian

"Rex" <rexswenn@hushmail.com> wrote in message 
news:Zj6Ue.9134$_84.9109@newsread1.news.atl.earthlink.net...
>I have lurked here for some time. I must admit my profession is developer, 
>not mathematician, however I am learning all I can in my efforts.
>
>
>
> I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.
>
>
>
> My problem is this, if i attain the magnitude of each bin, and graph it, i 
> see nice peaks where I would expect, and very little spectral leakage or 
> background noise.
>
> When I attempt to normalize the bin magnitudes, what used to be one bin 
> wide, mushrooms out to more than 10, and the background comes up 40-50 db.
>
>
>
> C# source below:
>
>
>
> How I am calculating magnitude :
>
> double normBinMag = 2 * Math.Sqrt((re*re)+(im*im)) / (nFourierPoints);
>
>
>
> How I am attempting to normalize it :
>
> double amplitude = 20 * Math.Log10( normBinMag );
>
>
>
> I have played with this for some time now, and although the normalization 
> method above normalizes the output, it has the above mentioned flaw.
>
>
>
> What am I doing wrong? I am assuming a window will not help this.
>
> Thanks in advance.

Taking the log is not "normalization".  And, you'd have to define the 
normalization you want in order to have someone comment.
One normalization would be to divide all values by peak magnitude that 
occurs over all frequency samples.

The log has the property of compressing amplitudes - so what appears to be a 
narrow spectral line, if there's adjacent energy, will appear to spread out.
The same thing for noise.  If there is noise at a low level then it will be 
much more apparent after taking the log magnitude.

Or, are you taking log log?  ... because you mention dB relative to 
pre-"normalization".

Fred 

Rex wrote:
> I have lurked here for some time. I must admit my profession is developer,
> not mathematician, however I am learning all I can in my efforts.
>
>
>
> I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.
>
>
>
> My problem is this, if i attain the magnitude of each bin, and graph it, i
> see nice peaks where I would expect, and very little spectral leakage or
> background noise.
>
> When I attempt to normalize the bin magnitudes, what used to be one bin
> wide, mushrooms out to more than 10, and the background comes up 40-50 db.

One "obvious" detail to check, is the number formats. How are the
numbers represented internally to your routine? Float? Integer?

If you load the data as integers, and normalize them as integers,
I would expect quantization noise to become a lot more dominant
than without the normalization.

If you haven't done so already, convert from int to float as
early as possible in your processing chain. 

Rune

Rex wrote:
> I have lurked here for some time. I must admit my profession is developer, 
> not mathematician, however I am learning all I can in my efforts.
> 
> 
> 
> I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.
> 
> 
> 
> My problem is this, if i attain the magnitude of each bin, and graph it, i 
> see nice peaks where I would expect, and very little spectral leakage or 
> background noise.
> 
> When I attempt to normalize the bin magnitudes, what used to be one bin 
> wide, mushrooms out to more than 10, and the background comes up 40-50 db.
> 
> 
> 
> C# source below:
> 
> 
> 
> How I am calculating magnitude :
> 
> double normBinMag = 2 * Math.Sqrt((re*re)+(im*im)) / (nFourierPoints);
> 
> 
> 
> How I am attempting to normalize it :
> 
> double amplitude = 20 * Math.Log10( normBinMag );
> 
> 
> 
> I have played with this for some time now, and although the normalization 
> method above normalizes the output, it has the above mentioned flaw.
> 
> 
> 
> What am I doing wrong? I am assuming a window will not help this.
> 
> Thanks in advance.
> 
> 
> 
> Brian
> 
> 

If your normalization is just doing 20*log10() and that is where you're 
problem comes in, then how do you know the noise comes up 40-50 dB? What 
   are you comparing it with to say that.

Normalization has a few different meanings, so you should clarify what 
exactly it is you are trying to accomplish.

Cheers,
David

"Fred Marshall" <fmarshallx@remove_the_x.acm.org> wrote in message 
news:uNidndWJybG2ibzeRVn-pA@centurytel.net...
>
> "Rex" <rexswenn@hushmail.com> wrote in message 
> news:Zj6Ue.9134$_84.9109@newsread1.news.atl.earthlink.net...
>>I have lurked here for some time. I must admit my profession is developer, 
>>not mathematician, however I am learning all I can in my efforts.
>>
>>
>>
>> I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.
>>
>>
>>
>> My problem is this, if i attain the magnitude of each bin, and graph it, 
>> i see nice peaks where I would expect, and very little spectral leakage 
>> or background noise.
>>
>> When I attempt to normalize the bin magnitudes, what used to be one bin 
>> wide, mushrooms out to more than 10, and the background comes up 40-50 
>> db.
>>
>>
>>
>> C# source below:
>>
>>
>>
>> How I am calculating magnitude :
>>
>> double normBinMag = 2 * Math.Sqrt((re*re)+(im*im)) / (nFourierPoints);
>>
>>
>>
>> How I am attempting to normalize it :
>>
>> double amplitude = 20 * Math.Log10( normBinMag );
>>
>>
>>
>> I have played with this for some time now, and although the normalization 
>> method above normalizes the output, it has the above mentioned flaw.
>>
>>
>>
>> What am I doing wrong? I am assuming a window will not help this.
>>
>> Thanks in advance.
>
> Taking the log is not "normalization".  And, you'd have to define the 
> normalization you want in order to have someone comment.
> One normalization would be to divide all values by peak magnitude that 
> occurs over all frequency samples.
>
> The log has the property of compressing amplitudes - so what appears to be 
> a narrow spectral line, if there's adjacent energy, will appear to spread 
> out.
> The same thing for noise.  If there is noise at a low level then it will 
> be much more apparent after taking the log magnitude.
>
> Or, are you taking log log?  ... because you mention dB relative to 
> pre-"normalization".
>
> Fred
>

I apologize for using the wrong term. The function i found in C that i 
mimicked in C#, said it was normalizing the Bin Magnitude.

What is the proper way to do this? Divide them all by the Bin with the 
largeest value?

I am trying to get some kind of normalization, so I can compare multiple 
audio files of the same format, and differentiate the bin magnitude.

Thank you for tolerating my newness to DSP. 

This :

    double amplitude = 20 * Math.Log10( normBinMag );

Seems to "normalize" the output Bin Magnitudes to decibels. I may be 
completely wrong though.
However when this happens, any noise seems to be grossly amplified.

A crude example :

Before "normalization"

|
|                 |
|                 |
|                 |
|                 |
|____--_--- -______________
|------------------------------------------
After :


|                 |
|               |    |
| ----_-- |       |___------______
|
 |
|------------------------------------------

All help is greatly appreciated.

Brian

"David Kirkland" <spam@netscape.net> wrote in message 
news:fgfUe.19366$I02.1104211@news20.bellglobal.com...
> Rex wrote:
>> I have lurked here for some time. I must admit my profession is 
>> developer, not mathematician, however I am learning all I can in my 
>> efforts.
>>
>>
>>
>> I am performing a 1-d fft on real data. 16bit, 8000Hz, mono.
>>
>>
>>
>> My problem is this, if i attain the magnitude of each bin, and graph it, 
>> i see nice peaks where I would expect, and very little spectral leakage 
>> or background noise.
>>
>> When I attempt to normalize the bin magnitudes, what used to be one bin 
>> wide, mushrooms out to more than 10, and the background comes up 40-50 
>> db.
>>
>>
>>
>> C# source below:
>>
>>
>>
>> How I am calculating magnitude :
>>
>> double normBinMag = 2 * Math.Sqrt((re*re)+(im*im)) / (nFourierPoints);
>>
>>
>>
>> How I am attempting to normalize it :
>>
>> double amplitude = 20 * Math.Log10( normBinMag );
>>
>>
>>
>> I have played with this for some time now, and although the normalization 
>> method above normalizes the output, it has the above mentioned flaw.
>>
>>
>>
>> What am I doing wrong? I am assuming a window will not help this.
>>
>> Thanks in advance.
>>
>>
>>
>> Brian
>>
>>
>
> If your normalization is just doing 20*log10() and that is where you're 
> problem comes in, then how do you know the noise comes up 40-50 dB? What 
> are you comparing it with to say that.
>
> Normalization has a few different meanings, so you should clarify what 
> exactly it is you are trying to accomplish.
>
> Cheers,
> David 

Rex -- "normalizing" a data vector usually means linearly scaling its
values so they meet some "normal" condition, such as the maximum equals
some value or the sum of the values equals 1.

OTOH, the 20*log10() operation you show is simply converting the
spectral amplitudes to decibels.  Your respondents (including me) are
somewhat confused by your description of this operation as
"normalizing".

As Fred has already pointed out, depicting the data on a log scale has
precisely the effects that you have described.  It hasn't changed the
underlying data or the SNR; it's just displaying it differently.

Perhaps you could explain in more detail what you're trying to
accomplish.

cheers,
  jerry

Thank you,

So I am not in fact trying to normalize the data.

My goal is to convert if you will, the Bin magnitudes to decibels.
Why do the peaks become so less distinct in doing this?

Without applying the log10, the peaks are well formed, and it looks very 
clean. After the conversion to decibels, it "looks" like the peaks get much 
wider (and more inaccurate), and the "bottom end"/"noise" comes up 
immensely.

Am I converting to decibels incorrectly? Are there other ways to do this?

Thank you all for bearing with me.

Brian

"Jerry Wolf" <sl_jerry1@hotmail.com> wrote in message 
news:1126282303.609881.269080@f14g2000cwb.googlegroups.com...
> Rex -- "normalizing" a data vector usually means linearly scaling its
> values so they meet some "normal" condition, such as the maximum equals
> some value or the sum of the values equals 1.
>
> OTOH, the 20*log10() operation you show is simply converting the
> spectral amplitudes to decibels.  Your respondents (including me) are
> somewhat confused by your description of this operation as
> "normalizing".
>
> As Fred has already pointed out, depicting the data on a log scale has
> precisely the effects that you have described.  It hasn't changed the
> underlying data or the SNR; it's just displaying it differently.
>
> Perhaps you could explain in more detail what you're trying to
> accomplish.
>
> cheers,
>  jerry
> 

Rex wrote:
> Thank you,
> 
> So I am not in fact trying to normalize the data.
> 
> My goal is to convert if you will, the Bin magnitudes to decibels.
> Why do the peaks become so less distinct in doing this?
> 
> Without applying the log10, the peaks are well formed, and it looks very 
> clean. After the conversion to decibels, it "looks" like the peaks get much 
> wider (and more inaccurate), and the "bottom end"/"noise" comes up 
> immensely.
> 
> Am I converting to decibels incorrectly? Are there other ways to do this?
> 
> Thank you all for bearing with me.

log10(1000)=3
log10(100)=2

Think about it.


> 
> Brian
> 
> "Jerry Wolf" <sl_jerry1@hotmail.com> wrote in message 
> news:1126282303.609881.269080@f14g2000cwb.googlegroups.com...
> 
>>Rex -- "normalizing" a data vector usually means linearly scaling its
>>values so they meet some "normal" condition, such as the maximum equals
>>some value or the sum of the values equals 1.
>>
>>OTOH, the 20*log10() operation you show is simply converting the
>>spectral amplitudes to decibels.  Your respondents (including me) are
>>somewhat confused by your description of this operation as
>>"normalizing".
>>
>>As Fred has already pointed out, depicting the data on a log scale has
>>precisely the effects that you have described.  It hasn't changed the
>>underlying data or the SNR; it's just displaying it differently.
>>
>>Perhaps you could explain in more detail what you're trying to
>>accomplish.
>>
>>cheers,
>> jerry
>>
> 
> 
> 

I am sorry, but it still is not clear to me. I understand how log10 is 
skewing the once well formed peaks now.

However, am I doing something terribly wrong? What way should I be doing 
this to avoid the current issue?

thank you

Brian
"Stan Pawlukiewicz" <spam@spam.mitre.org> wrote in message 
news:dfsd05$eft$1@newslocal.mitre.org...
> Rex wrote:
>> Thank you,
>>
>> So I am not in fact trying to normalize the data.
>>
>> My goal is to convert if you will, the Bin magnitudes to decibels.
>> Why do the peaks become so less distinct in doing this?
>>
>> Without applying the log10, the peaks are well formed, and it looks very 
>> clean. After the conversion to decibels, it "looks" like the peaks get 
>> much wider (and more inaccurate), and the "bottom end"/"noise" comes up 
>> immensely.
>>
>> Am I converting to decibels incorrectly? Are there other ways to do this?
>>
>> Thank you all for bearing with me.
>
> log10(1000)=3
> log10(100)=2
>
> Think about it.
>
>
>>
>> Brian
>>
>> "Jerry Wolf" <sl_jerry1@hotmail.com> wrote in message 
>> news:1126282303.609881.269080@f14g2000cwb.googlegroups.com...
>>
>>>Rex -- "normalizing" a data vector usually means linearly scaling its
>>>values so they meet some "normal" condition, such as the maximum equals
>>>some value or the sum of the values equals 1.
>>>
>>>OTOH, the 20*log10() operation you show is simply converting the
>>>spectral amplitudes to decibels.  Your respondents (including me) are
>>>somewhat confused by your description of this operation as
>>>"normalizing".
>>>
>>>As Fred has already pointed out, depicting the data on a log scale has
>>>precisely the effects that you have described.  It hasn't changed the
>>>underlying data or the SNR; it's just displaying it differently.
>>>
>>>Perhaps you could explain in more detail what you're trying to
>>>accomplish.
>>>
>>>cheers,
>>> jerry
>>>
>>
>>