Reply by glen herrmannsfeldt●July 18, 20132013-07-18
Robert Scott <no-one@notreal.invalid> wrote:
(snip on noise removal)
> The FFT methods rely on the fact that your signal has one general
> distribution and your noise has a different distribution. If you
> favor the frequency bands where your signal/noise ratio is the
> highest, the overall signal/noise ratio will be higher.
> But don't forget other methods of noise removal that are not based on
> conversion to the frequency domain. One example is when the noise is
> highly-structured, such as ignition noise in an automobile electrical
> system. In this case the noise pulses are of very short duration and
> have high energy. If you look at this noise in the frequency domain,
> it may spread throughout your desireable signal frequency range and be
> difficult to separate from the signal. But if you look at it in the
> time domain, a simple clipper that limits time domain amplitude will
> be very effective at reducing the noise.
I could never afford one, (maybe now the price is down) but there
used to be boxes that would remove the pop noise from vinyl disk
recordings. When the stylus goes over a piece of dust, or scratch,
you get a sharp peak that is easy to see in the time domain.
Somewhat similar to CD error concealment (when ECC fails), you
smooth over the pop. Might work for ignition noise, too, but I
haven't heard of anyone doing it.
-- glen
>On Tuesday, July 16, 2013 7:09:30 PM UTC-7, muhammadus...@gmail.com wrote:
>> Hi mates,
>> I need to know somehow, the parts of overall structure setup for removing background noise and determining background noise sources.
>>
>> I am working on a problem to solve background noise emitted from test objects.
>>
>> Here's written all on the document, I have
>> https://www.dropbox.com/sh/38hbzpufqiajm7n/D_vTPEP9mb
>> ...
>
>The type of structure you show is common to many signal processing applications. A general example is the class of instruments called dynamic spectrum analyzers. Combining that term in a Google with: application notes points to a variety of resources supplied by many venders if instrumentation.
>
>
>### General Background ###
>
>http://www.measurement.net.au/g/5788/application-notes.html
>
>The Fundamentals of
>Signal Analysis
>Application Note 243
>http://literature.agilent.com/litweb/pdf/5952-8898E.pdf
>
>### Example Instruments ###
>
>http://www.me.ua.edu/me360/docs/HP-DSA.pdf
>Effective Machinery
>Measurements using
>Dynamic Signal Analyzers
>Application Note 243-1
>
>The SR785 Data Sheet
>SR785c.pdf
>available from site:
>http://www.thinksrs.com/products/SR785.htm
>User Manual
>available through web site:
>http://www.thinksrs.com/mult/SR785m.htm
>
>### Consider looking at the library at:
>http://www.bksv.com/library/
>for examples of applications, algorithms and instruments
>for example:
>http://www.bksv.com/doc/TechnicalReview1996-2a.pdf
>
>Browse and learn. Come back with better questions.
>
>Dale B. Dalrymple
Hi Dale,
Wow! Those are great PDF files. Thanks.
[-Rick-]
Reply by Robert Scott●July 18, 20132013-07-18
On Wed, 17 Jul 2013 01:45:30 -0700 (PDT),
muhammadusman.khalil@gmail.com wrote:
>I studied until now that there's no generic way for removal of noise from c=
>ontaminated signal.
>
That is right. As Tim Wescott said in another posting, successful
noise removal methods depend on specific differences between the noise
and the signal. The more you know about the signal and/or the noise
in a particular instance the more you can separate them. But as long
as they are both "generic", there is no way.
The FFT methods rely on the fact that your signal has one general
distribution and your noise has a different distribution. If you
favor the frequency bands where your signal/noise ratio is the
highest, the overall signal/noise ratio will be higher.
But don't forget other methods of noise removal that are not based on
conversion to the frequency domain. One example is when the noise is
highly-structured, such as ignition noise in an automobile electrical
system. In this case the noise pulses are of very short duration and
have high energy. If you look at this noise in the frequency domain,
it may spread throughout your desireable signal frequency range and be
difficult to separate from the signal. But if you look at it in the
time domain, a simple clipper that limits time domain amplitude will
be very effective at reducing the noise.
Are you still looking for generic methods? Or does your project allow
the detailed consideration of the characteristics of the noise and the
signal?
Robert Scott
Hopkins, MN
Reply by glen herrmannsfeldt●July 18, 20132013-07-18
robert bristow-johnson <rbj@audioimagination.com> wrote:
(snip, I wrote)
>> Also, in audio (and likely others) there are two-sided
>> (is that the right term) and single-ended (I believe that
>> is right) noise reduction systems.
> i dunno what the terms are, but i know what you mean. the two-sided is
> like using pre-emphasis and de-emphasis. or, in another sense, like
> companding; do something to the signal to make it stand up better to the
> anticipated added noise (or noise generated from other mechanisms, like
> quantization), then undo that something to get your original signal
> back, hopefully with less error than it would have if you didn't do this
> something and unsomething.
There are references to single ended, but I don't see any reference
naming the other one. Sometimes they aren't so obvious. Note that
the opposite of single-ended SCSI is differential, as that is the
way it works electrically.
>> The more popular dolby and dbx systems are two-sided,
>> encoding the system before noise (vinyl disk or magnetic
>> tape) recording, and decoding it on playback.
> the vinyl wasn't dolby or dbx, i think it was called RIAA
> and it was static pre- and de-emphasis.
> (dolby was dynamic pre- and de-emphasis.)
There were dbx coded vinyl records, though I never saw one.
I have a dbx tape deck that has the ability to decode them.
I beleive they were somewhat expensive, so didn't sell well enough
to get popular. Also, as with dbx tapes, they sounds bad played
without the decoder.
The big advantage of dolby B is that it doesn't sound too bad
without the decoder, and might even sound a little better,
it you turn the treble down a little bit. I did used to have
a car tape player with a dolby B decoder, but they weren't
all that popular. A little compression helps in a car, to
keep above road noise.
>> Single-ended (on playback only) is more difficult, but
>> sometimes must be done. Since audio, and especial spoken
>> word, has little in the high frequencies,
> except for them nasty fricatives. but otherwize you're quite right. at
> least when Fs is 44100 Hz or similar. when Fs is 8K, then what is meant
> by "high frequencies" gets to be a little different.
Seems that AM radio is limited to 10.2kHz in the US.
I sometimes listen to baseball games on a noisy AM radio, adjusting
the tone control until it sounds best. I suppose I could work on
the radio, instead. It is part of a Nutone intercom system, and
I do have the service manual for it. (Installed by the previous
owner, who also supplied the service manual.)
> but, even for music, the far greater portion of energy is in the bottom
> 5 or 6 octaves of normal human hearing. so the top 2 or 3 or 4 octaves
> have very little energy. i have used this fact to advantage in
> designing the sinc-like impulse response for sample rate conversion alg.
> you can inexpensively put notches right on top of all multiples of Fs
> (except for the 0th multiple of Fs) and kill most of the energy in the
> images. the technique (notches at m*Fs except for m=0) is pretty
> obvious but i can't say it (just infer around it).
Besides, other than babies most of us don't hear anywhere near 20kHz.
There are stories of kids with ring tones on cell phones above the
teachers hearing, but below the kids' hearing limit.
-- glen
Reply by robert bristow-johnson●July 18, 20132013-07-18
On 7/17/13 12:09 PM, glen herrmannsfeldt wrote:
>
> Also, in audio (and likely others) there are two-sided
> (is that the right term) and single-ended (I believe that
> is right) noise reduction systems.
>
i dunno what the terms are, but i know what you mean. the two-sided is
like using pre-emphasis and de-emphasis. or, in another sense, like
companding; do something to the signal to make it stand up better to the
anticipated added noise (or noise generated from other mechanisms, like
quantization), then undo that something to get your original signal
back, hopefully with less error than it would have if you didn't do this
something and unsomething.
> The more popular dolby and dbx systems are two-sided,
> encoding the system before noise (vinyl disk or magnetic
> tape) recording, and decoding it on playback.
the vinyl wasn't dolby or dbx, i think it was called RIAA and it was
static pre- and de-emphasis. (dolby was dynamic pre- and de-emphasis.)
>
> Single-ended (on playback only) is more difficult, but
> sometimes must be done. Since audio, and especial spoken
> word, has little in the high frequencies,
except for them nasty fricatives. but otherwize you're quite right. at
least when Fs is 44100 Hz or similar. when Fs is 8K, then what is meant
by "high frequencies" gets to be a little different.
but, even for music, the far greater portion of energy is in the bottom
5 or 6 octaves of normal human hearing. so the top 2 or 3 or 4 octaves
have very little energy. i have used this fact to advantage in
designing the sinc-like impulse response for sample rate conversion alg.
you can inexpensively put notches right on top of all multiples of Fs
(except for the 0th multiple of Fs) and kill most of the energy in the
images. the technique (notches at m*Fs except for m=0) is pretty
obvious but i can't say it (just infer around it).
> but noise is
> often uniform in frequency space, a low pass filter
> is sometimes a fine noise reduction system.
yup.
--
r b-j rbj@audioimagination.com
"Imagination is more important than knowledge."
Reply by ●July 18, 20132013-07-18
Amount of time required to get a basic fft noise removal algorithm running; 1 week
Time required to eliminate " musical noise" artifacts; about 2 years.
Everyone who succeeds keeps their musical-noise-reduction recipe to themselves.
Bob
Reply by dbd●July 17, 20132013-07-17
On Tuesday, July 16, 2013 7:09:30 PM UTC-7, muhammadus...@gmail.com wrote:
> Hi mates,
> I need to know somehow, the parts of overall structure setup for removing background noise and determining background noise sources.
>
> I am working on a problem to solve background noise emitted from test objects.
>
> Here's written all on the document, I have
> https://www.dropbox.com/sh/38hbzpufqiajm7n/D_vTPEP9mb
> ...
Reply by glen herrmannsfeldt●July 17, 20132013-07-17
Rick Lyons <R.Lyons@_bogus_ieee.org> wrote:
(big snip)
> I don't know if you volunteered to solve your
> "noise removal" problem, or if some German professor
> assigned this problem to you. But in either case,
> I believe someone has made a BIG mistake.
> It seems to me that solving some sort of "noise
> removal" problem, the details of which are
> unknown at this time, would require a talanted
> DSP engineer who has years of DSP experience.
> Even then, success is not guaranteed. I say that
> because, without knowing any details about your
> noisy signals and what is meant by "noise removal",
> it may not be possible achieve acceptable
> noise removal.
I suppose this is all true, but sometimes we live with
it anyway. As for audio noise reduction, which many of
us know better than others, sometimes you just turn
down the tone control or treble control. It isn't the
best, but sometimes it works well enough.
Also, in audio (and likely others) there are two-sided
(is that the right term) and single-ended (I believe that
is right) noise reduction systems.
The more popular dolby and dbx systems are two-sided,
encoding the system before noise (vinyl disk or magnetic
tape) recording, and decoding it on playback.
Single-ended (on playback only) is more difficult, but
sometimes must be done. Since audio, and especial spoken
word, has little in the high frequencies, but noise is
often uniform in frequency space, a low pass filter
is sometimes a fine noise reduction system.
-- glen
Reply by Tim Wescott●July 17, 20132013-07-17
On Wed, 17 Jul 2013 01:45:30 -0700, muhammadusman.khalil wrote:
> Am Mittwoch, 17. Juli 2013 04:09:30 UTC+2 schrieb
> muhammadus...@gmail.com:
>> Hi mates,
>>
>>
>>
>> I need to know somehow, the parts of overall structure setup for
>> removing background noise and determining background noise sources.
>>
>>
>>
>> I am working on a problem to solve background noise emitted from test
>> objects.
>>
>>
>>
>> Here's written all on the document, I have
>>
>>
>>
>> https://www.dropbox.com/sh/38hbzpufqiajm7n/D_vTPEP9mb
>>
>>
>>
>> What all I am looking to know ,
>>
>>
>>
>> What these mathematical modules could be?
>>
>> What these algorithms could serve for?
>>
>> Are they contains some logic to remove noise?
>>
>> If so, what could be that logic?
>>
>>
>>
>> Once I heard , it might be Sqrt(), I don't know what it has to do with
>> removal of noise in this structure.
>>
>>
>>
>> I am sorry if this question is not elaborated at all, I am Informatics
>> student , but somehow I have to solve this problem and build
>> architecture to develop a software in .NET. But before that I have to
>> remove noise as it is an objective of my thesis part.
>>
>>
>>
>>
>>
>> Does these math algorithms needs to be replaces from other algorithms?
>>
>>
>>
>>
>>
>> If some body helps me out, I would be too much obliged for this.
>>
>>
>>
>>
>>
>> Regards
>>
>> Muhammad Usman Khalil
>
> Mates,
>
> I am not able to properly elaborate the problem I had. Sorry for this.
>
> Being a non DSP student, I am working on noise removal problem due to
> some reason. It has become my Thesis study of Master's. But I would have
> to implement a system that would follow the structure ( I mentioned as
> URL ). All I am working hard to grasp some concepts regarding signals
> and FFT.
>
> I know my data stream would be transformed from time domain to Frequency
> Domain after FFT. The results of FFT has to be "process" somehow, so
> that noise can be removed.
>
> I studied until now that there's no generic way for removal of noise
> from contaminated signal.
>
> 1 - Either you must have cleaned signal and noise signal ( Apply Wiener
> Filter or other DSP Filters ).
>
> 2 - Or Kalman Filter for taking more better results.
>
> This is all what I have researched until now.
> But the problem is : I have this template from my past workplace side
> and I am not able to get the answer from those guys somehow, why these
> Mathematical Modules are there? What kind of modules these are? Once
> they said these modules would be like "Sqrt".
>
> I am sorry , I am not DSP student but Informatics student. So, I am new
> bie in this domain. But I have got the knowledge about my other modules
> ( e.g FFT and Block Seperator ). But I have no idea about "Maths"
>
> That's why I posted here. Hope you will not angry this time.
We're not angry. We have some members that can be impatient with
students, more fools they. RBJ was just borrowing trouble.
I fear that you're being assigned a project by people that don't
necessarily know what they're asking for. You have my sympathy.
Since you mention Kalman filtering I have to comment: a Kalman filter
isn't necessarily going to give you better results than a Wiener filter.
The filtering technique that you need to use depends entirely on the
nature of the signal, of the noise, and what you care about. For most
filtering applications where a Wiener filter is helpful, the only thing
that a Kalman filter gets you is quicker acquisition. In fact, for those
filtering applications where a Wiener filter is indicated, a Kalman
filter that has been run for a good long time ends up being a Wiener
filter.
The "pure" Kalman filter is helpful when you need to have the very best
estimate of a system's states (or of signal vs. noise) very early in the
process, or when the system is time-varying in some known way -- and
that's about it.
You can take systems that are nonlinear in a known way, and treat them as
if they were linear, time-varying systems, and apply Kalman-filter
techniques. These aren't "pure" Kalman filters, though they are based on
them (see "extended Kalman" and "unscented Kalman").
Kalman (and Wiener) filters depend on having a near-perfect knowledge of
the system to work correctly. There are dodges to making them work well
when the system is not known (usually one pretends that there's more
noise than there really is, and hopes), but these dodges pretty much
immediately take you away from a nice mathematically guaranteed optimal
filter and dumps you right into the Land of Seat of the Pants
Engineering. You can do the same thing in a more structured way with a
so-called H-infinity filter, but these have a much greater computational
load if you are trying to estimate the states of a time-varying system.
--
Tim Wescott
Wescott Design Services
http://www.wescottdesign.com
Reply by Tim Wescott●July 17, 20132013-07-17
On Tue, 16 Jul 2013 19:09:30 -0700, muhammadusman.khalil wrote:
> Hi mates,
>
> I need to know somehow, the parts of overall structure setup for
> removing background noise and determining background noise sources.
>
> I am working on a problem to solve background noise emitted from test
> objects.
>
> Here's written all on the document, I have
>
> https://www.dropbox.com/sh/38hbzpufqiajm7n/D_vTPEP9mb
>
> What all I am looking to know ,
>
> What these mathematical modules could be?
> What these algorithms could serve for?
> Are they contains some logic to remove noise?
> If so, what could be that logic?
>
> Once I heard , it might be Sqrt(), I don't know what it has to do with
> removal of noise in this structure.
>
> I am sorry if this question is not elaborated at all, I am Informatics
> student , but somehow I have to solve this problem and build
> architecture to develop a software in .NET. But before that I have to
> remove noise as it is an objective of my thesis part.
>
>
> Does these math algorithms needs to be replaces from other algorithms?
>
>
> If some body helps me out, I would be too much obliged for this.
>
>
> Regards Muhammad Usman Khalil
We may all be over-thinking this. I'm not sure just what your prof has
in mind (if anything). Frankly, someone on your team (prof included)
should be able to answer these questions for the result to do anyone any
good other than as an elementary exercise in .NET programming.
So, background first, then comments on what might need to go into these
mysterious "math" blocks.
Background:
In order to remove your background noise from the signal, the noise has
to be somehow different from the signal. If you know the difference, and
if you can figure out how to make an algorithm that sorts out the
difference, then you can separate noise from signal. This is what a
filter does -- just like a coffee filter separates coffee grounds from
what goes into your cup. The coffee filter does it because coffee
grounds are a lot bigger than the particles in coffee. Most filters do
it because the noise is completely or partially in a different frequency
spectrum than the desired signal.
Comments:
You're already sorting the signal by frequency, so it's hard to see what
more might be done with it. Certainly presenting the signal in a manner
that makes it easy to see is good. If it hasn't been done, one of the
"math" blocks should be something that takes the pairs of complex numbers
out of the FFT and finds their magnitude while discarding their phase.
(Phase doesn't mean much in your application. Phase can mean a lot in
other places, but not here). Another useful "math" block would be to
take the logarithm of the amplitude, because people tend to perceive
amplitudes logarithmically. Without displaying your graphs
logarithmically, you will be suppressing low-amplitude effects, making
perfectly audible sounds too small to see on the graph.
Beyond that, I don't know what you might do. The block diagram that you
have is for something that will be useful to present visual data; if you
need to actually filter the signal from the noise, either for listening
or for further mechanical processing, then all you'll be doing with the
simple useful filtering techniques is zeroing out FFT bins; that'll just
show up as a band of "always zero" on the waterfall plot, but it won't
tell you anything more about what you want to see.
--
Tim Wescott
Wescott Design Services
http://www.wescottdesign.com