Convolution Tutorial

On Dec 28, 8:31&#4294967295;am, Tim Wescott <t...@seemywebsite.com> wrote:
> On Sun, 27 Dec 2009 10:01:07 +0000, invalid wrote:
> > "brent" <buleg...@columbus.rr.com> wrote in message
> > news:0fd6f825-e7ad-4642-
>
> a5fe-83de8ff8f...@x18g2000vbd.googlegroups.com...
>
> >>I have created a tutorial on the convolution integral. It uses an
> >> interactive flash program with embedded audio files. It is located
> >> here:
> >>http://www.fourier-series.com/Convolution/index.html
>
> > You start off by saying that convolution is a mathematical operation, at
> > which point I switched off.
>
> > Convolution is the way that real systems in the real world (such as
> > pianoforte strings)
> > respond to stimuli that are continuous (such as a sine wave from a
> > loudspeaker in close proximity)
>
> Convolution is _not_ the way that real systems in the real world respond
> to stimuli of any sort. &#4294967295;Convolution is just a _mathematical operation_
> that _approximates_ what real systems do. &#4294967295;Sometimes it even does it well.
>
> All real systems are nonlinear. &#4294967295;The convolution operation is one way to
> implement a linear model of a system. &#4294967295;Thus, the convolution operation
> does not model any real system with 100% accuracy. &#4294967295;As a model, the
> convolution operation is only as good as the fit between its bedrock
> assumption of linearity and the system's actual conformity to linear
> behavior.
>
> For many systems, using convolution is a horribly indirect way to
> implement what should be a simple, limited-state, ordinary linear
> differential equation.
>
> > and not just impulses (such as when hit
> > with a hammer). I had difficulty with Convolution for years until it was
> > explained to me in this practical way at which point it became
> > meaningful
> > instead of being some arcane mathematical operation which I did not
> > really trust.
>
> > Unless you introduce the student to the practical basis of why you would
> > want to undertake such a weird operation, then you might as well give
> > up.
>
> > Mathematical analysis should come after practical experience and not
> > before.
>
> I do agree that mathematical analysis should be kept firmly in the
> context of what is real -- when I teach control systems I try to draw
> examples from the real world as often as possible, and I try to keep a
> clear distinction between the thing you're interested in and the
> mathematical model that you've made of it.
>
> But then, you've already wandered away from reality if you're claiming
> that real systems convolve their input signals with unfailing accuracy.
>
> In today's world I don't think you can ask for practical experience
> before theoretical knowledge, though -- with that assumption, engineering
> schools would only take technicians who had already been through an
> apprenticeship, which severely cuts down on the available candidate pool.
>
> --www.wescottdesign.com

Agreed but it's pretty dammed close. I mean how linear is an R-C
network? Try adding two sine waves and passing them through an RC
network.
What cross-spectral terms to you get percentage wise? I would be
interested to know.


Hardy

HardySpicer wrote:


> I mean how linear is an R-C
> network? Try adding two sine waves and passing them through an RC
> network.

RC can have substantial nonlinearity as any real life RC depends on 
voltage and temperature. Ceramic capacitors are particurlarly bad in 
this regard.

> What cross-spectral terms to you get percentage wise? I would be
> interested to know.

It depends. How much of distortion do you need? Always check with the 
datasheets.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

HardySpicer wrote:

   ...

> Agreed but it's pretty dammed close. I mean how linear is an R-C
> network? Try adding two sine waves and passing them through an RC
> network.
> What cross-spectral terms to you get percentage wise? I would be
> interested to know.

Electrolytics are notoriously non-linear. High-value ceramic capacitors 
-- calcium titanate, for example -- are also non-linear. Dielectric 
absorption is a continuing problem with sample-and-hold capacitors.

Non-linear resistors abound. Hewlwtt and Packard had the brilliance to 
use a tungsten resistor to stabilize the amplitude of a Wien-bridge 
oscillator and founded a company on the parent. (The movie Fantasia was 
an early user of their tunable audio oscillators.)

The components we commonly use are explicitly selected to be adeqquately 
linear under their normal conditions of use. Most of the world doesn't 
work that way.

Jerry
-- 
Engineering is the art of making what you want from things you can get.
�����������������������������������������������������������������������

On Dec 27, 2:42&#4294967295;pm, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 27 Des, 11:01, "invalid" <inva...@invalid.invalid> wrote:
>
> > "brent" <buleg...@columbus.rr.com> wrote in message
>
> >news:0fd6f825-e7ad-4642-a5fe-83de8ff8f7f6@x18g2000vbd.googlegroups.com...
>
> > >I have created a tutorial on the convolution integral. It uses an
> > > interactive flash program with embedded audio files.
> > > It is located here:
> > >http://www.fourier-series.com/Convolution/index.html
>
> > You start off by saying that convolution is a mathematical operation,
> > at which point I switched off.
>
> Then you have a problem. Like it or not, DSP is applied maths.
>
> > Convolution is the way that real systems in the real world (such as
> > pianoforte strings)
> > respond to stimuli that are continuous (such as a sine wave from a
> > loudspeaker in close proximity) and not just impulses (such as when
> > hit with a hammer).
>
> It is an *idealized* *representation* of what happens.
>
> > I had difficulty with Convolution for years until it
> > was explained to me in this practical way at which point it became
> > meaningful
>
> Did you pay tuition fees to anyone for teaching you DSP
> before that? If so, you might have a law case for them not
> delivering what you paid them for.
>
> > instead of being some arcane mathematical operation which I did not
> > really trust.
>
> Do you trust that 2+2 = 4? Or that you go bankrupt if you
> spend more $$$ than your income can sustain? If so you will
> have to trust convolution.
>
> > Unless you introduce the student to the practical basis of why you would
> > want to undertake such a weird operation, then you might as well give up.
>
> Nope. Get new students.
>
> Abstractions and engineering are hard intellectual work. Throughout
> history only a slect small percentage of the population have turned
> out to be able to handle such concepts. If people who can not cope
> with these kinds of things try to learn DSP, it is *their* problem;
> not the subject's.
>

Abstractions are hard work.  For me, I have spent much of my life
trying to visualize the math.  Once I can visualize it, then I
understand it.  It is the long process of constructing the model in my
head that is hard. My tutorial website is intended to help people
visualize the math and concepts of various topics.  In building these
pages I have been able to also solidify the model and visualization in
my head to a much fuller degree.

 To some degree, I agree with "invalid" about trying to make
convolution meaningful in a real world way.  In the tutorial I
actually stated that convolution needs to be looked at from two
distinct points of views, one point of view is the mathematical
operation and the other point of view is the practical application.  I
agree with invalid because as an engineer I ultimately only care about
math to the extent that I can get something done with the math, so it
is more than a math operation and one needs to know how convolution
helps solve real world problems.



> > Mathematical analysis should come after practical experience and not
> > before.
>
> No.
>

I agree with both of you here.  Speaking for myself, I want to know
how things work and how to design things.  The math then becomes a
required chore to get to my end goal.  The nice thing about hard math
is that it makes the barrier to entry more difficult, which then makes
me feel all the more special about myself (ok - kidding - not
completely kidding but mostly).


> > IMHO.
>
> You are plain wrong.
>
> Rune

On Dec 27, 2:31&#4294967295;pm, Tim Wescott <t...@seemywebsite.com> wrote:

> Convolution is _not_ the way that real systems in the real world respond
> to stimuli of any sort. &#4294967295;Convolution is just a _mathematical operation_
> that _approximates_ what real systems do. &#4294967295;Sometimes it even does it well.
>

I have read your thoughts on non-linear systems a couple of times.  I
see what you are saying, but a good  understanding of the ideal linear
system is first required (not an insignificant hurdle)



> All real systems are nonlinear. &#4294967295;The convolution operation is one way to
> implement a linear model of a system. &#4294967295;Thus, the convolution operation
> does not model any real system with 100% accuracy. &#4294967295;As a model, the
> convolution operation is only as good as the fit between its bedrock
> assumption of linearity and the system's actual conformity to linear
> behavior.

In the end, nothing can be modeled with absolute certainty.  In the
end, statistical processes are going to need to be applied.  But that
does not take away from the absolute need for good modeling.  Part of
the engineers job is to determine which model to use.

>
> For many systems, using convolution is a horribly indirect way to
> implement what should be a simple, limited-state, ordinary linear
> differential equation.
>
>
> I do agree that mathematical analysis should be kept firmly in the
> context of what is real -- when I teach control systems I try to draw
> examples from the real world as often as possible, and I try to keep a
> clear distinction between the thing you're interested in and the
> mathematical model that you've made of it.
>
> But then, you've already wandered away from reality if you're claiming
> that real systems convolve their input signals with unfailing accuracy.
>

But nobody is claiming unfailing accuracy.

> In today's world I don't think you can ask for practical experience
> before theoretical knowledge, though -- with that assumption, engineering
> schools would only take technicians who had already been through an
> apprenticeship, which severely cuts down on the available candidate pool.
>

For me it has been a steady growth, simultaneously of practical
knowledge and theoretical knowledge.  They feed off of each other.

I agree with what you say, but because you seem to support global
warming legislation, I just had to jump in and find something to
disagree with you on.

brent

On Dec 28, 10:50&#4294967295;am, Jerry Avins <j...@ieee.org> wrote:
> HardySpicer wrote:
>
> &#4294967295; &#4294967295;...
>
> > Agreed but it's pretty dammed close. I mean how linear is an R-C
> > network? Try adding two sine waves and passing them through an RC
> > network.
> > What cross-spectral terms to you get percentage wise? I would be
> > interested to know.
>
> Electrolytics are notoriously non-linear. High-value ceramic capacitors
> -- calcium titanate, for example -- are also non-linear. Dielectric
> absorption is a continuing problem with sample-and-hold capacitors.
>
> Non-linear resistors abound. Hewlwtt and Packard had the brilliance to
> use a tungsten resistor to stabilize the amplitude of a Wien-bridge
> oscillator and founded a company on the parent. (The movie Fantasia was
> an early user of their tunable audio oscillators.)
>
> The components we commonly use are explicitly selected to be adeqquately
> linear under their normal conditions of use. Most of the world doesn't
> work that way.
>
> Jerry
> --
> Engineering is the art of making what you want from things you can get.

Don't for get that linear convolution is a special case of the non-
linear problem which has an infinite series of nested  convolutions.
So convolution still holds (sort of).


Hardy

On Dec 27, 8:21&#4294967295;pm, HardySpicer <gyansor...@gmail.com> wrote:
>
> Don't forget that linear convolution is a special case of the non-
> linear problem which has an infinite series of nested &#4294967295;convolutions.

i dunno what you mean by "nested convolutions".  even the convolution
of a convolution is a linear operation which can be modeled with its
own aggregate convolution.

if what you mean that "linear convolution" is a special case of the
more general Volterra series model for a general time-invariant, non-
linear system with memory, then i agree with that.

> So convolution still holds (sort of).

Hardy, despite all the people picking on you, i'm more on your side.
you need not back down too far...

On Dec 27, 4:04 pm, HardySpicer <gyansor...@gmail.com> wrote:
> On Dec 28, 8:31 am, Tim Wescott <t...@seemywebsite.com> wrote:
>
>
>
> > On Sun, 27 Dec 2009 10:01:07 +0000, invalid wrote:
> > > "brent" <buleg...@columbus.rr.com> wrote in message
> > > news:0fd6f825-e7ad-4642-
>
> > a5fe-83de8ff8f...@x18g2000vbd.googlegroups.com...
>
> > >>I have created a tutorial on the convolution integral. It uses an
> > >> interactive flash program with embedded audio files. It is located
> > >> here:
> > >>http://www.fourier-series.com/Convolution/index.html
>
> > > You start off by saying that convolution is a mathematical operation, at
> > > which point I switched off.
>
> > > Convolution is the way that real systems in the real world (such as
> > > pianoforte strings)
> > > respond to stimuli that are continuous (such as a sine wave from a
> > > loudspeaker in close proximity)
>
> > Convolution is _not_ the way that real systems in the real world respond
> > to stimuli of any sort.

here, Tim, i might have to disagree.  if the real system in the real
world is sufficiently linear (operating within specs of which the
nonlinearities lie outside), then convolution is *precisely* the way
it responds to stimuli.

the real system that is operating essentially linearly is responding
(at the "present" time, t) to a stimulus t0 seconds ago (at time t-t0)
with a scaling that is proportional to h(t0), how it would respond to
an isolated impulse-like stimulus positioned at a time t0 seconds
ago.  and the real system ("sufficiently" linear) has superposition
applying; how it responds to a stimulus with two components, one 5 ms
ago and another 10 ms ago, will be the sum of what it would do for the
two of them as isolated stimuli.  for an RC circuit, that's what the
*physics* says as long as we keep the Volume knob set to something
below Arc-Weld (or "11" on Spinal Tap).

> >  Convolution is just a _mathematical operation_
> > that _approximates_ what real systems do.  Sometimes it even does it well.
>
> > All real systems are nonlinear.

even more generally, all systems deviate from whatever model we
describe them with.  that doesn't stop us from using models to
describe systems.  sometimes a linear model suffices.  sometimes you
can separate out a non-linear component from a linear subsystem with
memory.  when you can't, i guess there is Volterra series to fall back
on.

> > The convolution operation is one way to
> > implement a linear model of a system.

it's the *only* way, but there are multiple ways to implement
convolution.  but, physically, why real (analog) LTI systems
themselves perform convolution as the primary way that they operate is
because the primary manner that they operate is linear (superposition
applies) and time-invariant.  if, upon analysis of a real system, you
conclude that it intrinsically behaves linearly with superposition and
has no built-in clock (or someone twisting a knob), then you know it's
LTI and it *naturally* is performing convolution in its intrinsic
operation.

> >  Thus, the convolution operation
> > does not model any real system with 100% accuracy.

*only* because LTI doesn't.  but if you stay out of saturation, if
there aren't other little nonlinearities in the present state of the
physical system (like crossover, hysteresis, etc), then that LTI
system is *naturally* doing convolution.  it is not (likely) doing
convolution by continuously performing a Fourier Transform,
multiplying in the frequency domain, and inverse Fouriering back, but
for the RC circuit, as long as that capacitor is *accumulating* charge
in an additive manner, and as long as the resistor isn't burning up,
that RC filter is itself doing the convolution integral.

> >  As a model, the
> > convolution operation is only as good as the fit between its bedrock
> > assumption of linearity and the system's actual conformity to linear
> > behavior.
>
> > For many systems, using convolution is a horribly indirect way to
> > implement what should be a simple, limited-state, ordinary linear
> > differential equation.

actually implementing an IIR filter is just another way to do
convolution where a recursive expression satisfies the model.  it's
recursive convolution.

> > > and not just impulses (such as when hit
> > > with a hammer). I had difficulty with Convolution for years until it was
> > > explained to me in this practical way at which point it became
> > > meaningful
> > > instead of being some arcane mathematical operation which I did not
> > > really trust.
>
> > > Unless you introduce the student to the practical basis of why you would
> > > want to undertake such a weird operation, then you might as well give
> > > up.
>
> > > Mathematical analysis should come after practical experience and not
> > > before.
>
> > I do agree that mathematical analysis should be kept firmly in the
> > context of what is real -- when I teach control systems I try to draw
> > examples from the real world as often as possible, and I try to keep a
> > clear distinction between the thing you're interested in and the
> > mathematical model that you've made of it.
>
> > But then, you've already wandered away from reality if you're claiming
> > that real systems convolve their input signals with unfailing accuracy.
>
> > In today's world I don't think you can ask for practical experience
> > before theoretical knowledge, though -- with that assumption, engineering
> > schools would only take technicians who had already been through an
> > apprenticeship, which severely cuts down on the available candidate pool.
>
> Agreed but it's pretty dammed close. I mean how linear is an R-C
> network? Try adding two sine waves and passing them through an RC
> network.

...


that's my story and i'm sticking to it.

r b-j

>On Sun, 27 Dec 2009 10:01:07 +0000, invalid wrote:
>
>> "brent" <bulegoge@columbus.rr.com> wrote in message
>> news:0fd6f825-e7ad-4642-
>a5fe-83de8ff8f7f6@x18g2000vbd.googlegroups.com...
>>>I have created a tutorial on the convolution integral. It uses an
>>> interactive flash program with embedded audio files. It is located
>>> here:
>>> http://www.fourier-series.com/Convolution/index.html
>> 
>> You start off by saying that convolution is a mathematical operation,
at
>> which point I switched off.
>> 
>> Convolution is the way that real systems in the real world (such as
>> pianoforte strings)
>> respond to stimuli that are continuous (such as a sine wave from a
>> loudspeaker in close proximity) 
>
>Convolution is _not_ the way that real systems in the real world respond

>to stimuli of any sort.  Convolution is just a _mathematical operation_ 
>that _approximates_ what real systems do.  Sometimes it even does it
well.

Convolution *is* the way many real systems behave. Its not some arcane
mathematical trick. Its the direct mathematical representation of the
underlying physical process. How well it fits reality is generally a matter
of how much the system is affected by second order effects. This is pretty
much like any other area of science and engineering.

>All real systems are nonlinear.  The convolution operation is one way to

>implement a linear model of a system.  Thus, the convolution operation 
>does not model any real system with 100% accuracy.  As a model, the 
>convolution operation is only as good as the fit between its bedrock 
>assumption of linearity and the system's actual conformity to linear 
>behavior.

You must absolutely loath the entire scientific education system. Almost
everything is taught as if it obeys relatively simple relationships, and
that's pretty much always a first order approximation. Often the higher
order elements are so small you can largely ignore them. If you want
accuracy, you'd better scrap Newton's laws of motion.

If you really want to complain about people being taught about stuff like
its an real accurate model, look at the real villans, like how capacitors
are taught. The number of engineers who treat them like they are linear
devices is truly sad. They demand that the latest silicon can do A/D
conversion at high speed with >16 bits precision, and then surround them
with tiny surface mount capacitors who's characteristics are bizarrely
funky.

Steve

>On Sun, 27 Dec 2009 10:01:07 +0000, invalid wrote:
>
>> "brent" <bulegoge@columbus.rr.com> wrote in message
>> news:0fd6f825-e7ad-4642-
>a5fe-83de8ff8f7f6@x18g2000vbd.googlegroups.com...
>>>I have created a tutorial on the convolution integral. It uses an
>>> interactive flash program with embedded audio files. It is located
>>> here:
>>> http://www.fourier-series.com/Convolution/index.html
>> 
>> You start off by saying that convolution is a mathematical operation,
at
>> which point I switched off.
>> 
>> Convolution is the way that real systems in the real world (such as
>> pianoforte strings)
>> respond to stimuli that are continuous (such as a sine wave from a
>> loudspeaker in close proximity) 
>
>Convolution is _not_ the way that real systems in the real world respond

>to stimuli of any sort.  Convolution is just a _mathematical operation_ 
>that _approximates_ what real systems do.  Sometimes it even does it
well.

Convolution *is* the way many real systems behave. Its not some arcane
mathematical trick. Its the direct mathematical representation of the
underlying physical process. How well it fits reality is generally a matter
of how much the system is affected by second order effects. This is pretty
much like any other area of science and engineering.

>All real systems are nonlinear.  The convolution operation is one way to

>implement a linear model of a system.  Thus, the convolution operation 
>does not model any real system with 100% accuracy.  As a model, the 
>convolution operation is only as good as the fit between its bedrock 
>assumption of linearity and the system's actual conformity to linear 
>behavior.

You must absolutely loath the entire scientific education system. Almost
everything is taught as if it obeys relatively simple relationships, and
that's pretty much always a first order approximation. Often the higher
order elements are so small you can largely ignore them. If you want
accuracy, you'd better scrap Newton's laws of motion.

If you really want to complain about people being taught about stuff like
its an real accurate model, look at the real villans, like how capacitors
are taught. The number of engineers who treat them like they are linear
devices is truly sad. They demand that the latest silicon can do A/D
conversion at high speed with >16 bits precision, and then surround them
with tiny surface mount capacitors who's characteristics are bizarrely
funky.

Steve

>On Sun, 27 Dec 2009 10:01:07 +0000, invalid wrote:
>
>> "brent" <bulegoge@columbus.rr.com> wrote in message
>> news:0fd6f825-e7ad-4642-
>a5fe-83de8ff8f7f6@x18g2000vbd.googlegroups.com...
>>>I have created a tutorial on the convolution integral. It uses an
>>> interactive flash program with embedded audio files. It is located
>>> here:
>>> http://www.fourier-series.com/Convolution/index.html
>> 
>> You start off by saying that convolution is a mathematical operation,
at
>> which point I switched off.
>> 
>> Convolution is the way that real systems in the real world (such as
>> pianoforte strings)
>> respond to stimuli that are continuous (such as a sine wave from a
>> loudspeaker in close proximity) 
>
>Convolution is _not_ the way that real systems in the real world respond

>to stimuli of any sort.  Convolution is just a _mathematical operation_ 
>that _approximates_ what real systems do.  Sometimes it even does it
well.

Convolution *is* the way many real systems behave. Its not some arcane
mathematical trick. Its the direct mathematical representation of the
underlying physical process. How well it fits reality is generally a matter
of how much the system is affected by second order effects. This is pretty
much like any other area of science and engineering.

>All real systems are nonlinear.  The convolution operation is one way to

>implement a linear model of a system.  Thus, the convolution operation 
>does not model any real system with 100% accuracy.  As a model, the 
>convolution operation is only as good as the fit between its bedrock 
>assumption of linearity and the system's actual conformity to linear 
>behavior.

You must absolutely loath the entire scientific education system. Almost
everything is taught as if it obeys relatively simple relationships, and
that's pretty much always a first order approximation. Often the higher
order elements are so small you can largely ignore them. If you want
accuracy, you'd better scrap Newton's laws of motion.

If you really want to complain about people being taught about stuff like
its an real accurate model, look at the real villans, like how capacitors
are taught. The number of engineers who treat them like they are linear
devices is truly sad. They demand that the latest silicon can do A/D
conversion at high speed with >16 bits precision, and then surround them
with tiny surface mount capacitors who's characteristics are bizarrely
funky.

Steve