MLE : Random Vs Deterministic

Hi,

I am coming across a number of papers in signal processing where they
discuss using the Maximum Likelihood method with either a
deterministic or random signal model.

I can understand the deterministic signal model: you have some known
number of stationary signals with fixed, but unknown parameters which
has an additive noise component. From a physical point of view, this
makes absolute sense to me.

OTOH, I am having a difficult time grasping the concept behind the
random signal model. Is there a simple *intuitive* explanation that
someone might be able to offer me or perhaps a decent text/paper that
explains what is behind it?

MTIA,

Matt

On Jan 31, 4:40 am, junoexpress <MTBrenne...@gmail.com> wrote:
> Hi,
>
> I am coming across a number of papers in signal processing where they
> discuss using the Maximum Likelihood method with either a
> deterministic or random signal model.
>
> I can understand the deterministic signal model: you have some known
> number of stationary signals with fixed, but unknown parameters which
> has an additive noise component. From a physical point of view, this
> makes absolute sense to me.
>
> OTOH, I am having a difficult time grasping the concept behind the
> random signal model. Is there a simple *intuitive* explanation that
> someone might be able to offer me or perhaps a decent text/paper that
> explains what is behind it?
>
> MTIA,
>
> Matt

Dear Matt
I will explain you the probabilistic model, as I understand it, I beg
the group to correct me where I am wrong.

In the case of a deterministic signal, you have a fully deterministic
function that determines how each element of the function domain is
mapped onto the codomain. Example, the sine funcion, maps x from 0 to
infinite as sin(x). This is done deterministically.

In the case of a random signal, you are talking about a sampling
space. One of the most simple examples is a function sin(x+phi(x)),
where phi(x) is random variable that represents an aleatory phase, for
this particular example. The output probability of this random
variable is determined by its distribution function. In that case, we
say "phi(x) has a normal distrubution", or whatever distribution.
Usually these distributions have parameters by themselves, for
example, the gaussian distribution has an average mu and a deviation
sigma.

You can find a simple introduction in http://cnx.org/content/m10649/latest/.
Also, you can get the book Fundamentals of Statistical Processing,
Volume I: Estimation Theory, by Steven Kay, quite complete though a
little bit difficult.

If I am incorrect, I beg the group to correct me.

Hope this helps.

Juan Pablo

On 30 Jan, 20:40, junoexpress <MTBrenne...@gmail.com> wrote:
> Hi,
>
> I am coming across a number of papers in signal processing where they
> discuss using the Maximum Likelihood method with either a
> deterministic or random signal model.
>
> I can understand the deterministic signal model: you have some known
> number of stationary signals with fixed, but unknown parameters which
> has an additive noise component. From a physical point of view, this
> makes absolute sense to me.
>
> OTOH, I am having a difficult time grasping the concept behind the
> random signal model. Is there a simple *intuitive* explanation that
> someone might be able to offer me or perhaps a decent text/paper that
> explains what is behind it?

Start with your own explanation of the deterministic model above,
and note in particular the word "known."

If you have a situation where you *know* what type of signals
you will get, then deterministic methods might work. However,
that's a powerful constraint. Popular methods to analyze the
sum-of-sines model, like MUSIC and ESPRIT, will fail if you
pretend to *know* the number of sinusoidals present, but for
some reason or another get it wrong. This particular case
is conveniantly handled by inserting an order estimator
like Akaikes Information Criterion or Rissanen and Scwartz'
Minimum Description Length in the computations. The order
estimators use internal results of the MUSIC and ESPRIT
routines and try to estimate the actual number of sinusoidals
present.

But do note that this still reqires that the sum-of-sines model
is valid, which may or may not be the case.

Once you start questioning the fundamental assumptions like
that, and modify the philosophy (and computations) accordingly,
you end up with the stochastic model.

Rune

Hello.

> Start with your own explanation of the deterministic model above,
> and note in particular the word "known."
> 
> If you have a situation where you *know* what type of signals
> you will get, then deterministic methods might work. However,
> that's a powerful constraint. Popular methods to analyze the
> sum-of-sines model, like MUSIC and ESPRIT, will fail if you
> pretend to *know* the number of sinusoidals present, but for
> some reason or another get it wrong. This particular case
> is conveniantly handled by inserting an order estimator
> like Akaikes Information Criterion or Rissanen and Scwartz'
> Minimum Description Length in the computations. The order
> estimators use internal results of the MUSIC and ESPRIT
> routines and try to estimate the actual number of sinusoidals
> present.

Rune, you can find the order in a mathematically consistent way by using 
the orthogonality property of MUSIC and the shift-invariance property of 
ESPRIT by observing that those properties only hold for the right order. 
Both work under some mild conditions regardless of the noise pdf. 
There's no need for the statistical methods that require knowledge of 
the noise pdf.

Regards,
Mads

On 31 Jan, 13:21, Mads Gr&#4294967295;sb&#4294967295;ll Christensen <m...@nospam.es.aau.dk>
wrote:

> Rune, you can find the order in a mathematically consistent way by using
> the orthogonality property of MUSIC and the shift-invariance property of
> ESPRIT by observing that those properties only hold for the right order.
> Both work under some mild conditions regardless of the noise pdf.

The 'mild' conditions being

1) That the data comply to the sum-of-sines model
2) That the designer has specified a valid range where to
   search for the 'true' order

Both will get you into serious trouble in real-life applications.
I don't know of any real-life scenarios where these methods
are actually used.

Rune

>> Rune, you can find the order in a mathematically consistent way by using
>> the orthogonality property of MUSIC and the shift-invariance property of
>> ESPRIT by observing that those properties only hold for the right order.
>> Both work under some mild conditions regardless of the noise pdf.
> 
> The 'mild' conditions being
> 
> 1) That the data comply to the sum-of-sines model
> 2) That the designer has specified a valid range where to
>    search for the 'true' order
> 
> Both will get you into serious trouble in real-life applications.
> I don't know of any real-life scenarios where these methods
> are actually used.

Rune, you already stated that those were the assumptions behind MUSIC 
and ESPRIT and those are not the conditions I was referring to. I was 
addressing the issue of selecting the right order, not the validity of 
the model.

Regards,
Mads

On 31 Jan, 13:30, Mads Gr&#4294967295;sb&#4294967295;ll Christensen <m...@nospam.es.aau.dk>
wrote:
> >> Rune, you can find the order in a mathematically consistent way by using
> >> the orthogonality property of MUSIC and the shift-invariance property of
> >> ESPRIT by observing that those properties only hold for the right order.
> >> Both work under some mild conditions regardless of the noise pdf.
>
> > The 'mild' conditions being
>
> > 1) That the data comply to the sum-of-sines model
> > 2) That the designer has specified a valid range where to
> > &#4294967295; &#4294967295;search for the 'true' order
>
> > Both will get you into serious trouble in real-life applications.
> > I don't know of any real-life scenarios where these methods
> > are actually used.
>
> Rune, you already stated that those were the assumptions behind MUSIC
> and ESPRIT and those are not the conditions I was referring to. I was
> addressing the issue of selecting the right order, not the validity of
> the model.

But selecting the order doesn't make sense unless the model
fits the data. From a numerical point of view both MUSIC
and ESPRIT are rather robust wrt both noise and certain types
of model mismatch.

I happened to work interactively with full-scale seismic surveys
with arrays containing hundreds or thousands of elements. The
scale of the surveys ensured that there were sufficient numbers
or sensors for my array analysis, and by checking everything
interactively I could see what data sets were (not) fit for
analysis, as well as what parameter ranges to use.

My conclusion, after having tested these methods in seismic
data processing, was that the system designer needs ridiculously
detailed prior knowledge of what the output of these methods
would be, before specifying the analysis parameters.
I got access to that information because of the redundancy
of data and the interactive way of working, but I can't see
any other real-life setting where these sorts of methods
would work.

Rune

> But selecting the order doesn't make sense unless the model
> fits the data. From a numerical point of view both MUSIC
> and ESPRIT are rather robust wrt both noise and certain types
> of model mismatch.
> 
> I happened to work interactively with full-scale seismic surveys
> with arrays containing hundreds or thousands of elements. The
> scale of the surveys ensured that there were sufficient numbers
> or sensors for my array analysis, and by checking everything
> interactively I could see what data sets were (not) fit for
> analysis, as well as what parameter ranges to use.
> 
> My conclusion, after having tested these methods in seismic
> data processing, was that the system designer needs ridiculously
> detailed prior knowledge of what the output of these methods
> would be, before specifying the analysis parameters.
> I got access to that information because of the redundancy
> of data and the interactive way of working, but I can't see
> any other real-life setting where these sorts of methods
> would work.
> 
> Rune

I know. You have written this many times before. This was not the point 
of my post. I just commented that you don't need to involve the methods 
you wrote about. I have no idea why you seek to make this discussion 
about something else.

Both MUSIC and ESPRIT are in my experience helpful methods for analysis, 
modeling, etc., of speech and audio signals which is my area. There are 
certain problems that one needs to be aware of, though. But that goes 
for any method, even non-parametric ones.

Regards,
Mads.

On 31 Jan, 13:46, Mads Gr&#4294967295;sb&#4294967295;ll Christensen <m...@nospam.es.aau.dk>
wrote:
> I just commented that you don't need to involve the methods
> you wrote about.

The AIC and the MDL? I can't imagine anything simpler for
estimating the order. For MUSIC and ESPRIT they are basically
weighted sums of eigenvalues of the data ovariance matrix.
The hard part is not to compute them, but to ensure that
the order of the covariance matrix is larger than the
number of sinusoids present.

> I have no idea why you seek to make this discussion
> about something else.

The details don't make sense unless one already use the
correct model. 'A known model' is a far cry from a more
cautious 'hypothesis testing based on statistics' approach.
Which was where this thread started.

Rune

>> I just commented that you don't need to involve the methods
>> you wrote about.
> 
> The AIC and the MDL? I can't imagine anything simpler for
> estimating the order. For MUSIC and ESPRIT they are basically
> weighted sums of eigenvalues of the data ovariance matrix.
> The hard part is not to compute them, but to ensure that
> the order of the covariance matrix is larger than the
> number of sinusoids present.

You cannot evaluate the likelihood of those eigenvalues without making 
additional assumptions on the noise pdf, assumptions that are not 
required by ESPRIT and MUSIC. The fewer and less restrictive assumptions 
the better, no?

>> I have no idea why you seek to make this discussion
>> about something else.
> 
> The details don't make sense unless one already use the
> correct model. 'A known model' is a far cry from a more
> cautious 'hypothesis testing based on statistics' approach.
> Which was where this thread started.

Where did I write anything to the contrary? It has nothing to do with 
what I wrote.

Regards,
Mads