On Sat, 03 Dec 2011 21:28:40 -0800, zqchen wrote:> On 12月4日, 上午1时23分, Rune Allnor <all...@tele.ntnu.no> wrote: >> On 3 Des, 18:04, Tim Wescott <t...@seemywebsite.com> wrote: >> >> > On Sat, 03 Dec 2011 05:41:48 -0800, zqchen wrote: >> > > Consider an unknown DC level in a white Gaussian noise with zero >> > > mean and unknown variance. The value of the DC level is restricted >> > > to a range of [A0, A1], the observations, that is the DC level plus >> > > WGN are also restricted in this range. Then what is the optimal >> > > estimator for the DC level and the variance of the noise? >> >> > Nonlinear. >> >> > Tell us more. If it's a real application tell us what it is. If >> > it's homework, tell us that, or go talk to your TA. >> >> > We _will_ help you with homework -- we'll just help you differently >> > than we would if it were real work. >> >> And we *can* tell the difference, if you try and disguise a homework >> assignment as real work. >> >> Rune > > Think about a pixel with unknown grayscale value corrupted by WGN. The > grayscale value and the observation are all restricted in the range of 0 > to 255. If the value and noise variance are to be estimated, what are > the best estimator?Well, that depends on what's important to you. That's why I asked "best in what sense"? For a linear system (which this isn't) with Gaussian noise, the MMSE estimator of the signal and the maximum liklihood estimator are the same. But this is not the case in general. You're not even using the word "optimal", which would at least restrict your question to mathematics. No, you're using the word "best". Perhaps I think the best estimate is MMSE. Perhaps Rune thinks its ML. Perhaps you think it's whatever makes your customer pay you well and on time -- but since you do not share this information, we can't help you. The same problem exists with your variance estimator -- do you want maximum likelihood? MMSE? Do you want an unbiased estimate? Is biased OK if it's otherwise closer to 'truth'? If the variance of the Gaussian noise is small enough that your pixel data mostly stays in bounds, then just pretend that the signal isn't limited and be happy -- your estimate will be pretty close. If the variance of the Gaussian noise is so big that your pixel data is always pegged at 0 or 255 then you're pretty much out of luck. If it's somewhere in between, then an answer can probably be found -- but for the most part it'll be of academic interest only, a lot of work to find, and different for each different optimality criterion. -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
What is the optimal estimator?
Started by ●December 3, 2011
Reply by ●December 4, 20112011-12-04
Reply by ●December 4, 20112011-12-04
On Dec 4, 1:55 pm, Tim <t...@seemywebsite.please> wrote:> On Sat, 03 Dec 2011 21:28:40 -0800, zqchen wrote: > > On 12月4日, 上午1时23分, Rune Allnor <all...@tele.ntnu.no> wrote: > >> On 3 Des, 18:04, Tim Wescott <t...@seemywebsite.com> wrote: > > >> > On Sat, 03 Dec 2011 05:41:48 -0800, zqchen wrote: > >> > > Consider an unknown DC level in a white Gaussian noise with zero > >> > > mean and unknown variance. The value of the DC level is restricted > >> > > to a range of [A0, A1], the observations, that is the DC level plus > >> > > WGN are also restricted in this range. Then what is the optimal > >> > > estimator for the DC level and the variance of the noise? > > >> > Nonlinear. > > >> > Tell us more. If it's a real application tell us what it is. If > >> > it's homework, tell us that, or go talk to your TA. > > >> > We _will_ help you with homework -- we'll just help you differently > >> > than we would if it were real work. > > >> And we *can* tell the difference, if you try and disguise a homework > >> assignment as real work. > > >> Rune > > > Think about a pixel with unknown grayscale value corrupted by WGN. The > > grayscale value and the observation are all restricted in the range of 0 > > to 255. If the value and noise variance are to be estimated, what are > > the best estimator? > > Well, that depends on what's important to you. That's why I asked "best > in what sense"? For a linear system (which this isn't) with Gaussian > noise, the MMSE estimator of the signal and the maximum liklihood > estimator are the same. But this is not the case in general. > > You're not even using the word "optimal", which would at least restrict > your question to mathematics. No, you're using the word "best". Perhaps > I think the best estimate is MMSE. Perhaps Rune thinks its ML. Perhaps > you think it's whatever makes your customer pay you well and on time -- > but since you do not share this information, we can't help you. > > The same problem exists with your variance estimator -- do you want > maximum likelihood? MMSE? Do you want an unbiased estimate? Is biased > OK if it's otherwise closer to 'truth'? > > If the variance of the Gaussian noise is small enough that your pixel > data mostly stays in bounds, then just pretend that the signal isn't > limited and be happy -- your estimate will be pretty close. If the > variance of the Gaussian noise is so big that your pixel data is always > pegged at 0 or 255 then you're pretty much out of luck. If it's > somewhere in between, then an answer can probably be found -- but for the > most part it'll be of academic interest only, a lot of work to find, and > different for each different optimality criterion. > > -- > Tim Wescott > Control system and signal processing consultingwww.wescottdesign.comThe noise is zero mean. What is the ML estimator and what is the MMSE estimator for this case?
Reply by ●December 4, 20112011-12-04
On Dec 3, 11:57�pm, HardySpicer <gyansor...@gmail.com> wrote:> On Dec 4, 2:41�am, zqchen <zhiqun.c...@gmail.com> wrote: > > > Consider an unknown DC level in a white Gaussian noise with zero mean > > and unknown variance. The value of the DC level is restricted to arange of [A0, A1], the observations, that is the DC level plus WGN are > > also restricted in this range. Then what is the optimal estimator for > > the DC level and the variance of the noise? > > Just integrate it ie a low pass filter. (since a pure integrator would > drift off to infinity due to the dc level). Can't see how you would > get better than that. > > HardyYou can do better because there is information in the problem statement that you haven't used i.e. that the DC level is limited to a particular range. So for instance you could assume a uniform probability distribution for the DC portion of the signal (this may or not be true). Even with uniform apriori distribution I seem to recall that the solution does not have an analytical solution - see Steven Kay's book on estimation theory. Another approach is to use biased estimation - no apriori distribution is required. Since apparently you know the range of the DC values you can reduce the variance of the estimator over the DC parameter range. So in essence you give up the idea that on average I get the correct value for the DC level, but what you get is on average I'm closer to the correct DC level i.e. you have a smaller variance. This approach is discussed in an issue of the IEEE Signal Processing Magazine - 'Rethinking Biased Estimation" by Steven Kay and Yonina Eldar. I think this exact problem is discussed there. Cheers, Dave
Reply by ●December 4, 20112011-12-04
On Dec 4, 12:28 am, zqchen <zhiqun.c...@gmail.com> wrote:> On 12��4��, ����1ʱ23��, Rune Allnor <all...@tele.ntnu.no> wrote: > > > > > > > > > > > On 3 Des, 18:04, Tim Wescott <t...@seemywebsite.com> wrote: > > > > On Sat, 03 Dec 2011 05:41:48 -0800, zqchen wrote: > > > > Consider an unknown DC level in a white Gaussian noise with zero mean > > > > and unknown variance. The value of the DC level is restricted to a range > > > > of [A0, A1], the observations, that is the DC level plus WGN are also > > > > restricted in this range. Then what is the optimal estimator for the DC > > > > level and the variance of the noise? > > > > Nonlinear. > > > > Tell us more. If it's a real application tell us what it is. If it's > > > homework, tell us that, or go talk to your TA. > > > > We _will_ help you with homework -- we'll just help you differently than > > > we would if it were real work. > > > And we *can* tell the difference, if you try and > > disguise a homework assignment as real work. > > > Rune > > Think about a pixel with unknown grayscale value corrupted by WGN. The > grayscale value and the observation are all restricted in the range of > 0 to 255. If the value and noise variance are to be estimated, what > are the best estimator?I think you'll find this is more probablematic. Since you not only have the limited range of the value of the true pixel, but the limited range of pixel plus noise - so you're dealing with a nonlinear system. The degrees of nonlinearity will depend on the number of pixels that are close to either 0 or 255, and the noise power. The more noise power you have then the more likely the values of the pixels will be clipped to either 0 or 255. Dave
Reply by ●December 4, 20112011-12-04
On Sat, 03 Dec 2011 23:24:33 -0800, zqchen wrote:> The noise is zero mean. > > What is the ML estimator and what is the MMSE estimator for this case?Real live work to calculate. I'm sorry if Rune and I haven't gotten this across yet. The estimator is nonlinear, which means that there is no canonical answer. Someone is going to have to apply pencil to paper, hand to pencil, and brain to hand in order to figure out the answer. When they are done they will generate an equation whose practical use will may or may not be be fraught with complications, depending on the details of the situation, and which therefore should only be used by someone who understands what the heck it means, or in close consultation with that someone. Closer consultation than you'll get for free on a USENET newsgroup, by far. I would start this computation by writing a probability density function of the measured pixel value, then try to find a set of statistics that I could pull out of that PDF that would indicate the quantities that you want. If I actually found a couple of sufficient statistics I would be quite happy; if not I would try to pull out what I could and at least estimate the goodness of fit between my estimator and whatever the underlying reality might be. As Rune pointed out, this is all straight out of Van Trees, volume 1. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●December 5, 20112011-12-05
Why is the problem statement not a contradiction in terms? Pixels are all positive. That's a reasonable assertion. The noise is all positive. That's another reasonable assertion. If so, then there's no way to separate out the "DC" and the noise. A lowpass on the noise will yield a positive value. If the noise is white Gaussian, that must be before quantization and limiting then. So, the noise amplitude distribution must have a spike at 255 - maybe big and maybe small. Maybe the value of that spike helps measure the noise variance?? And, the distribution of values will peak at zero also won't it? (because it's zero mean). Anyway, it's not Gaussian any more is it? So when does that description apply? To be "DC" it can't change .. ever. There was no description of the number of temporal samples of these pixels. There seems to be an assumption that there is more than one sample in time. Maybe reasonable, maybe implied but not stated. Fred
Reply by ●December 5, 20112011-12-05
On 12��5��, ����2ʱ46��, Fred Marshall <fmarshallxremove_th...@acm.org> wrote:> Why is the problem statement not a contradiction in terms? > > Pixels are all positive. That's a reasonable assertion. > The noise is all positive. That's another reasonable assertion. > If so, then there's no way to separate out the "DC" and the noise. > A lowpass on the noise will yield a positive value. > > If the noise is white Gaussian, that must be before quantization and > limiting then. So, the noise amplitude distribution must have a spike > at 255 - maybe big and maybe small. Maybe the value of that spike helps > measure the noise variance?? And, the distribution of values will peak > at zero also won't it? (because it's zero mean). Anyway, it's not > Gaussian any more is it? So when does that description apply? > > To be "DC" it can't change .. ever. > > There was no description of the number of temporal samples of these > pixels. There seems to be an assumption that there is more than one > sample in time. Maybe reasonable, maybe implied but not stated. > > FredIt may be viewed as multiple samples from a gaussian distribution with unknown mean and unknown variance. And the sample values are clipped in some range. The mean also lies in this range. It will get worse if assume a distribution other than gaussian, right?
Reply by ●December 5, 20112011-12-05
On 5 Des, 15:02, zqchen <zhiqun.c...@gmail.com> wrote:> On 12��5��, ����2ʱ46��, Fred Marshall <fmarshallxremove_th...@acm.org> > wrote: > > > > > > > Why is the problem statement not a contradiction in terms? > > > Pixels are all positive. That's a reasonable assertion. > > The noise is all positive. That's another reasonable assertion. > > If so, then there's no way to separate out the "DC" and the noise. > > A lowpass on the noise will yield a positive value. > > > If the noise is white Gaussian, that must be before quantization and > > limiting then. So, the noise amplitude distribution must have a spike > > at 255 - maybe big and maybe small. Maybe the value of that spike helps > > measure the noise variance?? And, the distribution of values will peak > > at zero also won't it? (because it's zero mean). Anyway, it's not > > Gaussian any more is it? So when does that description apply? > > > To be "DC" it can't change .. ever. > > > There was no description of the number of temporal samples of these > > pixels. There seems to be an assumption that there is more than one > > sample in time. Maybe reasonable, maybe implied but not stated. > > > Fred > > It may be viewed as multiple samples from a gaussian distribution with > unknown mean and unknown variance. And the sample values are clipped > in some range. The mean also lies in this range. It will get worse if > assume a distribution other than gaussian, right?You can do this in two ways: 1) Assume a simple Gaussian distribution which is known to be wrong for the problem, as Fred explained, but where the estimators you ask for are known and listed in textbooks. 2) Find a different distirbution that conforms to the actual situation, but where you have to the hard work yourself, as previously explained by Tim. Rune
Reply by ●December 5, 20112011-12-05
On 12/5/2011 1:46 AM, Fred Marshall wrote:> Why is the problem statement not a contradiction in terms? > > Pixels are all positive. That's a reasonable assertion. > The noise is all positive. That's another reasonable assertion. > If so, then there's no way to separate out the "DC" and the noise. > A lowpass on the noise will yield a positive value. > > If the noise is white Gaussian, that must be before quantization and > limiting then. So, the noise amplitude distribution must have a spike at > 255 - maybe big and maybe small. Maybe the value of that spike helps > measure the noise variance?? And, the distribution of values will peak > at zero also won't it? (because it's zero mean). Anyway, it's not > Gaussian any more is it? So when does that description apply? > > To be "DC" it can't change .. ever. > > There was no description of the number of temporal samples of these > pixels. There seems to be an assumption that there is more than one > sample in time. Maybe reasonable, maybe implied but not stated.I keep coming back to my earlier question. How can there be a substantial DC level in a signal specified to be zero mean? Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●December 5, 20112011-12-05
On 12/5/2011 7:13 AM, Jerry Avins wrote:> On 12/5/2011 1:46 AM, Fred Marshall wrote: >> Why is the problem statement not a contradiction in terms? >> >> Pixels are all positive. That's a reasonable assertion. >> The noise is all positive. That's another reasonable assertion. >> If so, then there's no way to separate out the "DC" and the noise. >> A lowpass on the noise will yield a positive value. >> >> If the noise is white Gaussian, that must be before quantization and >> limiting then. So, the noise amplitude distribution must have a spike at >> 255 - maybe big and maybe small. Maybe the value of that spike helps >> measure the noise variance?? And, the distribution of values will peak >> at zero also won't it? (because it's zero mean). Anyway, it's not >> Gaussian any more is it? So when does that description apply? >> >> To be "DC" it can't change .. ever. >> >> There was no description of the number of temporal samples of these >> pixels. There seems to be an assumption that there is more than one >> sample in time. Maybe reasonable, maybe implied but not stated. > > I keep coming back to my earlier question. How can there be a > substantial DC level in a signal specified to be zero mean? > > JerryJerry, I thought only the noise was stated to be zero mean. Although the sentence might leave some question. Fred
