What is consistency as regards random variables. I know that to be unbiased we have E[x] = the actual mean but does consistency mean that an estimator could be unbiased yet not consistent? What in words does this mean?
consistency
Started by ●November 10, 2008
Reply by ●November 10, 20082008-11-10
HardySpicer <gyansorova@gmail.com> writes:> What is consistency as regards random variables. I know that to be > unbiased we have E[x] = the actual mean but does consistency mean that > an estimator could be unbiased yet not consistent? What in words does > this mean?Hi Hardy, [mcdonough] defines an estimator x_n(_y_), where _y_ is a length-n vector, as "consistent" when it has the property that, for an small number e, lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. This is also called "convergence in probability" (see, e.g., [viniotis]). --Randy @book{mcdonough, title = "Detection of Signals in Noise", author = "{Robert~N.~McDonough and Anthony~D.~Whalen}", publisher = "Academic Press", year = "1995"} @BOOK{viniotis, title = "{Probability and Random Processes for Electrical Engineers}", author = "{Yannis~Viniotis}", publisher = "WCB McGraw-Hill", year = "1998"} -- % Randy Yates % "Though you ride on the wheels of tomorrow, %% Fuquay-Varina, NC % you still wander the fields of your %%% 919-577-9882 % sorrow." %%%% <yates@ieee.org> % '21st Century Man', *Time*, ELO http://www.digitalsignallabs.com
Reply by ●November 10, 20082008-11-10
On 10 Nov, 14:32, Randy Yates <ya...@ieee.org> wrote:> HardySpicer <gyansor...@gmail.com> writes: > > What is consistency as regards random variables. I know that to be > > unbiased we have E[x] = the actual mean but does consistency mean that > > an estimator could be unbiased yet not consistent? What in words does > > this mean? > > Hi Hardy, > > [mcdonough] defines an estimator x_n(_y_), where _y_ is a > length-n vector, as "consistent" when it has the property > that, for an small number e, > > � lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. > > This is also called "convergence in probability" (see, e.g., > [viniotis]).Maybe this is the same as you said, but my understanding is that a consistent estimator is an estimator which has the two properties that 1) the estimate is unbiased 2) the variance vanishes as the number of samples goes to infinity. Rune
Reply by ●November 10, 20082008-11-10
Rune Allnor <allnor@tele.ntnu.no> writes:> On 10 Nov, 14:32, Randy Yates <ya...@ieee.org> wrote: >> HardySpicer <gyansor...@gmail.com> writes: >> > What is consistency as regards random variables. I know that to be >> > unbiased we have E[x] = the actual mean but does consistency mean that >> > an estimator could be unbiased yet not consistent? What in words does >> > this mean? >> >> Hi Hardy, >> >> [mcdonough] defines an estimator x_n(_y_), where _y_ is a >> length-n vector, as "consistent" when it has the property >> that, for an small number e, >> >> � lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. >> >> This is also called "convergence in probability" (see, e.g., >> [viniotis]). > > Maybe this is the same as you said, but my understanding > is that a consistent estimator is an estimator which > has the two properties that > > 1) the estimate is unbiasedI think that's true.> 2) the variance vanishes as the number of samples goes to > infinity.I think so, Rune. I'm not a master at this analysis stuff so that's just my amateur understanding. It's the same property that the "estimate the power spectrum with one long n-point FFT" approach to power spectrum estimation fails, and is why we have to take the extra data, cut it into sections, and average. -- % Randy Yates % "She tells me that she likes me very much, %% Fuquay-Varina, NC % but when I try to touch, she makes it %%% 919-577-9882 % all too clear." %%%% <yates@ieee.org> % 'Yours Truly, 2095', *Time*, ELO http://www.digitalsignallabs.com
Reply by ●November 10, 20082008-11-10
Randy Yates <yates@ieee.org> writes:> [...] > for an small number eShould be "for any small number e". -- % Randy Yates % "She's sweet on Wagner-I think she'd die for Beethoven. %% Fuquay-Varina, NC % She love the way Puccini lays down a tune, and %%% 919-577-9882 % Verdi's always creepin' from her room." %%%% <yates@ieee.org> % "Rockaria", *A New World Record*, ELO http://www.digitalsignallabs.com
Reply by ●November 10, 20082008-11-10
PS: Here is my "cheatsheet" for the random processes class I had at NCSU. See the "Modes of Convergence" section for a more complete definition of convergence in probability and other modes. http://galois.digitalsignallabs.com/cheatsheet.pdf --Randy Randy Yates <yates@ieee.org> writes:> HardySpicer <gyansorova@gmail.com> writes: > >> What is consistency as regards random variables. I know that to be >> unbiased we have E[x] = the actual mean but does consistency mean that >> an estimator could be unbiased yet not consistent? What in words does >> this mean? > > Hi Hardy, > > [mcdonough] defines an estimator x_n(_y_), where _y_ is a > length-n vector, as "consistent" when it has the property > that, for an small number e, > > lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. > > This is also called "convergence in probability" (see, e.g., > [viniotis]). > > --Randy > > @book{mcdonough, > title = "Detection of Signals in Noise", > author = "{Robert~N.~McDonough and Anthony~D.~Whalen}", > publisher = "Academic Press", > year = "1995"} > @BOOK{viniotis, > title = "{Probability and Random Processes for Electrical Engineers}", > author = "{Yannis~Viniotis}", > publisher = "WCB McGraw-Hill", > year = "1998"} > > -- > % Randy Yates % "Though you ride on the wheels of tomorrow, > %% Fuquay-Varina, NC % you still wander the fields of your > %%% 919-577-9882 % sorrow." > %%%% <yates@ieee.org> % '21st Century Man', *Time*, ELO > http://www.digitalsignallabs.com-- % Randy Yates % "Midnight, on the water... %% Fuquay-Varina, NC % I saw... the ocean's daughter." %%% 919-577-9882 % 'Can't Get It Out Of My Head' %%%% <yates@ieee.org> % *El Dorado*, Electric Light Orchestra http://www.digitalsignallabs.com
Reply by ●November 11, 20082008-11-11
Randy Yates wrote:> Rune Allnor <all...@tele.ntnu.no> writes: > > On 10 Nov, 14:32, Randy Yates <ya...@ieee.org> wrote: > >> HardySpicer <gyansor...@gmail.com> writes: > >> > What is consistency as regards random variables. I know that to be > >> > unbiased we have E[x] = the actual mean but does consistency mean that > >> > an estimator could be unbiased yet not consistent? What in words does > >> > this mean? > > >> Hi Hardy, > > >> [mcdonough] defines an estimator x_n(_y_), where _y_ is a > >> length-n vector, as "consistent" when it has the property > >> that, for an small number e, > > >> � lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. > > >> This is also called "convergence in probability" (see, e.g., > >> [viniotis]). > > > Maybe this is the same as you said, but my understanding > > is that a consistent estimator is an estimator which > > has the two properties that > > > 1) the estimate is unbiased > > I think that's true.But it's not true (see below).> > > 2) the variance vanishes as the number of samples goes to > > � �infinity. > > I think so, Rune.This second point is necessary for consistency. If you assume that the estimator x_n results from estimating some parameter from n i.i.d. observations and let n->oo, then a consistent estimator may be biased for all n but the bias has to disappear as n->oo ("asymptotically unbiased") and the variance has to disappear,too. An example of an unbiased estimator where the variance does not disappear is a bin of the periodogramm (estimating the PSD at that frequency). An example of a biased estimator that is consistent is the sample variance (estimating the distribution variance). Regards, Andor
Reply by ●November 12, 20082008-11-12
Andor <andor.bariska@gmail.com> writes:> Randy Yates wrote: >> Rune Allnor <all...@tele.ntnu.no> writes: >> > On 10 Nov, 14:32, Randy Yates <ya...@ieee.org> wrote: >> >> HardySpicer <gyansor...@gmail.com> writes: >> >> > What is consistency as regards random variables. I know that to be >> >> > unbiased we have E[x] = the actual mean but does consistency mean that >> >> > an estimator could be unbiased yet not consistent? What in words does >> >> > this mean? >> >> >> Hi Hardy, >> >> >> [mcdonough] defines an estimator x_n(_y_), where _y_ is a >> >> length-n vector, as "consistent" when it has the property >> >> that, for an small number e, >> >> >> � lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. >> >> >> This is also called "convergence in probability" (see, e.g., >> >> [viniotis]). >> >> > Maybe this is the same as you said, but my understanding >> > is that a consistent estimator is an estimator which >> > has the two properties that >> >> > 1) the estimate is unbiased >> >> I think that's true. > > But it's not true (see below). > >> >> > 2) the variance vanishes as the number of samples goes to >> > � �infinity. >> >> I think so, Rune. > > This second point is necessary for consistency. > > If you assume that the estimator x_n results from estimating some > parameter from n i.i.d. observations and let n->oo, then a consistent > estimator may be biased for all n but the bias has to disappear as n- >>oo ("asymptotically unbiased") and the variance has to disappear, > too.Hi Andor, At this point, all I can do is say "OK, if you say so." I'm not sure how to show this to myself rigorously (analytically), so I guess I'll just shutup... It would be very interesting to me, however, if you could explain why you had to condition that statement with the assumption that the estimator inputs are iid. Aren't there potentially consistent-but-biased estimators for other classes of random processes? -- % Randy Yates % "Ticket to the moon, flight leaves here today %% Fuquay-Varina, NC % from Satellite 2" %%% 919-577-9882 % 'Ticket To The Moon' %%%% <yates@ieee.org> % *Time*, Electric Light Orchestra http://www.digitalsignallabs.com
Reply by ●November 13, 20082008-11-13
On 13 Nov., 02:00, Randy Yates <ya...@ieee.org> wrote:> Andor <andor.bari...@gmail.com> writes: > > Randy Yates wrote: > >> Rune Allnor <all...@tele.ntnu.no> writes: > >> > On 10 Nov, 14:32, Randy Yates <ya...@ieee.org> wrote: > >> >> HardySpicer <gyansor...@gmail.com> writes: > >> >> > What is consistency as regards random variables. I know that to be > >> >> > unbiased we have E[x] = the actual mean but does consistency mean that > >> >> > an estimator could be unbiased yet not consistent? What in words does > >> >> > this mean? > > >> >> Hi Hardy, > > >> >> [mcdonough] defines an estimator x_n(_y_), where _y_ is a > >> >> length-n vector, as "consistent" when it has the property > >> >> that, for an small number e, > > >> >> � lim_{n --> infty} P( | x_n(_y_) - x | > e) = 0. > > >> >> This is also called "convergence in probability" (see, e.g., > >> >> [viniotis]). > > >> > Maybe this is the same as you said, but my understanding > >> > is that a consistent estimator is an estimator which > >> > has the two properties that > > >> > 1) the estimate is unbiased > > >> I think that's true. > > > But it's not true (see below). > > >> > 2) the variance vanishes as the number of samples goes to > >> > � �infinity. > > >> I think so, Rune. > > > This second point is necessary for consistency. > > > If you assume that the estimator x_n results from estimating some > > parameter from n i.i.d. observations and let n->oo, then a consistent > > estimator may be biased for all n but the bias has to disappear as n- > >>oo ("asymptotically unbiased") and the variance has to disappear, > > too. > > Hi Andor, > > At this point, all I can do is say "OK, if you say so." I'm > not sure how to show this to myself rigorously (analytically), > so I guess I'll just shutup... > > It would be very interesting to me, however, if you could explain why > you had to condition that statement with the assumption that the > estimator inputs are iid. Aren't there potentially consistent-but-biased > estimators for other classes of random processes?Hi Randy It's more a matter of convenince. You need a sequence of random variables x_n in the first place (consistency is a convergence result, so you need a sequence that converges to somewhere). Estimators on growing sets of "data" (itself random varibles) are typical sequences that can converge. Assuming the "data" to be i.i.d just makes things simple. Regards, Andor
Reply by ●November 14, 20082008-11-14
On 13 Nov, 02:00, Randy Yates <ya...@ieee.org> wrote:> At this point, all I can do is say "OK, if you say so." I'm > not sure how to show this to myself rigorously (analytically), > so I guess I'll just shutup...Don't. Don't shut up. It doesn't matter what the analysis says. The accepted use[*] of the term 'consistent estimator' is exactly what I already said: That the estimator is unbiased *and* the variance vanishes when the number of samples become large. [[*] I don't use the term 'definition' since I haven't found anything that qualifies as a definition in the textbooks; that's probably because the term is considered to be so trivial everyone are expected to know. ] So when the OP asks "what does it mean that an estimator is unbiased but not consistent" the answer is that "the estimator's variance doesn't vanish as the number of data increases." You just can't analyze your way out of that. The estimate for the mean is either biased or unbiased. The estimate for the variance either vanishes or doesn't when the number of samples increases to infinity. The mean doesn't govern the asymptotic behaviour of the variance, so you can tell nothing about the asymptotic behaviour of the variance by analyzing the mean. Which is the very reason why one uses *both* mean *and* variance to characterize statistics. (If they were that closely interconnected, one would not need to estimate both.) This is statistics 101. One example of a non-consistent estimator the periodogram, where the mean is E[periodogram(f,N)] = Power Spectrum Density(f) where periodogram(f,N) means "periodogram coefficient at freqency f based on N data points". I am sure you have seen somewhere (or know where to look up) that Var(periodogram(f,N)) = PSD(f)^2 That is, the variance doesn't depend on the number of samples so it will not improve (that is, reduce) as the number of samples increases. So the periodogram is not a consistent estimator for the PSD. Which is the direct reason why there are so many roundabout, awkward ways to estimate the PSD. Randy, I know you have taken a few classes and put in a lot of work over the past few years, so I am confident that you know where to look to check these things out for yourself. As you do, and grow more confident that all that work you invested in the classes actually paied off, one unfortunate side effect is that you might start evaluating what you hear from colleagues and peers, and you will come to see that not everything they say make sense. That's OK, no one can be right every time. The difficult part is how people relate to this simple fact of life. Some people are just too stupid or too stubborn to realize that they need to learn, and refuse to go back to the basics more or less as a matter of principle. There's nothing you can do about that, except map out who those people might be. Such people will inhibit you in your struggles to get a pay-off for you hard-earned education. So you need to know what to look for, in order to identify (and maybe handle) them. Some common traits: - They have a carreer in a different field than where they have their education or base training - They very often have a high academic degree (PhD) in a more 'aristocratic' field than where they have their carreer - They may have decades of seniority (but in administrative positions) - They hold hold positions of percieved authority - They have never been challenged or had to defend a position in professional matters - They may have been over-promoted and try to compensate for professional insecurity Rune
