Randy Yates <yates@digitalsignallabs.com> wrote: (snip)> Yes. See the table under the "Internal Representation" section here:> http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems> The point I was making is exactly why the "Bits Precision" column is one > greater than the "Significand" column for Single and Double in that > table.But people aren't all that good at counting digits. Most will say that 1.00 and 9.99 both have three significant digits, but 1.00 has (just about) the same significance as 0.99. That is, 1.00 has log10(100) significant digits and 9.99 has log10(999) significant digits. Similarly, IEEE double has between 52 and 53 bits of significance, depending on the value. When I learned significant digits, you rounded to one more than the significant digits that you might otherwise expect, partly to allow for this. -- glen
Convert to higher sample rate
Started by ●April 13, 2014
Reply by ●April 17, 20142014-04-17
Reply by ●April 17, 20142014-04-17
On 4/17/14 4:27 PM, Randy Yates wrote:> robert bristow-johnson<rbj@audioimagination.com> writes: > >> On 4/17/14 12:38 AM, Randy Yates wrote: >>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>> [...] >>>> the real problem is with the cosine of teeny values. because then >>>> it's only in the *difference* between cos(theta) and 1 that tells you >>>> anything about what theta is. and that hidden 1 bit you get in floats >>>> doesn't save your ass as it does with sin(theta). >>> >>> Why doesn't it? The "hidden 1" bit is about normalization, right? It >>> allows you to use one more bit for the mantissa than you otherwise would >>> have, >> >> no, > > Yes. See the table under the "Internal Representation" section here: > > http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems > > The point I was making is exactly why the "Bits Precision" column is one > greater than the "Significand" column for Single and Double in that > table.Randy, i think you missed the point or i just didn't explain it good enough. when your fixed-point binary number looks like 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv single floating point preserves all of the "x" bits but not the "v" bits. when your fixed-point binary number looks like 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv single precision floating point will *not* preserve all of the "x" bits. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●April 17, 20142014-04-17
>> I forsee a day when all bands will write songs that are 2^n sampleslong. Well, it might slow down to the point where you'll actually notice the difference. So what? _____________________________ Posted through www.DSPRelated.com
Reply by ●April 17, 20142014-04-17
On 4/17/14 5:47 PM, robert bristow-johnson wrote: ...> when your fixed-point binary number looks like > > 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv > > > single floating point preserves all of the "x" bits but not the "v" bits.i was off by one bit. should be: when your fixed-point binary number looks like 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxvvvvvvvv single-precision floating point preserves all of the "x" bits but not the "v" bits. when your fixed-point binary number looks like 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxvvvvvvvv single-precision floating point will *not* preserve all of the "x" bits. it will drop 14 of the "x" bits. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●April 18, 20142014-04-18
robert bristow-johnson <rbj@audioimagination.com> writes:> On 4/17/14 4:27 PM, Randy Yates wrote: >> robert bristow-johnson<rbj@audioimagination.com> writes: >> >>> On 4/17/14 12:38 AM, Randy Yates wrote: >>>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>>> [...] >>>>> the real problem is with the cosine of teeny values. because then >>>>> it's only in the *difference* between cos(theta) and 1 that tells you >>>>> anything about what theta is. and that hidden 1 bit you get in floats >>>>> doesn't save your ass as it does with sin(theta). >>>> >>>> Why doesn't it? The "hidden 1" bit is about normalization, right? It >>>> allows you to use one more bit for the mantissa than you otherwise would >>>> have, >>> >>> no, >> >> Yes. See the table under the "Internal Representation" section here: >> >> http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems >> >> The point I was making is exactly why the "Bits Precision" column is one >> greater than the "Significand" column for Single and Double in that >> table. > > Randy, i think you missed the point or i just didn't explain it good > enough. when your fixed-point binary number looks like > > 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv > > > single floating point preserves all of the "x" bits but not the "v" bits. > > when your fixed-point binary number looks like > > 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv > > single precision floating point will *not* preserve all of the "x" bits.Robert, I'm not sure we agree on the real issue here. Perhaps that's the reason for some of the confusion. FACT: Both large numbers (e.g., those near 1) and small numbers (those near 0) have the same number of significant digits in IEEE-754 floating point format. FACT: The ONLY purpose the "hidden bit" serves is to allow one more significant digit than you otherwise would have with a direct representation. FACT: The number of significant digits is directly related to the "relative error" (see section 1.2 in [higham]). So.., while your statement above is true, I would say "So what?"! The number of significant digits in the second value (53) is greater than the number in the first value (39). So if you're using a representation that has less than 53 bits but more than 38 bits, you'll preserve the first but not the second. Of course. That sort of analysis doesn't prove that numbers near one are less accurate than those near zero. In fact the IEEE 754 format ensures both are represented with the same accuracy (or equivalently, the same relative error). As Higham shows so well in section 1.7, the problem is that SUBTRACTIVE CANCELLATION MAGNIFIES RELATIVE ERRORS. In fact the same magnification could happen with numbers near zero. Thus when you attribute the hidden bit as the cause of these sorts of problems, and implying that this only happens when subtracting numbers near 1, I think you're misidentifying the issue. --Randy @BOOK{higham, title = "{Accuracy and Stability of Numerical Algorithms}", author = "{Nicholas~J.~Higham}", publisher = "SIAM", year = "1996"} -- Randy Yates Digital Signal Labs http://www.digitalsignallabs.com
Reply by ●April 18, 20142014-04-18
PS: 1. Single precision would not preserve all the bits of either of the values you presented since it only has 24 bits of precision. 2. The "represents all number with the same relative error" statement only works with numbers within the "dynamic" range of the representation. For example, if the numbers get too small, you "underflow" the exponent and can no longer normalize --Randy Randy Yates <yates@digitalsignallabs.com> writes:> robert bristow-johnson <rbj@audioimagination.com> writes: > >> On 4/17/14 4:27 PM, Randy Yates wrote: >>> robert bristow-johnson<rbj@audioimagination.com> writes: >>> >>>> On 4/17/14 12:38 AM, Randy Yates wrote: >>>>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>>>> [...] >>>>>> the real problem is with the cosine of teeny values. because then >>>>>> it's only in the *difference* between cos(theta) and 1 that tells you >>>>>> anything about what theta is. and that hidden 1 bit you get in floats >>>>>> doesn't save your ass as it does with sin(theta). >>>>> >>>>> Why doesn't it? The "hidden 1" bit is about normalization, right? It >>>>> allows you to use one more bit for the mantissa than you otherwise would >>>>> have, >>>> >>>> no, >>> >>> Yes. See the table under the "Internal Representation" section here: >>> >>> http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems >>> >>> The point I was making is exactly why the "Bits Precision" column is one >>> greater than the "Significand" column for Single and Double in that >>> table. >> >> Randy, i think you missed the point or i just didn't explain it good >> enough. when your fixed-point binary number looks like >> >> 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >> >> >> single floating point preserves all of the "x" bits but not the "v" bits. >> >> when your fixed-point binary number looks like >> >> 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >> >> single precision floating point will *not* preserve all of the "x" bits. > > Robert, > > I'm not sure we agree on the real issue here. Perhaps that's the reason > for some of the confusion. > > FACT: Both large numbers (e.g., those near 1) and small numbers (those > near 0) have the same number of significant digits in IEEE-754 floating > point format. > > FACT: The ONLY purpose the "hidden bit" serves is to allow one more > significant digit than you otherwise would have with a direct > representation. > > FACT: The number of significant digits is directly related to the > "relative error" (see section 1.2 in [higham]). > > So.., while your statement above is true, I would say "So what?"! The > number of significant digits in the second value (53) is greater than > the number in the first value (39). So if you're using a representation > that has less than 53 bits but more than 38 bits, you'll preserve the > first but not the second. Of course. > > That sort of analysis doesn't prove that numbers near one are less > accurate than those near zero. In fact the IEEE 754 format ensures both > are represented with the same accuracy (or equivalently, the same > relative error). > > As Higham shows so well in section 1.7, the problem is that SUBTRACTIVE > CANCELLATION MAGNIFIES RELATIVE ERRORS. > > In fact the same magnification could happen with numbers near zero. > > Thus when you attribute the hidden bit as the cause of these sorts of > problems, and implying that this only happens when subtracting numbers > near 1, I think you're misidentifying the issue. > > --Randy > > > @BOOK{higham, > title = "{Accuracy and Stability of Numerical Algorithms}", > author = "{Nicholas~J.~Higham}", > publisher = "SIAM", > year = "1996"}-- Randy Yates Digital Signal Labs http://www.digitalsignallabs.com
Reply by ●April 18, 20142014-04-18
On 4/18/14 10:23 AM, Randy Yates wrote:> PS: > > 1. Single precision would not preserve all the bits of either of the > values you presented since it only has 24 bits of precision. > > 2. The "represents all number with the same relative error" statement > only works with numbers within the "dynamic" range of the > representation. For example, if the numbers get too small, you > "underflow" the exponent and can no longer normalize > > --Randy > > > > Randy Yates<yates@digitalsignallabs.com> writes: > >> robert bristow-johnson<rbj@audioimagination.com> writes: >> >>> On 4/17/14 4:27 PM, Randy Yates wrote: >>>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>> >>>>> On 4/17/14 12:38 AM, Randy Yates wrote: >>>>>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>>>>> [...] >>>>>>> the real problem is with the cosine of teeny values. because then >>>>>>> it's only in the *difference* between cos(theta) and 1 that tells you >>>>>>> anything about what theta is. and that hidden 1 bit you get in floats >>>>>>> doesn't save your ass as it does with sin(theta). >>>>>> >>>>>> Why doesn't it? The "hidden 1" bit is about normalization, right? It >>>>>> allows you to use one more bit for the mantissa than you otherwise would >>>>>> have, >>>>> >>>>> no, >>>> >>>> Yes. See the table under the "Internal Representation" section here: >>>> >>>> http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems >>>> >>>> The point I was making is exactly why the "Bits Precision" column is one >>>> greater than the "Significand" column for Single and Double in that >>>> table. >>> >>> Randy, i think you missed the point or i just didn't explain it good >>> enough. when your fixed-point binary number looks like >>> >>> 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >>> >>> >>> single floating point preserves all of the "x" bits but not the "v" bits. >>> >>> when your fixed-point binary number looks like >>> >>> 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >>> >>> single precision floating point will *not* preserve all of the "x" bits. >> >> Robert, >> >> I'm not sure we agree on the real issue here. Perhaps that's the reason >> for some of the confusion. >> >> FACT: Both large numbers (e.g., those near 1) and small numbers (those >> near 0) have the same number of significant digits in IEEE-754 floating >> point format. >> >> FACT: The ONLY purpose the "hidden bit" serves is to allow one more >> significant digit than you otherwise would have with a direct >> representation. >> >> FACT: The number of significant digits is directly related to the >> "relative error" (see section 1.2 in [higham]). >> >> So.., while your statement above is true, I would say "So what?"! The >> number of significant digits in the second value (53) is greater than >> the number in the first value (39). So if you're using a representation >> that has less than 53 bits but more than 38 bits, you'll preserve the >> first but not the second. Of course. >> >> That sort of analysis doesn't prove that numbers near one are less >> accurate than those near zero. In fact the IEEE 754 format ensures both >> are represented with the same accuracy (or equivalently, the same >> relative error). >> >> As Higham shows so well in section 1.7, the problem is that SUBTRACTIVE >> CANCELLATION MAGNIFIES RELATIVE ERRORS. >> >> In fact the same magnification could happen with numbers near zero. >> >> Thus when you attribute the hidden bit as the cause of these sorts of >> problems, and implying that this only happens when subtracting numbers >> near 1, I think you're misidentifying the issue. >> >> --Randy >> >> >> @BOOK{higham, >> title = "{Accuracy and Stability of Numerical Algorithms}", >> author = "{Nicholas~J.~Higham}", >> publisher = "SIAM", >> year = "1996"} >-- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●April 18, 20142014-04-18
On 4/18/14 10:15 AM, Randy Yates wrote:> robert bristow-johnson<rbj@audioimagination.com> writes: > >> On 4/17/14 4:27 PM, Randy Yates wrote: >>> robert bristow-johnson<rbj@audioimagination.com> writes: >>> >>>> On 4/17/14 12:38 AM, Randy Yates wrote: >>>>> robert bristow-johnson<rbj@audioimagination.com> writes: >>>>>> [...] >>>>>> the real problem is with the cosine of teeny values. because then >>>>>> it's only in the *difference* between cos(theta) and 1 that tells you >>>>>> anything about what theta is. and that hidden 1 bit you get in floats >>>>>> doesn't save your ass as it does with sin(theta). >>>>> >>>>> Why doesn't it? The "hidden 1" bit is about normalization, right? It >>>>> allows you to use one more bit for the mantissa than you otherwise would >>>>> have, >>>> >>>> no, >>> >>> Yes. See the table under the "Internal Representation" section here: >>> >>> http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems >>> >>> The point I was making is exactly why the "Bits Precision" column is one >>> greater than the "Significand" column for Single and Double in that >>> table. >> >> Randy, i think you missed the point or i just didn't explain it good >> enough. when your fixed-point binary number looks like >> >> 0.000000000000001xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >> >> >> single floating point preserves all of the "x" bits but not the "v" bits. >> >> when your fixed-point binary number looks like >> >> 0.111111111111111xxxxxxxxxxxxxxxxxxxxxxxxvvvvvvvvvvvvvv >> >> single precision floating point will *not* preserve all of the "x" bits. > > Robert, > > I'm not sure we agree on the real issue here. Perhaps that's the reason > for some of the confusion. > > FACT: Both large numbers (e.g., those near 1) and small numbers (those > near 0) have the same number of significant digits in IEEE-754 floating > point format.depends on what you mean by "significant". floating-point representation uses a method of data compression (essentially the exponent value) to tell you how many leading zeros there are and not waste precious bits in the mantissa on all those leading zeros. but they don't do that for leading 1s. so, for small theta, bits fall off the edge for cos(theta) that do not for sin(theta).> FACT: The ONLY purpose the "hidden bit" serves is to allow one more > significant digit than you otherwise would have with a direct > representation. > > FACT: The number of significant digits is directly related to the > "relative error" (see section 1.2 in [higham]). > > So.., while your statement above is true, I would say "So what?"!so what is epsilon and delta for 0 < theta << 1 ? float epsilon, delta, theta = 1.0e-12; epsilon = (float)acos( (double)((float)(cos((double)theta))) ) - theta; delta = (float)asin( (double)((float)(sin((double)theta))) ) - theta;> The > number of significant digits in the second value (53) is greater than > the number in the first value (39). So if you're using a representation > that has less than 53 bits but more than 38 bits, you'll preserve the > first but not the second. Of course. > > That sort of analysis doesn't prove that numbers near one are less > accurate than those near zero. In fact the IEEE 754 format ensures both > are represented with the same accuracy (or equivalently, the same > relative error). > > As Higham shows so well in section 1.7, the problem is that SUBTRACTIVE > CANCELLATION MAGNIFIES RELATIVE ERRORS.sure. but it's in the *difference* that cos(theta) is from 1 that tells you anything about what theta is. that's why there is a cosine problem and not a corresponding sine problem.> In fact the same magnification could happen with numbers near zero.arcsin( sin(tiny) ) does better than arccos( cos(tiny) ) so i don't see your point.> Thus when you attribute the hidden bit as the cause of these sorts of > problems, and implying that this only happens when subtracting numbers > near 1, I think you're misidentifying the issue.i'm not misidentifying the cosine problem. as to the OP, with single precision you won't know the difference between cos(2*pi*2^(-13)) and cos(2*pi*2(-14)). i'm not saying that the hidden 1 bit causes problems. i was saying that floating-point format (with or without a hidden 1 bit) doesn't solve the precision problem with cos(theta) with very small theta. On 4/18/14 10:23 AM, Randy Yates wrote:> PS: > > 1. Single precision would not preserve all the bits of either of the > values you presented since it only has 24 bits of precision.but it does better with sin() than it does with cos().> > 2. The "represents all number with the same relative error" statement > only works with numbers within the "dynamic" range of the > representation. For example, if the numbers get too small, you > "underflow" the exponent and can no longer normalizebut, with cos(theta), it's not in the exponent that there is any problem. in fixed-point, you will get into equal trouble as bits fall off to the right, whether your number is approaching 0 or approaching 1. floating-point solves the problem for approaching 0, but does not for approaching 1. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●April 18, 20142014-04-18
Randy Yates <yates@digitalsignallabs.com> wrote: (snip)> I'm not sure we agree on the real issue here. Perhaps that's > the reason for some of the confusion.> FACT: Both large numbers (e.g., those near 1) and small numbers (those > near 0) have the same number of significant digits in IEEE-754 floating > point format. > > FACT: The ONLY purpose the "hidden bit" serves is to allow one more > significant digit than you otherwise would have with a direct > representation.> FACT: The number of significant digits is directly related to the > "relative error" (see section 1.2 in [higham]).> So.., while your statement above is true, I would say "So what?"! > The number of significant digits in the second value (53) is > greater than the number in the first value (39). So if you're > using a representation that has less than 53 bits but more than > 38 bits, you'll preserve the first but not the second. Of course.> That sort of analysis doesn't prove that numbers near one are less > accurate than those near zero. In fact the IEEE 754 format ensures both > are represented with the same accuracy (or equivalently, the same > relative error).> As Higham shows so well in section 1.7, the problem is that SUBTRACTIVE > CANCELLATION MAGNIFIES RELATIVE ERRORS.> In fact the same magnification could happen with numbers near zero.I believe all those are true, but the original question was for the FFT, and I don't believe that is enough to explain the error model for the FFT. I heard from a reliable source (but I forget which one) that the FFT rounding is much better than the DFT rounding. All the intermediate roundings tend to average out in a way that one big rounding doesn't. As far as I know, though, it isn't easy to show in general how close one comes to the right answer. -- glen
Reply by ●April 18, 20142014-04-18
robert bristow-johnson <rbj@audioimagination.com> wrote: (snip, someone wrote)>> FACT: Both large numbers (e.g., those near 1) and small numbers (those >> near 0) have the same number of significant digits in IEEE-754 floating >> point format.> depends on what you mean by "significant". floating-point > representation uses a method of data compression (essentially the > exponent value) to tell you how many leading zeros there are and not > waste precious bits in the mantissa on all those leading zeros.Well, floating point helps in the case of relative error. If you don't know very well the size of the values, but need to preserve some amount of relative error, it is easy with floating point.> but they don't do that for leading 1s. so, for small theta, > bits fall off the edge for cos(theta) that do not for sin(theta).Yes, but those bits weren't going to do much, anyway. If you have a tiny rounding error, and then add another value with a much larger rounding error, you can pretty much forget about the tiny one. FFT does a lot of add and subtract with those values.>> FACT: The ONLY purpose the "hidden bit" serves is to allow one more >> significant digit than you otherwise would have with a direct >> representation.>> FACT: The number of significant digits is directly related to the >> "relative error" (see section 1.2 in [higham]).>> So.., while your statement above is true, I would say "So what?"!> so what is epsilon and delta for 0 < theta << 1 ?> float epsilon, delta, theta = 1.0e-12;> epsilon = (float)acos( (double)((float)(cos((double)theta))) ) - theta; > delta = (float)asin( (double)((float)(sin((double)theta))) ) - theta;Well, except that what you really want is X*cos(theta) +/- Y*sin(theta) where X and Y are about the same size. -- glen
