Let's say there is a voiceband communication channel suffering from noise, linear and nonlinear distortions, interference, fading, frequency shift, or any other severe analog or digital artifacts. The quality of the channel could be considered as marginally acceptable for conversation. Is there an objective method to measure the quality of such channel ? So one could compare the impact of the different impairments on the quality? Vladimir Vassilevsky DSP and Mixed Signal Design Consultant http://www.abvolt.com
An objective measure of intelligibility of speech ?
Started by ●May 25, 2011
Reply by ●May 25, 20112011-05-25
On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote:> > Let's say there is a voiceband communication channel suffering from > noise, linear and nonlinear distortions, interference, fading, frequency > shift, or any other severe analog or digital artifacts. The quality of > the channel could be considered as marginally acceptable for conversation. > > Is there an objective method to measure the quality of such channel ? > So one could compare the impact of the different impairments on the > quality?I don't know of any, but if you find one external to the group, could you please post the answer? -- Tim Wescott Wescott Design Services http://www.wescottdesign.com Do you need to implement control loops in software? "Applied Control Theory for Embedded Systems" was written for you. See details at http://www.wescottdesign.com/actfes/actfes.html
Reply by ●May 25, 20112011-05-25
Tim Wescott wrote:> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: >> >> Let's say there is a voiceband communication channel suffering from >> noise, linear and nonlinear distortions, interference, fading, frequency >> shift, or any other severe analog or digital artifacts. The quality of >> the channel could be considered as marginally acceptable for >> conversation. >> >> Is there an objective method to measure the quality of such channel ? >> So one could compare the impact of the different impairments on the >> quality? > > I don't know of any, but if you find one external to the group, could > you please post the answer? >Does Mean Opinion Score count? There are DSP implementations ( notably from Spirent ) that I've used on the past. http://en.wikipedia.org/wiki/Mean_opinion_score http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadband/PAB/Abacus/Abacus5000_IP_Telephony_Migration_Test_System_Datasheet.ashx Finally, http://www.bandcalc.com/ -- Les Cargill
Reply by ●May 26, 20112011-05-26
>Tim Wescott wrote: >> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: >>> >>> Let's say there is a voiceband communication channel suffering from >>> noise, linear and nonlinear distortions, interference, fading,frequency>>> shift, or any other severe analog or digital artifacts. The quality of >>> the channel could be considered as marginally acceptable for >>> conversation. >>> >>> Is there an objective method to measure the quality of such channel ? >>> So one could compare the impact of the different impairments on the >>> quality? >> >> I don't know of any, but if you find one external to the group, could >> you please post the answer? >> > > >Does Mean Opinion Score count? There are DSP implementations ( notably >from Spirent ) that I've used on the past. > >http://en.wikipedia.org/wiki/Mean_opinion_score > >http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadband/PAB/Abacus/Abacus5000_IP_Telephony_Migration_Test_System_Datasheet.ashx > >Finally, >http://www.bandcalc.com/ >There are no DSP implementations of MOS outside of marketing departments. Its a mean *opinion* score. DSP systems don't have opinions. That said, there are a number of algorithmic schemes for assessing the intelligibility of channels which get close enough to human opinions often enough to be useful - e.g. PESQ. Be careful to separate schemes for assessing interactive quality and one way quality. A good scheme for assessing interactive quality puts a fairly high weighting on low latency. For one way communication latency is largely unimportant. This might be relevant when comparing things like FEC schemes, which might work over very different periods. Steve
Reply by ●May 26, 20112011-05-26
On 05/25/2011 11:29 PM, Vladimir Vassilevsky wrote:> > Let's say there is a voiceband communication channel suffering from > noise, linear and nonlinear distortions, interference, fading, frequency > shift, or any other severe analog or digital artifacts. The quality of > the channel could be considered as marginally acceptable for conversation. > > Is there an objective method to measure the quality of such channel ? > So one could compare the impact of the different impairments on the > quality? > >There is the SII (Speech Intelligibility Index), standardized under ANSI S3.5-1997, which is exactly what you are looking for. You can find some information in http://www.sii.to/ and also sample implementations/programs. Regards, Laurent
Reply by ●May 26, 20112011-05-26
>Steve Underwood wrote: >>> Tim Wescott wrote: >>>> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: >>>>> >>>>> Let's say there is a voiceband communication channel suffering from >>>>> noise, linear and nonlinear distortions, interference, fading, >> frequency >>>>> shift, or any other severe analog or digital artifacts. The qualityof>>>>> the channel could be considered as marginally acceptable for >>>>> conversation. >>>>> >>>>> Is there an objective method to measure the quality of such channel?>>>>> So one could compare the impact of the different impairments on the >>>>> quality? >>>> >>>> I don't know of any, but if you find one external to the group, could >>>> you please post the answer? >>>> >>> >>> >>> Does Mean Opinion Score count? There are DSP implementations ( notably >>>from Spirent ) that I've used on the past. >>> >>> http://en.wikipedia.org/wiki/Mean_opinion_score >>> >>>http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadband/PAB/Abacus/Abacus5000_IP_Telephony_Migration_Test_System_Datasheet.ashx>>> >>> Finally, >>> http://www.bandcalc.com/ >>> >> >> There are no DSP implementations of MOS outside of marketingdepartments.>> Its a mean *opinion* score. DSP systems don't have opinions. >> > >... yet :) > >> That said, there are a number of algorithmic schemes for assessing the >> intelligibility of channels which get close enough to human opinionsoften>> enough to be useful - e.g. PESQ. > >Right. But... > >The ITU begs to differ. >http://www.itu.int/rec/T-REC-P.862.1/en >"Mapping function for transforming P.862 raw result scores to MOS-LQO"MOS-LQO is not real MOS. Its a recently made up term, which was pushed into an ITU spec to meet the needs of vendors peddling their wares. LQO == Listening only quality, objective. Even that name is designed to deceive. Objective sounds good, but it just means its an algorithmically cooked up value. The alternative MOS-LQS (listening only quality, subjective) measure sounds weaker. Subjective is bad, isn't it? Well, no. Its what real humans were able to make of the audio, and its the original MOS.> >> Be careful to separate schemes for >> assessing interactive quality and one way quality. A good scheme for >> assessing interactive quality puts a fairly high weighting on lowlatency.> >PESQ itself does not grade based on latency that I have been able to >tell.PESQ doesn't take latency into account. This is one of its serious drawbacks for assessing conversational quality. Some other assessment schemes do take latency into account. Some embedded assessments, such as in dynamic jitter buffers, take latency into account quite aggressively, as conversationally its better to have some packet loss and other major distortions than let the latency rise.>> For one way communication latency is largely unimportant. This might be >> relevant when comparing things like FEC schemes, which might work oververy>> different periods. >> >> Steve >> > >-- >Les Cargill >Steve
Reply by ●May 26, 20112011-05-26
Steve Underwood wrote:>> Tim Wescott wrote: >>> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: >>>> >>>> Let's say there is a voiceband communication channel suffering from >>>> noise, linear and nonlinear distortions, interference, fading, > frequency >>>> shift, or any other severe analog or digital artifacts. The quality of >>>> the channel could be considered as marginally acceptable for >>>> conversation. >>>> >>>> Is there an objective method to measure the quality of such channel ? >>>> So one could compare the impact of the different impairments on the >>>> quality? >>> >>> I don't know of any, but if you find one external to the group, could >>> you please post the answer? >>> >> >> >> Does Mean Opinion Score count? There are DSP implementations ( notably >>from Spirent ) that I've used on the past. >> >> http://en.wikipedia.org/wiki/Mean_opinion_score >> >> http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadband/PAB/Abacus/Abacus5000_IP_Telephony_Migration_Test_System_Datasheet.ashx >> >> Finally, >> http://www.bandcalc.com/ >> > > There are no DSP implementations of MOS outside of marketing departments. > Its a mean *opinion* score. DSP systems don't have opinions. >... yet :)> That said, there are a number of algorithmic schemes for assessing the > intelligibility of channels which get close enough to human opinions often > enough to be useful - e.g. PESQ.Right. But... The ITU begs to differ. http://www.itu.int/rec/T-REC-P.862.1/en "Mapping function for transforming P.862 raw result scores to MOS-LQO" >> Be careful to separate schemes for> assessing interactive quality and one way quality. A good scheme for > assessing interactive quality puts a fairly high weighting on low latency.PESQ itself does not grade based on latency that I have been able to tell.> For one way communication latency is largely unimportant. This might be > relevant when comparing things like FEC schemes, which might work over very > different periods. > > Steve >-- Les Cargill
Reply by ●May 26, 20112011-05-26
On May 25, 10:40�pm, "Steve Underwood" <coppice@n_o_s_p_a_m.coppice.org> wrote:> >Tim Wescott wrote: > >> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: > > >>> Let's say there is a voiceband communication channel suffering from > >>> noise, linear and nonlinear distortions, interference, fading, > frequency > >>> shift, or any other severe analog or digital artifacts. The quality of > >>> the channel could be considered as marginally acceptable for > >>> conversation. > > >>> Is there an objective method to measure the quality of such channel ? > >>> So one could compare the impact of the different impairments on the > >>> quality? > > >> I don't know of any, but if you find one external to the group, could > >> you please post the answer? > > >Does Mean Opinion Score count? There are DSP implementations ( notably > >from Spirent ) that I've used on the past. > > >http://en.wikipedia.org/wiki/Mean_opinion_score > > >http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadba... > > >Finally, > >http://www.bandcalc.com/ > > There are no DSP implementations of MOS outside of marketing departments. > Its a mean *opinion* score. DSP systems don't have opinions. > > That said, there are a number of algorithmic schemes for assessing the > intelligibility of channels which get close enough to human opinions often > enough to be useful - e.g. PESQ. Be careful to separate schemes for > assessing interactive quality and one way quality. A good scheme for > assessing interactive quality puts a fairly high weighting on low latency. > For one way communication latency is largely unimportant. This might be > relevant when comparing things like FEC schemes, which might work over very > different periods. > > Steve- Hide quoted text - > > - Show quoted text -Be careful with MOS and PESQ for accessing speech quality. PESQ was originally designed to evaluate coders. At one point I had an argument with the ITU working group responsible for PESQ because they wanted to try to use it for determining speech quality of speech enhancement algorithms (different animal than coders). Since I left the ITU, I have been out of touch with further developments on this issue. I know the working group I was with was looking into a quality measure for speech enhancement devices. Vlad, I speak with some of the group about once or twice a year, if interested I can ask what advancements they have made in this area. Maurice Givens
Reply by ●May 26, 20112011-05-26
Les, Steve and Maurice, Thank you for the very useful advice. That provides a lot of things to consider; so far it looks like PESQ is the preferred method.> Be careful with MOS and PESQ for accessing speech quality. PESQ was > originally designed to evaluate coders. At one point I had an argument > with the ITU working group responsible for PESQ because they wanted to > try to use it for determining speech quality of speech enhancement > algorithms (different animal than coders).That's what I would like to do: an objective comparison of the different distortions created by the different processing schemes. I am primarily interested in the operation at the marginally acceptable quality.> Since I left the ITU, I > have been out of touch with further developments on this issue. I know > the working group I was with was looking into a quality measure for > speech enhancement devices. > > Vlad, I speak with some of the group about once or twice a year, if > interested I can ask what advancements they have made in this area.That would be very interesting to know. With Best Regards, Vladimir Vassilevsky
Reply by ●May 26, 20112011-05-26
steveu wrote:>> Steve Underwood wrote: >>>> Tim Wescott wrote: >>>>> On 05/25/2011 02:29 PM, Vladimir Vassilevsky wrote: >>>>>> >>>>>> Let's say there is a voiceband communication channel suffering from >>>>>> noise, linear and nonlinear distortions, interference, fading, >>> frequency >>>>>> shift, or any other severe analog or digital artifacts. The quality > of >>>>>> the channel could be considered as marginally acceptable for >>>>>> conversation. >>>>>> >>>>>> Is there an objective method to measure the quality of such channel > ? >>>>>> So one could compare the impact of the different impairments on the >>>>>> quality? >>>>> >>>>> I don't know of any, but if you find one external to the group, could >>>>> you please post the answer? >>>>> >>>> >>>> >>>> Does Mean Opinion Score count? There are DSP implementations ( notably >>> >from Spirent ) that I've used on the past. >>>> >>>> http://en.wikipedia.org/wiki/Mean_opinion_score >>>> >>>> > http://www.spirent.com/Solutions-Directory/~/media/Datasheets/Broadband/PAB/Abacus/Abacus5000_IP_Telephony_Migration_Test_System_Datasheet.ashx >>>> >>>> Finally, >>>> http://www.bandcalc.com/ >>>> >>> >>> There are no DSP implementations of MOS outside of marketing > departments. >>> Its a mean *opinion* score. DSP systems don't have opinions. >>> >> >> ... yet :) >> >>> That said, there are a number of algorithmic schemes for assessing the >>> intelligibility of channels which get close enough to human opinions > often >>> enough to be useful - e.g. PESQ. >> >> Right. But... >> >> The ITU begs to differ. >> http://www.itu.int/rec/T-REC-P.862.1/en >> "Mapping function for transforming P.862 raw result scores to MOS-LQO" > > MOS-LQO is not real MOS.Right. It just scales a PESQ measurement on a MOS scale.> Its a recently made up term, which was pushed into > an ITU spec to meet the needs of vendors peddling their wares. LQO == > Listening only quality, objective. Even that name is designed to deceive. > Objective sounds good, but it just means its an algorithmically cooked up > value. The alternative MOS-LQS (listening only quality, subjective) measure > sounds weaker. Subjective is bad, isn't it? Well, no. Its what real humans > were able to make of the audio, and its the original MOS. >Agreed. But the test equipment approach provides some advantages, mainly of being able to do perturbation analysis. It's also considerably cheaper than a full on subjective test. Life is full of tradeoffs :) And these days, I've seen people just quote the upper limit for the CODEC as published on Wikipedia and go with that, rather than even use test equipment.>>>> Be careful to separate schemes for >>> assessing interactive quality and one way quality. A good scheme for >>> assessing interactive quality puts a fairly high weighting on low > latency. >> >> PESQ itself does not grade based on latency that I have been able to >> tell. > > PESQ doesn't take latency into account. This is one of its serious > drawbacks for assessing conversational quality. Some other assessment > schemes do take latency into account. Some embedded assessments, such as in > dynamic jitter buffers, take latency into account quite aggressively, as > conversationally its better to have some packet loss and other major > distortions than let the latency rise. >Depending on the codec, packet loss in dribs and drabs may not be that bad.>>> For one way communication latency is largely unimportant. This might be >>> relevant when comparing things like FEC schemes, which might work over > very >>> different periods. >>> >>> Steve >>> >> >> -- >> Les Cargill >> > > Steve-- Les Cargill
