About PESQ scores(audio quality evaluation)

Started by nang...@gmail.com January 25, 2011
I have a question on PESQ

If I want to verify the PESQ score of G711 and G729 codec, what procedures should I take?

my idea is:

1) prepare a audio .wav file, whose format is PCM 8000Hz, signed 16 bit, let me name it ori.wav

2) encode it into G711 or G729 format and get enc.wav

3) convert enc.wav into PCM 8000Hz, signed 16 bit format, and I get deg.wav

4) run PESQ algorithm over ori.wav and deg.wav, ./pesq +8000 ori.wav enc.wav

my question is, are my procedures right or wrong? or should I run PESQ algorithm over ori.wav and enc.wav? I found that if I run PESQ algorithm over enc.wav directly, the PESQ score is very low.

I have two pieces of software to for converting ori.wav into PCM g711 u-law, Goldwave and TotalRecorder.
If I follow 1) 2) 3) using Goldwave, the PESQ score between ori.wav and deg.wav is 4.5,
while if I use TotalRecorder, the PESQ score is 4.2, it is very strange

So, can you tell me what procedures should I follow? and what ori.wav should I use(where can I get)?

Thank you so much!
Nangergong-

> I have a question on PESQ
>
> If I want to verify the PESQ score of G711 and G729 codec,
> what procedures should I take?
>
> my idea is:
>
> 1) prepare a audio .wav file, whose format is PCM 8000Hz,
> signed 16 bit, let me name it ori.wav
>
> 2) encode it into G711 or G729 format and get enc.wav

I assume when you're using ITU reference software to do the encode?
>
> 3) convert enc.wav into PCM 8000Hz, signed 16 bit format,
> and I get deg.wav
>
> 4) run PESQ algorithm over ori.wav and deg.wav,
> ./pesq +8000 ori.wav enc.wav
>
> my question is, are my procedures right or wrong?

For G729, you have to run enc.wav through decode. Using your naming convention, you
would get dec.wav, and then compare that with ori.wav (PESQ). The output of G729 is
a series of packets with bit fields -- completely non-linear, nothing that should be
treated as wav data (unless you're trying to decipher the bit fields).

> or should I run PESQ algorithm over ori.wav and enc.wav? I found
> that if I run PESQ algorithm over enc.wav directly, the PESQ
> score is very low.
>
> I have two pieces of software to for converting ori.wav into
> PCM g711 u-law, Goldwave and TotalRecorder.
> If I follow 1) 2) 3) using Goldwave, the PESQ score between
> ori.wav and deg.wav is 4.5,

I assume you're using Goldwave for step 3). If so then I think the reason this works
for G711 is that Goldwave recognizes encoded output of G711 as uLaw and stores in
.wav file format. That would not happen for G729 unless you have a G729 license from
the software vendor (G729 is still under patent protection).

> while if I use TotalRecorder, the PESQ score is 4.2, it
> is very strange

What did you use in step 2)? Maybe it generated G711 format .wav file not compatible
with TotalRecorder.

-Jeff
Nangergong-
> Thank you so much. I think I'm more clearer than before!
>
> I got to know from some paper that the PESQ performance is 4.1 for G711 while
> 3.6 for G729.
>
> I have a G711-encoding .raw file directly from VOIP program, and I used Goldwave
> to open it and convert it into PCM signed 16 bit 8000Hz .wav file.

Sounds like you're capturing RTP streams using Wireshark? What data format is in the
.raw file? What are the max/min values? You should verify the .raw file contains
u-Law data -- there should be no "fuzziness" in your procedure. Each step should be
well-defined and well-understood.
> If I compare this .wav file with the original using PESQ tool, the PESQ score is
> about 4.1-4.2, so I think this procedure works fine.
> And I think Goldwave can do the decoding automatically, do you think so.

If you want to be sure then listen to the .wav file... if it sounds as expected, and
you verified .raw data as noted above, then yes Goldwave has done the conversion.
> While for G729, I found a G729 codec on windows platform. Do you know where can I
> download G729 codec on linux platform? I searched a lot but failed to find it.

Locate G729 source on the ITU website and build under Gcc.

-Jeff

PS. Post to the group, not to me.
> On Sun, Jan 30, 2011 at 6:17 AM, Jeff Brower wrote:
>
> Nangergong-
>
> > I have a question on PESQ
> >
> > If I want to verify the PESQ score of G711 and G729 codec,
> > what procedures should I take?
> >
> > my idea is:
> >
> > 1) prepare a audio .wav file, whose format is PCM 8000Hz,
> > signed 16 bit, let me name it ori.wav
> >
> > 2) encode it into G711 or G729 format and get enc.wav
> I assume when you're using ITU reference software to do the encode?
> >
> > 3) convert enc.wav into PCM 8000Hz, signed 16 bit format,
> > and I get deg.wav
> >
> > 4) run PESQ algorithm over ori.wav and deg.wav,
> > ./pesq +8000 ori.wav enc.wav
> >
> > my question is, are my procedures right or wrong?
> For G729, you have to run enc.wav through decode. Using your naming
> convention, you
> would get dec.wav, and then compare that with ori.wav (PESQ). The output
> of G729 is
> a series of packets with bit fields -- completely non-linear, nothing
> that should be
> treated as wav data (unless you're trying to decipher the bit fields).
>
> > or should I run PESQ algorithm over ori.wav and enc.wav? I found
> > that if I run PESQ algorithm over enc.wav directly, the PESQ
> > score is very low.
> >
> > I have two pieces of software to for converting ori.wav into
> > PCM g711 u-law, Goldwave and TotalRecorder.
> > If I follow 1) 2) 3) using Goldwave, the PESQ score between
> > ori.wav and deg.wav is 4.5,
> I assume you're using Goldwave for step 3). If so then I think the
> reason this works
> for G711 is that Goldwave recognizes encoded output of G711 as uLaw and
> stores in
> .wav file format. That would not happen for G729 unless you have a G729
> license from
> the software vendor (G729 is still under patent protection).
>
> > while if I use TotalRecorder, the PESQ score is 4.2, it
> > is very strange
> What did you use in step 2)? Maybe it generated G711 format .wav file
> not compatible
> with TotalRecorder.
>
> -Jeff
>