I am using Burg's method of linear prediction to replace invalid sections of audio data. The algorithm works 90% of the time with reasonable prediction accuracy but occasionally goes wrong with something that looks like numerical instability. At some point in the iterative process the reflection coefficient becomes > 1 and the filter coefficients at the end of the process are very large and sometimes overflow.(Prediction filter order is 256 but smaller orders have been tried). I do not think there is an error in the code as I have run someone else's code (courtesy of Cedric Colomb) in parallel with my own and it produces almost identical coefficient values. The code is Visual C++ using an old version of Visual Studio and double precision floating point is used throughout so rounding errors should not be a problem. I have tried different window functions (Hamming, Hanning, Rectangular) and the results do differ but usually with some 'instability' in there somewhere. When I pick a section where the algorithm is predicting well I find that comparisons between Frequency response of the Prediction Filter inverse and a FFT of the data block are a reasonable match. Does anyone have experience of this kind of problem?

The problem with the Burg method is that it is fairly easy to compute a pole that lies very nearly on the unit circle.  About the only solution is to reduce the order of the transfer response.  A reasonably safe solution is to set the number of coefficients to 40% of the length of data.  You can obtain very smooth predictions by dramatically reducing the number of coefficients.  The reason is clear - a low order transfer response simply cannot follow all the squiggles in the input data.

Thanks for that John. I will try it.

Do you know what is the normal procedure as regards use of Window functions?. With Levinson Durbin I got significantly better results by using Hanning or Hanning but this does not seem to be the case with Burg.

The Burg algorithm creates a poles-only transfer response that is optimally matched to the data presented.  In essence, windowing has "no" meaning.  The output is characterized by the length of the data in the "window" and the order of the transfer response polynomial.  As I said, you get a stable response when the order of the polynomial is less, sometimes much less, than the length of the input data. 

Having said that, I preprocess the data to select only the piece of the spectrum in which I am interested to be passed to the Burg algorithm.  I do this by serially filtering in a two pole lowpass filter at the high frequency end and a two pole highpass filter at the low frequency end of the spectrum.  In my case, I am seriously concerned with latency.  Therefore, my window length is quite small (which is why I use the Burg algorithm in the first place).