Hi all,

I want to compute power spectral density of the noise data. The underwater noise is measured at 500K Sample/sec for few hours. The overall data file is very large (>10gb). I want to compute the PSD of this data. There are many methods given in MATLAB like Welch's power spectral density estimate, periodogram method, etc. Which method is best. Also, since data is so large, I cannot compute the PSD in one go because RAM will be insufficient. So, how can I set the frame length or window length to compute PSD in chunks of data and somehow apply averaging (something like moving average)?

Please suggest the efficient way to do this. 

Thanks in advance!

Hi -

"Best" is always a dangerous word when talking about estimators. "Best" depends on your application. When you say "best", what you are really asking is which is optimal, and even then you need to ask - optimal in what sense of measure (e.g. MMSE, MVU, ML, etc.). I suggest you look at the Estimation Theory book by Kay and also a book on Statisitical Signal Processing by Hayes.

Here is a decent "starter" intro -

http://www.laurent-duval.eu/Documents-Common/Schus...

But in short, the periodiogram, while simple to understand and apply, is a biased estimator whose variance does not diminish even with growing data record. Use either Barlett's or Welch's methods, as those not only reduce variance but require some overlap of the data records to do so - which you say you cannot use all your data at once anyway.

Thank you so much for your reply. 

I understood the Welch's method and it is giving good results. It is basically averaging to reduce the variance of the periodogram. Its implementation on MATLAB is also easy. Can you also explain me in brief about bartlett method? I could not find much documentation on bartlett method and its implementation. Which is more optimal (not best :P) for PSD estimation?

Also, is cpsd (cross power spectral denisty) is different? is it worth to try cpsd too?

I would definitely use Welch's method, which is based on the periodogram.  Since your only issue is doing it sequentially through a file instead of on one giant input array, just write your own variant in Matlab.  You're just averaging squared-magnitude FFTs:

https://ccrma.stanford.edu/~jos/sasp/Matlab_Welch_...