There are two code snippets associated with this blog post:

Flat-Top Windowing Function for the Accurate Measurement of a Sinusoid's Peak Amplitude Based on FFT Data

and

Testing the Flat-Top Windowing Function

This blog discusses an accurate method of estimating time-domain sinewave peak amplitudes based on fast Fourier transform (FFT) data. Such an operation sounds simple, but the scalloping loss characteristic of FFTs complicates the process. We eliminate that complication by implementing 'flat-top' windowing by way of frequency-domain convolution in a way that greatly reduces scalloping-loss amplitude-estimation problems.

FFT SCALLOPING LOSS REVISITED
There are many applications that require the estimation of a time-domain sinewave's peak amplitude based on FFT data. Such applications include oscillator and analog-to-digital converter performance measurements, as well as standard total harmonic distortion (THD) testing. However, the scalloping loss inherent in FFTs creates an uncertainty in such time-domain peak amplitude estimations. This section provides a brief review of FFT scalloping loss.

As you know, if we perform an N-point FFT on N real-valued time-domain samples of a discrete sinewave, whose frequency is an integer multiple of f_s/N (f_s is the sample rate in Hz), the peak magnitude of the sinewave's positive-frequency spectral component will be

where A is the peak amplitude of the time-domain sinewave. That phrase "whose frequency is an integer multiple of f_s/N" means that the sinewave's frequency is located exactly at one of the FFT's bin centers.

Now, if an FFT's input sinewave's frequency is between two FFT bin centers (equal to a non-integer multiple of f_s/N) the FFT magnitude of that spectral component will be less that the value of M in Eq. (1). Figure 1 illustrates this behavior. Figure 1(a) shows the frequency responses of individual FFT bins where, for simplicity, we show only the mainlobes (no sidelobes) of the FFT bins' responses. What this means is that if we were to apply a sinewave to an FFT, and scan the frequency of that sinewave over multiple bins, the magnitude of the FFT's largest normalized magnitude sample value will follow the curve in Figure 1(b). That curve describes what is called the "scalloping loss" of an FFT [1].
(As an aside, the word scallop is not related to my favorite shellfish. As it turns out, some window drapery, and table cloths, do not have linear borders. Rather they have a series of circular segments, or loops, of fabric defining their decorative borders. Those loops of fabric are called scallops.)

What Figure 1(b) tells us is that if we examine the N-point FFT magnitude sample of an arbitrary-frequency, peak amplitude = A sinewave, that spectral component's measured peak magnitude M_peak can be in anywhere in the range of:

depending on the frequency of that sinewave. This is shown as the rectangular window curve in Figure 2, where the maximum scalloping error occurs at a frequency midpoint between two FFT bins.

The variable M in Figure (2) is the M from (1). So if we want to estimate a sinewave's time-domain peak amplitude A, by measuring its maximum FFT spectral peak magnitude M_peak, our estimated value of A, from Eq. (1), using

can have an error as great as 36.3%. In many spectrum analysis applications such a large potential error, equivalent to 3.9 dB, is unacceptable. Hanning and Hamming windowing of the FFT input data reduce the unpleasant frequency-dependent fluctuations in a measured spectral M_peak value, as shown in Figure 2, but not nearly enough to satisfy many applications.

One solution to this frequency-dependent, FFT-based, measured amplitude uncertainty is to multiply the original N time-domain samples by an N-sample flat-top window function and then perform a new FFT on the windowed data. Flat-top window functions are designed to overcome the scallop loss inherent in rectangular-windowed FFTs. While such a flat-top-windowed FFT technique will work, there are more computationally-efficient methods to solve our signal peak amplitude estimation uncertainty problem.

An Accurate Measurement Process
We will solve our sinusoidal-peak measurement problem by performing convolution in the frequency domain as opposed to window-function multiplication in the time domain [2,3]. Consider the FFT magnitude samples shown in Figure 3, and let's assume we want to estimate the peak amplitude of the time-domain sinusoid corresponding to the large |X(k)| FFT sample.

We can compute the complex Xft(k) sample using

Once we have the value of the complex X_ft(k) sample from Eq. (4), we use

in Eq. (3) to yield an accurate estimate of the sinewave's time-domain peak amplitude A. Our maximum error in an estimated value of peak amplitude A, using the |X_ft(k)| from Eq. (4), is roughly 0.02 dB, and is depicted as the essentially-horizontal Flat-top curve in Figure 2. That's significantly smaller than the rectangular-windowed maximum error of 3.9 dB.

Three Important Issues
First, the flat-top window frequency-domain convolutions are most useful in accurately measuring the time-domain amplitude of a sinusoidal signal when that signal's spectral component is not contaminated by sidelobe leakage from a nearby spectral component. For example, if a positive-frequency spectral component is low in frequency, i.e., located in the first few FFT bins, leakage from a spectral component's corresponding negative-frequency spectral component will contaminate that positive-frequency spectral component. As such, this code should not be used for frequencies below the sixth FFT bin or above the (N/2–5)th FFT bin.

Second, the flat-top window frequency-domain convolutions are most useful when the FFT spectral component being measured is well above the background spectral noise floor.

Third, reference [3] gives more details regarding this flat-top windowing process plus some clever methods of efficiently implementing the process in programmable hardware.

In the Matlab code associated with this FFT-based sinewave peak amplitude estimation method, we perform time-domain flat-top windowing of FFT samples by way of frequency-domain convolution. The input to the code is a sequence of complex-valued FFT samples, and the output of the code is a sequence of complex-valued flat-top-windowed FFT samples. If the FFT's input was a sinusoidal signal, the amplitude of that sinusoid can be accurately estimated by using Eq. (3) where M_peak is the maximum magnitude sample of the code's output sequence.

References
[1] F. Harris, "On the use of windows for harmonic analysis with the discrete Fourier transform," Proceedings of the IEEE, Vol. 66, No. 1, pp. 51-83, January 1978.

[2] E. Jacobsen, and R. Lyons, "The Sliding DFT", IEEE Signal Processing Magazine, DSP Tips & Tricks column, March, 2003.

[3] R. Lyons, "Reducing FFT Scalloping Loss Errors Without Multiplication", IEEE Signal Processing Magazine, DSP Tips & Tricks column, March, 2011, pp. 112-116.