Design Square-Root Nyquist Filters

Neil RobertsonJuly 13, 2020

In his book on multirate signal processing, harris presents a nifty technique for designing square-root Nyquist FIR filters with good stopband attenuation [1].  In this post, I describe the method and provide a Matlab function for designing the filters.  You can find a Matlab function by harris for designing the filters at [2].


Single-carrier modulation, such as QAM, uses filters to limit the bandwidth of the signal.  Figure 1 shows a simplified QAM system block diagram with I and Q filters in the transmitter and receiver.  The impulse response of the filters generally has ringing that causes inter-symbol interference, or ISI, between the transmitted symbols.  We can limit the ISI by judicious filter design.  It turns out that a filter with a frequency response as shown in Figure 2a has theoretically zero ISI.  This so-called Nyquist filter [3] has a magnitude response of 0.5 at fsymbol/2, and the response is odd-symmetric about fsymbol/2.  It also must have linear phase.  The stopband edge is at:

$$f= (1+\alpha)\frac{f_{symbol}}{2},\qquad 0<\alpha<1 \qquad (1) $$

This article is available in PDF format for easy printing

Given that the minimum bandwidth to transmit a symbol sequence is fsymbol/2, the parameter α is called the excess bandwidth.

Figure 2b plots the impulse response of a Nyquist filter that has four samples per transmitted symbol.  The samples that occur at symbol intervals are shown in red.  As desired, all of these samples other than the center one are near zero (near zero ISI).

We could place I and Q Nyquist filters in the transmitter, and everything would work fine … except for noise.  For optimal performance with noise, the receive filter must have a response that is a time-reversed version of the transmit filter [4,5].  For symmetrical filters, this means the transmit and receive filters must have identical impulse responses, i.e., they must be matched filters.  How can we achieve zero ISI with matched filters?  In the frequency domain, the matched filter response H(f) must satisfy:

$$ |H(f)|\cdot|H(f)|= |H_{nyquist}(f)| $$

where Hnyquist(f) is the response of a Nyquist filter.  We then have:

$$|H(f)|= |H_{nyquist}(f)|^{1/2} \qquad (2) $$

H(f) is called a square-root Nyquist response.  Figure 3 plots a square-root Nyquist response and a “full” Nyquist response (dashed).  As shown, the response of the square-root Nyquist filter at fsymbol/2 is the square-root of 0.5, or -3.01 dB.  Besides providing optimal noise performance, the square-root Nyquist response in the receive filter allows for good adjacent-channel rejection.

Figure 1.  QAM Transmitter and Receiver Simplified Block Diagram.  Wide lines = complex (I/Q).

Figure 2.  a) Nyquist filter magnitude response (linear amplitude scale).

                b) Impulse response of a Nyquist filter with 4 samples per symbol.

Figure 3.  Square-root Nyquist magnitude response (in red), linear amplitude scale.

A Matlab function to design square-root Nyquist FIR filters

Appendix A in the PDF version lists the Matlab function root_nyq.m for designing square-root Nyquist FIR filters.  It is based on the technique presented in reference [1].  The function uses the Matlab Parks-McClellan filter design function firpm, with stopband edge of (1 + α)*fsymbol/2.  The strategy is to find a passband edge frequency fpass that results in:

$$|H(f_{symbol}/2)|=\sqrt{1/2}\qquad(3) $$

as shown in Figure 3.  A gradient search is performed over fpass, and a new filter is synthesized for each fpass value, until the goal response is met with acceptable error.

Note that using fpass= (1 - α)*fsymbol/2 results in a response of 0.5 at fsymbol/2, which exceeds the attenuation goal.  Thus, this value of fpass is always below the desired value.  It is convenient to start the search at this frequency, with each step adjusting the value of fpass until the error in the response is minimized.  

Even though the function optimizes the transition band response at just one frequency, ISI is quite good.  And, for a given number of taps, filters designed using root_nyq.m have much higher stopband attenuation than square-root raised cosine (SRRC) filters [1].

Example 1

In root_nyq.m, symbol time Tsymbol has units of samples.  For this example, let Tsymbol = 4 samples and excess bandwidth alpha = 0.25.  Then, to design a 65-tap filter, we use the following Matlab code, which also computes the filter’s magnitude response, assuming sample rate of 200Hz:

    ntaps= 65;
    Tsymbol= 4;                % samples  symbol time
    alpha= .25;                % excess bandwidth
    b= root_nyq(ntaps,Tsymbol,alpha);
    fs= 200;
    [h,f]= freqz(b,1,512,fs);
    H= 20*log10(abs(h));

The magnitude response is plotted in Figure 4.  Given 4 samples per symbol, fsymbol = fs/4 = 50 Hz, and fsymbol/2 = fs/8 = 25 Hz.  As desired, the response is -3 dB at 25 Hz.  The stopband edge is at (1 + alpha)*fsymbol/2 = 1.25*25 = 31.25 Hz.

Next, we’ll display an eyeplot of random binary symbols for the cascade of two identical square-root Nyquist filters.  The following code generates random binary data, upsamples by four, and applies the upsampled signal to the cascade of two filters.

    b_full= conv(b,b);           % cascade two root-nyquist filters
    N= 1024;                     % number of symbols
    u= round(rand(1,N));
    x= u -1/2;                   % binary symbols at 1 sample/symbol
    x_up= zeros(1,4*N);
    x_up(1:4:4*N)= 4*x;          % upsample to 4 samples/symbol
    y= filter(b_full,1,x_up);    % apply x_up to full-Nyquist filter

The eyeplot is shown in figure 5 (The function eyeplot.m is listed in Appendix B of the PDF version).  Note that it is allowable to have transmit and receive square-root Nyquist filters with different numbers of taps.  However, in simulations, it is then sometimes necessary to resample the receive filter output to get optimal ISI.

Figure 4.  Square-root Nyquist filter magnitude response, ntaps= 65, fs= 200 Hz, fsymbol/2 = 25 Hz, and alpha= 0.25.

Figure 5.  Eyeplot at output of cascaded square-root Nyquist filters (Tsymbol = 4 samples).

Example 2

Let Tsymbol = 2 samples, ntaps = 42 and excess bandwidth alpha = 0.2.  The Matlab code to design the square-root Nyquist filter is as follows:

    ntaps= 42;
    Tsymbol= 2;
    alpha= .2;
    b= root_nyq(ntaps,Tsymbol,alpha);

The magnitude response is plotted in Figure 6, where we have again assumed sample frequency of 200 Hz.  fsymbol/2 = 50 Hz and stopband edge is 1.2*50 = 60 Hz.

As in example 1, we can find the eyeplot for the cascade of two identical square-root Nyquist filters using random binary symbols.  As a way to quantify the ISI, we will also compute the Modulation Error Ratio (MER) of the cascaded filters, assuming BPSK modulation.  I described MER calculation in an earlier post [6].  The Matlab code is as follows:

    b_full= conv(b,b);             % cascade two root-nyquist filters
    % generate eye plot
    N= 1024;                       % number of symbols
    u= round(rand(1,N));
    x= u -1/2;                     % random symbols at 1 sample/symbol
    x_up= zeros(1,2*N);
    x_up(1:2:2*N)= 2*x;            % upsample to 2 samples/symbol
    y= filter(b_full,1,x_up);      % apply x_up to full-Nyquist filter
    % compute MER of BPSK due to root-Nyquist ISI
    Pavg= -6.02;                   % dB avg power of constellation [-.5 .5]
    I= y(62:2:end);                % decimate to one sample/symbol
    dI= abs(I) - .5;               % error
    e_sq= dI.^2;                   % error squared
    e_sq_avg= sum(e_sq)/length(I);
    mer_dB= Pavg - 10*log10(e_sq_avg)

The eyeplot is shown in Figure 7.  The MER varies slightly depending on the random symbol sequence, and is approximately 51 dB.  We can compute MER for several values of ntaps – the result is plotted, along with stopband attenuation, in Figure 8.  ntaps should be chosen to achieve MER that puts minimal stress on the forward error correction algorithm.  If an adaptive equalizer is used, it may be able to adapt to achieve MER greater than that of the cascaded matched filters.

Figure 6.  Square-root Nyquist filter magnitude response, ntaps= 42, fs= 200 Hz, fsymbol/2 = 50 Hz, and alpha= 0.2.

Figure 7.  Eyeplot at output of cascaded square-root Nyquist filters (Tsymbol = 2 samples).

Figure 8.  MER and Stopband Attenuation vs. ntaps for alpha= 0.2 and Tsymbol = 2 samples.

See PDF version for Appendix A and B.


1.  harris, fredric j., Multirate Signal Processing for Communications Systems, Prentice Hall, 2004, section 4.3.2

2.  Website for reference 1, https://www.informit.com/store/multirate-signal-processing-for-communication-systems-9780131465114

3.  Rice, Michael, Digital Communications, A Discrete-Time Approach, Pearson, 2009, section A.2

4.  harris, p 86-88.

5.  Sklar, Bernard, Digital Communications:  Fundamentals and Applications, Prentice Hall, 1988, section 2.9.2.

6.  Robertson, Neil, “Compute Modulation Error Ratio (MER) for QAM", dspRelated.com, Nov., 2019, https://www.dsprelated.com/showarticle/1305.php

7.  Rice, section A.2.3.

Neil Robertson     July, 2020                       

To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.

Please login (on the right) if you already have an account on this platform.

Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers: