I have a DAC which is assumed to be nonlinear, such that it produces unwanted harmonic distortions at integer multiples of the input frequencies. Any other nonlinear distortions (such as intermodulation products) are assumed to be negligible

If the harmonics fall outside the bandwidth of the analog bandpass filter, then they are assumed to be eliminated:

However, if a harmonic falls within the bandpass filter’s passband, then the unwanted distortion remains. In an attempt to avoid this from happening, I can adjust the sample rate of the DAC. (This changes the Nyquist frequency, which changes the frequency of the harmonic after it is aliased into the Nyquist zone of interest).

Example: Assume the following:

• Baseband signal bandwidth = 1 GHz.
• Center frequency = 5.5 GHz.
• Bandpass filter passband is 5 GHz - 6 GHz (exactly covering the signal).
• 2^nd and 3^rd order harmonics are nonzero, but all higher order harmonics are negligible.
• The DAC supports up to 10 GS/s.

If we try sampling at 8 GS/s, then the 2^nd order harmonics (HD2) wrap into the signal band:

If we try sampling at 10 GS/s, then the 3^rd order harmonics (HD3) wrap into the signal band:

However, 9 GS/s appears to be a perfect choice in this case. Neither HD2 nor HD3 are wrapped into the signal band:

My problem is that I found this result by brute force. It's not obvious to me how to analyze these harmonics more generally because I don't know how to characterize the aliasing in a convenient way. Could anyone offer any suggestions?

My goal would be to answer more general questions such as (for some maximum sampling rate):

1. If the signal band is from A MHz to B MHz and the filter passband is from X MHz to Y MHz, then how many harmonics (HD2, then HD3, then HD4, etc) can be avoided (and how)?
2. If I want to avoid the first N harmonics (given a signal band from A MHz to B MHz), then what is the widest possible filter passband?

This may not directly answer your questions, but Analog devices has an online tool for finding aliases of a signal and its harmonics:

https://www.analog.com/en/design-center/interactiv...

Also, you can take a look at a post I wrote: "Find aliased ADC or DAC harmonics"

https://www.dsprelated.com/showarticle/1380.php

-- Neil

@neilrober Thank you for your comments and links. Your article is very clear and gives me confidence that I'm on the right track.

The Analog Devices tool you linked is more user-friendly than the equivalent tool I found from Texas Instruments (via the "Reference" section of of this EE Times article: link).

Both tools look equivalent to what I hand-coded in MATLAB, so it seems that the expectation from the chip manufacturers is that frequency planning should be done by manual brute force.

At least with my own scripts, I can upgrade that to "automated brute force", even if I can't find an elegant/analytical (non-brute-force) solution.

I would do exactly what you did. But if you do try any derivation analytically, you should use modular arithmetic algebra.

The assumption about the Nyquist harmonics and sample frequency isn't quite correct, as it (harmonics etc) also depends on where the non-linearity is within the DACs range, and which particular data value sequences are generated by the change in data rate. 

If there is a step discontinuity at, say, +V/2, in a range of -V..+V, then the distortion only happens as you pass that signal level. So if you have two sine waves beating together (peaks and troughs, some not reaching +V/2) then you get a mixture of transients. Reality is never easy..