Looks like quite an interesting project!

As for how to test it, you'll need a good model 1st.  Absolutely you'll have to test your model and get that working and then implement and test the RTL.  Matlab is good.

I like Simulink but that does require an accurate implementation for every block which you don't get right out of the box.  Not in my experience.  There are toolboxes to generate FPGA code from Simulink designs but I've yet to use them.  I can say for sure that my hand coded RTL is very efficient in term of fabric utilization.

My method is to implement blocks in C MEX files on anything that has to be bit accurate.  Then build up functionality from there.  Hammer test the model.  Then write the RTL block by block to bit accurately match the model.  Doesn't say that the RTL is correct but it does match the model.

I'm kind of married to Simulink at this point but I don't see why the modeling couldn't be done with Python as well.

FWIW,

Mark Napier

I always recommend writing a bit-accurate software model (typically in Python or MATLAB) to use as the basis for verification. You can then generate any variety of input signal in a convenient software environment, and drive those same inputs through your software model and through your HDL implementation (in simulation). The HDL simulation then just checks that the output of the RTL is numerically identical to the software model. Depending on your specific needs, a few good test cases may be sufficient, but once that workflow is set up, you can easily test many thousands of configurations if you need to.

Assuming your RTL implementation is fixed-point, you'll benefit greatly from a decent fixed-point math library. There are many bad ones and not many good ones. MATLAB's fixed-point toolbox is relatively good (but it's not free).

In the case of RFSoC, I usually test my receiver system in hardware by developing a transmitter system also on RFSoC (verified the same way as the receiver). That may be overkill in your case, but it gives you full control over what you transmit, so it can be a powerful tool.

I can't see your design because the picture is small, but 256 MHz is a slow clock for an RFSoC. If you're experienced with pipelined design, then you will have no trouble at 512 MHz. Of course, it depends on the speed grade of your device and other factors, but these devices are fast everywhere except the block memories (particularly the URAMs, which are limited to ~600 MHz). If you're new to HDL implementation, then stick with 256 MHz for now. RFSoCs are monstrously power hungry, so a slower clock will also save you from melting anything, or hitting the 50 A current limit of the main power pin.

To me it sounds like this is an academic project, and in addition to tests during development (which other answers discussed), you're also asking how you can test that it's functioning for a demonstration.

If you can automate your test equipment, I recommend inserting sine waves at various input frequencies to then measure the power output from each channel.  By choosing a large number of closely-spaced input frequencies, you can trace out the channel amplitude response of every output channel.  You can do this first on the RTL, using an RTL testbench.  (Matlab is overkill; generating sine waves in RTL testbenches is trivial.)  Then move the test to real-world signal generators and power meters.  It's easiest to just measure the power output, as a simple test.  More advanced tests might make coherent measurements at the outputs, but since the input and output frequencies are different these become increasingly challenging as they become more informative.  Offhand, I'm not even sure that S21 measurements (which you mention) can be applied when input and output frequencies are different.  (I'm not immensely familiar with them.)  However, there are other coherent or semi-coherent measurements that are applicable, and they're better than power measurements for determining things like noise and aliasing power levels.  Fully coherent tests are especially difficult, because they need to take into account the phase of the ADC and DAC clocks.  Semi-coherent tests only look that the output is at the correct frequency, and measure power at the correct frequency vs power at incorrect frequencies.  These ignore the output phase and are thus a lot easier.  However they require lots of automated measurements using a spectrum analyzer and additional analysis that simple power measurements don't require.

If it happens that this isn't just an academic demonstration project, and the point is to have such a design for some other purpose, I recommend that you don't do this at all.  If you're a low-budget academic, you should look into the CASPER project, which automates construction of such designs.  If this is for a company where your time is valuable, you can just purchase this (from me!), ready-to-go, complete with RTL tests.