Your normalization is not clear.
When you are shifted by an integer number of chips, then the autocorrelation would be zero if the PRBS had precisely the same number of ones as zeros (since half of the bits would give the same sine phases and half antiphase). If you have any imbalance, which there must be since there are an odd number of bits, there is residual autocorrelation in the FSK as well.
Regarding normalisation, the integrate and dump block is integrating for k samples where k is the number of samples in the length of the prbs. I then multiply the end result by 1/k to scale the result to unity.
I don't think the residual autocorrelation should be nowhere near as much as the 0.5 I am seeing. It should be approx 1/n where n is number of chips in my prbs.
I agree. If you are normalizing to 1 when the lag is zero, then the side lobes should not be 1/2. Now that I read your question more carefully, I understand that your modulator is mixed down to center frequency zero (so that your frequencies are -w and w). These two frequencies are not orthogonal! The idea of FSK is that when you have two different frequencies the integral of the product should be zero (not minus 1) at least if the bit is long enough. So mix up to some frequency and make the signals real.
Yes, I've used the standard BASEBAND m-fsk block from the SIMULINK library because simulation at baseband is encouraged in the SIMULINK literature (to reduce required sample rates). I thought there might be a way to do the correlation with the baseband signals but perhaps not.
I will try up converting the baseband signal and then correlating. I could also just use the passband m-fsk block to get the same passband signal. I'll try this out tomorrow and let you know how I get on.
Not sure what you mean by the last part of your reply, "make the signals real"?
Up conversion didn't help (got the same results).
However, after a bit more pondering I believe I have discovered the problem. The orthogonality (or the lack of it) depends on the frequencies chosen for binary 1 and binary 0, plus the symbol rate. My original choices were +200k and -200k for 1 and zero with a 200kchip rate. The up converted signal's '1' and '0' symbols are still nowhere near orthogonal.
I changed the frequencies so they're not integer related to the symbol rate and the correlation value at integer chip delays is now more like the value I would expect.