Sorry, there was one typo (was k now R) in the code. Fixed now
- Bill

>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
len=100
s= (1.0*cos(2*pi*3.0*(1:len)/len))'

%correlation matrix
rho= [1 .8 ; .8 1]  %correlation between noise vectors is 0.8

%variance-covariance matrix
R=chol(rho)

%generate correlated noise variables n(:,1), n(:,2)
n=randn(len,2)
n= n * R

plot(n(:,1), n(:,2),'o')

r1=s+n(:,1)
r2=n(:,2)
r=[r1,r2]

% Rows give test statistic for r1 and r2
lambda= (r * inv(R))' * s

% cross-correlation is how to do it if white noise
cc = r' * s

% signal is sinewave or 0 -- make s and inv(R) conformable
s0 = [s zeros(len,1)]

%this should be mean and var of test statistic
lambda0 = (s0 * inv(R))' * s

I have a signal detection problem.
The observer is presented 2 waveforms r1 and r2 on every trial.
One waveform contains the signal + noise, and the other just noise. Eg
r1 = s + n1
r2 = n2
The noise in the two waveforms is correlated Gaussian noise with covariance
matrix R. Ie corr(n1,n2) is nonzero.

If there were no correlation, the optimal way to make the decision would be
to cross-correlate r1 with s and r2 with s. If the cross-correlation is
greater than some criterion, decide 'signal in first r1' else 'signal in
r2'.

I am trying to work through the coloured noise case. As I understand it,
the strategy now is to cross-correlate r1 and r2 with s *after* whitening
the received waveforms. The whitening is done by multiplying with the
inverse of the covariance matrix.

I have done simulations and am trying to reconcile the results with what is
expected theoretically. From Whalen p 189:
test statistic: lambda = s' * inv(R) * r
where inv(R) is inverse of covariance matrix R
However this is not conformable. Instead I do
lambda= (r * inv(R))' * s
Does this seem reasonable?

From Whalen
H0: lambda ~ N(0, s' * inv(R) * s)
H1: lambda ~ N(s' * inv(R) * s, s' * inv(R) * s)
However s and inv(R) are not conformable. What if I tack on a column of
zeros alongside s? The thinking is that s= sinewave or zero.

What I find in the simulations is that
signal absent: lambda ~ N(?,?)
signal present: lambda ~ N(E,sigma^2*E)
The ?s mean I am not sure what is going on for signal absent.
Lambda does *not* have a mean of zero when the signal is absent. The mean
is zero only when the correlation is zero. Otherwise the mean is negative,
and it becomes more negative as the correlation increases.

Can anyone please tell me why I am finding results at odds with what is
expected theoretically? Maybe I have theory wrong, in which case please
correct me. [What I *really* want to know is how
d = (mean(signal+noise distrib) - mean(noise distrib)) / sd(noise distrib)
varies with rho.]

Below is some Matlab code (translated from my native language R) that shows
what I have been doing.

Thanks very much for any help!

Cheers
Bill

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

len=100
s= (1.0*cos(2*pi*3.0*(1:len)/len))'

%correlation matrix
rho= [1 .8 ; .8 1]  %correlation between noise vectors is 0.8

%variance-covariance matrix
R=chol(rho)

%generate correlated noise variables n(:,1), n(:,2)
n=randn(len,2)
n= n * k

plot(n(:,1), n(:,2),'o')

r1=s+n(:,1)
r2=n(:,2)
r=[r1,r2]

% Rows give test statistic for r1 and r2
lambda= (r * inv(R))' * s

% cross-correlation is how to do it if white noise
cc = r' * s

% signal is sinewave or 0 -- make s and inv(R) conformable
s0 = [s zeros(len,1)]

%this should be mean and var of test statistic
lambda0 = (s0 * inv(R))' * s