spatial autocorelation methods

i'm facing a problem where i need to do some spacial autocorelation on some
data.  the problem is that all the methods i've found (moran's i, k-means,
etc.) work on rectangular input data sets with no notion of 'missing'.  what i
have is a rectangle with some data and some missing and need to run whatever
calculation i end up using in a way that does not factor in the missing data.
is anyone out there aware of algorithims or, better, code which might to this?

kind regards.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================

"Ara.T.Howard" <Ara.T.Howard@noaa.gov> wrote in message 
news:Pine.LNX.4.60.0410121555140.13675@harp.ngdc.noaa.gov...
>
> i'm facing a problem where i need to do some spacial autocorelation on 
> some
> data.  the problem is that all the methods i've found (moran's i, k-means,
> etc.) work on rectangular input data sets with no notion of 'missing'. 
> what i
> have is a rectangle with some data and some missing and need to run 
> whatever
> calculation i end up using in a way that does not factor in the missing 
> data.
> is anyone out there aware of algorithims or, better, code which might to 
> this?

Is the data subject to lateral movement?  Or, are the positions fixed and 
you just have new data sets and are trying to find the best match?

2-D correlation is the same as 1-D correlation but with a double integral - 
which you might envision as a raster scan with different degrees of overlap 
perpendicular to the raster scan direction - if the coordinate system is 
rectangular.  No matter what it's still multiply and integrate (i.e. 
accumulate) at each value of offset.  So, if there's no lateral motion 
involved then there's no need to look at offset registration and you do a 
single zero-valued offset multiply and integrate.

As far as missing data is concerned, there would be a couple of issues:
- ignoring those coordinates
- weighting for the loss of inputs to the integration.
Assume that there is a grid for NxN data points and that the number of 
missing data points is [I J K L M .... ]

The potential sum of unit samples is N^2.  If the real potential sum is 
N^2-I, N^2-J, ... etc. then I think you might weight the results by 
N^2/[N^2-m] where m is the number of missing points for each integration - 
so that a plane of unit samples (with some missing) will always have a 
weighted integral of N^2.  The same applies if you have to shift the data 
and compute multiple points of the correlation function and you might treat 
the nonoverlapping areas the same way - as "missing".  If you do, then the 
edges will be noisy!!  So, maybe it's better to let the nonoverlap be zeros.

All this is just logical to me - no special knowledge on my part.

Fred 

On Tue, 12 Oct 2004, Fred Marshall wrote:

> Is the data subject to lateral movement?  Or, are the positions fixed and
> you just have new data sets and are trying to find the best match?

hmmm.  i'm not positive what you are asking?  the data are satelite images
(essentially masked) designating fires as detected by a particular sensor.
eg.

   0 0 0 0
   0 1 1 0
   0 1 1 1
   0 0 0 1

where the ones might identify a 'cluster' of fire pixels.  the task is to
identify the 'clusteredness' of the pixels en route to some statistical
verifaction against another data set (which set of variables is corelated with
a 'detect' in the other data set).   i had initially been looking into
algorithms like moran's i, which yield a value indicating the level of
clusteredness, but am leaning towards brute force identification of clusters
and their sizes and all algorithms looked at so far requir input data to be
square.

> 2-D correlation is the same as 1-D correlation but with a double integral -
> which you might envision as a raster scan with different degrees of overlap
> perpendicular to the raster scan direction - if the coordinate system is
> rectangular.  No matter what it's still multiply and integrate (i.e.
> accumulate) at each value of offset.  So, if there's no lateral motion
> involved then there's no need to look at offset registration and you do a
> single zero-valued offset multiply and integrate.

at a fundemental level this is, of course, always true since memory is flat.
i guess what i am trying to say is that my math has degraded to the point
where i'm unsure i can 'flatten' the existing algorithms out there.  ;-)

> As far as missing data is concerned, there would be a couple of issues:
> - ignoring those coordinates

the easiest part in general, but tricky along edges of ragged shapes...

> - weighting for the loss of inputs to the integration.
> Assume that there is a grid for NxN data points and that the number of
> missing data points is [I J K L M .... ]
>
> The potential sum of unit samples is N^2.  If the real potential sum is
> N^2-I, N^2-J, ... etc. then I think you might weight the results by
> N^2/[N^2-m] where m is the number of missing points for each integration -
> so that a plane of unit samples (with some missing) will always have a
> weighted integral of N^2.  The same applies if you have to shift the data
> and compute multiple points of the correlation function and you might treat
> the nonoverlapping areas the same way - as "missing".  If you do, then the
> edges will be noisy!!  So, maybe it's better to let the nonoverlap be zeros.

the moran's i impl. i have pads the edges of the deviation array with zero for
this reason.  again, i'm a bit fuzy here.  sorry.

> All this is just logical to me - no special knowledge on my part.

more than i have ;-)  forgive me if i'm a bit slow on the uptake here - it's
been a long time since i've done any math except that involving powers of two.

regards.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================

<Ara.T.Howard@noaa.gov> wrote in message 
news:Pine.LNX.4.60.0410122127200.14596@harp.ngdc.noaa.gov...
> On Tue, 12 Oct 2004, Fred Marshall wrote:
>
>> Is the data subject to lateral movement?  Or, are the positions fixed and
>> you just have new data sets and are trying to find the best match?
>
> hmmm.  i'm not positive what you are asking?  the data are satelite images
> (essentially masked) designating fires as detected by a particular sensor.
> eg.
>
>   0 0 0 0
>   0 1 1 0
>   0 1 1 1
>   0 0 0 1
>
> where the ones might identify a 'cluster' of fire pixels.  the task is to
> identify the 'clusteredness' of the pixels en route to some statistical
> verifaction against another data set (which set of variables is corelated 
> with
> a 'detect' in the other data set).   i had initially been looking into
> algorithms like moran's i, which yield a value indicating the level of
> clusteredness, but am leaning towards brute force identification of 
> clusters
> and their sizes and all algorithms looked at so far requir input data to 
> be
> square.
>
>> 2-D correlation is the same as 1-D correlation but with a double 
>> integral -
>> which you might envision as a raster scan with different degrees of 
>> overlap
>> perpendicular to the raster scan direction - if the coordinate system is
>> rectangular.  No matter what it's still multiply and integrate (i.e.
>> accumulate) at each value of offset.  So, if there's no lateral motion
>> involved then there's no need to look at offset registration and you do a
>> single zero-valued offset multiply and integrate.
>
> at a fundemental level this is, of course, always true since memory is 
> flat.
> i guess what i am trying to say is that my math has degraded to the point
> where i'm unsure i can 'flatten' the existing algorithms out there.  ;-)
>
>> As far as missing data is concerned, there would be a couple of issues:
>> - ignoring those coordinates
>
> the easiest part in general, but tricky along edges of ragged shapes...
>
>> - weighting for the loss of inputs to the integration.
>> Assume that there is a grid for NxN data points and that the number of
>> missing data points is [I J K L M .... ]
>>
>> The potential sum of unit samples is N^2.  If the real potential sum is
>> N^2-I, N^2-J, ... etc. then I think you might weight the results by
>> N^2/[N^2-m] where m is the number of missing points for each 
>> integration -
>> so that a plane of unit samples (with some missing) will always have a
>> weighted integral of N^2.  The same applies if you have to shift the data
>> and compute multiple points of the correlation function and you might 
>> treat
>> the nonoverlapping areas the same way - as "missing".  If you do, then 
>> the
>> edges will be noisy!!  So, maybe it's better to let the nonoverlap be 
>> zeros.
>
> the moran's i impl. i have pads the edges of the deviation array with zero 
> for
> this reason.  again, i'm a bit fuzy here.  sorry.
>
>> All this is just logical to me - no special knowledge on my part.
>
> more than i have ;-)  forgive me if i'm a bit slow on the uptake here - 
> it's
> been a long time since i've done any math except that involving powers of 
> two.
>
> regards.
>
> -a

Developing clusters is a different thing than correlation I should think. 
Are you also perhaps tracking the clusters?

As far as lateral movement I meant if the satellite images are of the same 
piece of real estate with only a shift in time but not in x or y.

Fred 

You can take a look at the process of "normalized convolution" at
http://www.isy.liu.se/cvl/Research/Filter/normconv.html

Maybe you can translate that to do autocorrelation.

Ara.T.Howard@noaa.gov wrote:
> 
> On Tue, 12 Oct 2004, Fred Marshall wrote:
> 
> > Is the data subject to lateral movement?  Or, are the positions fixed and
> > you just have new data sets and are trying to find the best match?
> 
> hmmm.  i'm not positive what you are asking?  the data are satelite images
> (essentially masked) designating fires as detected by a particular sensor.
> eg.
> 
>    0 0 0 0
>    0 1 1 0
>    0 1 1 1
>    0 0 0 1
> 
> where the ones might identify a 'cluster' of fire pixels.  the task is to
> identify the 'clusteredness' of the pixels en route to some statistical
> verifaction against another data set (which set of variables is corelated with
> a 'detect' in the other data set).   i had initially been looking into
> algorithms like moran's i, which yield a value indicating the level of
> clusteredness, but am leaning towards brute force identification of clusters
> and their sizes and all algorithms looked at so far requir input data to be
> square.
> 

Hi Ara
	Your not being very clear in defining the problem. I don't understand
what the problem is with input data needing to be square. It seems to me
your data *is* rectangular and should be easily divisible into squares.
Also, what do you mean by cluster and clusteredness? What exactly are
you trying to measure?
	 It sounds like you have data in a rectangular grid where each square
(meter^2, acre, square mile) of the grid is assigned a value of 1 (true)
or zero (false) depending on whether a wildfire is present or not. Also
it appears that you want to in some way quantify when groups of
neighboring squares reach some measure of critical mass.
	Assuming the above is correct, you may want to look at dilation and
erosion filters applied to the data to simplify the problem. Erosion
would involve iterating through the array, so that for each square with
a 1, if some threshold (say more than half) of the 8 neighbors is 0,
than change to 0 otherwise do nothing. For dilation iterate through the
array, and for each square with 0 if some threshold (say more than half)
of the 8 neighbors is 1, than change to 1. These sort of filters are
used in similar problems identifying certain scattered tissue types in
medical imaging. 
	Your algorithm might work like this: you grow the areas with 1 with
dilation filter (possibly more than one pass). And then apply erosion
filter (again maybe several passes) to shrink the areas marked as 1.
After sufficient erosion what is left that is still 1's will be the
areas that were of sufficient critical mass to survive. 
	The trick of course is to find the appropriate thresholds for dilating
and eroding, and also the appropriate number of iterations to apply them
so that you get the best fit predictive model of what your data
represents and what you are trying to measure. 
	
	Also, there is the issue that some sort of metric of clusteredness has
already been applied to the data when at some point it was originally
quantified that each square was either a 1 or a 0. To what extent the
sensitivity of that measurement will affect the outcome is an open
question.


-jim

> > 2-D correlation is the same as 1-D correlation but with a double integral -
> > which you might envision as a raster scan with different degrees of overlap
> > perpendicular to the raster scan direction - if the coordinate system is
> > rectangular.  No matter what it's still multiply and integrate (i.e.
> > accumulate) at each value of offset.  So, if there's no lateral motion
> > involved then there's no need to look at offset registration and you do a
> > single zero-valued offset multiply and integrate.
> 
> at a fundemental level this is, of course, always true since memory is flat.
> i guess what i am trying to say is that my math has degraded to the point
> where i'm unsure i can 'flatten' the existing algorithms out there.  ;-)
> 
> > As far as missing data is concerned, there would be a couple of issues:
> > - ignoring those coordinates
> 
> the easiest part in general, but tricky along edges of ragged shapes...
> 
> > - weighting for the loss of inputs to the integration.
> > Assume that there is a grid for NxN data points and that the number of
> > missing data points is [I J K L M .... ]
> >
> > The potential sum of unit samples is N^2.  If the real potential sum is
> > N^2-I, N^2-J, ... etc. then I think you might weight the results by
> > N^2/[N^2-m] where m is the number of missing points for each integration -
> > so that a plane of unit samples (with some missing) will always have a
> > weighted integral of N^2.  The same applies if you have to shift the data
> > and compute multiple points of the correlation function and you might treat
> > the nonoverlapping areas the same way - as "missing".  If you do, then the
> > edges will be noisy!!  So, maybe it's better to let the nonoverlap be zeros.
> 
> the moran's i impl. i have pads the edges of the deviation array with zero for
> this reason.  again, i'm a bit fuzy here.  sorry.
> 
> > All this is just logical to me - no special knowledge on my part.
> 
> more than i have ;-)  forgive me if i'm a bit slow on the uptake here - it's
> been a long time since i've done any math except that involving powers of two.
> 
> regards.
> 
> -a
> --
> ===============================================================================
> | EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
> | PHONE   :: 303.497.6469
> | When you do something, you should burn yourself completely, like a good
> | bonfire, leaving no trace of yourself.  --Shunryu Suzuki
> ===============================================================================


----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= East/West-Coast Server Farms - Total Privacy via Encryption =---

On Tue, 12 Oct 2004, Fred Marshall wrote:

> Developing clusters is a different thing than correlation I should think.

right...

> Are you also perhaps tracking the clusters?

perhaps - but only if moran's index turns out not to be suitable.  the idea is
that if there isn't a conventient algorithim to work on our data set we can
use some existing code which develops clusters.  having developed clusters we
could the the number and average size of them instead of correlation for our
study.  make sense?

> As far as lateral movement I meant if the satellite images are of the same
> piece of real estate with only a shift in time but not in x or y.

yes and no.  we have one large mosaic of data (the reference data set) and
several small peices of overlapping data sets to study.

  --------
  | ref  |
  |-------
  |\      \
  | \samp  \
  |  \______\
  |      |
  --------

where there are several samples.  so for any given sample there is no lateral
movement and there is time shift (we have several dates over same area) but
between samples there is shift.  why do you ask?

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================

On Wed, 13 Oct 2004, jim wrote:

> Your not being very clear in defining the problem.

   sorry - i'm discovering the language to describe it as i go.  i'll elaborate
   as best i can.  in essence we are comparing two data sets using one as the
   reference.  the reference data set is of much finer resolution than the data
   set for the study.  both data sets are byte images where the interpretation
   is that values above a certain value (say 11) are fires.  they are not
   binary images.

> I don't understand what the problem is with input data needing to be square.
> It seems to me your data *is* rectangular and should be easily divisible
> into squares.

   it seems that way, but it's not.  the reason is that one of the data sets is
   of very coarse resolution and has been reprojected/regridded onto the finer
   resolution in order to do the comparison.  further, the grid averaging uses
   nearest neighbor and the sensor in question has massive pixel overlap all of
   which leads to an __original__ course pixel (on coarse grid) that might look
   like this

       -----
       |   |
       |   |___
       |      |
       |   ----
       |   |
       ----

   then this coarse grid is reprojected/regridded onto the fine grid so that
   this shape ends up looking the same to the eye, but being comprised of, say,
   10000 pixels or more.  this set of pixels in the fine grid originating from
   one coarse pixel set (representaon of one pixel in the coarse grid) we are
   calling the pixel 'footprint'.  the reprojecting/regridding to fine res.
   step is merely to make determining the footprint an O(1) operation - it
   could be calculated on the fly of course...

   the task then is to do an analysis of the coarse pixels using the reference
   data as 'truth'.  so what is needed is to 'cut out' (mask) all the reference
   pixels from that footprint giving you a set of reference/fine pixels looking
   something like this

       -----
       |...|
       |.x.|___
       |.xx...|
       |..x----
       |x..|
       ----

   where '.' is < 11 (not fireness) and 'x' is > 11 (fireness).

   remember that the coarse pixel set at this point looks like

       -----
       |nnn|
       |nnn|___
       |nnnnnn|
       |nnn----
       |nnn|
       ----

   a 'set' of pixels with all the same value since the set is merely a result
   of reproj/regrid.  this is of no matter - we consider this set as one sensor
   pixel and then are doing a one to many (data to reference footprint)
   comparison.


   the question is then:  what set of factors in the reference (fine) data
   caused a detect in the test (coarse) data?

   obviously we are looking at things like:

     - how many total fine fire pixels where there
     - what is there total brightness
     - etc.

   we also want some measure of clusteredness and here the problem starts:


   all the code and translations of moran's index i've found work on a
   rectangular image region and i've got something like this

       -----
       |...|
       |.x.|___
       |.xx...|
       |..x----
       |x..|
       ----

   i can clip it out and then stuff it into a minimum bounding box so that it's
   something like

      ----------
      |-----   |
      ||...|   |
      ||.x.|___|
      ||.xx...||
      ||..x----|
      ||x..|   |
      |----    |
      ---------

   but then what to fill the missing values with.  all the code i've read just
   isn't prepared to deal with the situation and the uses the fact that the
   data is rectangular when computing mean, weights, etc.  bascially i need to
   create my own method where the input data is flat, but the weight function
   is aware of scanline adjacency (2dness).

   my original post was asking if this has been done somewhere.


> Also, what do you mean by cluster and clusteredness?

   we don't know.  we are searching for a some measure that will give an
   idicator.  i'm open to suggestion on the meaning of this.

my post is growing too long and so i'll stop here.  hopefully i've clarified
rather than confused.  again, sorry i'm not able to describe the problem
better - this problem is outside my normal domain.

kind regards.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================

Ara.T.Howard@noaa.gov wrote:
> 
> On Wed, 13 Oct 2004, jim wrote:
> 
> > Your not being very clear in defining the problem.
> 
>    sorry - i'm discovering the language to describe it as i go.  i'll elaborate
>    as best i can.  in essence we are comparing two data sets using one as the
>    reference.  the reference data set is of much finer resolution than the data
>    set for the study.  both data sets are byte images where the interpretation
>    is that values above a certain value (say 11) are fires.  they are not
>    binary images.
> 
> > I don't understand what the problem is with input data needing to be square.
> > It seems to me your data *is* rectangular and should be easily divisible
> > into squares.
> 
>    it seems that way, but it's not.  the reason is that one of the data sets is
>    of very coarse resolution and has been reprojected/regridded onto the finer
>    resolution in order to do the comparison.  further, the grid averaging uses
>    nearest neighbor and the sensor in question has massive pixel overlap all of
>    which leads to an __original__ course pixel (on coarse grid) that might look
>    like this
> 
>        -----
>        |   |
>        |   |___
>        |      |
>        |   ----
>        |   |
>        ----
> 
>    then this coarse grid is reprojected/regridded onto the fine grid so that
>    this shape ends up looking the same to the eye, but being comprised of, say,
>    10000 pixels or more.  this set of pixels in the fine grid originating from
>    one coarse pixel set (representaon of one pixel in the coarse grid) we are
>    calling the pixel 'footprint'.  the reprojecting/regridding to fine res.
>    step is merely to make determining the footprint an O(1) operation - it
>    could be calculated on the fly of course...
> 
>    the task then is to do an analysis of the coarse pixels using the reference
>    data as 'truth'.  so what is needed is to 'cut out' (mask) all the reference
>    pixels from that footprint giving you a set of reference/fine pixels looking
>    something like this
> 
>        -----
>        |...|
>        |.x.|___
>        |.xx...|
>        |..x----
>        |x..|
>        ----
> 
>    where '.' is < 11 (not fireness) and 'x' is > 11 (fireness).
> 
>    remember that the coarse pixel set at this point looks like
> 
>        -----
>        |nnn|
>        |nnn|___
>        |nnnnnn|
>        |nnn----
>        |nnn|
>        ----
> 
>    a 'set' of pixels with all the same value since the set is merely a result
>    of reproj/regrid.  this is of no matter - we consider this set as one sensor
>    pixel and then are doing a one to many (data to reference footprint)
>    comparison.
> 
>    the question is then:  what set of factors in the reference (fine) data
>    caused a detect in the test (coarse) data?


        Ok. Let's see if I understand. You have an image sensor that
produces a
coarse image that is binary (I'm guessing it arrived as binary, but
maybe you converted it). You also have the good fortune to have a high
resolution image taken at the same time (hopefully exactly the same
time). Your task essentially is to find out how the low resolution image
came to be using the high resolution as reference. I will assume you are
fairly confident the high resolution image is pretty much free of noise
and registration of the 2 images is fairly accurate. I'm also assuming
both images are in a rectangular array in computer memory.
        
        Your approach to this is to identify where groups of coarse
pixels
exist and compare them to the corresponding pixels in the fine image.
This really doesn't make sense to me. Think of it this way - aren't you
just as interested in why a pixel in the coarse image ended up as 0 as
you are in why another became a 1. 
        What you really would like to know is the "point spread
function" of
each pixel in the coarse image. That is for each coarse pixel what is
the contribution of each fine pixel in the surrounding neighborhood of
that pixel (only the light within a certain radius will contribute to
that pixel). This will probably be a more or less triangle shape
function where the fine pixel at the center of the corresponding coarse
pixel will have the greatest weight and the influence of the fine pixels
will decrease as you move farther from the center. This should be fairly
uniform throughout the image if noise doesn't play any significant role,
so you should really use all the pixels in both images.
        
        The other problem is determining the threshold that is used to
make the
coarse image. That complicates things significantly. It would really be
a lot simpler if the coarse image was grayscale also. Once you throw
information away there really isn't any good way of figuring out how to
retrieve it. After you have figured out what gray value the coarse
pixels should be (correlation will tell how well you did at that),
arriving at the threshold level used should be pretty trivial.

-jim


> 
>    obviously we are looking at things like:
> 
>      - how many total fine fire pixels where there
>      - what is there total brightness
>      - etc.
> 
>    we also want some measure of clusteredness and here the problem starts:
> 
>    all the code and translations of moran's index i've found work on a
>    rectangular image region and i've got something like this
> 
>        -----
>        |...|
>        |.x.|___
>        |.xx...|
>        |..x----
>        |x..|
>        ----
> 
>    i can clip it out and then stuff it into a minimum bounding box so that it's
>    something like
> 
>       ----------
>       |-----   |
>       ||...|   |
>       ||.x.|___|
>       ||.xx...||
>       ||..x----|
>       ||x..|   |
>       |----    |
>       ---------
> 
>    but then what to fill the missing values with.  all the code i've read just
>    isn't prepared to deal with the situation and the uses the fact that the
>    data is rectangular when computing mean, weights, etc.  bascially i need to
>    create my own method where the input data is flat, but the weight function
>    is aware of scanline adjacency (2dness).
> 
>    my original post was asking if this has been done somewhere.
> 
> > Also, what do you mean by cluster and clusteredness?
> 
>    we don't know.  we are searching for a some measure that will give an
>    idicator.  i'm open to suggestion on the meaning of this.
> 
> my post is growing too long and so i'll stop here.  hopefully i've clarified
> rather than confused.  again, sorry i'm not able to describe the problem
> better - this problem is outside my normal domain.
> 
> kind regards.
> 
> -a
> --
> ===============================================================================
> | EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
> | PHONE   :: 303.497.6469
> | When you do something, you should burn yourself completely, like a good
> | bonfire, leaving no trace of yourself.  --Shunryu Suzuki
> ===============================================================================


----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= East/West-Coast Server Farms - Total Privacy via Encryption =---

On Wed, 13 Oct 2004, jim wrote:

> Ok. Let's see if I understand. You have an image sensor that produces a
> coarse image that is binary (I'm guessing it arrived as binary, but maybe
> you converted it). You also have the good fortune to have a high resolution
> image taken at the same time (hopefully exactly the same time). Your task
> essentially is to find out how the low resolution image came to be using the
> high resolution as reference. I will assume you are fairly confident the
> high resolution image is pretty much free of noise and registration of the 2
> images is fairly accurate. I'm also assuming both images are in a
> rectangular array in computer memory.

yes on all points.

> Your approach to this is to identify where groups of coarse pixels exist and
> compare them to the corresponding pixels in the fine image.  This really
> doesn't make sense to me. Think of it this way - aren't you just as
> interested in why a pixel in the coarse image ended up as 0 as you are in
> why another became a 1.

not quite.  the approach is

   for each pixel transmitted from satelite

     determine it's irregular footprint in the reference data

       reference_set = reference[footprint]

       sa = calculate_spatial_autocorrelation reference_set

       record pixel, sa, other_stats

     end

   end

afterwords we'll be asking - did high sa (and of course other measures) prove
to be a significant factor in high satelite pixel value (fire).

the coarse/fine/proj/grid business is simply a method to form a uniq mapping
of satelite pixel to footprints and can be ignored except for the fact that
the footprints end up not being square, which is an artifact of the
methodology and the crux of my problem.  in this way each and every satelite
pixel will be compared against one, and only one, footprint/reference set in
such a way that ALL reference pixels eventually will be accounted for.  think
of it as the reference data being a jigsaw puzzle where each satelite pixel
maps to one exactly one peice, and that peice is many reference pixels.  so
you are correct in that we are interested in why a pixel in the coarse image
ended up as 0 or > 1 and thus are, in fact, accounting for ALL pixels.


> What you really would like to know is the "point spread function" of each
> pixel in the coarse image. That is for each coarse pixel what is the
> contribution of each fine pixel in the surrounding neighborhood of that
> pixel (only the light within a certain radius will contribute to that
> pixel). This will probably be a more or less triangle shape function where
> the fine pixel at the center of the corresponding coarse pixel will have the
> greatest weight and the influence of the fine pixels will decrease as you
> move farther from the center. This should be fairly uniform throughout the
> image if noise doesn't play any significant role, so you should really use
> all the pixels in both images.

not quite.  we want to know if the fine pixels affect the coarse pixel
differently based on how the bright ones are clusted.  eg, if the fine data
were:

   --------
   |x.....|
   |..x...---
   |....xx..|
   ----------

is it more or less likely to cause a signal in the coarse pixel than, say:

   --------
   |..xx..|
   |..xx..---
   |........|
   ----------

in otherwords for two scenes of equal mean brightness, does clustering of
bright pixels increase detection?  our hypothesis is that the nature of the
sensor suggests that clustering _should_ increase detection.

> The other problem is determining the threshold that is used to make the
> coarse image. That complicates things significantly. It would really be a
> lot simpler if the coarse image was grayscale also. Once you throw
> information away there really isn't any good way of figuring out how to
> retrieve it. After you have figured out what gray value the coarse pixels
> should be (correlation will tell how well you did at that), arriving at the
> threshold level used should be pretty trivial.

it is 'sort of' greyscale:  all pixels below a threshold are zero'd - all
others retain the original value.  it's a mask of fires where the non-zero
pixels tell you 'fireness'.  this may not be ideal - but the task to evaluate
the accuracy of this image - so modifying it is out be definition.  eg. we
want to say that, for a person looking at the image, how accurate is the image
w.r.t. geolocation and detection, and also give an estimation of the ground
factors likely to be detected vs. those that aren't.

kind regards.

ps.  thanks for taking the time here - i really appreciate the feedback.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================