[Openspace] LISA used in Geoda and ArcInfo

Wed May 14 03:56:27 CDT 2008

On Wed, 14 May 2008, Patrick Yang wrote:

> Dear everyone,
>
> Thank you for all the discussion. It is definitely that GeoDa is a fantastic
> tool for EDSA and there is no bug on GeoDa side. I want to report here
> is that in the latest ArcGIS 9.2, by using the option of standarized row for
> weights, I got the same result as GeoDa for LISA and global Moran I. The
> difference is only that ArcGIS use Z while GeoDa use permutation, as said by
> Julia and Anslein, so the test is different. With personal communication
> with Prof. Anslein, I totally agree with him that permutation is more
> reliable.
>
> The problem I met days ago is due to the projection. Remove the projection
> and process shapefile in Geoda, and then the results become the same.

OK, so the local Moran's I values differed between GeoDa and Arc because 
the weights were different, and I suppose distance weights rather than 
contiguity weights.

Has anyone ever seen any documentation of the Arc Weights_Matrix_File 
optional argument to the toolbox functions - read_user_weights seems to 
read something like a GWT file? If we can work that out, we can run FROM 
TO WEIGHT text files from GeoDa or R (or Matlab) in Arc, and cross-check.

Roger

>
> Thank you for Roger, Julia and Anslein for your great help and effective
> discussion.
>
>
> Bests,
> Zhenshan
>
> On Wed, May 14, 2008 at 12:11 AM, Julia Koschinsky <koschins at uiuc.edu>
> wrote:
>
>> Dear all,
>>
>> We've been aware of this discrepancy for a while (discussed first on Dec.
>> 17, 2004 on this list). In the past, one of the differences was that Arc
>> applications used analytic reference distributions (e.g., normal) for LISAs
>> while GeoDa uses computational reference distributions based on permutations
>> (the latter are consistent with the original 1995 LISA paper, so the "bug"
>> is not on GeoDa's side).
>>
>> Julia
>>
>> ---- Original message ----
>>> Date: Tue, 13 May 2008 23:39:34 +0200 (CEST)
>>> From: Roger Bivand <Roger.Bivand at nhh.no>
>>> Subject: Re: [Openspace] LISA used in Geoda and ArcInfo
>>> To: Patrick Yang <yangzspat at gmail.com>
>>> Cc: openspace at sal.uiuc.edu
>>>
>>> On Tue, 13 May 2008, Patrick Yang wrote:
>>>
>>>> Thank you Roger,
>>>>
>>>> I just find there is perhaps a bug in GeoDa. I caculated global Moran I
>>>> manually as sugested by the GeoDa tutorial. The spatial lag value,
>> based on
>>>> Rook weight, are obviously problematic, not computed as the average of
>> the
>>>> neighbouring values. I am trying now to compare this by the data of
>> Geoda
>>>> samples.
>>>
>>> Please wait until you are using shared data. I have posted:
>>>
>>> http://spatial.nhh.no/R/etc/sids2a.zip
>>>
>>> which contains a shapefile (the GeoDa distribution shapefile, with 4
>> extra
>>> fields: RR is the SID74/BIR74 raw rate, and the saved results from a
>>> univariate LISA using the enclosed contiguous county queen weights. I
>> also
>>> include a short R script showing how the results match (they small
>>> difference is caused by GeoDa usin (n-1) in computing the sample
>> variance,
>>> which is arguably better than localmoran() in spdep which just divides by
>>> n to agree with other work.
>>>
>>> Try to get Arc to give you numeric results using this shapefile and these
>>> weights if you can. My Arc 9.1 toolbox is broken, and another list member
>>> tried but failed to get 9.2 to compute anything, so no working ArcGIS
>>> installations with working Local Moran Python scripts have come forward.
>>> Can anyone help? The most likely cause of the difference is that the Arc
>>> script does not generate the same weights, and the weights have to be the
>>> same to get the same answer.
>>>
>>> Roger
>>>
>>>>
>>>> With best regards,
>>>> Zhenshan
>>>>
>>>>
>>>>
>>>> On Tue, May 13, 2008 at 10:06 AM, Roger Bivand <Roger.Bivand at nhh.no>
>> wrote:
>>>>
>>>>> On Mon, 12 May 2008, Patrick Yang wrote:
>>>>>
>>>>>  Dear Roger and other fellows,
>>>>>> Thank you very much. In deed I think the results from Geoda are more
>>>>>> reliable since it is original. But in terms of actual implications, I
>>>>>> much
>>>>>> believe the result of Arc Info. I attached the outcomes in graphics
>> and
>>>>>> also
>>>>>> included my operation process for your reference. Hopefully you can
>> help
>>>>>> me.
>>>>>>
>>>>>
>>>>> Patrick sent me his bulky Word document off-line, but unfortunately,
>> it
>>>>> did not contain anything helpful. I asked for screen shots not of the
>> output
>>>>> cartography, but of the actual input menus used to generate the
>> results.
>>>>> Sadly, neither of the software implementations is scripted, so no
>> proper
>>>>> history or journal is available (although all the argument values are
>> shown
>>>>> on the input forms, and also on Arc's Python report screen). So on
>> Windows
>>>>> Ctrl-Alt-PrintScreen of the input and report windows, please, or
>> screen
>>>>> capture to a paint program.
>>>>>
>>>>> What is needed is:
>>>>>
>>>>> The original data - shapefile as read into both Arc and GeoDa,
>> indicating
>>>>> which column is the problematic one;
>>>>>
>>>>> The GeoDa GAL file used (it can be read into Arc too, if need be);
>>>>>
>>>>> Screen shots of the input arguments as PNG files.
>>>>>
>>>>> It is impossible to debug software from output pictures - we have to
>> know
>>>>> what the input was.
>>>>>
>>>>> If your data are very private, use a sample data set from GeoDa, and
>> try
>>>>> to reproduce the problem. Your report of the Arc input includes
>> "Euclidean
>>>>> distance", which suggests that you may not have got the contiguity
>> weights
>>>>> you asked for, but rather inverse distance. I don't have access to Arc
>> 9.2,
>>>>> and 9.1 was known to be broken in some weighting settings, so we'll
>> need
>>>>> someone else to join in who has Arc 9.2 access.
>>>>>
>>>>> Roger
>>>>>
>>>>>
>>>>>
>>>>> I am much appreciated.
>>>>>>
>>>>>> The software version I used GeoDa 0.9.5-i5 (Aug 3, 2004) and Arc Info
>>>>>> V9.2.
>>>>>>
>>>>>> Thank you,
>>>>>> Zhenshan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, May 12, 2008 at 8:04 PM, Roger Bivand <Roger.Bivand at nhh.no>
>>>>>> wrote:
>>>>>>
>>>>>> On Mon, 12 May 2008, Patrick Yang wrote:
>>>>>>>
>>>>>>> Hello everybody,
>>>>>>>
>>>>>>>>
>>>>>>>> Does any one have both experiences of computing LISA in Geoda and
>>>>>>>> ArcInfo
>>>>>>>> package of Spatial Statistic Tools. I ran the same set of areal
>> data
>>>>>>>> based
>>>>>>>> on the same spatial weight matrix (the first order contiguity), but
>>>>>>>> got
>>>>>>>> very
>>>>>>>> different outcomes:
>>>>>>>>
>>>>>>>>
>>>>>>> Dear Patrick,
>>>>>>>
>>>>>>> In situations like this, it really helps to provide example data
>> (for
>>>>>>> instance a downloadable compressed archive file with the data files
>>>>>>> and
>>>>>>> screen shots of the two software systems showing the selected
>> argument
>>>>>>> values). I'm sure you have checked every possibility that this is
>> not
>>>>>>> a
>>>>>>> difference in your use of the software, but a difference in the
>>>>>>> software
>>>>>>> itself, but more eyes seeing your situation may see something that
>> you
>>>>>>> have
>>>>>>> not thought of.
>>>>>>>
>>>>>>> The key thing here is that the local Moran I values differ. This can
>>>>>>> come
>>>>>>> from three things: different data, different weights, and/or
>> different
>>>>>>> software implementations. As both implementations are widely used,
>> the
>>>>>>> community would benefit greatly if we could establish what is going
>>>>>>> on. In
>>>>>>> the Arc case, the Python code can be read if need be, but GeoDa
>> local
>>>>>>> Moran's I values typically agree with the R spdep localmoran()
>>>>>>> function
>>>>>>> output for the same data and weights. In other words, GeoDa is
>> pretty
>>>>>>> reliable, and indeed by the author of the original local Moran's I
>>>>>>> paper.
>>>>>>>
>>>>>>> Can you provide a documented example showing the problem? Could you
>>>>>>> also
>>>>>>> please say which build of both implementation you are using - it may
>>>>>>> be
>>>>>>> something that has been fixed in a later release?
>>>>>>>
>>>>>>> Once the local Moran's I values agree, other differences will be
>>>>>>> related
>>>>>>> to how any testing is carried out, and here there is more room for
>>>>>>> random
>>>>>>> differences (different random number streams in GeoDa itself can
>> give
>>>>>>> slightly different results). So let's take this step by step.
>>>>>>>
>>>>>>> Hope this helps,
>>>>>>>
>>>>>>> Roger
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - most part of area are recoginized as low-low significally in
>> Geoda,
>>>>>>>> but as
>>>>>>>> random process based on the Z-core (cut off by 1.96) in ArcInfo
>>>>>>>> - There is no high-high combination in Geoda, but can find hot
>> spots
>>>>>>>> in
>>>>>>>> ArcInfo
>>>>>>>> - The caculated local Moran I is different in the two softwares
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Can anybody kindly help me to explain the reason? PS: I choose the
>>>>>>>> standarized row for the weight matrix in ArcInfo but I guess in
>>>>>>>> GeoDa it
>>>>>>>> is
>>>>>>>> the same configuration. if it is not so, please point it out. Any
>>>>>>>> what
>>>>>>>> other
>>>>>>>> reasons? Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Zhenshan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>> Roger Bivand
>>>>>>> Economic Geography Section, Department of Economics, Norwegian
>> School
>>>>>>> of
>>>>>>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>>>>>>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>>>>>>> e-mail: Roger.Bivand at nhh.no
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Roger Bivand
>>>>> Economic Geography Section, Department of Economics, Norwegian School
>> of
>>>>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>>>>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>>>>> e-mail: Roger.Bivand at nhh.no
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Roger Bivand
>>> Economic Geography Section, Department of Economics, Norwegian School of
>>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>>> e-mail: Roger.Bivand at nhh.no
>>>
>>> _______________________________________________
>>> Openspace mailing list
>>> Openspace at sal.uiuc.edu
>>> http://sal.uiuc.edu/mailman/listinfo/openspace
>>
>
>
>
>

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no