Re: [Swarm-Modelling] comparing models

[gepr]

I wanted to comment on this before; but, I haven't had much time.

The _best_ way to go about comparing models is pretty simple, actually.

First off, completely avoid drawing conclusions about what the model(s)
_mean_.  Don't run off saying that the results of a model imply we
should, say, "change economic policy", "kill all the mosquitos in a 
5-mile radius of Cleveland", "invest a bunch of money in cloning", 
etc.  This point may be obvious; but, sometimes it bears repeating.

"Modeling" is _not_ science and it's not engineering.  As such, it
doesn't _produce_ anything useful in and of itself.

This holds true for "verification" and "validation", as well.  When
you go about "validating" your model by comparing its outputs to
either the output of a real system or the outputs of another model,
all you're doing is measuring the differences between two sets of
data.  That data says, literally, nothing about where it came from.
You could have gotten either data set by decrypting messages from the
Dog Star.

Second, when presenting results from a model, simply present the 
motivation for the model, the process by which you created the model,
the model, the process by which measurements are taken off the 
model, and the measurements of the model.  (Present the same collection
when you present other models or the real system.)

When presenting the "validation" or comparisons and contrasts to other
models or a real system, simply present the two sets of data, present
the motivations for how you compare the two, how you compare the two,
and what the result of the comparison is.

There will always be legitimate reasons to question any one part of
this collection, including the motivation for doing the work, the
processes for creating the models, taking measurements from the real
system, etc.  Even if you're "Bob-the-God-of-this-domain", there will
always be valid objections to any given part of what you've done.
The reasons this is true (and will always be true) is because modeling
is not an automatable process.

So, as to your questions about which techniques are best, just pick a
few, do the work, write down the results.  Pick a few more, do the
work, write down the results.  Etc.  If a sizable sampling of
techniques (e.g. 3 statistical, 2 from feature extraction, 1
state-space reconstruction, 2 in signal analysis) all give you a
certain result (e.g. model 1 and model 2 lead to the same
conclusions), then it may be worth pointing that out to some audience.


Steve Railsback writes:
 > Scott Christley wrote:
 > > 
 > > Anybody familiar with Robert Axelrod's cultural dissemination model?
 > > And the follow up paper by Axtell, Axelrod, Epstein, and Cohen which
 > > "docks" Sugarscape with Axelrod's model?  Even if you don't, that's
 > > fine, my questions are general enough.
 > 
 > By the way, the book these are in is an important read for anyone doing
 > ABMs.
 >  
 > > ...
 > 
 > > I've implemented Axelrod's model in Swarm, now I am going to implement
 > > the model again but with a fundamentally different algorithm(cultural
 > > dissemination rule) underneath.  Then I want to compare them to see if
 > > they are equivalent.
 > 
 > I am currently revising a book chapter on analyzing ecological ABMs, so
 > I should have something to say, but this is certainly an under-developed
 > field. In brief, what we're recommending is:
 > 
 > a. The most important question in comparing two versions of a model is
 > whether the two versions lead to the same conclusions about the system
 > you're modeling. 
 > 
 > b. Often, the best way to make this comparison is by identifying some
 > patterns that "capture the essence" of the system, then see which
 > versions of the model cause those patterns to emerge. In other words,
 > the 'weak equivalence' is the most important.
 > 
 > c. There are a number of potential pitfalls to statistical comparison,
 > some of which you've identified. The sample size is arbitrary- how many
 > times you run the model determines how "significant" differences are in
 > the distribution of results. The comparison can depend on parameter
 > values that may not be well defined. And the distribution of results you
 > get from replicate model runs is completely an artifact of how you use
 > random numbers in your model; so if you compare two algorithms that
 > differ in the degree to which their results depend on random numbers,
 > this difference could really affect the "significance" of differences in
 > results.
 > 
 > Some of these issues are discussed in a little paper whose title
 > (Getting "results"...) arose from a question Paul Johnson posted here
 > several years ago. You can download it here:
 > http://math.humboldt.edu/~simsys/Products.html
 >  
 > > Now statistics is not my strong suit, so I hope that somebody can give
 > > me some pointers or suggest some reading material.
 > > * In the docking paper, they mention two statistical tests: two-sided
 > > Mann-Whitney U statistic and the Kolmogorov-Smirnov (K-S) test.  Are
 > > there any others?  Any good books or papers that talk about these types
 > > of tests, pros-cons, underlying assumptions, etc?
 > 
 > What you need is just a basic statistics text book. I (being equally
 > ignorant of statistics) would just go to the library and look through
 > the statistics books (at the section on comparing distributions) until
 > you find one you can understand and that answers your questions. But I
 > would also run my analysis past somebody that knows statistics before
 > attempting to publish them.
 > 
 > Our experience has been that reviewers tend to be very picky about
 > statistical analysis of results from ABMs---apparently the whole idea of
 > "data" produced by a model makes them nervous, so they are even more
 > critical than usual.
 > 
 > And one solution we've had to use actually might be perfectly adequate
 > for you: just run each version of the model a bunch of times, then draw
 > histograms of the output, and compare them visually. This actually tells
 > you more, with less gobbledygook and fewer things for reviewers to snipe
 > at, than running statistics. Talk about how similar the shapes are, what
 > parts of the distributions are different, etc. It may be useful to also
 > throw in a K-S test etc.- perhaps to reinforce a point that is already
 > clear from the vision comparison (e.g., that the two distributions were
 > nearly identical or really different). 
 > 
 > Finally, according to my co-author Bret Harvey, who Knows All concerning
 > statistics, if you just want to compare the *means* of some model output
 > between two versions of a model, you use one-way ANOVAs followed by
 > pairwise comparisons using Bonferroni t
 > tests. An example is in the paper "Analysis of habitat selection
 > rules..." at the same web site. 
 >  
 > Steve R.
 > 
 > -- 
 > Lang Railsback & Assoc.
 > 250 California Ave.
 > Arcata CA  USA 95521
 > 707-822-0453; fax 822-1868
 > _______________________________________________
 > Modelling mailing list
 > address@hidden
 > http://www.swarm.org/mailman/listinfo/modelling

-- 
glen e. p. ropella              =><=                           Hail Eris!
H: 503.630.4505                              http://www.ropella.net/~gepr
M: 971.219.3846                               http://www.tempusdictum.com