Re: Formal methods for comparing model runs

[Jan Kreft]

Hi Steve,

On Wed, 3 Nov 1999, M. Lang / S. Railsback wrote:

> I'm trying to write up some comparisons of model scenarios; for example,
> comparing the distributions of water depth and velocity used by fish
> when we make two alternative assumptions about how they select habitat.
> 
> At first it seemed obvious to compare the scenarios with a
> Kolmogorov-Smirnov test for differences between distributions. Now,
> though, it seems just as obvious that hypothesis-testing statistics like
> these are not useful for comparing agent-based model runs. First, the
> basic purpose of hypothesis-testing methods is to distinguish
> "information" from "noise". Our model's habitat selection methods,
> though, are completely mechanistic and don't have any noise.

I wouldn't be too sure about that. Incidentally, I just spent two days
trying to measure noise in my simulation. Though it is unclear whether the
fluctuations I observe are artefacts, thus noise, or real fluctuations, or
a combination of both, these fluctuations are clearly there. Now let me
explain. I have a 2D array storing biomass amounts in grid cells i,j. The
agents (bacteria) are spheres in continuous space. The array is filled by
doling out the biomass of each agent into the grid cells this agent
covers, on an area coverage percentage basis. This is done every timestep,
say 1 min. The agents wiggle about as a result of minimizing overlap
between them. Now, if I plot the biomass in all non-empty grid cells i,j
over time with an interval of 1 min, I get many traces without
fluctuations, many with minor fluctuations, and some with huge
fluctuations (CV of up to 0.5). The fluctuations are short-term, say 10
min, compared to the generation time of the agents (few hours). My
conclusion from this is that when comparing biomass arrays from different
runs, I have to use more abstract descriptors that are independent of
these fluctuations/noise rather than computing and analyzing the
difference lattice. 

In your case, there is no randomness in it, right? But you could still
have artefacts from discretizing space and time that could render a
smooth curve rugged. So, rather than doing 1000 runs with the same
timestep and gridsize, I would check that changing these doesn't affect
results. Also, these artefacts could grow if there is some positive
feedback in the model, probably not in your case.

>  Second,
> with a model, the sample size can be arbitrarily large. If I want, I can
> compare model runs with 800 fish each observed on each of 10 days
> (sample size: 8000), or observed on each of 20 days (sample size:
> 16,000). Because the test results depend on sample size, I'm finding
> that even the tiniest differences in results between scenarios are
> statistically significant. 

Of course. Be glad you don't have to use statistics. But there are other
agent-based simulations where this wouldn't hold. If the results depend
very much on (random) initial conditions, and the number of possible
initial conditions is huge, you can't simulate all cases and you will have
sampled only a small part of the population.

> So the consequence is that statistically significant results clearly are
> not biologically significant. (People that count real fish would be very
> jealous.) Consequently, my current plan is to show the distributions
> resulting from each model scenario and talk about the potential
> biological significance of the differences in distributions.
> 
> Anybody know where this issue is discussed in the literature,
> specifically for agent-based models? 
> 
> Steve

Jan.

Jan Kreft                               Phone +44 (0)1222 875278
Cardiff School of Biosciences           Fax   +44 (0)1222 874305
Cardiff University                      E-mail address@hidden
PO Box 915, Cardiff CF10 3TL, UK




                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================