Testing and Validation

inSTREAM is a large, complex model so we take testing and validation very seriously. Unlike simple ecological models, we cannot assume that reasonable values for all parameters can be found via calibration, or that any mistakes will be apparent from model results. Instead, we have developed, applied, and published the following methods for (1) making sure the model's assumptions, parameters, and input are reasonable and useful, and (2) eliminating mistakes from the software. (See also: Ropella et al. 2002, “Software engineering considerations for individual-based models” on the Products page.)

• Independent design, testing, and calibration of submodels. Each process in the model is treated as a separate submodel that is calibrated and tested independently to the extent possible. For example, the equations for how a fish feeds and grows were implemented in a spreadsheet and tested to ensure they produce reasonable results over wide ranges of habitat conditions and fish sizes.
• Software reviews. The computer code for inSTREAM is reviewed by someone other than the programmer who wrote it. The review not only searches for mistakes in implementing the model formulation, but also ensures that the software is well-organized, self-documenting, and easy to understand and modify.
• Graphical testing. The graphical user interfaces are used to (1) look for unreasonable behaviors and (2) spot-check specific calculations.
• Double implementation of submodels. All the submodels for fish actions (e.g., feeding and growth, mortality risks, habitat selection, deciding when and where to spawn) are programmed twice and tested by making sure both implementations produce the same results over thousands of test cases. Usually, the submodels are coded in a spreadsheet in addition to the inSTREAM software. Intermediate output from the inSTREAM software is then imported to the spreadsheet for comparison. An example of these test spreadsheets can be downloaded here (CodeTest.zip : zipped Excel file, 938 kb).
• Pattern-oriented testing of key theoretical assumptions. The key theoretical innovation of inSTREAM is its approach to modeling how individual trout select habitat: how they move to make good tradeoffs between mortality risks and growth. In Version 3, this theory is expanded to also include activity selection: the decision of whether to feed or hide each time step. Because the theory used to model habitat (and activity) selection is critical yet new, we tested the theory by showing that it can reproduce a wide range of behavior patterns observed in real fish. For a description of this testing process, see the Products page for Railsback (2001; “Getting ‘results': the pattern-oriented approach to analyzing natural systems with individual-based models”). For results of tests on inSTREAM, see Railsback and Harvey (2002; “Analysis of habitat selection rules using an individual-based model”) and Railsback et al. (in preparation; “Tests of a theory for diel variation in activity and habitat”).
• Pattern-oriented analysis of inSTREAM predictions. We tested inSTREAM's ability to predict population-level responses to habitat and biological conditions by examining its ability to reproduce a variety of patterns observed in real populations. See the Products page for Railsback et al. (2002; “Population-level analysis and validation of an individual-based cutthroat trout model”).