Criteria for evaluating model deficiency for groundwater models and the effects of eliminating deficient models on multi-model analysis using AICC, KIC, AIC, and BIC

Abstract

Multi-model analysis (MMA) considers multiple model interpretations of a system. MMA provides a more realistic assessment of uncertainty associated with model predictions because both uncertainty of individual models and uncertainty associated with different model structures are considered. Models are evaluated for the strength of evidence that they represent an unknown system using different Information Criteria (IC) equations. IC equations are designed to assess the likelihood that a model in a set of models represents the true but unknown system. IC equations do not include a component which identifies a deficient model. Therefore, inclusion of deficient models in the set of models leads to poor model-averaged results. Evaluation of models to assess whether available observation data sufficiently support the model structure is an important step in MMA. Measures for evaluating models include: 1) failure to reach proper convergence during non-linear regression; 2) unreasonable parameter estimates; 3) unreasonable confidence intervals on parameters or a coefficient of variation greater than ten for one or more parameters; 4) high correlations between parameters; 5) determinant of the correlation matrix less than 1x10-12; 6) condition number of the Jacobian matrix greater than 2000; and 7) unreasonable confidence intervals on predictions. Experiments presented herein are designed to evaluate how components of AIC, AICc, BIC, and KIC rank models and assign model probabilities, and to demonstrate how removing deficient models improves MMA results. Synthetic models are used to represent true but unknown systems in contrast to experimental models that are created to simulate a simplified version of the unknown system based on observation data taken from the synthetic models. AIC, AICc, BIC, and KIC generally assign high probability to deficient models. AICc generally assigns high probability to deficient models if 1) there are many observation data or 2) there are few observation data and the model fits the data well. KIC generally assigns high probability to deficient models because these models have low Fisher Information. AIC and BIC are influenced by the goodness-of-fit and are more likely to assign high probability to more complex models because these models are generally over-fitted. Removing deficient models results in improved MMA results using AIC, AICc, BIC and KIC.