Comments on Computer Models and the Bay Program

• The purpose of a model should be clearly explained. This is important to define the “abilities” needed within the model and to properly define the use of model results.
• Model results should be clearly and comprehensively explained. It is far too easy for model results to be used in a context for which the results are not defensible. Model results from the watershed model are typically used to represent “actual” reductions in nutrient and sediment loads to Chesapeake Bay. The watershed model has no ability to provide estimates of “actual” reductions. This is not an inherent problem if model results are properly explained.
• The accuracy of model results should be defined and the results presented in a format consistent with the accuracy. Currently when model results are downloaded from the Bay Program website, the results are reported to an apparent level of significant figures far beyond any reasonable level of possible accuracy. As the Bay Program moves into the regulatory TMDL mode, the importance of understanding model accuracy will be essential. EPA anticipates using the watershed model to determine whether or not the States have attained the level of reductions indicated by the State’s 2-year milestones. Furthermore, EPA is considering legal “consequences” against a jurisdiction not reaching the 2-year milestone. If a state is within the accuracy of the model to project “attainment” of the milestone, but has not reached the exact number, how important would that be?
• Many would say that it is not the responsibility of the modeling community to appropriately explain model results. However, if the modeling community does not sufficiently explain how their models work and what the model represents in an understandable way, those who communicate the results may not have the understanding or desire to do so.
• Future Bay program modeling should be more watershed/state specific than the current Phase 5 model. The intended use of the watershed model is different between the Bay states. For example, Maryland will be using the Phase 5 model for a portion of its TMDL program. This is why Maryland provided funding in support of Phase 5 development. Pennsylvania does not use HSPF for its TMDL program, but does use BMP efficiencies for its trading program. Also, states with Bay waters are more concerned with oysters, crabs and menhaden. Headwater states are more concerned with Ag and urban issues. Obviously the outlook and stakeholder interest between the two are totally different.
• The desire for more specific watershed/state models would suggest individual basin oriented models. The present concept of uniformity of calibration as a form of “equity“ in modeling can ignore the inherent difference in the distribution and proportion of source loads within the states. An “automated” calibration is being used for the Phase 5 model because of its complexity and segment scale. To what extent does this approach affect the accuracy of the model at the individual state level?
• The issue of model complexity is relevant to a modeler; but less so to stakeholders and program managers. It is only necessary that model results are packaged in a usable manner. As you move up the chain of command, the detail of information needed gets less and less. The bottom line becomes more important. However, if model complexity requires additional data from the intended users that goes beyond the perceived intended use of the model, then complexity becomes a handicap and not a benefit.
• Model accuracy should not be construed as complexity. Complexity is only useful if sufficient data and understanding exists to represent such in a model. It is possible to have a model that reasonably matches the calibration data, but subsequent “what-if” projections are in error. The ability of the model to simulate actual conditions will determine whether the model is giving the right answer for the right reason. Otherwise, application of the model to make projections could be misleading. For example, the Bay water-quality model poorly defined an all-forest scenario when the existing allocations were derived. This was not the fault of the model, but clearly showed that there was insufficient information and understanding on how to simulation such a condition. Granted, this is an extreme example, but it shows the need to understand the limits of existing data and scientific understanding when modeling.
• Most stakeholders care little for being involved in the modeling process. Most stakeholder care more for actual water quality. Whether the water getting better is by far the most heard question and not, What does the model say?

I wonder if the TMDLs produced with existing modeling tools will be eagerly accepted by the community at large, and will drive the further remediation efforts? What are the chances that there will be disagreement, and that only more conflict and, possibly, court hearings are quite likely? Are the existing modeling tools viewed as bullet-proof and secure from any challenge from parties that are unhappy with the measures required?

Bill makes a very good point that “very large one size fits all watershed model” may not be the most efficient approach, especially if we have different goals for various watersheds and patchy data sets. Should we expand the monitoring efforts to provide the same level of data for the whole watershed? I don’t see this as feasible. Especially in the economic situation we have now and with the many other looming priorities coming from climate change and resource depletion.

Maybe it would be more prudent to rely on existing data sets, which, in fact are probably the best we can find for an ecosystem of this level of complexity and size. Maybe instead of a constant race for more data, we should adapt our models to the data that are available. This would then also mean that we should have the capability to zoom in to areas that have better data, using more detailed models, and, also zoom out to present the whole watershed and estuary with something fairly simple and well grounded in the existing monitoring results.

After all, we don’t want to forget about the big elephant in the room. The population in the Bay area is projected to keep increasing and may reach 20 mln by the year 2030 (http://www.bayjournal.com/image.cfm?row=5&article=3581). It is unlikely that the TMDLs that are to be implemented can compensate for that, even less will they be able to get the ecosystem to show any improvement compared to the existing conditions. What’s the point of refining our monitoring and modeling capabilities if they ignore the elephant?

What good will a 3D model that simulates in a more physically based way groundwater interactions and lag times if we don’t have any significant amount of groundwater monitoring to calibrate the model against? I think instead of working on a very large one size fits all watershed model that we need to take each river basin and mine what data we have and construct tributary models that would each rival the computational requirements of the existing phase 5.x WSM. In those few basins that might have groundwater monitoring sufficient we could employ more sophisticated 3D models. Where we do not maybe we have a modified HSPF or other watershed model. We need more ability to accurately distribute loadings and land uses within a river basin than what many lumped parameter models can provide so I would favor a model that allows a true distribution of land uses and interaction with the entire stream network not just the large 3rd, 4th and greater order streams.

I would also submit that once the TMDL and possible UAA’s are done the modeling effort may become less important to managers. Once we have established implementation and targeting goals with the existing models we can measure success by how many of the BMPs have we installed and what does the actual water quality monitoring indicate. If the states are implementing measures the monitoring should indicate reduction in loadings and or improvement in the estuary. Non of this requires any modeling and could redirect funding to do less modeling and more monitoring and implementation. Remember the de-listing of the impairments on any waterbody is based on actual monitoring not on any modeled “progress”. Many persons see the modeling effort as an unnecessary expense draining resources that could be used toward getting implementation done. Without the need to develop a TMDL this perception would likely be more widespread. And the more distance one is from Annapolis or Philly the output of the modeling is trusted less and less.

So here’s my perspective on this:

In 25 years, I’ll be dead…

(or wheeling around in Depends) and my kids will have to deal with the Bay’s recovery and the huge cost involved. If we cannot develop useful information from predictive models that policymakers at all levels – esp local government – can use in the near future, then this is not a useful expenditure.

We need to go back to basics and look at the inputs to these models, truth them (I was upset to learn about the lack of field testing of BMP performance, for example), make best guesses, and realize that the policy process is dealing in whole numbers while the science/modeling effort seems to be seeking some certainty x digits to the right of the decimal point.

There is no question that the next generation of CMDSS models will need to be much more flexible than the current CBP modeling system. Rather than one big model, we should move toward developing the next CMDSS as a suite of models that can be applied at a variety of scales. The scaling issue has been particularly problematic, i.e., it is very difficult to use a model that has been developed for the entire Chesapeake Bay to address questions related to small sub-estuaries and inlets.

As Alexey points out, the next CMDSS will also need to be much more transparent. The CBP has made great strides in recent years towards increasing the transparency of the models (i.e., making their codes accessible to the scientific community and the public). However, the next generation CMDSS needs to be developed from the outset using an open source approach. For example, following the approach that has been used in the development of the ROMS hydrodynamic, biogeochemical and sediment transport models. This will not only require a very different approach to the model development, but also a very different oversight structure than that which is currently in place.

One key issue that must be addressed is the need to eliminate conflict of interest among those who are asked to provide oversight and judge the functionality of the models and those who are funded to work on them. Obviously, no members any CMDSS oversight/advisory committee should be simultaneously receiving funding to work on CMDSS model development.

1. The Chesapeake is not getting any better, and most of the problems are becoming only more pressing;
2. It is unlikely that we will be enjoying the same funding opportunities as before. Resources are going to be getting only tighter as climate change and peak oil are kicking in.

Therefore, clearly, we will need to achieve more at lower cost.

One possible way to optimize our efforts is to try to make our models more driven by the stakeholders and decision makers. What exactly do they need? What are the most critical issues that they are facing? How models can come to a rescue?

We need an on-going open discussion to solicit this feedback and translate those needs into model functionality. I was very much hoping to start this dialogue with managers and stakeholders during the upcoming conference on “Ecosystem Based Management: the Chesapeake Basin & Other Systems”. Unfortunately our session proposal was rejected, so we will have to find other ways to engage the stakeholders in our discussion.

It would be nice if we could ask the decision makers that we know to contribute to this blog. Tell us what are their concerns, what are the needs.

We have pretty good ideas about how models should be open source, interoperable, transparent, component based, etc. But we do want to try to avoid the “one size fits all” syndrome and to make sure that the CMDSS can be configured towards the special needs of various stakeholders.

I expect development of the the next Chesapeake Modeling and Decision Support System (CMDSS) will undergo a similar transformation. But unlike the voluntary efforts that guided our current suite of models, this next generation will be driven by a regulatory framework. The next applications should be coupled with more highly specialized models to address local to regional needs of local, state and federal interests. Such needs will require more flexibility employing the next generation of computers and models. This will have to be integrated and guided by the scientific community.

The goals will need to be highly adaptive with often unrealistic deadlines. While the past 25 years have been an exciting challenge, I think the next 25 will be even more exciting. Good luck and have fun!!