SSC / CPT Assessment Comments & Recommendations

[adletaw]

C5 Crab Plan Team Report, February 2017:

BBRKC GMACS Model Update (pp. 9-11)

• "Regarding the GMACS model being slow (runs ~15 minutes w/Hessian) for BBRKC, a total of 13,440 dimensions are currently being evaluated vs. 2,520 dimensions in the 2016 model. The CPT noted that several numbers-at-length bins have no data (e.g., old shell females), but the model code is still running calculations on these empty bins, which increases run time. Some suggestions were to set up ragged matrices, or to use vectored parameters, either of which could address only the valid data bins instead of multiplying out every size, sex, and shell class."
• Create/revise the calendar for event timing
• Output likelihoods for all components by observation to look for outliers
• Look at parameterization of fishing mortality
• Work on initial year parameters
• Set all parameter values in GMACS to the BBRKC 2016 model values and compare output
• Set catch at zero and compare output to try to determine why Gmacs output is different than the BBRKC 2016 model

C7 Crab Plan Team Report, May 2017:

Gmacs BBRKC Model (pp. 10-12)

D’Arcy and Jim also discussed issues that remain to be resolved between Gmacs and the assessment model. These include being able to:

• estimate initial numbers-at-length in the same manner as is done in the assessment model (the current method does not give results similar to those from the assessment model) and
• estimate the growth transition matrix as in the assessment model (for the results presented to the CPT, the growth matrices were fixed to those estimated in the assessment).

Key issues that need to be addressed before the Gmacs model can be approved for use in the assessment include:

• processes in the current assessment model that are based on estimated parameters should also be based on estimated parameters in the Gmacs version (i.e., should not have to fix growth matrices, initial abundances to achieve similar results)
• the trajectories for recruitment and MMB predicted by the Gmacs model should be similar, but not necessarily identical, to those based on the current assessment model
• management quantities from the Gmacs model should be similar, but not necessarily identical, to those based on the current assessment

C3 Crab Plan Team Report, September 2017:

Bristol Bay red king crab (pp. 11-13)

Jie Zheng (ADFG) presented the stock assessment for Bristol Bay red king crab (BBRKC)

No updates for the Gmacs BBRKC model

2016 Crab SAFE for King and Tanner Crab Fisheries:

General Recommendations

• The team recommends that all assessment authors document assumptions and simulate data under those assumptions to test the ability of the model to estimate key parameters in an unbiased manner. These simulations would be used to demonstrate precision and bias in estimated model parameters.
• The CPT recommends that weighting factors be expressed as sigmas or CVs or effective sample sizes. The team requests all authors to follow the Guidelines for SAFE preparation and to follow the Terms of Reference as listed therein as applicable by individual assessment for both content and diagnostics.
• Authors should focus on displaying information on revised models as compared to last year’s model
  rather than focusing on aspects of the assessment that have not changed from the previous year.
• The current approach for fitting length-composition data accounts for sampling error but ignores the fact that selectivity among size classes is not constant within years; a small change in the selectivity on small animals could lead to a very large change in the catch of such animals (as may have happened for NSRKC). Authors are encouraged to develop approaches for accounting for this source of process error. This issue is generic to assessments of crab and groundfish stocks.
• Authors are reminded that assessments should include the time series of stock estimates at the time of survey for at least the author's recommended model in that year.
• Consider stepwise changes to data as individual model runs instead of changing multiple parameters at once so that changes in model performance may be attributed to specific data

SSC Minutes, October 2016:

BBRKC (pp. 7-8)

• The CPT recommended using Model 2 and the SSC concurred that this model should be used for estimation of the 2016 OFL and ABC (Model 2: the same as Model 1 in the SAFE report in September 2015 using BSFRF survey data in 2007, 2008, 2013-2016. The BSFRF survey is treated as an independent survey. The assumption that BSFRF survey capture probabilities are 1.0 for all length groups. Under this assumption, NMFS survey selectivity was the product of crab availability and NMFS survey capture probabilities.)
• The SSC recommends that the authors examine whether the current time period for estimation of
  biological reference points is indicative of the expected range of recruitment given current
  environmental conditions.

SSC Minutes, June 2017:

BBRKC (pp.13-14)

The SSC supported the CPT’s recommendations for further GMACS improvements including:

• estimating initial numbers-at-length in the same manner as is done in the assessment model; and
• estimating the growth transition matrix as in the assessment model.

These and other changes are likely necessary to achieve a close enough match between the two model results to provide a starting point for transition to GMACS for future assessments. The SSC also supports the CPT’s proposal to wait until 2018 to use the GMACS platform for the stock assessment in order to provide sufficient time to reconcile these remaining issues.

• The SSC noted that only scenarios utilizing Francis weighting methods were proposed for evaluation in the fall. As noted earlier regarding general guidance to the CPT and assessment authors, the SSC encourages stock assessment authors and the CPT to continue to consider alternative approaches, as data weighting is not a ‘one-size-fits-all’ problem. The best method for data weighting will depend on the quality of the data, the time-series length, the conflict among data sources and other factors unique to a specific assessment. Thus, the BBRKC stock assessment author should retain sufficient latitude to use a method appropriate for this particular assessment, noting that internal consistency is more important than blanket consistency across assessments dealing with a variety of unique data configurations and estimation issues. Evaluation of alternative data weighting approaches can be a useful diagnostic tool to better understand conflicts among data sources within the BBRKC assessment.
• Also, the SSC encourages the BBRKC author to objectively define the terminal year of recruitment to include in reference point calculations in this assessment. For BBRKC, where all recent recruitment years have been used in the past, dropping one or more years at the end of the time-series might be warranted. A general rule could be based on the variance of the estimated recruitments and/or the youngest ages of crabs sampled by the fishing gear and/or survey gear included in the model.