Skip to content

Version 3.11.1
Compare
Choose a tag to compare
@eeholmes eeholmes released this 19 Aug 18:26
· 379 commits to master since this release
v3.11.1
c1a9909

Version 3.11.1 is focused on addition of the predict, forecast, fitted and residuals functions along with plotting functions for the output. Documentation for these functions along with background literature and the derivation of the residuals algorithms have been updated. Residuals in state-space models are complex as there are two processes (observation and state), three types of conditioning (data to t-1, t or T), and four types of standardization used in the literature (none, marginal, Cholesky on the full variance matrix, and Cholesky on only model or state residual variance). The MARSS package computes all the variants of residuals. Many of the predict changes are listed below for 3.10.13 release on GitHub. New chapters illustrating structural equation models using MARSS versus StructTS and the KFAS package were added. The KFAS chapter compares the KFAS residuals functions to the MARSS residuals functions. The two packages use different algorithms and different semantics to compute the same residuals.

ENHANCEMENTS

  • ldiag() convenience function added to make list diagonal matrices. This replaces having to do code like a <- matrix(list(0),2,2); diag(a) <- list(2,"a"). Now you can call ldiag(list(2,"a")).
  • Added accurancy.marssMLE() and accuracy.marssPredict() which returns accuracy metrics sensu the forecast package.
  • Added is.unitcircle() utility function and added tol so that it does not fail if abs(eigenvalue) is above 1 by machine tolerance.
  • Added ACF plots for model and state innovation residuals to plot.marssMLE() and autoplot.marssMLE().
  • Revamped residuals.marssMLE(). Got rid of augment.marssMLE() and renamed it residuals.marssMLE(). The old residuals.marssMLE() became MARSSresiduals(). There was too much duplication between residuals.marssMLE() and augment.marssMLE() and between augment.marssMLE() and fitted.marssMLE(). Also I want to minimize dependency on other packages and augment is a class in the broom package. This required changes to the glance.marssMLE(), plot.marssMLE() and autoplot.marssMLE() code.
  • Revamped tidy.marssMLE. tsSmooth.marssMLE now returns the estimates from the Kalman filter or smoother which tidy.marssMLE had returned. tidy.marssMLE only returns a data frame for the parameter estimates.
  • V0T was computed with an inverse of Vtt1[,,1]. This led to unstable numerics when V00 was like matrix(big, m, m). Changed to use solve(t(Vtt1[,,1]), B%*%V00) which should be faster and seems to have lower numerical error.
  • predict.marssMLE updated to return ytt1, ytt, ytt1.
  • Added state innovations and contemporaneous residuals to MARSSresiduals.tt1 and MARSSresiduals.tt but not returned by residuals.marssMLE() (unless clean=FALSE). Only returned by MARSSresiduals().
  • Added Block Cholesky standardized residuals.

BUGS

  • This bug affected residuals() in cases where R=0. In v 3.10.12, I introduced a bug into MARSSkfss() for cases where R has 0s on diagonal. History: To limit propagation of numerical errors when R=0, the row/col of Vtt for the fully determined x need to be set to 0. In v 3.10.11 and earlier, my algorithm for finding these x was not robust and zero-d out Vtt row/cols when it should not have if Z was under-determined. This bug (in < 3.10.12) only affected underdetermined models (such as models with a stochastic trend and AR-1 errors). To fix I added a utility function fully.spec.x(). This returns the x that are fully determined by the data. There was a bug in these corrections which made MARSSkfss()$xtT wrong whenever there were 0s on diagonal of R. This would show up in residuals() since that was using MARSSkfss() (in order to get some output that MARSSkfas() doesn't provide.) The problem was fully.spec.x(). It did not recognize when Z.R0 (the Z for the R=0) was all 0 for an x and thus was not (could not be) fully specified by the data. Fix was simple check that colSums of Z.R0 was not all 0.
  • When computing the Cholesky standardized residuals, the lower triangle of the Cholesky decomposition should be used so that the residuals are standardized to a variance of 1. base::chol() returns the upper triangle. Thus the lines in MARSSresiduals.tT() and MARSSresiduals.tt1() that applied the standardization need t(chol()).
  • trace=1 would fail because MARSSapplynames() did not recognize that kf$xtt and kf$Innov were a message instead of a matrix. I had changed the MARSSkfas() behavior to not return these due to some questions about the values returned by the KFAS function.
  • is.validvarcov() used eigenvalues >= 0 as passing positive-definite test. Should be strictly positive so > 0.
  • MARSSkfas() had bug on the line where V0T was computed when tinitx=0. It was using * instead of %*% for the last J0 multiplication. It would affect models with a non-zero V0 under certain B matrices, such as structural models fit by StructTS().
  • The following bug was in MARSSresiduals.tT() and MARSSresiduals.tt1() but was in the old residuals.marssMLE() also. If MLE object had the kf element, kf was not assigned in code since there was no kf <- MLEobj$kf line for that case. Normally MLE objects do not have the kf element, but it could be added or is added for some settings of control$trace. This caused residuals() to fail if trace=2.

DOCUMENTATION and MAN FILES

  • Added covariates and example to MARSS_dfa.Rd
  • Added chapter on Structural time series models which compares StructTS() to MARSS() output.
  • Removed all mention of augment() from documentation and manuals. Replaced with residuals().
  • predict.marssMLE.Rd (help page) had bug in the examples. remove Q=Q from the model list in the first example.
  • Cleaned-up the man pages for predict() and predict.marssMLE().
  • In the chapter on structural breaks and outliers, Koopman et al (1998) use the marginal residuals in their example rather than the Cholesky standardized residuals. Changed to use marginal residuals to follow their example.
  • In Covariates.Rnw I show the acf of residuals. This should use innovations instead of smoothations. Only the former are temporally independent.
  • Added derivation for joint variance-covariance matrices for innovations model and state residuals.

OTHER

  • Changed fitted.marssMLE to have column with .x which is conceptually the same as the y column for observations. It is the left-side of the x equation (with error term) while fitted is the right-side without the error term.
  • Changed the x0 estimation behavior for predict.marssMLE() when no data passed in.
  • Added x0 argument to predict.marssMLE() so that user can specify x0 if needed.
  • Removed the tibble class from the data frames returned by residuals.marssMLE(). The data frames are still in tibble form. Removed all reference to tibbles in the documentation.
  • When trace = -1 some tests were still being done. I added a test for trace = -1 to a few more test lines in MARSS.R and MARSS_marxss.R.
  • Changed the default behavior of residuals.marssMLE() to return innovations instead of smoothations.
  • Columns of model estimated values in fitted.marssMLE, residuals.marssMLE, and tsSmooth.marssMLE have a leading ".".
Assets 2
Loading