From 8f63b3c393341e3589968bd75ce59cbc6a5f1481 Mon Sep 17 00:00:00 2001 From: Geoff Pleiss <824157+gpleiss@users.noreply.github.com> Date: Wed, 2 Oct 2024 17:24:11 -0700 Subject: [PATCH] Update basis expansion slides --- .../execute-results/html.json | 5 +- .../figure-revealjs/sim-results-1.svg | 441 ++--- .../figure-revealjs/unnamed-chunk-1-1.svg | 1705 ++++++++++++----- .../figure-revealjs/unnamed-chunk-2-1.svg | 772 ++++---- .../figure-revealjs/unnamed-chunk-3-1.svg | 582 ++++++ .../figure-revealjs/unnamed-chunk-5-1.svg | 756 ++++++++ .../figure-revealjs/unnamed-chunk-6-1.svg | 547 ++++++ .../figure-revealjs/unnamed-chunk-7-1.svg | 547 ++++++ .../figure-revealjs/unnamed-chunk-8-1.svg | 444 +++++ .../site_libs/revealjs/dist/theme/quarto.css | 2 +- schedule/slides/10-basis-expansions.qmd | 196 +- 11 files changed, 4888 insertions(+), 1109 deletions(-) create mode 100644 _freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-3-1.svg create mode 100644 _freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-5-1.svg create mode 100644 _freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-6-1.svg create mode 100644 _freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-7-1.svg create mode 100644 _freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-8-1.svg diff --git a/_freeze/schedule/slides/10-basis-expansions/execute-results/html.json b/_freeze/schedule/slides/10-basis-expansions/execute-results/html.json index 969b8b1..dd4f58a 100644 --- a/_freeze/schedule/slides/10-basis-expansions/execute-results/html.json +++ b/_freeze/schedule/slides/10-basis-expansions/execute-results/html.json @@ -1,7 +1,8 @@ { - "hash": "a08eef7388ec9662d837aaf688c89fca", + "hash": "601fd57f15e5bf84bf7f2bb36c16566b", "result": { - "markdown": "---\nlecture: \"10 Basis expansions\"\nformat: revealjs\nmetadata-files: \n - _metadata.yml\n---\n---\n---\n\n## {{< meta lecture >}} {.large background-image=\"gfx/smooths.svg\" background-opacity=\"0.3\"}\n\n[Stat 406]{.secondary}\n\n[{{< meta author >}}]{.secondary}\n\nLast modified -- 27 September 2023\n\n\n\n$$\n\\DeclareMathOperator*{\\argmin}{argmin}\n\\DeclareMathOperator*{\\argmax}{argmax}\n\\DeclareMathOperator*{\\minimize}{minimize}\n\\DeclareMathOperator*{\\maximize}{maximize}\n\\DeclareMathOperator*{\\find}{find}\n\\DeclareMathOperator{\\st}{subject\\,\\,to}\n\\newcommand{\\E}{E}\n\\newcommand{\\Expect}[1]{\\E\\left[ #1 \\right]}\n\\newcommand{\\Var}[1]{\\mathrm{Var}\\left[ #1 \\right]}\n\\newcommand{\\Cov}[2]{\\mathrm{Cov}\\left[#1,\\ #2\\right]}\n\\newcommand{\\given}{\\ \\vert\\ }\n\\newcommand{\\X}{\\mathbf{X}}\n\\newcommand{\\x}{\\mathbf{x}}\n\\newcommand{\\y}{\\mathbf{y}}\n\\newcommand{\\P}{\\mathcal{P}}\n\\newcommand{\\R}{\\mathbb{R}}\n\\newcommand{\\norm}[1]{\\left\\lVert #1 \\right\\rVert}\n\\newcommand{\\snorm}[1]{\\lVert #1 \\rVert}\n\\newcommand{\\tr}[1]{\\mbox{tr}(#1)}\n\\newcommand{\\brt}{\\widehat{\\beta}^R_{s}}\n\\newcommand{\\brl}{\\widehat{\\beta}^R_{\\lambda}}\n\\newcommand{\\bls}{\\widehat{\\beta}_{ols}}\n\\newcommand{\\blt}{\\widehat{\\beta}^L_{s}}\n\\newcommand{\\bll}{\\widehat{\\beta}^L_{\\lambda}}\n$$\n\n\n\n\n\n\n## What about nonlinear things\n\n\n$$\\Expect{Y \\given X=x} = \\sum_{j=1}^p x_j\\beta_j$$\n\nNow we relax this assumption of linearity:\n\n$$\\Expect{Y \\given X=x} = f(x)$$\n\nHow do we estimate $f$?\n\n. . . \n\nFor this lecture, we use $x \\in \\R$ (1 dimensional)\n\nHigher dimensions are possible, but complexity grows [exponentially]{.secondary}.\n\nWe'll see some special techniques for $x\\in\\R^p$ later this Module.\n\n\n## Start simple\n\nFor any $f : \\R \\rightarrow [0,1]$\n\n$$f(x) = f(x_0) + f'(x_0)(x-x_0) + \\frac{1}{2}f''(x_0)(x-x_0)^2 + \\frac{1}{3!}f'''(x_0)(x-x_0)^3 + R_3(x-x_0)$$\n\nSo we can linearly regress $y_i = f(x_i)$ on the polynomials.\n\nThe more terms we use, the smaller $R$.\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nset.seed(406406)\ndata(arcuate, package = \"Stat406\") \narcuate <- arcuate |> slice_sample(n = 220)\narcuate %>% \n ggplot(aes(position, fa)) + \n geom_point(color = blue) +\n geom_smooth(color = orange, formula = y ~ poly(x, 3), method = \"lm\", se = FALSE)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-1-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Same thing, different orders\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\narcuate %>% \n ggplot(aes(position, fa)) + \n geom_point(color = blue) + \n geom_smooth(aes(color = \"a\"), formula = y ~ poly(x, 4), method = \"lm\", se = FALSE) +\n geom_smooth(aes(color = \"b\"), formula = y ~ poly(x, 7), method = \"lm\", se = FALSE) +\n geom_smooth(aes(color = \"c\"), formula = y ~ poly(x, 25), method = \"lm\", se = FALSE) +\n scale_color_manual(name = \"Taylor order\",\n values = c(green, red, orange), labels = c(\"4 terms\", \"7 terms\", \"25 terms\"))\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-2-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Still a \"linear smoother\"\n\nReally, this is still linear regression, just in a transformed space.\n\nIt's not linear in $x$, but it is linear in $(x,x^2,x^3)$ (for the 3rd-order case)\n\nSo, we're still doing OLS with\n\n$$\\X=\\begin{bmatrix}1& x_1 & x_1^2 & x_1^3 \\\\ \\vdots&&&\\vdots\\\\1& x_n & x_n^2 & x_n^3\\end{bmatrix}$$\n\nSo we can still use our nice formulas for LOO-CV, GCV, Cp, AIC, etc.\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nmax_deg <- 20\ncv_nice <- function(mdl) mean( residuals(mdl)^2 / (1 - hatvalues(mdl))^2 ) \ncvscores <- map_dbl(seq_len(max_deg), ~ cv_nice(lm(fa ~ poly(position, .), data = arcuate)))\n```\n:::\n\n\n## \n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nlibrary(cowplot)\ng1 <- ggplot(tibble(cvscores, degrees = seq(max_deg)), aes(degrees, cvscores)) +\n geom_point(colour = blue) +\n geom_line(colour = blue) + \n labs(ylab = 'LOO-CV', xlab = 'polynomial degree') +\n geom_vline(xintercept = which.min(cvscores), linetype = \"dotted\") \ng2 <- ggplot(arcuate, aes(position, fa)) + \n geom_point(colour = blue) + \n geom_smooth(\n colour = orange, \n formula = y ~ poly(x, which.min(cvscores)), \n method = \"lm\", \n se = FALSE\n )\nplot_grid(g1, g2, ncol = 2)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-4-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Other bases\n\nPolynomials\n: $x \\mapsto \\left(1,\\ x,\\ x^2, \\ldots, x^p\\right)$ (technically, not quite this, they are orthogonalized)\n\nLinear splines\n: $x \\mapsto \\bigg(1,\\ x,\\ (x-k_1)_+,\\ (x-k_2)_+,\\ldots, (x-k_p)_+\\bigg)$ for some choices $\\{k_1,\\ldots,k_p\\}$\n\nCubic splines\n: $x \\mapsto \\bigg(1,\\ x,\\ x^2,\\ x^3,\\ (x-k_1)^3_+,\\ (x-k_2)^3_+,\\ldots, (x-k_p)^3_+\\bigg)$ for some choices $\\{k_1,\\ldots,k_p\\}$\n\nFourier series\n: $x \\mapsto \\bigg(1,\\ \\cos(2\\pi x),\\ \\sin(2\\pi x),\\ \\cos(2\\pi 2 x),\\ \\sin(2\\pi 2 x), \\ldots, \\cos(2\\pi p x),\\ \\sin(2\\pi p x)\\bigg)$\n\n\n\n## How do you choose?\n\n[Procedure 1:]{.secondary}\n\n1. Pick your favorite basis. This is not as easy as it sounds. For instance, if $f$ is a step function, linear splines will do well with good knots, but polynomials will be terrible unless you have __lots__ of terms.\n\n2. Perform OLS on different orders.\n\n3. Use model selection criterion to choose the order.\n\n[Procedure 2:]{.secondary}\n\n1. Use a bunch of high-order bases, say Linear splines and Fourier series and whatever else you like.\n\n2. Use Lasso or Ridge regression or elastic net. (combining bases can lead to multicollinearity, but we may not care)\n\n3. Use model selection criteria to choose the tuning parameter.\n\n\n## Try both procedures\n\n1. Split `arcuate` into 75% training data and 25% testing data.\n\n2. Estimate polynomials up to 20 as before and choose best order.\n\n3. Do ridge, lasso and elastic net $\\alpha=.5$ on 20th order polynomials, B splines with 20 knots, and Fourier series with $p=20$. Choose tuning parameter (using `lambda.1se`).\n\n4. Repeat 1-3 10 times (different splits)\n\n\n##\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlibrary(glmnet)\nmapto01 <- function(x, pad = .005) (x - min(x) + pad) / (max(x) - min(x) + 2 * pad)\nx <- mapto01(arcuate$position)\nXmat <- cbind(\n poly(x, 20), \n splines::bs(x, df = 20), \n cos(2 * pi * outer(x, 1:20)), sin(2 * pi * outer(x, 1:20))\n)\ny <- arcuate$fa\nrmse <- function(z, s) sqrt(mean( (z - s)^2 ))\nnzero <- function(x) with(x, nzero[match(lambda.1se, lambda)])\nsim <- function(maxdeg = 20, train_frac = 0.75) {\n n <- nrow(arcuate)\n train <- as.logical(rbinom(n, 1, train_frac))\n test <- !train # not precisely 25%, but on average\n polycv <- map_dbl(seq(maxdeg), ~ cv_nice(lm(y ~ Xmat[,seq(.)], subset = train))) # figure out which order to use\n bpoly <- lm(y[train] ~ Xmat[train, seq(which.min(polycv))]) # now use it\n lasso <- cv.glmnet(Xmat[train, ], y[train])\n ridge <- cv.glmnet(Xmat[train, ], y[train], alpha = 0)\n elnet <- cv.glmnet(Xmat[train, ], y[train], alpha = .5)\n tibble(\n methods = c(\"poly\", \"lasso\", \"ridge\", \"elnet\"),\n rmses = c(\n rmse(y[test], cbind(1, Xmat[test, 1:which.min(polycv)]) %*% coef(bpoly)),\n rmse(y[test], predict(lasso, Xmat[test,])),\n rmse(y[test], predict(ridge, Xmat[test,])),\n rmse(y[test], predict(elnet, Xmat[test,]))\n ),\n nvars = c(which.min(polycv), nzero(lasso), nzero(ridge), nzero(elnet))\n )\n}\nset.seed(12345)\nsim_results <- map(seq(20), sim) |> list_rbind() # repeat it 20 times\n```\n:::\n\n\n## \n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nsim_results |> \n pivot_longer(-methods) |> \n ggplot(aes(methods, value, fill = methods)) + \n geom_boxplot() +\n facet_wrap(~ name, scales = \"free_y\") + \n ylab(\"\") +\n theme(legend.position = \"none\") + \n xlab(\"\") +\n scale_fill_viridis_d(begin = .2, end = 1)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/sim-results-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Common elements\n\nIn all these cases, we transformed $x$ to a [higher-dimensional space]{.secondary}\n\nUsed $p+1$ dimensions with polynomials\n\nUsed $p+4$ dimensions with cubic splines\n\nUsed $2p+1$ dimensions with Fourier basis\n\n## Featurization\n\nEach case applied a [feature map]{.secondary} to $x$, call it $\\Phi$\n\nWe used new \"features\" $\\Phi(x) = \\bigg(\\phi_1(x),\\ \\phi_2(x),\\ldots,\\phi_k(x)\\bigg)$\n\nNeural networks (coming in module 4) use this idea\n\nYou've also probably seen it in earlier courses when you added interaction terms or other transformations.\n\n. . .\n\nSome methods (notably Support Vector Machines and Ridge regression) allow $k=\\infty$\n\nSee [ISLR] 9.3.2 for baby overview or [ESL] 5.8 (note 😱)\n\n\n# Next time...\n\nKernel regression and nearest neighbors\n", + "engine": "knitr", + "markdown": "---\nlecture: \"10 Basis expansions\"\nformat: revealjs\nmetadata-files: \n - _metadata.yml\n---\n\n\n## {{< meta lecture >}} {.large background-image=\"gfx/smooths.svg\" background-opacity=\"0.3\"}\n\n[Stat 406]{.secondary}\n\n[{{< meta author >}}]{.secondary}\n\nLast modified -- 02 October 2024\n\n\n\n\n\n$$\n\\DeclareMathOperator*{\\argmin}{argmin}\n\\DeclareMathOperator*{\\argmax}{argmax}\n\\DeclareMathOperator*{\\minimize}{minimize}\n\\DeclareMathOperator*{\\maximize}{maximize}\n\\DeclareMathOperator*{\\find}{find}\n\\DeclareMathOperator{\\st}{subject\\,\\,to}\n\\newcommand{\\E}{E}\n\\newcommand{\\Expect}[1]{\\E\\left[ #1 \\right]}\n\\newcommand{\\Var}[1]{\\mathrm{Var}\\left[ #1 \\right]}\n\\newcommand{\\Cov}[2]{\\mathrm{Cov}\\left[#1,\\ #2\\right]}\n\\newcommand{\\given}{\\ \\vert\\ }\n\\newcommand{\\X}{\\mathbf{X}}\n\\newcommand{\\x}{\\mathbf{x}}\n\\newcommand{\\y}{\\mathbf{y}}\n\\newcommand{\\P}{\\mathcal{P}}\n\\newcommand{\\R}{\\mathbb{R}}\n\\newcommand{\\norm}[1]{\\left\\lVert #1 \\right\\rVert}\n\\newcommand{\\snorm}[1]{\\lVert #1 \\rVert}\n\\newcommand{\\tr}[1]{\\mbox{tr}(#1)}\n\\newcommand{\\brt}{\\widehat{\\beta}^R_{s}}\n\\newcommand{\\brl}{\\widehat{\\beta}^R_{\\lambda}}\n\\newcommand{\\bls}{\\widehat{\\beta}_{ols}}\n\\newcommand{\\blt}{\\widehat{\\beta}^L_{s}}\n\\newcommand{\\bll}{\\widehat{\\beta}^L_{\\lambda}}\n\\newcommand{\\U}{\\mathbf{U}}\n\\newcommand{\\D}{\\mathbf{D}}\n\\newcommand{\\V}{\\mathbf{V}}\n$$\n\n\n\n\n# Quick Review of Ridge and Lasso\n\n## OLS: low bias, (potentially) high variance\n\n$$\n\\begin{gathered}\n\\text{Model:} \\quad y = x^\\top \\beta + \\epsilon, \\qquad \\epsilon \\sim N(0, \\sigma^2)\n\\\\\n\\text{OLS:} \\quad \\bls = \\argmin_\\beta \\| \\y - \\X\\beta\\|_2^2\\quad\n\\end{gathered}\n$$\n\n- Bias: $\\E[\\bls] - \\beta = \\E[\\E[\\bls \\mid \\X]] - \\beta = \\ldots = 0$\n- variance: $\\Var{\\bls} = \\sigma^2(\\X^\\top \\X)^{-1}$\n\n[When is $(\\X^\\top \\X)^{-1}$ large?]{.secondary}\n\n. . . \n\nWhen we have *nearly colinear features*\\\nNearly colinear features $\\Rightarrow$ small singular values $\\Rightarrow$ large matrix inverse\\\n[(Colinearity is more likely when $n$ is small)]{.secondary}\n\n\n## Reducing variance (at the cost of additional bias)\n\n0. *Manual variable selection*\n1. *Ridge regression*: $\\min_\\beta \\| \\y - \\X\\beta\\|_2^2 + \\lambda \\Vert \\beta \\Vert_2^2$\n2. *Lasso*: $\\min_\\beta \\| \\y - \\X\\beta\\|_2^2 + \\lambda \\Vert \\beta \\Vert_1$\n\n. . .\n\n*Ridge* shrinks all parameters towards 0.\\\n*Lasso* performs automatic variable selection.\\\n\n. . .\n\n\\\nIncreasing $\\lambda$ *increases bias* and *decreases variance*.\\\n[(For ridge, larger lambda $\\rightarrow$ smaller $\\beta$.)]{.small}\\\n[(For lasso, larger lambda $\\rightarrow$ sparser $\\beta$.)]{.small}\n\n\n---\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-1-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n\n## Computing ridge and lasso predictors\n\n- *OLS:* $\\bls = (\\X^\\top \\X)^{-1}\\X^\\top \\y$\n- *Ridge:* $\\brl = (\\X^\\top \\X + \\lambda \\mathbf{I})^{-1}\\X^\\top \\y$\n- *Lasso:* No closed form solution 😔\n - (Convex optimization problem solvable with iterative algorithms.)\n\n\n## (Optional) Proof that ridge shrinks parameters\n\n(This proof is not too hard if you use facts about SVDs and eigenvalues. I recommend working through it.)\n\nLet $\\mathbf{UDV^\\top} = X$ be the SVD of $\\mathbf X$.\n\n$$\n\\begin{align}\n\\brl &= (\\X^\\top \\X + \\lambda \\mathbf{I})^{-1} \\X^\\top \\y\n \\\\\n &= (\\X^\\top \\X + \\lambda \\mathbf{I})^{-1} {\\color{blue} \\X \\X^\\top (\\X^\\top \\X)^{-1}} \\X^\\top \\y\n \\\\\n &= (\\X^\\top \\X + \\lambda \\mathbf{I})^{-1} \\X \\X^\\top \\underbrace{\\left( (\\X^\\top \\X)^{-1} \\X^\\top \\y \\right)}_{\\bls}\n \\\\\n &= (\\mathbf V \\mathbf D^2 \\mathbf V^\\top + \\lambda \\mathbf{I})^{-1} \\mathbf V \\mathbf D^2 \\mathbf V^\\top \\bls\n \\\\\n &= (\\mathbf V (\\mathbf D^2 + \\lambda I) \\mathbf V^\\top)^{-1} \\mathbf V \\mathbf D^2 \\mathbf V^\\top \\bls\n \\\\\n &= \\mathbf V (\\mathbf D^2)(\\mathbf D^2 + \\lambda I)^{-1} \\mathbf V^\\top \\bls\n\\end{align}\n$$\n\n---\n\n$(\\mathbf D^2)(\\mathbf D^2 + \\lambda I)^{-1}$ is a diagonal matrix with entries $d_i^2/(d_i^2 + \\lambda) < 1$.\n\nSo $\\mathbf V (\\mathbf D^2)(\\mathbf D^2 + \\lambda I)^{-1} \\mathbf V^\\top$ is a matrix that *shrinks* all coefficients of any vector multiplied against it.\n\n$$\n\\Vert \\mathbf V (\\mathbf D^2)(\\mathbf D^2 + \\lambda I)^{-1} \\mathbf V^\\top \\bls \\Vert_2 < \\Vert \\bls \\Vert_2.\n$$\nSo $\\Vert \\brl \\Vert_2 < \\Vert \\bls \\Vert_2$\n\n\n# Now onto new stuff\n(But first, more clickers!)\n\n## What about nonlinear things\n\n\n$$\\text{Our usual model:} \\quad \\Expect{Y \\given X=x} = \\sum_{j=1}^p x_j\\beta_j$$\n\nNow we relax this assumption of linearity:\n\n$$\\Expect{Y \\given X=x} = f(x)$$\n\nHow do we estimate $f$?\n\n. . . \n\nFor this lecture, we use $x \\in \\R$ (1 dimensional)\n\nHigher dimensions are possible, but complexity grows [exponentially]{.secondary}.\n\nWe'll see some special techniques for $x\\in\\R^p$ later this Module.\n\n\n## Start simple\n\nFor any $f : \\R \\rightarrow [0,1]$\n\n$$f(x) = f(x_0) + f'(x_0)(x-x_0) + \\frac{1}{2}f''(x_0)(x-x_0)^2 + \\frac{1}{3!}f'''(x_0)(x-x_0)^3 + R_3(x-x_0)$$\n\nSo we can linearly regress $y_i = f(x_i)$ on the polynomials.\n\nThe more terms we use, the smaller $R$.\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nset.seed(406406)\ndata(arcuate, package = \"Stat406\") \narcuate <- arcuate |> slice_sample(n = 220)\narcuate %>% \n ggplot(aes(position, fa)) + \n geom_point(color = blue) +\n geom_smooth(color = orange, formula = y ~ poly(x, 3), method = \"lm\", se = FALSE)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-2-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n\n## Same thing, different orders\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\narcuate %>% \n ggplot(aes(position, fa)) + \n geom_point(color = blue) + \n geom_smooth(aes(color = \"a\"), formula = y ~ poly(x, 4), method = \"lm\", se = FALSE) +\n geom_smooth(aes(color = \"b\"), formula = y ~ poly(x, 7), method = \"lm\", se = FALSE) +\n geom_smooth(aes(color = \"c\"), formula = y ~ poly(x, 25), method = \"lm\", se = FALSE) +\n scale_color_manual(name = \"Taylor order\",\n values = c(green, red, orange), labels = c(\"4 terms\", \"7 terms\", \"25 terms\"))\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-3-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n\n## Still a \"linear smoother\"\n\nReally, this is still linear regression, just in a transformed space.\n\nIt's not linear in $x$, but it is linear in $(x,x^2,x^3)$ (for the 3rd-order case)\n\nSo, we're still doing OLS with\n\n$$\\X=\\begin{bmatrix}1& x_1 & x_1^2 & x_1^3 \\\\ \\vdots&&&\\vdots\\\\1& x_n & x_n^2 & x_n^3\\end{bmatrix}$$\n\nSo we can still use our nice formulas for LOO-CV, GCV, Cp, AIC, etc.\n\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nmax_deg <- 20\ncv_nice <- function(mdl) mean( residuals(mdl)^2 / (1 - hatvalues(mdl))^2 ) \ncvscores <- map_dbl(seq_len(max_deg), ~ cv_nice(lm(fa ~ poly(position, .), data = arcuate)))\n```\n:::\n\n\n\n## \n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nlibrary(cowplot)\ng1 <- ggplot(tibble(cvscores, degrees = seq(max_deg)), aes(degrees, cvscores)) +\n geom_point(colour = blue) +\n geom_line(colour = blue) + \n labs(ylab = 'LOO-CV', xlab = 'polynomial degree') +\n geom_vline(xintercept = which.min(cvscores), linetype = \"dotted\") \ng2 <- ggplot(arcuate, aes(position, fa)) + \n geom_point(colour = blue) + \n geom_smooth(\n colour = orange, \n formula = y ~ poly(x, which.min(cvscores)), \n method = \"lm\", \n se = FALSE\n )\nplot_grid(g1, g2, ncol = 2)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-5-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n\n## Other bases\n\nPolynomials\n: $x \\mapsto \\left(1,\\ x,\\ x^2, \\ldots, x^p\\right)$ (technically, not quite this, they are orthogonalized)\n\nLinear splines\n: $x \\mapsto \\bigg(1,\\ x,\\ (x-k_1)_+,\\ (x-k_2)_+,\\ldots, (x-k_p)_+\\bigg)$ for some $\\{k_1,\\ldots,k_p\\}$\n\n\n\nFourier series\n: $x \\mapsto \\bigg(1,\\ \\cos(2\\pi x),\\ \\sin(2\\pi x),\\ \\cos(2\\pi 2 x),\\ \\sin(2\\pi 2 x), \\ldots, \\cos(2\\pi p x),\\ \\sin(2\\pi p x)\\bigg)$\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nlibrary(cowplot)\nlibrary(ggplot2)\n\nrelu_shifted <- function(x, shift) {pmax(0, x - shift)}\n\n# Create a sequence of x values\nx_vals <- seq(-3, 3, length.out = 1000)\n\n# Create a data frame with all the shifted functions\ndata <- data.frame(\n x = rep(x_vals, 5),\n polynomial = c(x_vals, x_vals^2, x_vals^3, x_vals^4, x_vals^5),\n linear.splines = c(relu_shifted(x_vals, 2), relu_shifted(x_vals, 1), relu_shifted(x_vals, 0), relu_shifted(x_vals, -1), relu_shifted(x_vals, -2)),\n fourier = c(cos(pi / 2 * x_vals), sin(pi / 2 * x_vals), cos(pi / 4 * x_vals), sin(pi / 4 * x_vals), cos(pi * x_vals)),\n function_label = rep(c(\"f1\", \"f2\", \"f3\", \"f4\", \"f5\"), each = length(x_vals))\n)\n\n# Plot using ggplot2\ng1 <- ggplot(data, aes(x = x, y = polynomial, color = function_label)) +\n geom_line(size = 1, show.legend=FALSE) +\n theme(axis.text.y=element_blank())\ng2 <- ggplot(data, aes(x = x, y = linear.splines, color = function_label)) +\n geom_line(size = 1, show.legend=FALSE) +\n theme(axis.text.y=element_blank())\ng3 <- ggplot(data, aes(x = x, y = fourier, color = function_label)) +\n geom_line(size = 1, show.legend=FALSE) +\n theme(axis.text.y=element_blank())\n\nplot_grid(g1, g2, g3, ncol = 3)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-6-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## How do you choose?\n\n[Procedure 1:]{.secondary}\n\n1. Pick your favorite basis. (Think if the data might \"prefer\" one basis over another.)\n - [How \"smooth\" is the response you're trying to model?]{.small}\n \n\n\n2. Perform OLS on different orders.\n\n3. Use model selection criterion to choose the order.\n\n---\n\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-7-1.svg){fig-align='center'}\n:::\n:::\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/unnamed-chunk-8-1.svg){fig-align='center'}\n:::\n:::\n\n\n\nWhat bases do you think will work best for $f1$, $f2$ and $f3$?\n\n. . .\n\n[*Answer: $f1$ was made from polynomial bases, $f2$ from fourier, $f3$ from linear splines*]{.secondary}\n\n\n---\n\n## How do you choose?\n\n[Procedure 2:]{.secondary}\n\n1. Use a bunch of high-order bases, say Linear splines and Fourier series and whatever else you like.\n\n2. Use Lasso or Ridge regression or elastic net. (combining bases can lead to multicollinearity, but we may not care)\n\n3. Use model selection criteria to choose the tuning parameter.\n\n\n## Try both procedures\n\n1. Split `arcuate` into 75% training data and 25% testing data.\n\n2. Estimate polynomials up to 20 as before and choose best order.\n\n3. Do ridge, lasso and elastic net $\\alpha=.5$ on 20th order polynomials, splines with 20 knots, and Fourier series with $p=20$. Choose tuning parameter (using `lambda.1se`).\n\n4. Repeat 1-3 10 times (different splits)\n\n\n##\n\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlibrary(glmnet)\nmapto01 <- function(x, pad = .005) (x - min(x) + pad) / (max(x) - min(x) + 2 * pad)\nx <- mapto01(arcuate$position)\nXmat <- cbind(\n poly(x, 20), \n splines::bs(x, df = 20, degree = 1), \n cos(2 * pi * outer(x, 1:20)), sin(2 * pi * outer(x, 1:20))\n)\ny <- arcuate$fa\nrmse <- function(z, s) sqrt(mean( (z - s)^2 ))\nnzero <- function(x) with(x, nzero[match(lambda.1se, lambda)])\nsim <- function(maxdeg = 20, train_frac = 0.75) {\n n <- nrow(arcuate)\n train <- as.logical(rbinom(n, 1, train_frac))\n test <- !train # not precisely 25%, but on average\n polycv <- map_dbl(seq(maxdeg), ~ cv_nice(lm(y ~ Xmat[,seq(.)], subset = train))) # figure out which order to use\n bpoly <- lm(y[train] ~ Xmat[train, seq(which.min(polycv))]) # now use it\n lasso <- cv.glmnet(Xmat[train, ], y[train])\n ridge <- cv.glmnet(Xmat[train, ], y[train], alpha = 0)\n elnet <- cv.glmnet(Xmat[train, ], y[train], alpha = .5)\n tibble(\n methods = c(\"poly\", \"lasso\", \"ridge\", \"elnet\"),\n rmses = c(\n rmse(y[test], cbind(1, Xmat[test, 1:which.min(polycv)]) %*% coef(bpoly)),\n rmse(y[test], predict(lasso, Xmat[test,])),\n rmse(y[test], predict(ridge, Xmat[test,])),\n rmse(y[test], predict(elnet, Xmat[test,]))\n ),\n nvars = c(which.min(polycv), nzero(lasso), nzero(ridge), nzero(elnet))\n )\n}\nset.seed(12345)\nsim_results <- map(seq(20), sim) |> list_rbind() # repeat it 20 times\n```\n:::\n\n\n\n## \n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nsim_results |> \n pivot_longer(-methods) |> \n ggplot(aes(methods, value, fill = methods)) + \n geom_boxplot() +\n facet_wrap(~ name, scales = \"free_y\") + \n ylab(\"\") +\n theme(legend.position = \"none\") + \n xlab(\"\") +\n scale_fill_viridis_d(begin = .2, end = 1)\n```\n\n::: {.cell-output-display}\n![](10-basis-expansions_files/figure-revealjs/sim-results-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n\n## Common elements\n\nIn all these cases, we transformed $x$ to a [higher-dimensional space]{.secondary}\n\nUsed $p+1$ dimensions with polynomials\n\nUsed $p+4$ dimensions with cubic splines\n\nUsed $2p+1$ dimensions with Fourier basis\n\n## Featurization\n\nEach case applied a [feature map]{.secondary} to $x$, call it $\\Phi$\n\nWe used new \"features\" $\\Phi(x) = \\bigg(\\phi_1(x),\\ \\phi_2(x),\\ldots,\\phi_k(x)\\bigg)$\nw/ a linear model\n\n$$f(x) = \\Phi(x)^\\top \\beta$$ \n\nNeural networks (coming in module 4) build upon this idea\n\n\n\n. . .\n\n\\\nSome methods (notably Support Vector Machines and other Kernel Machines) allow $k=\\infty$\\\n[See [ISLR] 9.3.2 for baby overview or [ESL] 5.8 (note 😱)]{.small}\n\n\n# Next time...\n\nKernel regression and nearest neighbors\n", "supporting": [ "10-basis-expansions_files" ], diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/sim-results-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/sim-results-1.svg index d832284..ab9fcc3 100644 --- a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/sim-results-1.svg +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/sim-results-1.svg @@ -3,417 +3,420 @@ - + - + - + - + - + - + - + - + - + - + - + - - - - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - + + + + - - - - - + + + + + - - + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - - - - - + + + + + - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - - - - - + + + + + - - - - + + + + - - - - - + + + + + - - - - + + + + - - - - + + + + - - - - - - - - + + + + + + + + - + - - + + - - + + - - + + - - - - - - - - + + + + + + + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-1-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-1-1.svg index b9bd2f5..4bad50c 100644 --- a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-1-1.svg +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-1-1.svg @@ -1,500 +1,1329 @@ - + - + - + - + - + - + - + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-2-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-2-1.svg index eccf150..e8c7b02 100644 --- a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-2-1.svg +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-2-1.svg @@ -3,595 +3,489 @@ - - - - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - + - - - - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - + - - + + - - - + + + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - + + + + + + + + - - - - - - - - + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-3-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-3-1.svg new file mode 100644 index 0000000..eb61a8c --- /dev/null +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-3-1.svg @@ -0,0 +1,582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-5-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-5-1.svg new file mode 100644 index 0000000..4996b17 --- /dev/null +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-5-1.svg @@ -0,0 +1,756 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-6-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-6-1.svg new file mode 100644 index 0000000..0fd2174 --- /dev/null +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-6-1.svg @@ -0,0 +1,547 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-7-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-7-1.svg new file mode 100644 index 0000000..eb66aa1 --- /dev/null +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-7-1.svg @@ -0,0 +1,547 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-8-1.svg b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-8-1.svg new file mode 100644 index 0000000..9ce7c88 --- /dev/null +++ b/_freeze/schedule/slides/10-basis-expansions/figure-revealjs/unnamed-chunk-8-1.svg @@ -0,0 +1,444 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/site_libs/revealjs/dist/theme/quarto.css b/_freeze/site_libs/revealjs/dist/theme/quarto.css index 62c5b6b..2be515e 100644 --- a/_freeze/site_libs/revealjs/dist/theme/quarto.css +++ b/_freeze/site_libs/revealjs/dist/theme/quarto.css @@ -5,4 +5,4 @@ * we also add `bright-[color]-` synonyms for the `-[color]-intense` classes since * that seems to be what ansi_up emits * -*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-black,.ansi-bright-black-fg{color:#282c36}.ansi-black-intense-black,.ansi-bright-black-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-red,.ansi-bright-red-fg{color:#b22b31}.ansi-red-intense-red,.ansi-bright-red-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-green,.ansi-bright-green-fg{color:#007427}.ansi-green-intense-green,.ansi-bright-green-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-yellow,.ansi-bright-yellow-fg{color:#b27d12}.ansi-yellow-intense-yellow,.ansi-bright-yellow-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-blue,.ansi-bright-blue-fg{color:#0065ca}.ansi-blue-intense-blue,.ansi-bright-blue-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-magenta,.ansi-bright-magenta-fg{color:#a03196}.ansi-magenta-intense-magenta,.ansi-bright-magenta-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-cyan,.ansi-bright-cyan-fg{color:#258f8f}.ansi-cyan-intense-cyan,.ansi-bright-cyan-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-white,.ansi-bright-white-fg{color:#a1a6b2}.ansi-white-intense-white,.ansi-bright-white-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #fefefe;--quarto-body-color: #222;--quarto-text-muted: #6f6f6f;--quarto-border-color: #bbbbbb;--quarto-border-width: 1px;--quarto-border-radius: 4px}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:relative;float:right;background-color:rgba(0,0,0,0)}input[type=checkbox]{margin-right:.5ch}:root{--mermaid-bg-color: #fefefe;--mermaid-edge-color: #e98a15;--mermaid-node-fg-color: #222;--mermaid-fg-color: #222;--mermaid-fg-color--lighter: #3c3c3c;--mermaid-fg-color--lightest: #555555;--mermaid-font-family: Commissioner, Source Sans Pro, Helvetica, sans-serif;--mermaid-label-bg-color: #fefefe;--mermaid-label-fg-color: #2c365e;--mermaid-node-bg-color: rgba(44, 54, 94, 0.1);--mermaid-node-fg-color: #222}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}pre.sourceCode:hover>.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}pre.sourceCode:hover>.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}pre.sourceCode:hover>.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}pre.sourceCode:hover>.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}.panel-tabset [role=tablist]{border-bottom:1px solid #bbb;list-style:none;margin:0;padding:0;width:100%}.panel-tabset [role=tablist] *{-webkit-box-sizing:border-box;box-sizing:border-box}@media(min-width: 30em){.panel-tabset [role=tablist] li{display:inline-block}}.panel-tabset [role=tab]{border:1px solid rgba(0,0,0,0);border-top-color:#bbb;display:block;padding:.5em 1em;text-decoration:none}@media(min-width: 30em){.panel-tabset [role=tab]{border-top-color:rgba(0,0,0,0);display:inline-block;margin-bottom:-1px}}.panel-tabset [role=tab][aria-selected=true]{background-color:#bbb}@media(min-width: 30em){.panel-tabset [role=tab][aria-selected=true]{background-color:rgba(0,0,0,0);border:1px solid #bbb;border-bottom-color:#fefefe}}@media(min-width: 30em){.panel-tabset [role=tab]:hover:not([aria-selected=true]){border:1px solid #bbb}}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}a.external:after{content:"";background-image:url('data:image/svg+xml,');background-size:contain;background-repeat:no-repeat;background-position:center center;margin-left:.2em;padding-right:.75em}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.reveal.center .slide aside,.reveal.center .slide div.aside{position:initial}section.has-light-background,section.has-light-background h1,section.has-light-background h2,section.has-light-background h3,section.has-light-background h4,section.has-light-background h5,section.has-light-background h6{color:#222}section.has-light-background a,section.has-light-background a:hover{color:#2a76dd}section.has-light-background code{color:#4758ab}section.has-dark-background,section.has-dark-background h1,section.has-dark-background h2,section.has-dark-background h3,section.has-dark-background h4,section.has-dark-background h5,section.has-dark-background h6{color:#fff}section.has-dark-background a,section.has-dark-background a:hover{color:#42affa}section.has-dark-background code{color:#ffa07a}#title-slide,div.reveal div.slides section.quarto-title-block{text-align:center}#title-slide .subtitle,div.reveal div.slides section.quarto-title-block .subtitle{margin-bottom:2.5rem}.reveal .slides{text-align:left}.reveal .title-slide h1{font-size:1.6em}.reveal[data-navigation-mode=linear] .title-slide h1{font-size:2.5em}.reveal div.sourceCode{border:1px solid #bbb;border-radius:4px}.reveal pre{width:100%;box-shadow:none;background-color:#fefefe;border:none;margin:0;font-size:.55em}.reveal code{color:var(--quarto-hl-fu-color);background-color:rgba(0,0,0,0);white-space:pre-wrap}.reveal pre.sourceCode code{background-color:#fefefe;padding:6px 9px;max-height:500px;white-space:pre}.reveal pre code{background-color:#fefefe;color:#222}.reveal .column-output-location{display:flex;align-items:stretch}.reveal .column-output-location .column:first-of-type div.sourceCode{height:100%;background-color:#fefefe}.reveal blockquote{display:block;position:relative;color:#6f6f6f;width:unset;margin:var(--r-block-margin) auto;padding:.625rem 1.75rem;border-left:.25rem solid #6f6f6f;font-style:normal;background:none;box-shadow:none}.reveal blockquote p:first-child,.reveal blockquote p:last-child{display:block}.reveal .slide aside,.reveal .slide div.aside{position:absolute;bottom:20px;font-size:0.7em;color:#6f6f6f}.reveal .slide sup{font-size:0.7em}.reveal .slide.scrollable aside,.reveal .slide.scrollable div.aside{position:relative;margin-top:1em}.reveal .slide aside .aside-footnotes{margin-bottom:0}.reveal .slide aside .aside-footnotes li:first-of-type{margin-top:0}.reveal .layout-sidebar{display:flex;width:100%;margin-top:.8em}.reveal .layout-sidebar .panel-sidebar{width:270px}.reveal .layout-sidebar-left .panel-sidebar{margin-right:calc(0.5em*2)}.reveal .layout-sidebar-right .panel-sidebar{margin-left:calc(0.5em*2)}.reveal .layout-sidebar .panel-fill,.reveal .layout-sidebar .panel-center,.reveal .layout-sidebar .panel-tabset{flex:1}.reveal .panel-input,.reveal .panel-sidebar{font-size:.5em;padding:.5em;border-style:solid;border-color:#bbb;border-width:1px;border-radius:4px;background-color:#f8f9fa}.reveal .panel-sidebar :first-child,.reveal .panel-fill :first-child{margin-top:0}.reveal .panel-sidebar :last-child,.reveal .panel-fill :last-child{margin-bottom:0}.panel-input>div,.panel-input>div>div{vertical-align:middle;padding-right:1em}.reveal p,.reveal .slides section,.reveal .slides section>section{line-height:1.3}.reveal.smaller .slides section,.reveal .slides section.smaller,.reveal .slides section .callout{font-size:0.7em}.reveal.smaller .slides section section{font-size:inherit}.reveal.smaller .slides h1,.reveal .slides section.smaller h1{font-size:calc(2.5em/0.7)}.reveal.smaller .slides h2,.reveal .slides section.smaller h2{font-size:calc(1.6em/0.7)}.reveal.smaller .slides h3,.reveal .slides section.smaller h3{font-size:calc(1.3em/0.7)}.reveal .columns>.column>:not(ul,ol){margin-left:.25em;margin-right:.25em}.reveal .columns>.column:first-child>:not(ul,ol){margin-right:.5em;margin-left:0}.reveal .columns>.column:last-child>:not(ul,ol){margin-right:0;margin-left:.5em}.reveal .slide-number{color:#eea143;background-color:#fefefe}.reveal .footer{color:#6f6f6f}.reveal .footer a{color:#e98a15}.reveal .footer.has-dark-background{color:#fff}.reveal .footer.has-dark-background a{color:#7bc6fa}.reveal .footer.has-light-background{color:#505050}.reveal .footer.has-light-background a{color:#6a9bdd}.reveal .slide-number{color:#6f6f6f}.reveal .slide-number.has-dark-background{color:#fff}.reveal .slide-number.has-light-background{color:#505050}.reveal .slide figure>figcaption,.reveal .slide img.stretch+p.caption,.reveal .slide img.r-stretch+p.caption{font-size:0.7em}@media screen and (min-width: 500px){.reveal .controls[data-controls-layout=edges] .navigate-left{left:.2em}.reveal .controls[data-controls-layout=edges] .navigate-right{right:.2em}.reveal .controls[data-controls-layout=edges] .navigate-up{top:.4em}.reveal .controls[data-controls-layout=edges] .navigate-down{bottom:2.3em}}.tippy-box[data-theme~=light-border]{background-color:#fefefe;color:#222;border-radius:4px;border:solid 1px #6f6f6f;font-size:.6em}.tippy-box[data-theme~=light-border] .tippy-arrow{color:#6f6f6f}.tippy-box[data-placement^=bottom]>.tippy-content{padding:7px 10px;z-index:1}.reveal .callout.callout-style-simple .callout-body,.reveal .callout.callout-style-default .callout-body,.reveal .callout.callout-style-simple div.callout-title,.reveal .callout.callout-style-default div.callout-title{font-size:inherit}.reveal .callout.callout-style-default .callout-icon::before,.reveal .callout.callout-style-simple .callout-icon::before{height:2rem;width:2rem;background-size:2rem 2rem}.reveal .callout.callout-titled .callout-title p{margin-top:.5em}.reveal .callout.callout-titled .callout-icon::before{margin-top:1rem}.reveal .callout.callout-titled .callout-body>.callout-content>:last-child{margin-bottom:1rem}.reveal .panel-tabset [role=tab]{padding:.25em .7em}.reveal .slide-menu-button .fa-bars::before{background-image:url('data:image/svg+xml,')}.reveal .slide-chalkboard-buttons .fa-easel2::before{background-image:url('data:image/svg+xml,')}.reveal .slide-chalkboard-buttons .fa-brush::before{background-image:url('data:image/svg+xml,')}/*! light */.reveal ol[type=a]{list-style-type:lower-alpha}.reveal ol[type=a s]{list-style-type:lower-alpha}.reveal ol[type=A s]{list-style-type:upper-alpha}.reveal ol[type=i]{list-style-type:lower-roman}.reveal ol[type=i s]{list-style-type:lower-roman}.reveal ol[type=I s]{list-style-type:upper-roman}.reveal ol[type="1"]{list-style-type:decimal}.reveal ul.task-list{list-style:none}.reveal ul.task-list li input[type=checkbox]{width:2em;height:2em;margin:0 1em .5em -1.6em;vertical-align:middle}div.cell-output-display div.pagedtable-wrapper table.table{font-size:.6em}.reveal .code-annotation-container-hidden{display:none}.reveal code.sourceCode button.code-annotation-anchor,.reveal code.sourceCode .code-annotation-anchor{font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;font-size:.7em;line-height:1.2em;margin-top:2px;user-select:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;-o-user-select:none}.reveal code.sourceCode button.code-annotation-anchor{cursor:pointer}.reveal code.sourceCode a.code-annotation-anchor{text-align:center;vertical-align:middle;text-decoration:none;cursor:default;height:1.2em;width:1.2em}.reveal code.sourceCode.fragment a.code-annotation-anchor{left:auto}.reveal #code-annotation-line-highlight-gutter{width:100%;border-top:solid var(--quarto-hl-co-color) 1px;border-bottom:solid var(--quarto-hl-co-color) 1px;z-index:2}.reveal #code-annotation-line-highlight{margin-left:-8em;width:calc(100% + 4em);border-top:solid var(--quarto-hl-co-color) 1px;border-bottom:solid var(--quarto-hl-co-color) 1px;z-index:2;margin-bottom:-2px}.reveal code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#fefefe;font-weight:bolder}.reveal pre.code-annotation-code{padding-top:0;padding-bottom:0}.reveal pre.code-annotation-code code{z-index:3;padding-left:0px}.reveal dl.code-annotation-container-grid{margin-left:.1em}.reveal dl.code-annotation-container-grid dt{margin-top:.65rem;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;border:solid #222 1px;border-radius:50%;height:1.3em;width:1.3em;line-height:1.3em;font-size:.5em;text-align:center;vertical-align:middle;text-decoration:none}.reveal dl.code-annotation-container-grid dd{margin-left:.25em}.reveal .scrollable ol li:first-child:nth-last-child(n+10),.reveal .scrollable ol li:first-child:nth-last-child(n+10)~li{margin-left:1em}html.print-pdf .reveal .slides .pdf-page:last-child{page-break-after:avoid}.reveal .quarto-title-block .quarto-title-authors{display:flex;justify-content:center}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author{padding-left:.5em;padding-right:.5em}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:hover,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:visited,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:active{color:inherit;text-decoration:none}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-name{margin-bottom:.1rem}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-email{margin-top:0px;margin-bottom:.4em;font-size:.6em}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-orcid img{margin-bottom:4px}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-affiliation{font-size:.7em;margin-top:0px;margin-bottom:8px}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-affiliation:first{margin-top:12px}ol{padding-left:.5em}ul{list-style:none}ul li::marker{color:#2c365e}dt{color:#e98a15}#title-slide{text-align:left}#title-slide .title{color:#2c365e}#title-slide .author{text-align:left;color:#e98a15;font-weight:bold}#title-slide .institute{padding-bottom:200px}.small{font-size:.75em}.smallest{font-size:.5em}.large{font-size:1.5em}.medium{font-size:.9em}.center-align{text-align:center}.hand{font-family:"Gochi Hand",cursive;font-size:125%}.hand-blue{font-family:"Gochi Hand",cursive;color:var(--primary);font-size:125%}/*# sourceMappingURL=f95d2bded9c28492b788fe14c3e9f347.css.map */ +*/.ansi-black-fg{color:#3e424d}.ansi-black-bg{background-color:#3e424d}.ansi-black-intense-black,.ansi-bright-black-fg{color:#282c36}.ansi-black-intense-black,.ansi-bright-black-bg{background-color:#282c36}.ansi-red-fg{color:#e75c58}.ansi-red-bg{background-color:#e75c58}.ansi-red-intense-red,.ansi-bright-red-fg{color:#b22b31}.ansi-red-intense-red,.ansi-bright-red-bg{background-color:#b22b31}.ansi-green-fg{color:#00a250}.ansi-green-bg{background-color:#00a250}.ansi-green-intense-green,.ansi-bright-green-fg{color:#007427}.ansi-green-intense-green,.ansi-bright-green-bg{background-color:#007427}.ansi-yellow-fg{color:#ddb62b}.ansi-yellow-bg{background-color:#ddb62b}.ansi-yellow-intense-yellow,.ansi-bright-yellow-fg{color:#b27d12}.ansi-yellow-intense-yellow,.ansi-bright-yellow-bg{background-color:#b27d12}.ansi-blue-fg{color:#208ffb}.ansi-blue-bg{background-color:#208ffb}.ansi-blue-intense-blue,.ansi-bright-blue-fg{color:#0065ca}.ansi-blue-intense-blue,.ansi-bright-blue-bg{background-color:#0065ca}.ansi-magenta-fg{color:#d160c4}.ansi-magenta-bg{background-color:#d160c4}.ansi-magenta-intense-magenta,.ansi-bright-magenta-fg{color:#a03196}.ansi-magenta-intense-magenta,.ansi-bright-magenta-bg{background-color:#a03196}.ansi-cyan-fg{color:#60c6c8}.ansi-cyan-bg{background-color:#60c6c8}.ansi-cyan-intense-cyan,.ansi-bright-cyan-fg{color:#258f8f}.ansi-cyan-intense-cyan,.ansi-bright-cyan-bg{background-color:#258f8f}.ansi-white-fg{color:#c5c1b4}.ansi-white-bg{background-color:#c5c1b4}.ansi-white-intense-white,.ansi-bright-white-fg{color:#a1a6b2}.ansi-white-intense-white,.ansi-bright-white-bg{background-color:#a1a6b2}.ansi-default-inverse-fg{color:#fff}.ansi-default-inverse-bg{background-color:#000}.ansi-bold{font-weight:bold}.ansi-underline{text-decoration:underline}:root{--quarto-body-bg: #fefefe;--quarto-body-color: #222;--quarto-text-muted: #6f6f6f;--quarto-border-color: #bbbbbb;--quarto-border-width: 1px;--quarto-border-radius: 4px}table.gt_table{color:var(--quarto-body-color);font-size:1em;width:100%;background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_column_spanner_outer{color:var(--quarto-body-color);background-color:rgba(0,0,0,0);border-top-width:inherit;border-bottom-width:inherit;border-color:var(--quarto-border-color)}table.gt_table th.gt_col_heading{color:var(--quarto-body-color);font-weight:bold;background-color:rgba(0,0,0,0)}table.gt_table thead.gt_col_headings{border-bottom:1px solid currentColor;border-top-width:inherit;border-top-color:var(--quarto-border-color)}table.gt_table thead.gt_col_headings:not(:first-child){border-top-width:1px;border-top-color:var(--quarto-border-color)}table.gt_table td.gt_row{border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-width:0px}table.gt_table tbody.gt_table_body{border-top-width:1px;border-bottom-width:1px;border-bottom-color:var(--quarto-border-color);border-top-color:currentColor}div.columns{display:initial;gap:initial}div.column{display:inline-block;overflow-x:initial;vertical-align:top;width:50%}.code-annotation-tip-content{word-wrap:break-word}.code-annotation-container-hidden{display:none !important}dl.code-annotation-container-grid{display:grid;grid-template-columns:min-content auto}dl.code-annotation-container-grid dt{grid-column:1}dl.code-annotation-container-grid dd{grid-column:2}pre.sourceCode.code-annotation-code{padding-right:0}code.sourceCode .code-annotation-anchor{z-index:100;position:relative;float:right;background-color:rgba(0,0,0,0)}input[type=checkbox]{margin-right:.5ch}:root{--mermaid-bg-color: #fefefe;--mermaid-edge-color: #e98a15;--mermaid-node-fg-color: #222;--mermaid-fg-color: #222;--mermaid-fg-color--lighter: #3c3c3c;--mermaid-fg-color--lightest: #555555;--mermaid-font-family: Commissioner, Source Sans Pro, Helvetica, sans-serif;--mermaid-label-bg-color: #fefefe;--mermaid-label-fg-color: #2c365e;--mermaid-node-bg-color: rgba(44, 54, 94, 0.1);--mermaid-node-fg-color: #222}@media print{:root{font-size:11pt}#quarto-sidebar,#TOC,.nav-page{display:none}.page-columns .content{grid-column-start:page-start}.fixed-top{position:relative}.panel-caption,.figure-caption,figcaption{color:#666}}.code-copy-button{position:absolute;top:0;right:0;border:0;margin-top:5px;margin-right:5px;background-color:rgba(0,0,0,0);z-index:3}.code-copy-button:focus{outline:none}.code-copy-button-tooltip{font-size:.75em}pre.sourceCode:hover>.code-copy-button>.bi::before{display:inline-block;height:1rem;width:1rem;content:"";vertical-align:-0.125em;background-image:url('data:image/svg+xml,');background-repeat:no-repeat;background-size:1rem 1rem}pre.sourceCode:hover>.code-copy-button-checked>.bi::before{background-image:url('data:image/svg+xml,')}pre.sourceCode:hover>.code-copy-button:hover>.bi::before{background-image:url('data:image/svg+xml,')}pre.sourceCode:hover>.code-copy-button-checked:hover>.bi::before{background-image:url('data:image/svg+xml,')}.panel-tabset [role=tablist]{border-bottom:1px solid #bbb;list-style:none;margin:0;padding:0;width:100%}.panel-tabset [role=tablist] *{-webkit-box-sizing:border-box;box-sizing:border-box}@media(min-width: 30em){.panel-tabset [role=tablist] li{display:inline-block}}.panel-tabset [role=tab]{border:1px solid rgba(0,0,0,0);border-top-color:#bbb;display:block;padding:.5em 1em;text-decoration:none}@media(min-width: 30em){.panel-tabset [role=tab]{border-top-color:rgba(0,0,0,0);display:inline-block;margin-bottom:-1px}}.panel-tabset [role=tab][aria-selected=true]{background-color:#bbb}@media(min-width: 30em){.panel-tabset [role=tab][aria-selected=true]{background-color:rgba(0,0,0,0);border:1px solid #bbb;border-bottom-color:#fefefe}}@media(min-width: 30em){.panel-tabset [role=tab]:hover:not([aria-selected=true]){border:1px solid #bbb}}.code-with-filename .code-with-filename-file{margin-bottom:0;padding-bottom:2px;padding-top:2px;padding-left:.7em;border:var(--quarto-border-width) solid var(--quarto-border-color);border-radius:var(--quarto-border-radius);border-bottom:0;border-bottom-left-radius:0%;border-bottom-right-radius:0%}.code-with-filename div.sourceCode,.reveal .code-with-filename div.sourceCode{margin-top:0;border-top-left-radius:0%;border-top-right-radius:0%}.code-with-filename .code-with-filename-file pre{margin-bottom:0}.code-with-filename .code-with-filename-file{background-color:rgba(219,219,219,.8)}.quarto-dark .code-with-filename .code-with-filename-file{background-color:#555}.code-with-filename .code-with-filename-file strong{font-weight:400}a.external:after{content:"";background-image:url('data:image/svg+xml,');background-size:contain;background-repeat:no-repeat;background-position:center center;margin-left:.2em;padding-right:.75em}div.sourceCode code a.external:after{content:none}a.external:after:hover{cursor:pointer}.quarto-ext-icon{display:inline-block;font-size:.75em;padding-left:.3em}.reveal.center .slide aside,.reveal.center .slide div.aside{position:initial}section.has-light-background,section.has-light-background h1,section.has-light-background h2,section.has-light-background h3,section.has-light-background h4,section.has-light-background h5,section.has-light-background h6{color:#222}section.has-light-background a,section.has-light-background a:hover{color:#2a76dd}section.has-light-background code{color:#4758ab}section.has-dark-background,section.has-dark-background h1,section.has-dark-background h2,section.has-dark-background h3,section.has-dark-background h4,section.has-dark-background h5,section.has-dark-background h6{color:#fff}section.has-dark-background a,section.has-dark-background a:hover{color:#42affa}section.has-dark-background code{color:#ffa07a}#title-slide,div.reveal div.slides section.quarto-title-block{text-align:center}#title-slide .subtitle,div.reveal div.slides section.quarto-title-block .subtitle{margin-bottom:2.5rem}.reveal .slides{text-align:left}.reveal .title-slide h1{font-size:1.6em}.reveal[data-navigation-mode=linear] .title-slide h1{font-size:2.5em}.reveal div.sourceCode{border:1px solid #bbb;border-radius:4px}.reveal pre{width:100%;box-shadow:none;background-color:#fefefe;border:none;margin:0;font-size:.55em}.reveal .code-with-filename .code-with-filename-file pre{background-color:unset}.reveal code{color:var(--quarto-hl-fu-color);background-color:rgba(0,0,0,0);white-space:pre-wrap}.reveal pre.sourceCode code{background-color:#fefefe;padding:6px 9px;max-height:500px;white-space:pre}.reveal pre code{background-color:#fefefe;color:#222}.reveal .column-output-location{display:flex;align-items:stretch}.reveal .column-output-location .column:first-of-type div.sourceCode{height:100%;background-color:#fefefe}.reveal blockquote{display:block;position:relative;color:#6f6f6f;width:unset;margin:var(--r-block-margin) auto;padding:.625rem 1.75rem;border-left:.25rem solid #6f6f6f;font-style:normal;background:none;box-shadow:none}.reveal blockquote p:first-child,.reveal blockquote p:last-child{display:block}.reveal .slide aside,.reveal .slide div.aside{position:absolute;bottom:20px;font-size:0.7em;color:#6f6f6f}.reveal .slide sup{font-size:0.7em}.reveal .slide.scrollable aside,.reveal .slide.scrollable div.aside{position:relative;margin-top:1em}.reveal .slide aside .aside-footnotes{margin-bottom:0}.reveal .slide aside .aside-footnotes li:first-of-type{margin-top:0}.reveal .layout-sidebar{display:flex;width:100%;margin-top:.8em}.reveal .layout-sidebar .panel-sidebar{width:270px}.reveal .layout-sidebar-left .panel-sidebar{margin-right:calc(0.5em*2)}.reveal .layout-sidebar-right .panel-sidebar{margin-left:calc(0.5em*2)}.reveal .layout-sidebar .panel-fill,.reveal .layout-sidebar .panel-center,.reveal .layout-sidebar .panel-tabset{flex:1}.reveal .panel-input,.reveal .panel-sidebar{font-size:.5em;padding:.5em;border-style:solid;border-color:#bbb;border-width:1px;border-radius:4px;background-color:#f8f9fa}.reveal .panel-sidebar :first-child,.reveal .panel-fill :first-child{margin-top:0}.reveal .panel-sidebar :last-child,.reveal .panel-fill :last-child{margin-bottom:0}.panel-input>div,.panel-input>div>div{vertical-align:middle;padding-right:1em}.reveal p,.reveal .slides section,.reveal .slides section>section{line-height:1.3}.reveal.smaller .slides section,.reveal .slides section.smaller,.reveal .slides section .callout{font-size:0.7em}.reveal.smaller .slides section section{font-size:inherit}.reveal.smaller .slides h1,.reveal .slides section.smaller h1{font-size:calc(2.5em/0.7)}.reveal.smaller .slides h2,.reveal .slides section.smaller h2{font-size:calc(1.6em/0.7)}.reveal.smaller .slides h3,.reveal .slides section.smaller h3{font-size:calc(1.3em/0.7)}.reveal .columns>.column>:not(ul,ol){margin-left:.25em;margin-right:.25em}.reveal .columns>.column:first-child>:not(ul,ol){margin-right:.5em;margin-left:0}.reveal .columns>.column:last-child>:not(ul,ol){margin-right:0;margin-left:.5em}.reveal .slide-number{color:#eea143;background-color:#fefefe}.reveal .footer{color:#6f6f6f}.reveal .footer a{color:#e98a15}.reveal .footer.has-dark-background{color:#fff}.reveal .footer.has-dark-background a{color:#7bc6fa}.reveal .footer.has-light-background{color:#505050}.reveal .footer.has-light-background a{color:#6a9bdd}.reveal .slide-number{color:#6f6f6f}.reveal .slide-number.has-dark-background{color:#fff}.reveal .slide-number.has-light-background{color:#505050}.reveal .slide figure>figcaption,.reveal .slide img.stretch+p.caption,.reveal .slide img.r-stretch+p.caption{font-size:0.7em}@media screen and (min-width: 500px){.reveal .controls[data-controls-layout=edges] .navigate-left{left:.2em}.reveal .controls[data-controls-layout=edges] .navigate-right{right:.2em}.reveal .controls[data-controls-layout=edges] .navigate-up{top:.4em}.reveal .controls[data-controls-layout=edges] .navigate-down{bottom:2.3em}}.tippy-box[data-theme~=light-border]{background-color:#fefefe;color:#222;border-radius:4px;border:solid 1px #6f6f6f;font-size:.6em}.tippy-box[data-theme~=light-border] .tippy-arrow{color:#6f6f6f}.tippy-box[data-placement^=bottom]>.tippy-content{padding:7px 10px;z-index:1}.reveal .callout.callout-style-simple .callout-body,.reveal .callout.callout-style-default .callout-body,.reveal .callout.callout-style-simple div.callout-title,.reveal .callout.callout-style-default div.callout-title{font-size:inherit}.reveal .callout.callout-style-default .callout-icon::before,.reveal .callout.callout-style-simple .callout-icon::before{height:2rem;width:2rem;background-size:2rem 2rem}.reveal .callout.callout-titled .callout-title p{margin-top:.5em}.reveal .callout.callout-titled .callout-icon::before{margin-top:1rem}.reveal .callout.callout-titled .callout-body>.callout-content>:last-child{margin-bottom:1rem}.reveal .panel-tabset [role=tab]{padding:.25em .7em}.reveal .slide-menu-button .fa-bars::before{background-image:url('data:image/svg+xml,')}.reveal .slide-chalkboard-buttons .fa-easel2::before{background-image:url('data:image/svg+xml,')}.reveal .slide-chalkboard-buttons .fa-brush::before{background-image:url('data:image/svg+xml,')}/*! light */.reveal ol[type=a]{list-style-type:lower-alpha}.reveal ol[type=a s]{list-style-type:lower-alpha}.reveal ol[type=A s]{list-style-type:upper-alpha}.reveal ol[type=i]{list-style-type:lower-roman}.reveal ol[type=i s]{list-style-type:lower-roman}.reveal ol[type=I s]{list-style-type:upper-roman}.reveal ol[type="1"]{list-style-type:decimal}.reveal ul.task-list{list-style:none}.reveal ul.task-list li input[type=checkbox]{width:2em;height:2em;margin:0 1em .5em -1.6em;vertical-align:middle}div.cell-output-display div.pagedtable-wrapper table.table{font-size:.6em}.reveal .code-annotation-container-hidden{display:none}.reveal code.sourceCode button.code-annotation-anchor,.reveal code.sourceCode .code-annotation-anchor{font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;color:var(--quarto-hl-co-color);border:solid var(--quarto-hl-co-color) 1px;border-radius:50%;font-size:.7em;line-height:1.2em;margin-top:2px;user-select:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;-o-user-select:none}.reveal code.sourceCode button.code-annotation-anchor{cursor:pointer}.reveal code.sourceCode a.code-annotation-anchor{text-align:center;vertical-align:middle;text-decoration:none;cursor:default;height:1.2em;width:1.2em}.reveal code.sourceCode.fragment a.code-annotation-anchor{left:auto}.reveal #code-annotation-line-highlight-gutter{width:100%;border-top:solid var(--quarto-hl-co-color) 1px;border-bottom:solid var(--quarto-hl-co-color) 1px;z-index:2}.reveal #code-annotation-line-highlight{margin-left:-8em;width:calc(100% + 4em);border-top:solid var(--quarto-hl-co-color) 1px;border-bottom:solid var(--quarto-hl-co-color) 1px;z-index:2;margin-bottom:-2px}.reveal code.sourceCode .code-annotation-anchor.code-annotation-active{background-color:var(--quarto-hl-normal-color, #aaaaaa);border:solid var(--quarto-hl-normal-color, #aaaaaa) 1px;color:#fefefe;font-weight:bolder}.reveal pre.code-annotation-code{padding-top:0;padding-bottom:0}.reveal pre.code-annotation-code code{z-index:3;padding-left:0px}.reveal dl.code-annotation-container-grid{margin-left:.1em}.reveal dl.code-annotation-container-grid dt{margin-top:.65rem;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;border:solid #222 1px;border-radius:50%;height:1.3em;width:1.3em;line-height:1.3em;font-size:.5em;text-align:center;vertical-align:middle;text-decoration:none}.reveal dl.code-annotation-container-grid dd{margin-left:.25em}.reveal .scrollable ol li:first-child:nth-last-child(n+10),.reveal .scrollable ol li:first-child:nth-last-child(n+10)~li{margin-left:1em}html.print-pdf .reveal .slides .pdf-page:last-child{page-break-after:avoid}.reveal .quarto-title-block .quarto-title-authors{display:flex;justify-content:center}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author{padding-left:.5em;padding-right:.5em}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:hover,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:visited,.reveal .quarto-title-block .quarto-title-authors .quarto-title-author a:active{color:inherit;text-decoration:none}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-name{margin-bottom:.1rem}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-email{margin-top:0px;margin-bottom:.4em;font-size:.6em}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-author-orcid img{margin-bottom:4px}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-affiliation{font-size:.7em;margin-top:0px;margin-bottom:8px}.reveal .quarto-title-block .quarto-title-authors .quarto-title-author .quarto-title-affiliation:first{margin-top:12px}ol{padding-left:.5em}ul{list-style:none}ul li::marker{color:#2c365e}dt{color:#e98a15}#title-slide{text-align:left}#title-slide .title{color:#2c365e}#title-slide .author{text-align:left;color:#e98a15;font-weight:bold}#title-slide .institute{padding-bottom:200px}.small{font-size:.75em}.smallest{font-size:.5em}.large{font-size:1.5em}.medium{font-size:.9em}.center-align{text-align:center}.hand{font-family:"Gochi Hand",cursive;font-size:125%}.hand-blue{font-family:"Gochi Hand",cursive;color:var(--primary);font-size:125%}/*# sourceMappingURL=f95d2bded9c28492b788fe14c3e9f347.css.map */ diff --git a/schedule/slides/10-basis-expansions.qmd b/schedule/slides/10-basis-expansions.qmd index cfebf56..903274a 100644 --- a/schedule/slides/10-basis-expansions.qmd +++ b/schedule/slides/10-basis-expansions.qmd @@ -7,12 +7,117 @@ metadata-files: {{< include _titleslide.qmd >}} +# Quick Review of Ridge and Lasso +## OLS: low bias, (potentially) high variance + +$$ +\begin{gathered} +\text{Model:} \quad y = x^\top \beta + \epsilon, \qquad \epsilon \sim N(0, \sigma^2) +\\ +\text{OLS:} \quad \bls = \argmin_\beta \| \y - \X\beta\|_2^2\quad +\end{gathered} +$$ + +- Bias: $\E[\bls] - \beta = \E[\E[\bls \mid \X]] - \beta = \ldots = 0$ +- variance: $\Var{\bls} = \sigma^2(\X^\top \X)^{-1}$ + +[When is $(\X^\top \X)^{-1}$ large?]{.secondary} + +. . . + +When we have *nearly colinear features*\ +Nearly colinear features $\Rightarrow$ small singular values $\Rightarrow$ large matrix inverse\ +[(Colinearity is more likely when $n$ is small)]{.secondary} + + +## Reducing variance (at the cost of additional bias) + +0. *Manual variable selection* +1. *Ridge regression*: $\min_\beta \| \y - \X\beta\|_2^2 + \lambda \Vert \beta \Vert_2^2$ +2. *Lasso*: $\min_\beta \| \y - \X\beta\|_2^2 + \lambda \Vert \beta \Vert_1$ + +. . . + +*Ridge* shrinks all parameters towards 0.\ +*Lasso* performs automatic variable selection.\ + +. . . + +\ +Increasing $\lambda$ *increases bias* and *decreases variance*.\ +[(For ridge, larger lambda $\rightarrow$ smaller $\beta$.)]{.small}\ +[(For lasso, larger lambda $\rightarrow$ sparser $\beta$.)]{.small} + + +--- + +```{r, fig.width=11,fig.align="center",dev="svg",fig.height=7, echo=FALSE} +library(glmnet) +data(prostate, package = "ElemStatLearn") +X <- prostate |> dplyr::select(-train, -lpsa) |> as.matrix() +Y <- prostate$lpsa +ridge <- glmnet(X, Y, alpha = 0, lambda.min.ratio = 1e-10) # added to get a minimum +lasso <- glmnet(X, Y, alpha = 1, lambda.min.ratio = 1e-10) # added to get a minimum +ridge.cv <- cv.glmnet(X, Y, alpha = 0, lambda.min.ratio = 1e-10) # added to get a minimum +lasso.cv <- cv.glmnet(X, Y, alpha = 1, lambda.min.ratio = 1e-10) # added to get a minimum +par(mfrow = c(2, 2)) +plot(ridge, main = "Ridge", xvar = "lambda") +plot(lasso, main = "Lasso", xvar = "lambda") +plot(ridge.cv, main = "Ridge", xvar = "lambda") +plot(lasso.cv, main = "Lasso", xvar = "lambda") +``` + + +## Computing ridge and lasso predictors + +- *OLS:* $\bls = (\X^\top \X)^{-1}\X^\top \y$ +- *Ridge:* $\brl = (\X^\top \X + \lambda \mathbf{I})^{-1}\X^\top \y$ +- *Lasso:* No closed form solution 😔 + - (Convex optimization problem solvable with iterative algorithms.) + + +## (Optional) Proof that ridge shrinks parameters + +(This proof is not too hard if you use facts about SVDs and eigenvalues. I recommend working through it.) + +Let $\mathbf{UDV^\top} = X$ be the SVD of $\mathbf X$. + +$$ +\begin{align} +\brl &= (\X^\top \X + \lambda \mathbf{I})^{-1} \X^\top \y + \\ + &= (\X^\top \X + \lambda \mathbf{I})^{-1} {\color{blue} \X \X^\top (\X^\top \X)^{-1}} \X^\top \y + \\ + &= (\X^\top \X + \lambda \mathbf{I})^{-1} \X \X^\top \underbrace{\left( (\X^\top \X)^{-1} \X^\top \y \right)}_{\bls} + \\ + &= (\mathbf V \mathbf D^2 \mathbf V^\top + \lambda \mathbf{I})^{-1} \mathbf V \mathbf D^2 \mathbf V^\top \bls + \\ + &= (\mathbf V (\mathbf D^2 + \lambda I) \mathbf V^\top)^{-1} \mathbf V \mathbf D^2 \mathbf V^\top \bls + \\ + &= \mathbf V (\mathbf D^2)(\mathbf D^2 + \lambda I)^{-1} \mathbf V^\top \bls +\end{align} +$$ + +--- + +$(\mathbf D^2)(\mathbf D^2 + \lambda I)^{-1}$ is a diagonal matrix with entries $d_i^2/(d_i^2 + \lambda) < 1$. + +So $\mathbf V (\mathbf D^2)(\mathbf D^2 + \lambda I)^{-1} \mathbf V^\top$ is a matrix that *shrinks* all coefficients of any vector multiplied against it. + +$$ +\Vert \mathbf V (\mathbf D^2)(\mathbf D^2 + \lambda I)^{-1} \mathbf V^\top \bls \Vert_2 < \Vert \bls \Vert_2. +$$ +So $\Vert \brl \Vert_2 < \Vert \bls \Vert_2$ + + +# Now onto new stuff +(But first, more clickers!) ## What about nonlinear things -$$\Expect{Y \given X=x} = \sum_{j=1}^p x_j\beta_j$$ +$$\text{Our usual model:} \quad \Expect{Y \given X=x} = \sum_{j=1}^p x_j\beta_j$$ Now we relax this assumption of linearity: @@ -113,26 +218,94 @@ Polynomials : $x \mapsto \left(1,\ x,\ x^2, \ldots, x^p\right)$ (technically, not quite this, they are orthogonalized) Linear splines -: $x \mapsto \bigg(1,\ x,\ (x-k_1)_+,\ (x-k_2)_+,\ldots, (x-k_p)_+\bigg)$ for some choices $\{k_1,\ldots,k_p\}$ +: $x \mapsto \bigg(1,\ x,\ (x-k_1)_+,\ (x-k_2)_+,\ldots, (x-k_p)_+\bigg)$ for some $\{k_1,\ldots,k_p\}$ + Fourier series : $x \mapsto \bigg(1,\ \cos(2\pi x),\ \sin(2\pi x),\ \cos(2\pi 2 x),\ \sin(2\pi 2 x), \ldots, \cos(2\pi p x),\ \sin(2\pi p x)\bigg)$ +```{r fig.height=3, fig.width=9} +#| code-fold: true +library(cowplot) +library(ggplot2) + +relu_shifted <- function(x, shift) {pmax(0, x - shift)} + +# Create a sequence of x values +x_vals <- seq(-3, 3, length.out = 1000) +# Create a data frame with all the shifted functions +data <- data.frame( + x = rep(x_vals, 5), + polynomial = c(x_vals, x_vals^2, x_vals^3, x_vals^4, x_vals^5), + linear.splines = c(relu_shifted(x_vals, 2), relu_shifted(x_vals, 1), relu_shifted(x_vals, 0), relu_shifted(x_vals, -1), relu_shifted(x_vals, -2)), + fourier = c(cos(pi / 2 * x_vals), sin(pi / 2 * x_vals), cos(pi / 4 * x_vals), sin(pi / 4 * x_vals), cos(pi * x_vals)), + function_label = rep(c("f1", "f2", "f3", "f4", "f5"), each = length(x_vals)) +) + +# Plot using ggplot2 +g1 <- ggplot(data, aes(x = x, y = polynomial, color = function_label)) + + geom_line(size = 1, show.legend=FALSE) + + theme(axis.text.y=element_blank()) +g2 <- ggplot(data, aes(x = x, y = linear.splines, color = function_label)) + + geom_line(size = 1, show.legend=FALSE) + + theme(axis.text.y=element_blank()) +g3 <- ggplot(data, aes(x = x, y = fourier, color = function_label)) + + geom_line(size = 1, show.legend=FALSE) + + theme(axis.text.y=element_blank()) + +plot_grid(g1, g2, g3, ncol = 3) +``` ## How do you choose? [Procedure 1:]{.secondary} -1. Pick your favorite basis. This is not as easy as it sounds. For instance, if $f$ is a step function, linear splines will do well with good knots, but polynomials will be terrible unless you have __lots__ of terms. +1. Pick your favorite basis. (Think if the data might "prefer" one basis over another.) + - [How "smooth" is the response you're trying to model?]{.small} + + 2. Perform OLS on different orders. 3. Use model selection criterion to choose the order. +--- + +```{r echo=FALSE, fig.height=2, fig.width=6} +plot_grid(g1, g2, g3, ncol = 3) +``` + +```{r fig.height=4, fig.width=12, echo=FALSE} +set.seed(12345) +coeff <- rnorm(5) +funcs <- data.frame(x = x_vals, + f1 = (matrix(data$polynomial, nrow = length(x_vals), ncol = 5) %*% coeff), + f2 = (matrix(data$fourier, nrow = length(x_vals), ncol = 5) %*% coeff), + f3 = (matrix(data$linear.splines, nrow = length(x_vals), ncol = 5) %*% coeff)) + +g4 <- ggplot(funcs, aes(x=x, y=f1)) + geom_line(color = "blue") + theme() +g5 <- ggplot(funcs, aes(x=x, y=f2)) + geom_line(color = "blue") + theme() +g6 <- ggplot(funcs, aes(x=x, y=f3)) + geom_line(color = "blue") + theme() + +plot_grid(g4, g5, g6, ncol = 3) +``` + +What bases do you think will work best for $f1$, $f2$ and $f3$? + +. . . + +[*Answer: $f1$ was made from polynomial bases, $f2$ from fourier, $f3$ from linear splines*]{.secondary} + + +--- + +## How do you choose? + [Procedure 2:]{.secondary} 1. Use a bunch of high-order bases, say Linear splines and Fourier series and whatever else you like. @@ -148,7 +321,7 @@ Fourier series 2. Estimate polynomials up to 20 as before and choose best order. -3. Do ridge, lasso and elastic net $\alpha=.5$ on 20th order polynomials, B splines with 20 knots, and Fourier series with $p=20$. Choose tuning parameter (using `lambda.1se`). +3. Do ridge, lasso and elastic net $\alpha=.5$ on 20th order polynomials, splines with 20 knots, and Fourier series with $p=20$. Choose tuning parameter (using `lambda.1se`). 4. Repeat 1-3 10 times (different splits) @@ -162,7 +335,7 @@ mapto01 <- function(x, pad = .005) (x - min(x) + pad) / (max(x) - min(x) + 2 * p x <- mapto01(arcuate$position) Xmat <- cbind( poly(x, 20), - splines::bs(x, df = 20), + splines::bs(x, df = 20, degree = 1), cos(2 * pi * outer(x, 1:20)), sin(2 * pi * outer(x, 1:20)) ) y <- arcuate$fa @@ -223,16 +396,19 @@ Used $2p+1$ dimensions with Fourier basis Each case applied a [feature map]{.secondary} to $x$, call it $\Phi$ We used new "features" $\Phi(x) = \bigg(\phi_1(x),\ \phi_2(x),\ldots,\phi_k(x)\bigg)$ +w/ a linear model -Neural networks (coming in module 4) use this idea +$$f(x) = \Phi(x)^\top \beta$$ -You've also probably seen it in earlier courses when you added interaction terms or other transformations. +Neural networks (coming in module 4) build upon this idea -. . . + -Some methods (notably Support Vector Machines and Ridge regression) allow $k=\infty$ +. . . -See [ISLR] 9.3.2 for baby overview or [ESL] 5.8 (note 😱) +\ +Some methods (notably Support Vector Machines and other Kernel Machines) allow $k=\infty$\ +[See [ISLR] 9.3.2 for baby overview or [ESL] 5.8 (note 😱)]{.small} # Next time...