Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

Yarn 3 workspaces are not cached at all #615

Open
marcofugaro opened this issue Aug 25, 2021 · 3 comments
Open

Yarn 3 workspaces are not cached at all #615

marcofugaro opened this issue Aug 25, 2021 · 3 comments

Comments

@marcofugaro
Copy link

marcofugaro commented Aug 25, 2021

Related: #612 #432 #319

We are using yarn workspaces with yarn's latest version (3.0.1) and yarn is not cached at all, it takes 1m 54s in best cases and 3m 17s in worst cases, including the cache restore phase.

On the contrary, on GitHub actions with the yarn cache action, the Fetch phase is completely skipped, and it just runs the linking phase. It takes ~20 seconds.

I pasted the logs at the end of the issue, the relevant part are these:

And there is this warning from yarn, it may be relevant:

image

Full Netlify build logs (excluding build phase)
4:18:09 PM: Build ready to start
4:18:11 PM: build-image version: c6001ed68662a13e5deb24abec2b46058c58248a
4:18:11 PM: build-image tag: v3.9.0
4:18:11 PM: buildbot version: d2795bf5a96b85e585e90944f15f15381ce164a7
4:18:11 PM: Fetching cached dependencies
4:18:11 PM: Starting to download cache of 802.8MB
4:18:15 PM: Finished downloading cache in 3.951667197s
4:18:15 PM: Starting to extract cache
4:18:31 PM: Finished extracting cache in 16.395963806s
4:18:32 PM: Finished fetching cache in 20.470693303s
4:18:32 PM: Starting to prepare the repo for build
4:18:32 PM: Preparing Git Reference pull/1028/head
4:18:35 PM: Parsing package.json dependencies
4:18:35 PM: Starting build script
4:18:35 PM: Installing dependencies
4:18:35 PM: Python version set to 2.7
4:18:36 PM: Started restoring cached node version
4:18:38 PM: Finished restoring cached node version
4:18:39 PM: v16.7.0 is already installed.
4:18:40 PM: Now using node v16.7.0 (npm v7.20.3)
4:18:40 PM: Started restoring cached build plugins
4:18:40 PM: Finished restoring cached build plugins
4:18:40 PM: Attempting ruby version 2.7.1, read from environment
4:18:40 PM: Started restoring cached ruby version
4:18:40 PM: Finished restoring cached ruby version
4:18:42 PM: Using ruby version 2.7.1
4:18:42 PM: Using PHP version 5.6
4:18:42 PM: Started restoring cached yarn cache
4:18:42 PM: Finished restoring cached yarn cache
4:18:44 PM: Found yarn version (3.0.1) that doesn't match expected (1.22.4)
4:18:44 PM: up to date, audited 1 package in 261ms
4:18:44 PM: found 0 vulnerabilities
4:18:44 PM: Installing yarn at version 1.22.4
4:18:44 PM: Installing Yarn!
4:18:44 PM: > Downloading tarball...
4:18:44 PM: [1/2]: https://yarnpkg.com/downloads/1.22.4/yarn-v1.22.4.tar.gz --> /tmp/yarn.tar.gz.uavJVbtnLD
4:18:44 PM:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
4:18:44 PM:                                  Dload  Upload   Total   Spent    Left  Speed
4:18:45 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
4:18:45 PM: 100    79  100    79    0     0    351      0 --:--:-- --:--:-- --:--:--   352
4:18:45 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
4:18:45 PM: 100    93  100    93    0     0    146      0 --:--:-- --:--:-- --:--:--   451
4:18:45 PM: 100   625  100   625    0     0    684      0 --:--:-- --:--:-- --:--:--   684
4:18:46 PM: 100 1215k  100 1215k    0     0  1006k      0  0:00:01  0:00:01 --:--:-- 1006k
4:18:46 PM: [2/2]: https://yarnpkg.com/downloads/1.22.4/yarn-v1.22.4.tar.gz.asc --> /tmp/yarn.tar.gz.uavJVbtnLD.asc
4:18:46 PM: 100    83  100    83    0     0   1853      0 --:--:-- --:--:-- --:--:--  1853
4:18:46 PM: 100    97  100    97    0     0   1318      0 --:--:-- --:--:-- --:--:--  1318
4:18:46 PM: 100   629  100   629    0     0   6041      0 --:--:-- --:--:-- --:--:--  6041
4:18:46 PM: 100  1028  100  1028    0     0   8914      0 --:--:-- --:--:-- --:--:--  8914
4:18:46 PM: > Verifying integrity...
4:18:46 PM: gpg: Signature made Mon 09 Mar 2020 03:52:13 PM UTC using RSA key ID 69475BAA
4:18:46 PM: gpg: Good signature from "Yarn Packaging <[email protected]>"
4:18:46 PM: gpg: WARNING: This key is not certified with a trusted signature!
4:18:46 PM: gpg:          There is no indication that the signature belongs to the owner.
4:18:46 PM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
4:18:46 PM:      Subkey fingerprint: 6D98 490C 6F1A CDDD 448E  4595 4F77 6793 6947 5BAA
4:18:46 PM: > GPG signature looks good
4:18:46 PM: > Extracting to ~/.yarn...
4:18:46 PM: > Adding to $PATH...
4:18:46 PM: > Successfully installed Yarn 1.22.4! Please open another terminal where the `yarn` command will now be available.
4:18:46 PM: Yarn workspaces detected
4:18:46 PM: Started restoring workspace packages/editor node modules
4:18:46 PM: Finished restoring workspace packages/editor node modules
4:18:46 PM: Started restoring workspace root node modules
4:18:46 PM: Finished restoring workspace root node modules
4:18:47 PM: Installing NPM modules using Yarn version 3.0.1
4:18:48 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
4:18:49 PM: ➤ YN0000: ┌ Resolution step
4:18:49 PM: ➤ YN0002: │ @mdx-js/loader@npm:1.6.22 doesn't provide react (p12550), requested by @mdx-js/react
➤ YN0002: │ @react-aria/menu@npm:3.1.4 [6b3b6] doesn't provide react-dom (p7793e), requested by @react-aria/overlays
➤ YN0002: │ @react-aria/tooltip@npm:3.1.1 [6b3b6] doesn't provide react-dom (p25457), requested by @react-aria/overlays
➤ YN0002: │ @storybook/core@npm:6.1.21 [13e62] doesn't provide typescript (p4c2e3), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide react-refresh (pf9dee), requested by @pmmmwh/react-refresh-webpack-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide typescript (p0ad40), requested by react-docgen-typescript-plugin
➤ YN0002: │ @storybook/preset-create-react-app@npm:3.1.7 [7e6ab] doesn't provide webpack (p82525), requested by @pmmmwh/react-refresh-webpack-plugin
➤ YN0002: │ @storybook/react@npm:6.1.21 [7e6ab] doesn't provide typescript (p62333), requested by react-docgen-typescript-plugin
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p170a2), requested by babel-plugin-named-asset-import
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (pf8a98), requested by @storybook/addon-essentials
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (pc2008), requested by @storybook/preset-create-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p09571), requested by @storybook/react
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @babel/core (p1c95e), requested by babel-jest
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @testing-library/dom (p52ade), requested by @testing-library/user-event
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @typescript-eslint/eslint-plugin (pae93d), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide @typescript-eslint/parser (p47980), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide babel-loader (pf2344), requested by @storybook/addon-docs
➤ YN0002: │ editor@workspace:packages/editor doesn't provide babel-loader (pd8908), requested by @storybook/addon-essentials
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-flowtype (pdeceb), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-import (p284ba), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-jsx-a11y (p2f943), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-react (p4b4fd), requested by eslint-config-react-app
➤ YN0002: │ editor@workspace:packages/editor doesn't provide eslint-plugin-react-hooks (pb5a16), requested by eslint-config-react-app
➤ YN0060: │ editor@workspace:packages/editor provides react (pdd751) with version 17.0.1, which doesn't satisfy what @storybook/addon-actions and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react (pa0ab2) with version 17.0.1, which doesn't satisfy what @storybook/addon-docs and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react (p03409) with version 17.0.1, which doesn't satisfy what react-multi-email requests
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (p60341) with version 17.0.1, which doesn't satisfy what @storybook/addon-actions and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (pdc343) with version 17.0.1, which doesn't satisfy what @storybook/addon-docs and some of its descendants request
➤ YN0060: │ editor@workspace:packages/editor provides react-dom (pce5e4) with version 17.0.1, which doesn't satisfy what react-multi-email requests
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (pa7cad), requested by raw-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p68732), requested by esbuild-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p4ed41), requested by fast-sass-loader
➤ YN0002: │ editor@workspace:packages/editor doesn't provide webpack (p92b4b), requested by speed-measure-webpack-plugin
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pbc588), requested by @react-aria/i18n
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p8a0d0), requested by @react-aria/interactions
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p48a46), requested by @react-aria/link
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p109e9), requested by @react-aria/utils
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc15d4), requested by @react-aria/focus
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p6b6be), requested by @react-aria/label
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p16962), requested by @react-aria/ssr
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p83932), requested by @react-aria/overlays
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p3ae77), requested by @react-aria/progress
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p80dff), requested by @react-aria/visually-hidden
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p3a742), requested by @react-aria/textfield
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pabf72), requested by @react-aria/menu
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pfc846), requested by @react-aria/breadcrumbs
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pcd1d8), requested by @react-aria/button
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p09bf0), requested by @react-aria/checkbox
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p51ee3), requested by @react-aria/dialog
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pb7f39), requested by @react-aria/listbox
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p4847b), requested by @react-aria/meter
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc8bc1), requested by @react-aria/radio
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pc2954), requested by @react-aria/searchfield
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pfeef5), requested by @react-aria/select
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pede51), requested by @react-aria/separator
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p9db53), requested by @react-aria/slider
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (pe751f), requested by @react-aria/switch
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react (p5cd5d), requested by @react-aria/tooltip
➤ YN0002: │ react-aria@npm:3.4.0 doesn't provide react-dom (p337a3), requested by @react-aria/overlays
➤ YN0002: │ react-dev-utils@npm:11.0.4 doesn't provide typescript (p79ddf), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ react-dev-utils@npm:11.0.4 doesn't provide webpack (p2af19), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ react-hot-toast@npm:2.0.0 [7e6ab] doesn't provide csstype (p287b3), requested by goober
➤ YN0002: │ react-stately@npm:3.3.0 doesn't provide react (p85986), requested by @react-stately/data
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide @typescript-eslint/eslint-plugin (pdf9d2), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide @typescript-eslint/parser (pd2e4e), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide babel-eslint (peb234), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide cypress (pa68a9), requested by @testing-library/cypress
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide cypress (p85358), requested by cypress-localstorage-commands
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-flowtype (pfa493), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-import (pbae0d), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-jsx-a11y (pc5b66), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-react (p45cfd), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide eslint-plugin-react-hooks (pc0ad7), requested by eslint-config-react-app
➤ YN0002: │ root-workspace-0b6124@workspace:. doesn't provide react (p5a3eb), requested by react-scripts
➤ YN0002: │ runtime@workspace:packages/runtime doesn't provide webpack (p462c7), requested by raw-loader
➤ YN0002: │ runtime@workspace:packages/runtime [7e6ab] doesn't provide webpack (pc8bdf), requested by raw-loader
➤ YN0002: │ spe@workspace:packages/spe doesn't provide three (p422b3), requested by postprocessing
➤ YN0002: │ spe@workspace:packages/spe doesn't provide webpack (pfa9bb), requested by file-loader
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
4:18:49 PM: ➤ YN0000: └ Completed in 0s 608ms
4:18:49 PM: ➤ YN0000: ┌ Fetch step
4:20:23 PM: ➤ YN0013: │ 8 packages were already cached, 2408 had to be fetched
4:20:23 PM: ➤ YN0000: └ Completed in 1m 34s
4:20:23 PM: ➤ YN0000: ┌ Link step
4:20:24 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.3.2#~builtin<compat/fsevents>::version=2.3.2&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
4:20:25 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A1.2.13#~builtin<compat/fsevents>::version=1.2.13&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
4:20:29 PM: ➤ YN0007: │ husky@npm:4.3.7 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js@npm:3.9.1 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ electron@npm:8.5.5 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ esbuild@npm:0.12.19 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ ejs@npm:2.7.4 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js-pure@npm:3.9.1 must be built because it never has been before or the last one failed
4:20:29 PM: ➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT husky > Setting up git hooks
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT CI detected, skipping Git hooks installation.
4:20:30 PM: ➤ YN0000: │ husky@npm:4.3.7 STDOUT husky > Done
4:20:31 PM: ➤ YN0000: └ Completed in 7s 690ms
4:20:31 PM: ➤ YN0000: Done with warnings in 1m 43s
4:20:32 PM: NPM modules installed using Yarn
4:20:33 PM: Started restoring cached go cache
4:20:33 PM: Finished restoring cached go cache
4:20:33 PM: go version go1.14.4 linux/amd64
4:20:33 PM: go version go1.14.4 linux/amd64
4:20:33 PM: Installing missing commands
4:20:33 PM: Verify run directory
4:20:34 PM: ​
4:20:34 PM: ────────────────────────────────────────────────────────────────
4:20:34 PM:   Netlify Build
4:20:34 PM: ────────────────────────────────────────────────────────────────
4:20:34 PM: ​
4:20:34 PM: ❯ Version
4:20:34 PM:   @netlify/build 18.5.0
4:20:34 PM: ​
4:20:34 PM: ❯ Flags
4:20:34 PM:   baseRelDir: true
4:20:34 PM:   buildId: 612651213fe4cc0008c22123
4:20:34 PM:   deployId: 612651213fe4cc0008c22125
4:20:34 PM: ​
4:20:34 PM: ❯ Current directory
4:20:34 PM:   /opt/build/repo
4:20:34 PM: ​
4:20:34 PM: ❯ Config file
4:20:34 PM:   /opt/build/repo/netlify.toml
4:20:34 PM: ​
4:20:34 PM: ❯ Context
4:20:34 PM:   deploy-preview
4:20:35 PM: ​
4:20:35 PM: ────────────────────────────────────────────────────────────────
4:20:35 PM:   1. Build command from Netlify app
4:20:35 PM: ────────────────────────────────────────────────────────────────
4:20:35 PM: ​
4:20:35 PM: $ yarn build
@JGAntunes
Copy link
Contributor

JGAntunes commented Aug 30, 2021

Thanks for raising this @marcofugaro. This is definitely related with #612 and #319 and our usage of the --cache-folder property 👍 however it isn't particular to your workspace settings I believe, just the yarn version used.

@kachkaev
Copy link

kachkaev commented Jan 25, 2022

This issue is five months old today and there is still no solution for Yarn 3 in the public. I tried finding a workaround today but gave up: netlify/build#1535 (comment) HALP! 🙏

UPD: Workaround: netlify/build#1535 (comment)

@piecyk
Copy link

piecyk commented May 27, 2022

Hi, manage to create a workaround for yarn3 using a custom plugin, till #612

First in netlify.toml use NPM_FLAGS hack to disable default npm install, here we cannot use NETLIFY_USE_YARN becase of #779

[build.environment]
  NODE_VERSION = "16"
  NPM_FLAGS = "--version"

[[plugins]]
  package = "/.netlify/netlify-plugin-yarn"

then create .netlify/netlify-plugin-yarn folder with files

  • manifest.yml
  • index.js

manifest.yml is just

name: netlify-plugin-yarn
inputs: []

and index.js is responsible for moving cache folder, setting YARN_CACHE_FOLDER and enabling corepack for packageManager in package.json

module.exports = {
  onPreBuild: async ({ utils: { build, run } }) => {
    const cacheDir = '.yarn_cache'

    process.env['YARN_CACHE_FOLDER'] = `${process.env.HOME}/${cacheDir}`
    process.env['NETLIFY_CACHE_DIR'] = `${process.env.NETLIFY_BUILD_BASE}/cache`

    try {
      await run.command(`mv ${process.env.NETLIFY_CACHE_DIR}/${cacheDir} ${process.env.YARN_CACHE_FOLDER}`)
    } catch (error) {
      console.log('No yarn cache found')
    }

    try {
      await run.command('corepack enable')
    } catch (error) {
      return build.failBuild(error)
    }
  },
}

package.json

{
  "private": true,
  "engines": {
    "node": "16.x",
    "yarn": "3.2.1"
  },
  "packageManager": "[email protected]"
}

Edited ---

Extracted this code into package, https://github.com/piecyk/yarn-berry-netlify-plugin

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants