Cache flags #28331
Replies: 9 comments 24 replies
-
|
FYI, I'm enabling // gatsby-config.js
flags: {
FAST_DEV: true,
FAST_REFRESH: true,
},info The following flags are active:
- FAST_DEV · Enable all experiments aimed at improving develop server start time
- FAST_REFRESH · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/28390)) · Use React Fast Refresh instead of the legacy react-hot-loader for instantaneous feedback in your development server. Recommended for versions of React >= 17.0.
- DEV_SSR · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/28138)) · SSR pages on full reloads during develop. Helps you detect SSR bugs and fix them without needing to do full builds.
- QUERY_ON_DEMAND · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/27620)) · Only run queries when needed instead of running all queries upfront. Speeds starting the develop server.
- LAZY_IMAGES · EXPERIMENTAL · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/27603)) · Don't process images during development until they're requested from the browser. Speeds starting the develop server.
There are 2 other flags available that you might be interested in:
- PRESERVE_WEBPACK_CACHE · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/28331)) · Don't delete webpack's cache when changing gatsby-node.js & gatsby-config.js files.
- PRESERVE_FILE_DOWNLOAD_CACHE · (Umbrella Issue (https://github.com/gatsbyjs/gatsby/discussions/28331)) · Don't delete the downloaded files cache when changing gatsby-node.js & gatsby-config.js files. |
Beta Was this translation helpful? Give feedback.
-
|
oops, we didn't actually add it yet to FAST_DEV. Thanks for the reminder! PR #28447 |
Beta Was this translation helpful? Give feedback.
-
|
Nice call out in the CLI, it's a great way to discover those goodies. The instructions are not specific enough though. Not sure where to put flags. |
Beta Was this translation helpful? Give feedback.
-
|
Is the instructions in the body of the discussion more clear? Thanks for letting us know — we'll keep iterating on the instructions to make them more clear. |
Beta Was this translation helpful? Give feedback.
-
|
This was much clearer: |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Can you add a screenshot of the error? |
Beta Was this translation helpful? Give feedback.
-
|
Apologies, please delete! |
Beta Was this translation helpful? Give feedback.
-
|
Im experiencing an issue where Build / Develop cli logs are not visible if one of the flags are set. DescriptionWhen one of the new dev flags are set, Steps to reproduceOn the latest cli code, in Expected resultNotice about flags are shown, Actual resultNotice about flags are shown, Anyone else? |
Beta Was this translation helpful? Give feedback.
-
|
Same for me. I updated and the behavior started. Unfortunatelty I can seem to fix it temporarily by setting the flags to false or did I miss any? Edit: Added PARALLEL_SOURCING to the mix without success on gatsby-cli@latest. |
Beta Was this translation helpful? Give feedback.
-
|
Downgrading gatsby-cli to 2.15.1 resolved the issue, testing 2.16.1 reintroduced it. |
Beta Was this translation helpful? Give feedback.
-
|
I have also experienced this issue on Gatsby v 2.31.0 |
Beta Was this translation helpful? Give feedback.
-
|
I've been having some issues on windows where it doesn't seem to preserve the download cache. I think it's due to how 'del' works. I've added in a line to not delete the caches folder and it appears to work.
I think this is sort of indicated by the documentation for del https://github.com/sindresorhus/del#beware I believe it is working for me because my builds are now faster if I have this change in my locally modified node_modules. |
Beta Was this translation helpful? Give feedback.
-
|
So I was testing our your PR against an image from unsplash and it still wasn't working. I looked into it more and realized that we've only ever added support for Looking around, I think we should switch to using https://www.npmjs.com/package/make-fetch-happen as it promises us — "Built-in request caching following full HTTP caching rules (Cache-Control, ETag, 304s, cache fallback on error, etc)." |
Beta Was this translation helpful? Give feedback.
-
|
heh actually I tested it quickly against the unsplash URLs and it works as expected |
Beta Was this translation helpful? Give feedback.
-
|
I'll work on a PR to add caching support next week |
Beta Was this translation helpful? Give feedback.
-
|
hmmm actually (another yet again) — https://github.com/kornelski/http-cache-semantics is what we want. The "Got" cache means that we'd need to read & write to our cache each time — So this would work like our current system but have much more robust support for different caching setups. I might just post an issue though as I think the |
Beta Was this translation helpful? Give feedback.
-
|
One plus to changing is right now we do revalidate every time where w/ the above, we often wouldn't — which would save a lot of http requests. |
Beta Was this translation helpful? Give feedback.
-
DescriptionWhen using are emitted at build time. Steps to reproduceA minimal reproduction is available here Running Expected resultSuccessful build-time module resolution. Actual resultAn error at build time. Environmentconst flags = {
PRESERVE_WEBPACK_CACHE: true
}; |
Beta Was this translation helpful? Give feedback.
-
|
I got the "Hi from the Gatsby maintainers!" message about the PRESERVE_WEBPACK_CACHE flag in a CI build. Should I check the .cache folder into source control to take advantage of this? It's in my .gitignore but I'm not sure if I put it there or Gatsby did. |
Beta Was this translation helpful? Give feedback.
-
|
Don't check it in — just setup your CI to preserve and restore the |
Beta Was this translation helpful? Give feedback.
-
|
Can we have support for gatsby config files in I am using gatsby with ts-node and I get this error: Error: <w> [webpack.cache.PackFileCacheStrategy] Caching failed for pack: Error: Can't resolve '/Users/phil/Code/newrade-core/packages/ze-design-website/gatsby-node.js'Also I tried to disable or modify the Relevant code is here: |
Beta Was this translation helpful? Give feedback.
-
|
My current cache workaround: config.cache = {
type: 'filesystem',
buildDependencies: {
config: [
__filename,
path.resolve(process.cwd(), 'package.json'),
path.resolve(process.cwd(), 'tsconfig.json'),
...store
.getState()
.flattenedPlugins.filter((plugin: any) =>
plugin.nodeAPIs.includes(`onCreateWebpackConfig`)
)
.map((plugin: any) => {
const pluginPath = path.join(plugin.resolve, `gatsby-node.js`);
const pluginPathTs = path.join(plugin.resolve, `gatsby-node.ts`);
let foundPluginPath;
try {
fs.readFileSync(pluginPath);
foundPluginPath = pluginPath;
return foundPluginPath;
} catch (error: any) {
reporter.info(`[${pluginOptions.pluginName}] could not find file: ${pluginPath}`);
}
try {
reporter.info(`[${pluginOptions.pluginName}] trying file in ts: ${pluginPathTs}`);
fs.readFileSync(pluginPathTs);
reporter.info(`[${pluginOptions.pluginName}] found file: ${pluginPathTs}`);
foundPluginPath = pluginPathTs;
} catch (error: any) {
reporter.info(`[${pluginOptions.pluginName}] could not find file: ${pluginPathTs}`);
}
return foundPluginPath;
}),
],
},
};Output: It replaces the cache entries for gatsby config files with the correct .ts versions |
Beta Was this translation helpful? Give feedback.
-
|
The above workaround works for v3.13.0, just make sure that you set the |
Beta Was this translation helpful? Give feedback.
-
|
Been a while since something was added to this discussion... But I still think there's a major issue with overly aggressive clearing of thumbnail cache for thumbnails generated by This was discussed here #24822 but then the issue was closed because part of the problem was addressed by Query on Demand and Lazy Image flags - but this only addresses the issue of slow dev server builds, the issue of production builds was not addressed. Ideally we could have something like PRESERVE_FILE_DOWNLOAD_CACHE but for image thumbnails. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it's an idea… I've also lost sooooooo… many time ! |
Beta Was this translation helpful? Give feedback.
-
Cache clearing
We're investigating ways to be smarter about clearing caches. Initial PR
Currently we delete the entire cache when your gatsby-node.js, gatsby-config.js, or package versions change or you run gatsby clean. This is unnecessarily aggressive — changing these things don't invalidate most caches. The current cache clearing behavior means very expensive operations like downloading and processing images have to be repeated quite often which makes iterating on the website a lot more frustrating than it should.
There are currently two cache clearing flags you can try:
PRESERVE_FILE_DOWNLOAD_CACHEprevents file downloads from being deleted during cache clearing events (other thangatsby cleanwhich still deletes the entire cache).DEV_WEBPACK_CACHEprevents the webpack cache from deleted duringgatsby developwith the same caveat forgatsby cleanWe will continue to clear other plugin caches e.g. those created by source & transformer plugins. We're looking at ways to safely making more selective clearing those caches as well.
To enable the new cache behavior, add to your gatsby-config.js:
These will also be enabled if you add the
FAST_DEVflag which enables a number of WIP ways to speed up the dev server.Utilising webpack persistent caching
In addition to preserving webpack caches when invalidating gatsby cache,
PRESERVE_FILE_DOWNLOAD_CACHEflag (since[email protected]) also utilize webpack persitent caching for production builds. This makes webpack steps in subsequentgatsby buildsignificantly faster.Beta Was this translation helpful? Give feedback.
All reactions