Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SessionLoader: batch load #4439

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Conversation

soulomoon
Copy link
Collaborator

@soulomoon soulomoon commented Oct 24, 2024

An attempt to fix #4381

The main idea, is keeping track of pending files, succ loaded files(old files) and fail loaded files. Try to load as much new files as possible at one time. And fall back to load file one by one for those fail to load in batch.

Detail:

  1. Whenever we want to load a file, we add to the pending files, and start loading.
  2. we load a file in two cases:
    1. if a file is not in fail loaded file, we load this file with extra files: pending files - fail loaded files, and old files.
    2. if a file is in the fail loaded file, we load this file with only old files.
  3. load result in two cases:
    1. if load succ, we remove actual loaded files from pending files and fail loaded files, then add them to succ loaded files.
    2. if load fail, we check if we are loading with 2.I or 2.II, for 2.I, we put all the extra files to fail loaded files. and start 2 again. otherwise we return with error as before and put the file to fail loaded files.

Completed Tasks:

  1. Implemented Batch Loading with Session Loader’s Pending Files Queue: Developed a mechanism to queue files for loading and optimize file loading by batching as many files as possible into the current cradle being processed.
  2. Enhanced Session Loader for Consistency in Build Results: Refined the session loader to ensure more consistent build outcomes by halting the Shake session before executing the initial load.

Potential Improvements (Constrained by Current Hiebios Limitations):

  1. Error distribution over files when we batch load. This PR provide workaround for by fall back to non-batch load when error happens in batch loading.
  2. cradleToOptsAndLibDir should return what we have tried to load even on failure the workaround can be more accurate.

@soulomoon soulomoon added the performance Issues about memory consumption, responsiveness, etc. label Oct 25, 2024
@soulomoon soulomoon marked this pull request as ready for review November 3, 2024 07:14
@soulomoon

This comment was marked as outdated.

@soulomoon soulomoon marked this pull request as draft November 3, 2024 07:25
@soulomoon
Copy link
Collaborator Author

@fendor I've implemented a workable batch loader, do you want to try it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Issues about memory consumption, responsiveness, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Initial load of multiple home units is slow with lazy component loading
1 participant