Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimally choose division count by interpreting min-cpu-threads in task model #2

Open
krunch3r76 opened this issue Apr 29, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@krunch3r76
Copy link
Owner

krunch3r76 commented Apr 29, 2022

because of network latency, the task model lags when dividing to many nodes.

it may make sense to use the service model in which min-cpu-threads dictates how many threads the work would be parallelized across on a presumably hefty provider.

but the (simple) service model implies multiple work consecutively, so for a single file, task model makes sense. service model may come into play with archiving... see comments below for plans on optimizing single file compression in current task model

@krunch3r76 krunch3r76 added the enhancement New feature or request label Apr 29, 2022
@krunch3r76 krunch3r76 self-assigned this Apr 29, 2022
@krunch3r76
Copy link
Owner Author

krunch3r76 commented Apr 29, 2022

initial heuristics model:
if min-cpu-threads exceeds the number of divisions, the division count may be reduced.
suppose divisions is 10 (default) and min-cpu-threads is 24
then divisions should be adjusted down to 1
suppose divisions is 10 (default) and min-cpu-threads is 11
then divisions should be adjusted down to 1
supposed divisions is 10 (default) and min-cpu-threads is 8
then divisions should be adjusted down to 2 the first multiple of 8 above division count

@krunch3r76 krunch3r76 changed the title reinterpret min-cpu-threads in a service model reinterpret min-cpu-threads in task or potentially service model Apr 29, 2022
@krunch3r76 krunch3r76 changed the title reinterpret min-cpu-threads in task or potentially service model reinterpret min-cpu-threads in task model Apr 29, 2022
@krunch3r76 krunch3r76 changed the title reinterpret min-cpu-threads in task model reinterpret min-cpu-threads in task model for optimal division count Apr 29, 2022
@krunch3r76 krunch3r76 changed the title reinterpret min-cpu-threads in task model for optimal division count optimally choose division count by interpreting min-cpu-threads in task model Apr 29, 2022
@krunch3r76
Copy link
Owner Author

krunch3r76 commented Apr 30, 2022

currently work is divided into blocks of 64MiB to use 1 thread. the thread count should be increased to 2 for files 128MiB or larger. when this is done this issue shall be resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant