Set vmen correctly, fixes #66 #70
Conversation
|
fixes #66 |
erna/scripts/process_fact_mc.py
Outdated
| stream_runner = stream_runner_local | ||
| else: | ||
| run = [jar, xml, df] | ||
| run = [jar, xml, df, None, '{mb}'.format(0.9*vmem)] |
There was a problem hiding this comment.
This will raise a KeyError, I think you ment '{}mb'
There was a problem hiding this comment.
Actually I think it should be '{}m' as this will work, I don't know if java works with 'mb', it should be checked while fixing it.
There was a problem hiding this comment.
shit I though I had them all. Missed that one. Good catch. mb works. worked before
| @@ -21,7 +21,7 @@ def make_jobs(jar, xml, aux_source_path, output_directory, df_mapping, engine, | |||
| for num, indices in enumerate(split_indices): | |||
| df = df_mapping[indices.min(): indices.max()] | |||
|
|
|||
| job = Job(stream_runner.run, [jar, xml, df, aux_source_path], | |||
| job = Job(stream_runner.run, [jar, xml, df, aux_source_path, '{}mb'.format(0.9*vmem)], | |||
There was a problem hiding this comment.
This can result in bad errors, if vmem is lower then 10GB (the memory of everything else is set to around 1 GB). I think going for vmem-1000 and making sure that vmem can't be set lower then 2 GB for example, should be good. To create a check for vmem change its click type to: type=click.IntRange(2000, 1000000) (A maximum of 1 TB should be good enough for a little while).
There was a problem hiding this comment.
I don't understand what you mean. This way the heap_size is always set to 90% vmem. so in case of 10GB vmem to 9GB heap_size and 1GB vmem to 900MB heap_size. Why should vmem-1000 help?
erna/scripts/process_fact_mc.py
Outdated
| stream_runner = stream_runner_local | ||
| else: | ||
| run = [jar, xml, df] | ||
| run = [jar, xml, df, None, '{mb}'.format(0.9*vmem)] |
There was a problem hiding this comment.
Actually I think it should be '{}m' as this will work, I don't know if java works with 'mb', it should be checked while fixing it.
WIP