generic.concat keep average #739
Conversation
| temp_dfs.Close() | ||
|
|
||
| # Calculate average | ||
| averaged_data = (existing_data + current_data) / 2 |
There was a problem hiding this comment.
This is calculating the average as soon as the dfs2's overlap
| averaged_data = (existing_data + current_data) / 2 | ||
|
|
||
| # Write averaged data | ||
| dfs_o.WriteItemTimeStepNext(0, averaged_data) |
There was a problem hiding this comment.
But this line is wrong, since it is 'appending' at the end.
I tried with WriteItemTimeStep but could not make it work.
There was a problem hiding this comment.
@ryan-kipawa Is this something that you would like to dig into?
| dfs_o.Flush() | ||
| temp_dfs = DfsFileFactory.DfsGenericOpen( | ||
| str(outfilename) | ||
| ) |
There was a problem hiding this comment.
This is also kind of crazy. It crashes and complains that it cannot open outfilename, but if I do dfs_o.Flush() again after it crashed, then it works.
its almost like I need to add a try/except and flush twice to make it run
There was a problem hiding this comment.
I will make a new failing test on Monday, I have some data I know the expected output
There was a problem hiding this comment.
Ok I already uploaded a failing unit test
| @@ -552,6 +552,82 @@ def concat( | |||
| ) # get end time from current file | |||
| dfs_i.Close() | |||
|
|
|||
| if keep == "average": | |||
| if i == 0: | |||
| # For first file, write all timesteps normally | |||
There was a problem hiding this comment.
Isn't it better to calculate the timestep where the overlap occurs, calcuate the average, write that and then the rest, instead of writing the entire contents and then trying to overwrite?
There was a problem hiding this comment.
Agree. I would suggest an algorithm like this (file1 starts before file2):
- open file1
- write file1 timesteps until start of file file2
- open file2
- write overlapping steps by with alpha*data1 + (1-alpha)*data2 until end of file1
- close file1
- write rest of file2
- close file2
using the alpha notation above it will be super simple to later add a keep="linear" that linear mixes the two in the overlapping period.
There was a problem hiding this comment.
don't shoot the messenger, I chat'gpted this solution due to my lack of mikecore competences
License badges
I got it semi working, but not yet.
I need help with mikeCore I think.
I basically managed to create a concatenated file, but the overlap is not being written properly, and after the overlap, I think the second file is being written but with a messed up index.