Refactor IO Directory Processing and Compression Logic #63
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First, the directory processing has been completely redone to replace the outdated and unmaintainable code used for old directory processing. Now, it simply invokes process_single_file.
Second, the logic has been changed so that it no longer creates a new folder named "foldername-(de-)compressed". Instead, (de-)compressed files are now placed in the same folder.
Third, I've reverted to the previous logic; if a file is unsupported (for example, raw), the code shouldn't stop. Instead, the file should be skipped.
Note: All the changes were discussed in meeting with Carlos.
P.S. The integration tests are really great. I've modified the directory tests due to the change in logic.
P.S. As discussed with Carlos, if the changes are deemed not beneficial enough and/or if the trade-offs(such as returning Options instead of just results from the readers) are deemed significant enough, I'll close this PR and eliminate all the directory processing