Ingestion path structure breaking changes and shared lib re-factoring

• Added ability to archive data files. To activate this feature, set the Cloud Function environment variable ARCHIVE_FILES=false.
• Added support for new directory structure.
• Added validation logic.
• Re-factored common shared library.
• Integrated e2e test for file processing to mocha unit test.