output a ZIP with JSON and CSV versions of all outputs

[cjyetman]

• closes feat: compress all output json files into a archive.zip #27

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 24 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (adebb07) to head (61b784f).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
R/prepare_pacta_dashboard_data.R	0.00%	24 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main     #36   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         21      21           
  Lines        997    1021   +24     
=====================================
- Misses       997    1021   +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jdhoffa]

I get this error when running locally:

workflow.pacta.dashboard-1  | zip error: Nothing to do! (outputs/data_archive.zip)

[cjyetman]

I get this error when running locally:

workflow.pacta.dashboard-1  | zip error: Nothing to do! (outputs/data_archive.zip)

how does one "run locally" now? like, what exactly does that mean?

let me explain better so it doesn't sound snarky....

if one were to load/install the workflow.pacta.dashboard and run workflow.pacta.dashboard:::zip_outputs("test") where "test" was an empty directory, than I wouldn't be surprised with a message like that, though I can imagine a discussion following that about implementing more robust error checking even though it's a minor, internal utility function. likewise if some larger, fuller run process is happening but is also not fully setup, e.g. if ultimately it does not create any files in the output directory and therefore there's nothing for zip_outputs() to zip

if it's happening in an environment that you're sure there are outputs in the directory defined by output_dir in the prepare_pacta_dashboard_data(), then I'd appreciate some more detail so I can reproduce, because that would definitely be weird

[jdhoffa]

how does one "run locally" now? like, what exactly does that mean?

I have a .env file that contains:

analysis_output_dir: /PATH/TO/pacta_analysis_outputs
benchmarks_dir: /PATH/TO/pacta-data/2023Q4
dashboard_data_dir: /PATH/TO/json_assets_outputs 

and call:

docker compose up --build

[cjyetman]

I have a .env file that contains:

analysis_output_dir: /PATH/TO/pacta_analysis_outputs
benchmarks_dir: /PATH/TO/pacta-data/2023Q4
dashboard_data_dir: /PATH/TO/json_assets_outputs 

hmmm, ok I don't really know how that works and don't see it documented anywhere... I'll try and see if I can run it somehow and maybe get a similar result as to what you're getting

[jdhoffa]

And for flavour, when I do this from main, I get:

I have a .env file that contains:

analysis_output_dir: /PATH/TO/pacta_analysis_outputs
benchmarks_dir: /PATH/TO/pacta-data/2023Q4
dashboard_data_dir: /PATH/TO/json_assets_outputs 

hmmm, ok I don't really know how that works and don't see it documented anywhere... I'll try and see if I can run it somehow and maybe get a similar result as to what you're getting

Created a new issue for documentation 😊

[jdhoffa]

if it's happening in an environment that you're sure there are outputs in the directory defined by output_dir in the prepare_pacta_dashboard_data(), then I'd appreciate some more detail so I can reproduce, because that would definitely be weird

And for reference, when I run the same thing (same .env file and everything) from main I get an output directory with expected datasets:

 lsd  --group-dirs first -A ~/Downloads/test_dashboard_outputs/
 data_company_bubble.json         data_techexposure_company_companies.json
 data_emissions.json              data_techexposure_company_portfolio.json
 data_emissions_pie_bonds.json    data_techmix_sector.json
 data_emissions_pie_equity.json   data_trajectory_alignment.json
 data_exposure_stats.json         data_value_pie_bonds.json
 data_included_table.json         data_value_pie_equity.json
 data_techexposure.json  

and I get an identical output when running from this branch, so the zipping doesn't seem to be happening

[cjyetman]

if it's happening in an environment that you're sure there are outputs in the directory defined by output_dir in the prepare_pacta_dashboard_data(), then I'd appreciate some more detail so I can reproduce, because that would definitely be weird

And for reference, when I run the same thing (same .env file and everything) from main I get an output directory with expected datasets:

 lsd  --group-dirs first -A ~/Downloads/test_dashboard_outputs/
 data_company_bubble.json         data_techexposure_company_companies.json
 data_emissions.json              data_techexposure_company_portfolio.json
 data_emissions_pie_bonds.json    data_techmix_sector.json
 data_emissions_pie_equity.json   data_trajectory_alignment.json
 data_exposure_stats.json         data_value_pie_bonds.json
 data_included_table.json         data_value_pie_equity.json
 data_techexposure.json  

and I get an identical output when running from this branch, so the zipping doesn't seem to be happening

ok, I'll have to wait until I can run it the way you're running it because I can't replicate any of that. If I just run the functions themselves without any fancy stuff, they work as expected, which makes me suspect that something is different in that environment with the directories it is pointing or, possibly that the Docker environment is resistant to the function making a temp directory? 🤷🏻

[jdhoffa]

We can pair and have a look tomorrow