2022Q4 Scenario prep exits with warnings

[AlexAxthelm]

From a recent run of 2022Q4 Scenario prep:

Warning message:
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat) 
INFO [2024-04-16 10:16:33] Finished processing scenarios.

likely relates to RMI-PACTA/pacta.scenario.data.preparation#29 (comment)

[cjyetman]

also relevant ticket on private ADO is here https://dev.azure.com/RMI-PACTA/2DegreesInvesting/_workitems/edit/10608
though it's "Done", so probably a new related ticket needs to be opened (but the necessary details are there) @jdhoffa

ℹ️ ADO ticket was re-opened with the relevant problem here as an acceptance criteria

[cjyetman]

@AlexAxthelm for detail...

In the file pactarawdata/scenario-sources/00Imported/WEO2022 extended data/WEO2022 extended data/raw_data_from_provider/used_in_pacta.scenario_preparation/IEA-EV-dataEV salesCarsProjection-APS.csv

currently Line 2 ends with ,https://www.iea.org/reports/global-ev-outlook-2022/executive-summary

for it to be a valid CSV, ,https://www.iea.org/reports/global-ev-outlook-2022/executive-summary needs to be added to every line after that (L3:L49)

and technically the file should end with a new line

---

whether you want to modify the existing file in place, or create a new modified file and update the config here I leave up to you

[AlexAxthelm]

I think my preferred solution would be to not touch the file itself, but deal with it as part of reading. Maybe we could use read.table(filename, fill = TRUE)?

[cjyetman]

I suppose I could adapt the workflow here to read in plain text and then pass that on to pacta.scenario.data.prep to process. FYI though... Antoine added that string to Line 2, so it's not like it's truly a "raw" file.

[jdhoffa]

TBH, given that the column that we are concerned with was added manually to begin with, seems like it makes more sense just to fill it in properly... (or remove it entirely?)

[cjyetman]

will be resolved by:

• deal with the malformed CSV text from IEA-EV-dataEVsalesCarsProjection-APS.csv pacta.scenario.data.preparation#46
• import and pass the raw text of weo_2022_electric_sales_aps_auto_raw_full_filename #49

[AlexAxthelm]

Closing, as I have altered the source file to include the column in all rows.