Problem processing a large solution file (.zip)

[jhoncr]

process_solution(location_solution)
*** rplexos debug: - Solution: 'results.zip'
*** rplexos debug: Reading XML file and saving content
*** rplexos debug: PLEXOS version: '7.400 R01 Gold '
*** rplexos debug: rplexos version: '1.1.10'
*** rplexos debug: Adding extra tables to the database
*** rplexos debug: t_key has 462844 rows
*** rplexos debug: temp_key has 462844 rows
*** rplexos debug: Creating final database and adding basic data
*** rplexos debug: Creating data tables and views
*** rplexos debug: Reading interval binary data
*** rplexos debug: Max position: 1785730104
*** rplexos debug: Adjusted max position: 1785730104
*** rplexos debug: Sum of length: 1785822360
*** rplexos debug: Sum of adjusted length: 1785822360
Warning: Problem with length of 't_key_index' for period 0
in file 'results-temp.db'
*** rplexos debug: 1785729360 values read
Error: Problem reading interval binary data (reached end of file).
8760 values requested, 3360 returned.

[danielsjf]

Yes I had this problem as well for some DBs. Not sure yet what the problem might be. Could you share the db? I cannot guarantee that I can solve it though. But if you want to investigate it yourself, I can always accept a pull request.

[danielsjf]

Could also be related to #27 if you're on a Unix system.

[jhoncr]

Ok, I'm running it on a Win64 environment. Where is the warning coming from? Do you have the same warning proceeding the same problem?

[danielsjf]

The error message is generated from the code here:

rplexos/R/process_solution.R

Line 272 in de84e8b

stop("Problem reading ", period.name, " binary data (reached end of file).\n",

I am not the original developer of the code, but this is how I understand it. In the Plexos solution.zip, you have a few files. One is an xml, the others are binary files for the various aggregations (interval, day, week, month, year,...). Rplexos first reads the xml. This xml also contains a mapping for the binary files. The binary files are read in different chunks (and immediately written to the sqlite db). These chunks are identified with the time, collection and item (e.g. data_interval_generator_generation) and each chunk forms one table in the final sqlite db.

What is basically happening here is that rplexos tries to read a chunk of the binary file (which should be in that position according to the xml), but then the file ends before the full chunk was read. This could be due to a corrupt Plexos solution file or an edge case in which there is a bug in rplexos.

Unfortunately, I don't have a db anymore that produces the error. It was a former colleague that encountered the problem.