Further efficiency increases

[edwardchalstrey1]

Branches

• adjust-loading just does some of the below, see in progress below
• Processing speedups - June 6th 2024 #159 actually has significant speedups by other means than the below, simply reducing for loopage etc

TODO 6th June

• Finish Processing speedups - June 6th 2024 #159
• Finish adjust-loading
  • Hide loading indicators judiciously
  • If the slider moves, run map_view_one_year again
    • can we update this such that if the displayYear != enteredYear whilst the world_map_all fetch is in progress, we exit that fetch to run the map_view_one_year again before running the world_map_all fetch again
      • In progress - adjust-loading sort of does this, but it doesn't quite work as expected, keep fiddling and see if it could work, bit don't spend long and otherwise
        • remove this idea and merge progress on this branch from before b3c4732a2bd69776fe7ce288d8fef836c5fbe92d into Processing speedups - June 6th 2024 #159

TODO

• Refactor such that assign_variables_to_shapes and assign_categorical_variables_to_shapes are called after the shapes are added to the map. It would require a separate django view and django object which could be a dictionary with the shape id as the key and the var info as the value
  • First, refactor so that there is a view function called map_view_all_with_vars that gets run last and overwrites the shapeData instead of adding to it as the others do. The other views should not include variables.
    • Ensure the selected polity is retained
    • Ensure the variables dropdown is populated
    • Ensure the categorical variables dropdown is populated
    • The variables dropdown should be greyed out until this has loaded.
    • The previous variable button should not be displayed until after everything has loaded and should be the local stored variable
• Write structured doc on this idea including the below
• Time things
  • Model -> view
  • View completion Processing speedups - June 6th 2024 #159
  • View -> JS
• Set the initial polityBorderWeight in Python, not JS, to avoid looping though dict before page load running the plotPolities() function is instant, nothing to improve here
• Is it faster on a GPU machine? I don't think this is relevant since the live site won't have it
• Can we load points faster than shapes?
• Could we store the results of the full map view as a temporary JSON file which gets loaded if it exists already instead of using the view? Edit: perhaps try this on a separate branch from the stuff above
  • If so, this might be what we want to use parquet/arrow or something fast for
• Speed up assign_variables_to_shapes Update: this has already been sped up via refactoring so further speed increases may not be needed
  • Alternatively load these as a separate thing to the shapes and suppress at first that's pointless because the issue is running the function, it shouldn't slow down the Python=>JS transition - yes but you could have a separate view that runs this and updates the shapes?
  • Try just not doing it for the first year and suppress the variable changer until all shapes loaded
    • Problem is that the shapes loaded for the initial year don't have the colour info, so we'll need to make sure that gets added in
• Speed up categorical variables function
• Look into GDAL/GEOS versions used for Seshat (maybe older versions are slower on VM) #154
• Time the loading of the non-simplified shapes on the VM, is it any slower?

[edwardchalstrey1]

Bounding box loading idea

What if the initial map view was zoomed in and we load the shapes in the viewed bounding box first (first one year, then across all years), then allow zooming out after it has loaded

[edwardchalstrey1]

Could we use GeoArrow in Django?

My Q:

Would it be better to/ is it possible to somehow use shape data directly from file, rather than the db. Perhaps the logic in the view should be moved to the data loading command script, creating a new file. Could the new file be in more efficient "GeoArrow" instead of GeoJSON?

Copilot answer:

Apache Arrow (GeoArrow) is a binary format and is not directly readable in a HTML template. You would need to convert it to a format that can be read in the template, such as GeoJSON.
Remember, this approach is not recommended for larger files, as it can lead to performance issues. It's generally better to use a view to handle the file and pass only the necessary data to the template.

For using GeoJSON, basically you would go via a view as we are