Experimental code generation

[brandomr]

This issue is related to code generation using obscure/new modeling and simulation frameworks that won't be as well understood as, say, Pandas 🙄

For example, if given the entirety of Oceananigans.jl documentation and a paper whose model/simulation relies on Oceananigans, can GPT-4 recreate the simulation code referenced in the paper ("ground truth").

Note this example may not be ideal since the code uses outdated Oceananigans.jl paradigms.

Additional areas to explore might be:

• whether the whole code base for the library can be ingested into context and/or used as part of a RAG approach
• whether stackexchange or other user provided forums can be leveraged for assistance in the generation process (e.g. GH discussions or GH wiki

[brandomr]

via @ryanholtschneider2:

An update to this issue--we are now shifting focus to look at auto generation of model configurations (e.g. MITGCM or GraphCast).

• Extracting Config Components - Given a code repo extract any command line arguments and configuration files (and variables). This should be easy and should save the annotator time and potentially prevent annotation mistakes.
• Find run command(s): Given a code repo determine the command(s) needed to run the model. This may be more difficult depending on the complexity of the repo and saves less time than extracting config components, probably less important.
• Installation: Find and (optionally run/verify/debug) installation of the package and dependencies. This may be difficult or near impossible in some cases and trivial in others. Either way, it could save some headaches but not too much time generally.

Probably higher hanging fruit with a lot of corner cases.

• Annotate Output File: Given a code repo or files which generate the output file or print to console, create a natural language description/summary of the output for the user. Depending on how we want to interface the LLM and the simulation with the user, this may be useful, what are your thoughts?

For acting as a natural language interface my thoughts are as follows:
This is very similar to other LLM applications where we are trying to translate information from a conversation or single shot request from the user and translate it to choice of tool and inputs to a tool (or code gen, etc..). This portion is necessary and most likely the most helpful and time saving for the user. It is also likely one of the easiest if we have properly defined a dictionary describing potential configuration options during the repo analysis task. However there is the issue of "needing" an expert to be able to "properly" run the model which we could attempt to address by making suggestions to the user or inform ing them of configuration variables which likely need to be changed given their request. This may be difficult to do but would be quite satisfying and very useful.

[ryanholtschneider2]

Extracting configuration, output and run command info working pretty well on auto-config. Take a look at https://github.com/DARPA-ASKEM/auto-config for a demo video, code and a system diagram