Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for libyt #209

Open
wants to merge 144 commits into
base: main
Choose a base branch
from
Open

Conversation

matthewturk
Copy link
Contributor

This adds support for libyt to enzo. This week in San Diego, @cindytsai will talk more about it, and she can say more here, but for posterity here are a few bits:

  • This adds a libyt configuration option; outside of that, it should not change compilation
  • libyt, counterintuitively, does not require yt, but instead manages calling Python (3) for any purpose, supplying the appropriate information from Enzo. Python is not directly linked, but instead connected via libyt.
  • This enables interactive mode for libyt as well.
  • This can be considered a replacement for the inline python calls, using libyt to manage all the data etc. It manages parallelism, etc, and avoids copies of data at all costs.

There are a few things left, but as it stands this can be run:

  • Enzo-side derived fields like temperature currently are not implemented, as a result of needing to delete the fields afterward
  • Particle fields currently need to be supplied using function pointers to libyt, which isn't directly mappable to Enzo's hierarchy system, as it would require a global lookup table for grids. I'm working on a way around this.
  • the previous inline python system has calling conventions that aren't yet implemented, like N subcycles, calling at top/bottom of hierarchy, etc
  • there are a few outstanding PRs needed to the primary libyt repo and the main yt repo

@clairekope clairekope self-requested a review February 21, 2024 17:18
Copy link
Contributor

@clairekope clairekope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't yet reviewed the brand new source files but I wanted to submit my review so my comments about the docs & where the in situ analysis is called can be viewed & discussed by all. Once the comments from @dcollins4096 and I have been addressed I want to try running the in situ analysis myself before approving, as I noted in a comment on the new doc page. Super excited about this feature and I want to make sure its broadly accessible!

doc/manual/source/user_guide/InSituPythonAnalysis.rst Outdated Show resolved Hide resolved
* **How to activate in situ Python analysis process?**

The full process is encapsulated inside ``CallInSitulibyt`` function in ``src/enzo/CallInSitulibyt.C``.
We can put this function everywhere we want in Enzo to start in situ analysis.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing uses of CallInSitulibyt are indeed only in EvolveLevel and after the recursive loop. Given that the simulation is supposed to finish evolution before handing over data, this makes sense. I don't think CallInSitulibyt should (much less can) be added to any other location because of this as it would be unwise to allow data to be passed to in situ analysis while the level hierarchy is still being evolved. Though, see my notes on EvolveLevel.C for my thoughts on its current placement.

Overall, I don't think the docs should advertise that CallInSitulibyt can be added anywhere else than where it already is.

Comment on lines +849 to +855
#ifdef USE_LIBYT
LCAPERF_START("CallInSitulibyt");
//CallEmptyInteractivePython(LevelArray, MetaData, level, 0);
CallInSitulibyt(LevelArray, MetaData, level, 0);
LCAPERF_STOP("CallInSitulibyt");
#endif

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is CallInSitulibyt called before flux correction? Would it be better to call it from EvolveHierarchy? It's already being called after EvolveLevel's recursive loop.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I did this exclusively to mimic the behavior of the existing python in situ stuff. I think as long as it's able to be called from either the top or bottom of the W cycle it's fine to call it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will calling it before flux correction have any strange consequences on cells at the boundaries of grids with different resolution? I know yt will automatically use data at the finest resolution (unless you do a lot of manual intervention)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure, nor which would be more desired. I think I was attempting to replicate as best I could the interpolated output machinery.

src/enzo/global_data.h Show resolved Hide resolved
doc/manual/source/user_guide/InSituPythonAnalysis.rst Outdated Show resolved Hide resolved
doc/manual/source/user_guide/InSituPythonAnalysis.rst Outdated Show resolved Hide resolved
If an error occurred when running inline Python functions or Enzo detects ``LIBYT_STOP`` file, then it will enter reloading script phase.
Document about ``yt_run_ReloadScript`` is `here <https://libyt.readthedocs.io/en/latest/libyt-api/yt_run_reloadscript.html>`__.

* **How to reload a script?**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me why I would want to use this feature; an example use case would really help. Even just a few sentences.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed.

doc/manual/source/user_guide/InSituPythonAnalysis.rst Outdated Show resolved Hide resolved

// TODO: yt_run_Function and yt_run_FunctionArguments
// Put yt_run_Function and yt_run_FunctionArguments here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @dcollins4096; ideally I'd like to try running the in situ analysis myself before approving this PR. That way I can identify gaps in the docs.

@cindytsai
Copy link
Contributor

cindytsai commented Mar 7, 2024

Hi @clairekope and @dcollins4096,

The majority of the comments are fixed, except these:

Why is CallInSitulibyt called before flux correction? Would it be better to call it from EvolveHierarchy? It's already being called after EvolveLevel's recursive loop.

Will calling it before flux correction have any strange consequences on cells at the boundaries of grids with different resolution? I know yt will automatically use data at the finest resolution (unless you do a lot of manual intervention)

  • Where should Enzo call CallInSitulibyt.

The docs are all up-to-date, I would be really happy if you could give me some feedback about libyt!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants