RFC Configuration Management

[hlee9212]

Migrated from previous repository. The RFC was placed on hold and the PR was kept open in the previous repository. No need to review now, this PR is not intended to be merged until we decide to implement.

[vkakerbeck]

I thought the process was that an RFC is merged once it is accepted? That doesn't mean that we have to implement it right away or that the author has to implement it (https://thousandbrainsproject.readme.io/docs/request-for-comments-rfc#implementing-an-rfc). I think a good next step would be for @nielsleadholm to review this RFC and then for us to have a meeting to talk through whether we want to change our config setup how you propose (we can set the timeline for it later). There is no rush to do decide this right now, I just wanted to point out that per our RFC process we can merge as soon as we decide that we want this. We don't have to wait until we have someone assigned to implement it right away.

[nielsleadholm]

Thanks again for putting this together Hojae. Following our in-person discussion about the RFC, it sounded like the biggest missing feature at the moment is verification of the configs - basically checking that a config is valid, before going into an actual experiment and potentially getting an error raised at that point.

My understanding is Hydra itself does not necessarily provide that much in the way of checking for misconfigurations - e.g. that two particular parameters are incompatible. However, this could be a great use-case of the post_init method in data-classes.

The main feature Hydra might bring then is hierarchical structuring of configs, but again I think this is something we can already do with inheritance in data-classes.

Finally, other features like accepting experiment arguments via the command-line could also be supported through appropriate parsing of arguments passed by the user using standard Python.

As such, my current impression is that using data-classes more could address our requirements, while avoiding adding a new dependency/library. But I think we agreed that it would be useful to get feedback from everyone (including anyone in the community) running experiments, to better understand what the most acute pain-points are, and then we can decide on the best solution. It's definitely the case that we want to make our configs easier to use and build-on, and this RFC is a great way to explore how we do that.

[vkakerbeck]

Sounds good, that makes sense. And yes, that was my impression as well, that so far it is unclear what advantage Hydra gives that we can't achieve with our Python configs (even though we want to improve those for sure!). I like this plan to wait and see about feedback once we have more people using the code :)