Add support for Observables to MESA

[quaquel]

see #2281

This PR adds various new classes to MESA. The key public-facing ones are, at the moment, Observable, ObservableList, and HasObservables. Observable is a descriptor that is used to declare a given attribute to be observable. Being observable means changing the attribute using the assignment operator = willl result in the sending of a signal. Callbacks can subscribe to this signal. ObservableList is a proof of principle on how the idea of an observable can be expanded to collections. Future extensions could include, e.g., ObservableDict. HasObservables is a mixin that can be used with e.g., Model or Agent. It is required if you want to use Observable or ObservableList. HasObservable contains the key logic for subscribing to signals and the emitting of signals.

Observable emits a change signal when its attribute value is changed via the assignment operator (i.e., =). ObservableList at the moment defines five signal types: "replace", "remove", "insert", "append", "change". "change" is emmited when the entire list is changed via the assignment operator (i.e., =). The other signals are tied to the respective list operations. Currently, signals are following the approach used in traitlets. The main reason for making the signals identical to traitlets is that it should make integration with solara easier, which already supports traitlets.

The emitted signals are instances of an AttributeDict. This is in line with Traitlets. It is just a dict with attribute-style access to the fields. By default, a signal has four fields: "owner", "type", "old", and "new". It is possible to add additiona fields if the user wants. For example, the ObservableList signals (accept "change") all also include "index".

I have also added a WIP Computed and Computable approach. In javascript-style signals, but, e.g., also in solara, a Computed is a callable that is dependent on one or more signals. When it receives a signal that any of its dependencies has changed, it will mark itself as being outdated. It will only update its value if it is called to do so. This can be used to set up a hybrid push/pull structure where Obervables push signals, but a Computed is only recalculated on a pull if it knows its underlying data has changed. A Computed is also Observable, so it will emit a signal if it knows it is outdated. This allows the chaining of observables and computed's.

API

Below, there is a quick sketch of the resulting API. We define Obervable attributes at the class level. We can use the attribute normally in the rest of the code.

class MyAgent(Agent, HasObservables):
    wealth = Observable()
    income = Observable() 

   def __init__(self, model)
       super().__init__(model)
       self.wealth = 5  # we can use wealth as a normal attribute, Observable hides all the magic

class MyModel(Model, HasObservables):
    gini = Computable()
    agent_wealth = ObservableList()

    def __init__(self, random=random):
        super().__init__(random=random)
        
        agent_wealth = [agent.wealth for agent in self.agents]  # ObservableList can be used as a normal list
        self.gini = Computed(calculate_gini, agent_wealth)

model = MyModel()
agent = MyAgent(model)

# whenever wealth is changed some_callback_function will be called
agent.observe("wealth", "change", some_callback_function)

# Whenever either income or wealth is changed, generic_callback_function
# is called. All() is a new helper class
agent.observe(All(), "change", generic_callback_function)

# observables is a class-level attribute with all observables defined for the class
print(agent.observables)

Implementation Details

After a long discussion in #2281, I decided to implement everything from scratch. Existing libraries like traitlets or psygnal add too much overhead because they are not tailored to the specific use case of ABMs with potentially many updates to an attribute.

future extensions

This PR puts in place the basic machinery, but there are ample options for further improvements:

• At the moment, anything that wants to emit a signal needs to be registered as being observable. Ideally, something like model.agents should emit agent_added and agent_removed signals. At the moment, there is no machinery in place to make it easy to add this functionality. I don't think it is overly difficult, but I want to take a closer look soon.
• I currently include only an ObservableList as an example of how we can use observables to also observe changes in collections. Other collections can easily be added if and when desired.
• The current implementation of Computed is heavily inspired by signals and Solara. It involves the autodiscovery of dependencies. The params library, instead, forces users to pass the list of dependencies explicitly. I am not sure which approach is better.
• I don't like the current API for Computed. It requires the user to do 2 things: declare an attribute as being computable at the class level and assign a computed instance to this attribute during object initialization. If I fix the first bullet, it might be possible to improve this API and eliminate the Computable declaration.
• The way in which the signals that a given object can emit is declared is less than ideal. Observable and ObservableList both have a list of strings that enumerate the signals that can be sent. However, there is no explicit declaration anywhere of the fields that are part of any signal. A signal is an instance of an AttributeDict, following traitlets, but it is completely up to the user to decide what fields will be in it. This means that the user is dependend on the docstring, or some test code to discover what is in a signal, rather than that it can be declared and checked explicitly via e.g., a type checker.
• At the moment, the notify method, which is responsible for emitting the signals requires the name of the observable (name), the old value (old), the new value (new), and the signal type (type). Further elements can be added as desired. For example, most signals from ObservableList include index. However, for e.g., an append signal, having old in there makes little sense.

[EwoutH]

Looks really interesting! I will review it in detail tomorrow, and try to play with it.

[EwoutH]

Could you update or rebase this branch on main?

[quaquel]

A quick update: I started adding tests. The Observable and HasObservable are now fully tested. Still need to do the test for computed/computable and for the SignalList.

[github-actions]

Performance benchmarks:

Model	Size	Init time [95% CI]	Run time [95% CI]
BoltzmannWealth	small	🔵 -2.1% [-3.5%, -0.6%]	🔵 -2.6% [-2.8%, -2.4%]
BoltzmannWealth	large	🔵 -0.9% [-1.5%, -0.3%]	🔵 -3.8% [-5.8%, -1.8%]
Schelling	small	🟢 -3.6% [-4.0%, -3.3%]	🔵 -1.8% [-2.1%, -1.4%]
Schelling	large	🔵 -3.0% [-3.4%, -2.6%]	🟢 -5.7% [-8.2%, -3.2%]
WolfSheep	small	🔵 -0.3% [-0.9%, +0.3%]	🔵 -0.5% [-1.1%, +0.2%]
WolfSheep	large	🔵 -3.4% [-4.5%, -2.0%]	🟢 -9.1% [-12.3%, -5.1%]
BoidFlockers	small	🔵 -2.3% [-3.2%, -1.3%]	🔵 -0.9% [-2.0%, +0.2%]
BoidFlockers	large	🔵 -1.3% [-2.0%, -0.4%]	🔵 -0.8% [-1.6%, -0.1%]

[EwoutH]

I didn't review every line of code, but I would like to see this move forward. The user API is clean enough, considering what's possible in Python, and it's all in the experimental space.

Thanks for working on this, curious where it will go!

[EwoutH]

Would an ObservableDict (also) be useful, to be able to track which Agent the value originated from?

[quaquel]

An ObservableDict would indeed also be useful. Not sure what you mean with

to be able to track which Agent the value originated from

[tpike3]

I think I see what @EwoutH means here where it would be something like {agent.unique_id: <observables>}

[tpike3]

Just curious why this is here? Did we maybe want to add this to the docs, so users have an example to follow? Maybe subset of the getting started and experimental how to??

[quaquel]

This will be removed before merging, was just there during development. Good catch!

[tpike3]

This is awesome! I think as the develops this and the datacollection update will inextricably intertwined.

Let us know what you want to complete before we merge.

[quaquel]

Let us know what you want to complete before we merge.

I want to do a bit of further code cleaning (like the tests.py file) and improve code coverage a bit further. Beyond that, any feedback or ideas are welcome here or in #2281.

[Corvince]

Not directly related to this PR, but I just watched a video about alien-signals, a new signals library for JavaScript. Its supposed to be very fast and will probably be used by Vue.js. Vue.js is interesting because it previously switched from a push-based system to a pull-based system and alien-signals is now push-pull-based. So maybe that is interesting for you @quaquel.

[quaquel]

Thanks, I'll take a look. There might be useful ideas that we can borrow.

[EwoutH]

Congratulations on merging this!