Causal discovery API

[robertness]

Discussion thread for the causal discovery API. The proposed API is here.

[petergtz]

Hey @robertness, thanks for writing all these things up and also iterating on your proposal. I'm curious about a sketch of the implementation of learn_graph. Would it look something like this:

def learn_graph(context: Context, method: Discoverer=SomeDefaultDiscoverer(...)]) -> MixedGraph:
    return method.discover(context)

If so, and if the reason to provide this, is to provide an easy-to-use function to beginners, I'm wondering how much value it adds.

BTW, this could even be achieved by this shorter version:

learn_graph = SomeDefaultDiscoverer(...).discover

However, let me try and introduce a different angle on this: In the same way that ML libraries don't provide "default nets", I wonder if a different way to go would be to help beginners through the new user guide. Just like e.g. PyTorch introduces beginners to ML by setting up a neural net through a sophisticated set of functions, we could do the same and start the discovery chapter of the DoWhy User Guide by introducing the simplest discovery algorithms first and then move through the different kinds explaining each one's advantages and disadvantages.

My main point suggesting this is that I'm not sure, giving a beginner a learn_graph function is helpful. That person won't understand how reliable the results are, because the person won't understand under which condition a certain algorithm performs well and under which it doesn't.

Coming back to a suggestion that @emrekiciman pointed out recently, where learn_graph actually exercises a whole set of methods and can then combine the results, or find out which result is the best, and actually return something that has a high likelihood of being correct, would probably add a lot of value, even to beginners. It's basically the AutoGluon of causal discovery. But I'm not sure we're there yet in terms of reliability of the methods.

But maybe I'm thinking in the wrong direction here. If so, I'd appreciate further insight.

[robertness]

Hi @petergtz , sorry for the late reply, I'm now getting back to full speed after travel and other obligations.

Just like e.g. PyTorch introduces beginners to ML by setting up a neural net through a sophisticated set of functions, we could do the same and start the discovery chapter of the DoWhy User Guide by introducing the simplest discovery algorithms first and then move through the different kinds explaining each one's advantages and disadvantages.

I think we should do this for developers who are interested in paying the price of learning more about the algorithms in order to achieve better results.

However, I would suggest that for the majority of modelers, we would want to take a different tacked. Here are my arguments:

There is no explicit representation of domain knowledge and inductive biases in neural network frameworks. The only thing you can do is learn various elements of architecture and configurations of the training procedure. Only after you learn these things can you build an intuition for the inductive bias a given architecture proffers on a given problem.

In contrast, causal models not only make it possible to explicitly represent domain knowledge and other explicit representations of inductive bias, they require it for provable identifiability.

Note, this is not to say I want to prevent developers from seeing the algorithms and their details, I just don't want knowledge of these algorithms to be required for making use of these tools. Once a developer is onboarded, is comfortable using this technology with their problem, and once to achieve further optimal results, they should be able to follow tutorials that help them explore more sophisticated modeling.

I also have in mind some failure modes I've observed with researchers encountering causal discovery for the first time. Often, they don't have a clear idea of the causal problem they want to solve in their head, and turn to causal discovery as a place to start.
Then they try out a causal discovery method, it returns results that don't align with their assumptions, then they go down some rabbit-hole of trying to find the algorithmic "philosopher's stone" of causal discovery algorithms that solves all their problems. Meanwhile, they manage to avoid defining the ultimate problem the need to solve.

Alternatively, they know their problem, and causal discovery is just a first step. But they still get mired down in causal discovery's algorithmic imperfections.

So with the proposal, we'd like to see people be able to achieve a "quick and dirty" causal discovery, adding it as a key component in a complete workflow. Then, giving them the ability improve that workflow.

And as @amit-sharma and @emrekiciman suggest, we want to give them the ability to compare and contrast different algorithms without having to understand them in depth. I'll add a todo to add a comparable results section to the API in the same spirit of DoWhy's estimators.

[amit-sharma]

thanks for the updates on the API @robertness . I think it is getting there :)

Coming back to a suggestion that @emrekiciman pointed out recently, where learn_graph actually exercises a whole set of methods and can then combine the results, or find out which result is the best, and actually return something that has a high likelihood of being correct, would probably add a lot of value, even to beginners. It's basically the AutoGluon of causal discovery. But I'm not sure we're there yet in terms of reliability of the methods.

agree, this will be the version of learn_graph that we can aspire to. Meanwhile, I think that we can already have a useful learn_graph function that analyzes the context provided by a user and applies the method that uses the context in the best way. For example, as a start, we can restrict it simply by the requirements/capabilities of the individual algorithms. The motivation is that a new user may bring with them different context pieces that they have assembled, and it will be easy if dowhy automatically chose a method a supported all parts of their context. If such an algorithm exists, we can select it as default, if not, we can raise an Error.

@petergtz would this be a good usecase for including this function?

[petergtz]

Sounds good, @amit-sharma 👍

[robertness]

I like this idea of learn_graph choosing a default based on context @amit-sharma. I created an issue.

[robertness]

I've begun adding issues to the dodiscovery library and will update this page as these abstractions become more clear.