DiD questions for an experiment involving group splitting

[nicktgr15]

Hi

As a follow-up to the discussion I had with @PhilippBach I'm posting a few questions I have regarding an ongoing analysis.

Context:
I'm running an experiment where we split a set of routes into two groups (A and B) and then apply an intervention on group B routes only.

Questions:

1. Since we do the route splitting into two groups I'm wondering if we don't need the propensity score model in our case (i.e. no selection bias) and we should set score='experimental' - is that assumption correct?
2. With cross sectional data how to decide how granular we want to be with time (using days at the moment)? e.g. if we go down to 1 second resolution we'll probably get very few events per row.
3. How to evaluate RMSE figures for the chosen model and parameters? What's "good enough" for a DiD analysis? Should we check the fit summary and coefficients with a few different types of models/params?
4. We have a few categorical features which we'd like to use with a catboost regressor as it handles categorical features without any pre-processing but it looks like DoubleMLData doesn't support non-numerical features, so we used one-hot-encoding. Ideally we'd like to avoid that, is there any workaround to use categorical features with catboost in double ml DiD without pre-processing?
5. Finally, we use sensitivity analysis in a very basic way with the default params i.e. .sensitivity_analysis(cf_y=0.04, cf_d=0.03). That works nicely out of the box, and when we get zero included in the interval we follow up with benchmarking but so far we haven't been able to identify (through trial and error) any covariates with strong confounding effect. Is there anything else we can try in that space? (I understand that in some cases it might be impossible to identify the unobserved confounder because it might be not feasible, not available in the data etc.)

Cheers,
Nik

[SvenKlaassen]

Hi @nicktgr15,

1. Yes, only the proportion of treated units will be used to update the "propensity score"/ treatment probability.
2. That really depends on the use case.
3. Again, this is use case dependent and how noisy your observations are. Yes, I would suggest to check e.g. linear- and nonlinear learners and try some different combinations.
4. Sadly not. The only option would be to use external predictions (see https://docs.doubleml.org/stable/guide/learners.html#advanced-external-predictions)
5. I would try to compare the model to a model with score=observational and a corresponding sensitivity analysis. Remark that the current version of experimental scores might give too small sensitivity estimates (Rework the DID Sensitvitiy  #262).

Best,
Sven