Strange covar module in MixedSingleTaskGP

[K-Kielak]

Its pydocs (https://github.com/pytorch/botorch/blob/98503e4db839dcf92e60fa0ed8e5917ba6223239/botorch/models/gp_regression_mixed.py#L36:L38) say that its kernel is of the form:

K((x1, c1), (x2, c2)) = K_cont_1(x1, x2) + K_cat_1(c1, c2) + K_cont_2(x1, x2) * K_cat_2(c1, c2)

That makes sense. Based on the above I would expect the code to look something like:

sum_kernel = MaternKernel(...) + CategoricalKernel(...)
prod_kernel = MaternKernel(...) * CategoricalKernel(...)
covar_module = ScaleKernel(sum_kernel + prod_kernel)

However, the code https://github.com/pytorch/botorch/blob/98503e4db839dcf92e60fa0ed8e5917ba6223239/botorch/models/gp_regression_mixed.py#L152:L180 is a bit more complicated and I don't really understand why, nor can I see any consistent pattern. Namely, it does:

# sum_kernel broken into two steps to avoid brackets hell
sum_kernel = MaternKernel(...) + ScaleKernel(CategoricalKernel(...))
sum_kernel = ScaleKernel(sum_kernel)
prod_kernel = ScaleKernel(MaternKernel(...) * CategoricalKernel(...))
covar_module = sum_kernel + prod_kernel

The most confusing part is that the CategoricalKernel from sum_kernel has two layers of ScaleKernel on top of it, isn't there a mistake with one set of brackets reaching too far, wasn't the idea to do:

sum_kernel = ScaleKernel(MaternKernel(...)) + ScaleKernel(CategoricalKernel(...))
prod_kernel = ScaleKernel(MaternKernel(...) * CategoricalKernel(...))
covar_module = sum_kernel + prod_kernel

or

sum_kernel = ScaleKernel(MaternKernel(...)) + CategoricalKernel(...))
prod_kernel = ScaleKernel(MaternKernel(...) * CategoricalKernel(...))
covar_module = sum_kernel + prod_kernel

?

Especially that both Matern and Categorical kernel have default output scale of [0, 1]. I'm happy to open a PR that fixes it, just not sure if it actually is an issue or I'm just missing something.

[saitcakmak]

Especially that both Matern and Categorical kernel have default output scale of [0, 1].

In GPyTorch, the kernels do not have an output scale. The ScaleKernel is used to add an output scale to any given kernel. So, that's where that bit comes into play.

Onto the structure of the sum kernel: This is actually equivalent to sum_kernel = ScaleKernel(MaternKernel(...)) + ScaleKernel(CategoricalKernel(...)) in theory. Since the output scale in ScaleKernel is a trained parameter, there isn't a difference between the two. The original reads

sum_kernel = scale_1 * (matern + scale_2 * categorical) = scale_1 * matern + scale_1 * scale_2 * categorical

If you define scale_3 = scale_1 * scale_2, you get the other definition.

I don't know why it was implemented one way vs the other. If I had to guess, I'd say that the current implementation produced better results during model training (they're equivalent in theory, but numerical optimization is another story...).

[K-Kielak]

In GPyTorch, the kernels do not have an output scale. The ScaleKernel is used to add an output scale to any given kernel. So, that's where that bit comes into play.

Everything has some sort of output scale in a literal meaning of this word. What I meant is that both Matern and Categorical will always output values within the [0, 1] range so there's no need to use assymetric scale kernel to scale them to comparable values. I get that ScaleKernel is used in GPyTorch to add learnable outputscale.

Onto the structure of the sum kernel: This is actually equivalent to sum_kernel = ScaleKernel(MaternKernel(...)) + ScaleKernel(CategoricalKernel(...)) in theory. Since the output scale in ScaleKernel is a trained parameter, there isn't a difference between the two. The original reads...

Of course! Not sure how I missed that one. I'm still curious about the reasoning though, even if it's just for numerical optimization. I'll have to think about this one though 🤔