MDS

[msluszniak]

Closes #163

Currently, because we call isotonic regression inside defn I changed an implementation of isotonic regression so it would be able to runing MDS. Ofc, commenting whole sections of code is temporal solution and I'm wondering what we should do with it.

[josevalim]

Do we need to preprocess? Because if we need to preprocess, then the best approach is likely to give the result of IsotonicRegression as an input to MDS.

[msluszniak]

The preprocessing is generally needed for points that have the same x coordinate, so in real cases there are probably not much of these cases. The problem with passing results of IsotonicRegression is that this function is called multiple times inside while loop

[josevalim]

So I think we can isotonic regression fit and then run the linear regression ourselves?

[msluszniak]

I think I don't understand the concept

[josevalim]

What I mean is that you defined a defn special_preprocess in IsotonicRegression that does what you need. :)

[msluszniak]

Yeah, that's probably the best idea

[msluszniak]

I don't know why the test results here are so strange. I mean, I know that when I invoke x() and key() directly inside the EXLA.jit_apply then I will achieve the same results as here, but they aren't reasonable (in contradiction to what I achieved locally calling x and key before)

[josevalim]

@msluszniak what exactly is strange about it?

[msluszniak]

For example, when I choose a smaller epsilon :eps (even veeery small) the result does not change in this setup as if there was a perfect convergence after only 5 or 6 steps, which is rather impossible

[msluszniak]

Locally, the response for smaller epsilon was as expected: much more iterations and smaller stress loss. And they correspond to what sklearn implemenation returns.

[josevalim]

@msluszniak I see. In this case I can't help much, it seems to be related to the algorithm. :( maybe print_value can help debug.

[msluszniak]

Yeap, there is definitely something wrong with smacof procedure since it's mathematically proven that the stress shouldn't increase but currently sometime it does for bigger number of samples

[msluszniak]

Actually I debug the Sklearn implementation and there is the same behaviour here, so maybe there is nothing wrong here 😅

[msluszniak]

Ok, I found a problem. The problem was in pairwise distance. For a function that calculates pairwise distance on a single tensor, there was only a check for negative values (that occures sometimes because of numerical instability). However, there is also a check needed for positive values on the main diagonal. Ideally, there should be only zeros, but because of numerical instability sometimes there are positive values that in my specific case cause that loss explode to huge values

[msluszniak]

@josevalim why on one CI two test fails and on the other three? And why do even these tests fail? 😅

[josevalim]

@msluszniak for what is worth, it is very common to have precision differences across machines and compiler versions. You will have to either increase the tolerance of the matrix comparison or test based on a property (I don't know if there is such a property for MDS).

[msluszniak]

Ok, I understand. Unfortunately, there is no such property in MDS. This algorithm is very sensitive to any numerical changes, and the final result will be completely different (but also correct, only embedded in different space). I guess that we may flag these tests as @to_skip or something like that because I think it's good to have them just in case.

[josevalim]

So how do you know MDS is correct? :D

[msluszniak]

Actually, via visualization and empirical check. Even Sklearn has barely any tests for MDS https://github.com/scikit-learn/scikit-learn/blob/093e0cf14aff026cca6097e8c42f83b735d26358/sklearn/manifold/tests/test_mds.py. We can check the stress majorization procedure like they do, but that's all

[josevalim]

Sounds like I plan to me.

[msluszniak]

I think it's ready to go

[josevalim]

💚 💙 💜 💛 ❤️