[backport, r24.11] Make scikit-learn optional again (#596)

[backport, r24.11] Make scikit-learn optional again (#596) #296