FEAT Add QuadraticHessian datafit

[tvayer]

Objectives

The overall objective is to make a fast solver for $\ell_1$ penalized quadratic program based on Anderson CD. Precisely,

• Add a datafit corresponding to the loss $x \to \frac{1}{2} x^\top A x + \langle b, x\rangle$
• Create a $\ell_1$ penalized QP solver $\min_x \frac{1}{2} x^\top A x + \langle b, x\rangle + \lambda |x|_1$ based on AndersonCD

What has been done:

• In datafits: A HessianQuadratic that implements the corresponding loss
• In estimators: A L1PenalizedQP that solves the $\ell_1$ penalized QP

Usage and comparison with the Lasso:

import numpy as np
from skglm.estimators import L1PenalizedQP, Lasso
import scipy as sp

n = 100
d = 3
G = np.random.randn(n, d)
A = G.T @ G + np.eye(d)
b = np.random.randn(d)
U, s, Vh = sp.linalg.svd(A, full_matrices=False)
S = np.diag(s)
A_half = U @ np.diag(s**(0.5)) @ Vh
A_minus_half = U @ np.diag(s**(-0.5)) @ Vh
X = np.sqrt(d)*A_half # to get the same problem we find the corresponding X and y
y = -np.sqrt(d)*A_minus_half @ b 

alpha = 1e-1
qp_solver = L1PenalizedQP(alpha=alpha, fit_intercept=False, max_iter=500)
lasso = Lasso(alpha=alpha, fit_intercept=False, max_iter=500)

qp_solver.fit(A, b)
lasso.fit(X, y)

print(f'{qp_solver.coef_=}') # same results
print(f'{lasso.coef_=}')

To do

• Documentation
• Test functions
• Write examples

[mathurinm]

I removed the dedicated estiamtor in order to limit the numbers of estimators we implement ; you can use it with GeneralizedLinearEstimator:

    pen = L1(alpha)
    solv = AndersonCD(warm_start=False, verbose=2, fit_intercept=False)
    qpl1 = GeneralizedLinearEstimator(QuadraticHessian(), pen, solv).fit(A, b)

For now it will only be compatible with AndersonCD, and (by design) does not support fitting an intercept.