LTR Baseline

A collection of LTR baselines using various algorithms from publicly availble implementations.

Datasets

• Yahoo LTR Challenge
• MSLR
• Istella

Results

Tables are orderd by NDCG@10. Statistical significance is indicated using a pairwise t-test with Bonferroni correction. Entries marked with * and ^ indicate 95% and 99% confidence intervals respectively. Comparisons are relative to LightGBM.

Yahoo! Set 1

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
GBDT	XGBoost	0.7685^	0.6409^	0.4600	0.4745	0.4783	0.7439	0.7858	0.8272
LambdaRank	LightGBM	0.7653	0.6379	0.4616	0.4761	0.4799	0.7425	0.7845	0.8253
LambdaRank	jforests	0.7650	0.6377	0.4615	0.4760	0.4798	0.7431	0.7842	0.8256
LambdaMART	QuickRank	0.7645	0.6372	0.4603	0.4749	0.4787	0.7408	0.7827	0.8237
X-DART	QuickRank	0.7582^	0.6332^	0.4546^	0.4695^	0.4735^	0.7237^	0.7688^	0.8124^

Yahoo! Set 2

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
GBDT	XGBoost	0.8465	0.7493	0.4422	0.4587	0.4651	0.7381	0.7746	0.8296
LambdaRank	LightGBM	0.8406	0.7446	0.4453	0.4613	0.4677	0.7389	0.7735	0.8285
LambdaRank	jforests	0.8396	0.7443	0.4439	0.4600	0.4665	0.7360	0.7714	0.8274
LambdaMART	QuickRank	0.8361	0.7412	0.4421	0.4580	0.4646	0.7327	0.7662	0.8228
X-DART	QuickRank	0.8311	0.7382	0.4415	0.4575	0.4642	0.7223	0.7580	0.8173

MSLR-WEB10K

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
LambdaRank	LightGBM	0.6886	0.6353	0.3548	0.3735	0.3815	0.4724	0.4913	0.5225
GBDT	XGBoost	0.6917^	0.6398^	0.3513*	0.3702*	0.3784*	0.4694*	0.4893	0.5213
LambdaRank	jforests	0.6832^	0.6306^	0.3536	0.3724	0.3804	0.4672^	0.4865^	0.5180^
LambdaMART	QuickRank	0.6771^	0.6250^	0.3470^	0.3663^	0.3745^	0.4582^	0.4782^	0.5102^
X-DART	QuickRank	0.6738^	0.6201^	0.3458^	0.3647^	0.3729^	0.4552^	0.4721^	0.5023^

MSLR-WEB30K

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
LambdaRank	LightGBM	0.6970	0.6434	0.3698	0.3881	0.3959	0.4865	0.5046	0.5354
GBDT	XGBoost	0.6989^	0.6472^	0.3651^	0.3836^	0.3914^	0.4847	0.5034	0.5349
LambdaRank	jforests	0.6938^	0.6406^	0.3680^	0.3864^	0.3942^	0.4843^	0.5025^	0.5335^
LambdaMART	QuickRank	0.6894^	0.6358^	0.3636^	0.3821^	0.3899^	0.4773^	0.4942^	0.5243^
X-DART	QuickRank	0.6834^	0.6294^	0.3568^	0.3754^	0.3834^	0.4660^	0.4824^	0.5125^

Istella-S LETOR

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
LambdaRank	LightGBM	0.8026	0.6213	0.7512	0.7566	0.7572	0.7000	0.7614	0.8215
LambdaRank	jforests	0.8033	0.6221^	0.7479	0.7532	0.7538	0.6983	0.7612	0.8207
LambdaMART	QuickRank	0.8030	0.6215	0.7416^	0.7469^	0.7475^	0.6921^	0.7559^	0.8157^
GBDT	XGBoost	0.8157^	0.6299^	0.7311^	0.7373^	0.7379^	0.6798^	0.7490^	0.8120^
X-DART	QuickRank	0.7821^	0.6055^	0.7215^	0.7281^	0.7289^	0.6613^	0.7263^	0.7914^

Istella LETOR

Ranker	Framework	[email protected]	[email protected]	ERR@5	ERR@10	ERR@20	NDCG@5	NDCG@10	NDCG@20
LambdaRank	LightGBM	0.7454	0.5717	0.7569	0.7625	0.7635	0.6684	0.7165	0.7745
LambdaRank	jforests	0.7442	0.5715	0.7529*	0.7587*	0.7596*	0.6634^	0.7133^	0.7718^
X-DART	QuickRank	0.7397^	0.5664^	0.7478^	0.7536^	0.7547^	0.6594^	0.7068^	0.7650^
LambdaMART	QuickRank	0.7395^	0.5665^	0.7466^	0.7524^	0.7535^	0.6564^	0.7049^	0.7634^
GBDT	XGBoost	0.7576^	0.5796^	0.7347^	0.7411^	0.7422^	0.6468^	0.7015^	0.7624^