Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/consider short term params when clipping post-lapse stability #257

Merged
merged 3 commits into from
Dec 10, 2024

Conversation

L-M-Sherlock
Copy link
Member

@L-M-Sherlock L-M-Sherlock commented Dec 9, 2024

refer to:

benchmark result:

Model: FSRS-rs-new
Total number of users: 9999
Total number of reviews: 349923850
Weighted average by reviews:
FSRS-rs LogLoss (mean±std): 0.3276±0.1524
FSRS-rs RMSE(bins) (mean±std): 0.0517±0.0333
FSRS-rs AUC (mean±std): 0.7003±0.0780

Weighted average by log(reviews):
FSRS-rs LogLoss (mean±std): 0.3532±0.1694
FSRS-rs RMSE(bins) (mean±std): 0.0710±0.0461
FSRS-rs AUC (mean±std): 0.6991±0.0884

Weighted average by users:
FSRS-rs LogLoss (mean±std): 0.3565±0.1719
FSRS-rs RMSE(bins) (mean±std): 0.0739±0.0478
FSRS-rs AUC (mean±std): 0.6983±0.0904

parameters: [0.4364, 1.1666, 3.2263, 15.8143, 7.1405, 0.5257, 1.7751, 0.0102, 1.5151, 0.1304, 1.0091, 1.9183, 0.1034, 0.3016, 2.3447, 0.2315, 3.0099, 0.4436, 0.6239]

Model: FSRS-rs-old
Total number of users: 9999
Total number of reviews: 349923850
Weighted average by reviews:
FSRS-rs-1 LogLoss (mean±std): 0.3271±0.1521
FSRS-rs-1 RMSE(bins) (mean±std): 0.0512±0.0331
FSRS-rs-1 AUC (mean±std): 0.7014±0.0778

Weighted average by log(reviews):
FSRS-rs-1 LogLoss (mean±std): 0.3528±0.1691
FSRS-rs-1 RMSE(bins) (mean±std): 0.0705±0.0460
FSRS-rs-1 AUC (mean±std): 0.7001±0.0885

Weighted average by users:
FSRS-rs-1 LogLoss (mean±std): 0.3561±0.1716
FSRS-rs-1 RMSE(bins) (mean±std): 0.0734±0.0478
FSRS-rs-1 AUC (mean±std): 0.6992±0.0905

parameters: [0.4127, 1.1488, 3.1878, 15.8143, 7.1333, 0.5271, 1.7733, 0.0084, 1.5148, 0.1191, 1.003, 1.9051, 0.1122, 0.2962, 2.3266, 0.2272, 3.0122, 0.5077, 0.6404]

@L-M-Sherlock L-M-Sherlock changed the title Fix/consider short term params when clipping pls Fix/consider short term params when clipping post-lapse stability Dec 9, 2024
@L-M-Sherlock L-M-Sherlock requested a review from dae December 9, 2024 10:51
@L-M-Sherlock
Copy link
Member Author

@dae, do you plan to release a trivial version for Anki 24.11? If so, please consider to include this update.

L-M-Sherlock added a commit to open-spaced-repetition/srs-benchmark that referenced this pull request Dec 10, 2024
@L-M-Sherlock L-M-Sherlock merged commit 359744c into main Dec 10, 2024
3 checks passed
@L-M-Sherlock L-M-Sherlock deleted the Fix/consider-short-term-params-when-clipping-PLS branch December 10, 2024 04:04
@dae
Copy link
Collaborator

dae commented Dec 14, 2024

So far there hasn't been any bugfixes that need to go out quickly. I'll merge this into main, and if we need to get another bugfix release out quickly, I can cherry-pick this into the branch (please remind me if I forget)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants