Quantifying memorization progress

[SplittyDev]

Hey, what would be the best way to quantify the progress in order to tell the user something like "Your mastery of this card is x%"?

At first I thought the retrievability $R$ would be a good candidate because $0 \leq R \leq 1$. But even though it's a good value to turn into a percentage due to its normalized nature, it doesn't accurately reflect the "mastery" of the card, since the recall will always be high shortly after a review session.

I then thought that $S$ could be a good value to use, but it's not bounded or normalized in any sensible way that would make it suitable for quantifying the learning progress.

For lack of a better option, I was calculating the "mastery" value as $M = \frac{\max(S, 365)}{365}$ and assuming that $S \geq 365 \implies R > 0.9$ is good enough to qualify as having mastered the card, even though it's not technically correct.

I've now switched to calculating mastery as retrievability(for date: now + 1 year) (pseudocode), since the numbers seem more sensible to me than the previous stability method, but it still suffers from some of the same issues.

Since it's quite important to show the user a sensible approximation of their progress for the sake of motivation, it would be great to know how to best approach this.

[Expertium]

I'm not sure if showing the user a measure of "mastery" is a good idea to begin with. FSRS is already quite complex, stability and retrievability are confusing for a lot of people. I would say that stability itself is a good measure of mastery. Also, it's technically bounded: in FSRS, the upper limit is 36500 days aka 100 years; though I'd argue that 10 years is already mastery. I don't doubt that it's possible to come up with a clever metric that uses clever math, I doubt that most users' reaction would be anything other than "Oh, it's another mysterious FSRS number, whatever".

Btw, while this isn't what you're looking for, there is a metric that can be compared across different people: knowledge acquisition rate. Install the Helper add-on and Shift + Left Mouse Click on Stats to find it.

[SplittyDev]

Thanks for the answer!

I'm developing a language self-learning app, which uses FSRS as the backing algorithm.

My app has a "word bank" with a list of all words, and these words have attached FSRS cards, but that's just an implementation detail the user neither knows nor cares about. FSRS and the card system are used for flashcard review, but the users don't concern themselves with the details of that. They just go to the review tab and go through the cards.

Since my app is targeting casual learners and not exactly the same crowd that uses Anki with custom plugins, settings, algorithms and parameters, I want the users to have some kind of idea how their learning is going.

Not showing anything isn't gonna make them feel like they're making progress, and gamifying this experience a bit is surely gonna help people stay motivated in the long run. Seeing continual progress is always a good motivator.

I've switched to a tiered metric now:

• First, use the 1-year retrievability to show progress.
• Once that reaches 90%+, give the user an "upgrade" to the next tier and use the 2-year retrievability
• Same thing again, once it reaches 90%, user gets an "upgrade" to 5-year retrievability
• And finally, once that is done, the final mastery tier is reached, which uses the 10-year retrievability

The progress is nicely and colorfully rendered next to each word, so the user can see "this word has a nice and green badge with two stars, and I'm close to reaching the third", gamifying the experience and keeping them motivated to learn more.

The flashcard system in the app will keep working regardless of the calculated mastery. The user will always be able to learn more and improve their retrievability way beyond the app's visual 10-year cap, but I strongly believe that the system is gonna help people keep the learning up and not give up because they don't feel any progress.

You could compare this system to Memrise, which uses the growing stages of plants to signify progress, or Duolingo, which uses a star-system. Each of those systems might not perfectly quantify the user's knowledge/memory journey, but they're a big factor in why these apps are popular.

I'm just trying to find a system that works well with FSRS and gives the user a decent approximation of their progress, ideally without gaslighting them into thinking they've "mastered" a word when they really really haven't.

I'd argue that having 90%+ successful recall probability in 10 years from now is quite the good indicator of learning progress, but using that as the initial number would discourage users, since they barely see any progress. The tiered system is the best I could come up with so far, but I'd appreciate different ideas a lot.

In the end, I'm just trying to make learning fun, in addition to being efficient and actually helping the user on their language learning journey without actively and purposefully misleading them (as some other apps are certainly doing).

[Expertium]

I see. Yeah, that's a different situation. I like your idea with tiered retrievability, maybe I'll try to think of something too.

Btw, in your app will parameters be optimized automatically and periodically?

[SplittyDev]

I'm thinking of doing that, but I'll have to look a bit deeper into how much that actually helps and how hard it'll be to implement.

I've already done the work of porting the Swift algorithm to FSRS 4.5 from 4.0 (and upstreamed it), but porting the optimizer too might be a little bit too much for me at the moment 😄

From what I have gathered so far, I'd assume that the default parameters are good enough^TM for the average user, since they're derived from a fairly large data set. I'm also not sure how much additional efficiency there is to be gained from doing automatic optimization, or if it could even hurt and result in regressed performance in the long term.

[ishiko732]

Currently, the fsrs-browser can be used for parameter optimization in webview that support wasm and worker, but I'm not sure if Swift can be embedded in the webview to execute. In my tests, both Safari and Chrome were able to run successfully. Currently, it is still a bit difficult to simplify the calling of parameter optimization on the web. If you want to understand the implementation details, you can check the related code in the ts-fsrs-demo. https://github.com/ishiko732/ts-fsrs-demo/tree/main/src/components/train

[Expertium]

or if it could even hurt and result in regressed performance in the long term.

No, that's very unlikely. The more reviews the user does, the more data there is for the optimizer to work with, the better it can adapt to this user. It's the lack of optimization that would hurt, especially in cases where there is plenty of review data (like, tens of thousand of reviews) and all that data is unused. According to the benchmark, optimized FSRS works better than unoptimized FSRS pretty much universally.
Also, FSRS-5 is out.

[joshdavham]

Late to this conversation, but I think that trying to find a reasonable-ish way to quantify memory progress in a way that learners feel that they understand is very important. I specifically work in the field of "immersion learning" and one of the biggest pain points of immersion learners is feeling like they're not making progress, so finding a way to show them that they actually are making progress is super duper important.

It was actually a funny realization I had about a year ago, but it was that one of the primary functions of anki for an immersion learner is gamification. They get to see their numbers go up, they get to graduate cards and they can even get some pretty graphs. Without that, they feel like they're not making progress.

[L-M-Sherlock]

Here is a repo you may be interested in:

https://github.com/xiety/AnkiHistoryVisualization