Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --handicap-rank-difference-{small,19x19} options #51

Merged
merged 1 commit into from
Dec 17, 2023

Conversation

dexonsmith
Copy link
Contributor

@dexonsmith dexonsmith commented Dec 16, 2023

Add options to compute the effective handicap rank difference, incorporating board size, komi, and the point value of handicap stones.

This hits a math domain error when tallying because the head start for black is sometimes enormous, such as 8 stones on a 9x9 board.

Processing OGS data
   2,739,321 /   30,811,698 games processed.  192.3s remainingTraceback (most recent call last):
  File "/Users/dexonsmith/Repos/online-go/goratings/analysis/analyze_glicko2_one_game_at_a_time.py", line 99, in <module>
    tally.add_glicko2_analytics(analytics)
  File "/Users/dexonsmith/Repos/online-go/goratings/analysis/util/TallyGameAnalytics.py", line 102, in add_glicko2_analytics
    self.prediction_cost[size][speed][rank][handicap] += - math.log(result.expected_win_rate if black_won else 1 - result.expected_win_rate)
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: math domain error

Since the options are off-by-default, it'd fine to land as-is and then decide what to do. Or we could fix first.

Here are the choices for what to do (either before or after this lands):

  • Update the tallying code to check for 0 before calling math.log().
  • Clamp effective rank between (e.g.) 40kyu and 15dan
  • Stop rating games with a handicap rank difference bigger than (e.g.) 9
  • Some combination above the above.

Relates to #45. Replaces #46.

Add options to compute the effective handicap rank difference, incorporating
board size, komi, and the point value of handicap stones.

This hits a math domain error when tallying because the head start for black is
sometimes enormous, such as 8 stones on a 9x9 board.

```
Processing OGS data
   2,739,321 /   30,811,698 games processed.  192.3s remainingTraceback (most recent call last):
  File "/Users/dexonsmith/Repos/online-go/goratings/analysis/analyze_glicko2_one_game_at_a_time.py", line 99, in <module>
    tally.add_glicko2_analytics(analytics)
  File "/Users/dexonsmith/Repos/online-go/goratings/analysis/util/TallyGameAnalytics.py", line 102, in add_glicko2_analytics
    self.prediction_cost[size][speed][rank][handicap] += - math.log(result.expected_win_rate if black_won else 1 - result.expected_win_rate)
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: math domain error
```

Since the options are off-by-default, it'd fine to land as-is and then decide
what to do. Or we could fix first.

Here are the options:

- Update the tallying code to check for 0 before calling `math.log()`.
- Clamp effective rank between (e.g.) 40kyu and 15dan
- Stop rating games with a handicap rank difference bigger than (e.g.) 9
- Some combination above the above.
@anoek
Copy link
Member

anoek commented Dec 17, 2023

Stop rating games with a handicap rank difference bigger than (e.g.) 9

I suspect this is the answer, especially for some of those games where the effective rank difference is computed to be something like 52 or whatnot. When that happens your going to have the effective player ratings be something like 1000 vs 5000, and that's fine so long as the 5000 wins (which should net basically 0 points to both parties), but if they do something like resign that game for whatever reason, all of a sudden you're going to extremely tank one and vault the other a ridiculous amount, not to mention probably jacking up their RD's high so as to further increase their rating volatility going forward. I suspect this is what's leading to those math domain errors.

So yeah I don't think considering those types of games is going to do anything but harm the rating pool. If we want to experiment with considering some past 9 but maybe not all the way to 50+, that might be something to consider, but intuitively I wouldn't expect many of those games to be beneficial.

@anoek anoek merged commit 70c4d78 into online-go:master Dec 17, 2023
@dexonsmith dexonsmith deleted the handicap-rank-difference branch January 11, 2024 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants