Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use UTC for daily stats calculations #2359

Merged
merged 9 commits into from
Sep 9, 2021
Merged

Use UTC for daily stats calculations #2359

merged 9 commits into from
Sep 9, 2021

Conversation

eatyourgreens
Copy link
Contributor

@eatyourgreens eatyourgreens commented Aug 17, 2021

Ignore the local time zone and use UTC when calculating your daily classification counts. With this change, the daily stats chart should always begin on a Monday, regardless of which time zone you're in. Classifications should be shown on the day that you made them, not the day before.

Screenshot of the daily stats chart with mock data, in the project storybook.
http://localhost:9001/?path=/story/project-app-screens-classify-daily-classifications-chart--plain

I've also upgraded the chart component to remove inject, and upgraded the stories to remove storiesOf.

Package:
app-project

Closes #2244.

Review Checklist

General

  • Are the tests passing locally and on Travis?
  • Is the documentation up to date?

Components

Apps

  • Does it work in all major browsers: Firefox, Chrome, Edge, Safari?
  • Does it work on mobile?
  • Can you yarn panic && yarn bootstrap or docker-compose up --build and app works as expected?

Publishing

  • Is the changelog updated?
  • Are the dependencies updated for apps and libraries that are using the newly published library?

Post-merging

@eatyourgreens eatyourgreens added the bug Something isn't working label Aug 17, 2021
@eatyourgreens eatyourgreens requested a review from srallen August 17, 2021 10:11
@eatyourgreens eatyourgreens force-pushed the daily-stats-utc branch 3 times, most recently from 2a51a1e to ac41fc6 Compare August 17, 2021 10:46
@eatyourgreens
Copy link
Contributor Author

https://local.zooniverse.org:3000/projects/eatyourgreens/-project-testing-ground/classify/workflow/3223?env=staging is a Yes/No workflow on staging, which can be used to check today's classification count. I don't think daily stats are stored for staging projects.

const locale = counterpart.getLocale()
const stats = thisWeek.map(({ count: statsCount, period }) => {
const day = new Date(period)
const isToday = day.getUTCDay() === TODAY.getDay()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could put the day numbers and local names for each day in the store, so that each daily count is { count, dayNumber, label, longLabel, period }. They never change, so only need to be generated once. Maybe something for a future rewrite of this component?

Copy link
Contributor

@srallen srallen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately none of the tests for the DailyClassificationChartContainer pass running in Chicago CDT time. Everything is a day later compared to the expectation it looks like. Do you want a copy of the test failure outputs?

@srallen
Copy link
Contributor

srallen commented Aug 25, 2021

It's 11:22 AM if that helps.

@eatyourgreens
Copy link
Contributor Author

I think these lines in the tests are generating the local day name in Chicago, at midnight UTC.

const dayNameShort = day.toLocaleDateString('en', { weekday: 'narrow' })
const dayNameFull = day.toLocaleDateString('en', { weekday: 'long' })

@eatyourgreens
Copy link
Contributor Author

@srallen I've fixed the expected day names, so the tests should pass now. I've changed the wording to make it explicit, which day is being tested for each date. This is what you should see:

Component > DailyClassificationsChartContainer
    ✓ should render without crashing
    ✓ should render the `DailyClassificationsChart` component
    daily stats counts
      ✓ should exist for the entire week
      2019-09-30
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Monday
        ✓ should have an accessible description
      2019-10-01
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Tuesday
        ✓ should have an accessible description
      2019-10-02
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Wednesday
        ✓ should have an accessible description
      2019-10-03
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Thursday
        ✓ should have an accessible description
      2019-10-04
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Friday
        ✓ should have an accessible description
      2019-10-05
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Saturday
        ✓ should have an accessible description
      2019-10-06
        ✓ should have a count
        ✓ should have a period
        ✓ should have a short label
        ✓ should be Sunday
        ✓ should have an accessible description

Copy link
Contributor

@srallen srallen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still seeing failures, having to do with mismatches between Saturday and Sunday:

1) Component > DailyClassificationsChartContainer
       daily stats counts
         2019-10-05
           should have a count:

      AssertionError: expected 25 to equal 8
      + expected - actual

      -25
      +8

      at Context.<anonymous> (src/screens/ClassifyPage/components/YourStats/components/DailyClassificationsChart/DailyClassificationsChartContainer.spec.js:121:33)
      at processImmediate (internal/timers.js:464:21)

  2) Component > DailyClassificationsChartContainer
       daily stats counts
         2019-10-05
           should have an accessible description:

      AssertionError: expected 'Saturday: 25' to equal 'Saturday: 8'
      + expected - actual

      -Saturday: 25
      +Saturday: 8

      at Context.<anonymous> (src/screens/ClassifyPage/components/YourStats/components/DailyClassificationsChart/DailyClassificationsChartContainer.spec.js:137:31)
      at processImmediate (internal/timers.js:464:21)

  3) Component > DailyClassificationsChartContainer
       daily stats counts
         2019-10-06
           should have a count:

      AssertionError: expected 15 to equal 25
      + expected - actual

      -15
      +25

      at Context.<anonymous> (src/screens/ClassifyPage/components/YourStats/components/DailyClassificationsChart/DailyClassificationsChartContainer.spec.js:121:33)
      at processImmediate (internal/timers.js:464:21)

  4) Component > DailyClassificationsChartContainer
       daily stats counts
         2019-10-06
           should have an accessible description:

      AssertionError: expected 'Sunday: 15' to equal 'Sunday: 25'
      + expected - actual

      -Sunday: 15
      +Sunday: 25

      at Context.<anonymous> (src/screens/ClassifyPage/components/YourStats/components/DailyClassificationsChart/DailyClassificationsChartContainer.spec.js:137:31)
      at processImmediate (internal/timers.js:464:21)

Ignore the local time zone and use UTC when calculating your daily classification counts. With this change, the daily stats chart should always begin on a Monday, regardless of which time zone you're in. Classifications should be shown on the day that you made them, not the day before.
Use UTC when getting day names for dates from the stats server.
Today's day number should be calculated from the local clock. Day numbers for stats data should use UTC.
Calculation of the daily counts is a bit hard to follow. This might help clarify it.
@eatyourgreens
Copy link
Contributor Author

eatyourgreens commented Sep 7, 2021

That's weird. sinon.useFakeTimers({ now: new Date('2019-10-06'), toFake: ['Date'] }) is setting the day to Saturday when you run the tests, but not when I run the tests here (in BST) or in CI (in UTC.) My guess would be that it should use UTCDate. I'll try that.

EDIT: So the fix seems to be to run the mock clock from noon on 6th Oct 2019, which should be 7am Sunday in CST?

Don't localise the expected day names. They should be the same, regardless of which time zone and language the tests are running in.
Run the mock clock at noon.
Copy link
Contributor

@srallen srallen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest updates resolved it. Thanks!

@github-actions github-actions bot added the approved This PR is approved for merging label Sep 9, 2021
@eatyourgreens
Copy link
Contributor Author

Cool! Let's see if this resolves any issues for people who are seeing their classifications plotted on the wrong day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved This PR is approved for merging bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Daily count stats bug
2 participants