✨ wizard: anomalist (first draft) #3363

lucasrodes · 2024-10-03T08:25:36Z

First implementation of: #3340

Re-organisation of grapher db/s3 util functions.
Draft of anomalist page in Wizard.

owidbot · 2024-10-03T08:30:07Z

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-wizard-anomalies

chart-diff: ✅

2/2 reviewed charts

Modified: 2/2
New: 0/0

data-diff: ❌ Found differences

= Dataset garden/artificial_intelligence/2024-02-05/chess
  = Table chess
    ~ Column elo_rating (changed data)
        ~ Changed values: 5 / 39 (12.82%)
                         entity  year  elo_rating -  elo_rating +
          Computer chess engine  1994          2321          2314
          Computer chess engine  2009          3237          3227
          Computer chess engine  2010          3237          3227
          Computer chess engine  2011          3237          3236
          Computer chess engine  2023          3591          3586
= Dataset garden/artificial_intelligence/2024-06-28/ai_bills
  = Table ai_bills
~ Dataset garden/war/2024-08-26/ucdp
-   -   This dataset provides information on armed conflicts, using data from the UCDP Georeferenced Event Dataset (version 24.1), the UCDP/PRIO Armed Conflict Dataset (version 24.1), and the UCDP Battle-Related Deaths Dataset (version 24.1).
    ?                                                                                                                        ^                                                    ^                                                          ^
+   +   This dataset provides information on armed conflicts, using data from the UCDP Georeferenced Event Dataset (version 23.1), the UCDP/PRIO Armed Conflict Dataset (version 23.1), and the UCDP Battle-Related Deaths Dataset (version 23.1).
    ?                                                                                                                        ^                                                    ^                                                          ^
  = Table ucdp_country
  = Table ucdp
  = Table ucdp_locations
~ Dataset garden/war/2024-08-26/ucdp_prio
-   -   This dataset provides information on armed conflicts, using data from the UCDP Georeferenced Event Dataset (version 24.1), the UCDP/PRIO Armed Conflict Dataset (version 24.1), and the UCDP Battle-Related Deaths Dataset (version 24.1).
    ?                                                                                                                        ^                                                    ^                                                          ^
+   +   This dataset provides information on armed conflicts, using data from the UCDP Georeferenced Event Dataset (version 23.1), the UCDP/PRIO Armed Conflict Dataset (version 23.1), and the UCDP Battle-Related Deaths Dataset (version 23.1).
    ?                                                                                                                        ^                                                    ^                                                          ^
  = Table ucdp_prio
= Dataset garden/who/2024-09-09/flu_test
  = Table flu_test
    ~ Dim country
+       + New values: 1 / 71581 (0.00%)
                date country
          2024-09-16  Mexico
-       - Removed values: 23 / 71581 (0.03%)
                date   country
          2024-09-23     India
          2024-09-23 Indonesia
          2024-09-23   Jamaica
          2024-09-16   Vietnam
          2024-09-23   Vietnam
    ~ Dim date
+       + New values: 1 / 71581 (0.00%)
          country       date
           Mexico 2024-09-16
-       - Removed values: 23 / 71581 (0.03%)
            country       date
              India 2024-09-23
          Indonesia 2024-09-23
            Jamaica 2024-09-23
            Vietnam 2024-09-16
            Vietnam 2024-09-23
    ~ Column denomcombined (new data, changed data)
+       + New values: 1 / 71581 (0.00%)
          country       date  denomcombined
           Mexico 2024-09-16            482
-       - Removed values: 23 / 71581 (0.03%)
            country       date  denomcombined
              India 2024-09-23            113
          Indonesia 2024-09-23             24
            Jamaica 2024-09-23              5
            Vietnam 2024-09-16             50
            Vietnam 2024-09-23             21
        ~ Changed values: 92 / 71581 (0.13%)
             country       date  denomcombined -  denomcombined +
           Argentina 2024-08-26              115              105
              Brazil 2024-05-20             7369             7326
              Brazil 2024-06-10             7006             6979
              Brazil 2024-07-22             6140             6130
          Costa Rica 2024-08-19              724              681
    ~ Column pcnt_poscombined (new data, changed data)
+       + New values: 1 / 71581 (0.00%)
          country       date  pcnt_poscombined
           Mexico 2024-09-16          5.394191
-       - Removed values: 23 / 71581 (0.03%)
            country       date  pcnt_poscombined
              India 2024-09-23         23.008850
          Indonesia 2024-09-23          8.333333
            Jamaica 2024-09-23         20.000000
            Vietnam 2024-09-16         32.000000
            Vietnam 2024-09-23         38.095238
        ~ Changed values: 104 / 71581 (0.15%)
            country       date  pcnt_poscombined -  pcnt_poscombined +
          Argentina 2024-06-10           24.615160           24.248644
             Brazil 2024-08-12            4.376248            4.378633
            Jamaica 2024-08-05           16.666666           16.326530
               Mali 2024-09-09           20.000000           18.367348
            Vietnam 2024-01-22           31.081081           32.432434


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Edited: 2024-10-09 10:35:32 UTC
Execution time: 4.22 seconds

lucasrodes · 2024-10-03T17:23:35Z

implemented the foundations of the app.

next tasks:

create plot interface (entity→ country)
add button to generate anomaly list, per indicator
- can we have sth to list those we want to create lists for? checkboxes per container; maybe too long. multiselect at first + individual buttons? ー second option sounds better
- individual button action should be kept in fragment ー avoid whole page rerun
think plot for upgrade mode
export anomalies somewhere, maybe indicator metadata?

lucasrodes · 2024-10-04T16:04:23Z

Managed to use Grapher to chart indicators (user might want to see the time series if an anomaly is detected)

Currently working on streaming the output from OpenAI and render streamlit objects.

Relevant links:

Structured output: https://platform.openai.com/docs/guides/structured-outputs/supported-schemas
Streaming structured output: https://community.openai.com/t/streaming-using-structured-outputs/925799/8
intro to structured output: https://cookbook.openai.com/examples/structured_outputs_intro

Marigold · 2024-10-08T08:38:58Z

etl/grapher_model.py

@@ -1251,6 +1277,24 @@ def override_yaml_path(self) -> Path:
        """Return path to indicator YAML file."""
        return self.step_path.with_suffix(".meta.override.yml")

+    def get_data(self, session: Optional[Session] = None) -> pd.DataFrame:


There's already a function variable_data_df_from_s3 for reading data from S3, you can reuse it.

* wip * db -> db_utils * io -> db * move things db_utils -> db * db -> grapher_io * db -> grapher_io, db_utils -> db * docstring * db_utils -> db * wip * remove indicator * add overloads * ci/cd * wip * cicd * wip * deprecation warnings * missing import

Marigold

It's hard to review such a massive PR, but it looks good after skimming through it. Don't forget to squash it.

* ✨ wizard: anomalies * wip * bump streamlit * wip * wip: chart * wip * todo * plot indicator * re-structure * wip: loading indicators * fix API grapher_chart * deprecate chart_html * chart_html -> grapher_chart * clean * ci/cd * wip * wip * changed module name * custom components module * add methods to get uris * new alias * get dataset uris * update import * update gpt pricing * update import * wip * provide entity-context for anomaly * wip: anomalist v2 * wip * wip * lock * ✨ anomalist: improve utils (#3385) * wip * db -> db_utils * io -> db * move things db_utils -> db * db -> grapher_io * db -> grapher_io, db_utils -> db * docstring * db_utils -> db * wip * remove indicator * add overloads * ci/cd * wip * cicd * wip * deprecation warnings * missing import * hide anomalist in wizard

✨ wizard: anomalies

5aa850d

github-actions bot assigned lucasrodes Oct 3, 2024

lucasrodes added 6 commits October 3, 2024 13:00

wip

5e61b90

bump streamlit

ccf180c

wip

54c3f37

wip: chart

3f45b64

wip

0fc1b09

todo

1555f21

lucasrodes added 12 commits October 4, 2024 11:46

Merge branch 'master' into wizard-anomalies

5b4509b

plot indicator

b4d417d

re-structure

63b751b

Merge branch 'master' into wizard-anomalies

dfff33b

wip: loading indicators

1062536

fix API grapher_chart

58c612e

deprecate chart_html

11a3c49

chart_html -> grapher_chart

5f73abd

clean

69bb65c

ci/cd

e9ed4b3

wip

00d3eb9

wip

42675ca

lucasrodes added 7 commits October 6, 2024 23:34

changed module name

2c86454

custom components module

718a2b7

add methods to get uris

c3fb56a

new alias

15b621f

get dataset uris

3ac30d6

update import

b6d039c

update gpt pricing

86a9659

lucasrodes added 6 commits October 6, 2024 23:35

update import

4910aae

wip

58128e6

provide entity-context for anomaly

2c55201

wip: anomalist v2

12ad899

wip

68f60fc

wip

baab9be

Marigold reviewed Oct 8, 2024

View reviewed changes

lucasrodes added 4 commits October 9, 2024 11:11

Merge branch 'master' into wizard-anomalies

7f27c5f

lock

723c3c4

hide anomalist in wizard

db169ce

lucasrodes changed the title ~~✨ wizard: anomalies~~ ✨ wizard: anomalist (first draft) Oct 9, 2024

Marigold approved these changes Oct 9, 2024

View reviewed changes

lucasrodes marked this pull request as ready for review October 9, 2024 11:05

lucasrodes merged commit d0afe39 into master Oct 9, 2024
9 of 10 checks passed

lucasrodes deleted the wizard-anomalies branch October 9, 2024 11:05

lucasrodes mentioned this pull request Oct 9, 2024

🎉 wizard: anomalist (v2) #3388

Merged

lucasrodes mentioned this pull request Oct 7, 2024

Create anomaly detection workflow #3340

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ wizard: anomalist (first draft) #3363

✨ wizard: anomalist (first draft) #3363

lucasrodes commented Oct 3, 2024 •

edited

Loading

owidbot commented Oct 3, 2024 •

edited

Loading

lucasrodes commented Oct 3, 2024 •

edited

Loading

lucasrodes commented Oct 4, 2024 •

edited

Loading

Marigold Oct 8, 2024

Marigold left a comment

✨ wizard: anomalist (first draft) #3363

✨ wizard: anomalist (first draft) #3363

Conversation

lucasrodes commented Oct 3, 2024 • edited Loading

owidbot commented Oct 3, 2024 • edited Loading

lucasrodes commented Oct 3, 2024 • edited Loading

lucasrodes commented Oct 4, 2024 • edited Loading

Marigold Oct 8, 2024

Choose a reason for hiding this comment

Marigold left a comment

Choose a reason for hiding this comment

lucasrodes commented Oct 3, 2024 •

edited

Loading

owidbot commented Oct 3, 2024 •

edited

Loading

lucasrodes commented Oct 3, 2024 •

edited

Loading

lucasrodes commented Oct 4, 2024 •

edited

Loading