Add kind that demonstrates how to modify the upstream graph in a transform #438
firefoxci-taskcluster / tests-taskgraph-diff
failed
Feb 12, 2024 in 2m 33s
FirefoxCI (pull_request)
Create diffs of taskgraphs vs. base revision.
Details
View task in Taskcluster
View logs in Taskcluster
[task 2024-02-12T20:00:06.660Z] 'use-opuscleaner': 'true',
[task 2024-02-12T20:00:06.660Z] 'vocab': 'NOT-YET-SUPPORTED'},
[task 2024-02-12T20:00:06.660Z] 'marian-args': {'decoding-backward': {'mini-batch-words': '2000'},
[task 2024-02-12T20:00:06.660Z] 'decoding-teacher': {'mini-batch-words': '1000',
[task 2024-02-12T20:00:06.660Z] 'precision': 'float16'},
[task 2024-02-12T20:00:06.660Z] 'training-backward': {'after': '50u',
[task 2024-02-12T20:00:06.660Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.660Z] '1000',
[task 2024-02-12T20:00:06.660Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.660Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.660Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.660Z] 'training-student': {'after': '50u',
[task 2024-02-12T20:00:06.660Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.660Z] '1000',
[task 2024-02-12T20:00:06.660Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.660Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.660Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.660Z] 'training-student-finetuned': {'after': '50u',
[task 2024-02-12T20:00:06.660Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.660Z] '1000',
[task 2024-02-12T20:00:06.660Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.660Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.660Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.660Z] 'training-teacher': {'after': '50u',
[task 2024-02-12T20:00:06.660Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.660Z] '1000',
[task 2024-02-12T20:00:06.660Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.660Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.660Z] 'task': 'transformer-base',
[task 2024-02-12T20:00:06.660Z] 'valid-freq': '50'}},
[task 2024-02-12T20:00:06.660Z] 'target-stage': 'all',
[task 2024-02-12T20:00:06.660Z] 'taskcluster': {'split-chunks': 2}},
[task 2024-02-12T20:00:06.660Z] 'version': None}
[task 2024-02-12T20:00:06.660Z]
[task 2024-02-12T20:00:06.660Z] During handling of the above exception, another exception occurred:
[task 2024-02-12T20:00:06.660Z]
[task 2024-02-12T20:00:06.660Z] Traceback (most recent call last):
[task 2024-02-12T20:00:06.660Z] File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
[task 2024-02-12T20:00:06.660Z] r = call_item.fn(*call_item.args, **call_item.kwargs)
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/main.py", line 148, in format_taskgraph
[task 2024-02-12T20:00:06.660Z] tg = getattr(tgg, options["graph_attr"])
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 170, in full_task_graph
[task 2024-02-12T20:00:06.660Z] return self._run_until("full_task_graph")
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 425, in _run_until
[task 2024-02-12T20:00:06.660Z] k, v = next(self._run)
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 264, in _run
[task 2024-02-12T20:00:06.660Z] parameters = self._parameters(graph_config)
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/parameters.py", line 373, in get_parameters
[task 2024-02-12T20:00:06.660Z] parameters.check()
[task 2024-02-12T20:00:06.660Z] File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/parameters.py", line 220, in check
[task 2024-02-12T20:00:06.660Z] raise ParameterMismatch(str(e))
[task 2024-02-12T20:00:06.660Z] taskgraph.parameters.ParameterMismatch: Invalid parameters:
[task 2024-02-12T20:00:06.660Z] required key not provided @ data['training_config']['experiment']['bicleaner']['disable']
[task 2024-02-12T20:00:06.660Z] {'base_ref': 'origin/main',
[task 2024-02-12T20:00:06.660Z] 'base_repository': 'https://github.com/mozilla/firefox-translations-training',
[task 2024-02-12T20:00:06.660Z] 'base_rev': 'fd3125862741e231e5b4a43e2bde715e1ee05482',
[task 2024-02-12T20:00:06.660Z] 'build_date': 1700603004,
[task 2024-02-12T20:00:06.660Z] 'build_number': 1,
[task 2024-02-12T20:00:06.660Z] 'do_not_optimize': [],
[task 2024-02-12T20:00:06.660Z] 'enable_always_target': True,
[task 2024-02-12T20:00:06.660Z] 'existing_tasks': {},
[task 2024-02-12T20:00:06.660Z] 'filters': ['target_tasks_method'],
[task 2024-02-12T20:00:06.660Z] 'head_ref': 'refs/heads/main',
[task 2024-02-12T20:00:06.660Z] 'head_repository': 'https://github.com/mozilla/firefox-translations-training',
[task 2024-02-12T20:00:06.660Z] 'head_rev': 'fd3125862741e231e5b4a43e2bde715e1ee05482',
[task 2024-02-12T20:00:06.660Z] 'head_tag': '',
[task 2024-02-12T20:00:06.660Z] 'level': '1',
[task 2024-02-12T20:00:06.660Z] 'moz_build_date': '20231121214324',
[task 2024-02-12T20:00:06.660Z] 'next_version': None,
[task 2024-02-12T20:00:06.660Z] 'optimize_strategies': None,
[task 2024-02-12T20:00:06.660Z] 'optimize_target_tasks': True,
[task 2024-02-12T20:00:06.660Z] 'owner': '[email protected]',
[task 2024-02-12T20:00:06.660Z] 'project': 'firefox-translations-training',
[task 2024-02-12T20:00:06.660Z] 'pushdate': 0,
[task 2024-02-12T20:00:06.660Z] 'pushlog_id': '0',
[task 2024-02-12T20:00:06.660Z] 'repository_type': 'git',
[task 2024-02-12T20:00:06.660Z] 'target-kinds': [],
[task 2024-02-12T20:00:06.660Z] 'target_tasks_method': 'default',
[task 2024-02-12T20:00:06.660Z] 'tasks_for': 'github-push',
[task 2024-02-12T20:00:06.660Z] 'training_config': {'datasets': {'devtest': ['flores_dev',
[task 2024-02-12T20:00:06.660Z] 'sacrebleu_aug-mix_wmt19'],
[task 2024-02-12T20:00:06.660Z] 'mono-src': ['news-crawl_news.2008'],
[task 2024-02-12T20:00:06.660Z] 'mono-trg': ['news-crawl_news.2007'],
[task 2024-02-12T20:00:06.660Z] 'test': ['flores_devtest'],
[task 2024-02-12T20:00:06.660Z] 'train': ['opus_ada83/v1',
[task 2024-02-12T20:00:06.660Z] 'opus_ELRC-3075-wikipedia_health/v1']},
[task 2024-02-12T20:00:06.660Z] 'experiment': {'backward-model': 'NOT-YET-SUPPORTED',
[task 2024-02-12T20:00:06.660Z] 'best-model': 'chrf',
[task 2024-02-12T20:00:06.660Z] 'bicleaner': {'dataset-thresholds': {'opus_ELRC-3075-wikipedia_health/v1': 0.6,
[task 2024-02-12T20:00:06.660Z] 'opus_ada83/v1': 0.0},
[task 2024-02-12T20:00:06.660Z] 'default-threshold': 0.5},
[task 2024-02-12T20:00:06.660Z] 'mono-max-sentences-src': 10000,
[task 2024-02-12T20:00:06.660Z] 'mono-max-sentences-trg': 10000,
[task 2024-02-12T20:00:06.660Z] 'name': 'ci',
[task 2024-02-12T20:00:06.660Z] 'split-length': 5000,
[task 2024-02-12T20:00:06.660Z] 'spm-sample-size': 10000,
[task 2024-02-12T20:00:06.660Z] 'spm-vocab-size': 1000,
[task 2024-02-12T20:00:06.660Z] 'src': 'ru',
[task 2024-02-12T20:00:06.660Z] 'teacher-ensemble': 1,
[task 2024-02-12T20:00:06.661Z] 'trg': 'en',
[task 2024-02-12T20:00:06.661Z] 'use-opuscleaner': 'true',
[task 2024-02-12T20:00:06.661Z] 'vocab': 'NOT-YET-SUPPORTED'},
[task 2024-02-12T20:00:06.661Z] 'marian-args': {'decoding-backward': {'mini-batch-words': '2000'},
[task 2024-02-12T20:00:06.661Z] 'decoding-teacher': {'mini-batch-words': '1000',
[task 2024-02-12T20:00:06.661Z] 'precision': 'float16'},
[task 2024-02-12T20:00:06.661Z] 'training-backward': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-student': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-student-finetuned': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-teacher': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'task': 'transformer-base',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'}},
[task 2024-02-12T20:00:06.661Z] 'target-stage': 'all',
[task 2024-02-12T20:00:06.661Z] 'taskcluster': {'split-chunks': 2}},
[task 2024-02-12T20:00:06.661Z] 'version': None}
[task 2024-02-12T20:00:06.661Z] """
[task 2024-02-12T20:00:06.661Z]
[task 2024-02-12T20:00:06.661Z] The above exception was the direct cause of the following exception:
[task 2024-02-12T20:00:06.661Z]
[task 2024-02-12T20:00:06.661Z] taskgraph.parameters.ParameterMismatch: Invalid parameters:
[task 2024-02-12T20:00:06.661Z] required key not provided @ data['training_config']['experiment']['bicleaner']['disable']
[task 2024-02-12T20:00:06.661Z] {'base_ref': 'origin/main',
[task 2024-02-12T20:00:06.661Z] 'base_repository': 'https://github.com/mozilla/firefox-translations-training',
[task 2024-02-12T20:00:06.661Z] 'base_rev': 'fd3125862741e231e5b4a43e2bde715e1ee05482',
[task 2024-02-12T20:00:06.661Z] 'build_date': 1700603004,
[task 2024-02-12T20:00:06.661Z] 'build_number': 1,
[task 2024-02-12T20:00:06.661Z] 'do_not_optimize': [],
[task 2024-02-12T20:00:06.661Z] 'enable_always_target': True,
[task 2024-02-12T20:00:06.661Z] 'existing_tasks': {},
[task 2024-02-12T20:00:06.661Z] 'filters': ['target_tasks_method'],
[task 2024-02-12T20:00:06.661Z] 'head_ref': 'refs/heads/main',
[task 2024-02-12T20:00:06.661Z] 'head_repository': 'https://github.com/mozilla/firefox-translations-training',
[task 2024-02-12T20:00:06.661Z] 'head_rev': 'fd3125862741e231e5b4a43e2bde715e1ee05482',
[task 2024-02-12T20:00:06.661Z] 'head_tag': '',
[task 2024-02-12T20:00:06.661Z] 'level': '1',
[task 2024-02-12T20:00:06.661Z] 'moz_build_date': '20231121214324',
[task 2024-02-12T20:00:06.661Z] 'next_version': None,
[task 2024-02-12T20:00:06.661Z] 'optimize_strategies': None,
[task 2024-02-12T20:00:06.661Z] 'optimize_target_tasks': True,
[task 2024-02-12T20:00:06.661Z] 'owner': '[email protected]',
[task 2024-02-12T20:00:06.661Z] 'project': 'firefox-translations-training',
[task 2024-02-12T20:00:06.661Z] 'pushdate': 0,
[task 2024-02-12T20:00:06.661Z] 'pushlog_id': '0',
[task 2024-02-12T20:00:06.661Z] 'repository_type': 'git',
[task 2024-02-12T20:00:06.661Z] 'target-kinds': [],
[task 2024-02-12T20:00:06.661Z] 'target_tasks_method': 'default',
[task 2024-02-12T20:00:06.661Z] 'tasks_for': 'github-push',
[task 2024-02-12T20:00:06.661Z] 'training_config': {'datasets': {'devtest': ['flores_dev',
[task 2024-02-12T20:00:06.661Z] 'sacrebleu_aug-mix_wmt19'],
[task 2024-02-12T20:00:06.661Z] 'mono-src': ['news-crawl_news.2008'],
[task 2024-02-12T20:00:06.661Z] 'mono-trg': ['news-crawl_news.2007'],
[task 2024-02-12T20:00:06.661Z] 'test': ['flores_devtest'],
[task 2024-02-12T20:00:06.661Z] 'train': ['opus_ada83/v1',
[task 2024-02-12T20:00:06.661Z] 'opus_ELRC-3075-wikipedia_health/v1']},
[task 2024-02-12T20:00:06.661Z] 'experiment': {'backward-model': 'NOT-YET-SUPPORTED',
[task 2024-02-12T20:00:06.661Z] 'best-model': 'chrf',
[task 2024-02-12T20:00:06.661Z] 'bicleaner': {'dataset-thresholds': {'opus_ELRC-3075-wikipedia_health/v1': 0.6,
[task 2024-02-12T20:00:06.661Z] 'opus_ada83/v1': 0.0},
[task 2024-02-12T20:00:06.661Z] 'default-threshold': 0.5},
[task 2024-02-12T20:00:06.661Z] 'mono-max-sentences-src': 10000,
[task 2024-02-12T20:00:06.661Z] 'mono-max-sentences-trg': 10000,
[task 2024-02-12T20:00:06.661Z] 'name': 'ci',
[task 2024-02-12T20:00:06.661Z] 'split-length': 5000,
[task 2024-02-12T20:00:06.661Z] 'spm-sample-size': 10000,
[task 2024-02-12T20:00:06.661Z] 'spm-vocab-size': 1000,
[task 2024-02-12T20:00:06.661Z] 'src': 'ru',
[task 2024-02-12T20:00:06.661Z] 'teacher-ensemble': 1,
[task 2024-02-12T20:00:06.661Z] 'trg': 'en',
[task 2024-02-12T20:00:06.661Z] 'use-opuscleaner': 'true',
[task 2024-02-12T20:00:06.661Z] 'vocab': 'NOT-YET-SUPPORTED'},
[task 2024-02-12T20:00:06.661Z] 'marian-args': {'decoding-backward': {'mini-batch-words': '2000'},
[task 2024-02-12T20:00:06.661Z] 'decoding-teacher': {'mini-batch-words': '1000',
[task 2024-02-12T20:00:06.661Z] 'precision': 'float16'},
[task 2024-02-12T20:00:06.661Z] 'training-backward': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-student': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-student-finetuned': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'},
[task 2024-02-12T20:00:06.661Z] 'training-teacher': {'after': '50u',
[task 2024-02-12T20:00:06.661Z] 'dim-vocabs': '1000 '
[task 2024-02-12T20:00:06.661Z] '1000',
[task 2024-02-12T20:00:06.661Z] 'disp-freq': '1',
[task 2024-02-12T20:00:06.661Z] 'save-freq': '25',
[task 2024-02-12T20:00:06.661Z] 'task': 'transformer-base',
[task 2024-02-12T20:00:06.661Z] 'valid-freq': '50'}},
[task 2024-02-12T20:00:06.661Z] 'target-stage': 'all',
[task 2024-02-12T20:00:06.661Z] 'taskcluster': {'split-chunks': 2}},
[task 2024-02-12T20:00:06.661Z] 'version': None}
[task 2024-02-12T20:00:06.661Z]
[task 2024-02-12T20:00:06.682Z] Note: switching to '9ebfd13903e28015d8008eb814ccc6ea7d7d735a'.
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] You are in 'detached HEAD' state. You can look around, make experimental
[task 2024-02-12T20:00:06.682Z] changes and commit them, and you can discard any commits you make in this
[task 2024-02-12T20:00:06.682Z] state without impacting any branches by switching back to a branch.
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] If you want to create a new branch to retain commits you create, you may
[task 2024-02-12T20:00:06.682Z] do so (now or later) by using -c with the switch command. Example:
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] git switch -c <new-branch-name>
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] Or undo this operation with:
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] git switch -
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] Turn off this advice by setting config variable advice.detachedHead to false
[task 2024-02-12T20:00:06.682Z]
[task 2024-02-12T20:00:06.682Z] HEAD is now at 9ebfd13 Publish YAML configuration to group_logs run (#386)
[task 2024-02-12T20:00:06.684Z] Generating full_task_graph @ 9ebfd13903e2
[task 2024-02-12T20:00:06.689Z]
[task 2024-02-12T20:00:07.902Z] Previous HEAD position was 9ebfd13 Publish YAML configuration to group_logs run (#386)
[task 2024-02-12T20:00:07.902Z] Switched to branch 'graph-mod'
[task 2024-02-12T20:00:07.903Z] WARNING: Diff skipped for the following generations due to failures:
[task 2024-02-12T20:00:07.903Z] full_task_graph_9ebfd13903e28015d8008eb814ccc6ea7d7d735a_large-lt-en
[task 2024-02-12T20:00:07.903Z] full_task_graph_9ebfd13903e28015d8008eb814ccc6ea7d7d735a_small-ru-en
[task 2024-02-12T20:00:07.903Z] See '/builds/worker/.cache/taskgraph/log/vcs' for logs
[task 2024-02-12T20:00:07.933Z] make: *** [Makefile:72: diff-taskgraph] Error 1
[taskcluster 2024-02-12 20:00:08.444Z] === Task Finished ===
[taskcluster 2024-02-12 20:00:08.444Z] Unsuccessful task run with exit code: 2 completed in 33.974 seconds
Loading