Skip to content

Latest commit

 

History

History
62 lines (40 loc) · 2.12 KB

README.md

File metadata and controls

62 lines (40 loc) · 2.12 KB
Raw

Dataset viewer maintenance job

Job to run maintenance actions on the dataset viewer.

Available actions:

  • backfill: backfill the cache (i.e. create jobs to add the missing entries or update the outdated entries)
  • backfill-retryable-errors: backfill the cache for retryable errors
  • collect-cache-metrics: compute and store the cache metrics
  • collect-queue-metrics: compute and store the queue metrics
  • post-messages: post messages in Hub discussions
  • skip: do nothing

Configuration

The script can be configured using environment variables. They are grouped by scope.

  • CACHE_MAINTENANCE_ACTION: the action to launch, among backfill, backfill-retryable-errors, collect-cache-metrics, collect-queue-metrics and post-messages. Defaults to skip.

Backfill job configurations

See ../../libs/libcommon/README.md for the following configurations:

  • Cached Assets
  • Assets
  • S3
  • Cache
  • Queue

Collect Cache job configurations

See ../../libs/libcommon/README.md for the following configurations:

  • Cache

Collect Queue job configurations

See ../../libs/libcommon/README.md for the following configurations:

  • Queue

Clean Directory job configurations

  • DIRECTORY_CLEANING_CACHE_DIRECTORY: directory location to clean up.
  • DIRECTORY_CLEANING_SUBFOLDER_PATTERN: sub folder pattern inside the cache directory.
  • DIRECTORY_CLEANING_EXPIRED_TIME_INTERVAL_SECONDS: time in seconds after a file is deleted since its last accessed time.

Post Messages job configurations

Set environment variables to configure the post-messages job:

  • DISCUSSIONS_BOT_ASSOCIATED_USER_NAME: name of the Hub user associated with the dataset viewer bot app.
  • DISCUSSIONS_BOT_TOKEN: token of the dataset viewer bot used to post messages in Hub discussions.
  • DISCUSSIONS_PARQUET_REVISION: revision (branch) where the converted Parquet files are stored.

Common

See ../../libs/libcommon/README.md for more information about the common configuration.

Launch

make run