feat(chart): auto deploy when secrets change #2911

rtrompier · 2024-06-14T08:16:54Z

Will automatically redeploy applications when secrets are changed in Infisical (max 1 min after the change).

github-actions · 2024-06-14T08:17:33Z

ArgoCD Diff for commit `5737fee`

Updated at 6/14/2024, 10:17:32 AM CEST

App: datasets-server-prod
YAML generation: Success 🟢
App sync status: Synced ✅

===== apps/Deployment datasets-server/prod-datasets-server-admin ======
--- /tmp/argocd-diff1179860489/prod-datasets-server-admin-live.yaml	2024-06-14 08:17:31.075136626 +0000
+++ /tmp/argocd-diff1179860489/prod-datasets-server-admin	2024-06-14 08:17:31.071136631 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "535"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-admin","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":2,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-admin","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"ADMIN_HF_ORGANIZATION","value":"huggingface"},{"name":"ADMIN_CACHE_REPORTS_NUM_RESULTS","value":"1000"},{"name":"ADMIN_CACHE_REPORTS_WITH_CONTENT_NUM_RESULTS","value":"100"},{"name":"ADMIN_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"ADMIN_HF_WHOAMI_PATH","value":"/api/whoami-v2"},{"name":"ADMIN_MAX_AGE","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"ADMIN_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"ADMIN_UVICORN_NUM_WORKERS","value":"9"},{"name":"ADMIN_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-admin:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/admin/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-admin","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/admin/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":4,"memory":"8Gi"},"requests":{"cpu":1,"memory":"8Gi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":true,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server","operator":"Exists"}],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 542
   labels:
     app.kubernetes.io/component: prod-datasets-server-admin

===== apps/Deployment datasets-server/prod-datasets-server-api ======
--- /tmp/argocd-diff2744611402/prod-datasets-server-api-live.yaml	2024-06-14 08:17:31.091136609 +0000
+++ /tmp/argocd-diff2744611402/prod-datasets-server-api	2024-06-14 08:17:31.091136609 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "540"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-api","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":20,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"API_HF_WEBHOOK_SECRET","valueFrom":{"secretKeyRef":{"key":"WEBHOOK_SECRET","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"9"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-api:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-api","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":4,"memory":"14Gi"},"requests":{"cpu":4,"memory":"14Gi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{"role-datasets-server-api":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-api","operator":"Exists"}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 548
   labels:
     app.kubernetes.io/component: prod-datasets-server-api

===== apps/Deployment datasets-server/prod-datasets-server-rows ======
--- /tmp/argocd-diff1264759971/prod-datasets-server-rows-live.yaml	2024-06-14 08:17:31.111136587 +0000
+++ /tmp/argocd-diff1264759971/prod-datasets-server-rows	2024-06-14 08:17:31.111136587 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "374"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-rows","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-rows","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":12,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-rows","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-rows","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"API_HF_WEBHOOK_SECRET","valueFrom":{"secretKeyRef":{"key":"WEBHOOK_SECRET","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"2"},{"name":"API_UVICORN_PORT","value":"8080"},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"}],"image":"huggingface/datasets-server-services-rows:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-rows","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":4,"memory":"6500Mi"},"requests":{"cpu":1,"memory":"6500Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":true,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server-rows":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-rows","operator":"Exists"}],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 383
   labels:
     app.kubernetes.io/component: prod-datasets-server-rows

===== apps/Deployment datasets-server/prod-datasets-server-search ======
--- /tmp/argocd-diff1766530365/prod-datasets-server-search-live.yaml	2024-06-14 08:17:31.127136570 +0000
+++ /tmp/argocd-diff1766530365/prod-datasets-server-search	2024-06-14 08:17:31.127136570 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "327"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-search","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-search","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":3,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-search","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-search","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"API_HF_WEBHOOK_SECRET","valueFrom":{"secretKeyRef":{"key":"WEBHOOK_SECRET","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"2"},{"name":"API_UVICORN_PORT","value":"8080"},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_CACHE_CLEAN_CACHE_PROBA","value":"0.05"},{"name":"DUCKDB_INDEX_CACHE_EXPIRED_TIME_INTERVAL_SECONDS","value":"43200"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"}],"image":"huggingface/datasets-server-services-search:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-search","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":64,"memory":"250Gi"},"requests":{"cpu":1,"memory":"250Gi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{"role-datasets-server-search":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-search","operator":"Exists"}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 333
   labels:
     app.kubernetes.io/component: prod-datasets-server-search

===== apps/Deployment datasets-server/prod-datasets-server-sse-api ======
--- /tmp/argocd-diff2649158477/prod-datasets-server-sse-api-live.yaml	2024-06-14 08:17:31.143136553 +0000
+++ /tmp/argocd-diff2649158477/prod-datasets-server-sse-api	2024-06-14 08:17:31.143136553 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "266"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-sse-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-sse-api","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":2,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-sse-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-sse-api","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"API_HF_WEBHOOK_SECRET","valueFrom":{"secretKeyRef":{"key":"WEBHOOK_SECRET","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-sse-api:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/sse/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-sse-api","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/sse/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":4,"memory":"4Gi"},"requests":{"cpu":1,"memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{"role-datasets-server":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server","operator":"Exists"}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 268
   labels:
     app.kubernetes.io/component: prod-datasets-server-sse-api

===== apps/Deployment datasets-server/prod-datasets-server-webhook ======
--- /tmp/argocd-diff4068251205/prod-datasets-server-webhook-live.yaml	2024-06-14 08:17:31.159136535 +0000
+++ /tmp/argocd-diff4068251205/prod-datasets-server-webhook	2024-06-14 08:17:31.155136539 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "17"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-webhook","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-webhook","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":4,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-webhook","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-webhook","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"prod-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"1.5"},{"name":"API_HF_WEBHOOK_SECRET","valueFrom":{"secretKeyRef":{"key":"WEBHOOK_SECRET","name":"prod-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"9"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-webhook:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-webhook","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":4,"memory":"4Gi"},"requests":{"cpu":4,"memory":"4Gi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{"role-datasets-server-webhook":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-webhook","operator":"Exists"}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 17
   labels:
     app.kubernetes.io/component: prod-datasets-server-webhook

===== apps/Deployment datasets-server/prod-datasets-server-worker-heavy ======
--- /tmp/argocd-diff2151338037/prod-datasets-server-worker-heavy-live.yaml	2024-06-14 08:17:31.187136505 +0000
+++ /tmp/argocd-diff2151338037/prod-datasets-server-worker-heavy	2024-06-14 08:17:31.183136509 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "109"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-heavy","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-worker-heavy","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-worker-heavy","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-heavy","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"HF_MODULES_CACHE","value":"/tmp/modules-cache"},{"name":"HF_DATASETS_TRUST_REMOTE_CODE","value":"0"},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"WORKER_CONTENT_MAX_BYTES","value":"10_000_000"},{"name":"WORKER_HEARTBEAT_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_KILL_ZOMBIES_INTERVAL_SECONDS","value":"600"},{"name":"WORKER_KILL_LONG_JOB_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_MAX_JOB_DURATION_SECONDS","value":"2400"},{"name":"WORKER_MAX_LOAD_PCT","value":"0"},{"name":"WORKER_MAX_MEMORY_PCT","value":"0"},{"name":"WORKER_MAX_MISSING_HEARTBEATS","value":"5"},{"name":"WORKER_SLEEP_SECONDS","value":"5"},{"name":"TMPDIR","value":"/tmp"},{"name":"FIRST_ROWS_MAX_BYTES","value":"200_000"},{"name":"FIRST_ROWS_MIN_CELL_BYTES","value":"100"},{"name":"FIRST_ROWS_MIN_NUMBER","value":"10"},{"name":"FIRST_ROWS_COLUMNS_MAX_NUMBER","value":"1000"},{"name":"PARQUET_AND_INFO_COMMIT_MESSAGE","value":"Update parquet files"},{"name":"PARQUET_AND_INFO_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"PARQUET_AND_INFO_MAX_DATASET_SIZE_BYTES","value":"5_000_000_000"},{"name":"PARQUET_AND_INFO_MAX_EXTERNAL_DATA_FILES","value":"10_000"},{"name":"PARQUET_AND_INFO_MAX_ROW_GROUP_BYTE_SIZE_FOR_COPY","value":"300_000_000"},{"name":"PARQUET_AND_INFO_SOURCE_REVISION","value":"main"},{"name":"PARQUET_AND_INFO_TARGET_REVISION","value":"refs/convert/parquet"},{"name":"PARQUET_AND_INFO_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"OPT_IN_OUT_URLS_SCAN_COLUMNS_MAX_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_CONCURRENT_REQUESTS_NUMBER","value":"100"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_REQUESTS_PER_SECOND","value":"50"},{"name":"OPT_IN_OUT_URLS_SCAN_ROWS_MAX_NUMBER","value":"100000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_TOKEN","valueFrom":{"secretKeyRef":{"key":"SPAWNING_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"OPT_IN_OUT_URLS_SCAN_URLS_NUMBER_PER_BATCH","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_URL","value":"https://opts-api.spawningaiapi.com/api/v2/query/urls"},{"name":"CONFIG_NAMES_MAX_NUMBER","value":"3000"},{"name":"DUCKDB_INDEX_COMMIT_MESSAGE","value":"Update duckdb index files"},{"name":"DUCKDB_INDEX_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"DUCKDB_INDEX_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"DESCRIPTIVE_STATISTICS_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY","value":"/tmp/stats-cache"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"},{"name":"DATASETS_BASED_HF_DATASETS_CACHE","value":"/tmp/hf-datasets-cache/heavy/datasets"},{"name":"WORKER_DIFFICULTY_MAX","value":"100"},{"name":"WORKER_DIFFICULTY_MIN","value":"70"},{"name":"WORKER_JOB_TYPES_BLOCKED","value":""},{"name":"WORKER_JOB_TYPES_ONLY","value":""},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"WORKER_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"WORKER_UVICORN_NUM_WORKERS","value":"1"},{"name":"WORKER_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-worker:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-worker","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":8,"memory":"44Gi"},"requests":{"cpu":8,"memory":"34Gi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server-worker":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-worker","operator":"Exists"}],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 31997
   labels:
     app.kubernetes.io/component: prod-datasets-server-worker-heavy

===== apps/Deployment datasets-server/prod-datasets-server-worker-light ======
--- /tmp/argocd-diff2389520848/prod-datasets-server-worker-light-live.yaml	2024-06-14 08:17:31.211136478 +0000
+++ /tmp/argocd-diff2389520848/prod-datasets-server-worker-light	2024-06-14 08:17:31.207136483 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "109"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-light","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-worker-light","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-worker-light","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-light","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"HF_MODULES_CACHE","value":"/tmp/modules-cache"},{"name":"HF_DATASETS_TRUST_REMOTE_CODE","value":"0"},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"WORKER_CONTENT_MAX_BYTES","value":"10_000_000"},{"name":"WORKER_HEARTBEAT_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_KILL_ZOMBIES_INTERVAL_SECONDS","value":"600"},{"name":"WORKER_KILL_LONG_JOB_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_MAX_JOB_DURATION_SECONDS","value":"2400"},{"name":"WORKER_MAX_LOAD_PCT","value":"0"},{"name":"WORKER_MAX_MEMORY_PCT","value":"0"},{"name":"WORKER_MAX_MISSING_HEARTBEATS","value":"5"},{"name":"WORKER_SLEEP_SECONDS","value":"5"},{"name":"TMPDIR","value":"/tmp"},{"name":"FIRST_ROWS_MAX_BYTES","value":"200_000"},{"name":"FIRST_ROWS_MIN_CELL_BYTES","value":"100"},{"name":"FIRST_ROWS_MIN_NUMBER","value":"10"},{"name":"FIRST_ROWS_COLUMNS_MAX_NUMBER","value":"1000"},{"name":"PARQUET_AND_INFO_COMMIT_MESSAGE","value":"Update parquet files"},{"name":"PARQUET_AND_INFO_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"PARQUET_AND_INFO_MAX_DATASET_SIZE_BYTES","value":"5_000_000_000"},{"name":"PARQUET_AND_INFO_MAX_EXTERNAL_DATA_FILES","value":"10_000"},{"name":"PARQUET_AND_INFO_MAX_ROW_GROUP_BYTE_SIZE_FOR_COPY","value":"300_000_000"},{"name":"PARQUET_AND_INFO_SOURCE_REVISION","value":"main"},{"name":"PARQUET_AND_INFO_TARGET_REVISION","value":"refs/convert/parquet"},{"name":"PARQUET_AND_INFO_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"OPT_IN_OUT_URLS_SCAN_COLUMNS_MAX_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_CONCURRENT_REQUESTS_NUMBER","value":"100"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_REQUESTS_PER_SECOND","value":"50"},{"name":"OPT_IN_OUT_URLS_SCAN_ROWS_MAX_NUMBER","value":"100000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_TOKEN","valueFrom":{"secretKeyRef":{"key":"SPAWNING_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"OPT_IN_OUT_URLS_SCAN_URLS_NUMBER_PER_BATCH","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_URL","value":"https://opts-api.spawningaiapi.com/api/v2/query/urls"},{"name":"CONFIG_NAMES_MAX_NUMBER","value":"3000"},{"name":"DUCKDB_INDEX_COMMIT_MESSAGE","value":"Update duckdb index files"},{"name":"DUCKDB_INDEX_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"DUCKDB_INDEX_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"DESCRIPTIVE_STATISTICS_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY","value":"/tmp/stats-cache"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"},{"name":"DATASETS_BASED_HF_DATASETS_CACHE","value":"/tmp/hf-datasets-cache/light/datasets"},{"name":"WORKER_DIFFICULTY_MAX","value":"40"},{"name":"WORKER_DIFFICULTY_MIN","value":"0"},{"name":"WORKER_JOB_TYPES_BLOCKED","value":""},{"name":"WORKER_JOB_TYPES_ONLY","value":""},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"WORKER_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"WORKER_UVICORN_NUM_WORKERS","value":"1"},{"name":"WORKER_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-worker:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-worker","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":2,"memory":"1Gi"},"requests":{"cpu":"200m","memory":"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server-worker-light":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-worker-light","operator":"Exists"}],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 45128
   labels:
     app.kubernetes.io/component: prod-datasets-server-worker-light

===== apps/Deployment datasets-server/prod-datasets-server-worker-medium ======
--- /tmp/argocd-diff1835806833/prod-datasets-server-worker-medium-live.yaml	2024-06-14 08:17:31.235136452 +0000
+++ /tmp/argocd-diff1835806833/prod-datasets-server-worker-medium	2024-06-14 08:17:31.231136456 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "109"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-medium","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","argocd.argoproj.io/instance":"datasets-server-prod","helm.sh/chart":"datasets-server"},"name":"prod-datasets-server-worker-medium","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"prod-datasets-server-worker-medium","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"prod-datasets-server-worker-medium","app.kubernetes.io/instance":"prod","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prod-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.huggingface.co/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"prod-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"prod-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"HF_MODULES_CACHE","value":"/tmp/modules-cache"},{"name":"HF_DATASETS_TRUST_REMOTE_CODE","value":"0"},{"name":"LOG_LEVEL","value":"INFO"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"prod-datasets-server-secs","optional":false}}},{"name":"WORKER_CONTENT_MAX_BYTES","value":"10_000_000"},{"name":"WORKER_HEARTBEAT_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_KILL_ZOMBIES_INTERVAL_SECONDS","value":"600"},{"name":"WORKER_KILL_LONG_JOB_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_MAX_JOB_DURATION_SECONDS","value":"2400"},{"name":"WORKER_MAX_LOAD_PCT","value":"0"},{"name":"WORKER_MAX_MEMORY_PCT","value":"0"},{"name":"WORKER_MAX_MISSING_HEARTBEATS","value":"5"},{"name":"WORKER_SLEEP_SECONDS","value":"5"},{"name":"TMPDIR","value":"/tmp"},{"name":"FIRST_ROWS_MAX_BYTES","value":"200_000"},{"name":"FIRST_ROWS_MIN_CELL_BYTES","value":"100"},{"name":"FIRST_ROWS_MIN_NUMBER","value":"10"},{"name":"FIRST_ROWS_COLUMNS_MAX_NUMBER","value":"1000"},{"name":"PARQUET_AND_INFO_COMMIT_MESSAGE","value":"Update parquet files"},{"name":"PARQUET_AND_INFO_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"PARQUET_AND_INFO_MAX_DATASET_SIZE_BYTES","value":"5_000_000_000"},{"name":"PARQUET_AND_INFO_MAX_EXTERNAL_DATA_FILES","value":"10_000"},{"name":"PARQUET_AND_INFO_MAX_ROW_GROUP_BYTE_SIZE_FOR_COPY","value":"300_000_000"},{"name":"PARQUET_AND_INFO_SOURCE_REVISION","value":"main"},{"name":"PARQUET_AND_INFO_TARGET_REVISION","value":"refs/convert/parquet"},{"name":"PARQUET_AND_INFO_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"OPT_IN_OUT_URLS_SCAN_COLUMNS_MAX_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_CONCURRENT_REQUESTS_NUMBER","value":"100"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_REQUESTS_PER_SECOND","value":"50"},{"name":"OPT_IN_OUT_URLS_SCAN_ROWS_MAX_NUMBER","value":"100000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_TOKEN","valueFrom":{"secretKeyRef":{"key":"SPAWNING_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"OPT_IN_OUT_URLS_SCAN_URLS_NUMBER_PER_BATCH","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_URL","value":"https://opts-api.spawningaiapi.com/api/v2/query/urls"},{"name":"CONFIG_NAMES_MAX_NUMBER","value":"3000"},{"name":"DUCKDB_INDEX_COMMIT_MESSAGE","value":"Update duckdb index files"},{"name":"DUCKDB_INDEX_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"prod-datasets-server-secs","optional":false}}},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"DUCKDB_INDEX_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"DESCRIPTIVE_STATISTICS_MAX_SPLIT_SIZE_BYTES","value":"5_000_000_000"},{"name":"DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY","value":"/tmp/stats-cache"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"},{"name":"DATASETS_BASED_HF_DATASETS_CACHE","value":"/tmp/hf-datasets-cache/medium/datasets"},{"name":"WORKER_DIFFICULTY_MAX","value":"70"},{"name":"WORKER_DIFFICULTY_MIN","value":"40"},{"name":"WORKER_JOB_TYPES_BLOCKED","value":""},{"name":"WORKER_JOB_TYPES_ONLY","value":""},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"WORKER_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"WORKER_UVICORN_NUM_WORKERS","value":"1"},{"name":"WORKER_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-worker:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"prod-datasets-server-worker","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":2,"memory":"14Gi"},"requests":{"cpu":2,"memory":"14Gi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/prod/parquet-metadata/"}]}],"nodeSelector":{"role-datasets-server-worker":"true"},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[{"effect":"NoSchedule","key":"huggingface.co/datasets-server-worker","operator":"Exists"}],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 50540
   labels:
     app.kubernetes.io/component: prod-datasets-server-worker-medium

App: datasets-server-staging
YAML generation: Success 🟢
App sync status: Synced ✅

===== apps/Deployment datasets-server/staging-datasets-server-admin ======
--- /tmp/argocd-diff3011583892/staging-datasets-server-admin-live.yaml	2024-06-14 08:17:32.235135364 +0000
+++ /tmp/argocd-diff3011583892/staging-datasets-server-admin	2024-06-14 08:17:32.227135373 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "254"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-admin","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-admin","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"ADMIN_HF_ORGANIZATION","value":"huggingface"},{"name":"ADMIN_CACHE_REPORTS_NUM_RESULTS","value":"100"},{"name":"ADMIN_CACHE_REPORTS_WITH_CONTENT_NUM_RESULTS","value":"100"},{"name":"ADMIN_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"ADMIN_HF_WHOAMI_PATH","value":"/api/whoami-v2"},{"name":"ADMIN_MAX_AGE","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"ADMIN_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"ADMIN_UVICORN_NUM_WORKERS","value":"1"},{"name":"ADMIN_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-admin:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/admin/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-admin","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/admin/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":true,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 260
   labels:
     app.kubernetes.io/component: staging-datasets-server-admin

===== apps/Deployment datasets-server/staging-datasets-server-api ======
--- /tmp/argocd-diff3312995887/staging-datasets-server-api-live.yaml	2024-06-14 08:17:32.251135347 +0000
+++ /tmp/argocd-diff3312995887/staging-datasets-server-api	2024-06-14 08:17:32.247135351 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "453"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-api","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"API_HF_WEBHOOK_SECRET","value":""},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-api:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-api","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 453
   labels:
     app.kubernetes.io/component: staging-datasets-server-api

===== apps/Deployment datasets-server/staging-datasets-server-rows ======
--- /tmp/argocd-diff1317962782/staging-datasets-server-rows-live.yaml	2024-06-14 08:17:32.271135325 +0000
+++ /tmp/argocd-diff1317962782/staging-datasets-server-rows	2024-06-14 08:17:32.267135329 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "419"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-rows","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-rows","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-rows","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-rows","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"API_HF_WEBHOOK_SECRET","value":""},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"}],"image":"huggingface/datasets-server-services-rows:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-rows","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":true,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[],"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 419
   labels:
     app.kubernetes.io/component: staging-datasets-server-rows

===== apps/Deployment datasets-server/staging-datasets-server-search ======
--- /tmp/argocd-diff2291286691/staging-datasets-server-search-live.yaml	2024-06-14 08:17:32.287135308 +0000
+++ /tmp/argocd-diff2291286691/staging-datasets-server-search	2024-06-14 08:17:32.283135312 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "367"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-search","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-search","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-search","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-search","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"API_HF_WEBHOOK_SECRET","value":""},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_CACHE_CLEAN_CACHE_PROBA","value":"0.05"},{"name":"DUCKDB_INDEX_CACHE_EXPIRED_TIME_INTERVAL_SECONDS","value":"43200"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"}],"image":"huggingface/datasets-server-services-search:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-search","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 369
   labels:
     app.kubernetes.io/component: staging-datasets-server-search

===== apps/Deployment datasets-server/staging-datasets-server-sse-api ======
--- /tmp/argocd-diff628473750/staging-datasets-server-sse-api-live.yaml	2024-06-14 08:17:32.303135290 +0000
+++ /tmp/argocd-diff628473750/staging-datasets-server-sse-api	2024-06-14 08:17:32.299135294 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "295"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-sse-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-sse-api","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-sse-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-sse-api","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"API_HF_WEBHOOK_SECRET","value":""},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-sse-api:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/sse/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-sse-api","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/sse/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":0},"requests":{"cpu":0}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 295
   labels:
     app.kubernetes.io/component: staging-datasets-server-sse-api

===== apps/Deployment datasets-server/staging-datasets-server-webhook ======
--- /tmp/argocd-diff277795173/staging-datasets-server-webhook-live.yaml	2024-06-14 08:17:32.315135277 +0000
+++ /tmp/argocd-diff277795173/staging-datasets-server-webhook	2024-06-14 08:17:32.315135277 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "17"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-webhook","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-webhook","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-webhook","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-webhook","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_EXPIRATION_SECONDS","value":"3600"},{"name":"CLOUDFRONT_KEY_PAIR_ID","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_KEY_PAIR_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"CLOUDFRONT_PRIVATE_KEY","valueFrom":{"secretKeyRef":{"key":"CLOUDFRONT_PRIVATE_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_AUTH_PATH","value":"/api/datasets/%s/auth-check"},{"name":"API_HF_JWT_PUBLIC_KEY_URL","value":"https://huggingface.co/api/keys/jwt"},{"name":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","valueFrom":{"secretKeyRef":{"key":"API_HF_JWT_ADDITIONAL_PUBLIC_KEYS","name":"staging-datasets-server-secs","optional":false}}},{"name":"API_HF_JWT_ALGORITHM","value":"EdDSA"},{"name":"API_HF_TIMEOUT_SECONDS","value":"0.2"},{"name":"API_HF_WEBHOOK_SECRET","value":""},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"CACHED_ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/cached-assets"},{"name":"CACHED_ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"API_MAX_AGE_LONG","value":"120"},{"name":"API_MAX_AGE_SHORT","value":"10"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"API_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"API_UVICORN_NUM_WORKERS","value":"1"},{"name":"API_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-webhook:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-webhook","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"nodeSelector":{},"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":[]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 17
   labels:
     app.kubernetes.io/component: staging-datasets-server-webhook

===== apps/Deployment datasets-server/staging-datasets-server-worker-all ======
--- /tmp/argocd-diff4071582071/staging-datasets-server-worker-all-live.yaml	2024-06-14 08:17:32.343135247 +0000
+++ /tmp/argocd-diff4071582071/staging-datasets-server-worker-all	2024-06-14 08:17:32.339135251 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "103"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-worker-all","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-worker-all","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-worker-all","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-worker-all","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"HF_MODULES_CACHE","value":"/tmp/modules-cache"},{"name":"HF_DATASETS_TRUST_REMOTE_CODE","value":"0"},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"WORKER_CONTENT_MAX_BYTES","value":"10_000_000"},{"name":"WORKER_HEARTBEAT_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_KILL_ZOMBIES_INTERVAL_SECONDS","value":"600"},{"name":"WORKER_KILL_LONG_JOB_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_MAX_JOB_DURATION_SECONDS","value":"2400"},{"name":"WORKER_MAX_LOAD_PCT","value":"0"},{"name":"WORKER_MAX_MEMORY_PCT","value":"0"},{"name":"WORKER_MAX_MISSING_HEARTBEATS","value":"5"},{"name":"WORKER_SLEEP_SECONDS","value":"5"},{"name":"TMPDIR","value":"/tmp"},{"name":"FIRST_ROWS_MAX_BYTES","value":"200_000"},{"name":"FIRST_ROWS_MIN_CELL_BYTES","value":"100"},{"name":"FIRST_ROWS_MIN_NUMBER","value":"10"},{"name":"FIRST_ROWS_COLUMNS_MAX_NUMBER","value":"1000"},{"name":"PARQUET_AND_INFO_COMMIT_MESSAGE","value":"Update parquet files"},{"name":"PARQUET_AND_INFO_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"PARQUET_AND_INFO_MAX_DATASET_SIZE_BYTES","value":"500_000_000"},{"name":"PARQUET_AND_INFO_MAX_EXTERNAL_DATA_FILES","value":"10_000"},{"name":"PARQUET_AND_INFO_MAX_ROW_GROUP_BYTE_SIZE_FOR_COPY","value":"300_000_000"},{"name":"PARQUET_AND_INFO_SOURCE_REVISION","value":"main"},{"name":"PARQUET_AND_INFO_TARGET_REVISION","value":"refs/convert/parquet"},{"name":"PARQUET_AND_INFO_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"OPT_IN_OUT_URLS_SCAN_COLUMNS_MAX_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_CONCURRENT_REQUESTS_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_REQUESTS_PER_SECOND","value":"20"},{"name":"OPT_IN_OUT_URLS_SCAN_ROWS_MAX_NUMBER","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_TOKEN","valueFrom":{"secretKeyRef":{"key":"SPAWNING_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"OPT_IN_OUT_URLS_SCAN_URLS_NUMBER_PER_BATCH","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_URL","value":"https://opts-api.spawningaiapi.com/api/v2/query/urls"},{"name":"CONFIG_NAMES_MAX_NUMBER","value":"3000"},{"name":"DUCKDB_INDEX_COMMIT_MESSAGE","value":"Update duckdb index files"},{"name":"DUCKDB_INDEX_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"DUCKDB_INDEX_MAX_SPLIT_SIZE_BYTES","value":"100_000_000"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"DESCRIPTIVE_STATISTICS_MAX_SPLIT_SIZE_BYTES","value":"100_000_000"},{"name":"DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY","value":"/tmp/stats-cache"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"},{"name":"DATASETS_BASED_HF_DATASETS_CACHE","value":"/tmp/hf-datasets-cache/all/datasets"},{"name":"WORKER_DIFFICULTY_MAX","value":"100"},{"name":"WORKER_DIFFICULTY_MIN","value":"0"},{"name":"WORKER_JOB_TYPES_BLOCKED","value":""},{"name":"WORKER_JOB_TYPES_ONLY","value":""},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"WORKER_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"WORKER_UVICORN_NUM_WORKERS","value":"1"},{"name":"WORKER_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-worker:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-worker","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"nodeSelector":null,"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":null,"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 132
   labels:
     app.kubernetes.io/component: staging-datasets-server-worker-all

===== apps/Deployment datasets-server/staging-datasets-server-worker-light ======
--- /tmp/argocd-diff3447319417/staging-datasets-server-worker-light-live.yaml	2024-06-14 08:17:32.371135217 +0000
+++ /tmp/argocd-diff3447319417/staging-datasets-server-worker-light	2024-06-14 08:17:32.367135221 +0000
@@ -5,6 +5,7 @@
     deployment.kubernetes.io/revision: "103"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"staging-datasets-server-worker-light","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","argocd.argoproj.io/instance":"datasets-server-staging","helm.sh/chart":"datasets-server"},"name":"staging-datasets-server-worker-light","namespace":"datasets-server"},"spec":{"progressDeadlineSeconds":600,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"staging-datasets-server-worker-light","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"staging-datasets-server-worker-light","app.kubernetes.io/instance":"staging","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"staging-datasets-server","helm.sh/chart":"datasets-server"}},"spec":{"containers":[{"env":[{"name":"ASSETS_BASE_URL","value":"https://datasets-server.us.dev.moon.huggingface.tech/assets"},{"name":"ASSETS_STORAGE_ROOT","value":"hf-datasets-server-statics-staging/assets"},{"name":"ASSETS_STORAGE_PROTOCOL","value":"s3"},{"name":"S3_REGION_NAME","value":"us-east-1"},{"name":"S3_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"staging-datasets-server-secs","optional":false}}},{"name":"S3_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"staging-datasets-server-secs","optional":false}}},{"name":"CACHE_MONGO_DATABASE","value":"datasets_server_cache"},{"name":"CACHE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"COMMON_BLOCKED_DATASETS","value":"open-llm-leaderboard/details_*,lunaluan/*,atom-in-the-universe/*,cot-leaderboard/cot-eval-traces,mitermix/yt-links,mcding-org/*"},{"name":"COMMON_DATASET_SCRIPTS_ALLOW_LIST","value":"hf-internal-testing/dataset_with_script,togethercomputer/RedPajama-Data-1T,togethercomputer/RedPajama-Data-V2,gaia-benchmark/GAIA,poloclub/diffusiondb,mozilla-foundation/common_voice_*,google/fleurs,speechcolab/gigaspeech,espnet/yodas,ylecun/mnist,zalando-datasets/fashion_mnist"},{"name":"COMMON_HF_ENDPOINT","value":"https://huggingface.co"},{"name":"HF_ENDPOINT","value":"https://huggingface.co"},{"name":"COMMON_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"HF_MODULES_CACHE","value":"/tmp/modules-cache"},{"name":"HF_DATASETS_TRUST_REMOTE_CODE","value":"0"},{"name":"LOG_LEVEL","value":"DEBUG"},{"name":"NUMBA_CACHE_DIR","value":"/tmp/numba-cache"},{"name":"PARQUET_METADATA_STORAGE_DIRECTORY","value":"/storage/parquet-metadata"},{"name":"QUEUE_MONGO_DATABASE","value":"datasets_server_queue"},{"name":"QUEUE_MONGO_URL","valueFrom":{"secretKeyRef":{"key":"MONGO_URL","name":"staging-datasets-server-secs","optional":false}}},{"name":"WORKER_CONTENT_MAX_BYTES","value":"10_000_000"},{"name":"WORKER_HEARTBEAT_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_KILL_ZOMBIES_INTERVAL_SECONDS","value":"600"},{"name":"WORKER_KILL_LONG_JOB_INTERVAL_SECONDS","value":"60"},{"name":"WORKER_MAX_JOB_DURATION_SECONDS","value":"2400"},{"name":"WORKER_MAX_LOAD_PCT","value":"0"},{"name":"WORKER_MAX_MEMORY_PCT","value":"0"},{"name":"WORKER_MAX_MISSING_HEARTBEATS","value":"5"},{"name":"WORKER_SLEEP_SECONDS","value":"5"},{"name":"TMPDIR","value":"/tmp"},{"name":"FIRST_ROWS_MAX_BYTES","value":"200_000"},{"name":"FIRST_ROWS_MIN_CELL_BYTES","value":"100"},{"name":"FIRST_ROWS_MIN_NUMBER","value":"10"},{"name":"FIRST_ROWS_COLUMNS_MAX_NUMBER","value":"1000"},{"name":"PARQUET_AND_INFO_COMMIT_MESSAGE","value":"Update parquet files"},{"name":"PARQUET_AND_INFO_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"PARQUET_AND_INFO_MAX_DATASET_SIZE_BYTES","value":"500_000_000"},{"name":"PARQUET_AND_INFO_MAX_EXTERNAL_DATA_FILES","value":"10_000"},{"name":"PARQUET_AND_INFO_MAX_ROW_GROUP_BYTE_SIZE_FOR_COPY","value":"300_000_000"},{"name":"PARQUET_AND_INFO_SOURCE_REVISION","value":"main"},{"name":"PARQUET_AND_INFO_TARGET_REVISION","value":"refs/convert/parquet"},{"name":"PARQUET_AND_INFO_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"OPT_IN_OUT_URLS_SCAN_COLUMNS_MAX_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_CONCURRENT_REQUESTS_NUMBER","value":"10"},{"name":"OPT_IN_OUT_URLS_SCAN_MAX_REQUESTS_PER_SECOND","value":"20"},{"name":"OPT_IN_OUT_URLS_SCAN_ROWS_MAX_NUMBER","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_TOKEN","valueFrom":{"secretKeyRef":{"key":"SPAWNING_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"OPT_IN_OUT_URLS_SCAN_URLS_NUMBER_PER_BATCH","value":"1000"},{"name":"OPT_IN_OUT_URLS_SCAN_SPAWNING_URL","value":"https://opts-api.spawningaiapi.com/api/v2/query/urls"},{"name":"CONFIG_NAMES_MAX_NUMBER","value":"3000"},{"name":"DUCKDB_INDEX_COMMIT_MESSAGE","value":"Update duckdb index files"},{"name":"DUCKDB_INDEX_COMMITTER_HF_TOKEN","valueFrom":{"secretKeyRef":{"key":"PARQUET_CONVERTER_HF_TOKEN","name":"staging-datasets-server-secs","optional":false}}},{"name":"DUCKDB_INDEX_TARGET_REVISION","value":"refs/convert/duckdb"},{"name":"DUCKDB_INDEX_URL_TEMPLATE","value":"/datasets/%s/resolve/%s/%s"},{"name":"DUCKDB_INDEX_MAX_SPLIT_SIZE_BYTES","value":"100_000_000"},{"name":"DUCKDB_INDEX_CACHE_DIRECTORY","value":"/tmp/duckdb-index"},{"name":"DUCKDB_INDEX_EXTENSIONS_DIRECTORY","value":"/tmp/duckdb-extensions"},{"name":"DESCRIPTIVE_STATISTICS_MAX_SPLIT_SIZE_BYTES","value":"100_000_000"},{"name":"DESCRIPTIVE_STATISTICS_CACHE_DIRECTORY","value":"/tmp/stats-cache"},{"name":"HF_HUB_ENABLE_HF_TRANSFER","value":"1"},{"name":"DATASETS_BASED_HF_DATASETS_CACHE","value":"/tmp/hf-datasets-cache/light/datasets"},{"name":"WORKER_DIFFICULTY_MAX","value":"40"},{"name":"WORKER_DIFFICULTY_MIN","value":"0"},{"name":"WORKER_JOB_TYPES_BLOCKED","value":""},{"name":"WORKER_JOB_TYPES_ONLY","value":""},{"name":"ROWS_INDEX_MAX_ARROW_DATA_IN_MEMORY","value":"300_000_000"},{"name":"PROMETHEUS_MULTIPROC_DIR","value":"/tmp"},{"name":"WORKER_UVICORN_HOSTNAME","value":"0.0.0.0"},{"name":"WORKER_UVICORN_NUM_WORKERS","value":"1"},{"name":"WORKER_UVICORN_PORT","value":"8080"}],"image":"huggingface/datasets-server-services-worker:sha-1f5c680","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"name":"staging-datasets-server-worker","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"failureThreshold":30,"httpGet":{"path":"/healthcheck","port":8080},"periodSeconds":5},"resources":{"limits":{"cpu":1,"memory":"4Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/storage/parquet-metadata","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"dnsConfig":{"options":[{"name":"ndots","value":"1"}]},"initContainers":[{"args":["chown 1000:3000 /mounted-path;"],"command":["/bin/sh","-c"],"image":"ubuntu:focal","imagePullPolicy":"IfNotPresent","name":"prepare-parquet-metadata","securityContext":{"runAsGroup":0,"runAsNonRoot":false,"runAsUser":0},"volumeMounts":[{"mountPath":"/mounted-path","mountPropagation":"None","name":"volume-parquet-metadata","readOnly":false,"subPath":"datasets-server/staging/parquet-metadata/"}]}],"nodeSelector":null,"securityContext":{"runAsGroup":3000,"runAsNonRoot":true,"runAsUser":1000},"tolerations":null,"volumes":[{"name":"volume-parquet-metadata","persistentVolumeClaim":{"claimName":"datasets-server-parquet-pvc"}}]}}}}
+    secrets.infisical.com/auto-reload: "true"
   generation: 127
   labels:
     app.kubernetes.io/component: staging-datasets-server-worker-light

Legend	Status
✅	The app is synced in ArgoCD, and diffs you see are solely from this PR.
⚠️	The app is out-of-sync in ArgoCD, and the diffs you see include those changes plus any from this PR.
🛑	There was an error generating the ArgoCD diffs due to changes in this PR.

feat(chart): auto deploy when secrets change

5737fee

rtrompier requested review from severo and AndreaFrancis June 14, 2024 08:16

severo approved these changes Jun 14, 2024

View reviewed changes

AndreaFrancis approved these changes Jun 14, 2024

View reviewed changes

rtrompier merged commit a525a87 into main Jun 26, 2024
3 checks passed

rtrompier deleted the feat/auto-deploy-on-secrets-change branch June 26, 2024 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(chart): auto deploy when secrets change #2911

feat(chart): auto deploy when secrets change #2911

rtrompier commented Jun 14, 2024 •

edited

Loading

github-actions bot commented Jun 14, 2024

feat(chart): auto deploy when secrets change #2911

feat(chart): auto deploy when secrets change #2911

Conversation

rtrompier commented Jun 14, 2024 • edited Loading

github-actions bot commented Jun 14, 2024

ArgoCD Diff for commit 5737fee

rtrompier commented Jun 14, 2024 •

edited

Loading

ArgoCD Diff for commit `5737fee`