Release 2.1.0 - Inpainting Challenges

Validator Update Steps

• Validators on autoupdate: No action required
• Validators not on autoupdate: Please restart your validator with the --clear-cache flag as shown below:

./start_validator --clear-cache

or, to restart with autoupdate:

pm2 start run_neuron.py -- --validator --clear-cache

TLDR

• First image-to-image (i2i) challenges starting with inpainting, along with cache and data generation refactors
• Expected challenge difficulty increase, as the inpaintings are the first semi-synthetics introduced to the subnet
• Inpainting challenges roll out with 20% representation of all image challenges
• Video rewards doubled from 10% to 20% weight

InPainting Challenges

2.1.0 introduces image-to-image (i2i) generation capabilities to SyntheticDataGenerator, starting with inpainting. Our first inpainting model is a Stable Diffusion XL pipeline, which allows us to take a real image input and generate high-quality image transformations using text annotations from prompt_generator.py, given a masked region of the original picture.

I2I_MODELS = {
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1": {
        "pipeline_cls": AutoPipelineForInpainting,
        "from_pretrained_args": {
            "use_safetensors": True,
            "torch_dtype": torch.float16,
            "variant": "fp16"
        },
        "generate_args": {
            "guidance_scale": 7.5,
            "num_inference_steps": 50,
            "strength": 0.99
        }
    }
}

Mask Generation

• Added random mask generation utility that creates either:
  • Rectangular masks with rounded corners
  • Circular masks with feathered edges
• Masks are sized proportionally to input image dimensions (25-50% of image dimensions)
• Centered placement with random offset
• Images are resized to 1024x1024 for optimal inpainting quality

Left to right: 1. Original, 2. Random mask (can be rectangular or circle mask), 3. In painting only 4. Final image
Prompt: "A cat with green eyes is photographed in a portrait style, standing outdoors with a green grass background."

Validator Challenge Flow

• 20% chance of selecting i2i (inpainting) when generating synthetic image challenges

1. Sample real image from cache
2. Generate dynamic caption using BLIP2
3. Create random inpainting mask
4. Run inpainting pipeline with original image, generated mask, and enhanced caption prompt
5. Cache generated image for future challenges

Prompt Generation Pipeline Updates

• New prompt enhancement step to improve prompt quality/descriptiveness, as well as to convey motion for more dynamic synthetic video results
• Renamed image_annotation_generator.py to prompt_generator.py

Video Rewards

• Video rewards bumped from 10% to 20%

QOL Changes

• Refactors for readability, including consolidating model constants and simplified cache output directories for t2v, t2i, and i2i in in the validator config.py
• --clear-cache option added to validator startup scripts

Release 2.0.3 - Bittensor 8.5.1

Upgrade Bittensor version to 8.5.1 for CR3

Release 2.0.2 - Validator Logging and VRAM Consumption

• Fix for info logs to show by default for validator process
• Ensure that processes that do not require GPU (cache updater, validator) do not allocate any VRAM for cuda/torch configurations.
  • This reduces overall VRAM usage by ~4-5GB
• try/except for wandb challenge data population to avoid errors when corrupt images are sampled (due to a failed extraction from parquet)

Release 2.0.1 - Bittensor 8.4.1

• Upgrading Bittensor to version 8.4.1
• Updating W&B logging to separate video from image performance metrics

Additional Information

Fix for libcudnn.so.9: cannot open shared object file: No such file or directory (courtesy of @Swamination, thank you!)

Checked to see if the LD_LIBRARY_PATH had been added (it was not present):

echo $LD_LIBRARY_PATH

Searched for the path to whatever version is installed:

find /home/ubuntu -type f -name libcudnn.so*

The v9 file was present, so I added the path to the lib's location as the library path (conda activate bitmind):

conda env config vars set LD_LIBRARY_PATH=/home/ubuntu/.local/lib/python3.10/site-packages/nvidia/cudnn/lib/

Release 2.0.0 - Video Challenges V1

Update Steps

Validator Update Steps

• Ensure you are running on a machine that meets the requirements specified in min_compute.yaml
  • Note the 80 GB VRAM GPU and recommended 600 GB storage
• If on autoupdate, no other action needed
• If not on autoupdate, run:

git pull
./setup_env.sh

pm2 delete bitmind_validator
pm2 delete run_neuron  # if using our run script

pm2 start run_neuron.py -- --validator

If you wish to turn off autoupdate or self healing restarts, you can instead start your validator with either

pm2 start run_neuron.py -- --validator --no-auto-update --no-self-heal

or

./start_validator.sh

NOTE Our startup script run_neuron.py (which calls start_validator.py, which is responsible for spawning the pm2 processes) now starts the processes shown below.
Do not manually run neruons/validator.py

┌────┬───────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name                      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼───────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 2  │ bitmind_cache_updater     │ default     │ N/A     │ fork    │ 3354129  │ 14s    │ 0    │ online    │ 0%       │ 1.7gb    │ user     │ disabled │
│ 3  │ bitmind_data_generator    │ default     │ N/A     │ fork    │ 3354163  │ 13s    │ 0    │ online    │ 0%       │ 1.8gb    │ user     │ disabled │
│ 1  │ bitmind_validator         │ default     │ N/A     │ fork    │ 3354098  │ 15s    │ 0    │ online    │ 0%       │ 1.8gb    │ user     │ disabled │
│ 0  │ run_neuron                │ default     │ N/A     │ fork    │ 3353994  │ 52s    │ 0    │ online    │ 0%       │ 10.5mb   │ user     │ disabled │
└────┴───────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

• run_neuron manages self-heal restarts and auto-update
• bitmind_validator is your validator process
• bitmind_data_generator is populating ~/.cache/sn34/synthetic with outputs from text-to-video and text-to-image models. This is the only sn34 validator process that uses GPU
• bitmind_cache_updater manages the cache of real images and videos at ~/.cache/sn34/real

Miner Update Steps

• For deployed miner instances, no immediate action is required
• For orientation around video detection model training and deployment, please see the Miner Updates section below
• Note that reward distribution will initially be split 90% to image challenges, 10% to video challenges to allow miners the opportunity to iterate without being significantly impacted.

Overview

v2.0.0 features our initial version of video challenges with three text-to-video models, and two large real video datasets amounting to nearly 10TB of video data. It also contains significant refactors of core subnet components, with an emphasis on our approach to sampling and generating data for validator challenges. More on this below.

It also includes code to train and deploy a deepfake video detection model called TALL (whitepaper) as a SN34 miner.

Reward distribution will initially be split 90% to image challenges, 10% to video challenges. This is meant to allow miners the freedom to explore and experiment with different video detection models without a significant on incentive distribution, while still allowing high video performance to provide upward mobility.

To test miner generalizability, video challenges include (chronologically ordered) random set of video frames sampled at a variable frame rate. Frames are extracted with a png codec, with randomly applied jpeg compression as part of our pre-existing augmentation pipeline

This release corresponds to our initial version of video challenges. Upcoming future releases will include:

• Improved output quality from existing text-to-video models
• Additional SoTA text-to-video models, like open-sora
• Additional real image dataset sources
• Improved VideoSynapse efficiency

Models

https://huggingface.co/genmo/mochi-1-preview
https://huggingface.co/THUDM/CogVideoX-5b
https://huggingface.co/ByteDance/AnimateDiff-Lightning

Datasets

Initial selection for providing real videos. More to come, these are subject to change.
https://huggingface.co/datasets/nkp37/OpenVid-1M
https://huggingface.co/datasets/shangxd/imagenet-vidvrd

Replacement of open-image-v7 URL dataset with 256x256 JPEG subset (1.9M images).
https://huggingface.co/datasets/bitmind/open-image-v7-256

Miner Updates

TALLDetector + VideoDataset

Model Training

• Miners can use base_miner/datasets/create_video_dataset.py (example usage in create_videos_dataset_example.sh) to transform a directory of mp4 files into a train-ready video frames dataset. This involves extracting individual frames from the mp4s and creating a local Huggingface dataset to reference the extracted frames during training.

Miner Deployment

• miner.env now has separate env vars for configuring both an image detection model and a video detection model.
• These models reside within the same miner process listening on the single miner axon port, and respectively respond to requests of type ImageSynapse and VideoSynapse.

Validator Optimizations and Refactors

SyntheticDataGenerator

• Formerly known as SyntheticImageGenerator, this class handles generation of videos in addition to its original functionality of prompt and image generation
• New functionality to run independently of the validator, continually generating and caching new synthetic video and images in ~/.cache/sn34/synthetic.
  • start_validator.sh now starts an additional pm2 process called bitmind_data_generator
  • This is an improvement over the previous "as-needed" approach to synthetic data generation, providing greater flexibility and higher generation throughput

ImageAnnotationGenerator

• Simplified, stripped of utility code used internally by the BitMind team for full synthetic dataset generation. The utility code will be updated and moved to our bitmind-utils repository

ImageCache and VideoCache

• Two new classes are introduced to reduce storage requirements for validators (given that the video files from OpenVid-1m alone takes up several TB)
• These classes are used to keep a fresh cache of compressed data sources (parquet for images, zips for videos), and a corresponding cache of extracted images and videos.
  • start_validator.sh now starts an additional pm2 process called bitmind_cache_updater that manages a real video cache and a real image cache. Each has its own asynchronous tasks to download new zip/parquet files on a regular interval, and extract random images and videos on a shorter interval.

Validator.forward

• Challenge generation has been updated according to the asynchronous nature of data sampling/generation described in the previous two sections.
• Rather than generating synthetic media on an as-needed basis and downloading entire image datasets, the BitMind Validator's forward function now samples random data from a local cache of real and synthetic images and videos. For videos, a random number of frames are sampled from a random mp4 file. This logic is handled by the VideoCache class

Additional Changes

• bitmind.constants.py had become quite a monolith, with the majority of its contents pertaining to validator operations. These variables have been moved to a more aptly named bitmind/validator/config.py, and model metadata dictionaries have been given a new structure that is more amenable to the nature of how the codebase interacts with them.
• Upgraded library versions, accommodating changes in behavior of huggingface file downloads.
• Added functionality to data augmentation pipeline to apply image transforms uniformly across all frames of a video
• Consolidated requirements.txt and setup_env.sh to avoid out of sync dependency versions
• For image challenges, we swapped out the open-images-v7 URL dataset for a JPEG subset of open-images-v7 to comply with our new caching system. This will also improve sampling reliability, as fetching URL images sometimes fails.

Release 1.2.9 - openjourney-v4 and animagine-xl-3.1

TLDR

• Two new diffusion models for validator challenges
  • prompthero/openjourney-v4
  • cagliostrolab/animagine-xl-3.1
• Better device management, now exposing --neuron.device through validator.env's DEVICE variable
• Deprecating unused prompt generation model due to the biases it displayed. We have replaced this with our ImageAnnotationGenerator pipeline.

New Model Details

We are introducing two additional popular open source image models for validator challenges, as part of our continuous effort to meaningfully expand the distribution of synthetic images that miners classify. Our goal is to encourage better miner generalizability to images produced by different deepfake methods.

For validators: No new dependencies/install requirements were added - models will automatically be downloaded to your local huggingface cache.

For miners: We foresee a difficulty increase in subnet challenges as miners adapt to the new models' image distributions.

The two models are:

• prompthero/openjourney-v4, a popular StableDiffusion model finetuned on +124k Midjourney v4 images, by PromptHero (158,394 downloads last month as of writing)
• cagliostrolab/animagine-xl-3.1, a popular StableDiffusionXL model finetuned on anime images, by Cagliostro Research Lab in collaboration with SeaArt.ai (411,846 downloads last month as of writing)

These were selected by

1. Manually reviewing state-of-the-art models (tech review blogs, AI news, Hugging Face models) and open source models finetuned on proprietary image generators (e.g. Midjourney) and/or popular generated image genres in-the-wild (e.g. anime)
2. Applying our subnet API benchmarking procedure to synthetic mirrors generated by candidate models.
   • Notably, our subnet scored worse on openjourney-v4 and animagine-xl-3.1 (68.76% and 83.8% accuracy) than any of our existing validator challenge models, indicating that miners have yet to optimally generalize to those model output distributions (supporting the addition of these models).
   • We sourced the real images for mirroring through a scraping method outlined in the technical blog post (link above).
   • API benchmarking for new model selection was conducted end-to-end within 24-HR spans to increase the likelihood that accuracy measures reflected a stable subnet state (as opposed to accuracy variations caused by significant miner changes on mainnet).

Release 1.2.8 - Validator Proxy Update

• Introducing hotkeys and coldkeys to validator proxy responses for flexibility in consumption through our api server
• Added rich param that allows caller to specify how much data they want
  • False will send back only preds and fqdn
  • true will send back the above along with uids, emissions, incentives, ranks, hotkeyks, coldkeys

Release 1.2.7: Introducing Data Augmentation Levels

We are introducing data augmentation levels in update 1.2.7, designed to better reward high-performing (better generalization) miners. TL;DR: 30% of data will be augmented with either 'medium'/level 2 or 'hard'/level 3 augmentations, which include distortions.

Augmentation Levels

• Level 0 (25%): No augmentations (base transforms).
• Level 1 (45%): Random transformations.
• Level 2 (15%): Random transformations + Medium distortions.
• Level 3 (15%): Random transformations + Hard distortions.

Distortion Intensity Levels* (indices 0 and 1 are used):

'CS': [0.4, 0.3, 0.2, 0.1, 0.0],    // smaller, worse
'CC': [0.85, 0.725, 0.6, 0.475, 0.35],    // smaller, worse
'BW': [16, 32, 48, 64, 80],    // larger, worse
'GNC': [0.001, 0.002, 0.005, 0.01, 0.05],    // larger, worse
'GB': [7, 9, 13, 17, 21],    // larger, worse
'JPEG': [2, 3, 4, 5, 6]    // larger, worse

*Detailed descriptions of constants in bitmind/image_transforms.py.

Medium Distortions
Applied Parameters: Color saturation, color contrast, JPEG compression.
Probability: 50% not applied, 50% applied at lowest intensity.

Hard Distortions
Applied Parameters: Color saturation, color contrast, JPEG compression, Gaussian noise color, Gaussian blur.
Probability: 33% not applied, 33% applied at lowest intensity, 33% applied at second intensity.

Release 1.2.6 - Hotfix: Real Image Sampling Edge Case

source_index = np.searchsorted(cumulative_sizes, index % (cumulative_sizes[-1]))
changed to
source_index = np.searchsorted(cumulative_sizes - 1, index % (cumulative_sizes[-1]))
to avoid an edge case where in the selected index == the size of one of the datasets, resulting in an out of bounds error

Release 1.2.5 - Expose Validator Proxy Port Arg

Adding VALIDATOR_PROXY_PORT to validator.env to allow validators on containerized services to easily configure a valid port