Older commit data can reappear on the bittensor network

[donaldknoller]

Given the following:

1. Miner registers onto a subnet
2. Miner creates a commit call
3. Miner gets deregistered

A miner can register again on the same subnet and their older commit data may still be retained. It is unclear how long the period for retaining this commit call on the network would last, but this may lead to inconsistency issues if left unchecked.

The above behavior can also be exploited when updating the huggingface repository details to effectively "submit" a new model under the guise of an older one.

Given that there are many implications for this current behavior, open discussion is encouraged here to decide on an optimal path forward

[Nadejde]

So in my view this is not necessarily a problem. If you look at it this way its a feature:

1. Miner registers on the subnet
2. Miner creates a commit call and the model is now in the queue awaiting evaluation
3. The queue is stuck or slow or very large and the key immunity expires until the model gets evaluated
4. Miner is deregistered after key immunity expires
5. Model eventually gets evaluated in the queue
6. Miner re-buys the same key again gets to keep the evaluated model at the original commit block.

This has happened to me a few times and I found it good that I could actually keep my original block for the model I submitted. With the evaluation being so slow right now I did have to pay an extra 2 tao but I have an option to not loose my submission to the cloners.

[donaldknoller]

It appears that this is a desired behavior, but will keep this open for further discussion down the road

[donaldknoller]

Update: for the specific case of modifying submissions after the commit data was sent, I have set up the following script as a temporary patch until a more thorough change is added:

import logging
from datetime import datetime
from huggingface_hub import HfApi
from model.scores import StatusEnum
from persistence import SupabaseState

logger = logging.getLogger(__name__)

import datetime

REFERENCE_BLOCK = 4041914
REFERENCE_TIME = datetime.datetime(2024, 10, 14, 14, 35, 30, tzinfo=datetime.timezone.utc)
BLOCK_INTERVAL = 12  # seconds

def utc_to_block(utc_time):
    """
    Convert a UTC datetime to the corresponding block number.
    
    :param utc_time: A datetime object in UTC
    :return: The estimated block number
    """
    if not utc_time.tzinfo:
        utc_time = utc_time.replace(tzinfo=datetime.timezone.utc)
    
    time_difference = utc_time - REFERENCE_TIME
    seconds_difference = time_difference.total_seconds()
    block_difference = int(seconds_difference / BLOCK_INTERVAL)
    
    return REFERENCE_BLOCK + block_difference

def block_to_utc(block_number):
    """
    Convert a block number to the corresponding UTC datetime.
    
    :param block_number: The block number
    :return: A datetime object in UTC
    """
    block_difference = block_number - REFERENCE_BLOCK
    seconds_difference = block_difference * BLOCK_INTERVAL
    
    return REFERENCE_TIME + datetime.timedelta(seconds=seconds_difference)


def check_model_history():
    supabase = SupabaseState()
    hf_api = HfApi()
    

    try:
        # Fetch all entries from the leaderboard
        response = supabase.supa_client().table("minerboard").select("*, leaderboard(*)").execute()
        minerboard_entries = response.data

        for minerboard_entry in minerboard_entries:
            # print(minerboard_entry)
            entry = minerboard_entry.get('leaderboard')
            repo_namespace = entry.get('repo_namespace')
            repo_name = entry.get('repo_name')

            if not repo_namespace or not repo_name:
                continue
            if entry['status'] != 'COMPLETED':

                continue
            if 'block' not in minerboard_entry:
                print(f"no block info available for {minerboard_entry}")
                continue
            entry['block'] = minerboard_entry['block']
            try:
                # Fetch repository details from Hugging Face
                repo_info = hf_api.repo_info(repo_id=f"{repo_namespace}/{repo_name}")
                f = repo_info.last_modified

                # print(f)
                # print(f"{f} maps to block {utc_to_block(f)}")
                modified_after_block = utc_to_block(f) > entry['block']
                if not modified_after_block:
                    continue
                print(f"entry: {repo_namespace}/{repo_name}")
                notes = f"Repository was last updated at {f} which corresponds to block {utc_to_block(f)}.\nThe commit data was submitted on {entry['block']} which suggests repository was modified after submission."
                print(notes)
                try:
                    response = (
                        supabase.supa_client().table("leaderboard")
                        .update(
                            {"status": StatusEnum.FAILED, "notes": notes},
                        )
                        .eq("hash", entry['hash'])
                        .execute()
                    )
                    
                except Exception as e:
                    print(f"Error updating leaderboard status for {hash}: {e}")
                    

                

            except Exception as e:
                logger.error(f"Error fetching info for {repo_namespace}/{repo_name}: {str(e)}")

    except Exception as e:
        logger.error(f"Error fetching leaderboard entries: {str(e)}")

if __name__ == "__main__":
    check_model_history()