Skip to content

Commit

Permalink
added SN dir
Browse files Browse the repository at this point in the history
  • Loading branch information
isabella618033 committed Mar 28, 2024
1 parent 8cc3b8c commit f61b15e
Show file tree
Hide file tree
Showing 3 changed files with 465 additions and 0 deletions.
155 changes: 155 additions & 0 deletions SN1_text_prompting/demo.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SN netuid - sn_name demo\n",
"\n",
"This demo notebook should fufill the following tasks.\n",
" - (A) Demonstrate the quality of the communication between miners and validators coming from the top miner.\n",
" - (B) Justify the difference in incentive for miners in different tiers (eg.quantile 1 VS quantile 3).\n",
" - (C) (If applicable) Show the landscape and variety of miners. \n",
" - (D) (If applicable) Demonstrate the effectiveness of the scoring mechanism.\n",
" - (E) (If applicable) Show the dataset that was used by the validator.\n",
"\n",
" - (F) (If applicable) Show the use of any API and/or links to a frontend.\n",
" \n",
" - ** If you have difficulty in completing any of the task and need an alternative, please feel free to contact Isabella(isabella618033)/ Eugene(eugene3684) from the Opentensor Foundation on Discord.\n",
"- ** We understand that SNs runs very differently, so please feel free to make any modification to the notebook that best suit your SN as long as it can demonstrate\n",
"- ** For any wallet/ dendrite calls needed, we will be using the foundation hotkey \n",
"\n",
"## Objective\n",
"> Please write the objective of the SN with precision. \n",
"> - The task of the miner (what task does the validator send to miner, and what are miner supposed to response with.)\n",
"> - The scoring mechanism (please specify is there is any model involved)\n",
">\n",
"> - The dataset used\n",
"\n",
"## Setup\n",
"> Please give instruction for how to run the notebook"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (A) Top miner responses\n",
"- This section should demonstrate the quality of the communication between miners and validators coming from the top miner.\n",
"\n",
"> - (1) Define a group of top miner.\n",
">\n",
"> - (2) Define a forward function. \n",
"> - (3) Call the forward function for the top miners. \n",
"> - (4) Show the responses from the miners."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (B) Justification for incentive distribution \n",
"- Justify the difference in incentive for miners in different incentive tiers (eg. sample 5 miners from quantile 1 VS 5 miners from quantile 3) with code.\n",
"- If there is no significant difference in the incentive distribution, you can also show that miners in the SN have about the same performance in multiple ways.\n",
"\n",
"- There could be many reasons for the difference in incentive for miners. \n",
" - Case 1: Difference in the quality of response\n",
" - Show that miners with higher incentive generally give better answer then those with lower incentive through the following ways\n",
" - lower loss; higher accuracy\n",
" - human eval for text/ image/ audio quality \n",
"\n",
" - Case 2: Difference in miner avalibility \n",
" - Then you can show that given a certain number of trials(100), there are more successful calls to higher incentive miners.\n",
" \n",
" - Case 3: Difference in latency.\n",
" - Then you can show that miners in Q1 generally response in faster time rather then miners in Q3.\n",
"\n",
" - Case 4: Please provide your own justification if the reasons above dosen't fit.\n",
"\n",
"> (1) Define the group of high incentive miner VS low incentive miner (you can have ~5 samples from each group, but please feel to make your own definition of high/low incentive miners)\n",
">\n",
"> (2) Make the forward call to the group of high/low incentive miners \n",
">\n",
"> (3) Show the difference in the quality of the high/low incentive miners \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (C) (If applicable) Miner landscape\n",
"- How many unique responses can we get from the network and how many miners are giving the same responses. It is perfectly fine even if all of the miners respond the same thing.\n",
"> (1) Send the same request to all miners over the SN\n",
">\n",
"> (2) Check the number of unique responses \n",
"> \n",
"> (3) Check the number of miners giving the same response\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ## (D) (If applicable) Demonstrate the effectiveness of the scoring mechanism.\n",
"- If you are using a reward/penalty model: \n",
" - Please load the reward or penalty model one by one and then show that the reward of a good response > the reward of a bad response\n",
" - Please also allow us to customise the input of the reward model\n",
"\n",
" > (1) Load the reward/penalty model one by one \n",
" >\n",
" > (2) Define the good/bad response\n",
" >\n",
" > (3) Score the response with the model\n",
"\n",
"- Otherwise, you may just give a brief explanation to how does your scoring mechanism works."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" ## (E) (If applicable) Show the dataset that was used by the validator.\n",
" > (1) Load the dataset \n",
" > \n",
" > (2) Show the first 10 samples of the dataset \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ## (F) (If applicable) Demonstrate the use of any API and/or links to a frontend.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Consent: Do you want this demo notebook to be public? Yes/No "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
155 changes: 155 additions & 0 deletions SN21_filetao/demo.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SN netuid - sn_name demo\n",
"\n",
"This demo notebook should fufill the following tasks.\n",
" - (A) Demonstrate the quality of the communication between miners and validators coming from the top miner.\n",
" - (B) Justify the difference in incentive for miners in different tiers (eg.quantile 1 VS quantile 3).\n",
" - (C) (If applicable) Show the landscape and variety of miners. \n",
" - (D) (If applicable) Demonstrate the effectiveness of the scoring mechanism.\n",
" - (E) (If applicable) Show the dataset that was used by the validator.\n",
"\n",
" - (F) (If applicable) Show the use of any API and/or links to a frontend.\n",
" \n",
" - ** If you have difficulty in completing any of the task and need an alternative, please feel free to contact Isabella(isabella618033)/ Eugene(eugene3684) from the Opentensor Foundation on Discord.\n",
"- ** We understand that SNs runs very differently, so please feel free to make any modification to the notebook that best suit your SN as long as it can demonstrate\n",
"- ** For any wallet/ dendrite calls needed, we will be using the foundation hotkey \n",
"\n",
"## Objective\n",
"> Please write the objective of the SN with precision. \n",
"> - The task of the miner (what task does the validator send to miner, and what are miner supposed to response with.)\n",
"> - The scoring mechanism (please specify is there is any model involved)\n",
">\n",
"> - The dataset used\n",
"\n",
"## Setup\n",
"> Please give instruction for how to run the notebook"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (A) Top miner responses\n",
"- This section should demonstrate the quality of the communication between miners and validators coming from the top miner.\n",
"\n",
"> - (1) Define a group of top miner.\n",
">\n",
"> - (2) Define a forward function. \n",
"> - (3) Call the forward function for the top miners. \n",
"> - (4) Show the responses from the miners."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (B) Justification for incentive distribution \n",
"- Justify the difference in incentive for miners in different incentive tiers (eg. sample 5 miners from quantile 1 VS 5 miners from quantile 3) with code.\n",
"- If there is no significant difference in the incentive distribution, you can also show that miners in the SN have about the same performance in multiple ways.\n",
"\n",
"- There could be many reasons for the difference in incentive for miners. \n",
" - Case 1: Difference in the quality of response\n",
" - Show that miners with higher incentive generally give better answer then those with lower incentive through the following ways\n",
" - lower loss; higher accuracy\n",
" - human eval for text/ image/ audio quality \n",
"\n",
" - Case 2: Difference in miner avalibility \n",
" - Then you can show that given a certain number of trials(100), there are more successful calls to higher incentive miners.\n",
" \n",
" - Case 3: Difference in latency.\n",
" - Then you can show that miners in Q1 generally response in faster time rather then miners in Q3.\n",
"\n",
" - Case 4: Please provide your own justification if the reasons above dosen't fit.\n",
"\n",
"> (1) Define the group of high incentive miner VS low incentive miner (you can have ~5 samples from each group, but please feel to make your own definition of high/low incentive miners)\n",
">\n",
"> (2) Make the forward call to the group of high/low incentive miners \n",
">\n",
"> (3) Show the difference in the quality of the high/low incentive miners \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (C) (If applicable) Miner landscape\n",
"- How many unique responses can we get from the network and how many miners are giving the same responses. It is perfectly fine even if all of the miners respond the same thing.\n",
"> (1) Send the same request to all miners over the SN\n",
">\n",
"> (2) Check the number of unique responses \n",
"> \n",
"> (3) Check the number of miners giving the same response\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ## (D) (If applicable) Demonstrate the effectiveness of the scoring mechanism.\n",
"- If you are using a reward/penalty model: \n",
" - Please load the reward or penalty model one by one and then show that the reward of a good response > the reward of a bad response\n",
" - Please also allow us to customise the input of the reward model\n",
"\n",
" > (1) Load the reward/penalty model one by one \n",
" >\n",
" > (2) Define the good/bad response\n",
" >\n",
" > (3) Score the response with the model\n",
"\n",
"- Otherwise, you may just give a brief explanation to how does your scoring mechanism works."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" ## (E) (If applicable) Show the dataset that was used by the validator.\n",
" > (1) Load the dataset \n",
" > \n",
" > (2) Show the first 10 samples of the dataset \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ## (F) (If applicable) Demonstrate the use of any API and/or links to a frontend.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Consent: Do you want this demo notebook to be public? Yes/No "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading

0 comments on commit f61b15e

Please sign in to comment.