TorchSharp NanoGpt

[GeorgeS2019]

https://github.com/biegehydra/NanoGptDotnet/blob/master/src/NanoGpt/NanoGpt.cs

[ejhg]

Thank you @GeorgeS2019 !

I can almost feel the AGI now!

[travisjj]

This is a nicely implemented version of Karpathy's first setup of reproducing GPT, but does not include the updates from GPT-2 such as some of the CUDA performance improvements, the checkmarks, and some minor tweaks to preactivation.

[GeorgeS2019]

@travisjj check these:
https://github.com/biegehydra/NanoGptDotnet
https://github.com/PentoreXannaci/NanoGPTNet

[ejhg]

Along these lines, I have been playing a lot with o1-preview. I copy and paste large chunks of transformer related python code and ask it to translate it to C#. It is mind-blowingly effective. The tasks I give it are particularly challenging because I'm asking it to rewrite llama from scratch without dependencies, or I ask it to unflatten tensor operations a certain way, etc. This approach could be effective in bringing over any of Karpathy's implementations to C#/TorchSharp.

[ejhg]

Example: I couldn't find any C# code to train a Llama model from a quick google search, so I asked it to write it for me from one of Karparthy's repos. (I still have to test it...)

https://chatgpt.com/share/672d06a0-e6fc-8009-b29a-408e0575a4bd

[travisjj]

Hi @GeorgeS2019 , one of those (NanoGPTNet) is a simple nano-gpt.com wrapper, which doesn't really include any torchsharp code unfortunately. Pentore is also from the older version released by Karpathy. A great indicator is the lack of GELU, but there are more subtleties as well such as all of the performance increases he gains in the subsequent version; for example with F.scaled_dot_product_attention inside of the CausalSelfAttention's forward pass.

I do have a working version I could share, perhaps with some polish... I am used to publishing c# enterprise code and this code isn't commented and the call to start the training itself is rather chicken scratch. I haven't played around too much with it, but on an Azure A100 it hit 600k tokens per second, and 200k tokens processed per second on gradient accumulation.

[travisjj]

Part of the intent with Karpathy making NanoGPT was to be able to mimic GPT-2 in such a way as to be able to load it's model weights. As I have gotten further into developing NanoGPT in TorchSharp I have noticed that the parameter count is slightly off from what it is in PyTorch.

I am using the hyper parameters that should be equivalent to GPT-2 small:

var gptConfig = new HyperParams()
{
    BlockSize = 1024,
    VocabSize = 50257,

    BlockLayers = 12,
    NumberOfHeads = 12,

    EmbedDimensions = 768
};

(huggingface config for GPT-2: https://huggingface.co/docs/transformers/en/model_doc/gpt2#transformers.GPT2Config)

Unfortunately this is yielding 163M (163037184), which is off from GPT-2 small's 124M. As a result, and this is the important part, I cannot load a pretrained model from huggingface. I tried using the same train.bin that Karpathy loads but it is a mismatch. If anyone was curious about training their own models, it looks like the 163M parameter model would take 3.7 days per epoch on an Azure NVidia A100.

Of the other implementations of NanoGPT in TorchSharp, has anyone seen any which will load a huggingface model using Model.Load that "just works"?

[travisjj]

I did a flat parameter count for 163M, but failed to notice the weight sharing aspect to gpt2. This brings my model to the exact 124M parameter count expected from gpt2. Having some issues loading the huggingface model though due to mismatch. It looks like the names in TorchSharp contain k__BackingField when there are auto implemented properties

If anyone has advice for loading pretrained huggingface models (especially the gpt-2 small) into TorchSharp I would greatly appreciate it.

[travisjj]

This appears to be related to a bug during the RegisterComponents method.

fieldInfo.ComponentName()

Is causing erroneous names to be created. When saved, the model.save does not care because it will always recreate it properly (since the conventions match); however, this makes loading pretrained models very difficult. Perhaps instead of using this reflection approach we just require the names to be registered explicitly, or use some regex to remove non variable named values inside of the name string. More to come.

[ejhg]

Are you loading from pickle files or what kind of format?

If you are using pickle (using the TorchSharp.PyBridge library), I was able to easily write a custom loader here: https://github.com/ejhg/llama-torchsharp/blob/main/src/unpickler/DelayedExecutionLoader.cs

[travisjj]

I managed to sort this all out. I am making a somewhat polished version to share. Currently applying to places so I can't just push out a code cowboy version unfortunately :) :D

Here's what was happening and what I managed to do to get it working.

At first the loading was all messed up because I was trying to load a pickled file, which just doesn't work. On top of that, the state dictionary for TorchSharp was kind of all over the place due to extra text being added in from reflection reading autoprop names as well as from RegisterComponents taking everything when some things were in fact nodes not edges of the graph.

I solved this by loading the huggingface model directly (not Karpathy's shard version), and then in python using exportsd.py (https://github.com/dotnet/TorchSharp/blob/main/src/Python/exportsd.py) which worked very well to get everything into a nice .dat file. In addition, I removed all of the auto properties and manually composed the component registration, using the specific names expected from the huggingface model's state dictionary.

The only thing that was interesting was the fact that part of the file came in transposed, but luckily Karpathy solved for that in his video and I adapated that part in as well (the .transpose in the qkv forward pass). I was able to take parts of TorchSharp's model.load functionality and modify it to allow transposition of shapes for this to load.

So now the whole thing loads. I will test a fine tune run on it and hopefully get some stuff shared that JustRuns™ by next week.

[ejhg]

Can you share your branch to repro?

[travisjj]

I definitely will, however, it is currently prepped to run with cuda enabled on a linux Azure instance and lacks comments; so... maybe I should repackage (to cpu? what is everyone training these on? hobby? home gpu? Bueller!?) prior to sharing this exact version. I was mostly looking to reproduce the functionality prior to making this anything larger. To be honest I fully agree with your sentiment about llama2 and that direction.

[ejhg]

(Just read this reply.) I run on my Macbook Air (apple silicon). I read and crack code--comments not needed ;)

[travisjj]

I was trying to make sure it worked as well as was readable. Looks like it is loading from huggingface now which is nice. See above comment for some expansion. Hopefully code will be ready next week to share here.

[travisjj]

@ejhg Full project available: https://github.com/travisjj/NanoGPT2

It doesn't include the model weights from huggingface though. The file is quite large, also, it seems that the weights don't fair well when device.CPU is true, so I will need to see about converting them in a GPU environment.

Interestingly, in Python, using a CPU to generate with the pipeline for GPT2 gives different results than when using a GPU.

[ejhg]

Congrats @travisjj ! I'll try to run over the weekend (or next week).

using a CPU to generate with the pipeline for GPT2 gives different results than when using a GPU

To be expected right? ...since they use a different pseudo random number sequence. Also floating point operations on the GPU are sometimes not as accurate as those on the CPU, so even if you set the temperature to 0 and initialize the same weights, there's a chance that numeric error accumulates differently AFAIK.

[ejhg]

Personally, I've shifted my attention away from nanogpt towards llama2.c instead: https://github.com/karpathy/llama2.c

IMO, the complexity of llama is comparable to nanogpt, yet llama uses a more modern architecture, and you have much better open source pre-trained models available to plug and play.

The neat thing about llama2.c is that you can run inference on the cpu, and the inference file is written in a single C file from scratch.

There's already a C# port of llama2.c, and I have my own port as well... If you compile with optimizations and AOT, it runs impressively fast, comparable to the original C version, though I don't have a benchmark.

[ejhg]

@travisjj I haven't tested this, but you would need to run Karpathy's script to convert the huggingface model into a .bin file, which you can then run in any of the llama2.c versions. This is the script: https://github.com/karpathy/llama2.c/blob/master/export.py

Here's a C# version of the CPU inference file: https://github.com/trrahul/llama2.cs

Here's my version before I broke it up into smaller files: https://github.com/ejhg/transformers/blob/21b0234f7de6d1f4bd135adc41baaed93128169f/llama.cs/model.cs

It's worth noting that I used o1-preview heavily to craft my version. Not only did it perform the initial translation, but I guided it to refactor it in certain ways. For instance, below is what my weights look like. Notice that I use multi-dimensional arrays instead of flat arrays like Karpathy and the other versions do. Since we're not running on GPUs, there's no point in flattening arrays and losing structure. It was very tricky to get this right, and some of the arrays can still be broken down more. I verified that this did not impact performance. I resorted to unsafe code in one of the matrix multiplies. The thread parallelization was ChatGPT's idea...

    public class LayerWeights
    {
        // Weights for RMSNorms
        public float[] rms_att_weight; // [dim]
        public float[] rms_ffn_weight; // [dim]

        // Weights for matmuls. note dim == n_heads * head_size
        public float[,] wq; // [n_heads * head_size][dim]
        public float[,] wk; // [n_kv_heads * head_size][dim]
        public float[,] wv; // [n_kv_heads * head_size][dim]
        public float[,] wo; // [dim][n_heads * head_size]

        // Weights for FFN
        public float[,] w1; // [hidden_dim][dim]
        public float[,] w2; // [dim][hidden_dim]
        public float[,] w3; // [hidden_dim][dim]
    }

[ejhg]

This is llama inference running on TorchSharp, and it does load hugging face files successfully: https://github.com/ejhg/llama-torchsharp

I tested it with both Llama 2 and 3. Llama 3 does work, but there is one scaling layer that isn't implemented, so it works, but it could be better.

I plan to reuse the loader from this repo and combine it with the CPU implementation mentioned above so Karpathy's python script can be bypassed.

[ejhg]

Do the llama2 pre-trained models load in the c# ported versions you have seen or created?

At least for my port, I wrote a custom loader to load *.pth (pickle) files (similar to PyBridge, but barebones and bypassing TorchSharp completely), but I haven't pushed it because there seems to be a bug in my port TBD...

[travisjj]

What kind of bug? I ported a model converter as well if you wanted me to share it. It's in python and produces a .dat. Although it seems to load in transposed for a few properties of GPT2 (Karpathy notes this, so I wonder if it is just there or if other places are transposed as well).

[ejhg]

Reminder: My goal is to implement a basic llama (min-gpt-ish) architecture from scratch using pure C#

I wish I had time to clean this up, but just pushed what I have in case you want to check it out: https://github.com/ejhg/transformers

You'll need to read and hack the Program.cs (or simply ask me) to load the models if you want to run it.

The model files for TinyStories can be downloaded from here https://huggingface.co/karpathy/tinyllamas/tree/main

Current status

Based on what I recall from testing a week ago.

command	status	description
llama.cs:stories-15m	inference works	loads from pickle file
llama.cs:stories-15m.bin	inference works	loads from Karpathy binary file
llama.cs:llama-3.2-1b-instruct	inference produces gibberish	loads from pickle file
llama.torchsharp:llama-3.2-1b-instruct	inference works	reference impl using torchsharp

Things I need to try out:

• Test with pickle llama 2 instead of llama 3
• The issue is probably in the attention mechanism refactor I made. I could test using someone else's c# port.

[travisjj]

I have a public version available for NanoGPT here: https://github.com/travisjj/NanoGPT2

Please let me know if anything seems off with it. It trains well, although it has the same pitfalls the original GPT-2 model had in general, and would take quite a lot of compute to train on openwebtext to a low loss rate.

[ejhg]

Sanity check: When you train it from scratch against shakespeare.txt does the loss go down, and does it start producing something sensible? If I recall correctly, the python implementation would start converging within less than 500-1000 epochs for the small model I tested.

Maybe if you include some sample benchmark timing or loss versus epoch using a minimal setup in your README, that could help others who are checking out your project. Also interesting, how does it perform against python for a basic setup?

Great job again @travisjj !!!

[travisjj]

I guess that depends on your definition of sensible :) Even the huggingface model for ChatGPT-2 in python on a gpu produces quite odd results.

You can run it from a bash command line in linux if cuda is installed with:

python -c "from transformers import pipeline, set_seed; \
generator = pipeline('text-generation', model='gpt2-medium'); \
set_seed(42); \
res = generator(\"Hello, I'm a language model,\", max_length=30, num_return_sequences=5); \
print(res)"

I ran NanoGPT2 for approximately 8 hours on openwebtext on an A100 and got to a loss of about 4.9 using gradient accumulation. It was interesting, but still not very sensible.

The temperature, topk, whether or not to use gradient accumulation, token (B,T), specific tokenization approaches, and vocab minimalization all seem to play large roles in the loss rates. As such, and because it is a little pricey to train while in some ways guessing, I haven't spent a large amount of time pursuing lower loss rates. It is possible to do though.

Using this, I trained a very simple, we will call it pico-sized transformer, on my Stack Overflow content which was mixed with code. It does attempt to mix code and verbiage, but not does not respond well to prompt with regards to alignment. It was also only trained for 4 hours.

Unfortunately, Visual Studio lacks a good visualization integration for displaying some of this data.

If you had something in mind with regards to what would go into benchmarking, I would be happy to run some of those and show what loss it gets to along with a sample of the output text from the training display and a generation or two. For example, should I use no temperature (1.0), 50 as topk (default for huggingface); or something else? Should I include gradient accumulation or not? What should the token batch and time frames look like for (B,T); currently I use (64,256). The most common setup I have been using is topk 50, temperature 1.0, (B,T) = (64,256) and GPT2-small with gradient accumulation. However, I am curious if (32,1024) and no gradient accumulation would reach a lower loss quicker, and if so, would it be as accurate?

Anyway, let me know, the benchmark idea is interesting and I would certainly entertain giving it a shot.

[ejhg]

I use this fork [email protected]:ejhg/minGPT.git as my reference. The value add of this fork vs the original repo is that the different examples that Karpathy provides are codified within the "tests" directory, so you can run them (and automate them in CI) easily.

To benchmark training on text, I simply run python tests/test_shakespeare.py.

This benchmark can be run within seconds/minutes on a laptop, and you can see within ~300 iterations below that the loss has already gone down to ~2.2, and the output starts to resemble the dataset. If your implementation can produce something similar, then you have reassurance that it is probably working alright.

The bond sim to disell the bluch
Sut talce for ford fresinctiong pa

>>  python tests/test_shakespeare.py
system:
model:
    model_type: gpt
    n_layer: 6
    n_head: 6
    n_embd: 192
    vocab_size: None
    block_size: None
    embd_pdrop: 0.1
    resid_pdrop: 0.1
    attn_pdrop: 0.1
data has 1115394 characters, 65 unique.
transformer parameters: 2.69M
total of parameters: 2.71M
running on device cpu
Theu t  u otl ete d  cossu clee sur eu d t fh,  ce   ds s eldr rhhe
iter_dt 0.00ms,iter 0,loss 4.19184
iter_dt 373.92ms,iter 10,loss 3.00594
iter_dt 369.32ms,iter 20,loss 2.75432
iter_dt 375.47ms,iter 30,loss 2.61354
iter_dt 373.97ms,iter 40,loss 2.56515
iter_dt 376.21ms,iter 50,loss 2.59632
iter_dt 369.33ms,iter 60,loss 2.48572
iter_dt 376.07ms,iter 70,loss 2.50596
iter_dt 376.47ms,iter 80,loss 2.49347
iter_dt 379.93ms,iter 90,loss 2.46803
The and sistes, be hand butones plimy outoul m tout thin
Thet wing 
iter_dt 378.10ms,iter 100,loss 2.45311
iter_dt 379.66ms,iter 110,loss 2.39890
iter_dt 375.51ms,iter 120,loss 2.41220
iter_dt 374.12ms,iter 130,loss 2.42018
iter_dt 377.36ms,iter 140,loss 2.39410
iter_dt 374.20ms,iter 150,loss 2.36301
iter_dt 378.70ms,iter 160,loss 2.34126
iter_dt 383.88ms,iter 170,loss 2.37171
iter_dt 438.32ms,iter 180,loss 2.34545
iter_dt 414.84ms,iter 190,loss 2.33489
The hteirt ind; meam t o nit thea the my simsteacord ily oug hancks
iter_dt 405.82ms,iter 200,loss 2.33699
iter_dt 426.65ms,iter 210,loss 2.31672
iter_dt 437.32ms,iter 220,loss 2.28934
iter_dt 425.87ms,iter 230,loss 2.27844
iter_dt 377.72ms,iter 240,loss 2.25207
iter_dt 372.05ms,iter 250,loss 2.23734
iter_dt 383.38ms,iter 260,loss 2.22233
iter_dt 376.74ms,iter 270,loss 2.22855
iter_dt 372.86ms,iter 280,loss 2.21433
iter_dt 374.41ms,iter 290,loss 2.19018
The bond sim to disell the bluch
Sut talce for ford fresinctiong pa
iter_dt 374.66ms,iter 300,loss 2.20630