Feat/compile with cmake #489
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I can test nvidia on Linux when it's ready, I'd prefer if we didn't change the default of compiling for metal when it's available (I think it's possible to detect) |
Metal should be enabled by default in llama.cpp and afaik it's automatically compiled by cmake of llama. See log (search for 'metal') cargo run --example usage qwen2-1_5b-instruct-q4_0.gguf
llama_model_loader: loaded meta data with 26 key-value pairs and 338 tensors from qwen2-1_5b-instruct-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = qwen2-1_5b-instruct
llama_model_loader: - kv 2: qwen2.block_count u32 = 28
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
llama_model_loader: - kv 4: qwen2.embedding_length u32 = 1536
llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 8960
llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 12
llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 2
llama_model_loader: - kv 8: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 9: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 10: general.file_type u32 = 2
llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 12: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 15: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 19: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 21: general.quantization_version u32 = 2
llama_model_loader: - kv 22: quantize.imatrix.file str = ../Qwen2/gguf/qwen2-1_5b-imatrix/imat...
llama_model_loader: - kv 23: quantize.imatrix.dataset str = ../sft_2406.txt
llama_model_loader: - kv 24: quantize.imatrix.entries_count i32 = 196
llama_model_loader: - kv 25: quantize.imatrix.chunks_count i32 = 1937
llama_model_loader: - type f32: 141 tensors
llama_model_loader: - type q4_0: 193 tensors
llama_model_loader: - type q4_1: 3 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens cache size = 293
llm_load_vocab: token to piece cache size = 0.9338 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 151936
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 1536
llm_load_print_meta: n_layer = 28
llm_load_print_meta: n_head = 12
llm_load_print_meta: n_head_kv = 2
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 6
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 8960
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 1.54 B
llm_load_print_meta: model size = 888.43 MiB (4.83 BPW)
llm_load_print_meta: general.name = qwen2-1_5b-instruct
llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token = 151645 '<|im_end|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
llm_load_print_meta: EOT token = 151645 '<|im_end|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size = 0.30 MiB
ggml_backend_metal_log_allocated_size: allocated buffer, size = 888.44 MiB, ( 888.50 / 10922.67)
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors: CPU buffer size = 182.57 MiB
llm_load_tensors: Metal buffer size = 888.43 MiB
....................................................................
llama_new_context_with_model: n_ctx = 512
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1
ggml_metal_init: picking default device: Apple M1
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name: Apple M1
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction support = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB
llama_kv_cache_init: Metal KV buffer size = 14.00 MiB
llama_new_context_with_model: KV self size = 14.00 MiB, K (f16): 7.00 MiB, V (f16): 7.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.58 MiB
llama_new_context_with_model: Metal compute buffer size = 299.75 MiB
llama_new_context_with_model: CPU compute buffer size = 4.01 MiB
llama_new_context_with_model: graph nodes = 986
llama_new_context_with_model: graph splits = 2
I'm just a computer program, so I don't have feelings or emotions. How can I assist you today?
ggml_metal_free: deallocating |
Nvidia should be ready already on Windows / Linux :) |
|
I think that the symbols renaming in the current build.rs are actually useful. See ggerganov/llama.cpp#9267 |
|
linux + cuda is working well. |
|
https://github.com/utilityai/llama-cpp-rs/actions/runs/10655804883/job/29576603420?pr=489 should be passing though. |
|
|
|
I pushed new code. It should fix that last error. |
That would be ideal. I wonder if both llama.cpp and whisper.cpp vendoring (sometimes patched) ggml would cause issues. |
MarcusDunn
reviewed
Sep 4, 2024
Comment on lines
+31
to
50
| fn copy_folder(src: &Path, dst: &Path) { | ||
| std::fs::create_dir_all(dst).expect("Failed to create dst directory"); | ||
| if cfg!(unix) { | ||
| std::process::Command::new("cp") | ||
| .arg("-rf") | ||
| .arg(src) | ||
| .arg(dst.parent().unwrap()) | ||
| .status() | ||
| .expect("Failed to execute cp command"); | ||
| } | ||
|
|
||
| if cfg!(windows) { | ||
| std::process::Command::new("robocopy.exe") | ||
| .arg("/e") | ||
| .arg(src) | ||
| .arg(dst) | ||
| .status() | ||
| .expect("Failed to execute robocopy command"); | ||
| } | ||
| } |
There was a problem hiding this comment.
this should probably fail if neither cfg! blocks run. (if unix / else if windows / else fail)
|
I think everything there looks good. I'll merge, and see if people start complaining about things being broken! |
MarcusDunn
approved these changes
Sep 4, 2024
|
https://github.com/utilityai/llama-cpp-rs/actions/runs/10710926865 failed, would you mind adding the required files to the include block? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Todo:
Resolve #486 #454
Update: llama.cpp with vulkan much faster than llama-cpp-rs with vulkan in usage example on Windows. On Linux it's fast.
maybe the context creation is the problem in the example