how to specify the number of cpu threads?

[raingart]

[imtuyethan]

"n_gl" or "n_gpu_layers" is a setting that controls how many layers of the AI model are loaded into the GPU memory. It's basically a way to balance between speed and memory usage:

• Higher n_gpu_layers = more GPU memory used, faster processing
• Lower n_gpu_layers = less GPU memory used, potentially slower processing

Suggestions

• For 7B models: Try starting with 32 layers (n_gpu_layers=32)
• For 13B models: Start with around 20-24 layers
• For larger models: You may need to go even lower, perhaps 16 or fewer layers