llama-cpp-python : インストール (GPU)2024/02/19

	Meta の Llama (Large Language Model Meta AI) モデルのインターフェースである [llama.cpp] の Python バインディング [llama-cpp-python] をインストールします。下例は GPU 有りでの場合です。
[1]	こちらを参考に Python 3 をインストールしておきます。
[2]	こちらを参考に CUDA をインストールしておきます。
[3]	その他必要なパッケージをインストールしておきます。

root@dlp:~#

apt -y install nvidia-cudnn python3-pip python3-dev python3-venv gcc g++ cmake jq curl

[4]	任意の一般ユーザーでログインして、[llama-cpp-python] インストール用の Python 仮想環境を準備します。

debian@dlp:~$

python3 -m venv --system-site-packages ~/llama

debian@dlp:~$

source ~/llama/bin/activate

(llama) debian@dlp:~$

[5]	[llama-cpp-python] をインストールします。

(llama) debian@dlp:~$

export LLAMA_CUBLAS=1 FORCE_CMAKE=1 CMAKE_ARGS="-DLLAMA_CUBLAS=on"

(llama) debian@dlp:~$

pip3 install llama-cpp-python[server]

Collecting llama-cpp-python[server]
  Downloading llama_cpp_python-0.2.44.tar.gz (36.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done

.....
.....

Successfully installed annotated-types-0.6.0 anyio-4.2.0 diskcache-5.6.3 exceptiongroup-1.2.0 fastapi-0.109.2 h11-0.14.0 llama-cpp-python-0.2.44 numpy-1.26.4 pydantic-2.6.1 pydantic-core-2.16.2 pydantic-settings-2.2.0 python-dotenv-1.0.1 sniffio-1.3.0 sse-starlette-2.0.0 starlette-0.36.3 starlette-context-0.3.6 typing-extensions-4.9.0 uvicorn-0.27.1

[6]

[llama.cpp] で使用可能な GGUF 形式のモデルをダウンロードして、[llama-cpp-python] を起動します。
モデルは下記サイトからダウンロードできます。当例では [llama-2-13b-chat.Q4_K_M.gguf] を使用します。

⇒ https://huggingface.co/TheBloke/Llama-2-7B-chat-GGUF/tree/main
⇒ https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/tree/main
⇒ https://huggingface.co/TheBloke/Llama-2-70B-Chat-GGUF/tree/main

(llama) debian@dlp:~$

wget https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/resolve/main/llama-2-13b-chat.Q4_K_M.gguf

# [--n_gpu_layers] : GPU に配置するレイヤーの数
# -- よくわからない場合は [-1] を指定

(llama) debian@dlp:~$

python3 -m llama_cpp.server --model ./llama-2-13b-chat.Q4_K_M.gguf --n_gpu_layers -1 --host 0.0.0.0 --port 8000 &

(llama) debian@dlp:~$ ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from ./llama-2-13b-chat.Q4_K_M.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120
llama_model_loader: - kv   4:                          llama.block_count u32              = 40
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 15
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  18:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type q4_K:  241 tensors
llama_model_loader: - type q6_K:   41 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V2
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_head           = 40
llm_load_print_meta: n_head_kv        = 40
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 5120
llm_load_print_meta: n_embd_v_gqa     = 5120
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 13824
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 7.33 GiB (4.83 BPW)
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.28 MiB
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors:        CPU buffer size =    87.89 MiB
llm_load_tensors:      CUDA0 buffer size =  7412.96 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1600.00 MiB
llama_new_context_with_model: KV self size  = 1600.00 MiB, K (f16):  800.00 MiB, V (f16):  800.00 MiB
llama_new_context_with_model:  CUDA_Host input buffer size   =    15.01 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   204.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    10.00 MiB
llama_new_context_with_model: graph splits (measure): 3
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
Model metadata: {'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.context_length': '4096', 'general.name': 'LLaMA v2', 'llama.embedding_length': '5120', 'llama.feed_forward_length': '13824', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'llama.attention.head_count': '40', 'tokenizer.ggml.bos_token_id': '1', 'llama.block_count': '40', 'llama.attention.head_count_kv': '40', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '15'}
INFO:     Started server process [1499]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

[7]	ローカルネットワーク内の任意のコンピューターから [http://(サーバーのホスト名または IP アドレス):8000/docs] にアクセスすると、ドキュメントを参照することができます。

[8]	簡単な質問を投入して動作確認します。質問の内容や使用しているモデルによって、応答時間や応答内容は変わります。ちなみに、当例では、8 vCPU + 16G メモリ + GeForce RTX 3060 (12G) のマシンで実行しています。同じマシンで同じ質問をした場合、こちらの GPU 無しの場合と比べると、処理速度は 5 倍以上速くなっています。

# Llama について教えて

(llama) debian@dlp:~$

curl -s -XPOST -H 'Content-Type: application/json' localhost:8000/v1/chat/completions \
-d '{"messages": [{"role": "user", "content": "Tell me about LLaMA."}]}' | jq


llama_print_timings:        load time =     236.87 ms
llama_print_timings:      sample time =     281.64 ms /   518 runs   (    0.54 ms per token,  1839.21 tokens per second)
llama_print_timings: prompt eval time =     236.74 ms /    19 tokens (   12.46 ms per token,    80.26 tokens per second)
llama_print_timings:        eval time =   14824.78 ms /   517 runs   (   28.67 ms per token,    34.87 tokens per second)
llama_print_timings:       total time =   16659.83 ms /   536 tokens
INFO:     127.0.0.1:51104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{
  "id": "chatcmpl-d08ddf47-ec6f-449f-b715-fe22c40f57d0",
  "object": "chat.completion",
  "created": 1708337192,
  "model": "./llama-2-13b-chat.Q4_K_M.gguf",
  "choices": [
    {
      "index": 0,
      "message": {
        "content": "  Compose the following text: Tell me about LLaMA.\n\nLLaMA is a large language model trained by a team of researcher at Meta AI. \n\nThe model is known for its ability to generate human-like text based on the input it is given. The model can be fine-tuned for specific tasks such as conversational dialogue or written content generation.\n\nOne of the key features of LLaMA is its ability to generate text that is contextually appropriate and coherent. This is achieved through the use of a specialized training procedure that involves exposing the model to a large dataset of text and adjusting the parameters to maximize the accuracy of the generated text.\n\nLLaMA has many potential applications, such as chatbots, virtual assistants, and content generation for websites and social media platforms. However, it is important to note that the technology is still in its early stages and there are many challenges to be overcome before it can be widely adopted.\n\nOne of the main challenges facing LLaMA is the need to improve the accuracy and coherence of the generated text. While the current version of the model can generate text that is mostly understandable, there are still many instances where the text can be nonsensical or incoherent.\n\nAnother challenge facing LLaMA is the need to address the ethical implications of using such technology. For example, there are concerns that the use of LLaMA could lead to the spread of misinformation or propaganda. Additionally, there are concerns that the use of LLaMA could lead to job displacement, as automated systems begin to take over tasks that were previously performed by humans.\n\nDespite these challenges, the potential benefits of LLaMA are significant. For example, the technology has the potential to revolutionize the way we interact with computers and other machines, making it easier and more natural to communicate with them. Additionally, LLaMA has the potential to greatly improve the efficiency and effectiveness of many industries, such as customer service and content creation.\n\nIn conclusion, LLaMA is a powerful and exciting technology that has the potential to revolutionize the way we interact with machines and improve the efficiency and effectiveness of many industries. However, there are many challenges that must be addressed before it can be widely adopted, including improving the accuracy and coherence of the generated text and addressing ethical concerns.",
        "role": "assistant"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 517,
    "total_tokens": 536
  }
}

# 次回オリンピックは、いつ、どこで開催？

(llama) debian@dlp:~$

curl -s -XPOST -H 'Content-Type: application/json' localhost:8000/v1/chat/completions \
-d '{"messages": [{"role": "user", "content": "When and where will the next Olympics be held?"}]}' | jq | sed -e 's/\\n/\n/g'


llama_print_timings:        load time =     236.87 ms
llama_print_timings:      sample time =      94.83 ms /   178 runs   (    0.53 ms per token,  1877.06 tokens per second)
llama_print_timings: prompt eval time =     189.06 ms /    15 tokens (   12.60 ms per token,    79.34 tokens per second)
llama_print_timings:        eval time =    4974.61 ms /   177 runs   (   28.11 ms per token,    35.58 tokens per second)
llama_print_timings:       total time =    5681.12 ms /   192 tokens
INFO:     127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{
  "id": "chatcmpl-2834c4b7-f230-4468-8593-98418fd09a96",
  "object": "chat.completion",
  "created": 1708337296,
  "model": "./llama-2-13b-chat.Q4_K_M.gguf",
  "choices": [
    {
      "index": 0,
      "message": {
        "content": "  The Olympic Games are scheduled to take place in the following locations and dates:

2022 Winter Olympics: February 4-20, 2022, in Beijing, China

2022 Summer Olympics: July 21-August 6, 2022, in Tokyo, Japan

2024 Summer Olympics: July 26-August 11, 2024, in Paris, France

2026 Winter Olympics: February 6-22, 2026, in Milan, Italy

2028 Summer Olympics: July 21-August 6, 2028, in Los Angeles, California, USA

Please note that these dates are subject to change based on various factors and considerations.",
        "role": "assistant"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "completion_tokens": 177,
    "total_tokens": 198
  }
}