一、大模型推理

https://github.com/hiyouga/LLaMA-Factory/blob/main/README_zh.mdhttps://github.com/hiyouga/LLaMA-Factory/blob/main/examples/README_zh.md

• 安装 v7.1

https://github.com/hiyouga/LLaMA-Factory/releases/tag/v0.7.1

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[torch,metrics]

• 推理

• 以 Llama-3-8B-Instruct 为例，这里全部用 vllm 作为后端，不写就是 transformers 后端

• 使用命令行接口

• llama3_vllm.yaml

model_name_or_path: eta-Llama-3-8B-Instruct
template: llama3
infer_backend: vllm
vllm_enforce_eager: true

CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat examples/inference/llama3_vllm.yaml

• 使用浏览器界面

CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat examples/inference/llama3_vllm.yaml

• 启动 OpenAI 风格 API

CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/llama3_vllm.yaml

• 模拟发起请求

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Meta-Llama-3-8B-Instruct",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Who won the world series in 2020?"
      },
      {
        "role": "assistant",
        "content": "The Los Angeles Dodgers won the World Series in 2020."
      },
      {
        "role": "user",
        "content": "Where was it played?"
      }
    ]
  }'

---