Llama 3.1模型推理实战
1、环境准备
首先，我们需要确保我们的服务器具备足够的硬件配置来支持Llama 3.1模型的运行。我们选择的是一台配备有4090型号GPU（24G显存）的服务器，基础镜像信息如下：ubuntu 22.04、python 3.12、cuda 12.1、pytorch 2.3.0。
2、安装依赖
首先 pip 换源加速下载并安装依赖包
# 升级pip
python -m pip install --upgrade pip
# 更换 pypi 源加速库的安装
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install fastapi==0.111.1
pip install uvicorn==0.30.3
pip install modelscope==1.16.1
pip install transformers==4.42.4
pip install accelerate==0.32.1安装完成如下：
3、模型下载
使用 modelscope 中的 snapshot_download 函数下载模型。第一个参数为模型名称，参数 cache_dir 用于指定模型的下载路径。
在 /root/autodl-tmp 路径下新建 d.py 文件，并在其中输入以下内容：
import torch
from modelscope import snapshot_download, AutoModel, AutoTokenizer
import os
model_dir = snapshot_download('LLM-Research/Meta-Llama-3.1-8B-Instruct', cache_dir='/root/autodl-tmp', revision='master')如下：
运行 python /root/autodl-tmp/d.py 执行下载。需注意，模型大小约为 15GB，下载模型大概需要 20 分钟，请耐心等待。
4、模型推理
1）推理测试
# 导入所需的库
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# 加载预训练的分词器和模型
model_name_or_path = '/root/autodl-tmp/LLM-Research/Meta-Llama-3___1-8B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", torch_dtype=torch.bfloat16)
# 定义对话消息列表，包含系统角色和用户角色的消息
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who are you?"}
]
# 使用分词器将对话消息转换为模型输入格式
input_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# 将输入转换为PyTorch张量并移动到GPU设备上
model_inputs = tokenizer([input_ids], return_tensors="pt").to('cuda')
# 使用模型生成回复
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512)
# 从生成的ID中提取回复部分
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
# 使用分词器将生成的ID解码为文本
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]执行成功如下：
查看响应结果
response结果如下：
"I'm an artificial intelligence model designed to assist and communicate with users in a helpful and informative way. I'm a type of chatbot, and my primary function is to provide information, answer questions, and engage in conversation to the best of my abilities.\n\nI don't have a personal identity or emotions, but I'm here to help you with any questions or topics you'd like to discuss. I can provide information on a wide range of subjects, from science and history to entertainment and culture. I can also help with tasks such as language translation, text summarization, and even creative writing.\n\nHow can I assist you today?"2）中文测试一
# 定义对话消息列表，包含系统角色和用户角色的消息
messages = [
{"role": "user", "content": "你会讲中文么?"}
]
# 使用分词器将对话消息转换为模型输入格式
input_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# 将输入转换为PyTorch张量并移动到GPU设备上
model_inputs = tokenizer([input_ids], return_tensors="pt").to('cuda')
# 使用模型生成回复
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512)
# 从生成的ID中提取回复部分
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
# 使用分词器将生成的ID解码为文本
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
response输出如下：
3）中文测试二
# 定义对话消息列表，包含系统角色和用户角色的消息
messages = [
{"role": "user", "content": "请以“夜晚”为题写一首诗"}
]
# 使用分词器将对话消息转换为模型输入格式
input_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# 将输入转换为PyTorch张量并移动到GPU设备上
model_inputs = tokenizer([input_ids], return_tensors="pt").to('cuda')
# 使用模型生成回复
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512)
# 从生成的ID中提取回复部分
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
# 使用分词器将生成的ID解码为文本
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
response输出如下：
注意：如果推理报错如下
File ~/miniconda3/lib/python3.10/site-packages/transformers/models/llama/configuration_llama.py:182, in LlamaConfig._rope_scaling_validation(self) 179 return 181 if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:--> 182 raise ValueError( 183 "`rope_scaling` must be a dictionary with two fields, `type` and `factor`, " f"got {self.rope_scaling}" 184 ) 185 rope_scaling_type = self.rope_scaling.get("type", None) 186 rope_scaling_factor = self.rope_scaling.get("factor", None)
ValueError: `rope_scaling` must be a dictionary with two fields, `type` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}则需要升级transformers：
pip install --upgrade transformers资源消耗如下：
文章最后
AI大模型作为人工智能领域的重要技术突破，正成为推动各行各业创新和转型的关键力量。抓住AI大模型的风口，掌握AI大模型的知识和技能将变得越来越重要。
学习AI大模型是一个系统的过程，需要从基础开始，逐步深入到更高级的技术。
