litellm 集成instructor 简单说明

原创

rongfengliang 2024-08-12 11:19:16 ©著作权

文章标签 参考资料 github python 文章分类 Html/CSS 前端开发

©著作权归作者所有：来自51CTO博客作者rongfengliang的原创作品，请联系作者获取转载授权，否则将追究法律责任

instructor 是一个很不错的llm 结构化输出工具litellm 与instructor 的集成模式实际也包含了两种，一种是基于patch 模式，一种是标准openai 模式，以下简单说明下

patch 模式

import litellm
from litellm import Router
import instructor
from pydantic import BaseModel
 
litellm.set_verbose = True
 
client = instructor.patch(
    Router(
        model_list=[
            {
                "model_name": "myqwen2",
                "litellm_params": {  # params for litellm completion/embedding call - e.g.: https://github.com/BerriAI/litellm/blob/62a591f90c99120e1a51a8445f5c3752586868ea/litellm/router.py#L111
                    "model": "ollama/qwen2:1.5b",
                    "api_key": "demo",
                    "api_base": "http://localhost:11434",
                },
            }
        ]
    )
)
 
 
class UserDetail(BaseModel):
    name: str
    age: int
 
 
user = client.chat.completions.create(
    model="myqwen2",
    response_model=UserDetail,
    messages=[
        {"role": "user", "content": "Extract Jason is 25 years old"},
    ],
)
 
print(user)

from litellm import Router

标准openai api 模式

核心是基于proxy 进行配置，之后就是使用key 进行访问
参考代码

import litellm
import instructor
from pydantic import BaseModel
from openai import OpenAI
litellm.set_verbose = True
# api_key  需要通过 /key/generate 或者ui 获取
openai = OpenAI(
    api_key="sk-26xW7syxk4GIsr1qcXf1bw",
    base_url="http://localhost:4000"
)
 
client = instructor.from_openai(openai)
 
class UserDetail(BaseModel):
    name: str
    age: int
 
 
user = client.chat.completions.create(
    model="myqwen2",
    response_model=UserDetail,
    messages=[
        {"role": "user", "content": "Extract Jason is 25 years old"},
    ],
)
print(user)