HTTP API 参考

Text Generation Inference 自定义 API

查看 API 文档以获取有关如何与 Text Generation Inference API 交互的更多信息。

OpenAI Messages API

Text Generation Inference (TGI) 现在支持 Messages API，它与 OpenAI Chat Completion API 完全兼容。此功能从 1.4.0 版本开始可用。您可以使用 OpenAI 的客户端库或期望 OpenAI 模式的第三方库来与 TGI 的 Messages API 进行交互。以下是如何利用此兼容性的一些示例。

注意：Messages API 从 TGI 1.4.0 及更高版本开始支持。请确保您使用的是兼容版本以访问此功能。

发起请求

您可以使用 curl 向 TGI 的 Messages API 发起请求。这是一个示例

curl localhost:3000/v1/chat/completions \
    -X POST \
    -d '{
  "model": "tgi",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is deep learning?"
    }
  ],
  "stream": true,
  "max_tokens": 20
}' \
    -H 'Content-Type: application/json'

流式传输

您还可以使用 OpenAI 的 Python 客户端库来发起流式请求。方法如下

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    base_url="https://127.0.0.1:3000/v1",
    api_key="-"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=True
)

# iterate and print stream
for message in chat_completion:
    print(message)

同步

如果您希望发起同步请求，可以这样做

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    base_url="https://127.0.0.1:3000/v1",
    api_key="-"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=False
)

print(chat_completion)

Hugging Face 推理端点

Messages API 与 Inference Endpoints 集成。每个使用带有 LLM 的 “Text Generation Inference” 的端点，只要它具有聊天模板，现在都可以使用。以下是如何使用 OpenAI 的 Python 客户端库将 IE 与 TGI 一起使用的示例

注意：请务必将 base_url 替换为您的端点 URL，并在 URL 末尾包含 v1/。 api_key 应替换为您的 Hugging Face API 密钥。

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    # replace with your endpoint url, make sure to include "v1/" at the end
    base_url="https://vlzz10eq3fol3429.us-east-1.aws.endpoints.huggingface.cloud/v1/",
    # replace with your API key
    api_key="hf_XXX"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=True
)

# iterate and print stream
for message in chat_completion:
    print(message.choices[0].delta.content, end="")

云服务提供商

TGI 可以部署在各种云服务提供商上，以实现可扩展且强大的文本生成。其中一个提供商是 Amazon SageMaker，它最近增加了对 TGI 的支持。以下是如何在 Amazon SageMaker 上部署 TGI

Amazon SageMaker

Amazon Sagemaker 原生支持 message API

import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
 role = sagemaker.get_execution_role()
except ValueError:
 iam = boto3.client('iam')
 role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
 'HF_MODEL_ID':'HuggingFaceH4/zephyr-7b-beta',
 'SM_NUM_GPUS': json.dumps(1),
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
 image_uri=get_huggingface_llm_image_uri("huggingface",version="3.2.2"),
 env=hub,
 role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
 initial_instance_count=1,
 instance_type="ml.g5.2xlarge",
 container_startup_health_check_timeout=300,
  )

# send request
predictor.predict({
"messages": [
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ]
})

< > 在 GitHub 上更新

text-generation-inference