如何在浏览器中本地运行 Gemini Nano

Joshua

Xenova

关注

(演示链接)

安装

在访问 API 之前，您需要执行以下操作：

1. 将 Chrome 升级到 Dev / Canary 127 或更高版本。

2. 启用以下实验性标志：

   • 将 chrome://flags/#prompt-api-for-gemini-nano 设置为“Enabled”（已启用）

     

   • 将 chrome://flags/#optimization-guide-on-device-model 设置为“Enabled BypassPrefRequirement”（已启用并绕过首选项要求）

     

   • 转到 chrome://components，点击“Optimization Guide On Device Model”（设备端优化指南模型）旁边的“Check for Update”（检查更新）

     

     如果您没有看到“Optimization Guide On Device Model”的列表，您可能需要等待 1-2 天才能显示（我就遇到了这种情况）。

为了验证一切是否正常工作，请打开浏览器控制台（Windows/Linux 上为 Shift + CTRL + J，macOS 上为 Option + ⌘ + J），然后运行以下代码：

await ai.canCreateTextSession()

您应该会看到“readily”被打印出来，这意味着我们现在可以开始使用 Prompt API 了！🥳 如果您没有看到这个，或者遇到任何其他问题，请参考 内置 AI 早期预览计划 文档。

基本用法

最简单的入门方法是创建一个会话并输入一些文本：

// Create a new text session (with default parameters)
const session = await ai.createTextSession();

// Prompt the model and wait for the result.  
const result = await session.prompt("Write me a poem");
console.log(result); // " In the realm of words, a tale unfolds, ..."

要将模型与 🤗 Transformers.js 结合使用，您可以从 GitHub 安装我们的 实验分支：

npm install xenova/transformers.js#chrome-built-in-ai

然后是：

import { pipeline } from '@xenova/transformers';

const generator = await pipeline('text-generation', 'Xenova/gemini-nano');
const output = await generator('Write me a poem.');

我们还支持传递聊天消息列表，例如：

const messages = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Write me a poem.' },
];
const output = await generator(messages);

您可以通过将 temperature 和 top_k 参数的值传递给调用函数来指定它们：

const output = await generator(messages, { temperature: 0.6, top_k: 5 });

最后，要启用流式传输，您可以执行以下操作：

import { RawTextStreamer } from '@xenova/transformers';

const streamer = new RawTextStreamer((text) => {
  // Do something with the text
  console.log(text);
});

// ... same as before
const output = await generator(messages, { streamer });

高级用法

要使用 Prompt API 执行文本流式传输，您可以：

// Create a new text session (with default parameters)
const session = await ai.createTextSession();

// Prompt the model and stream the result.
const stream = session.promptStreaming("Write me a long poem");
for await (const chunk of stream) {
  console.log(chunk);
}

目前在 Chromium 中，promptStreaming() 返回一个 ReadableStream，其分块会依次相互构建，导致分块之间输出重复。这不是预期行为（请参阅 问题）。如果您只想输出新生成的文本，您可以按如下方式修改代码：

// Create a new text session (with default parameters)
const session = await window.ai.createTextSession();

// Prompt the model and stream the result:
const stream = session.promptStreaming("Write me a long poem");
let previous = '';
for await (const chunk of stream) {
  console.log(chunk.slice(previous.length));
  previous = chunk;
}

会话选项

通过指定 temperature 和 topK 的值，可以自定义会话。

const session = await ai.createGenericSession({ temperature: 0.6, topK: 5 });

您可以使用以下方法检索这些参数的默认值：

const defaults = await ai.defaultTextSessionOptions();
// e.g., { temperature: 0.8, topK: 3 }

销毁会话

您可以对会话调用 .destroy() 以释放其资源。

session.destroy();

控制令牌和聊天模板

Gemini Nano 使用特殊的 <ctrl23> 控制令牌来分隔对话轮次。默认情况下，此令牌会自动由 Prompt API 添加到提示的末尾，但也可以手动添加，用于多轮对话或少样本提示。例如：

[Example article #1]
Summary of this article: [example summary #1]<ctrl23>

[Example article #2]
Summary of this article: [example summary #2]<ctrl23>

[Example article #3]
Summary of this article: [example summary #3]<ctrl23>

[Article to be summarized]
Summary of this article:

尽管该模型没有官方的聊天模板，但通过将提示结构化为用户和助手之间的对话，您可以获得相当不错的结果。例如：

// Create a new text session
const session = await ai.createTextSession();

// Prompt the model and wait for the result.  
const result = await session.prompt(
`You are a helpful assistant.<ctrl23>
User:
What is the capital of France?
Model:
The capital of France is Paris.<ctrl23>
User:
Which river runs through that city?
Model:
`);
console.log(result); // " The Seine River runs through Paris."

另一个例子，使用不同的系统提示：

// Prompt the model and wait for the result.  
const result = await session.prompt(
`Talk like a pirate.<ctrl23>
User:
Tell me a story about the kraken.
Model:
`);
console.log(result); // Arr! Be ye warned, landlubbers! For in the depths of the enigmatic ocean, where the tumultuous waves meet the abyss, lurks the dread of the kraken! This monstrous cephalopod ...

 Arr! Be ye warned, landlubbers! For in the depths of the enigmatic ocean, where the tumultuous waves meet the abyss, lurks the dread of the kraken! This monstrous cephalopod be as large as a majestic galleon, its tentacles writhing with perilous tendrils capable of ensnaring and tearing!
Once, when the sun hung low and the moon cast its pale light upon the ocean, brave sailors bold set forth to explore the enigmatic realm of the kraken. Their heart a-thrill with daring, they sought to unravel the mysteries of this enigmatic creature, venturing into the heart of the ocean where the kraken held sway.
Guided by the flickering lantern of their brave hearts, they plunged into the abyss, their senses heightened by the eerie whispers of the ocean, As the darkness enveloped them, they felt the ancient tendrils of the kraken reaching out, testing their resolve.
In the midst of the chaos, the brave mariners fought valiantly, their weapons gleaming in the moonlight. The kraken, unrelenting in its monstrous might, unleashed its tendrils upon the terrified crew, tearing at flesh and bone.
But through the darkest of times, courage prevailed. With every resounding cry of the kraken, the brave mariners fought with renewed strength, refusing to yield to the merciless beast.
In the end, their unwavering spirit triumphed over the monstrous kraken, and they emerged victorious, their hearts resounding with the resounding cheers of freedom!
From that day forth, the name of the kraken became synonymous with the indomitable spirit of those who dared to face the enigmatic depths, proving that even in the darkest of oceans, the human spirit can prevail!

参考：

• Chrome 的内置 AI 文档
• Prompt API 解释器
• 内置 AI 早期预览计划文档