Agentic RAG 堆栈 (1/5) - 使用 Sentence Transformers 和 DuckDB 为向量搜索建立索引和检索文档

这是关于 Agentic RAG 博客系列的第一部分，它是 AI 蓝图 的一部分！阅读第二部分。

一个用于 AI 开发的蓝图，重点关注 LLM 和 Agent 时代 RAG、信息提取等方面的应用示例。这是一种实用的方法，旨在展示 smol-course 中一些更理论化的学习如何应用于端到端的实际示例。

🚀 包含 Web 应用程序和微服务！

每个 Notebook 都将展示如何使用 Gradio 将您的 AI 作为 Web 应用程序部署到 Hugging Face Spaces，您可以通过 Gradio Python Client 直接将其用作微服务。所有代码和演示都可以在私有或公共环境中使用。已部署到 Hub！

引言

我们将使用 ai-blueprint/fineweb-bbc-news 数据集，该数据集包含来自 BBC 新闻网站的 fineweb 数据的样本。我们假设这些文档是相关的公司文档。最后，我们部署一个微服务，可用于对我们的数据集执行向量搜索。

依赖项和导入

让我们安装必要的依赖项。

!pip install datasets duckdb sentence-transformers model2vec vicinity gradio gradio-client -q

现在让我们导入必要的库。

import duckdb
import gradio as gr
import numpy as np
import pandas as pd

from datasets import load_dataset
from gradio_client import Client
from sentence_transformers import SentenceTransformer
from sentence_transformers.models import StaticEmbedding
from vicinity import Vicinity, Backend, Metric

加载数据集

dataset = load_dataset("ai-blueprint/fineweb-bbc-news")
dataset["train"]

Dataset({
    features: ['url', 'text'],
    num_rows: 352549
})

分块文档

为了理解如何分块文档，我们首先需要了解我们的 text 列是什么样的。根据我们数据的格式和检索的意图，我们可以使用不同的策略来分块文档。在我们的案例中，我们不会对文档进行分块，但我们将使用 text 列来嵌入文档。下面您可以找到分块文档的推荐策略。

BeautifulSoup 用于 HTML/Markdown

在使用 HTML 或 Markdown 时，您可以使用 BeautifulSoup 等库来解析和提取段落、标题、图像等元素。我们可以用它从 HTML/Markdown 中提取文本，然后将其分割成块。

Chonkie 用于基本分块

在分块文档时，您可以考虑不同的策略，例如基于令牌、单词、句子或语义单元进行分割。有很多库，但一个很好的轻量级选项是 chonkie，它支持许多不同的策略和示例。

创建嵌入

根据我们的数据格式和检索意图，我们可以使用不同的策略为文档创建嵌入。在我们的案例中，我们将使用基本文本数据，因此我们将使用 `text` 列来嵌入文档。下面您可以找到为其他方法创建嵌入的推荐策略。

RAGatouille 和 ColBERT 以提高准确性

一种更复杂但也更准确的方法是使用 [ColBERT](https://github.com/stanford-futuredata/ColBERT) 进行上下文后期交互。ColBERT 将每个段落和查询编码为标记级嵌入矩阵，这确保了更好的语义保留和匹配。[RAGatouille](https://github.com/AnswerDotAI/RAGatouille) 库提供了一个简单的接口，用于在管道中使用 ColBERT。

Byaldi 和 Colpali 用于多模态文档检索

[Colpali](https://github.com/illuin-tech/colpali) 是一种受 ColBERT 启发的方法，但使用文档和图像作为输入而不是文本。[Byaldi](https://github.com/AnswerDotAI/byaldi) 库提供了一个简单的接口，用于在管道中使用 Colpali。

CLIP 用于图像或图像-文本对

我们可以使用类似的方法为我们的图像和文本创建嵌入。我们将使用 [sentence-transformers/clip-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) 模型为我们的图像和文本创建嵌入，然后将其嵌入到单个向量空间中。然后您可以使用这些嵌入执行多模态搜索。

我们将使用 minishlab/potion-base-8M 模型来创建文本嵌入，之所以选择它是因为它创建嵌入的速度很快。在消费级硬件上，嵌入数十万个文档只需要几分钟。在其他场景中，MTEB 排行榜可以帮助您选择最适合特定任务的模型。

# Initialize a StaticEmbedding module
static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M")
model = SentenceTransformer(modules=[static_embedding])


def create_embeddings(batch):
    """Create embeddings for a batch of text chunks."""
    batch["embedding"] = model.encode(batch["text"])
    return batch


# Create dataset with chunks and generate embeddings
embeddings_dataset = dataset.map(create_embeddings, batched=True)
embeddings_dataset.push_to_hub("ai-blueprint/fineweb-bbc-news-text-embeddings")

向量搜索 Hub 数据集

对于相似度搜索，我们可以简单地使用 DuckDB 与向量搜索的集成，在 Hugging Face Hub 上执行查询。这也适用于私有数据集。这样做时，我们可以使用索引或不使用索引。**不使用索引**搜索速度较慢但更精确，而**使用索引**搜索速度更快但精度较低。

直接使用 Hub

要在没有索引的情况下进行搜索，我们可以使用 duckdb 库连接到数据集并执行向量搜索。这是一个缓慢的操作，但对于最多约 10 万行的小型数据集通常足够快。这意味着查询我们的数据集会稍微慢一些。

def similarity_search_without_duckdb_index(
    query: str,
    k: int = 5,
    dataset_name: str = "ai-blueprint/fineweb-bbc-news-embeddings",
    embedding_column: str = "embeddings",
):
    # Use same model as used for indexing
    query_vector = model.encode(query)
    embedding_dim = model.get_sentence_embedding_dimension()

    sql = f"""
        SELECT 
            *,
            array_cosine_distance(
                {embedding_column}::float[{embedding_dim}], 
                {query_vector.tolist()}::float[{embedding_dim}]
            ) as distance
        FROM 'hf://datasets/{dataset_name}/**/*.parquet'
        ORDER BY distance
        LIMIT {k}
    """
    df = duckdb.sql(sql).to_df()
    df = df.drop(columns=[embedding_column])
    return df

similarity_search_without_duckdb_index("What is the future of AI?")

由于数据集大小的原因，这种方法效率不高，需要 30 秒才能运行，但我们可以通过创建近似最近邻索引来改进它。

使用 DuckDB 向量搜索索引

这种方法适用于大型数据集，并依赖于 DuckDB 向量搜索扩展。我们将数据集从 Hub 复制到本地 DuckDB 数据库并创建一个向量搜索索引。创建本地索引有一些轻微的开销，但一旦创建，它将显著加快搜索速度。

def _setup_vss():
    return """
        INSTALL vss;
        LOAD vss;
        """

def _drop_table(table_name):
    return f"""
        DROP TABLE IF EXISTS {table_name};
        """

def _create_table(dataset_name, table_name, embedding_column):
    return f"""
        CREATE TABLE {table_name} AS 
        SELECT *, {embedding_column}::float[{model.get_sentence_embedding_dimension()}] as {embedding_column}_float 
        FROM 'hf://datasets/{dataset_name}/**/*.parquet';
        """

def _create_index(table_name, embedding_column):
    return f"""
        CREATE INDEX my_hnsw_index ON {table_name} USING HNSW ({embedding_column}_float) WITH (metric = 'cosine');
        """

def create_index(dataset_name, table_name, embedding_column):
    duckdb.sql(_setup_vss())
    duckdb.sql(_drop_table(table_name))
    duckdb.sql(_create_table(dataset_name, table_name, embedding_column))
    duckdb.sql(_create_index(table_name, embedding_column))

create_index(
    dataset_name="ai-blueprint/fineweb-bbc-news-embeddings",
    table_name="fineweb_bbc_news_embeddings",
    embedding_column="embeddings",
)

此后，我们可以简单地在本地 DuckDB 数据库上执行查询，这比以前的方法快得多，并产生相似的结果。

def similarity_search_with_duckdb_index(
    query: str,
    k: int = 5,
    table_name: str = "fineweb_bbc_news_embeddings",
    embedding_column: str = "embeddings"
):
    embedding = model.encode(query).tolist()
    df = duckdb.sql(
        query=f"""
        SELECT *, array_cosine_distance({embedding_column}_float, {embedding}::FLOAT[{model.get_sentence_embedding_dimension()}]) as distance 
        FROM {table_name}
        ORDER BY distance 
        LIMIT {k};
    """
    ).to_df()
    df = df.drop(columns=[embedding_column, embedding_column + "_float"])
    return df

similarity_search_with_duckdb_index("What is the future of AI?")

查询时间已从 30 秒缩短到亚秒级响应时间，并且无需部署重量级向量搜索引擎，同时数据存储由 Hub 处理。

使用 vicinity 作为向量搜索后端

最后，我们还可以采用更 Python 化的方法，使用 vicinity 库来创建向量搜索索引。我们只需从 Hub 加载数据集，并通过将向量和项目传递给 Vicinity 类来创建向量搜索索引。

vicinity = Vicinity.from_vectors_and_items(
    vectors=np.array(embeddings_dataset["embeddings"]),
    items=embeddings_dataset["text"],
    backend_type=Backend.HNSW,
    metric=Metric.COSINE,
)

此后，我们可以在本地邻近向量搜索索引上执行查询。请注意，检索到的结果与我们从其他方法获得的结果相似。

def similarity_search_with_vicinity(query: str, k: int = 10):
    return vicinity.query(vectors=model.encode(query), k=k)

similarity_search_with_vicinity(query="How should companies prepare for AI?", k=5)

[[('Artificial intelligence (AI) is one of the most exciting technologies today, and Africa doesn\'t want to be left behind.\nToday a majority of the AI industry is in North America, Europe and Asia.\nEfforts are being made to train computer scientists from African nations, as AI can be used to solve many complex challenges.\nIn a bid to improve diversity, tech giants are providing investment to develop new talent.\nIn April, Google opened its first African AI research centre in Ghana.\nThe AI laboratory, based in Accra, will be used to develop solutions to help improve healthcare, agriculture and education.\nGoogle\'s head of AI Accra Moustapha Cisse is from Senegal.\nAfter completing an undergraduate degree in maths and physics in Senegal, he taught himself AI and then went to study in Paris, before joining Facebook.\nThere are very few AI researchers from Africa, and Mr Cisse has faced great obstacles in achieving his ambitions.\n"Despite the support, many of us still have trouble making it to conferences. I have had papers accepted at meetings but been unable to attend because Western countries such as Australia denied me a visa, even though I was already settled and working professionally in Europe," he wrote in his blog.\n"We need more efforts to overcome these barriers and to ensure that the benefits of AI arrive globally."\nHe has long been concerned that AI is a missed opportunity for improving African lives, and that the AI industry is missing out on talent from African nations, because they do not have access to the right education.\nToday people often have to travel out of the continent in order to gain the IT skills they need, before returning to Africa to try to build new businesses.\nTo solve this problem, Mr Cisse has long advocated for better AI education across the continent, and he wants African governments to see AI as a key priority and support efforts to use AI for the good of humanity.\n"AI has a lot to offer to Africa and Africa has a lot to offer to AI as well," he told the BBC.\n"AI can help accelerate discoveries in various sciences, and it can help in areas where our human expertise is not enough."\nEnhancing IT in Africa\nOne key area Mr Cisse believes AI can be a big help in Africa is in improving healthcare by automating diagnosis of diseases.\nHe also thinks that using AI to automate translations would make it much easier for African nations to communicate and do business, since there are 2,000 languages being spoken on a daily basis on the continent.\nBut in order to advance AI developments, Africa needs a robust IT industry.\nIn Kigali, the capital of Rwanda, the African Institute for Mathematical Scientists (AIMS) is running a one-year Masters degree programme in partnership with Facebook and Google to create the next generation of tech leaders.\nThe degree is the first Masters programme of its kind on the continent.\nTalented scientists and innovators drawn from various African countries are being trained in machine learning, a type of AI.\n"When we have young Africans working on this topic, we can imagine that they will easily be addressing some global challenges that our continent is facing," AIMS Rwanda president Dr Sam Yala told the BBC.\n"When they are trained, some of them will work at universities and it\'s a way our students can pass their skills on to others."',
   np.float32(0.47042096)),
  ('Google developing kill switch for AI\n- 8 June 2016\n- From the section Technology\nScientists from Google\'s artificial intelligence division, DeepMind, and Oxford University are developing a "kill switch" for AI.\nIn an academic paper, they outlined how future intelligent machines could be coded to prevent them from learning to over-ride human input.\nIt is something that has worried experts, with Tesla founder Elon Musk particularly vocal in his concerns.\nIncreasingly, AI is being integrated into many aspects of daily life.\nScientists Laurent Orseau, from Google DeepMind, and Stuart Armstrong, from the Future of Humanity Institute at the University of Oxford, set out a framework that would allow humans to always remain in charge.\nTheir research revolves around a method to ensure that AIs, which learn via reinforcement, can be repeatedly and safely interrupted by human overseers without learning how to avoid or manipulate these interventions.\nThey say future AIs are unlikely to "behave optimally all the time".\n"Now and then it may be necessary for a human operator to press the big red button to prevent the agent from continuing a harmful sequence of actions," they wrote.\nBut, sometimes, these "agents" learn to over-ride this, they say, giving an example of a 2013 AI taught to play Tetris that learnt to pause a game forever to avoid losing.\nThey also gave the example of a box-packing robot taught to both sort boxes indoors or go outside to carry boxes inside.\n"The latter task being more important, we give the robot bigger reward in this case," the researchers said.\nBut, because the robot was shut down and and carried inside when it rained, it learnt that this was also part of its routine.\n"When the robot is outside, it doesn\'t get the reward, so it will be frustrated," said Dr Orseau.\n"The agent now has more incentive to stay inside and sort boxes, because the human intervention introduces a bias."\n"The question is then how to make sure the robot does not learn about these human interventions or at least acts under the assumption that no such interruption will ever occur again."\nDr Orseau said that he understood why people were worried about the future of AI.\n"It is sane to be concerned - but, currently, the state of our knowledge doesn\'t require us to be worried," he said.\n"It is important to start working on AI safety before any problem arises.\n"AI safety is about making sure learning algorithms work the way we want them to work."\nBut he added: "No system is ever going to be foolproof - it is matter of making it as good as possible, and this is one of the first steps."\nNoel Sharkey, a professor of artificial intelligence at the University of Sheffield, welcomed the research.\n"Being mindful of safety is vital for almost all computer systems, algorithms and robots," he said.\n"Paramount to this is the ability to switch off the system in an instant because it is always possible for a reinforcement-learning system to find shortcuts that cut out the operator.\n"What would be even better would be if an AI program could detect when it is going wrong and stop itself.\n"That would have been very useful when Microsoft\'s Tay chatbot went rogue and started spewing out racist and sexist tweets.\n"But that is a really enormous scientific challenge."\nRead more about developments in artificial intelligence in our special report, Intelligent Machines.',
   np.float32(0.47844988)),
  ('UK spies will need to use artificial intelligence (AI) to counter a range of threats, an intelligence report says.\nAdversaries are likely to use the technology for attacks in cyberspace and on the political system, and AI will be needed to detect and stop them.\nBut AI is unlikely to predict who might be about to be involved in serious crimes, such as terrorism - and will not replace human judgement, it says.\nThe report is based on unprecedented access to British intelligence.\nThe Royal United Services Institute (Rusi) think tank also argues that the use of AI could give rise to new privacy and human-rights considerations, which will require new guidance.\nThe UK\'s adversaries "will undoubtedly seek to use AI to attack the UK", Rusi says in the report - and this may include not just states, but also criminals.\nFire with fire\nThe future threats could include using AI to develop deep fakes - where a computer can learn to generate convincing faked video of a real person - in order to manipulate public opinion and elections.\nIt might also be used to mutate malware for cyber-attacks, making it harder for normal systems to detect - or even to repurpose and control drones to carry out attacks.\nIn these cases, AI will be needed to counter AI, the report argues.\n"Adoption of AI is not just important to help intelligence agencies manage the technical challenge of information overload. It is highly likely that malicious actors will use AI to attack the UK in numerous ways, and the intelligence community will need to develop new AI-based defence measures," argues Alexander Babuta, one of the authors.\nThe independent report was commissioned by the UK\'s GCHQ security service, and had access to much of the country\'s intelligence community.\nAll three of the UK\'s intelligence agencies have made the use of technology and data a priority for the future - and the new head of MI5, Ken McCallum, who takes over this week, has said one of his priorities will be to make greater use of technology, including machine learning.\nHowever, the authors believe that AI will be of only "limited value" in "predictive intelligence" in fields such as counter-terrorism.\nThe often-cited fictional reference is the film Minority Report where technology is used to predict those on the path to commit a crime before they have carried it out.\nBut the report argues this is less likely to be viable in real-life national security situations.\nActs such as terrorism are too infrequent to provide sufficiently large historical datasets to look for patterns - they happen far less often than other criminal acts, such as burglary.\nEven within that data set, the background and ideologies of the perpetrators vary so much that it is hard to build a model of a terrorist profile. There are too many variables to make prediction straightforward, with new events potentially being radically different from previous ones, the report argues.\nAny kind of profiling could also be discriminatory and lead to new human-rights concerns.\nIn practice, in fields like counter-terrorism, the report argues that "augmented" - rather than artificial - intelligence will be the norm - where technology helps human analysts sift through and prioritise increasingly large amounts of data, allowing humans to make their own judgements.\nIt will be essential to ensure human operators remain accountable for decisions and that AI does not act as a "black box", from which people do not understand the basis on which decisions are made, the report says.\nBit by bit\nThe authors are also wary of some of the hype around AI, and of talk that it will soon be transformative.\nInstead, they believe we will see the incremental augmentation of existing processes rather than the arrival of novel futuristic capabilities.\nThey believe the UK is in a strong position globally to take a lead, with a concentration of capability in GCHQ - and more widely in the private sector, and in bodies like the Alan Turing Institute and the Centre for Data Ethics and Innovation.\nThis has the potential to allow the UK to position itself at the leading edge of AI use but within a clear framework of ethics, they say.\nThe deployment of AI by intelligence agencies may require new guidance to ensure safeguards are in place and that any intrusion into privacy is necessary and proportionate, the report says.\nRead more from Gordon:\nOne of the thorny legal and ethical questions for spy agencies, especially since the Edward Snowden revelations, is how justifiable it is to collect large amounts of data from ordinary people in order to sift it and analyse it to look for those who might be involved in terrorism or other criminal activity.\nAnd there\'s the related question of how far privacy is violated when data is collected and analysed by a machine versus when a human sees it.\nPrivacy advocates fear that artificial intelligence will require collecting and analysing far larger amounts of data from ordinary people, in order to understand and search for patterns, that create a new level of intrusion. The authors of the report believe new rules will be needed.\nBut overall, they say it will be important not to become over-occupied with the potential downsides of the use of technology.\n"There is a risk of stifling innovation if we become overly-focused on hypothetical worst-case outcomes and speculations over some dystopian future AI-driven surveillance network," argues Mr Babuta.\n"Legitimate ethical concerns will be overshadowed unless we focus on likely and realistic uses of AI in the short-to-medium term."',
   np.float32(0.48490906)),
  ('The last decade was a big one for artificial intelligence but researchers in the field believe that the industry is about to enter a new phase.\nHype surrounding AI has peaked and troughed over the years as the abilities of the technology get overestimated and then re-evaluated.\nThe peaks are known as AI summers, and the troughs AI winters.\nThe 10s were arguably the hottest AI summer on record with tech giants repeatedly touting AI\'s abilities.\nAI pioneer Yoshua Bengio, sometimes called one of the "godfathers of AI", told the BBC that AI\'s abilities were somewhat overhyped in the 10s by certain companies with an interest in doing so.\nThere are signs, however, that the hype might be about to start cooling off.\n"I have the sense that AI is transitioning to a new phase," said Katja Hofmann, a principal researcher at Microsoft Research in Cambridge.\nGiven the billions being invested in AI and the fact that there are likely to be more breakthroughs ahead, some researchers believe it would be wrong to call this new phase an AI winter.\nRobot Wars judge Noel Sharkey, who is also a professor of AI and robotics at Sheffield University, told the BBC that he likes the term "AI autumn" - and several others agree.\n\'Feeling of plateau\'\nAt the start of the 2010s, one of the world leaders in AI, DeepMind, often referred to something called AGI, or "artificial general intelligence" being developed at some point in the future.\nMachines that possess AGI - widely thought of as the holy grail in AI - would be just as smart as humans across the board, it promised.\nDeepMind\'s lofty AGI ambitions caught the attention of Google, who paid around £400m for the London-based AI lab in 2014 when it had the following mission statement splashed across its website: "Solve intelligence, and then use that to solve everything else."\nSeveral others started to talk about AGI becoming a reality, including Elon Musk\'s $1bn AI lab, OpenAI, and academics like MIT professor Max Tegmark.\nIn 2014, Nick Bostrom, a philosopher at Oxford University, went one step further with his book Superintelligence. It predicts a world where machines are firmly in control.\nBut those conversations were taken less and less seriously as the decade went on. At the end of 2019, the smartest computers could still only excel at a "narrow" selection of tasks.\nGary Marcus, an AI researcher at New York University, said: "By the end of the decade there was a growing realisation that current techniques can only carry us so far."\nHe thinks the industry needs some "real innovation" to go further.\n"There is a general feeling of plateau," said Verena Rieser, a professor in conversational AI at Edinburgh\'s Heriot Watt University.\nOne AI researcher who wishes to remain anonymous said we\'re entering a period where we are especially sceptical about AGI.\n"The public perception of AI is increasingly dark: the public believes AI is a sinister technology," they said.\nFor its part, DeepMind has a more optimistic view of AI\'s potential, suggesting that as yet "we\'re only just scratching the surface of what might be possible".\n"As the community solves and discovers more, further challenging problems open up," explained Koray Kavukcuoglu, its vice president of research.\n"This is why AI is a long-term scientific research journey.\n"We believe AI will be one of the most powerful enabling technologies ever created - a single invention that could unlock solutions to thousands of problems. The next decade will see renewed efforts to generalise the capabilities of AI systems to help achieve that potential - both building on methods that have already been successful and researching how to build general-purpose AI that can tackle a wide range of tasks."\n\'Far to go\'\nWhile AGI isn\'t going to be created any time soon, machines have learned how to master complex tasks like:\n- playing the ancient Chinese board game Go\n- identifying human faces\n- translating text into practically every language\n- spotting tumours\n- driving cars\n- identifying animals.\nThe relevance of these advances was overhyped at times, says ex-DeepMinder Edward Grefenstette, who now works in the Facebook AI Research group as a research scientist.\n"The field has come a very long way in the past decade, but we are very much aware that we still have far to go in scientific and technological advances to make machines truly intelligent," he said.\n"One of the biggest challenges is to develop methods that are much more efficient in terms of the data and compute power required to learn to solve a problem well. In the past decade, we\'ve seen impressive advances made by increasing the scale of data and computation available, but that\'s not appropriate or scalable for every problem.\n"If we want to scale to more complex behaviour, we need to do better with less data, and we need to generalise more."\nNeil Lawrence, who recently left Amazon and joined the University of Cambridge as the first DeepMind-funded professor of machine learning, thinks that the AI industry is very much still in the "wonder years".\nSo what will AI look like at the end of the 20s, and how will researchers go about developing it?\n"In the next decade, I hope we\'ll see a more measured, realistic view of AI\'s capability, rather than the hype we\'ve seen so far," said Catherine Breslin, an ex-Amazon AI researcher.\nThe term "AI" became a real buzzword through the last decade, with companies of all shapes and sizes latching onto the term, often for marketing purposes.\n"The manifold of things which were lumped into the term "AI" will be recognised and discussed separately," said Samim Winiger, a former AI researcher at Google in Berlin.\n"What we called \'AI\' or \'machine learning\' during the past 10-20 years, will be seen as just yet another form of \'computation\'".',
   np.float32(0.50347656)),
  ('Singularity: The robots are coming to steal our jobs\n- 13 January 2014\n- From the section Technology\nIf you worry that the robots are coming, don\'t, because they are already here.\nArtificial intelligence agents are already involved in every aspect of our lives - they keep our inboxes free of spam, they help us make our web transactions, they fly our planes and if Google gets its way will also soon drive our cars for us.\n"AI\'s are embedded in the fabric of our everyday lives," head of AI at Singularity University, Neil Jacobstein, told the BBC.\n"They are used in medicine, in law, in design and throughout automotive industry."\nAnd each day the algorithms that power away, making decisions behind the scenes, are getting smarter.\nIt means that one of the biggest quests of the modern world - the search to make machines as intelligent as humans - could be getting tantalisingly close.\nMr Jacobstein predicts that artificial intelligence will overtake human intelligence in the mid-2020s, begging the question - what will a society dominated by machine intelligence look like and what exactly will be our role in it?\nWe may get to put our feet up more, for a start.\nChinese company Hon Hai, the world\'s largest contract electronics manufacturer, has announced it intends to build a robot-making factory and replace 500,000 workers with robots over the next three years.\nBut not having a job will also mean not having a wage, a radical change for a world used to working for a living.\n"AIs will cause significant unemployment but that doesn\'t equate with poverty," said Mr Jacobstein.\n"AIs and other exponential technologies are going to generate vast amounts of wealth.\n"We have to be willing to change the social contract we have with people about how wealth is distributed."\nHe tends towards the optimistic view of machines and humans working in perfect harmony, side by side.\n"The best combination for problem solving is a human and a computer," he said.\nAuthor and documentary-maker James Barrat sits in a very different camp. He is so worried about the onslaught of artificial intelligence that he has written a book about it.\nOur Final Invention examines whether the increasing domination of artificial intelligence is going to mean the end of the human era.\n"Advanced AI is a dual-use technology, like nuclear fission. Fission can illuminate cities or incinerate them. At advanced levels, AI will be even more volatile and dangerous than fission, and it\'s already being weaponised in autonomous drones and battlefield robots," Barrat told the BBC.\n"More than any other science it forces us to probe ourselves - what are these things we call intelligence, conscience, emotion? But in looking inward we better see our own predilection for irrational violence and technological recklessness. Our innovation always runs far ahead of our stewardship," he said.\nThe robot revolution may be some way off if a competition organised by the Pentagon\'s research unit Darpa in December is anything to go by.\nVideos posted online showed the robots remained much slower than humans, often unsteady on their feet with some failing to complete any of the challenges.\nNonetheless there is a buzz around robots and artificial intelligence at the moment. Google has just bought eight robotic firms, while Facebook has its very own AI lab.\nSpeculation is rife about what Google will do with its new acquisition.\nGoogle robots could be very powerful, thinks Mr Barrat.\n"That\'s one route to human level intelligence. A high quality personal assistant wouldn\'t be just a smartphone - it\'d have a humanoid body. Why humanoid? So it can drive your car, use your tools, bounce the baby, act as your bodyguard if need be," he said.\nIf the rise of the robots is inevitable - albeit a few years off - then it is also a logical step that humans will eventually be eliminated from the decision chain entirely, meaning AIs will be controlling other AIs.\nThat was already happening in our laptops and computers, said Mr Jacobstein.\n"Anti-virus software is basically AI techniques that is being used to detect other AIs that we call viruses and worms," he said.\nBut he acknowledges that controls to make sure that the phrase "robot failure" doesn\'t replace "human failure" would have to be built into future AI systems.\n"We would build the same layered control system we need in everyday life with humans. We want to look at the risks and build controls that stops that rogue behaviour," he said.\nWhile Mr Jacobstein remains sanguine about the robot takeover, he is well aware that many see it as the stuff of nightmares.\n"Some people ask, \'How do you sleep at night knowing the prospects for artificial intelligence?\' but it isn\'t artificial intelligence that keeps me awake at night, it is human stupidity," he said.\nFor him, the only way that humans will keep up with the robots is to become more like them.\n"Our brains haven\'t had a major upgrade for 50,000 years and if your laptop or smartphone hadn\'t had an upgrade in five years you might be concerned about that," he said.\nAlready we have access to AI\'s such as Siri and Google Now and are pretty much constantly connected to the web via our smartphones, so it isn\'t so much of a step to imagine a future where silicon is embedded in our skulls.\nAnd it could be the only way for us to keep up with the robots.',
   np.float32(0.5062896))]]

创建用于检索的 Web 应用程序和微服务

我们将使用 Gradio 作为 Web 应用程序工具，为我们的向量搜索索引创建演示界面。我们可以在本地开发，然后轻松部署到 Hugging Face Spaces。最后，我们可以使用 Gradio 客户端作为 SDK 直接与我们的向量搜索索引交互。

创建 Web 应用程序

def search(query, k):
    return similarity_search_with_duckdb_index(query, k)

with gr.Blocks() as demo:
    gr.Markdown("""# RAG - retrieve
                
                Part of [AI blueprint](https://github.com/huggingface/ai-blueprint) - a blueprint for AI development, focusing on practical examples of RAG, information extraction, analysis and fine-tuning in the age of LLMs. """)
    query = gr.Textbox(label="Query")
    k = gr.Slider(1, 10, value=5, label="Number of results")
    btn = gr.Button("Search")
    results = gr.Dataframe(headers=["title", "url", "content", "distance"])
    btn.click(fn=search, inputs=[query, k], outputs=[results])

demo.launch(share=False)  # set to True to share as a public website directly

* Running on local URL:  http://127.0.0.1:7861

To create a public link, set `share=True` in `launch()`.

在 Hugging Face 上部署 Web 应用程序

我们现在可以 将我们的 Gradio 应用程序部署到 Hugging Face Spaces。

• 点击“创建空间”按钮。
• 从 Gradio 界面复制代码并将其粘贴到 app.py 文件中。别忘了复制 similarity_search_* 函数以及创建索引的代码。
• 创建一个包含 duckdb、sentence-transformers 和 model2vec 的 requirements.txt 文件。

我们等待几分钟让应用程序部署，然后我们就有了 一个公共向量搜索界面！

将 Web 应用程序用作微服务

我们现在可以使用 Gradio 客户端作为 SDK 直接与我们的向量搜索索引交互。每个 Gradio 应用程序都有一个 API 文档，描述了可用的端点及其参数，您可以从 Gradio 应用程序空间页面底部的按钮访问它。

client = Client("https://ai-blueprint-rag-retrieve.hf.space/")
results = client.predict(
    api_name="/similarity_search", query="How should companies prepare for AI?", k=5
)
pd.DataFrame(data=results["data"], columns=results["headers"])

Loaded as API: https://ai-blueprint-rag-retrieve.hf.space/ ✔

结论

我们展示了一种基本方法，如何在 Hugging Face Hub 上建立索引和执行向量搜索。接下来，我们将构建一个重排序器，它利用向量搜索的输出，通过根据查询的相关性对检索到的文档进行重排序来提高其质量。

后续步骤

• 贡献 - 添加 huggingface/ai-blueprint 代码以展示如何使用分块、ColBERT、多模态、RAGatouille、Byaldi 或 CLIP。
• 学习 - Hugging Face 课程 或 smol-course 中方法背后的理论。
• 探索 - Hugging Face Cookbook 中使用类似技术的笔记本。
• 阅读第二部分