百度向量数据库¶

Baidu VectorDB 是由百度智能云精心打造并全托管的企业级分布式数据库服务，以其卓越的多维向量数据存储、检索与分析能力著称。该服务基于百度自研的"摩天轮"向量数据库内核，在提供高性能、高可用性和高安全性的同时，兼具出色的可扩展性和易用性。

该数据库服务支持多种索引类型和相似度计算方法，适用于各类应用场景。VectorDB 的突出特点在于能够管理高达百亿级别的超大规模向量数据，同时保持优异的查询性能——支持每秒百万级查询量(QPS)，且查询延迟控制在毫秒级别。

本笔记本展示了如何将 BaiduVectorDB 作为 LlamaIndex 中的向量存储库进行基础使用。

运行前需确保已创建数据库实例。

安装¶

如果您在 Colab 上打开此 Notebook，可能需要安装 LlamaIndex 🦙。

In [ ]:

%pip install llama-index-vector-stores-baiduvectordb

%pip install llama-index-vector-stores-baiduvectordb

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

!pip install pymochow

!pip install pymochow

In [ ]:

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
)
from llama_index.vector_stores.baiduvectordb import (
    BaiduVectorDB,
    TableParams,
    TableField,
)
import pymochow

from llama_index.core import ( VectorStoreIndex, SimpleDirectoryReader, StorageContext, ) from llama_index.vector_stores.baiduvectordb import ( BaiduVectorDB, TableParams, TableField, ) import pymochow

请提供 OpenAI 访问密钥¶

如需使用 OpenAI 的嵌入功能，您需要提供 OpenAI API 密钥：

In [ ]:

import openai

OPENAI_API_KEY = getpass.getpass("OpenAI API Key:")
openai.api_key = OPENAI_API_KEY

import openai OPENAI_API_KEY = getpass.getpass("OpenAI API Key:") openai.api_key = OPENAI_API_KEY

下载数据¶

In [ ]:

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

创建并填充向量数据库¶

现在您将从本地文件加载一些保罗·格雷厄姆（Paul Graham）的文章，并将它们存储到百度向量数据库（Baidu VectorDB）中。

In [ ]:

# load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print(f"Total documents: {len(documents)}")
print(f"First document, id: {documents[0].doc_id}")
print(f"First document, hash: {documents[0].hash}")
print(
    f"First document, text ({len(documents[0].text)} characters):\n{'='*20}\n{documents[0].text[:360]} ..."
)

# load documents documents = SimpleDirectoryReader("./data/paul_graham").load_data() print(f"Total documents: {len(documents)}") print(f"First document, id: {documents[0].doc_id}") print(f"First document, hash: {documents[0].hash}") print( f"First document, text ({len(documents[0].text)} characters):\n{'='*20}\n{documents[0].text[:360]} ..." )

初始化百度向量数据库¶

创建向量存储时，若底层数据库集合尚不存在，则需先创建该集合：

In [ ]:

vector_store = BaiduVectorDB(
    endpoint="http://192.168.X.X",
    api_key="*******",
    table_params=TableParams(dimension=1536, drop_exists=True),
)

vector_store = BaiduVectorDB( endpoint="http://192.168.X.X", api_key="*******", table_params=TableParams(dimension=1536, drop_exists=True), )

现在将这个存储封装到 index LlamaIndex 抽象中以供后续查询：

In [ ]:

storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context )

请注意，上述 from_documents 调用会同时执行多个操作：将输入文档分割为可管理大小的片段（"节点"），为每个节点计算嵌入向量，并将它们全部存储到百度向量数据库中。

查询存储¶

基础查询¶

In [ ]:

query_engine = index.as_query_engine()
response = query_engine.query("Why did the author choose to work on AI?")
print(response)

query_engine = index.as_query_engine() response = query_engine.query("Why did the author choose to work on AI?") print(response)

基于 MMR 的查询¶

MMR（最大边际相关性）方法旨在从存储中获取与查询同时相关但彼此差异尽可能大的文本片段，目的是为构建最终答案提供更广泛的上下文：

In [ ]:

query_engine = index.as_query_engine(vector_store_query_mode="mmr")
response = query_engine.query("Why did the author choose to work on AI?")
print(response)

query_engine = index.as_query_engine(vector_store_query_mode="mmr") response = query_engine.query("Why did the author choose to work on AI?") print(response)

连接到现有存储¶

由于该存储后端采用百度向量数据库（Baidu VectorDB），其本质具有持久化特性。因此，若需连接至先前已创建并填充数据的存储，可按以下步骤操作：

In [ ]:

vector_store = BaiduVectorDB(
    endpoint="http://192.168.X.X",
    api_key="*******",
    table_params=TableParams(dimension=1536, drop_exists=False),
)

# Create index (from preexisting stored vectors)
new_index_instance = VectorStoreIndex.from_vector_store(
    vector_store=new_vector_store
)

# now you can do querying, etc:
query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query(
    "What did the author study prior to working on AI?"
)
print(response)

vector_store = BaiduVectorDB( endpoint="http://192.168.X.X", api_key="*******", table_params=TableParams(dimension=1536, drop_exists=False), ) # Create index (from preexisting stored vectors) new_index_instance = VectorStoreIndex.from_vector_store( vector_store=new_vector_store ) # now you can do querying, etc: query_engine = index.as_query_engine(similarity_top_k=5) response = query_engine.query( "What did the author study prior to working on AI?" ) print(response)

元数据过滤¶

百度向量数据库（Baidu VectorDB）的向量存储支持在查询时通过精确匹配的 key=value 键值对形式进行元数据过滤。以下示例单元（基于全新创建的集合）展示了这一功能。

本演示中，为保持简洁，仅加载了单个源文档（即 ../data/paul_graham/paul_graham_essay.txt 文本文件）。不过我们会为该文档附加一些自定义元数据，以此说明如何通过文档所附元数据的条件限制来约束查询范围。

In [ ]:

filter_fields = [
    TableField(name="source_type"),
]

md_storage_context = StorageContext.from_defaults(
    vector_store=BaiduVectorDB(
        endpoint="http://192.168.X.X",
        api_key="="*******",",
        table_params=TableParams(
            dimension=1536, drop_exists=True, filter_fields=filter_fields
        ),
    )
)


def my_file_metadata(file_name: str):
    """Depending on the input file name, associate a different metadata."""
    if "essay" in file_name:
        source_type = "essay"
    elif "dinosaur" in file_name:
        # this (unfortunately) will not happen in this demo
        source_type = "dinos"
    else:
        source_type = "other"
    return {"source_type": source_type}


# Load documents and build index
md_documents = SimpleDirectoryReader(
    "../data/paul_graham", file_metadata=my_file_metadata
).load_data()
md_index = VectorStoreIndex.from_documents(
    md_documents, storage_context=md_storage_context
)

filter_fields = [ TableField(name="source_type"), ] md_storage_context = StorageContext.from_defaults( vector_store=BaiduVectorDB( endpoint="http://192.168.X.X", api_key="="*******",", table_params=TableParams( dimension=1536, drop_exists=True, filter_fields=filter_fields ), ) ) def my_file_metadata(file_name: str): """Depending on the input file name, associate a different metadata.""" if "essay" in file_name: source_type = "essay" elif "dinosaur" in file_name: # this (unfortunately) will not happen in this demo source_type = "dinos" else: source_type = "other" return {"source_type": source_type} # Load documents and build index md_documents = SimpleDirectoryReader( "../data/paul_graham", file_metadata=my_file_metadata ).load_data() md_index = VectorStoreIndex.from_documents( md_documents, storage_context=md_storage_context )

In [ ]:

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

In [ ]:

md_query_engine = md_index.as_query_engine(
    filters=MetadataFilters(
        filters=[MetadataFilter(key="source_type", value="essay")]
    )
)
md_response = md_query_engine.query(
    "How long it took the author to write his thesis?"
)
print(md_response.response)

md_query_engine = md_index.as_query_engine( filters=MetadataFilters( filters=[MetadataFilter(key="source_type", value="essay")] ) ) md_response = md_query_engine.query( "How long it took the author to write his thesis?" ) print(md_response.response)