Oracle AI 向量搜索：文档处理¶

Oracle AI 向量搜索专为人工智能（AI）工作负载设计，允许您基于语义而非关键词来查询数据。 Oracle AI 向量搜索的最大优势之一在于，非结构化数据的语义搜索可以与业务数据的关系型搜索在单一系统中结合使用。这不仅功能强大，而且效率显著提升，因为您无需额外部署专门的向量数据库，从而消除了多系统间数据碎片化的痛点。

此外，您的向量数据还能充分利用 Oracle 数据库的所有强大功能，例如：

本指南演示如何利用 Oracle AI 向量搜索中的文档处理能力，分别使用 OracleDocLoader 和 OracleTextSplitter 来加载和分块文档。

如果你是 Oracle 数据库的初学者，建议先尝试免费的 Oracle 23 AI，该资源能帮助你快速掌握数据库环境搭建。操作数据库时，通常应避免默认使用系统用户，而是创建专属用户以提升安全性和可定制性。关于用户创建的具体步骤，可参考我们的端到端指南，其中也演示了如何在 Oracle 中设置用户。此外，理解用户权限对于有效管理数据库安全至关重要，可通过官方Oracle 指南深入学习用户账户与安全管理。

前提条件¶

如需将 llama_index 与 Oracle AI 向量搜索功能结合使用，请先安装 Oracle Python 客户端驱动。

In [ ]:

Copied!

%pip install llama-index-readers-oracleai
%pip install llama-index-readers-oracleai

连接 Oracle 数据库¶

以下示例代码将展示如何连接 Oracle 数据库。默认情况下，python-oracledb 运行在 'Thin' 模式下，可直接连接至 Oracle 数据库。此模式无需 Oracle 客户端库，但使用这些库时可获得某些额外功能。当使用 Oracle 客户端库时，python-oracledb 被称为处于 'Thick' 模式。两种模式均全面支持 Python 数据库 API v2.0 规范。参阅以下指南了解各模式支持的功能特性。若无法使用 Thin 模式，可考虑切换至 Thick 模式。

In [ ]:

Copied!





import sys

import oracledb

# please update with your username, password, hostname and service_name
username = "<username>"
password = "<password>"
dsn = "<hostname>/<service_name>"

try:
    conn = oracledb.connect(user=username, password=password, dsn=dsn)
    print("Connection successful!")
except Exception as e:
    print("Connection failed!")
    sys.exit(1)
import sys

import oracledb

# please update with your username, password, hostname and service_name
username = ""
password = ""
dsn = "/"

try:
    conn = oracledb.connect(user=username, password=password, dsn=dsn)
    print("Connection successful!")
except Exception as e:
    print("Connection failed!")
    sys.exit(1)

现在让我们创建一个表并插入一些示例文档进行测试。

In [ ]:

Copied!





try:
    cursor = conn.cursor()

    drop_table_sql = """drop table if exists demo_tab"""
    cursor.execute(drop_table_sql)

    create_table_sql = """create table demo_tab (id number, data clob)"""
    cursor.execute(create_table_sql)

    insert_row_sql = """insert into demo_tab values (:1, :2)"""
    rows_to_insert = [
        (
            1,
            "If the answer to any preceding questions is yes, then the database stops the search and allocates space from the specified tablespace; otherwise, space is allocated from the database default shared temporary tablespace.",
        ),
        (
            2,
            "A tablespace can be online (accessible) or offline (not accessible) whenever the database is open.\nA tablespace is usually online so that its data is available to users. The SYSTEM tablespace and temporary tablespaces cannot be taken offline.",
        ),
        (
            3,
            "The database stores LOBs differently from other data types. Creating a LOB column implicitly creates a LOB segment and a LOB index. The tablespace containing the LOB segment and LOB index, which are always stored together, may be different from the tablespace containing the table.\nSometimes the database can store small amounts of LOB data in the table itself rather than in a separate LOB segment.",
        ),
    ]
    cursor.executemany(insert_row_sql, rows_to_insert)

    conn.commit()

    print("Table created and populated.")
    cursor.close()
except Exception as e:
    print("Table creation failed.")
    cursor.close()
    conn.close()
    sys.exit(1)
try:
    cursor = conn.cursor()

    drop_table_sql = """drop table if exists demo_tab"""
    cursor.execute(drop_table_sql)

    create_table_sql = """create table demo_tab (id number, data clob)"""
    cursor.execute(create_table_sql)

    insert_row_sql = """insert into demo_tab values (:1, :2)"""
    rows_to_insert = [
        (
            1,
            "If the answer to any preceding questions is yes, then the database stops the search and allocates space from the specified tablespace; otherwise, space is allocated from the database default shared temporary tablespace.",
        ),
        (
            2,
            "A tablespace can be online (accessible) or offline (not accessible) whenever the database is open.\nA tablespace is usually online so that its data is available to users. The SYSTEM tablespace and temporary tablespaces cannot be taken offline.",
        ),
        (
            3,
            "The database stores LOBs differently from other data types. Creating a LOB column implicitly creates a LOB segment and a LOB index. The tablespace containing the LOB segment and LOB index, which are always stored together, may be different from the tablespace containing the table.\nSometimes the database can store small amounts of LOB data in the table itself rather than in a separate LOB segment.",
        ),
    ]
    cursor.executemany(insert_row_sql, rows_to_insert)

    conn.commit()

    print("Table created and populated.")
    cursor.close()
except Exception as e:
    print("Table creation failed.")
    cursor.close()
    conn.close()
    sys.exit(1)

加载文档¶

用户可通过灵活配置加载器参数，选择从 Oracle 数据库、文件系统或同时从两者加载文档。有关这些参数的完整说明，请参阅 Oracle AI 向量搜索指南。

使用 OracleDocLoader 的显著优势在于其能处理超过 150 种不同文件格式，无需为不同文档类型配置多个加载器。完整支持的格式列表请参见 Oracle Text 支持文档格式。

以下示例代码片段展示了如何使用 OracleDocLoader

In [ ]:

Copied!





from llama_index.core.schema import Document
from llama_index.readers.oracleai import OracleReader

"""
# loading a local file
loader_params = {}
loader_params["file"] = "<file>"

# loading from a local directory
loader_params = {}
loader_params["dir"] = "<directory>"
"""

# loading from Oracle Database table
loader_params = {
    "owner": "<owner>",
    "tablename": "demo_tab",
    "colname": "data",
}

""" load the docs """
loader = OracleReader(conn=conn, params=loader_params)
docs = loader.load()

""" verify """
print(f"Number of docs loaded: {len(docs)}")
# print(f"Document-0: {docs[0].text}") # content
from llama_index.core.schema import Document
from llama_index.readers.oracleai import OracleReader

"""
# loading a local file
loader_params = {}
loader_params["file"] = ""

# loading from a local directory
loader_params = {}
loader_params["dir"] = ""
"""

# loading from Oracle Database table
loader_params = {
    "owner": "",
    "tablename": "demo_tab",
    "colname": "data",
}

""" load the docs """
loader = OracleReader(conn=conn, params=loader_params)
docs = loader.load()

""" verify """
print(f"Number of docs loaded: {len(docs)}")
# print(f"Document-0: {docs[0].text}") # content

文档分割¶

文档的尺寸可能差异很大，从很小到非常大不等。用户通常倾向于将文档分割成较小的部分，以便于生成嵌入向量。在此分割过程中，提供了丰富的自定义选项。有关这些参数的完整详细信息，请参阅 Oracle AI 向量搜索指南。

以下是一个演示如何实现此功能的示例代码：

In [ ]:

Copied!





from llama_index.core.schema import Document
from llama_index.readers.oracleai import OracleTextSplitter

"""
# Some examples
# split by chars, max 500 chars
splitter_params = {"split": "chars", "max": 500, "normalize": "all"}

# split by words, max 100 words
splitter_params = {"split": "words", "max": 100, "normalize": "all"}

# split by sentence, max 20 sentences
splitter_params = {"split": "sentence", "max": 20, "normalize": "all"}
"""

# split by default parameters
splitter_params = {"normalize": "all"}

# get the splitter instance
splitter = OracleTextSplitter(conn=conn, params=splitter_params)

list_chunks = []
for doc in docs:
    chunks = splitter.split_text(doc.text)
    list_chunks.extend(chunks)

""" verify """
print(f"Number of Chunks: {len(list_chunks)}")
# print(f"Chunk-0: {list_chunks[0]}") # content
from llama_index.core.schema import Document
from llama_index.readers.oracleai import OracleTextSplitter

"""
# Some examples
# split by chars, max 500 chars
splitter_params = {"split": "chars", "max": 500, "normalize": "all"}

# split by words, max 100 words
splitter_params = {"split": "words", "max": 100, "normalize": "all"}

# split by sentence, max 20 sentences
splitter_params = {"split": "sentence", "max": 20, "normalize": "all"}
"""

# split by default parameters
splitter_params = {"normalize": "all"}

# get the splitter instance
splitter = OracleTextSplitter(conn=conn, params=splitter_params)

list_chunks = []
for doc in docs:
    chunks = splitter.split_text(doc.text)
    list_chunks.extend(chunks)

""" verify """
print(f"Number of Chunks: {len(list_chunks)}")
# print(f"Chunk-0: {list_chunks[0]}") # content

端到端演示¶

请参阅我们的完整演示指南 Oracle AI 向量搜索端到端演示指南，借助 Oracle AI 向量搜索构建端到端 RAG 流水线。