Docling 阅读器¶

概述¶

Docling 能够将 PDF、DOCX、HTML 等文档格式提取为富文本表示（包含布局、表格等元素），并可导出为 Markdown 或 JSON 格式。

本笔记本展示的 Docling Reader 和 Docling Node Parser 实现了 Docling 与 LlamaIndex 的无缝集成，使您能够：

• 轻松快速地在 LLM 应用中使用各类文档
• 利用 Docling 的富文本格式实现高级的、文档原生的语义落地

安装配置¶

• 👉 为获得最佳转换速度，请尽可能使用 GPU 加速；例如在 Colab 上运行时，请启用 GPU 运行时环境。
• 本笔记本使用 HuggingFace 的 Inference API；如需提升大语言模型配额，可通过环境变量 HF_TOKEN 提供令牌。
• 依赖项可按以下方式安装（--no-warn-conflicts 参数专为 Colab 预置的 Python 环境设计，严格使用时可直接移除）：

In [ ]:

%pip install -q --progress-bar off --no-warn-conflicts llama-index-core llama-index-readers-docling llama-index-node-parser-docling llama-index-embeddings-huggingface llama-index-llms-huggingface-api llama-index-readers-file python-dotenv

%pip install -q --progress-bar off --no-warn-conflicts llama-index-core llama-index-readers-docling llama-index-node-parser-docling llama-index-embeddings-huggingface llama-index-llms-huggingface-api llama-index-readers-file python-dotenv

现在我们可以定义主要参数：

In [ ]:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI
import os
from dotenv import load_dotenv


def get_env_from_colab_or_os(key):
    try:
        from google.colab import userdata

        try:
            return userdata.get(key)
        except userdata.SecretNotFoundError:
            pass
    except ImportError:
        pass
    return os.getenv(key)


load_dotenv()
EMBED_MODEL = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
GEN_MODEL = HuggingFaceInferenceAPI(
    token=get_env_from_colab_or_os("HF_TOKEN"),
    model_name="mistralai/Mixtral-8x7B-Instruct-v0.1",
)
SOURCE = "https://arxiv.org/pdf/2408.09869"  # Docling Technical Report
QUERY = "Which are the main AI models in Docling?"

from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI import os from dotenv import load_dotenv def get_env_from_colab_or_os(key): try: from google.colab import userdata try: return userdata.get(key) except userdata.SecretNotFoundError: pass except ImportError: pass return os.getenv(key) load_dotenv() EMBED_MODEL = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5") GEN_MODEL = HuggingFaceInferenceAPI( token=get_env_from_colab_or_os("HF_TOKEN"), model_name="mistralai/Mixtral-8x7B-Instruct-v0.1", ) SOURCE = "https://arxiv.org/pdf/2408.09869" # Docling Technical Report QUERY = "Which are the main AI models in Docling?"

使用 Markdown 导出功能¶

要构建一个简单的 RAG 流水线，我们可以：

• 定义一个默认导出为 Markdown 格式的 DoclingReader，以及
• 为这些基于 Markdown 的文档使用标准节点解析器，例如 MarkdownNodeParser

In [ ]:

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser
from llama_index.readers.docling import DoclingReader

reader = DoclingReader()
node_parser = MarkdownNodeParser()

index = VectorStoreIndex.from_documents(
    documents=reader.load_data(SOURCE),
    transformations=[node_parser],
    embed_model=EMBED_MODEL,
)
result = index.as_query_engine(llm=GEN_MODEL).query(QUERY)
print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:")
display([(n.text, n.metadata) for n in result.source_nodes])

from llama_index.core import VectorStoreIndex from llama_index.core.node_parser import MarkdownNodeParser from llama_index.readers.docling import DoclingReader reader = DoclingReader() node_parser = MarkdownNodeParser() index = VectorStoreIndex.from_documents( documents=reader.load_data(SOURCE), transformations=[node_parser], embed_model=EMBED_MODEL, ) result = index.as_query_engine(llm=GEN_MODEL).query(QUERY) print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:") display([(n.text, n.metadata) for n in result.source_nodes])

Q: Which are the main AI models in Docling?
A: 1. A layout analysis model, an accurate object-detector for page elements. 2. TableFormer, a state-of-the-art table structure recognition model.

Sources:

[('3.2 AI models\n\nAs part of Docling, we initially release two highly capable AI models to the open-source community, which have been developed and published recently by our team. The first model is a layout analysis model, an accurate object-detector for page elements [13]. The second model is TableFormer [12, 9], a state-of-the-art table structure recognition model. We provide the pre-trained weights (hosted on huggingface) and a separate package for the inference code as docling-ibm-models . Both models are also powering the open-access deepsearch-experience, our cloud-native service for knowledge exploration tasks.',
  {'Header_2': '3.2 AI models'}),
 ("5 Applications\n\nThanks to the high-quality, richly structured document conversion achieved by Docling, its output qualifies for numerous downstream applications. For example, Docling can provide a base for detailed enterprise document search, passage retrieval or classification use-cases, or support knowledge extraction pipelines, allowing specific treatment of different structures in the document, such as tables, figures, section structure or references. For popular generative AI application patterns, such as retrieval-augmented generation (RAG), we provide quackling , an open-source package which capitalizes on Docling's feature-rich document output to enable document-native optimized vector embedding and chunking. It plugs in seamlessly with LLM frameworks such as LlamaIndex [8]. Since Docling is fast, stable and cheap to run, it also makes for an excellent choice to build document-derived datasets. With its powerful table structure recognition, it provides significant benefit to automated knowledge-base construction [11, 10]. Docling is also integrated within the open IBM data prep kit [6], which implements scalable data transforms to build large-scale multi-modal training datasets.",
  {'Header_2': '5 Applications'})]

使用 Docling 格式¶

为了充分利用 Docling 丰富的原生格式，我们：

• 创建一个导出类型为 JSON 的 DoclingReader，以及
• 使用 DoclingNodeParser 来正确解析该 Docling 格式。

请注意现在数据源还包含文档级别的定位信息（例如页码或边界框信息）：

In [ ]:

from llama_index.node_parser.docling import DoclingNodeParser

reader = DoclingReader(export_type=DoclingReader.ExportType.JSON)
node_parser = DoclingNodeParser()

index = VectorStoreIndex.from_documents(
    documents=reader.load_data(SOURCE),
    transformations=[node_parser],
    embed_model=EMBED_MODEL,
)
result = index.as_query_engine(llm=GEN_MODEL).query(QUERY)
print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:")
display([(n.text, n.metadata) for n in result.source_nodes])

from llama_index.node_parser.docling import DoclingNodeParser reader = DoclingReader(export_type=DoclingReader.ExportType.JSON) node_parser = DoclingNodeParser() index = VectorStoreIndex.from_documents( documents=reader.load_data(SOURCE), transformations=[node_parser], embed_model=EMBED_MODEL, ) result = index.as_query_engine(llm=GEN_MODEL).query(QUERY) print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:") display([(n.text, n.metadata) for n in result.source_nodes])

Q: Which are the main AI models in Docling?
A: The main AI models in Docling are a layout analysis model and TableFormer. The layout analysis model is an accurate object-detector for page elements, and TableFormer is a state-of-the-art table structure recognition model.

Sources:

[('As part of Docling, we initially release two highly capable AI models to the open-source community, which have been developed and published recently by our team. The first model is a layout analysis model, an accurate object-detector for page elements [13]. The second model is TableFormer [12, 9], a state-of-the-art table structure recognition model. We provide the pre-trained weights (hosted on huggingface) and a separate package for the inference code as docling-ibm-models . Both models are also powering the open-access deepsearch-experience, our cloud-native service for knowledge exploration tasks.',
  {'schema_name': 'docling_core.transforms.chunker.DocMeta',
   'version': '1.0.0',
   'doc_items': [{'self_ref': '#/texts/34',
     'parent': {'$ref': '#/body'},
     'children': [],
     'label': 'text',
     'prov': [{'page_no': 3,
       'bbox': {'l': 107.07593536376953,
        't': 406.1695251464844,
        'r': 504.1148681640625,
        'b': 330.2677307128906,
        'coord_origin': 'BOTTOMLEFT'},
       'charspan': [0, 608]}]}],
   'headings': ['3.2 AI models'],
   'origin': {'mimetype': 'application/pdf',
    'binary_hash': 14981478401387673002,
    'filename': '2408.09869v3.pdf'}}),
 ('With Docling , we open-source a very capable and efficient document conversion tool which builds on the powerful, specialized AI models and datasets for layout analysis and table structure recognition we developed and presented in the recent past [12, 13, 9]. Docling is designed as a simple, self-contained python library with permissive license, running entirely locally on commodity hardware. Its code architecture allows for easy extensibility and addition of new features and models.',
  {'schema_name': 'docling_core.transforms.chunker.DocMeta',
   'version': '1.0.0',
   'doc_items': [{'self_ref': '#/texts/9',
     'parent': {'$ref': '#/body'},
     'children': [],
     'label': 'text',
     'prov': [{'page_no': 1,
       'bbox': {'l': 107.0031967163086,
        't': 136.7283935546875,
        'r': 504.04998779296875,
        'b': 83.30133056640625,
        'coord_origin': 'BOTTOMLEFT'},
       'charspan': [0, 488]}]}],
   'headings': ['1 Introduction'],
   'origin': {'mimetype': 'application/pdf',
    'binary_hash': 14981478401387673002,
    'filename': '2408.09869v3.pdf'}})]

使用 Simple Directory Reader¶

为了演示这种使用模式，我们首先建立一个测试文档目录。

In [ ]:

from pathlib import Path
from tempfile import mkdtemp
import requests

tmp_dir_path = Path(mkdtemp())
r = requests.get(SOURCE)
with open(tmp_dir_path / f"{Path(SOURCE).name}.pdf", "wb") as out_file:
    out_file.write(r.content)

from pathlib import Path from tempfile import mkdtemp import requests tmp_dir_path = Path(mkdtemp()) r = requests.get(SOURCE) with open(tmp_dir_path / f"{Path(SOURCE).name}.pdf", "wb") as out_file: out_file.write(r.content)

使用上述任一变体中的 reader 和 node_parser 定义时，与 SimpleDirectoryReader 的配合使用方式如下：

In [ ]:

from llama_index.core import SimpleDirectoryReader

dir_reader = SimpleDirectoryReader(
    input_dir=tmp_dir_path,
    file_extractor={".pdf": reader},
)

index = VectorStoreIndex.from_documents(
    documents=dir_reader.load_data(SOURCE),
    transformations=[node_parser],
    embed_model=EMBED_MODEL,
)
result = index.as_query_engine(llm=GEN_MODEL).query(QUERY)
print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:")
display([(n.text, n.metadata) for n in result.source_nodes])

from llama_index.core import SimpleDirectoryReader dir_reader = SimpleDirectoryReader( input_dir=tmp_dir_path, file_extractor={".pdf": reader}, ) index = VectorStoreIndex.from_documents( documents=dir_reader.load_data(SOURCE), transformations=[node_parser], embed_model=EMBED_MODEL, ) result = index.as_query_engine(llm=GEN_MODEL).query(QUERY) print(f"Q: {QUERY}\nA: {result.response.strip()}\n\nSources:") display([(n.text, n.metadata) for n in result.source_nodes])

Q: Which are the main AI models in Docling?
A: The main AI models in Docling are a layout analysis model and TableFormer. The layout analysis model is an accurate object-detector for page elements, and TableFormer is a state-of-the-art table structure recognition model.

Sources:

[('As part of Docling, we initially release two highly capable AI models to the open-source community, which have been developed and published recently by our team. The first model is a layout analysis model, an accurate object-detector for page elements [13]. The second model is TableFormer [12, 9], a state-of-the-art table structure recognition model. We provide the pre-trained weights (hosted on huggingface) and a separate package for the inference code as docling-ibm-models . Both models are also powering the open-access deepsearch-experience, our cloud-native service for knowledge exploration tasks.',
  {'file_path': '/var/folders/76/4wwfs06x6835kcwj4186c0nc0000gn/T/tmpgwz4gpzx/2408.09869.pdf',
   'file_name': '2408.09869.pdf',
   'file_type': 'application/pdf',
   'file_size': 5566574,
   'creation_date': '2024-10-24',
   'last_modified_date': '2024-10-24',
   'schema_name': 'docling_core.transforms.chunker.DocMeta',
   'version': '1.0.0',
   'doc_items': [{'self_ref': '#/texts/34',
     'parent': {'$ref': '#/body'},
     'children': [],
     'label': 'text',
     'prov': [{'page_no': 3,
       'bbox': {'l': 107.07593536376953,
        't': 406.1695251464844,
        'r': 504.1148681640625,
        'b': 330.2677307128906,
        'coord_origin': 'BOTTOMLEFT'},
       'charspan': [0, 608]}]}],
   'headings': ['3.2 AI models'],
   'origin': {'mimetype': 'application/pdf',
    'binary_hash': 14981478401387673002,
    'filename': '2408.09869.pdf'}}),
 ('With Docling , we open-source a very capable and efficient document conversion tool which builds on the powerful, specialized AI models and datasets for layout analysis and table structure recognition we developed and presented in the recent past [12, 13, 9]. Docling is designed as a simple, self-contained python library with permissive license, running entirely locally on commodity hardware. Its code architecture allows for easy extensibility and addition of new features and models.',
  {'file_path': '/var/folders/76/4wwfs06x6835kcwj4186c0nc0000gn/T/tmpgwz4gpzx/2408.09869.pdf',
   'file_name': '2408.09869.pdf',
   'file_type': 'application/pdf',
   'file_size': 5566574,
   'creation_date': '2024-10-24',
   'last_modified_date': '2024-10-24',
   'schema_name': 'docling_core.transforms.chunker.DocMeta',
   'version': '1.0.0',
   'doc_items': [{'self_ref': '#/texts/9',
     'parent': {'$ref': '#/body'},
     'children': [],
     'label': 'text',
     'prov': [{'page_no': 1,
       'bbox': {'l': 107.0031967163086,
        't': 136.7283935546875,
        'r': 504.04998779296875,
        'b': 83.30133056640625,
        'coord_origin': 'BOTTOMLEFT'},
       'charspan': [0, 488]}]}],
   'headings': ['1 Introduction'],
   'origin': {'mimetype': 'application/pdf',
    'binary_hash': 14981478401387673002,
    'filename': '2408.09869.pdf'}})]