使用 Azure OpenAI GPT-4o mini 实现多模态LLM图像推理¶

本笔记本演示如何通过 Azure OpenAI LLM 类/抽象层使用 GPT-4o mini 进行图像理解与推理。如需查看完整示例，请访问此笔记本。

In [ ]:

%pip install llama-index-llms-azure-openai

%pip install llama-index-llms-azure-openai

先决条件¶

1. 创建 Azure 订阅 - 可在此免费申请
2. 申请访问 Azure OpenAI 服务请点击此处
3. 在 Azure 门户中创建资源
4. 在 Azure OpenAI Studio 中部署模型

更多细节可参考本指南。

请记录 "模型名称" 和 "部署名称"，连接 LLM 时将需要这些信息。

使用 GPT-4o mini 理解来自 URL / base64 的图像¶

In [ ]:

import os

os.environ["AZURE_OPENAI_API_KEY"] = "xxx"
os.environ["AZURE_OPENAI_ENDPOINT"] = "https://YOUR_URL.openai.azure.com/"
os.environ["OPENAI_API_VERSION"] = "2024-02-15-preview"

import os os.environ["AZURE_OPENAI_API_KEY"] = "xxx" os.environ["AZURE_OPENAI_ENDPOINT"] = "https://YOUR_URL.openai.azure.com/" os.environ["OPENAI_API_VERSION"] = "2024-02-15-preview"

初始化 AzureOpenAI 并加载 URL 中的图像¶

与常规 OpenAI 不同，除了 model 参数外，您还需要传递 engine 参数。engine 是您在 Azure OpenAI Studio 中部署模型时为其指定的名称。

In [ ]:

from llama_index.llms.azure_openai import AzureOpenAI

from llama_index.llms.azure_openai import AzureOpenAI

In [ ]:

azure_openai_llm = AzureOpenAI(
    engine="my-gpt-4o-mini",
    model="gpt-4o-mini",
    max_new_tokens=300,
)

azure_openai_llm = AzureOpenAI( engine="my-gpt-4o-mini", model="gpt-4o-mini", max_new_tokens=300, )

或者，您也可以跳过环境变量设置，直接通过构造函数传入参数。

In [ ]:

azure_openai_llm = AzureOpenAI(
    azure_endpoint="https://YOUR_URL.openai.azure.com/",
    engine="my-gpt-4o-mini",
    api_version="2024-02-15-preview",
    model="gpt-4o-mini",
    max_new_tokens=300,
    api_key="xxx",
    supports_content_blocks=True,
)

azure_openai_llm = AzureOpenAI( azure_endpoint="https://YOUR_URL.openai.azure.com/", engine="my-gpt-4o-mini", api_version="2024-02-15-preview", model="gpt-4o-mini", max_new_tokens=300, api_key="xxx", supports_content_blocks=True, )

In [ ]:

import base64
import requests
from llama_index.core.schema import Document, MediaResource

image_url = "https://www.visualcapitalist.com/wp-content/uploads/2023/10/US_Mortgage_Rate_Surge-Sept-11-1.jpg"

response = requests.get(image_url)
if response.status_code != 200:
    raise ValueError("Error: Could not retrieve image from URL.")
img_data = base64.b64encode(response.content)

image_document = Document(image_resource=MediaResource(data=img_data))

import base64 import requests from llama_index.core.schema import Document, MediaResource image_url = "https://www.visualcapitalist.com/wp-content/uploads/2023/10/US_Mortgage_Rate_Surge-Sept-11-1.jpg" response = requests.get(image_url) if response.status_code != 200: raise ValueError("Error: Could not retrieve image from URL.") img_data = base64.b64encode(response.content) image_document = Document(image_resource=MediaResource(data=img_data))

In [ ]:

from IPython.display import HTML

src = f'<img width=400 src="data:{image_document.image_resource.mimetype};base64,{image_document.image_resource.data.decode("utf-8")}"/>'
HTML(src)

from IPython.display import HTML src = f'' HTML(src)

Out[ ]:

通过图像补全提示词¶

In [ ]:

from llama_index.core.llms import (
    ChatMessage,
    ImageBlock,
    TextBlock,
    MessageRole,
)

msg = ChatMessage(
    role=MessageRole.USER,
    blocks=[
        TextBlock(text="Describe the images as an alternative text"),
        ImageBlock(image=image_document.image_resource.data),
    ],
)

response = azure_openai_llm.chat(messages=[msg])

from llama_index.core.llms import ( ChatMessage, ImageBlock, TextBlock, MessageRole, ) msg = ChatMessage( role=MessageRole.USER, blocks=[ TextBlock(text="Describe the images as an alternative text"), ImageBlock(image=image_document.image_resource.data), ], ) response = azure_openai_llm.chat(messages=[msg])

In [ ]:

print(response)

print(response)

assistant: The image presents a graph titled "The U.S. Mortgage Rate Surge," comparing the U.S. 30-year fixed-rate mortgage rates with existing home sales from 2014 to 2023. 

- The vertical axis on the left represents the mortgage rate, while the right vertical axis indicates the number of existing home sales, measured in millions.
- A blue line illustrates the trend of existing home sales, showing fluctuations over the years, peaking around 2020 and declining thereafter.
- A red line represents the mortgage rate, which has seen a significant increase, particularly in 2022 and 2023, reaching its highest level in over 20 years.
- The background includes a subtle grid, and the data sources are noted at the bottom. The overall design is clean and informative, aimed at highlighting the relationship between rising mortgage rates and declining home sales.