Key takeaway
- How multimodal agents work on Databricks in 2027: images and documents, Agent Bricks, Unity Gateway governance, limits, cost and an adoption plan.
A Databricks multimodal agent is an AI agent that reads more than text, such as images and scanned or photographed documents, and then acts on what it finds using the same governed data, tools and models as any other agent on the platform. In Databricks documentation the pieces are Unity Catalog volumes for the files, Databricks, query vision models for image-capable models, ai_parse_document for document parsing, Agent Bricks for building the agent and Unity Gateway for controlling it.
The enterprise question is less "can a model look at a picture?" and more "who is allowed to send which files to which model, and how do we trace what the agent did?". This guide maps each modality to a current Databricks feature, shows where the limits are, and sets out the governance, cost and adoption choices. For the lakehouse-wide view of agents see our guide to Databricks AI agents; for camera and video use see multimodal surveillance agents on Databricks.
What is a multimodal agent on Databricks?
A multimodal model accepts more than one kind of input, for example text plus images. An agent adds a loop around the model: it reads a request, chooses tools, acts and reports back. A multimodal agent therefore combines both: it can, for instance, receive a photographed form, extract its content, check it against a table and return a decision with a trace of the steps.
On Databricks, the agent itself is built with Agent Bricks, described on its page as "the Databricks agent developer platform". The page lists model access through Unity Gateway and Foundation Model APIs, compute through Agent Server and Agent Runtime, context from MCP servers, tools, AI Search and Genie Agents, quality through MLflow, and governance through Unity Catalog and Unity Gateway policies (updated 13 October 2026). That page does not itself state which input types are supported, so image support depends on the model you choose, covered below.
Which Databricks feature handles which modality?
The table below uses only what Databricks documents. Rows marked "not documented" are gaps in the pages we read in November 2026, not statements that the capability is absent; confirm them with Databricks before you design around them.
| Need | Databricks feature | Current status and limits (according to its site) |
|---|---|---|
| Store images, PDFs, audio and video under governance | Unity Catalog volumes | Volumes hold non-tabular data in any format, with examples including image, audio, video and PDF files. Managed volumes route all access through Unity Catalog; external volumes need extra controls for direct reads and writes from outside systems (page updated 12 October 2026). |
| Ask a model about an image | Foundation Model APIs and Unity Gateway, queried as described in Databricks, query vision models | Images go in as base64-encoded data URLs in the Chat Completions format, and several images can be sent in one request. Most models accept JPEG, PNG, WebP and non-animated GIF; Gemini image models accept JPEG, PNG and WebP only (page updated 11 November 2026). |
| Classify or describe images in SQL | The ai_query() function | Documented on the vision page as in Public Preview, reading image files from a Unity Catalog volume. |
| Turn documents into structured content | ai_parse_document | Parses PDF, JPG/JPEG, PNG, TIFF, DOC/DOCX and PPT/PPTX into a VARIANT result with layout elements such as headers, tables and footers. Needs Databricks Runtime 17.3 or above, is available only in some regions, and the page states no release stage (updated 4 November 2026). |
| Retrieve by similarity | Databricks AI Search | Formerly called Databricks Vector Search. The page lists image and video recognition among the uses of embeddings but does not describe a multimodal search feature, and some full-text features are marked Beta (updated 29 October 2026). |
| Ask questions of files and tables together | Genie Agents | Analysing files in volumes with a Genie Agent is listed as Public Preview, for unstructured files alongside structured data; the page does not mention images specifically (updated 2 November 2026). |
| Audio and video input | Not documented on the pages we read | We found no page confirming audio or video input for agents. Treat voice and video agents as something to prove in a pilot. |
Limits differ by model. The vision page lists, for example, up to 5 images per request and 10 MB total for Gemma 3 12B and Llama 4 Maverick, up to 50 images and 7 MB per image for Gemini image models, and up to 100 images per request for Claude models. Check the current table for the model you plan to use rather than designing to a single number.
How does the architecture fit together?
A practical pattern has five layers, each using a documented Databricks capability.
- Files and tables. Source images and documents live in Unity Catalog volumes; structured records live in Unity Catalog tables. One permission model covers both.
- Perception. A vision-capable model reached through Unity Gateway reads images, and ai_parse_document extracts content from documents. Keep the extracted text and fields as governed tables so they can be audited and re-used.
- Retrieval. Databricks AI Search indexes the extracted content so the agent can find relevant passages.
- Reasoning and action. An Agent Bricks agent chooses tools, which can include Genie Agents for questions over curated data and MCP servers or APIs for actions.
- Observability. MLflow on Databricks records agent traces including model calls, tool calls and retrieval, with inputs, outputs, latency, token usage and cost, stored in Unity Catalog where they can be queried with SQL (page updated 8 October 2026). That page does not discuss image tracing, so confirm how images appear in traces before relying on them for audit.
Teams that want the platform-level story across lakehouse, governance and silos can read how Databricks removes data silos. For serving capacity see Databricks autoscaling for AI inference.
What has changed in the Databricks AI naming and stack?
Older material, including earlier versions of many vendor and partner articles, talks about Mosaic AI, Mosaic AI Model Serving, the Mosaic AI Agent Framework and Vector Search. The current documentation pages we read describe Agent Bricks, Foundation Model APIs, Unity Gateway and Databricks AI Search (formerly Vector Search), and do not use the Mosaic AI names on those pages. Use current names in architecture documents and procurement, and confirm product names and release stages on the Databricks page at the time you buy or build.
| Earlier term | Where it appears now | Action for your team |
|---|---|---|
| Mosaic AI Agent Framework | Agent Bricks developer platform | Use the current name in designs; check the Agent Bricks page for the framework you need |
| Vector Search | Databricks AI Search, formerly Databricks Vector Search | Update diagrams and runbooks |
| Model serving and gateway controls | Unity Gateway over Foundation Model APIs | Route all model calls, including image calls, through the governed path |
| Model catalogue | The Foundation Model APIs list changes, and Databricks publishes retirement notices | Check the legacy and retired models page; for example the overview page lists DeepSeek V4.1 Flash retiring on 19 November 2026 |
What governance, security and cost controls does a multimodal agent need?
Images and scanned documents often contain personal data, signatures, faces or commercial secrets that never appear in a table. Treat each file as governed data.
- Access. Unity Gateway registers AI assets such as model services, external model providers, MCP tools and HTTP connections as Unity Catalog securables, so access is granted with the same privileges used for tables and volumes.
- Guardrails. Service policies can control how each request and response proceeds based on content and caller, and can be built-in, custom or external.
- Limits and cost. The Unity Gateway page lists rate limits on model and MCP services, budgets with per-user thresholds and hard caps, and cost attribution to services, models, principals and tags. Image requests are typically larger than text requests, so set budgets before a pilot, not after.
- Logging. Requests and responses can be logged to Unity Catalog Delta tables. Decide up front whether logged payloads may contain images or extracted text and who can read them.
- External volumes. Databricks notes that Unity Catalog does not govern direct reads and writes to an external volume from outside systems, so add short-lived credentials or cloud-native policies.
- Data location. Check the region availability of each AI function and model before sending regulated data; the document parsing page says availability varies by region.
For the broader governance model see governance and security in SAP Databricks, for spend see FinOps for Databricks, and for quality checks on the data the agent reads see data quality monitoring in Databricks. ElixirData describes its Context OS as one that keeps governed context, versioned policy and decision traces in one decision layer above AI agents, which is the layer where a multimodal agent's decisions and the policy behind them are recorded. AgentGRC keeps autonomous agent actions inside policy, with the audit trail risk teams need.
When does a multimodal agent fit, and when does it not?
| Situation | Fit | Why |
|---|---|---|
| High volumes of forms, scans or photos that staff read by hand | Good fit | Document parsing plus an agent that checks extracted fields against governed tables |
| Questions that span tables and a folder of files | Good fit, pilot first | Genie Agents file analysis is Public Preview, so test accuracy and access rules |
| Decisions with legal or safety consequences | Only with a human approving | Keep a person in the loop and log the evidence the agent used |
| Voice or video agents | Prove first | Audio and video input are not documented on the pages we read |
| Clean tabular data with clear rules | Probably not | A SQL rule or a text-only agent is cheaper and easier to audit |
| Files you may not send to a third-party model | Not until policy is settled | Check model provider, region and logging settings first |
How should an enterprise adopt multimodal agents?
- Pick one decision. Name the decision the agent supports, the files it reads and the table it checks against. Avoid starting from the modality.
- Land the files in governed volumes. Set privileges, retention and ownership before any model sees them.
- Test models on your own samples. Compare two or three models from the Foundation Model APIs list on real documents, scoring accuracy, cost and refusal behaviour. Do not rely on vendor benchmarks.
- Add tracing and evaluation early. Use MLflow traces and scorers, and turn failed cases into an evaluation set.
- Add approvals. Route low-confidence or high-impact outputs to a person.
- Scale with budgets. Set Unity Gateway limits, then widen the document set and the number of users.
If you want help with delivery, our team offers multimodal AI development services, and ElixirClaw is an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control. For plain-language questions over enterprise data, AgentAnalyst lets teams ask questions of enterprise data in plain language and get real-time insights.
To map one multimodal decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.
What should you plan for in 2027?
Databricks publishes model retirement dates, and agents tied to a retiring model need a plan.
- 4 February 2027: Zhipu AI GLM 5.2 retires for both pay per token and provisioned throughput, and Databricks recommends GLM 5.3 as the replacement (Databricks retired models policy, updated 12 November 2026).
- Now: the Foundation Model APIs overview already lists that model as legacy, so check which models your agents call and re-test image and document cases on any replacement before the date.
Keep model choice behind Unity Gateway so a swap is a configuration change, and keep your evaluation set ready to re-run.
FAQ: Databricks multimodal agents
What is a Databricks multimodal agent?
An agent that reads more than text, such as images and scanned documents, using Databricks features like Unity Catalog volumes, vision-capable models through Foundation Model APIs and Unity Gateway, document parsing and an agent built with Agent Bricks.
Which image formats can Databricks models read?
According to the Databricks vision page, most models accept JPEG, PNG, WebP and non-animated GIF, while Gemini image models accept only JPEG, PNG and WebP. Limits on image counts and size vary by model.
Can Databricks parse scanned documents and PDFs?
The ai_parse_document function parses PDF, JPG/JPEG, PNG, TIFF, DOC/DOCX and PPT/PPTX into structured content. It needs Databricks Runtime 17.3 or above, is available only in some regions, and its page does not state a release stage.
Does Databricks support audio and video agents?
We did not find a Databricks page confirming audio or video input for agents. Confirm with Databricks and prove it in a pilot before committing.
How do you govern images sent to an AI model?
Keep files in Unity Catalog volumes, route model calls through Unity Gateway so access, guardrail policies, rate limits and budgets apply, and log requests to Unity Catalog tables with clear rules on who can read them.
Is Mosaic AI still the name to use?
The current Databricks pages we read use Agent Bricks, Foundation Model APIs, Unity Gateway and Databricks AI Search, and do not use the Mosaic AI names. Check the Databricks page for current product names before you write architecture documents.
Next Steps in Implementing Databricks Multimodal Agents
Talk to our experts about implementing Databricks Multi-Modal AI systems. Learn how industries and different departments use agentic workflows and decision intelligence to become data-driven. Utilize AI to automate and optimize IT support and operations, improving efficiency and responsiveness.
Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE