Decision infrastructure and enterprise AI agents

Azure AI Search and Agentic Retrieval: Enterprise Guide

Written by Navdeep Singh Gill | 07 August 2024

Azure Cognitive Search is now called Azure AI Search, and it has moved from a single-query search engine to a service with two modes: classic search and agentic retrieval. Microsoft's documentation describes classic search as one request against one index, and agentic retrieval as a multi-query pipeline in which an LLM can plan subqueries, run them in parallel across one or more knowledge sources, rerank them and return grounded results for an agent to use. For most enterprises the practical question is no longer "search or AI" but which mode fits which workload, and what an agentic AI layer should add on top.

Our view is that retrieval and action should be designed separately. Azure AI Search should stay the governed retrieval layer, agentic retrieval should be switched on where questions are genuinely multi-part, and an agent layer should act on the answers only inside approved policy. This guide covers what changed, a decision table, governance and cost, an adoption sequence and the trade-offs.

What happened to Azure Cognitive Search, and what is Azure AI Search today?

They are the same product. Microsoft's FAQ states that Azure Search, Azure Cognitive Search and Azure AI Search are one product, rebranded in October 2019 and again in October 2023, and that you may still see the former names in programmatic interfaces. Microsoft's "What's new" page lists the sequence as Azure Search (March 2015), Azure Cognitive Search (October 2019) and Azure AI Search (November 2023). The semantic search feature was renamed semantic ranker in the same month. If your architecture documents still say "Cognitive Search", they describe the same service, but they may also describe a pre-agentic design.

Microsoft now describes Azure AI Search as a fully managed, cloud-hosted service that connects enterprise data to AI so that agents and LLMs can produce grounded answers. It supports full-text, vector, hybrid and multimodal queries, AI enrichment, relevance tuning, and security through Microsoft Entra ID, Azure Private Link, document-level access control and role-based access. It also underpins Foundry IQ, Microsoft's managed knowledge layer for agents in the Foundry portal. Source: Microsoft Learn, What is Azure AI Search.

What is agentic retrieval in Azure AI Search?

Agentic retrieval is a query pipeline that targets a knowledge base rather than a single index. A knowledge base references one or more knowledge sources, an optional LLM for query planning and answer synthesis, and parameters that govern retrieval. Microsoft's overview describes four steps: your application calls the knowledge base with a query and conversation history; the knowledge base plans focused subqueries (skipped at minimal reasoning effort); subqueries run simultaneously as keyword, vector or hybrid searches with semantic reranking; and the results are merged into a unified response, with optional source references and an activity log.

Knowledge sources can be indexed, backed by a search index on your service, or remote, retrieved live from an external platform at query time. Microsoft's November 2025 update added indexed OneLake, indexed SharePoint, remote SharePoint and web knowledge sources. For retrieval that must respect who is asking, Microsoft notes that remote SharePoint is designed for user-based permission inheritance. Source: Microsoft Learn, Agentic retrieval in Azure AI Search.

Teams that need enterprise search across unstructured knowledge with role-aware results can also look at AgentSearch, an ElixirData agent for context-aware search across unstructured knowledge bases that stays inside existing permissions.

Should you use classic search, agentic retrieval or a full agent layer?

Microsoft's guidance is direct: if you are building a traditional app without an agent or chatbot, direct index queries can meet most needs with lower cost and complexity. Agentic retrieval is for work across multiple knowledge sources or more advanced scenarios, and it adds latency compared with a single-query pipeline. The table below combines Microsoft's documented behaviour with our recommendation.

OptionWhat it doesWhen it fitsCurrent status (Microsoft Learn)
Classic searchOne request against one predefined index, ranked results in one response, no LLM planningPredictable, low-latency lookups; traditional search appsGenerally available
Agentic retrievalLLM-assisted or user-provided plan, parallel subqueries across knowledge sources, merged grounding dataMulti-part questions, chat context, several sources, agent groundingGenerally available, with some capabilities in preview (query planning, answer synthesis, higher reasoning effort)
Agent layer on topReasons over retrieved content and triggers actions in other systemsWhen the answer must start a workflow, such as a ticket or an approvalYour own design; keep actions inside policy and logged

Two production details matter. Microsoft says to use the 2026-04-01 REST API for production workloads that use generally available knowledge source types with minimal, extractive retrieval, and the 2026-08-01-preview API for preview capabilities, which are not covered by a service-level agreement. Objects created in the Azure or Foundry portals may use preview schemas and need migration when you move to the generally available API.

What has changed since the Cognitive Search era?

ChangeWhenWhy it matters for your design
Product renamed Azure AI SearchNovember 2023Update documents, tags and runbooks; API strings may still show the old name
Semantic search renamed semantic rankerNovember 2023Same L2 reranking, clearer name; it is used inside agentic retrieval
Knowledge agents renamed knowledge basesNovember 2025Breaking changes to REST routes and properties; follow Microsoft's migration guide
New knowledge source types (OneLake, SharePoint, web)November 2025Retrieval can span more of the estate without copying everything into one index
Generally available agentic retrieval path in the 2026-04-01 REST API2026A supported route for minimal, extractive retrieval in production

Source: Microsoft Learn, What's new in Azure AI Search.

What does an agent layer add to retrieval?

Retrieval answers a question. An agent layer decides what to do with the answer. In the original version of this article we described this as autonomy, contextual reasoning and action-oriented outcomes, and that framing still holds, with one correction: reasoning is no longer absent from the Microsoft stack, because agentic retrieval itself can plan queries with an LLM. What Azure AI Search does not do for you is execute business actions, so that is where AI agents and agentic workflows come in.

  • Compliance review: an agent retrieves policies and regulations from several sources, then drafts a summary with citations for a reviewer to approve.
  • IT operations: an agent searches incident history and runbooks, then proposes a ticket with the relevant documentation attached. See our view of agentic workflows for IT infrastructure.
  • Data and analytics: an agent combines indexed documents with structured sources, a pattern we describe in multimodal agents on Databricks.
  • Customer support: an agent grounds suggested replies in a knowledge base and records which passages it used.

These are illustrative patterns, not Microsoft features. For build, orchestration and governance of such agents we use ElixirClaw, an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control.

How should you govern, secure and budget for it?

  • Access control: use Microsoft Entra ID, role-based access and document-level access control, and decide early whether permissions are inherited from the source or enforced with security filters. Microsoft lists Azure Blob Storage, ADLS Gen2 and remote SharePoint as routes for permission inheritance and a security filter workaround for other cases.
  • Decision records: keep governed context, versioned policy and decision traces for what an agent retrieved and did. ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents. See also our primer on data governance.
  • Cost model: classic search is billed per query on semantic ranker use. Agentic retrieval is billed in tokens by Azure AI Search for subquery execution and semantic ranking, plus Azure OpenAI tokens for query planning and answer synthesis, so costs appear on two bills. Microsoft's cost tips are to reduce the number of knowledge sources, lower reasoning effort and organise content so fewer sources are needed.
  • Regions: agentic retrieval is available only in select regions, while classic search has no such restriction.

What is a sensible adoption sequence?

  1. Inventory content, owners and access rules before changing any retrieval.
  2. Baseline classic hybrid search with semantic ranker and measure relevance on real questions.
  3. Pilot one knowledge base with minimal reasoning effort and extractive output on the generally available API.
  4. Raise reasoning effort or add answer synthesis only where tests show a gain worth the added latency and cost, and accept the preview terms knowingly.
  5. Add an agent layer for one workflow, with human approval for any action that changes a record.
  6. Track retrieval quality, cost per answer and policy exceptions as standing metrics.

What are the trade-offs, and when should you not use agentic retrieval?

  • Skip it for simple, single-index lookups where latency and cost matter more than query planning.
  • Expect added latency, because planning and multi-query execution take longer than one request.
  • Plan for preview churn: Microsoft warns that preview features can change or be constrained before general availability.
  • Do not give an agent write access until retrieval quality, access control and audit logging have been tested.

To map one search-driven decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

Is Azure Cognitive Search the same as Azure AI Search?

Yes. Microsoft states they are the same product, renamed in October 2019 and again in October 2023, and former names can still appear at the programmatic level.

What is agentic retrieval in Azure AI Search?

It is a multi-query pipeline that targets a knowledge base, plans subqueries, runs them in parallel across knowledge sources, reranks them semantically and returns merged grounding data for agents and apps.

Is agentic retrieval generally available?

Partly. Microsoft lists it as generally available with some capabilities in preview. Query planning, answer synthesis and non-minimal reasoning effort are preview features.

Does agentic retrieval cost more than classic search?

It is billed differently. Classic search is query based, while agentic retrieval is token based and also incurs Azure OpenAI charges for planning and answer synthesis.

Do we need to replace our existing index to use agents?

No. Microsoft states that indexed knowledge sources use the same indexing and query engines as classic search, so existing indexes can sit behind a knowledge base.

Disclosure: XenonStack builds ElixirClaw, ElixirData and the ElixirData agents mentioned in this article.