Private Networking for Azure AI Services: The Part Everyone Underestimates

I recently put together an internal session for AI Apps Solution Engineers on private networking across Azure AI services. The more I worked through the material, the more obvious the real problem became.

Everybody wants the simple version: "turn off public access and add private endpoints."

That sentence is not wrong. It is just wildly incomplete.

In a normal web app conversation, private networking usually means a fairly familiar chain: client, DNS, gateway, backend, data store. There are still plenty of ways to mess that up, but the mental model is stable.

AI apps stretch that model. A single Azure AI workload might involve Microsoft Foundry, Agents, Azure AI Search, Storage, Cosmos DB, Key Vault, Container Apps, App Configuration, Application Insights, and sometimes API Management as an AI gateway. Some traffic comes from users. Some comes from app runtimes. Some comes from managed services like Search indexers. Some comes from agent infrastructure that does not behave like a VM you own.

So the better question is not "does this resource have a private endpoint?"

The better question is: which caller, from which network, under which identity, is trying to reach which dependency?

The AI Landing Zone is the right starting point

I grounded the session in the Azure AI Landing Zones work because it gives us an implementation-backed reference architecture instead of another pretty whiteboard diagram.

The AI Landing Zone is an application landing zone for AI Apps and Agents. It is meant to sit either on its own or inside a broader enterprise platform landing zone. That distinction matters.

That is usually the first design fork:

• Are we deploying a standalone AI app environment?
• Or are we integrating into an existing enterprise platform landing zone?

If you skip that question, the rest of the network conversation gets messy fast.

Private connectivity is a path, not a checkbox

This is the mental model I kept coming back to during the presentation.

Private networking only makes sense when you can describe the whole path:

That sounds basic, but it catches a lot of real-world misses. Consider three paths that might appear on the same architecture diagram:

The resources are related, but the network pattern for each path is different.

Foundry has three networking conversations

Microsoft Foundry network isolation is easier to reason about when you split it into three areas.

Azure AI Search is where people get tripped up

Azure AI Search has two private-link conversations, and mixing them up causes a lot of confusion.

Here is the field example:

You create a private endpoint for Azure AI Search. You disable public access on the Storage account. Then the indexer fails.

The private endpoint helped your clients reach Search. It did not help Search reach Storage. For that, Search needs a shared private link to the storage subresource, and the target resource owner has to approve the connection.

That one sentence has saved me a lot of whiteboard time.

DNS is usually where the design breaks

Private DNS is the part everyone nods at and then underestimates.

The connection string usually does not change. The hostname stays the same. What changes is name resolution from the network where the request starts.

From inside the linked VNet, the service hostname should resolve to the private endpoint IP. From outside, it may still resolve through the public CNAME chain. That is normal.

The dangerous assumption is treating DNS resolution as proof of access. It is not.

A public DNS lookup can prove a resource name exists. It does not prove the caller can reach the private endpoint. It does not prove public network access is enabled. It does not prove the caller has data-plane permission.

For customers with custom DNS or on-premises DNS, someone has to own forwarding or delegation for the privatelink zones. In a mature enterprise, that is usually the platform team, not the app team. This is why I keep asking the same boring question in design reviews:

Who owns DNS?

If nobody can answer that quickly, the private networking design is not done.

The "looks private" trap

There are a few patterns that look secure on paper but fail in production.

DNS resolves. Great. From where?

Public network access is disabled. Great. Does every required caller have a private path?

The portal opens. Great. Is that the same origin as the app runtime or Search indexer?

A private endpoint exists. Great. Is it approved, linked to the right DNS zone, reachable over the route table and firewall, and paired with the correct managed identity permissions?

The validation sequence should be boring and specific:

Anything less is a partial test.

APIM changes the conversation

API Management is not required for every Azure AI app. I would not force it into a simple design just to make the diagram look more enterprise-y.

But once a customer has multiple teams calling models, APIM starts to make a lot of sense as an AI gateway. It gives the platform a place to handle policy, routing, throttling, header normalization, backend selection, observability, and governance across model endpoints.

That is why the AI Landing Zones work includes both a Foundry Landing Zone and an AI Gateway Landing Zone. Some customers will deploy both. Some will integrate with a gateway the platform team already owns.

This is another place where the platform landing zone question matters. If the central team already runs APIM, the AI app should probably integrate with it. If the AI app team deploys its own APIM without talking to the platform team, they may have built a duplicate control plane by accident.

The field-ready checklist

This is the slide I would keep handy for any Azure AI private networking review.

Before calling an AI app privately networked, answer these five questions:

If you cannot answer all five, the architecture may be private in one place and broken in three others.

The part I would teach first

If I only had ten minutes with an app team, I would not start with the private endpoint wizard.

I would start with a blank page and draw the callers:

• Who is the user?
• Where does the app run?
• What does Search index?
• Where does agent state live?
• Which identity performs each action?
• Who owns DNS and egress?

Then I would map those answers back to the AI Landing Zone pattern.

Private networking for Azure AI is not one setting. It is a chain of named, routed, authorized paths across the app.

Once you see it that way, the service-specific details stop feeling random. Foundry, AI Search, Storage, Cosmos DB, Key Vault, and APIM all have their own knobs, but the review pattern stays the same:

origin, route, name, identity, dependency.

That is the part worth memorizing.

Resources:

• Azure AI Landing Zones
• Azure/AI-Landing-Zones on GitHub
• AI Landing Zone Portal Deployment Guide
• Configure network isolation for Microsoft Foundry
• Configure virtual networks for Foundry Tools
• Create a private endpoint for Azure AI Search
• Connect through a shared private link in Azure AI Search
• Azure Private Endpoint private DNS zone values