Azure Load Balancer Architecture Guide

Running a single virtual machine for a production service creates an obvious bottleneck. When that VM fails—due to hardware, a network issue, or a routine reboot—the service goes offline. Even when the VM is healthy, its network interface and private IP address limit how much traffic it can accept. A straightforward solution is to run multiple instances and place a load balancer in front of them. The load balancer provides a stable, routable entry point and distributes incoming connections among the healthy instances.

Azure Load Balancer is Microsoft’s managed service that does exactly this at Layer 4 of the OSI model. It works with TCP and UDP traffic, forwarding packets to a pool of backend virtual machines or Virtual Machine Scale Set instances based on a hash of the packet’s header fields. Because it operates at the transport layer, it does not inspect HTTP request paths, host headers, cookies, or application‑layer payloads. It does not terminate TLS connections or provide a web application firewall. Those responsibilities belong to other Azure services—or to the application itself.

This guide is written for engineers who need to design, operate, or troubleshoot Layer 4 load‑balancing architectures in Azure. It explains how the service behaves, which architectural decisions matter in production, and when a different Azure traffic service is the better choice.

Mental model of Azure Load Balancer

Azure Load Balancer is not a single appliance. It is a distributed, highly available service that runs inside the Azure software‑defined networking fabric. When you create a load balancer, you define a set of rules and pools. The service continuously evaluates health probes and, for each new TCP or UDP flow, selects a healthy backend instance to handle the traffic.

The components you configure are:

• Frontend IP configuration – A public or private IP address that clients target.
• Backend pool – The set of virtual machine NICs, scale set instances, or IP addresses that can receive traffic.
• Health probe – A periodic check that determines whether a backend instance is eligible to receive new flows.
• Load‑balancing rule – A mapping from a frontend IP:port and protocol to a backend pool and port, along with the health probe to use.
• Inbound NAT rule – A static mapping from a frontend port to a specific backend instance and port (port forwarding, not load balancing).
• Outbound rule – Configures Source Network Address Translation (SNAT) for outbound connections from backend instances that use the load balancer’s public frontend IP as the egress address.

A backend instance that is a member of the pool is a candidate for traffic. Only instances that pass the health probe become eligible. Eligibility can change in seconds; the load balancer’s data plane updates accordingly.

How a TCP connection flows through Azure Load Balancer

Consider a public load balancer with a frontend IP 203.0.113.10 listening on TCP port 443, a backend pool of three virtual machines, and an HTTP health probe on port 80 path /health.

1. A client sends a TCP SYN packet to 203.0.113.10:443.
2. Azure Load Balancer matches the packet to the load‑balancing rule for frontend port 443 and the backend pool.
3. The service computes a 5‑tuple hash over source IP, source port, destination IP, destination port, and protocol (TCP). This hash selects a backend instance from those currently healthy.
4. The destination IP address in the packet is rewritten to the private IP of the chosen backend. The source IP address remains unchanged (unless floating IP is enabled). The packet is forwarded to the backend’s network interface.
5. The backend receives the SYN, completes the TCP three‑way handshake, and processes the application request. Return traffic flows back through the same load balancer path because Azure networking ensures flow symmetry.
6. Meanwhile, the health probe continues to send HTTP requests to each backend on port 80. If a backend stops responding with 200 OK, the load balancer removes it from the eligible pool. Established connections to that backend are not immediately terminated; they persist until the idle timeout expires or the backend resets them.

The diagram separates the data path (solid lines) from the health‑check path (dashed line). The health probe does not carry client traffic; it is a parallel, continuous validation that runs between the load balancer and each backend.

Layer 4 behaviour and its limits

Because Azure Load Balancer works at the transport layer, its traffic‑distribution logic is simple and fast, but also limited in what it can do.

What the 5‑tuple hash means in practice. Every new TCP or UDP flow—defined by the combination of source IP, source port, destination IP, destination port, and protocol—is hashed to a backend instance. Once a flow is pinned to an instance, all packets in that flow go to the same backend for the lifetime of the load‑balancing rule. This preserves connection state without the load balancer needing to track application sessions.

Implications for application behaviour:

• Multiple HTTP requests sent over a single persistent connection (HTTP keep‑alive, HTTP/2 stream, gRPC channel) all arrive at the same backend instance. The load balancer does not distribute individual requests among backends.
• Traffic distribution by request count or CPU load is not guaranteed. A client that opens many short‑lived connections may, by hash coincidence, send more traffic to one backend than to another. This is normal and not a sign of a misconfiguration.
• Adding a new, healthy instance to the backend pool does not rebalance existing flows. Existing connections remain pinned to their current backends until they close. Only new flows can be routed to the new instance.

Session persistence (source IP affinity). You can configure a load‑balancing rule to use source IP affinity. This changes the hash to consider only source IP (and optionally protocol), so all connections from a given client IP land on the same backend. This can help legacy applications that store session state in local memory, but it reduces distribution granularity and makes failover coarser. Affinity is not a replacement for externalizing session state.

What Azure Load Balancer does not do:

• It cannot route based on URL paths, host headers, cookies, or any application‑layer data.
• It does not terminate TLS or inspect certificates.
• It does not provide a web application firewall (WAF).
• It does not perform content‑based request routing or header manipulation.

For HTTP workloads that need these features, Azure Application Gateway or Azure Front Door is the appropriate service. For simple TCP/UDP distribution where these features are unnecessary, Azure Load Balancer is lighter weight, lower latency, and easier to operate.

Public versus internal load balancers

The choice of frontend IP type defines whether the load balancer accepts traffic from the internet or only from within a private network.

Aspect	Public Load Balancer	Internal Load Balancer
Frontend IP	Public IP address, reachable from the internet	Private IP address from a VNet subnet
Typical clients	Internet users, external APIs, SaaS consumers	Application tiers within Azure, peered VNets, VPN/ExpressRoute‑connected networks
Use cases	Public‑facing TCP/UDP services, game servers, custom‑protocol endpoints	Internal microservices, database proxies, private API endpoints, middle‑tier services
Security considerations	Requires strict NSG rules and DDoS protection; avoid exposing management ports	Relies on network isolation; NSGs and host firewalls still required

Public load balancers are appropriate when clients exist outside Azure. They should be combined with Network Security Groups that allow only the intended ports and source IP ranges. The frontend IP is a public resource, but that does not mean every backend instance should have a public IP. In fact, backends typically remain in private subnets, and the load balancer acts as the sole internet‑facing entry point.

Internal load balancers are the foundation of multi‑tier architectures. A web tier that receives public traffic (via a public load balancer or Application Gateway) can call an application tier through an internal load balancer. The application VMs have no public IPs and are not reachable from the internet, reducing the attack surface. Internal load balancers are also used to expose private APIs, database read replicas, and platform services to other workloads in the same virtual network.

SKU selection and lifecycle

Azure Load Balancer comes in two SKUs: Basic and Standard.

Feature	Basic Load Balancer	Standard Load Balancer
Backend pool scope	Single availability set or Virtual Machine Scale Set	Any VMs, scale sets, or IP addresses in a region
Availability zones	Not supported	Zone‑redundant and zonal frontends
Health probes	TCP, HTTP	TCP, HTTP, HTTPS
Security model	Open by default; NSGs optional	Secure by default; NSGs required
Outbound connectivity	Automatic but not configurable	Explicit outbound rules or NAT Gateway
SLA	None	99.99% (with ≥2 healthy VMs)
Current status	Retired; not available for new deployments	Fully supported for all new workloads

Basic Load Balancer has been retired and is no longer available for new deployments (verify the latest official notice). All new designs must use the Standard SKU. Existing Basic deployments should be migrated; the differences in security posture and outbound connectivity behavior are substantial and can cause service disruption if not accounted for.

The Standard SKU’s secure‑by‑default model means that after creating a load balancer, you must explicitly allow traffic through Network Security Groups on the backend subnet or NICs. Traffic is blocked until you create the necessary rules. This includes allowing health probe traffic from the AzureLoadBalancer service tag.

Health probes: the gate to production traffic

A backend instance is not eligible to receive new flows until it passes its health probe. The probe is a periodic request sent by the load balancer to each backend instance. If the instance fails to respond correctly for a configurable number of consecutive probes (the unhealthy threshold), it is removed from the eligible pool. When the instance starts responding again, it is automatically added back.

Probe types:

• TCP probe – A TCP connection attempt to a specified port. If the three‑way handshake completes, the instance is considered healthy.
• HTTP probe – An HTTP GET request to a configurable path. The instance must return HTTP 200 OK.
• HTTPS probe – Similar to HTTP, but the connection uses TLS. The probe does not validate the server certificate; it only checks that a 200 response is received.

Choosing the right probe depth. A TCP probe on port 22 (SSH) proves that the operating system is alive and the SSH daemon is running, but it says nothing about your application. A TCP probe on the application’s own port (e.g., 8080) is better—it confirms the application is listening—but a deadlocked process may still accept TCP connections and pass the probe. An HTTP probe to a dedicated health endpoint (/healthz) can verify that the application can process a simple request.

However, a health endpoint that checks every downstream dependency—databases, caches, external APIs—can make the probe too brittle. A temporary slowdown in one dependency can cause all backend instances to fail their probes simultaneously, removing the entire service from rotation. The probe should verify only the minimum set of dependencies necessary for the instance to serve traffic.

Common probe failure patterns:

• The probe port is blocked by an NSG, a host firewall, or the application is bound to localhost instead of 0.0.0.0.
• The probe path returns a 404 or 503 because the application’s health endpoint was not deployed or is misconfigured.
• The probe interval or unhealthy threshold is set too aggressively, causing flapping during brief load spikes.

Inbound NAT rules and administrative access

An inbound NAT rule forwards traffic on a specific frontend port to a single backend instance and port. For example, frontend port 50001 can be mapped to the SSH port (22) of a particular VM. This is port forwarding, not load balancing.

Inbound NAT rules are sometimes used for direct administrative access to backend VMs. Exposing SSH or RDP to the internet is risky. Attackers continuously scan for open management ports. If you must use a NAT rule for administration, restrict the source IP address range to trusted networks. Better alternatives include:

• Azure Bastion – a fully managed jump host that provides browser‑based SSH/RDP access without exposing VMs to the internet.
• Just‑In‑Time (JIT) access – Microsoft Defender for Cloud can open management ports on demand for a limited time.
• Private management network – route administrative traffic through a VPN or ExpressRoute connection and keep management ports open only on private IPs.

Outbound connectivity is a separate design problem

A backend VM that receives inbound traffic through a load balancer does not automatically have internet access. Outbound connectivity must be explicitly designed.

When a backend VM initiates a connection to the internet—to download updates, call an external API, or send telemetry—its private source IP must be translated to a public IP. This is Source Network Address Translation (SNAT). The Standard Load Balancer can provide SNAT via outbound rules, but SNAT ports are a finite resource. If many concurrent outbound connections are established, the SNAT port pool can be exhausted, causing outbound connection failures.

The recommended outbound solution for most workloads is Azure NAT Gateway. It provides a larger, dedicated SNAT port allocation per VM, predictable public IP addresses, and better scalability. When you use NAT Gateway, you do not need outbound rules on the load balancer. In fact, using both can lead to asymmetric routing; the NAT Gateway takes precedence for outbound traffic when associated with a subnet.

Key architectural point: An internal load balancer alone cannot provide internet egress. If your backend VMs need outbound access, pair the internal load balancer with a NAT Gateway or another explicit outbound solution.

High availability and failure behaviour

Azure Load Balancer itself is a highly available service. Its data plane is distributed across Azure’s network fabric, and it can survive infrastructure failures without reconfiguration. The load balancer’s SLA (99.99% for Standard SKU with two or more healthy VMs) covers its ability to forward packets, not your application’s end‑to‑end availability.

To build a resilient application:

• Distribute backend instances across multiple Availability Zones. The load balancer’s zone‑redundant frontend (Standard SKU) remains reachable even if a zone fails. If you use a zonal frontend, it becomes unavailable during a zone outage.
• Ensure that you have enough backend capacity in the surviving zones to handle the full traffic load after a zone failure.
• Health probes detect failed instances and remove them from the eligible pool, but detection is not instantaneous. The unhealthy threshold determines how long a probe must fail before the instance is removed. Clients should implement retry logic with exponential back‑off to handle transient failures.
• Existing TCP connections to a failed instance are not migrated to another backend. The client must reconnect. Long‑lived connections, such as gRPC streams or database replication sessions, will be interrupted. Applications must be designed to reconnect.

What about global failover? Azure Load Balancer is a regional service. It does not perform cross‑region traffic distribution on its own. For multi‑region resilience, place a global service in front of regional load balancers: Azure Front Door for HTTP/HTTPS applications, Azure Traffic Manager for DNS‑based steering, or the Cross‑region Load Balancer for TCP/UDP (currently available and subject to its own limitations).

Choosing the right Azure traffic service

Azure offers multiple services that direct traffic. They are not interchangeable, but they are often combined.

Service	Layer	Scope	HTTP aware?	TLS termination	WAF	Typical use
Azure Load Balancer	4 (TCP/UDP)	Regional	No	No	No	High‑performance Layer 4 distribution for TCP/UDP
Azure Application Gateway	7 (HTTP/HTTPS)	Regional	Yes (URL, host, headers)	Yes	Yes	Web application delivery, API gateway, WAF
Azure Front Door	7 (HTTP/HTTPS)	Global	Yes (URL, host, headers)	Yes	Yes	Global web app acceleration and multi‑region failover
Azure Traffic Manager	DNS	Global	No	No	No	DNS‑based endpoint selection and failover
Azure NAT Gateway	3/4	Regional (subnet)	No	No	No	Managed outbound internet connectivity

Decision guide:

• Use Azure Load Balancer when you need a simple, high‑performance Layer 4 distributor for TCP or UDP, and your backend instances handle TLS and HTTP routing.
• Use Application Gateway when you need URL‑based routing, TLS offloading, or WAF for HTTP applications within a region.
• Use Front Door when you need a global HTTP/S entry point with edge caching, TLS, and WAF.
• Use Traffic Manager for DNS‑based global failover, especially when your endpoints are public and you don’t need application‑layer routing.
• Use NAT Gateway when your primary concern is managed, scalable outbound internet access for private VMs.

Security responsibilities

Security for a load‑balanced application requires coordination across several layers.

• Network Security Groups (NSGs) are mandatory with Standard Load Balancer. Create rules on the backend subnet or NIC that allow inbound traffic from clients on the required ports. Also allow health probe traffic from the AzureLoadBalancer service tag.
• Guest OS firewalls (Windows Firewall, iptables) must be configured to accept the same ports. An NSG rule does not override a host‑level block.
• Public exposure should be minimized. If only internal clients need access, use an internal load balancer. For public load balancers, only open the minimum ports and consider DDoS Protection Standard.
• TLS should be terminated on the backend instances or on a Layer 7 service placed in front of the load balancer. The load balancer itself does not decrypt traffic.
• Management ports must not be left open to the internet. Use Bastion, JIT, or a private management network.

warning

A load‑balancing rule makes the traffic path possible, but it does not guarantee reachability. NSGs, route tables, backend listeners, and host firewalls all participate. A missing NSG rule is the most common reason a service is unreachable through the load balancer.

Observability and troubleshooting

When traffic does not flow as expected, work through the evidence layer by layer.

Symptom	Likely causes	What to check
All backends show unhealthy	NSG blocking probe traffic; probe misconfiguration	NSG rules on backend subnet; probe port, protocol, path; backend listener
Backend reachable by IP but not via frontend	NSG blocking client traffic; backend bound to localhost	NSG rules for client IPs; netstat on backend
Some clients cannot connect	Client IP blocked by NSG or firewall	NSG deny logs; effective security rules
Traffic concentrated on few backends	Normal flow hash behavior; few clients with many connections	Instance metrics; consider Layer 7 if request‑level distribution is critical
Long‑lived TCP connections drop during deployment	Instance removed without draining; probe removed instance	Implement graceful shutdown: fail probe, drain connections, then terminate
Outbound connections fail intermittently	SNAT port exhaustion	SNAT metrics; consider migrating to NAT Gateway
Private clients cannot reach internal frontend	Client not in peered VNet; route missing	Network Watcher next hop; VNet peering status

Use Azure Monitor to track health probe status and data path availability. Enable NSG flow logs for detailed traffic analysis. Network Watcher’s connection troubleshoot can test reachability from a source to a destination, helping isolate the blocking layer.

A production scenario: private API tier

Imagine an internal microservice that serves a gRPC API for inference. The service must:

• Accept gRPC traffic on TCP port 9000 only from internal application workloads.
• Run on multiple VMs across three availability zones.
• Terminate TLS on the backend VMs (the load balancer passes TCP transparently).
• Allow outbound internet access for pulling model updates.

Architecture:

• Internal Standard Load Balancer with a private frontend IP 10.20.1.50.
• Backend pool: a Virtual Machine Scale Set with instances in zones 1, 2, and 3.
• Health probe: HTTP on port 8080 path /ready; verifies the gRPC server is running and the model is loaded.
• Load‑balancing rule: TCP port 9000 mapped to backend port 9000.
• Outbound connectivity: Azure NAT Gateway on the backend subnet.
• NSGs: allow inbound TCP 9000 from the application subnets; allow inbound TCP 8080 from AzureLoadBalancer; allow outbound to NAT Gateway.

Traffic flow:

1. Application workloads resolve the service name to 10.20.1.50.
2. They open a gRPC connection to 10.20.1.50:9000.
3. The load balancer selects a healthy backend using the flow hash.
4. The packet arrives at the instance, where the gRPC server handles the request.
5. Return traffic flows back through the load balancer.

Why this design works: The load balancer provides a stable private IP, distributes connections across zones, and the health probe ensures that only ready instances receive traffic. The NAT Gateway provides predictable outbound access without consuming SNAT ports on the load balancer. TLS is handled by the application, so no Layer 7 processing is needed at the load‑balancing layer. If HTTP‑level routing or WAF becomes necessary in the future, an Application Gateway or Front Door could be added in front of the internal load balancer.

Common design mistakes

• Treating Azure Load Balancer as a reverse proxy. It cannot route by URL, inspect headers, or terminate TLS. Use Application Gateway or Front Door for those features.
• Using a health probe that is too shallow or too deep. A TCP probe on the application port may miss a hung process. An HTTP probe that checks all downstream dependencies can cause cascading failures.
• Forgetting NSG rules on the backend subnet. Standard Load Balancer blocks all traffic until you allow it. Remember to open health probe traffic.
• Relying on implicit outbound access. Backend VMs behind a Standard Load Balancer do not have internet access unless you configure outbound rules or attach a NAT Gateway. The old default outbound access behaviour has been retired for new deployments.
• Exposing SSH/RDP to the internet. Use Azure Bastion, JIT, or a private management network instead.
• Assuming traffic will be evenly distributed by request count. Layer 4 hash distribution is connection‑based. Long‑lived connections and client behaviour can cause uneven load. This is normal.
• Deploying all backends in a single availability zone. A zone‑redundant frontend does not protect you if all backend instances are in the zone that failed. Distribute backends.
• Assuming existing connections fail over gracefully. Established connections are not migrated. Applications must reconnect.

Summary

Azure Load Balancer provides reliable, high‑performance Layer 4 traffic distribution for TCP and UDP workloads. It offers a stable frontend, health‑based backend selection, and tight integration with Azure virtual networks. Its simplicity is its strength: it does one thing well and leaves application‑layer concerns to the services and code that are designed for them.

Choose Azure Load Balancer when you need a regional, transport‑layer load balancer for protocols that don’t require HTTP awareness. Combine it with Application Gateway or Front Door when you need URL routing, TLS offloading, or WAF. Always design outbound connectivity explicitly—preferably with NAT Gateway—and validate your health probe, NSG, and host firewall configuration end to end. The load balancer is a building block, not a turnkey solution; it works best when the application, network policy, and deployment practices are designed with its behaviour in mind.