Friday, November 19, 2021

How do AKS and AKS on Azure Stack HCI compare?

Hey! I’m Abhilasha, a PM on the AKS-HCI team. In this blog, I wanted to share some of the similarities and differences between AKS-HCI and AKS.


AKS on Azure and has been generally available for years. AKS on Azure Stack HCI is an on-premises implementation of AKS, it runs in customer environments on customer managed hardware. We basically want to achieve world peace and support running AKS everywhere outside of Azure.


Feature Set



Kubernetes Management Cluster

(lifecycle operations)


Who manages the management cluster?

Split management – mostly managed by the service with some admin config (networking, required certs, etc).

Managed by AKS; hidden from your view

Where is the management cluster located?

In your datacenter. Currently, we support:

Azure Stack HCI 21H2

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

Azure cloud; hidden from your view

Well, who manages the hardware where this is running?

All physical hardware things – storage, network, servers, etc are managed by you

Storage, network, etc are created and managed for you when you create an AKS cluster; visible to you

How do you update the management cluster?

PS commands initiated by you (Update-Akshci)

Windows Admin Center GUI initiated by you

Note: Workload cluster update is covered in the below section.

Azure managed updates; hidden from customer view




Kubernetes Target Cluster

(lifecycle operations)    



Who manages the cluster?

Managed by you

Managed by you

Where is the cluster located?

In your datacenter alongside your AKS-HCI management cluster

Azure cloud

K8s cluster lifecycle management (create, scale, update and delete clusters)

PowerShell (PS)

Windows Admin Center (WAC)


Az PowerShell

Azure Portal

ARM templates


Pulumi and many more!

Can you use kubectl and other open-source Kubernetes tools?

Yes! AKS-HCI is a Cloud Native Computing Foundation (CNCF) certified Kubernetes distribution

Yes! AKS is a Cloud Native Computing Foundation (CNCF) certified Kubernetes distribution

Workload cluster updates

K8s version upgrade through PowerShell or WAC. Initiated by you.

node OS image update initiated by you;

Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.

Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions;

OS image patch upgrade;

Automatic upgrades;

Planned maintenance windows;


Kubernetes versions

~same as AKS:

1.19.11, 1.19.13, 1.20.7, 1.20.9 (default), 1.21.1, 1.21.2 (as of 11/5/2021)

1.19.11, 1.19.13, 1.20.7, 1.20.9 (default), 1.21.1, 1.21.2, 1.22.1, 1.22.2 (as of 11/5/2021)

Can you start/stop K8s clusters to save costs?

Yes, by stopping the underlying failover cluster

Yes, by using Az CLI commands or Azure Portal




Node Pools



Do you support running Linux and Windows node pools in the same cluster?


Linux nodes: CBL-Mariner

Windows nodes: Windows Server 2019 Datacenter


Linux nodes: Ubuntu 18.04

Windows nodes: Windows Server 2019 Datacenter

What’s your container runtime?

Linux nodes: containerd

Windows nodes: Docker

Linux nodes: containerd

Windows nodes: Docker (default), containerd (preview)

Can you scale node pools?


No auto-scaler yet


You can auto scale node pools in AKS

Scale down a cluster without deleting the K8s nodes


What about virtual nodes?

No, we do not have support for virtual nodes yet

Yes! Virtual nodes are only supported with Linux pods and nodes. This is based on the open-source project Virtual Kubelet.

Can you upgrade a node pool?

We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level.

You can perform node pool specific upgrades in an AKS cluster.







Who creates and manages the networks?

All networks (for both the management cluster and target K8s clusters) are created and managed by you

By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters

What type of network options are supported?

Support logical networks:

DHCP networks with/without VLAN ID

Static IP networks with/without VLAN ID

We do not have SDN support yet


Support Azure virtual networks in 2 network models:

Kubenet (default) – Networks created alongside AKS

Azure-CNI – use existing virtual networks and configurations

Load balancer

HAProxy – runs in a separate VM in the target K8s cluster

kubeVIP – runs as a K8s service in the control plane K8s node

Load balancers are always given Static IP addresses to ensure application and K8s cluster availability

Azure load balancer – Basic SKU or Standard SKU

Can also use internal load balancer

By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service

CNI/Network plugin


Calico (default)

Note: Network policies are covered in the Security and Authentication section.

Azure CNI


Note: Network policies are covered in the Security and Authentication section.

Load balancers

Default load balancer for services is HA Proxy, however, you can also use your own cluster load balancer like MetalLB


Ingress controllers

You can add ingress controllers like Nginx, Traefik etc. using common Kubernetes guidance.

Default ingress control in Azure is the Azure load balancer. You can deploy additional ingress controls like Nginx.

Egress controls

Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies.

Default setting is to not allow traffic from pod-to-pod. You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.

Egress types

Load Balancer is the only built-in egress type in AKS-HCI

Azure load balancer, managed NAT gateway and user defined routes are the supported egress types

Customize CoreDNS









Where is the storage provisioned?


Azure Storage. You can also deploy any custom storage class.

What types of persistent volumes are supported?

Read Write Once

Read Write Many

Read Write Once

Read Write Many

Do the storage drivers support Container Storage Interface (CSI)?



Is dynamic provisioning supported?



Is volume resizing supported?



Are volume snapshots supported?






Security and Authentication



How do you access your Kubernetes cluster?

Certificate based kubeconfig (default)

AD based kubeconfig

Certificate based kubeconfig (default)

AAD based kubeconfig

Azure Policy

Yes! Through Azure Arc enabled K8s (free for AKS-HCI customers)


Azure Security

Yes! Through Azure Arc enabled K8s
Azure Security Center
Azure Defender


Azure Security Center
Azure Defender

Network Policies

Yes! We support Calico network policies

Yes! We support Calico and Azure CNI network policies

Limit access to API server

Yes! By using VIP pools.

Yes! By using the “-api-server-authorized-ip-ranges” parameter and private clusters.

Cluster RBAC

Yes, with added setup (RBAC manager)

Yes! Azure RBAC, AAD

Certificate rotation and secrets encryption



Support for private cluster

Not supported yet

Yes! You can create private AKS clusters

Secrets store CSI driver



Support for disk encryption

Yes via bitlocker

Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys.

Hosts and locally attached disks can also be encrypted with encryption at host.

Support for node-to-node traffic encryption

Can be enabled via Calico Wireguard

Can be enabled via Calico Wireguard

Pod identity support

Not yet

Windows containers - gMSA

Pod identity is in public preview

Windows containers - gMSA




Monitoring and Logging



Support for Azure Monitor

Yes! Through Azure Arc enabled K8s


Collect logs

Yes! Through PS (Get-AksHciLogs) and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected.

Yes! Through Azure Portal, Az CLI, etc

SSH into K8s nodes to troubleshoot

Yes! You can SSH into Linux and Windows nodes

Yes! You can SSH into Linux and Windows nodes. However, we recommend using cloud native logging tools like AKS periscope or inspector gadget.

On-premises logging and monitoring tools

Yes! You can connect to Prometheus & Grafana using AKS-HCI PS commands (Install-AksHciMonitoring)

You can also setup EFK on your AKS-HCI clusters.

Not applicable. AKS is entirely based in Azure!




Develop and run applications



Azure App service

Yes! Through Azure Arc enabled K8s


Azure Functions

Yes! Through Azure Arc enabled K8s


Azure Logic Apps

Yes! Through Azure Arc enabled K8s


Install applications using Helm






Bridge to Kubernetes



Open Liberty/WebSphere Liberty



Run Spark jobs

Pending validation


Use GPUs

In preview



Pending validation




Pending validation


Service Mesh

Yes! We support Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.

Open Service Mesh



Consul Connect





Azure DevOps through Azure Arc enabled K8s. GitOps through Azure Arc enabled Kubernetes is free for AKS-HCI customers.



Azure DevOps

GitHub Actions





Product Pricing



Product pricing

Pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.

Unlimited free clusters, pay for on-demand compute of the worker nodes.


Azure Support

AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support.

AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.


We do not offer SLAs since AKS-HCI runs in your environment.

Paid uptime SLA clusters for production with fixed cost on the API + worker node compute cost.




Posted at