Wednesday, April 20, 2022

Protecting applications on private Azure Kubernetes Service clusters with Astra Control Service

 

Abstract

Introduction

Scenario

Create a private AKS cluster

Access the private AKS cluster

Enable the private AKS cluster to be managed from ACS

Discover and manage the private AKS cluster using ACS

Summary

Recommended reading/references

 

Abstract 

This technical blog explains how NetApp® Astra™ Control Service (ACS) can be used to protect applications on private Azure Kubernetes Service (AKS) clusters. Private AKS clusters restrict Kubernetes network traffic to an internal network. Familiarity with AKS clusters and general Azure principles is recommended.

 

Co-authors: Sayan Saha and Balasubramanian Ramesh Babu, NetApp

 

Introduction

Security is a growing concern for customers. For organizations that would like to steer clear of public IPs, Azure offers a convenient way of spinning up AKS clusters that are private. A standard AKS cluster provides a public IP address for the API server. A private AKS cluster, on the other hand, will use a private IP address for its API server endpoint. This protects the API server from being exposed to the outside world. In both cases, nodes present in the cluster will use a private network for assigning IPs to the worker nodes.

 

Scenario

This blog covers the following:

 

  • Creating a private AKS cluster: We will examine the options available to create private AKS clusters. Since the cluster is private, accessing it is not the same as a standard AKS cluster.
  • Accessing the private AKS cluster: There are multiple options available to configure access; this blog uses a bastion host.
  • Enabling the private AKS cluster to be managed from Astra Control Service: Connecting private AKS clusters to Astra Control Service is achieved by deploying the astra-connector-operator.
  • Discovering and managing the private AKS cluster using Astra Control Service: Once the required pieces are in place, the final step is to manage the cluster with Astra Control.

 

Create a private AKS cluster

Reference: https://docs.microsoft.com/azure/aks/private-clusters

 

Deploying a private AKS cluster requires using the Azure CLI. There are several options available to customize a private AKS cluster. For the sake of simplicity, this blog creates a private AKS cluster with basic networking, using the Kubenet plugin. If desired, clusters can be created using the Azure network plugin. This will allow clusters to leverage an existing Azure Vnet and subnets.

 

# Create a resource group and a private AKS cluster
az group create -l eastus -n rg-brameshb
az aks create -n private-cluster-eastus -g rg-brameshb --load-balancer-sku standard --enable-private-cluster

 

Once the cluster is deployed and is in the “Succeeded” status, additional details about the cluster can be obtained.

GeertVanTeylingen_0-1650383083701.png

 

As can be seen from the output, the AKS cluster is a resource present in `rg-brameshb` RG. Nodes present in the AKS cluster, however, are placed in a separate RG. `nodeResourceGroup` indicates the name of this RG. It can be obtained from the JSON output of the AKS cluster.

GeertVanTeylingen_1-1650383083716.png

 

 `MC_rg-brameshb_private-cluster-eastus_eastus` contains the Azure resources that make up `private-cluster-eastus`.

 

Access the private AKS cluster

Since the API server is not publicly routable, additional steps must be taken. Microsoft recommends choosing an approach from the following:

  1. An Azure VM that has access to the VNet used by the private AKS cluster.
  2. An Azure VM that is in a different VNet and is peered to the private AKS cluster VNet.
  3. VPN connection.
  4. AKS command invoke.
  5. Private endpoint.

 

This blog uses (1). A bastion host is created in the `MC_rg-brameshb_private-cluster-eastus_eastus` RG.

GeertVanTeylingen_2-1650383083734.png

 

Enable the private AKS cluster to be managed from ACS

To manage private AKS clusters with Astra Control Service, the Astra Connector Operator is used. The Astra Connector Operator needs to be deployed on the AKS cluster to enable communications with NetApp Astra Control Service. This step requires using the manifests provided here. Proceed to login to the bastion host and configure access to the AKS cluster.

 

#Access bastionhost
ssh -i brameshb-key.pem brameshb@20.232.168.67
#Login to Azure and access private AKS cluster
brameshb@bastionhost:~$ az login
brameshb@bastionhost:~$ az account set --subscription <subscription-id>
brameshb@bastionhost:~$ az aks get-credentials --resource-group rg-brameshb --name private-cluster-eastus
#Download kubectl
brameshb@bastionhost:~$ curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
brameshb@bastionhost:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
brameshb@bastionhost:~$ kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-25392440-vmss000000   Ready    agent   22h   v1.21.9
aks-nodepool1-25392440-vmss000001   Ready    agent   22h   v1.21.9
aks-nodepool1-25392440-vmss000002   Ready    agent   22h   v1.21.9

 

Deploy the Astra Connector operator.

brameshb@bastionhost:~$ kubectl create ns astra-connector-operator
brameshb@bastionhost:~$ git clone https://github.com/NetApp/astra-connector-operator.git
brameshb@bastionhost:~$ cd astra-connector-operator
#Apply operator manifest
brameshb@bastionhost:~$ kubectl apply -f astraconnector_operator.yaml -n astra-connector-operator
#Check if astra-connector-operator is up and running
brameshb@bastionhost:~$ kubectl get pods -n astra-connector-operator
NAME                                          READY   STATUS    RESTARTS   AGE
operator-controller-manager-bbf65d7cd-cpcfb   2/2     Running   0          4h18m

 

The next step is to create an AstraConnector object. It provides connectivity information required by Astra to discover the AKS cluster it is created on. To do this, an API token must be generated first:

generate-api-token.gif

 

Once the API token is generated, it is plugged into the AstraConnector spec. The name of the private cluster [clusterName] and Astra account ID [accountID] are also required. A private image registry [imageRegistry] can be used if needed. A sample config is available in the config/samples directory. For a complete description of the parameters in use, refer to the table provided here.

 

#Create AstraConnector to discover private AKS cluster
$ cat config/samples/astraconnector_v1.yaml
apiVersion: netapp.astraconnector.com/v1
kind: AstraConnector
metadata:
  name: astra-connector
spec:
  natssync-client:
    image: natssync-client:0.9.202202170408
    cloud-bridge-url: https://astra.netapp.io
  nats:
    image: nats:2.6.1-alpine3.14
  httpproxy-client:
    image: httpproxylet:0.9.202202170408
  echo-client:
    image: echo-proxylet:0.9.202202170408
  imageRegistry:
    name: theotw
  astra:
    token: <token>
    clusterName: <aks-cluster-name>
    accountId: <astra-account-id>
    acceptEULA: yes

 

Create the AstraConnector. `kubectl get astraconnector` informs if the private cluster is registered with Astra. If running into issues, examine logs from the `operator-controller-manager` pod in the astra-connector-operator namespace.

 

# Create a namespace and apply AstraConnector config
$ kubectl create ns astra-connector
$ kubectl apply -f config/samples/astraconnector_v1.yaml -n astra-connector
# Observe its status
$ kubectl get astraconnector -n astra-connector
NAME              REGISTERED   ASTRACONNECTORID
astra-connector   true         a740d092-d272-4b88-a605-a396d11373a4
# Examine logs from operator-controller-manager
$ kubectl logs operator-controller-manager-bbf65d7cd-cpcfb -n astra-connector-operator -c manager
.
.
.
1.6496804658713624e+09 INFO    controller.astraconnector      Registering cluster with Astra   {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680465871367e+09  INFO    controller.astraconnector      CloudBridgeURL for Astra Host    {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "CloudBridgeURL": "https://preview.astra.netapp.io"}
1.6496804658713708e+09 INFO    controller.astraconnector      Checking for a valid SA credential for cloud   {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudType": "Azure"}
1.6496804659914987e+09 INFO    controller.astraconnector      Found a valid SA credential for cloud      {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudType": "Azure", "credName": "brameshb-azure-sp"}
1.649680465991535e+09  INFO    controller.astraconnector      Fetching cloud ID        {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680466105789e+09  INFO    controller.astraconnector      Found cloud ID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudID": "883191fe-0cd1-499d-b2da-ce9104d49b74"}
1.6496804661058552e+09 INFO    controller.astraconnector      Finding cluster ID        {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680466224342e+09  INFO    controller.astraconnector      Found cluster ID        {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "clusterId": "dae554c0-0c2e-49f9-b26c-841f3d89936d"}
1.6496804663686578e+09 INFO    controller.astraconnector      successfully registered astraConnectorID       {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "astraConnectorID": "v1:a740d092-d272-4b88-a605-a396d11373a4"}
1.649680466368695e+09  INFO    controller.astraconnector      Registered cluster with Astra   {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}

 

To provide persistent storage for applications, Azure NetApp Files is recommended. An NetApp account is created in the resource group (rg-brameshb) that contains the private AKS cluster. In addition, virtual networks used by Azure NetApp Files and the AKS cluster must be peered.

 

ANF config.gif

 

Discover and manage the private AKS cluster using ACS

With all required dependencies in place, the private cluster can be discovered by Astra. Log in to the Astra dashboard and add the cluster. This requires an Azure credential. Refer to the Astra docs to understand how an Azure Service Principal can be created.

 

Adding cluster.gif

 

Applications can be deployed with persistent storage provided by Azure NetApp Files, Azure Disk, and/or Azure File. Astra Control protects application workloads. Protection policies can be created that specify backup schedules. To learn more about how this works, take a look at Disaster Recovery of AKS workloads with Astra Control Service and Azure NetApp Files.

 

Summary

This article explained the management of private AKS clusters with Astra Control, using the Astra Connector operator.

 

Recommended reading/references

 

  1. To learn about custom execution hooks and quiescing applications, take a look at Protecting MongoDB on AKS/ANF with Astra Control Service using custom execution hooks.
  2.  Astra Control Service Documentation.

 

Posted at https://sl.advdat.com/3k1Mk6Uhttps://sl.advdat.com/3k1Mk6U
Labels: ,