When I was doing the first round of AKS cluster upgrades at my current client, I noticed we were running a lot of pods with only 1 replica. I always try to lift my clients to the next level by leveraging Cloud Native technologies as much as possible. I’m therefore starting a project to always run applications with multiple replicas. However, running multiple replicas is not the only necessary improvement here. Even though a pod is running with multiple replicas, that does not mean that Kubernetes will always keep them alive. When you do an AKS cluster upgrade, nodes are drained one by one and the pods are moved to a node with the higher k8s version. Technically, when draining a node, Kubernetes could kill both of the pods at the same time if they are running on the same node. ...

Been working on a nice project lately. My current client has chosen to run Grafana Cloud and the instance was running in West US, which was leading to slower performance. I took up the project of provisioning a new instance in West Europe. Although I would have preferred to host Grafana ourselves or use Azure Managed Grafana, this is what the client has chosen so I that is what I’ll have to work with. ...

Nice article that explains why blogging is a satisfying activity. I’ve written the exact same reasons of why I blog on my about page where I wrote that it is very satisfying to scroll through your own feed and see what you have been up to. My blog is a time machine. https://brainbaking.com/post/2023/11/on-writing-for-yourself-in-public/ Links: 202311150711

Great news! KEDA finally went GA for AKS. https://learn.microsoft.com/en-us/azure/aks/keda-about Links: 202311081711

I use this bash code to get secrets from Azure key vaults using the Azure CLI. Make sure to set your subscription with az account set -s 12321424 kvname=kv-vo-rea-prd-weu-001 secretname=worker-ad-client-secret az keyvault secret show --vault-name $kvname --name $secretname --query value -o tsv Links: 202310312010 [[Azure]]

Chapter 11 Virtual WAN Easy Mode Hub-Spoke incorporates multiple hybrid connections (vpn/expressroute) incorporates multiple VNets Dynamically handles routing easily deploys NVA/Firewall Use case You technically don’t need to use it for anything, you can do the same with a normal hub-spoke built using VNets. However, when you do, you’re in charge of: add/update routes network gateway connections integration with virtual appliances / firewalls Virtual WAN makes these tasks easier. Components Virtual Wan ...

Upgrading clusters with databases When upgrading this type of cluster it is better to upgrade the control plane first and then upgrade the node pools one by one. The EDB operator prevents you to drain nodes with databases on them with PodDisruptionBudgets. In order to be able to run the AKS upgrade you will need to set the following: k cnp maintenance set --all-namespaces --reusePVC cnp is a kubectl plugin which you need to install to manage the EDB operator --reusePVC can only be used if all the nodes are in the same availability zone use this command to list all the zones: kubectl describe nodes | grep -e "Name:" -e "topology.kubernetes.io/zone" When the upgrade is complete, run k cnp maintenance set --all-namespaces ...

You can back up your AKS clusters using Azure Backup in preview! Preferably you have your clusters stateless and you can redeploy everything from code when sh*t hits the fan. However, I can think of a few enterprise use cases that will be relevant for this new feature. https://learn.microsoft.com/en-us/azure/backup/azure-kubernetes-service-cluster-backup Links: 202310130610 [[AKS]] [[Kubernetes]] [[Azure]]

This week I’ve been tackling an issue at work with akv2k8s. We are running CloudnativePG, similar to EnterpriseDB. After upgrading akv2k8s to v2.5.0 our database pods were not coming up anymore due to an error with the initcontainer: Error: container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root (pod: "vcs-pooler-rw-858bf7c954-vjzr4_vcs(e19bfa4c-26d8-4a6d-b4ad-bba8b52c01e6)", container: bootstrap-controller) Pods have a security context where you specify how the containers in the pod should be run. ...

Learned a new trick to force a new cert from letsencrypt. Had this error message: The certificate request has failed to complete and will be retried: Failed to wait for order resource "vcs-secret-j8gnj-3121219202" to become ready: order is in "errored" state: Failed to create Order: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: sadfsadf.com, retry after 2023-10-12T18:54:27Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/ I asked for too many certificates during a 168 hour period. However, there is a really easy fix. Just add a new subdomain and LetsEncrypt will treat it as an entirely new certificate! ...