How I did not create a Kubernetes Cluster

Some background

On the last days we try to run Cassandra working on a Kubernetes Cluster, it will be one of the keys of our infrastructure so it’s a good time for start to play with it.

How it’s written in the official page, “ Cassandra is the right choice when you need scalability and high availability without compromising performance. ”  and what is high availability if we’re just serving everything from a single country?

So we just have an objective

  • Deploy a Multi-Region Cassandra’s cluster in Azure.

At first, it looks like we don’t going to have any trouble with this…
Azure let us to launch some AKS easily (we used Terraform for make things quicker) and there’s a Helm Chart available in the incubator (the incubator is just like a place for official beta charts).

Once the cluster in running, is time to install the chart, it just need a single command if you want to go direct to the point but let’s take a look to the file ‘values.yaml’, here we have all the different variables that our chart will read in order to configure Cassandra, most of the time it will have enough for our purpose, but sometimes you need to modify the chart itself if you want something more special.
Here are some of the variables that we tweak for our Cassandra.

resources:

    requests:

        memory: 5Gi

        cpu: 1.5

    limits:

        memory: 6Gi

        cpu: 1.5

{…}

cluster_size: 1

seed_size: 1

{…}

persistence:

    enabled: true

    storageClass: “slow”

    accessMode: ReadWriteOnce

    size: 2Gi

Our cluster is just for development purpose so let’s put the resources at minimum.

We want persistence of our stored data so we need to enable it and give him a persistent volume, in this case we will use a managed azure-disk:

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

    name: slow

    labels:

kubernetes.io/cluster-service: ‘true’

provisioner: kubernetes.io/azure-disk

parameters:

    skuName: Standard_LRS

    kind: managed

It will be created with $ kubectl create -f AzureDisk.yaml.

Once we have our disk and our values ready, we add the incubator repo to our helm and launch the chart with the desired valuesyaml.

$ helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/

$ helm install --name "cassandra" -f values.yaml incubator/cassandra

Here comes the challenges

Everything worked as expected but now we want to go a step forward, we want Cassandra splitted into two different clusters, in different regions, and here our problems started.

To run another cluster it was easy, we just change our region in the terraform script, we choose Canada Central & Canada East cause they’re paired regions and we expected to get some value of that.

Our first pitfall is… how I can tell to Cassandra that we have another cluster and it must contact to him? Well, for that we have the Seed nodes, if we can launch Cassandra with some seed nodes from the other region, the gossip (https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureGossipAbout_c.html) protocol will contact to the nodes and everyone will be happy but… how I could get those node IP’s from one region and share it with the other one if everything is inside different K8 clusters?

VPN to the rescue (or not)

Paired regions looks like don’t have anything like share a network so we have two options, one that is the worst idea ever, and ‘the correct one’.

  • The first (bad) option will be to use nginx-ingress controller to expose all seed nodes and then just add them to the cluster list of seeds.
  • The second one is to use an VPN.

Cassandra Cluster

Configure an VPN in Azure is not the most straightforward thing in the world by far… one of the things that we must do to achieve this is to define a custom VNet for the AKS in order to avoid IP overlap, this stuff is available in ACS but not for AKS (we can follow a discussion about this here https://github.com/Azure/AKS/issues/27).

Sadly, here we found a dead end and until Azure establish in his roadmap a way to achieve this.

Another way to deploy Multi-Region Cassandra in Azure is following the official documentation https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/cassandra-nodejs but it does nothing with Kubernetes.

In conclusion

I must admit that I started all this cluster stuff expecting that everything will work like a charm…

I expected the same kind of options in AKS than in ACS and I should be more cautious when I saw the cassandra chart was not stable yet (for that was in incubator). So once I had everything suited up and only one step is missing, I found the problem with the VNet and I can not get rid of the feeling that I worked for nothing… which is not true. I was able to learn how to use and modify Helm charts, deploy AKS with terraform and learnt about Cassandra internal guts but in the end, I have no product, so next time the best way, will be to read all stuff about the topic that I want to cover and check how far I can go before start with it, be careful with preview software!

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.