Anthos Config Connector and Redis security

A short disclaimer. I am writing it in the middle of March 2022 and it is possible that when you read the blog the information published here is not relevant anymore. Cloud products are evolving very fast.

I write the post to share some observations and potential issues you might have with deploying GCP Memorystore for Redis instances through Anthos Config Connector (ACC) controller. If you are not familiar with ACCI, I strongly recommend reading at least a high level overview of the product. In essence this is a Kubernetes addon which allows you to automatically deploy and manage GCP services by applying a manifest file (YAML or Helm chart) to a Kubernetes cluster with the ACC controller. It allows you to use the Kubernetes cluster as a deployment tool for GCP resources in your organization. This is a really interesting approach and might transform your environment in the cloud. But it implies some challenges around security which I am going to discuss in the blog.

First, let me illustrate how the workflow of a deployment works when we use the ACC. Here is a simple diagram showing different stages of the deployment.

As we can see the deployment is going through different stages:

  • Create manifest for the Redis instance
  • Submit it to the repository
  • The image is picked up by either your CD tool or just applied by kubectl
  • The resource definition (CRD) tells that it is ACC resource
  • ACC is picking up the definition and creating the resource.

That sounds great and in combination with other tools allows you to deploy almost any kind of GCP resource as part of your Kubernetes application.

Let’s return back to the Memorystore for Redis and see what options we have for ACC Redis Instance Resource Definition. Here is a sample manifest for ACC to deploy a Redis Instance:

kind: RedisInstance
  name: redisstd-gleb-sandbox-02
  annotations: gleb-sandbox-project
    purpose: "sandbox"
  authEnabled: true
  # authString:  parameter exists but doesn't do any difference
    external: "projects/gleb-sandbox/global/networks/gleb-sandbox-net-01"
  displayName: redisstd-gleb-sandbox-01
  region: us-central1
  redisVersion: "REDIS_6_X"
  region: us-central1
  replicaCount: 1
  reservedIpRange: "redis-private-ip-03"
  transitEncryptionMode: "SERVER_AUTHENTICATION"
  memorySizeGb: 16

All parameters and options are described in the reference documentation. Today we will pay attention to some security properties and start from the connection layer.

The TLS can be enabled by parameter “transitEncryptionMode” and it is either on or off. The value “SERVER_AUTHENTICATION” turns it on and the value “DISABLED” turns it off. The certificate is self-signed by Google, has 2,048 bits length and valid for 10 years. It will be automatically rotated (issued a new one in addition to the existing) in 5 years.

This is probably quite sufficient for most businesses which don’t go through vigorous security controls and are not subjected by financial or PII standards. But for the latter it might be a challenge. The fact that the certificate is self-signed by Google is the first challenge you might face. It is not possible to use a Customer Managed Certificate Authority signed by the company or any other authority. The second problem is the rotation period. In many companies the max lifetime for a key is limited by two years by standards. In the GCP Redis you cannot modify the rotation interval. One more thing is about visibility for the public certificate but I am going to discuss it a bit later.

The second security part for your Redis Connection is the Redis AUTH string. It is defined by the parameter “authEnabled” and it is either “true” or “false”. In the ACC Resource Definition you also have the “authString” parameter which you can supposedly define but it doesn’t really do anything. My assumption is that it is supposed to be an output parameter for a synchronized type of deployment mechanism like Terraform. The ACC deployments though are asynchronous by nature. If the manifest is submitted and if it doesn’t have any syntax errors it will be accepted by ACC. When it happens you are not getting any output information except an error in the events if it is not able to deploy it successfully.

The asynchronous nature of the ACC deployment required that the AUTH string was read from the Redis configuration and placed to a secure solution for future use by application. If it is done manually it means the person who get the string will know it and in many organizations it is unacceptable from security point of view. If we use an automated approach we need a solution to be able to connect to the Redis instance using a Google Service Account, get the string and put it to the secure storage accessible by the application. It means we need a staged deployment when the Redis is deployed first, then the Certificate and the AUTH string should be somehow put to the secret store and only after that the application can be deployed to be able to use those values to securely connect to the Redis instance.

And one thing everybody has to keep in mind is access to the kubectl tool or to the cluster itself since the AUTH string and the Certificate for the instance are visible clearly when you run kubectl get or describe commands.

user@26c17a4143c6:~/gcp-functions-go/backupinst# kubectl get redisinstance -o jsonpath="{.items[*].spec.authString}{'\n'}"
user@26c17a4143c6:~/gcp-functions-go/backupinst# kubectl get redisinstance -o jsonpath="{.items[*].status.serverCaCerts[*].cert}{'\n'}"

It means that direct access to the GKE cluster should be prohibited and guarded as access to any secret store solution. Knowing the AUTH string, IP and the Certificate anybody with access to the network can read the data from the instance.

Here is short summary:

  • If you use Anthos Config Connector to deploy Redis you need to restrict access to the cluster which deploys the instance to prevent access to the Redis AUTH string, Certificate and IP address.
  • The AUTH, IP and Certificate can be retrieved only after successful deployment and it requires a tool which can verify the resource, get the resource and check it out to a secret solution for the application usage.
  • The deployment is asynchronous and the application deployment should be delayed or staged after successful deployment of the Redis instance and retrieval of the necessary connection information.

Leave a Reply

Your email address will not be published. Required fields are marked *