A short disclaimer. I am writing it in the middle of March 2022 and it is possible that when you read the blog the information published here is not relevant anymore. Cloud products are evolving very fast.
I write the post to share some observations and potential issues you might have with deploying GCP Memorystore for Redis instances through Anthos Config Connector (ACC) controller. If you are not familiar with ACCI, I strongly recommend reading at least a high level overview of the product. In essence this is a Kubernetes addon which allows you to automatically deploy and manage GCP services by applying a manifest file (YAML or Helm chart) to a Kubernetes cluster with the ACC controller. It allows you to use the Kubernetes cluster as a deployment tool for GCP resources in your organization. This is a really interesting approach and might transform your environment in the cloud. But it implies some challenges around security which I am going to discuss in the blog.
First, let me illustrate how the workflow of a deployment works when we use the ACC. Here is a simple diagram showing different stages of the deployment.
As we can see the deployment is going through different stages:
- Create manifest for the Redis instance
- Submit it to the repository
- The image is picked up by either your CD tool or just applied by kubectl
- The resource definition (CRD) tells that it is ACC resource
- ACC is picking up the definition and creating the resource.
That sounds great and in combination with other tools allows you to deploy almost any kind of GCP resource as part of your Kubernetes application.
Let’s return back to the Memorystore for Redis and see what options we have for ACC Redis Instance Resource Definition. Here is a sample manifest for ACC to deploy a Redis Instance:
apiVersion: redis.cnrm.cloud.google.com/v1beta1 kind: RedisInstance metadata: name: redisstd-gleb-sandbox-02 annotations: cnrm.cloud.google.com/project-id: gleb-sandbox-project labels: purpose: "sandbox" spec: authEnabled: true # authString: parameter exists but doesn't do any difference authorizedNetworkRef: external: "projects/gleb-sandbox/global/networks/gleb-sandbox-net-01" connectMode: "PRIVATE_SERVICE_ACCESS" displayName: redisstd-gleb-sandbox-01 region: us-central1 readReplicasMode: READ_REPLICAS_ENABLED redisVersion: "REDIS_6_X" region: us-central1 replicaCount: 1 reservedIpRange: "redis-private-ip-03" tier: STANDARD_HA transitEncryptionMode: "SERVER_AUTHENTICATION" memorySizeGb: 16
All parameters and options are described in the reference documentation. Today we will pay attention to some security properties and start from the connection layer.
The TLS can be enabled by parameter “transitEncryptionMode” and it is either on or off. The value “SERVER_AUTHENTICATION” turns it on and the value “DISABLED” turns it off. The certificate is self-signed by Google, has 2,048 bits length and valid for 10 years. It will be automatically rotated (issued a new one in addition to the existing) in 5 years.
This is probably quite sufficient for most businesses which don’t go through vigorous security controls and are not subjected by financial or PII standards. But for the latter it might be a challenge. The fact that the certificate is self-signed by Google is the first challenge you might face. It is not possible to use a Customer Managed Certificate Authority signed by the company or any other authority. The second problem is the rotation period. In many companies the max lifetime for a key is limited by two years by standards. In the GCP Redis you cannot modify the rotation interval. One more thing is about visibility for the public certificate but I am going to discuss it a bit later.
The second security part for your Redis Connection is the Redis AUTH string. It is defined by the parameter “authEnabled” and it is either “true” or “false”. In the ACC Resource Definition you also have the “authString” parameter which you can supposedly define but it doesn’t really do anything. My assumption is that it is supposed to be an output parameter for a synchronized type of deployment mechanism like Terraform. The ACC deployments though are asynchronous by nature. If the manifest is submitted and if it doesn’t have any syntax errors it will be accepted by ACC. When it happens you are not getting any output information except an error in the events if it is not able to deploy it successfully.
The asynchronous nature of the ACC deployment required that the AUTH string was read from the Redis configuration and placed to a secure solution for future use by application. If it is done manually it means the person who get the string will know it and in many organizations it is unacceptable from security point of view. If we use an automated approach we need a solution to be able to connect to the Redis instance using a Google Service Account, get the string and put it to the secure storage accessible by the application. It means we need a staged deployment when the Redis is deployed first, then the Certificate and the AUTH string should be somehow put to the secret store and only after that the application can be deployed to be able to use those values to securely connect to the Redis instance.
And one thing everybody has to keep in mind is access to the kubectl tool or to the cluster itself since the AUTH string and the Certificate for the instance are visible clearly when you run kubectl get or describe commands.
user@26c17a4143c6:~/gcp-functions-go/backupinst# kubectl get redisinstance -o jsonpath="{.items[*].spec.authString}{'\n'}" d8e56eba-1d8b-4c3d-80f5-0d0eea308660 user@26c17a4143c6:~/gcp-functions-go/backupinst# kubectl get redisinstance -o jsonpath="{.items[*].status.serverCaCerts[*].cert}{'\n'}" -----BEGIN CERTIFICATE----- MIIDnTCCAoWgAwIBAgIBADANBgkqhkiG9w0BAQsFADCBhTEtMCsGA1UELhMkMWQ0 YzgzNzItMTMzMS00MWQyLTkzMjgtNzViZTE5MjkwMTE4MTEwLwYDVQQDEyhHb29n bGUgQ2xvdWQgTWVtb3J5c3RvcmUgUmVkaXMgU2VydmVyIENBMRQwEgYDVQQKEwtH b29nbGUsIEluYzELMAkGA1UEBhMCVVMwHhcNMjIwMzIxMTQzMzU3WhcNMzIwMzE4 MTQzNDU3WjCBhTEtMCsGA1UELhMkMWQ0YzgzNzItMTMzMS00MWQyLTkzMjgtNzVi ZTE5MjkwMTE4MTEwLwYDVQQDEyhHb29nbGUgQ2xvdWQgTWVtb3J5c3RvcmUgUmVk aXMgU2VydmVyIENBMRQwEgYDVQQKEwtHb29nbGUsIEluYzELMAkGA1UEBhMCVVMw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7ZScHu3E7wifuplWRN5cu HqDXW3NTS4mtleTp6Z1cu3YGPWw6lzTH8tHcVJnLE0jjC8r0r517AvYmt/Ww9OzP Q/20AQOb4auFgAalbTSmvZ/++5ZIi7PRi8MvNJvGB9agmj+rEbDwcvu4Bdrs62kJ fpstTK5MGjdwCg/2pNXBERX71WgK+oT3i5AFJ2Ij/JlUpsc8A9Q2NrGctetz/fhH IgD2adxeKbhOqQLMGmU1adRgJlRriJcpgKFVfLZU9ZWqMD3u2S5hsGBYgjtNAjix gqSYcFcZNRm7TIYTf/08uUx0r/Ffj/Yp/L6qwnv5JNWNKKgU9FvWDo+/2874h99b AgMBAAGjFjAUMBIGA1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQELBQADggEB AJvKCuPLfrI1AVVKh8a3kiEQweCgqOmQTqqaAWKA9oaf4x4Iu5svRQj0Z2jIlzdg XCCX0JJdHpJrHrF8b9hze3/i/xSl86n1zKFAf0p8XUOTS7Kk9EnzSqZhbhw2RyZV VdMOPpAOSz9NbvbbldPgQm6EhXmwooGsrbFnHABYC8iCZD+oRqFngtJ1AvjngPPj zph8gcZAjpPrVwlBLA5cheAy2MKJU8pgvob6mFK6tuBJkcfRWsmrzoeDemHxh57w FMscG86vGwpibaI9CYEUENPmBDbPyzOxFOnShspkWobzXFj6PCH0IyyZXr6RbgPd TSMTaCndhVnL/z5b9N4RxuI= -----END CERTIFICATE----- user@26c17a4143c6:~/gcp-functions-go/backupinst#
It means that direct access to the GKE cluster should be prohibited and guarded as access to any secret store solution. Knowing the AUTH string, IP and the Certificate anybody with access to the network can read the data from the instance.
Here is short summary:
- If you use Anthos Config Connector to deploy Redis you need to restrict access to the cluster which deploys the instance to prevent access to the Redis AUTH string, Certificate and IP address.
- The AUTH, IP and Certificate can be retrieved only after successful deployment and it requires a tool which can verify the resource, get the resource and check it out to a secret solution for the application usage.
- The deployment is asynchronous and the application deployment should be delayed or staged after successful deployment of the Redis instance and retrieval of the necessary connection information.