Integrating Equinix Metal and AWS Secrets

As we've previously discussed in our deploy guides, there's a one rather tricky challenge with spinning up Kubernetes clusters on bare-metal; secrets. This problem doesn't really rear its ugly head until you try to build a highly available control plane, as to do so requires you to share the private keys with each control plane node. While kubeadm does store these temporarily as a secret, it requires each node to have a working join token and for the new nodes to join before the secret is deleted; which defaults to an hour.

Now, we could show you how to deploy Hashicorp's Vault on Equinix Metal, and we will soon, but if you're going to deploy something secret to bare metal, it needs secrets to keep its own secrets secret. Got it? So in-order to show you how to do that correctly, we're going to start by showing you how to integrate the KMS (Key Management Server) of another cloud provider to act as the substrate for your own secrets within Equinix Metal; and we'll do this in a secure fashion over a private, direct, internet free, connection.

Let's begin.

Prerequisites

• AWS Account
• Equinix Metal Account
• Equinix Fabric Portal Access

In-order for our demo to work, we need to configure a few things on the Equinix side and on the AWS side.

Equinix Configuration

Equinix Metal

First, let's use the Equinix Metal portal to create our interconnection request. You can find the Request a new interconnection button by finding the Interconnections menu on the Equinix Metal Portal.

You'll want to select Fabric VC and Metal Billed and then configure the interconnection with a name, metro, and speed.

Click Submit Request

This will provide you with a primary and secondary service token. Store these and keep them safe.

Equinix Fabric

You can now login to the Equinix Fabric Portal, where you can create a new connection. Dropdown the Connections menu and select Create Connection.

It'll ask where you want to connect to, select A Service Provider followed by AWS.

Paste your service tokens into the Service Token tab and validate them. Please note, you may only have a single token if you didn't create a redundant connection on the Equinix Metal side.

Lastly, enter your AWS Account ID that you wish to connect to and you need to select a speed. Please do ensure this matches the speed you requested on the Equinix Metal side also.

That's it! Now you wait for the connections to be provisioned, this usually takes around 15 minutes.

AWS Configuration

KMS VPC Endpoint

In order to authenticate with AWS KMS without requiring any secrets, we need to create a VPC endpoint to allow direct connections, without the internet, from a VPC IP address. This is made possible by AWS PrivateLink connected with Equinix Fabric.

To create a VPC Endpoint for KMS, you must first select a region for your KMS keyring. Let's assume that you want to use us-east-1.

resource "aws_vpc_endpoint" "kms" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.us-east-1.kms"
}
output "kms-endpoint" {
  value = aws_vpc_endpoint.kms.dns_entry
}

This HCL when used with Terraform is all you need to create the VPC Endpoint. Now within your main VPC, you can use the dns_entry attribute from the kmsresource to speak to the KMS API.

If you wish to check that this works, you can use a virtual machine within your VPC to confirm the HTTP availability of the KMS API. Please not that this aws CLI command must be manually configured with AWS credentials, at-least until we authorise invocations through IAM.

aws kms list-keys --endpoint-url $(terraform output kms-endpoint)

VPC Endpoint Policies

Now that we have a means to communicate with KMS without going over the internet, we can begin to configure that endpoint to be secure.

{
    "Id": "example-key-1",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM policies",
            "Effect": "Allow",
            "Principal": {"AWS":["111122223333"]},
            "Action": ["kms:*"],
            "Resource": "*"
        },
        {
            "Sid": "Restrict usage to my VPC endpoint",
            "Effect": "Deny",
            "Principal": "*",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:sourceVpce": "vpce-1234abcdf5678c90a"
                }
            }
        }
    ]
}

KMS Proxy

Even though we now have a direct connection from Equinix to the AWS KMS, we don't have any means to authenticate that connection. Now we could say Put the credentials on the metal, but we're going to continue to persevere and make this work sans credentials.

To do that, we need to deploy an AWS Lambda function that can use Workload Identity to gain access to the KMS keyring; and ensure that we allow anonymous access to said Lambda through another VPC Endpoint.

Let's take a look at our Lambda code.

Decrypt

const AWS = require('aws-sdk');
AWS.config.update({ region: 'eu-central-1' });

const encrypted = process.env['STRIPE_SECRET_KEY'];
let decrypted;

exports.handler = async (event) => {
    if (!decrypted) {
        const kms = new AWS.KMS();
        try {
            const req = { CiphertextBlob: Buffer.from(encrypted, 'base64') };
            const data = await kms.decrypt(req).promise();
            decrypted = data.Plaintext.toString('ascii');
        } catch (err) {
            console.log('Decrypt error:', err);
            throw err;
        }
    }
};

Summary

Equinix Fabric combined with AWS Direct Connect and PrivateLink offer a world of integration points to bridge the gap between bare metal and managed cloud; allowing you to pick and choose the hybrid architecture that makes sense for you.

We hope this helps you get up and running and we'll be back soon with the guide to using this method for a secure deployment of Hashicorp Vault.