Mark Young

Software. Microcontrollers. Beer.

Up and Running With Bless and Enforced MFA - Part 1

I finally got Netflix’s Bless running in production using a forked version of Lyft’s client. This post will focus on the first and easier portion: Bless in Lambda

Prerequisites

Time and terraform. I like terraform because I dislike cloudformation. Feel free to adapt.

Installing Bless

This is actually the easiest part, because it’s straight forward. Side note: I like bless because of its simplicity. It uses lambda + KMS and nothing more. The permissions are stupid simple. The purpose, after talking to one of the guys behind it, was to make it feel native and un-intrusive. It works.

Setup Config

This was the hardest to get going, because configuration always is. The readme says to paste in a function to lambda to generate your password. But thats not needed with the super awesome lambda container from lambci. To do this, create a file function.py with the contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import boto3
import base64
import os


def lambda_handler(event, context):
    region = os.environ['AWS_REGION']
    client = boto3.client('kms', region_name=region)
    response = client.encrypt(
    KeyId='kms_key_id',
    Plaintext='totallysecure'
    )

    ciphertext = response['CiphertextBlob']
    return base64.b64encode(ciphertext)

Now lets run that without using the real lambda. Make sure your access key/id have the required permissions to do kms:Encrypt:

1
2
3
4
5

docker run -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY -e AWS_REGION=us-east-1 -v "$PWD":/var/task lambci/lambda:python2.7 function.lambda_handler
START RequestId: 42f85048-91e1-4b48-9376-0563722616f0 Version: $LATEST
END RequestId: 42f85048-91e1-4b48-9376-0563722616f0
REPORT RequestId: 42f85048-91e1-4b48-9376-0563722616f0 Duration: 733 ms Billed Duration: 800 ms Memory Size: 1536 MB Max Memory Used: 25 MB
"some base64 encoded key"

The Config

This is an example of a finished config. Save it as lambda_configs/bless_deploy.cfg:

1
2
3
4
5
6
7
8
9
10
11
12
13

[Bless Options]
certificate_validity_after_seconds = 120
certificate_validity_before_seconds = 120
entropy_minimum_bits = 2048
random_seed_bytes = 256
logging_level = INFO

[Bless CA]

us-east-1_password = some base64 encoded key from lambci/lambda
ca_private_key_file = foo1.pem

[KMS Auth]

Build

1
2
3
4

$ virtualenv venv
$ . venv/bin/activate
$ pip install -r requirements.txt
$ make publish

Deploy

IAM

The first piece is the IAM portion for Bless. I need to allow it to:

  1. Generate random keys from KMS (obvious)
  2. Decrypt the KMS key that’s the password for Bless. This is generated and
  3. Create the log group (maybe not necesary since Lambda does that for you?)
  4. Push logs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

resource "aws_iam_role" "bless_lambda" {
    name = "bless_lambda"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "bless_lambda" {
    name = "bless_lambda"
    role = "${aws_iam_role.bless_lambda.id}"
    policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Sid": "Stmt1443036478000",
          "Effect": "Allow",
          "Action": [
              "kms:GenerateRandom",
              "kms:Decrypt"
          ],
          "Resource": [
              "${data.terraform_remote_state.kms.ops_bless_arn}"
          ]
        },
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:*:log-group:/aws/lambda/*"
            ]
        }
    ]
}
EOF
}

KMS

1
2
3
4
5
6
7
8
9
10
11
12
13

resource "aws_kms_key" "ops-bless" {
    description             = "KMS key for Bless"
    deletion_window_in_days = 7
}

resource "aws_kms_alias" "ops-bless" {
    name          = "alias/ops/bless"
    target_key_id = "${aws_kms_key.ops-bless.key_id}"
}

output "ops_bless_arn" {
  value = "${aws_kms_key.ops-bless.arn}"
}

Client

Client IAM role to assume

It basically just allows the invocation of our deployed function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

resource "aws_iam_role" "bless-client" {
    name               = "bless-client"
    path               = "/"
    assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "lambda:InvokeFunction",
            "Effect": "Allow",
            "Resource": [
                "${data.terraform_remote_state.lambda-bless.bless_arn}"
            ]
        }
    ]
}
EOF
}

I added this to our ops IAM group so that any ops person can assume the role or use kms:Encrypt from the bless KMS key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

resource "aws_iam_group_policy" "bless-client" {
    name = "bless-client_iam_policy"
    group = "${aws_iam_group.ops.id}"
    policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [
          "${data.terraform_remote_state.bless_client_iam.bless-client-role-arn}"
        ]
      },
      {
        "Action": "kms:Encrypt",
        "Effect": "Allow",
        "Resource": [
          "${data.terraform_remote_state.kms.ops_bless_arn}"
        ],
        "Condition": {
          "StringEquals": {
            "kms:EncryptionContext:user_type": "user",
            "kms:EncryptionContext:from": "$${aws:username}"
          }
        }
      }
    ]
}
EOF
}

Client config

This is an example of what my client config looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

[MAIN]
region_aliases: east
kms_service_name: bless
bastion_ips: 192.168.1.6
remote_user: ec2-user

[CLIENT]
domain_regex: (.*\.mycompany\.com|.*\.example\.net|\A10\.[0-9](?:\.[0-9]{1,3}){2}\Z)$
cache_dir: .bless/session
cache_file: bless_cache.json
mfa_cache_dir: .aws/session
mfa_cache_file: token_cache.json
ip_urls: https://api.ipify.org, https://canihazip.com
update_script: update_blessclient.sh

[LAMBDA]
user_role: bless-client
account_id: my account id
functionname: bless
functionversion: $LATEST
certlifetime: 120
ipcachelifetime: 120
timeout_connect: 30
timeout_read: 30

[REGION_EAST]
awsregion: us-east-1
kmsauthkey: my kms id that i used for the original password encrypt

Test

You should now have a lambda that can generate certificates to use. You’ll need to put the public keys from foo1.pub and foo2.pub into your servers at /etc/ssh/cas.pub and add TrustedUserCAKeys /etc/ssh/cas.pub to /etc/ssh/sshd_config You can test this with their client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

function bounce () {
  FILE="$(mktemp)"
  rm -rf ${FILE}
  ssh-keygen -f ${FILE} -N ""
  ./bless_client.py \
    us-east-1 \
    bless \
    testdev $(curl --silent https://ipecho.net/plain) \
    ec2-user $(ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1') \
    "" \
    ${FILE}.pub ${FILE}-cert.pub
  ssh -i ${FILE} ec2-user@$1 "${@:2}"
}

$ bounce 10.0.0.19