Rebrand to Mission Impossible Code

This blog started life as because I was generating a lot of sharable Infrastructure as Code and DevOps automation for Windows. While I still do this, my career focus has really broadened to cover plenty of Linux and more and more cloud specific technologies - which is reflected in my blog posts. The one common theme across the previous focus and the new one, is that the coding style I try for is consistently characterized by what I’ve come to call “Mission Impossible Code” (originally discussed here).

Read more

Share Comments

UPDATED: The Ultimate AWS AutoScaling Group ASG Kickstart and Lab Kit

While noodling uses for the Ultimate AWS ASG Kickstart and Lab Kit I realized it could use a couple new features and improvements.

The first is to have instances self-tag themselves as to whether they are a spot or on-demand instance. When supporting a mixed instances policy, the implementation requires a little more thought.

A second is the ever common need for a bucket that the ASG instances have access to - whether for AWS SSM results collection or inventory, for deployment artifacts to update instances, access to data or many other uses.

While a relatively small change, the title has also been updated to “The Ultimate AWS ASG Kickstart and Lab Kit” to indicate it is appropriate for experimentation and also as the foundation for a deployable configuration.

I was able unable to avoid the temptation to sneak in a couple other improvements as well.

Read more

Share Comments

NEW Oneliner to Tail the Windows Eventlog

Since switching focus to the cloud I am doing more and more pure CLI admin of Windows. One of the pains of windows admin from a console is accessing the windows eventlog. Since they are not simple text files like Linux, special PowerShell CMDLets must be used to retrieve them.

Due to the frequency of needing to do it, one of the biggest challenges is tailing an eventlog while waiting for results.

When following a text log, I simply use Get-Content logfilename -wait to emulate the Linux command tail -f logfilename

So I went in search of what I thought would be a quick find, but all my finds were all way to long and involved - so I made a new oneliner that follows the principles of Mission Impossible Coding.

Read more

Share Comments

The Ultimate AWS AutoScaling Group ASG Lab Kit

A while back I wrote a blog and companion Cloud Formation templates for experimenting with the ways an ELB creation template could be linked to an ASG. That iteration was based on an ASG template designed to show how to kernel patch linux and reboot without termination using ASG Lifecycle hooks.

I had a number of improvements I wanted to make to this template set and this blog represents that work.

The result is really the answer to the question “What would be a minimal, but production-useful working example to learn and experiment with AWS ASGs that use spot instances and proper lifecycle hooks?”

Since the last team I managed had to do all of our automation work for both Windows and Linux, I wanted the solution to work for both.

Read more

Share Comments

GitLab Commit - The Little Conference That Could - And Did

This post contains the links to my GitLab Commit Brooklyn session “Never Hire a Butler to do a Robot’s Job” as well as an interview I had with Alan Shimel at GitLab Commit.

I’ve spoken at a lot of conferences and I’ve experienced a good number of the many ways that conferences can fail to deliver.

GitLab Commit had three major things working against it that made me wonder what the experience would be like. First of all it was a first time conference for GitLab. Secondarily it was only one day. In my experience, many one day conferences are thinly veiled marketing events with shallow, slapped together sessions. And finally, session slots were 30 minutes - unless it is a 5 minute lightning session, I’ve never seen a session length that short.

My expectations weren’t high, but I was delighted to be wrong on all counts…

Pssst: You can still catch GitLab Commit in London on October 9th or San Franciso on January 14th.

Read more

Share Comments

Never Hire a Butler to do a Robot's Job - Born for DevOps CI

At Infor, my team is responsible for operating a internal, scalable, highly available implementation of Gitlab It is designed for scaling to service our thousands of SaaS developers building hundreds of applications. We have built Gitlab CI Runner deployment automation to supersede our previous deployment automation for a pseudo-high availability Jenkins configuration similar to the approach of CloudBees. We have put together a guide for developers to consider Gitlab CI whenever they have the opportunity to reconsider their CI.

Read more

Share Comments

Mission Impossible Code Part 2: Extreme Multilingual IaC (via Standard Code for Preflight TCP Connect Testing a List of Endpoints in Both Bash and PowerShell)

It is not possible for me to count the number of times this code has saved me support calls because I never get those calls ;) A huge part of my work is to build DevOps IaC automation code as tools in a company that runs around 50% Windows and %50% Linux across their many SaaS software stacks.

One of the main types of IaC my team builds is deployment automation for DevOps agents that are designed to run on any of the 10’s of thousands of instances at the company - agents for things like vulnerability scanning, malware scanning, log aggregation and monitoring. Generally these agents are wiring up to an internal or external cloud tenant environment for reporting and/or administration.

Everyday at my job I learn of a new environment I’ve never heard of before that someone is trying to run my team’s code in. Frequently the environment setup is at fault when these DevOps agents error out on their tenant registration calls. After way too many escalations that resulted in the discovery that the environment is at fault - I decided we need to preflight check the tenant URLs we would need to connect to and report failures in logging so that tooling users could easily distinguish when their environment was not allowing endpoint access.

Another common case for endpoint verification is when code depends on public or external package management endpoints for things like yum or chocolatey packages. However, the approach is solid for endpoints of all type whether public or private, local or remote.

If you take a look at a lot of your automation code it may make fundamental assumptions about available endpoints and if it will run in environments that are out of your control, endpoint connectivity validation will save you boatloads of support time :)

Read more

Share Comments

CloudFormation Stack Attack

I’ve been studying for the AWS Certified DevOps Engineer exam and CloudFormation is a big topic.

Understanding the more complex ways to configure interrelated stacks is a must know for this exam.

I like to learn by doing and I started to wonder if I could create a minimalist set of CloudFormation templates that could demonstrate all the ways of inter relating stacks.

This post is the result of that effort.

Read more

Share Comments

ASG Lifecycle Hook for Linux Kernel Patching with a Reboot In AWS Autoscaling Groups

Linux has a long and strong reputation for rarely needing a reboot - and it lives up to that reputation very well.

Recently I had to devise a solution for a case where it frequently needs a reboot, but you can’t easily take one.

AWS ASGs are notorious for being quick to terminate a rebooting linux instance because it deems them unhealthy. Making the health check long enough to accomodate the instance build and reboot will in many cases yield a health check that too long for daily production operations - which defeats the whole point of the health check.

Yet if you perform comprehensive OS patching during ASG provisioning of a new instance, you will eventually end up with a pending kernel patch due to the age of the AMI the ASG was commissioned with.

AWS Amazon Linux 1 is very stable and so new AMIs releases with updated patches can be 6 to 9 months or more apart - which increases the possibility of critical kernel vulnerability patches awaiting a reboot that will never happen.

Let’s look at a simple, effective solution to avoid this problem during ASG instance provisioning that can also be used to perform regularly patching of an autoscaling group of instances.

BTW - there is a lot of value to adding this pattern to your Windows instances as well - so you can read this article and the provided CloudFormation template with an eye to that as well!

Read more

Share Comments

Mission Impossible Code - Hyper-planning + Hyper-pragmatism = Get the Job Done Every Time (Part 1)

Super action spies like Ethan Hunt, Jason Borne and Evelyn Salt live in an ethos of getting the job done no matter what! They complete their missions in vastly diverse conditions and in the face of the unexpected.

Super spies make use of specialized tools and techniques when available (and working), but simple and pragmatic alternatives are always top of mind. They jump out of windows, walk across moving cars, use household objects as weapons and drive cars down staircases. They are consistently fashioning situational tools of whatever is found around them. They don’t think of objects and situations as having fixed purposes - but rather that objects and situations are flexible to serve their imposed purposes.

Is it possible to write code that acts like a super spy? Over time I have adhered to a set of coding design heuristics whose parallels to super spy priorities are intriguing.

Read more

Share Comments