Never Ask a Butler to do a Robot's Job - Born for DevOps CI

At Infor, my team is responsible for operating a internal, scalable, highly available implementation of Gitlab It is designed for scaling to service our thousands of SaaS developers building hundreds of applications. We have built Gitlab CI Runner deployment automation to supersede our previous deployment automation for a pseudo-high availability Jenkins configuration similar to the approach of CloudBees. We have put together a guide for developers to consider Gitlab CI whenever they have the opportunity to reconsider their CI.

Read more

Share Comments

Mission Impossible Code Part 2: Extreme Multilingual IaC (via Standard Code for Preflight TCP Connect Testing a List of Endpoints in Both Bash and PowerShell)

It is not possible for me to count the number of times this code has saved me support calls because I never get those calls ;) A huge part of my work is to build DevOps IaC automation code as tools in a company that runs around 50% Windows and %50% Linux across their many SaaS software stacks.

One of the main types of IaC my team builds is deployment automation for DevOps agents that are designed to run on any of the 10’s of thousands of instances at the company - agents for things like vulnerability scanning, malware scanning, log aggregation and monitoring. Generally these agents are wiring up to an internal or external cloud tenant environment for reporting and/or administration.

Everyday at my job I learn of a new environment I’ve never heard of before that someone is trying to run my team’s code in. Frequently the environment setup is at fault when these DevOps agents error out on their tenant registration calls. After way too many escalations that resulted in the discovery that the environment is at fault - I decided we need to preflight check the tenant URLs we would need to connect to and report failures in logging so that tooling users could easily distinguish when their environment was not allowing endpoint access.

Another common case for endpoint verification is when code depends on public or external package management endpoints for things like yum or chocolatey packages. However, the approach is solid for endpoints of all type whether public or private, local or remote.

If you take a look at a lot of your automation code it may make fundamental assumptions about available endpoints and if it will run in environments that are out of your control, endpoint connectivity validation will save you boatloads of support time :)


Read more

Share Comments

CloudFormation Stack Attack

I’ve been studying for the AWS Certified DevOps Engineer exam and CloudFormation is a big topic.

Understanding the more complex ways to configure interrelated stacks is a must know for this exam.

I like to learn by doing and I started to wonder if I could create a minimalist set of CloudFormation templates that could demonstrate all the ways of inter relating stacks.

This post is the result of that effort.


Read more

Share Comments

ASG Lifecycle Hook for Linux Kernel Patching with a Reboot In AWS Autoscaling Groups

Linux has a long and strong reputation for rarely needing a reboot - and it lives up to that reputation very well.

Recently I had to devise a solution for a case where it frequently needs a reboot, but you can’t easily take one.

AWS ASGs are notorious for being quick to terminate a rebooting linux instance because it deems them unhealthy. Making the health check long enough to accomodate the instance build and reboot will in many cases yield a health check that too long for daily production operations - which defeats the whole point of the health check.

Yet if you perform comprehensive OS patching during ASG provisioning of a new instance, you will eventually end up with a pending kernel patch due to the age of the AMI the ASG was commissioned with.

AWS Amazon Linux 1 is very stable and so new AMIs releases with updated patches can be 6 to 9 months or more apart - which increases the possibility of critical kernel vulnerability patches awaiting a reboot that will never happen.

Let’s look at a simple, effective solution to avoid this problem during ASG instance provisioning that can also be used to perform regularly patching of an autoscaling group of instances.

BTW - there is a lot of value to adding this pattern to your Windows instances as well - so you can read this article and the provided CloudFormation template with an eye to that as well!


Read more

Share Comments

Mission Impossible Code - Hyper-planning + Hyper-pragmatism = Get the Job Done Every Time (Part 1)

Super action spies like Ethan Hunt, Jason Borne and Evelyn Salt live in an ethos of getting the job done no matter what! They complete their missions in vastly diverse conditions and in the face of the unexpected.

Super spies make use of specialized tools and techniques when available (and working), but simple and pragmatic alternatives are always top of mind. They jump out of windows, walk across moving cars, use household objects as weapons and drive cars down staircases. They are consistently fashioning situational tools of whatever is found around them. They don’t think of objects and situations as having fixed purposes - but rather that objects and situations are flexible to serve their imposed purposes.

Is it possible to write code that acts like a super spy? Over time I have adhered to a set of coding design heuristics whose parallels to super spy priorities are intriguing.


Read more

Share Comments

Super Compact, DevOps-ish Pending Reboot Test for The Rebootiest Operating System in The Cloud

Windows and reboots - more than a few candles have been burned on both ends in understanding and resolving this relationship. Like it or not and despite Microsoft’s efforts - Windows is the most rebootiest operating system around.

There has to be a ton of code written around this - is it possible add a new contribution of real value?

I think it is - by being concise around my specific context of software deployment automation for DevOps in the cloud with a brutal eye to compactness.


Read more

Share Comments

No 7zip Allowed: Extracting Oracle's Gzipped Java Tarball On Windows to Create an Isolated, Zero Footprint Java Install for CIS CAT Pro

I had a project to package the CIS CAT Pro benchmark auditing tool for Windows and Linux. The unique Windows challenges I experienced are applicable anytime you either need to extract Java for Windows or extract any gzipped or tar archive on Windows - without using 7zip. CIS CAT Pro requires Java and I wanted to create a zero footprint Java install that could be cleanly wiped out by deleting a folder. This allows the automation to be more readily used on production systems because it won’t force a Java install, nor compete with an existing version of Java. (I find it ironic that CIS CAT requires Java - and then frequently flags the copy of Java it is using as a problem)

7zip has had a fair share of security vulnerabilities - consequently installing or using it can set off more than a few security bells where I work - so it was required to have a solution that was 7zip-less.

While it is more than a little frustrating that Java is only provided by Oracle as a gzipped tarball for Windows, this method will work fine for anything else that is only provided for Windows as a gzipped tarball.


Read more

Share Comments

Time Is Not Your Most Precious Resource

I used to feel that Time was the most valuable commodity I possessed. The reasoning is simple, seductive and often repeated. When this idea is tossed around in popular culture, it really seems to mean “Time is the most unchangable resource used for moving toward your goals.” It makes sense right?, because you can’t control your spend rate - it goes out the door at 60 seconds to the minute, 24 hours to the day.

One morning I woke up and realized my sleep time is not available to me to apply to my goals. (Yeah, self-evidential Eurekas are that sort of paradox) This got me wondering whether there were other natural limits to my usable time that I wasn’t immediate grasping?

To this day, I am a personal productivity geek who enjoys books on time perception and tracks every minute of my professional work activities, yet I have come to believe that time itself is NOT my most precious resource…


Read more

Share Comments

Automators Paradox - Never Put Your Career Management on Autopilot

My mind does little cartwheels when it experiences the confluence of two independent streams of thought into a larger, faster flow.

I have been listening to Stanley McChrystal’s “Team of Teams” which bursts with interesting insights. Recently someone forwarded a blog post by a colleague, Forrest Brazeal, titled “Cloud Irregular: The Creeping IT Apocalypse”.

The combination seems to be both delicious and nutritious.


Read more

Share Comments

Three Amazon Linux 2 Containers for Testing

I frequently have to test code on Amazon Linux 2 - both for work and for the PowerShell Core universal install script install-powershell.sh that I help maintain on the PowerShell Open Source project.

Spinning up an instance on Amazon is not a hassle, unless of course, you compare it to spinning up a container. Amazon Linux 2 container images are necessarily super-optimized to run as a container host for applications - so they have many packages removed compared to an Amazon Linux 2 EC2 AMI build.

But when I am testing something that will run on the full EC2 build, I’d still like to use a container.


Read more

Share Comments