I have been a consultant for boutique, medium-sized, and large consultancies. Before consulting, I was a developer. All of my consulting experience has been in helping companies adopt DevOps practices. Because I am a consultant, I get exposure to many more companies than I would if I were just an employee at them. So, I hope that if you don't know much about consulting, you get a view into my world, and if you're an existing consultant, then maybe I can help you with a challenge you've had.

Before we get started

I was inspired at my day job because lots of our recruits don't have previous consulting experience, so I've been reflecting on what it means to be a "consultant." I thought that others could benefit from the lessons I have learned over the years. I am also doing this for myself because it's helping me reflect.

These opinions are my own, dear employer, don't fire me. 🙃

For context, I live and work remotely for clients in the United States. I know there's an international audience through this medium.

What this blog post is not about

In my view, some "consulting" companies are staff augmentation companies. In other words, the "client" is a company that needs people with certain skills. The "client" can't find people because they don't have a strong brand or recruiting capabilities, their payscales don't allow for competitive compensation, or the company likes to spool up or down the workforce with contractors, etc. When a consulting company does a "staff augmentation" project, the "client" does not expect strong advisory services.

I believe a "consultant" is an advisor and change agent looking to help the "client" achieve their goals. So, in this post, I will share my opinions about projects where I was strictly in that role - an advisor.

Lastly, I consider "DevOps" to include people, process, and technology aspects. So, this will not be a blog post around tips and tricks for creating pipelines or cranking out containers. Spoiler alert: Technology is the easy part. 😱

Consultant Glossary

• Statement of Work (SOW) - The contract for a project between a consulting company and the client organization.
• Project Plan - A plan on how to deliver the project. Sometimes very Ghantt chart esque.
• Client Organization - The company that hires a consulting company.
• Client - The main point of contact or stakeholder that works for the client organization.

1. Do the hardest thing first

Often, as engineers, we tend to want to put our heads down and crank out code or fix technical issues. But we're human beings, and as humans, we are highly social creatures. There is no "avoiding people." And people also create processes to get stuff done.

So, do the hard things first. You're probably dreading the hard things, and they will most likely be problems related to "people" or "processes."

So, for example, here's a list of things I watch out for.

1. Go to production - Deploy to production as early as possible. Just because you "deploy" to production doesn't mean you have to "release" it to your users. More on this later.
2. Have that security review with Central IT - If you're part of a big traditional enterprise, you are likely required to have a "security review" with enterprise architects and security architects. Do it as early as possible, be ready for it, and sometimes you'll be surprised to find that they're more straightforward than you made them out to be in your head. However, you want to avoid the scenario where you wait until the big production deployment towards the end of the project to have that security review. Then you receive feedback that will fundamentally change your architecture.
3. Have that review with your enterprise architects - Like the previous point, propose a review at the beginning of the project in addition to the final one. The earlier, the better, so that you avoid surprises.
4. Get the "process" stuff out of the way - Have to integrate with systems you find dumb, but they're crucial to your project? Do it soon. Don't run a project on assumptions that things will work. Validate those assumptions.
5. Start that documentation early - Often, I have to provide long recommendation documents. So, I "write drunk" and "edit sober." Meaning writing the first draft without editing. If you try to write and edit, you will get stuck in writer's block or perfectionism.

Here's a strategy I borrowed from a client for greenfield apps - perform a walking skeleton production deployment early in the project. A walking skeleton is just a basic set of the infrastructure deployed to production with very little actual code. Also, a walking skeleton uses Infrastructure as Code and a CI/CD pipeline. The sheer nature of being in "production" will trigger all those processes you dread, and you can get them out of the way early. You will avoid nasty surprises and delays toward the end of the project. You will also gain a lot of practice and confidence deploying to production with the automation you established in the first place.

2. "Big bang deployment" is a bad word

Remember the "big bang" theory? There was a HUGE explosion, and then there was the universe. Deployments to production can be that way, too - a big bang deployment of infrastructure and code, chaos, and hopefully a working application.

You might be surprised, but so many companies today still wait until the end of the project to release to production. They hope and pray that a big bang will work. Sometimes, they ignorantly think nothing wrong will happen because of their skill and experience.

A big bang deployment is an explosion of Work in Progress (WIP). WIP is unfinished work. The work is unfinished because it hasn't been shipped to production, and you're not learning from it yet. The more WIP you have in the system at one time, the more risk.

Can you make a big bang less painful? Yes. I have had to suck it up and do big-bang deployments before. Here are some tactics that have helped me:

• Infrastructure as Code at the beginning of the project - Less environment drift between non-prod, the better. It's going to alleviate surprises on the production side. But it won't stop them completely. If you start at the beginning of the project, you'll get a lot more repetitions with the process and have the training needed for the big day.
• CI/CD pipeline at the beginning of the project - You become reliant on a pipeline to perform the deployments of the code. Therefore, you're less likely to have surprises on the big day.
• Create a deployment plan and review it thoroughly - Create a simple document and list step-by-step all the things the team will do to perform the deployment. If you can, deploy a whitespace deployment in a non-production environment to validate the document. A whitespace deployment is just a deployment from scratch, so re-create the infrastructure and redeploy the code.
• Break it up into chunks - Production "release" is not the same as a production "deployment." A release is when you make it available to your customers. So, if you deploy to production, you don't have to make it available to your customers yet. For example, you could deploy the cloud infrastructure, go to sleep, deploy the code, go to sleep, run some automated functional tests to validate, go to sleep, then perform database migration and do the DNS cutover.

Ideally, if I had a magic wand and I could get rid of Big Bang deployments, I would:

1. Create the infrastructure via code at the beginning of the project.
2. Set up CI/CD pipelines for my infrastructure and the application code at the beginning of the project. Then deploy to production frequently throughout the development of the app. Feature flags can also be very helpful here.
3. Perform whitespace deployments occasionally throughout the project.

The last thing on this topic, you might think that big-bang deployments only affect companies that are not using Agile. In my experience, just because a company is "using" Agile or Scrum or SAFe doesn't automatically mean they are free from big-bang deployments. I've worked with zero companies that identify as using Waterfall, and I have run into many, many, many big-bang deployments.

3. Listen to the client and bring them into the conversation. Co-create with them. But there's a catch.

When designing a system, I like to bring the client into the design conversations. Maybe have "whiteboarding" sessions or just a series of meetings where we talk. At some point, I begin to form an opinion and have an idea of what I would like to recommend. So, I want to fail fast and validate my assumptions with the client. I also recognize that I might not have all the data points, so if possible, I create a simple draft document or PowerPoint stating options on a part of the system that could be designed. I present the options to a core client team with the goal of learning more. Often, they provide minor feedback or significant feedback. Sometimes the options get smashed together into a new option. Sometimes, we need to create an experiment to validate one of the options to address any concerns. The key is that the client is involved and therefore feels in control of the project's outcome.

In my observations, traditional consulting companies have different project delivery methods. They create a detailed project plan and weekly email reports and assign a project manager to the project. The plan has a clear "design/document/build" phase. Depending on the size of the project, there could be several months before anything is built!

What happens if there are gaps in the design document? Will the implementation go according to plan? Does the project need to halt because the design needs to be flushed out? I'm not sure, but it does sound risky, right?

If the project does get delayed, the client would not feel good about it. Even worse, if the design phase is dragged out until it's "right," it could mean months without actually building something or driving outcomes for the client. In my opinion, the client typically doesn't care about a recommendations document - it just doesn't drive the outcome for them at all. The document is a method for communicating design decisions and rationale.

Overall, I think one huge problem with the traditional consulting approach: the client was not involved in the design process.

How would you feel if you attended 10+ meetings talking about system design, and when you asked a question, the consultant said that the final answer would be in a recommendations document? Then, several weeks or months later, you receive over a hundred pages of the document. So, either:

• A. You have no time to read it - You trust that there are appropriate recommendations based on the points you have made. Maybe you skim the document and can't find the things you're looking for, so you make a point to come back to it but never do.
• B. You make the time to read it, and you're disappointed - The document doesn't have enough detail and rationale behind the decisions. Or maybe, there are contradictions between recommendations. Or, more frustratingly, the questions and points you have made during the discussions were never addressed. After you read more and more, you start to question, "this document is generic; it just states obvious things that we already knew."

Alright, so what's the catch to co-creating and designing with your client?

You can't hide.

If you're designing with them in the open, you will get to a point where you have to say, "I don't know." Or "in my experience, that hasn't worked well," or "I haven't tried this solution yet."

So ultimately, the client will know how much you know and how much you don't know. It will come out if you don't know enough about the subject you're talking about.

I don't recommend this approach to someone that doesn't have a lot of experience in the topic.

4. Know when to step back. Know when to step in.

Active listening is crucial. Stating opinions is important too. So, like any normal conversation, I think talking to a client should be an ebb and flow of listening and stating opinions. This "ebb and flow" is more important the more people there are in the "room." I believe that the "ebb and flow" quality often translates to the customer experience. The reason is that sometimes clients bring in consultancies because they cannot work well with a different team; therefore, if you manage to have two or more teams talking and working together, that's a tremendous value.

Here are two extremes that I am not fond of:

• Too much talking - Talking so much to "control" the conversation and "impress" the client. Often, it just comes across like someone likes to hear themselves talk. Besides, if you do all the talking, there's very little listening, and if you want to surely fail a project, that's to not listen at all.
• Listening and punting - Too much of one thing is bad, and the same is valid for listening. Punting is when opinions are reserved until a later point and not "showing your cards." Being very introverted, I struggle with this.

5. The client doesn't give a shit about the SOW. The client wants results.

In my experience, it's normal for clients not to read the contract. Maybe, the stakeholder read it but didn't understand the technical fine points. Or maybe the stakeholder is not part of the day-day delivery of the project and did not share with the main people you're working with. Sometimes, I believe that they expect excellent customer service, and they trust that the consulting company understands their needs, so they never get around to reading it. Whatever the reason is, the SOW doesn't matter to them. What matters to them is that they can reach the outcome that they're looking for. It is very possible that you do everything in the SOW within the timeline, and the client is unsatisfied.

If the SOW does not state clearly the goals or outcomes of the engagement, ask the client. Sometimes the client will be a little annoyed since they would have had these conversations already during the pre-sales process. Still, it's better that they are annoyed that the project starts with complete misalignment from the beginning.

I'll share with you some high-level outcomes of my past projects so that you get a sense of how high-level we're talking.

• "We acquired a company with an extensive software system, and their system is not up to our security standards. We want to redo everything using DevOps practices and Infrastructure as Code to replicate these practices for other systems in the future and cut the lead time to onboard new applications to the cloud."
• "We want to modernize our monolithic application because we had a severe outage that lost us millions of dollars of revenue. If it happens again, it could be detrimental to the business. We feel this technical debt on the monolith is catching up to us; it's been growing for a decade."
• "We want to find a new branching strategy because our teams struggle to deliver value consistently during their sprints. Every sprint, we also have crazy four-hour deployment meetings at 3 AM because usually, we have challenges deploying to production without an issue. We have other pains like code freezes against our QA environment, large deployment checklists, and long hours before the end of the sprint to try to get work done."
• "We want to standardize our cloud infrastructure to cut down little decisions when we onboard applications to the cloud platform. By standardizing, we will be able to cut lead times to deploy new applications in the cloud and keep up with the demand of the enterprise so that we don't have to grow the cloud team."

Notice that I did not say, "we want ten Kubernetes clusters, twenty pipelines, and lots of shit doing fancy things."

6. Don’t hide behind the project manager, the salesperson, or the engineering director. Be vulnerable.

In my career, I have witnessed engineers that prefer to focus on the "technical" things while they lean on other teammates to do the "people" stuff. The same goes for consulting, and I have witnessed consultants that want to rely on the project manager to talk to the client. Heck, I have done this too. But, in my experience, this is what ends up happening:

1. Project manager asks what the team has done and delivered.
2. Engineers speak a bunch of technical gobbly doo.
3. Project manager has a project update with the stakeholders and presents all the technical gobbly doo, but because they don't understand it, they don't explain it well.
4. Client asks questions, and the project manager says, "I don't know," and asks the engineer during the meeting.

I believe it's much more effective for consultants to learn the soft skills necessary to say the hard things and communicate risks to the client. I'll give you an example of hard things I have said before:

• "I am afraid that if we don't stand up this Kubernetes cluster soon, we won't have enough time to perform a load test with the application on it and, therefore, not be ready for the traffic spike in the high season."
• "I am concerned that if we don't migrate this database to the cloud, then there's a risk that latency will be too large for the API. So, let's migrate it, but that will require us to complete that security review now."
• "I know this project is really important to you. I hear what you're saying. Unfortunately, we won't be able to start XYZ until the week of A because we have to finish B and an upcoming vacation."

I won't lie to you - it takes time and practice. If you're early on in your career, start small. You'll be much more comfortable asking questions when you're a senior team member practicing this skill for years.

7. Working under SAFe is really really hard

I'll be honest with you. I haven't taken the time to learn or understand the principles of SAFe. SAFe is short for Scaled Agile Framework, which was put together to help dictate how to do agile in a large enterprise. I can't speak to the details of exactly why it doesn't fit well with DevOps. But I can tell you my experience.

When I hear that a client is using SAFe, I bear for a long and slow burn-out-inducing project. It could be that 99% of my clients that have implemented SAFe have implemented it incorrectly, so it could be that it's not SAFe's fault. Based on opinions from others more well-versed in this topic, I don't think that's the case.

When I engage with a client organization that's typically already using SAFe, we have to use that delivery model to deliver a project. So, here are my observations about trying to deliver a project under SAFe:

• Work to do work - So much work just to do work. Not only are there the typical Scrum ceremonies, but there's also lots of other planning to ensure that other's team's planning is planned effectively. In the end, it's a lot of planning work to prepare to do the work.
• Super large meetings - To ensure that multiple teams plan effectively, they get shoved into a massive meeting. It could be a meeting with maybe ten or so teams or hundreds of teams. This large meeting is typically called "Program Increment" or "PI" for short. I have had clients with a PI planning event that's six hundred people large. Have you ever attended a six-hundred-person meeting? I have. Shoutout to Microsoft Teams for handling that massive scale.
• It's just disguised Scrumfall - Scrumfall is when a team uses Scrum in Waterfall fashion. If you're "iterating" for nine months before you deploy to production, it's probably Waterfall, not Agile. With SAFe, there's so much emphasis on planning and the illusion of iterations that it's hard to deploy or deliver anything to production for the first time under this model.

Did you know that big tech companies don't use SAFe? Even just Scrum, for that matter?

Here is the fantastic article mentioned in the tweet.

So, do you have to do what big tech companies do? Nope.

But this data point probably means that you don't need SAFe to be able to be successful, right?

Before trying to undo SAFe at your company, read my other points.

So, it's likely that you'll have to learn how to make the best out of it. Here's what worked well for me:

• Build excitement before a big planning increment - In SAFe, you have huge planning events once per quarter. If you can't convince another team to add some DevOps work during a program increment, then it's likely not going to happen. Therefore, before the big planning event, raise awareness before the big planning starts so that they'll add the work you're hoping to see onto their program increment. Do a brown bag, a demo, etc.
• Re-adjust your pace to match the client's pace - There is only so fast you can move under this world. I have had to learn multiple times to let go and be ok with moving much slower than I would like.

Edit: Lastly, here's a great guide from Agile thought leaders.

8. Sunk cost fallacy is a strong force. Add ego, and you are screwed.

The sunk cost fallacy is when someone decides to keep going in a specific direction because there's been too much time, money, and effort put into the project. I can't tell you how often a client has chosen to continue doing something a certain way just because they've spent too much time on the implementation.

The ego is a self-protection mechanism that we all humans have. The ego jumps out to help our self-esteem when we feel threatened. Maybe, we feel incompetent if XYZ were true, or we feel like we lost control if XYZ were true, or our bosses won't like us if XYZ were true. When someone has an extraordinary amount of ego, especially when they choose to listen to information based on the status of the person giving the information, it's going to be hard. Add some sunk-cost fallacy, and it becomes a fast-moving training heading for a cliff.

Here are some examples:

• "I went to an ivy-league school, and I created this fantastic library. We're going to keep using this library."
• "Microsoft uses Azure Service Fabric, so we'll keep using Azure Service Fabric."
• "Jimmy worked at Amazing Tech Company, so he's probably right on this. We're going to listen to Jimmy."

As a consultant, you can't control the client's decision. You have zero control. You only have the power of influence. If you run into someone with an extraordinary "ego," then the best thing you can try is to speak to his/her ego and provide information tied to status. In the end, if they don't value or respect your authority on the subject, they most likely won't listen. It might as well be an immovable object.

So, if I encounter a lot of ego and they're protecting something that cost them a lot of time and effort to implement, I pick a different problem to tackle. Usually, there are a lot of problems to solve, so this is not an issue. But if the problem they're protecting is crucial to the outcome for the client, then I'd have to lay it all out on the table and say the hard thing. In practice, I would probably escalate the issue and add other key personnel to the conversation to increase the stakes and the pressure.

9. Pick a hill to die on. Don’t fight all the battles.

If you picture a medieval times battlefield where armies fight close combat with swords, then the army on top of a hill usually has the advantage, and they try to defend the hill. The opposing army then fights uphill to defeat the enemy, but it will be more challenging. If you don't like the combative analogy, think of yourself as a love warrior trying to spread love 🙃. You will still only have enough energy to climb some of the hills.

When I start a project, I look for the "hills." The hills are challenging topics that will require a tremendous amount of emotional energy. Because of the energy, these initiatives will take, I pick one or two. Based on experience, I rule out some hills automatically because I know it probably won't help the project move forward.

Here are some of those "hills" that I always avoid:

• Trying to convince the client to use something other than JIRA.
• Trying to convince my client not to use SAFe for my project.

10. There needs to be trust between you and the client. If there’s no trust, they will not listen. In the beginning, they probably will not trust you. Get some quick wins to show value and you will earn trust.

When I make recommendations, I imagine having a "trust budget." Each big recommendation and big ask that will mean a lot of work for my client will consume from the "trust budget." Before I make these recommendations, I visualize how much of my trust budget I will consume. Often, I have a zero trust budget at the beginning of a project with a new client because I haven't delivered results yet.

So, if trust is earned via results, I quickly try to increase my trust budget so that I can make recommendations. Therefore, I am looking for some quick wins.

Quick wins are painful problems that are relatively easy to solve. When clients hear a low-effort solution to a nagging problem, they feel relief and excitement. And, if you're able to solve the silly problem along the way, then tell them afterward... they might make a statue of you outside their building.

Here are some examples of quick wins I have delivered:

• Document the onboarding steps you're going through and deliver them as an onboarding guide or checklist for future engineers.
• Add to their READMEs as you're learning about the systems.
• Tweaking ugly deployment scripts to make them cleaner.
• Provide easy recommendations on slashing cloud costs.
• Provide a demo of a potential solution to a nagging problem, ideally within the project's first week.
• Find a security vulnerability and raise awareness about it. Caveat: this only works if the company values security and doesn't see security as more work to do.

What if you can't find a quick win?

Then listen closely and ensure you'll solve the main problem extremely well. The key point is that when you deliver your recommendations, you want them to be heard. If you sense skepticism when talking to the client, that's probably an indication of a low "trust budget."

But skepticism is different than curiosity. Skepticism is "Do I trust this consultant that they can solve this problem? Have they done it before?". Curiosity is, "Okay, if this solution works, it would be amazing. How do I make this happen? What could go wrong?"

11. There’s no good and bad. Just effective or ineffective.

Don't be dogmatic. Don't hide behind "best practices." Explain why something is better or more effective.

I don't like "best practices." It implies that everyone should make this master list of things, and if you're not, then it's bad, and you're failing. They can also be a crutch to consultants.

I'm much more interested in the essence of a "best practice." Why is this a good idea? Why should your client do this? How much effort will they need to place to get the benefit?

For example, "it is usually best practice to usually keep secrets out of source control because if you decide to give read access to a repository to a contractor, then they can see those secrets and use them to gain access to an environment that they shouldn't have access to."

So, when you understand why something is considered "best practice," you can discuss risks vs. benefits with your client. If you're having trouble explaining why something is considered best practice, you probably don't understand it well enough yet.

12. When working with your team, call out ownership areas.

Some of the best projects I have delivered can be attributed to this single decision. Think of your team like a team of Avengers. Every person brings something different to the team, and each person has strengths and weaknesses. So, play to everyone's strengths.

For example:

• Jimmy used to work as an ops guy, so he should probably own the monitoring recommendations.
• Even though Anya is early in her career, she has a lot of drive. She can work on creating some demos that will take just as much time for someone senior to complete.
• Carlos used to be a developer, so he should probably own the Infrastructure as Code (IaC) recommendations.
• Sofia has the most experience, so she should probably design the Kubernetes cluster infrastructure since that's the most crucial part of this project.

When everyone agrees on what areas they will own upfront, it is much easier for the team to self-organize and react together. In contrast, if a team optimizes for "busyness" or time, then Jimmy, who doesn't have as much coding experience, might miss some crucial recommendations.

13. Sometimes, the customer experience matters more than the quality of the work. Sometimes, it’s more about how the client felt than how amazing the delivery was.

Although I cannot find the source, I heard of a study that found that average people thought the "best" doctors were the ones that had better bedside manners, not the more skilled ones. If this is true, it could also apply to many other aspects of life.

Receiving medical care can be a very vulnerable moment. Engaging with a consultant can also be a vulnerable moment since companies don't hire consultants when things are going well. Sometimes, the project that a client is sponsoring could mean a promotion or maybe the last ditch effort before they leave the company.

So, what is the client experiencing?

Do they feel listened to?

Do they feel understood?

Do they feel like they can trust their judgment?

I learned this lesson through a unique project. My consulting company has three teams working for a client organization. My team was the highest performer, and the client liked it best, but it wasn't because of me. It was because of my teammate.

At the time, I believed that delivering amazing client results was doing lots of great technical work. I put my head down, and I worked furiously to drive results. On the other hand, my teammate was not as ambitious; however, he did stop and strike up conversations with my client. He found out they both had military experience and loved Asian cuisine. Later, we ended up going to dinner with our client at a hole-in-the-wall yet amazing Korean BBQ place, and it was amazing.

I remember thinking, "Wow, what am I? Chopped liver? My client loves my team because of my teammate. I don't think it has anything to do with me."

My teammate just genuinely cared about the person in front of him. My client felt listened to and probably felt like they could talk openly when we had to make tough decisions at work. With time, after more dinners and lunches, we got to hear more stories and context from our clients. The relationship was growing into a friendship, and he likely felt very understood.

Wouldn't it be better to go through a vulnerable moment with someone you can trust?

Building a relationship with a client is very important.

14. Practice how you ask questions. The tone is so important. Avoid sounding combative. You’re trying to reduce the chances of people getting defensive.

You can't control how people feel. You can control how you say things in a way that minimizes the chances of someone being triggered and raising their defenses.

For example,

Consultant: Why are you doing it that way? That's not best-practice.

vs.

Consultant: Hmm. I am not tracking. Help me understand. What's the value of doing it this way one more time? I fear that we might not reach the project's goal with this approach.

When I think about how to ask questions, I think of two things:

• I want my question to come from a place of curiosity.
• I want to avoid the word "you."

To ask from a place of curiosity, I often have to take a deep breath, put myself in the client's shoes, be empathic for them, and assume that they are trying their best given their circumstances.

Also, by avoiding using the word "you" forces me to talk about my experience. No one can challenge how I feel or what I have experienced. It is my truth.

Don't get me wrong. I am still me, and I am still a person with flaws. I always have moments where I think to myself, "what the f*?." The difference is that when those feelings arise, I acknowledge them and take a moment to think about the person in front of me.

Side note: I was fortunate to work with an organizational psychologist at a previous consultancy. She trained me and fellow consultants to avoid certain words so we could speak more collaboratively.

We had this posted in our office so that we would remember to use the vocabulary.

15. When people lose their cool and start yelling, you can be the voice of reason.

As a consultant, you probably don't get hired into totally functional companies. From time to time, you might be caught in the middle of an internal political battle where someone ultimately loses their temper. That's a moment where you can add a lot of value.

Here's a recount of something that happened to me:

Person 1: We can't design the network this way. It leaves XYZ vulnerable. We'd need another ABC service and another firewall.

Person 2: Ah ok. But this part of the design is ok, right?

Person 1: No, it's not. We can't link these networks together because XYZ is vulnerable.

Person 2 gets frustrated because he wants to go live with the project.

Person 2: We need to start doing things in a new way! We can't be doing data center crap in the cloud!

Yelling continues, then stops for a second.

Me: Yeah, Person 1, I understand what you're saying. What if we get rid of the issue? Because this is a greenfield deployment, could we say that workloads that don't meet these criteria won't be able to be hosted here? That way, we can ship and don't have to design for all possible use cases, especially legacy ones.

Honestly, I can't tell you if this conversation moved them closer to the goal. I can tell you from more than one account that sometimes groups bring you in because they feel pain points in "dealing with another team." They want someone to help them work with another team. So, if I didn't try to be a mediator, I would have had zero chance of moving the outcome forward. If they weren't working well together before they hired me, then they will probably not magically begin to collaborate, right?

The best thing you can do is try; if it doesn't work, it's not any worse than before.

16. You don’t need the client to like you. They don’t need to be your friend. Say the hard things.

I have to tell myself this all the time. I want people to like me, and I enjoy working with people that are having a good time. But, sometimes, what the client needs is not a friend. Sometimes, they need to hear what is wrong or what could go wrong - directly and clearly.

"If we don't do this soon, I fear we'll not have time to go to production."

"I know the burndown chart says we're tracking to finish the project; I strongly feel that we will need more time than our board says."

"I know you have spent a lot of time on your solution. I want to present you with the alternatives so you know our stance and the surprises that might come up. I would hate for you to continue your current path without knowing the risks. Ultimately, it's your decision, and if you choose to continue on this path, you know the risks."

So don't avoid saying the hard thing because you are afraid they won't like you anymore. Also, remember, watch your tone!

17. As a consultant, you have no control. It’s all influence.

Early in my career, learning that I had no control was probably one of the biggest lessons that took me the longest to learn. I kept getting frustrated working with some clients because they were not listening fully to my advice. I had to learn that I have no control over what the client decides or does.

I have no "skin in the game." I don't work for the company that they work for. I probably won't get fired if this project does not go well. So, it's only fair that if they hold the risk, then they decide, right?

Instead, the only tool you have is influence.

How do you influence?

Think of social media influencers like YouTubers. What do they do?

• They share information in easily digestible ways.
• They tell you about other products with honest reviews.
• They share their experience and what's worked for them.
• They connect you with other great ideas and people.

All those things apply, minus the YouTube channel.

18. Do not word vomit. Use "top-down" communication.

Have you ever sat through a painful presentation where an engineer is presenting to upper management, and management seems disinterested? Was it painful? Did it seem like it was dragging on?

McKinsey is a world-renowned management consulting company. McKinsey uses "top-down" communication. Instead of telling a long story and finally providing the key point at the end, present the key point first and then explain.

This blog post is an example of that kind of communication. The key points are delivered as headings. The points might not make sense unless you read the explanations like this one. But, you have the option to determine if the key point is something you're interested in, and I'm not holding your attention hostage.

I think this blog post in story form would be much more difficult to read. You would have to read everything to get the twenty-eight lessons I am sharing.

As an engineer, you might cringue, but I find PowerPoints very useful. I find them useful because they're a tool that allow me to help break down complex ideas into small chunks and communicate them effectively. For example:

• When creating PowerPoints, I create a slide to share the main recommendations in the beginning of the presentation.
• I try to keep six items on the slide at once. Items include images and words. When I do this, it typically leads to people paying more attention to what I am saying because they are not caught reading slides. It also forces me to explain the key point and focus on it.

If you want to learn more about the McKinsey method of communication, here's a great video from an ex-McKinsey consultant.

19. Who are you talking to? IT doesn't want more work, and developers want to ship more.

Unless you have decades of industry experience and managed not to become a manager, you probably don't have experience being a professional system administrator and developer. So, you likely only have experience in the Dev or the Ops in DevOps. If you're a junior DevOps engineer, you have a serious disadvantage.

Part of playing politics and reaching agreements is figuring out what different people want to achieve. Very generally speaking:

• Developers want to ship more value. Developers at large enterprises they're often frustrated with all the processes that prevent them from shipping and how little control they have over infrastructure, even in the cloud. I was a developer at a large enterprise, and I felt this pain point deeply.
• IT wants less work to do. They include system administrators, identity admins, network engineers, infrastructure engineers, cloud admins, etc. They're usually underwater with work and often on call when things go wrong. They're responsible for large infrastructure and don't want to be called at night. Making things standardized makes their job easier. Also, if they're extremely busy, they will sometimes opt for tried and true technology and avoid new trends. Lastly, they also really care about top-down governance because they don't want people they don't know to cause more work for them.
• Security - They feel pressure to keep the company safe from breaches. There are generally different specialties within security. For example, some security engineers focus on network security, some on monitoring and alerting, and some on application security. They know that they are often seen as the folks that halt projects.

If you have a different perspective on each group of people care about, that's ok. However, let's leave out security for the following metaphor; otherwise, it is too complicated.

Imagine that your role as a DevOps enabler is to be a bridge for "work." At one end of the bridge, there's "development," and at the other end, "operations." You have made your customers happier when you ship one unit of "work" across the bridge.

So, to become an effective bridge, you must know a bit about "development" and "operations." If you started your career as a developer, then you probably have gaps in operations, as I did. If you started your IT career, you probably have development gaps. If you started your career as a junior DevOps engineer, you probably have gaps in both realms. The key here is to know enough about each role so that you can speak their language. Being able to speak about what someone cares about is a great way to build trust. The more you can speak in someone's language, the more they will think, "you are one of us." And in dysfunctional environments that need the most help, there's usually an "us vs. them" mentality. No pressure. The better bridge you can be, the better advisor you will be.

So truly, here are some of my standard tactics to build more trust:

• Go out of your way to add guardrails on behalf of IT. Guardrails are often policies that prevent people from doing insecure or unapproved things. Examples of guardrails are Azure Policies, HashiCorp Sentinel, and AWS Policies. After adding guardrails, you can then turn around and try to give developers more autonomy because there are guardrails that disallow them to do bad things.
• Go out of your way to increase security before the security engineers say something. By increasing security preemptively, you can gain favor and help design a solution that is not a hindrance.
• Show developers Kubernetes, Infrastructure as Code, and Application Performance Monitoring (APM) tools. They will be often be interested in doing little bit of "ops" work so that they can ship more.

20. You are a doctor

How would you interact with people if you thought of yourself as a doctor?

Would you try to find the root cause of a symptom instead of pushing pills first?

In this case, pills are tools. If you push tools all the time, you're not going to solve the root problems. Sometimes organizations require a lifestyle change, which means a process or culture change.

So, think about the care plan for the organization. It could start with a tool to alleviate some profound pain but then think about process improvements to solve the deeper issues.

21. You have a hidden force acting against your relationship with the client. It's hourly billing.

Consulting companies are businesses. Most consulting companies charge their clients by the hour. There is usually a margin between what they pay the consultant and what they charge the client. It's that simple.

So, because of the billing method, a consulting company makes more money when:

• Projects take longer, even if they are delayed.
• Solutions are more complex because they will take more time and/or people.
• Client organizations cannot maintain a solution, so they must continue hiring the consulting organization.

I'll be honest. I find these to be extremes. I wouldn't work for a consulting organization that's doing these practices on purpose to make more money.

And, humans can be exceptionally good at smelling intent.

Also, we humans can be very sensitive to perceived ill intent. Think used car salesman.

For example, one time, I was working with an excellent client. We had a great relationship. They had asked for a follow-up engagement and given me a list of items to help them with. However, it was in the middle of a heavy sales process change, and I needed approval from general managers and go through a "deal review." I was so focused on getting approval and fighting the spreadsheets that I lost sight of the hourly rate we were charging the client before the process changed. I was trying to get approval so we could start soon, but the new rate was considerably higher. We sent the proposal to the client; they expressed that they had sticker shock (the price was extremely high). The relationship ended.

I believe that they thought that because we did a good job and they were asking for more help, we were being bold and raising the rates because they needed us. In reality, I was scoping what they had asked for and made a mistake carrying the proposal through the heavy-handed sales process.

The lesson to be learned is to remember what your company is incentivized to do and consider how your client will feel. Don't overcomplicate solutions, be transparent about delays, communicate early and often, and be mindful that a client must maintain your solution, show urgency, etc.

I'll leave you with this: imagine you hired a good plumber to fix a leaky pipe and he was charging you $150/hour. How would you feel is he was taking his sweet time and moving slow and looking ineffective? If you had a low budget, I think that would give anyone a little anxiety.

22. Virtual desktops over laptops

If you're a consultant, you might like this one. Request a virtual desktop instead of a laptop when starting a project that's less than a year. A virtual desktop allows you to easily switch between your company's and the client's communications without needing a fancy KVM.

Side note: here's a product idea: a very fancy KVM for consultants. Before giving up and accepting virtual desktops, I vigorously searched the market for the perfect KVM that would be supported by MacOS and Windows so that I could keep my peripherals plugged in one place and switch between computers effortlessly.

Consultants often have to get access to the client's systems to perform the work. Many companies have security postures that don't accept using the consultant's workstation, hence, bringing their own device. Instead, many enterprises like to provide laptops to contractors so that they can connect to the network from a device they can trust and control.

Well, if you're a consultant, these devices can pile up. I have an ugly laptop that's been collecting dust next to me for six months after I finished the project. 😊

I was once doing a project with a client that also hired AWS Professional Services (AWS consulting). I overheard the AWS project manager tell the client that using virtual desktops instead of laptops was part of their contract. I found this interesting because it took me two months to get set up with a laptop. It took the AWS consultants two weeks to get set up. Then, I started asking my clients for virtual desktops for short/medium projects.

By using VDIs, I was relieved that I didn't have to agonize anymore about having all this clutter and physical devices everywhere. Also, starting the project with virtual desktops is often much faster than with laptops.

I hope this doesn't happen to you, but I once had to make a round-trip of seven hours to go to a location to get a proprietary badge and then drive another nine hours round trip to pick up a laptop. The badge unlocked the laptop. Then, another time, I had to drive an hour each way, sit outside a client's branch location at a Chinese fast food place, get on the client's Wi-Fi, then talk to the IT help desk for two hours to get my awesome laptop set-up.

23. Juggling multiple emails is a necessary evil

Another tip for a new consultant.

On top of getting a new laptop from clients, we are often given an email address by the client. So on top of our actual work email addresses, we end up with all these other identities and inboxes for each client.

At the beginning of my consulting career, I tried hard to solve this problem. The reality is that many enterprises lock down any customizations to their emails so that you can't use external OAuth applications. Heck, I had one client that monitored their emails so closely that I was strongly warned that if I ever forwarded an internal email to an external account, an automated system would pick it up and fire me. I was also strongly advised not to print anything because I would be automatically fired.

You will have multiple emails and identities. Just accept it. It's the consulting tax. 😀

24. Windows is sometimes a necessary evil

Last tip for an existing consultant, especially one that hates Windows. 😄

To be clear, I like Windows, even though I'm in the DevOps field. I'm writing this blog post from a Windows 11 laptop. 🤗

However, I wouldn't say I like Windows as a typical enterprise setting, and that's because of all the controls and bloatware that enterprises install. I bet everyone hates slow laptops and funky issues due to bloatware.

Just like you can ask for a virtual desktop, you can also ask for what operating systems they support. Rarely do enterprises support Linux for workstations. Sometimes, you might get lucky, and an enterprise supports MacOs. Most of the time, enterprises only support Windows.

25. KISS is relative. Simplicity is clear; it's not lazy.

Many engineers like to say Keep It Simply Stupid (KISS). I agree. But I think simplicity is relative.

What you might find simple, I might not find simple. What I could find simple, the client might find complicated. What the client might find simple, I might find an abomination.

I think simple takes work. I don't think arriving at the first solution is often the "simple" one. Doing the least amount of work can be "lazy." So, I'm going to do this... I'm going to give you a Steve Jobs quote. 😀

“Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains.” - Steve Jobs

There have been times that I worked hard to make a solution "simple" for clients. Here are some of the attributes I'm looking for:

• Simple is clear - When the client reviews the solution, does it have an "aha!" moment? Will it click? Many consulting solutions go to die because when the client looks at the solution, they find it too complex and don't have the time to reverse engineer it.
• Simple is maintainable - Assuming that the client will work to learn the foundations of technology, will they know enough to maintain and extend it? One time, I had a client demanding that we undo a CI/CD pipeline written in YAML because they wanted to use the old and comfortable GUI-based pipelines in Azure DevOps. I think this was lazy on the client's part because they wanted to do what they've always done instead of exploring an industry practice. But, if our client had learned the essentials of building a YAML-based pipeline, could they grab our pipeline and understand it enough to extend it?
• Simple is not a Rube-Goldberg machine - Is your solution starting to feel like a Rube-Goldberg machine? Maybe, it's simple on its face but complicated behind the scenes. For instance, I was reviewing an application that my client had built, three simple pages. It was a user sign-up, a login, and the main page. Behind the scenes, there were like twenty microservices! It took a lot of cognitive load to understand what was happening in the backend. Oof.

26. Avoid designing the bike shed

I learned this one from a co-worker recently. I think it applies to a lot of organizations.

What is described in this Wikipedia article, and what I have experienced, is that often we can easily focus on trivial things. To "avoid designing the bike shed" means that if you're designing something important like a nuclear power plant, then don't place as much importance on designing the bike shed next to the power plant.

Recently, I was on a project to design a very important enterprise-scale landing zone for the cloud, meaning the cloud foundation. The client was interested in discussing naming conventions and spending time on them. Naming is hard. I get it. So, I'd rather use a naming convention that a reputable source created rather than waste precious mental energy creating my naming convention that I'll probably get wrong.

Be aware of this for yourself, your teammates, and your clients. You might be stuck discussing something that will not improve the nuclear power plant.

27. Confidence is different from assertiveness. Confidence comes from practice.

Have you met someone that seems like they're exuding confidence? I'm not that person, and I've always envied that to some extent.

Instead, I'm naturally quite the opposite. I have a lot of self-doubts, and I double-guess myself. So, I've asked co-workers, "you sound so confident; how do you do it?."

Sometimes, some people may sound confident, but inside, they are still scared. Their physiological responses are ones that you might not see. For example, when I get nervous, you can tell. My ears turn red, I start to sweat a lot, and I start to stutter and think slower. So, some people might be nervous, yet they don't show those clear signs, and then you might make up a story in your mind that they're confident because they sound so assertive.

The true solution that I found that works for me is practice. The more I do something, the more confident I get. So, I accept it and give myself permission to be nervous when I'm not great at something yet. If I keep showing up and doing, I will eventually get better and be much more confident.

Here's a book that helped me and might help you: The Confidence Gap: A Guide to Overcoming Fear and Self-Doubt.

28. How you treat others is a reflection of how you treat yourself

Do you remember a time that you worked with someone that was really critical?

Maybe they weren't criticizing you, but you were listening to how they criticizing someone else. I've come to know someone with this tendency, and when I hear what they think of themselves, it's rough. I wouldn't want to be in their brain. There's a lot of stress and hate, so if they're hard on others, they're likely even harsher on themselves.

Watch how you show up with others. If you're passing judgment, you're likely to pass a lot of judgment on yourself. Be kind to yourself.

Also, it works the other way around. When working with someone overly critical of you, try thinking about how much harder they probably are on themselves. If I hear harsh criticism from someone, I find it useful to think about their inner struggle. It helps me center myself and lower my defenses because I realize they're also struggling.

Thank you for your attention!

I know there's so much content out there. I'm honored if you've read some or all of this article. Remember, I'm just a random dude from the internet, and some of my lessons might or might not help you.

TLDR; I wrote a Terraform composite action that you can just plug into your GitHub workflow to make this happen. Jump ahead to Putting it all together section.

Background

Here's the thing with GitHub - it favors automation running from pull requests and issues. For example, with other systems, Azure DevOps and GitLab, engineers can create "stages" to be able to separate environments and promotion through environments. And while GitHub has "environments" and "stages," I don't find them as effective as the other competing platforms. Even if it did, I like the pull-request workflow because it addresses some of the challenges I experienced at scale.

At scale, I ran into the following challenges:

• Accountability - For companies not interested in paying for Terraform Cloud, having other engineers consistently review the Terraform plans can be difficult. Making engineers go to a pipeline log to view the Terraform plan reduces the experience and increases the chance that someone will blindly approve without looking closely at the plan.
• Really large Terraform plans - While very much discouraged, sometimes they happen. Big bang Terraform deployments are sometimes unavoidable on the first promotion to an environment.

Note: This content was inspired by the README of the setup-terraform action and experiences in the real world.

What guide is not

• A beginner guide on Terraform
• An introduction to how to create GitHub Workflows/Actions
• A how-to on creating a simple pipeline that deploys to Azure

The process

This process uses a trunk-based branching strategy. Environments like "dev", "test", and "prod" are isolated via folders and there's a single long-lived branch, main.

.
├── production
│   └── main.tf
└── staging
    └── main.tf

Why folders? I find it the most elegant, and it's a whole topic. It also works well with a pull-request workflow. If you wonder about the advantages of separating different environments via folders vs. branches vs. workspaces, then Yevgeniy Brikman recently wrote an in-depth blog post series on this topic.

So, here's what the process would look like.

1. When someone opens a pull request, a GitHub workflow will kick off and run terraform plan.
2. When someone merges the pull request against main, the same GitHub workflow will kick off and run both terraform plan and terraform apply.
3. Depending on what files under which environment folder were changed, a different GitHub workflow will trigger. For instance, if someone changes the Terraform code inside the staging environment folder, it will trigger the staging GitHub workflow for the staging environment.

Note: Terragrunt will help you keep code DRY, but it's certainly not a requirement to use Terragrunt for environmental folders.

Writing the GitHub Actions

1. The trigger

When someone changes a Terraform file in the environment folder, trigger the appropriate action. You'll need a GitHub workflow file per environment.

# .github/workflows/staging.yaml
name: 'Terraform - Staging'

on:
  push:
    branches:
    - main
    paths: 
    - staging/**
  pull_request:
    branches:
    - main
    paths: 
    - staging/**

2. Create a GitHub environment

Under the settings of the repository you're using, create an environment like "Staging." Then, if you're deploying to Azure, you can create an Azure Service Principal and set secrets like CLIENT_ID and CLIENT_SECRET for the new environment. By having secrets scoped to the environment, you can use separate service principals per environment.

If you are not deploying to Azure, you can create whatever secrets your Terraform provider needs to authenticate.

3. Set the permissions for GITHUB_TOKEN

The following sets the permissions for the GITHUB_TOKEN to be able to add comments to the pull requests. You have to add this to your GitHub workflow file, which can be at the same level as on and jobs. The good news is that this configuration avoids you having to create a GitHub PAT.

# .github/workflows/staging.yaml
permissions:
  contents: read
  pull-requests: write

4. Set-up Terraform Action

GitHub-hosted runners don't have Terraform installed out of the box, and you'll want to install it on each workflow run. Even if you use self-hosted runners, you still might want to consider using this action so that each repository can fully control what version of Terraform they are using.

Also, in the following snippet, we're using the "Staging" environment we created.

# .github/workflows/staging.yaml
jobs: 
  terraform:
    runs-on: ubuntu-latest
    name: Terraform
    environment: staging 
    steps:
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
      with:
        terraform_version: 1.1.9
        terraform_wrapper: false

Note: terraform_wrapper is set to false. This flag means there won't be a wrapper that highjacks the STDOUT from the Terraform commands. We'll still get fancy to add the Terraform plan to the pull-request comments.

5. Terraform Init

This step requires that you have a Terraform backend configured. Otherwise, your pipeline won't work correctly since you will dispose of the Terraform state on each run. Here's how to set up a Terraform backend using Azure.

# .github/workflows/staging.yaml
- name: Terraform Init
  id: init
  run: terraform init
  env:
    ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }} # This doesn't have to be a secret. Client IDs are not sensitive.
    ARM_CLIENT_SECRET: ${{ secrets.CLIENT_SECRET }}
    ARM_SUBSCRIPTION_ID: "[Your Subscription ID]"
    ARM_TENANT_ID: "[Your Tenant ID]"
  working-directory: ./staging # Use your environment folder
  shell: bash

6. Terraform Plan

We're getting closer to the good part. There are some important arguments that we provide to Terraform when we "plan":

• -input=false - This flag tells Terraform to fail if there are some missing variable values that were not specified. Otherwise, Terraform will stop the pipeline because it will ask the user (that's not there) for a variable value.
• -no-color - This flag tells Terraform not to be cute and show colors. Otherwise, you'll see funky characters in the output stream of the GitHub actions.
• -out tf.plan - This tells Terraform to write the plan to a local file. It will be used by the terraform apply and terraform show.

# .github/workflows/staging.yaml
- name: Terraform Plan
  id: plan
  run: |
    terraform plan -input=false -no-color -out tf.plan
  env:
    ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }} 
    ARM_CLIENT_SECRET: ${{ secrets.CLIENT_SECRET }}
    ARM_SUBSCRIPTION_ID: "[Your Subscription ID]"
    ARM_TENANT_ID: "[Your Tenant ID]"
  working-directory: ./staging # Use your environment folder
  shell: bash

7. Terraform Show

The show command simply replays the information shown from terraform plan. During this action, we will pipe the output of terraform show to a local temporary file.

# .github/workflows/staging.yaml
- name: Terraform Show
  id: show
  run: terraform show -no-color tf.plan 2>&1 > /tmp/plan.txt
  working-directory: ./staging # Use your environment folder
  env:
    ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }} 
    ARM_CLIENT_SECRET: ${{ secrets.CLIENT_SECRET }}
    ARM_SUBSCRIPTION_ID: "[Your Subscription ID]"
    ARM_TENANT_ID: "[Your Tenant ID]"
  shell: bash

8. Write the plan to the pull-request

In this step, we use actions/github-script@v6 to create the GitHub pull-request comment. The action runs open-ended scripts that you create in JavaScript. The contents of the pull-request comment come from the temp file created in the earlier step.

# .github/workflows/staging.yaml
- uses: actions/github-script@v6
  if: github.event_name == 'pull_request'
  with:
    github-token: ${{ inputs.GITHUB_TOKEN }}
    script: |
      const fs = require("fs");
      const plan = fs.readFileSync("/tmp/plan.txt", "utf8");
      const maxGitHubBodyCharacters = 65536;

      function chunkSubstr(str, size) {
        const numChunks = Math.ceil(str.length / size)
        const chunks = new Array(numChunks)
        for (let i = 0, o = 0; i < numChunks; ++i, o += size) {
          chunks[i] = str.substr(o, size)
        }
        return chunks
      }

      // Split the Terraform plan into chunks if it's too big and can't fit into the GitHub Action
      var plans = chunkSubstr(plan, maxGitHubBodyCharacters); 
      for (let i = 0; i < plans.length; i++) {
        const output = `### ${{ inputs.plan-title }} Part # ${i + 1}
        #### Terraform Format and Style 🖌\`${{ steps.fmt.outcome }}\`
        #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
        #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`
        <details><summary>Show Plan</summary>
        \`\`\`\n
        ${plans[i]}
        \`\`\`
        </details>
        *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Working Directory: \`${{ inputs.working-directory }}\`, Workflow: \`${{ github.workflow }}\`*`;   

        await github.rest.issues.createComment({
          issue_number: context.issue.number,
          owner: context.repo.owner,
          repo: context.repo.repo,
          body: output
        })
      }

Lastly, if you look closely, there are some magic variables like maxGitHubBodyCharacters and a helper function chunkSubstr. These are my learnings from working with Terraform, specially really large plans. There is a known discussion that reveals that there is a limit to how big a comment can be, and it's tied to the underlying GitHub comment schema in their database. So, the maxGitHubBodyCharacters is the limit of characters allowed by GitHub's database, and chunkSubstr function breaks any huge comments into smaller parts.

Here's what the actual pull-request comment would look like.

The line of code that says <details><summary>Show Plan</summary> is the toggle "Show Plan" in the pull-request comment.

9. Terraform Apply

In this step, we will "apply" the plan file created during terraform plan. Notice there is a conditional if operator, so the apply only happens when the GitHub workflow runs from the main branch. If you use a different branch name like master, you would have to adjust this GitHub action.

# .github/workflows/staging.yaml
- name: Terraform Apply
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'
  run: terraform apply -input=false tf.plan
  env:
    ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }} 
    ARM_CLIENT_SECRET: ${{ secrets.CLIENT_SECRET }}
    ARM_SUBSCRIPTION_ID: "[Your Subscription ID]"
    ARM_TENANT_ID: "[Your Tenant ID]"
  working-directory: ./staging # Use your environment folder
  shell: bash

Putting it all together

Combining all these actions for each repository and environment can be overwhelming. So, I created a GitHub composite workflow that you can reuse. If you cannot use it because of organizational policies, you can copy the action source code.

And, here's how you would use it.

# .github/workflows/staging.yaml
name: 'Terraform - Staging'

on:
  push:
    branches:
    - main
    paths: 
    - staging/**
  pull_request:
    branches:
    - main
    paths: 
    - staging/**

permissions:
  contents: read
  pull-requests: write

jobs: 
  terraform:
    runs-on: ubuntu-latest
    name: Terraform
    environment: staging 
    steps:
      - uses: actions/checkout@v3
      - uses: boxboat/terraform-azure-actions@v1.0.0
        with:
          client-id: '<< service principal client id >>'
          client-secret: ${{ secrets.CLIENT_SECRET }}
          default-subscription-id: '<< some subscription id >>'
          tenant-id: '<< some aad tenant id >>'
          working-directory: './staging'
          plan-title: 'Staging :star:'
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          terraform-version: 1.1.9

That's it! 🎉

The Caveats

Here are the caveats that you should be aware of.

• Race conditions - There won't be anything preventing you from trying to deploy conflicting changes simultaneously. Let's say you have two pull requests against the same environment, then merge them one after the other. The GitHub actions will trigger twice at roughly the same time. In the best-case scenario, one of the pipelines fails to acquire the terraform lock on the backend provider. You'll have to be diligent about which pull request and the other in which you merge them.
• Large plans - As mentioned earlier in this post, large Terraform plans are never fun to work with. There is a limit on the size of the GitHub comment you can write to issues and pull requests. Therefore, to have a reliable workflow, your GitHub action should account for large Terraform plans and try to break up the plan into multiple comments. An alternative to the solution covered in this post is to trim the terraform plan when it gets past a specific size.
• Noise in pull-requests - While performing with a big bang deployment, we had to update pull-request multiple times. So, because each plan created about 20 comments, there was a lot of noise. A co-worker didn't like this, so he deleted old and irrelevant comments before each Git push. If you prefer to update the comment, check out the README for the setup-terraform action.

Conclusion

I love Terraform. Some of my coworkers call it "terrorform" because of the quirky challenges at scale. If companies don't pay for a solution like Terraform Cloud, adopt Atlantis, or use other paid solutions, they will have to engineer solutions like the ones discussed in this post. Hopefully, you found this helpful. Feel free to leave a comment below if you have questions. You can also tweet me at @gaunacode.

What this guide is not

• An introduction to Terraform

• An introduction to Terragrunt

• An introduction to Azure

Instead, this guide will cover how to set up an opinionated Terragrunt "live" repository so that you can deploy across multiple subscriptions, regions, resource groups, and application environments.

If you're looking for an overview of Terraform and how to use it with Azure, there's a great lab by HashiCorp that's free.

If you're looking for an intro to Terragrunt, the Terragrunt docs is a great place to start.

📕 Also, Terragrunt makes it easier to implement some of the ideas called out in the Terraform Up and Running book.

The demo

Here's a repository I used to create a demo to deploy to Azure.

Here's a backup link in case the fancy card stops working 👆🏽.

The directory structure looks like this.

 .
├── terragrunt.hcl
└── vs-enterprise 
    ├── staging 
    │   ├── eastus
    │   │   ├── region.hcl
    │   │   └── spoke-vnet
    │   │       └── terragrunt.hcl
    │   ├── env.hcl
    │   ├── global
    │   │   ├── region.hcl
    │   │   └── resource_groups
    │   │       ├── terragrunt.hcl
    │   └── westus
    │       ├── region.hcl
    │       └── spoke-vnet
    │           └── terragrunt.hcl
    └── subscription.hcl

This example is heavily inspired by the AWS terragrunt demo.

So, if you want to follow along, clone the repository, then I'll walk you through customizing to your scenario.

Okay, about the file structure

For me, the folder/file structure is one of the most appealing aspects of Terragrunt. I like that someone has put thought into and shared their experience on promoting Terraform across multiple environments for many large-scale applications.

• terragrunt.hcl - We'll talk about this last

• vs-enterprise - A folder for my Visual Studio Enterprise subscription

• vs-enterprise/staging - A folder for my staging environment

• vs-enterprise/staging/eastus - A folder for resources in the East US region

Let's walk through all the folders.

First, the subscriptions 🔑

Some organizations have separate subscriptions per workload and per environment. Some organizations just have a "non-production" and a "production subscription."

Therefore, you will want to create as many folders as you have subscriptions. As you'll see later, this will help you switch subscriptions by simply changing the directory from which you run terragrunt apply. I like it because it is more obvious to see which subscription I am deploying to.

So, if you have a "non-production" and "production," then you're going to have these folders in your repository.

.
├── azsub-non-production
└── azsub-production

If you have a more complex subscription strategy, like subscription per app and environment, then you're going to have more folders.

.
├── azsub-fabrikam-app-dev
├── azsub-fabrikam-app-prod
└── azsub-fabrikam-app-test

Please note: I pretend to use a fancy naming convention for my subscription. If you have messy subscription names, I think it's ok to have messy subscription folders.

Lastly, you'll want to create a subscription.hcl file in each subscription folder. In here, add the following contents. Make sure to add the subscription_id to each subscription file.

# vs-enterprise/subscription.hcl
locals {
    subscription_id = "<< your subscription id >>"
}

Next, the environment 📦

Like always, it depends on how many environments your applications will have. You could have just "staging" or "production" environments, or you could have "dev", "test", "prod."

I create a shared or management folder for resources that don't necessarily correspond to an environment. For example, you might have a Log Analytics workspace that's shared across all non-prod environments.

In my example, I am using a single environment - staging.

But then, in each environment folder, create an env.hcl file. Make sure to change the environment for each env.hcl file.

# vs-enterprise/staging/env.hcl
locals {
  environment = "staging"
}

The region 🌎

Depending on your architecture, create a folder for each region. I make a global folder for resources that don't necessarily correspond to a region.

Examples of global resources are resource groups, Azure Front Door, Azure DNS, Azure Traffic Manager, Log Analytics workspace, etc. If you're a person that doesn't like to have resources from different regions inside of the same resource group, then you don't have to consider it a "global" resource.

For my example, if I am deploying to multiple regions, then my folders could be:

• East - A spoke virtual network

• West - Another spoke virtual network

• Global - Create the resource groups necessary for the virtual networks. More on dependencies later.

In each region folder, create a region.hcl file.

# vs-enterprise/staging/eastus/region.hcl
locals {
  location = "eastus"
}

But what about the global folder? You'll still want to set a region. Most Azure global resources still require that you pick a region, for example, resource groups and Azure WAN. So, pick a "default" region for all of these global resources.

Finally, the Terragrunt file 📄

You will have to break up your system architecture into smaller pieces. This is one of the strengths of Terraform/Terragrunt. So, find "seams" in your architecture where you break things up into "modules." This takes some experience and practice and time.

Once you have the modules, you create the Terragrunt configuration. So, in my example, I will be deploying a virtual network.

So, I created a directory called spoke-vnet under the East and West US regions. Then, I created a Terragrunt.hcl file.

# vs-enterprise/staging/eastus/spoke-vnet/terragrunt.hcl
terraform {
  source = "tfr:///Azure/vnet/azurerm//?version=2.6.0"
}

include {
  path = find_in_parent_folders()
}

dependencies {
  paths = ["../../global/resource_groups"]
}

dependency "resource_groups" {
  config_path = "../../global/resource_groups"

  mock_outputs = {
    vnet_resource_group_name = "rg-terragrunt-mock-001"
  }
  mock_outputs_merge_with_state = true
}

locals {
  env_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
  environment = local.env_vars.locals.environment

  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  location = local.region_vars.locals.location
}

inputs = {
  vnet_name           = "vnet-spoke-${local.environment}-${local.location}-001"
  resource_group_name = dependency.resource_groups.outputs.vnet_resource_group_name
  address_space       = ["10.0.0.0/16"]
  subnet_prefixes     = ["10.0.1.0/26", "10.0.2.0/24", "10.0.3.0/24", "10.0.4.0/24"]
  subnet_names        = ["AzureBastionSubnet", "Management", "Tools", "Workloads"]
  location            = local.location

  tags = {
    environment = local.environment
  }
}

Then, I created another Terragrunt.hcl file in my westus folder. The contents are mostly the same with the exception of the address spaces (inputs.address_space and inputs.subnet_prefixes).

A couple of important things to call out:

• The source - Under the terraform block, there's the opportunity to configure the source of the Terraform module. In my example, I use a public Terraform module to create a virtual network.

• The include - We're saying to include the first terragrunt.hcl file in the parent directories. Because our other Terragrunt files like subscription.hcl don't follow the default name, they will be ignored with this statement. Later, we'll create a terragrunt.hcl file at the repository's root. 😊 When we create a terragrunt.hcl file in the root of the repository, then this include statement will discover it.

• The inputs - The inputs are the input variables passed into the module. Notice how I am using locals to build some inputs.

• The locals - Much like proper Terraform, you can have locals. These variables are truly local to the file. You won't be able to consume locals defined in the parent directories like the subscription.hcl or env.hcl. However, you can force this behavior by loading them explicitly:

locals {
    env_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
    environment = local.env_vars.locals.environment
}

• The dependencies - Remember how we never created a resource group for our virtual network? Well, we want to use Terraform to create the resource group as well. The problem is that we can't write proper Terraform inside of a Terragrunt configuration. We could create a resource_group.tf next to the terragrunt.hcl file and the resource group would be created, but there's no way to refer to the attributes of that resource or do a depends_on from Terragrunt file to Terraform configuration. Instead, the annoying part is that we'll have to create a separate module for just the resource groups to develop an inter-module dependency. Notice that there are two kinds of blocks: dependencies and dependency blocks The dependencies block is only required to specify the order of dependencies for when you use terragrunt run-all. The dependency block declares the actual dependency and refers to outputs from the depending module. For example, to get the name of the resource group that hasn't been created, we refer to the resource group module by calling dependency.resource_groups.outputs.vnet_resource_group_name.

Alright, fine, let's create the resource-group module. 😬

Create resource-groups folder. I placed mine under the global region. Here, you will define the resource groups that you want to create and deploy your infrastructure into.

Note: This pattern of creating a module also applied to me when I wanted to use data blocks since you can't use data blocks in your Terragrunt configuration. In this example, I am not using data elements.

# vs-enterprise/staging/global/resource_groups/resource_group.tf
resource "azurerm_resource_group" "vnet_resource_group" {
  name     = var.vnet_resource_group_name
  location = var.location
}

Then you need some variables.

# vs-enterprise/staging/global/resource_groups/variables.tf
variable "location" {
  type  = string
  description = "The Azure Region to use"
}

variable vnet_resource_group_name {
  type        = string
  description = "The name of the vnet resource group"
}

And, some outputs so that you can create inter-module dependencies.

# vs-enterprise/staging/global/resource_groups/outputs.tf
output vnet_resource_group_name {
  value = var.vnet_resource_group_name
  depends_on = [azurerm_resource_group.vnet_resource_group]
}

Lastly, a Terragrunt configuration file.

# vs-enterprise/staging/global/resource_groups/terragrunt.hcl
include {
  path = find_in_parent_folders()
}

inputs = {
  vnet_resource_group_name = "rg-example-vnet"
}

Dude, what about the backend? And... the provider?

Yes, I know. We haven't talked about where the Terraform state will be stored. Bad news. This part is a little complicated.

To define a Terraform backend, we could go into each terragrunt.hcl file for each component, environment, subscription, and region. But, that would mean copying and pasting code. Instead, we could define the backend configuration in one place.

As promised, remember this from our terragrunt.hcl file?

# vs-enterprise/staging/eastus/spoke-vnet/terragrunt.hcl
include {
  path = find_in_parent_folders()
}

Well, Terragrunt will be looking for another terragrunt.hcl file in the parent directories. So let's define it at the root of your repository.

# terragrunt.hcl
locals {
  # Automatically load subscription variables
  subscription_vars = read_terragrunt_config(find_in_parent_folders("subscription.hcl"))

  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))

  # Automatically load environment-level variables
  environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))

  location              = local.region_vars.locals.location
  environment       = local.environment_vars.locals.environment
  subscription_id   = local.subscription_vars.locals.subscription_id
}

# Generate Azure providers
generate "versions" {
  path      = "versions_override.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
    terraform {
      required_providers {
        azurerm = {
          source = "hashicorp/azurerm"
          version = "2.95.0"
        }
      }
    }

    provider "azurerm" {
        features {}
        subscription_id = "${local.subscription_id}"
    }
EOF
}

remote_state {
    backend = "azurerm"
    config = {
        subscription_id = "${local.subscription_id}"
        key = "${path_relative_to_include()}/terraform.tfstate"
        resource_group_name = "rg-terragrunt-example-001"
        storage_account_name = "stterragruntexample001"
        container_name = "environment-states"
    }
    generate = {
        path      = "backend.tf"
        if_exists = "overwrite_terragrunt"
    }
}

# Configure root level variables that all resources can inherit. This is especially helpful with multi-subscription configs
# where terraform_remote_state data sources are placed directly into the modules.
inputs = merge(
  local.subscription_vars.locals,
  local.region_vars.locals,
  local.environment_vars.locals
)

Did you shed a tear? 😢 Yes, it's ugly and it looks complicated. I'll walk you through it.

• The locals - The key to understanding the locals is the context where this terragrunt.hcl file will be run from. It always runs from the context of the leaf folder (the module folder). So, when we deploy our spoke-vnet module with Terragrunt, it will load the parent terragrunt.hcl file. Then, it will try to load subscription locals from a Terragrunt config file called subscription.hcl located in one of the parent directories.

• The generate block - Essentially, you will generate a block to specify the provider for all the leaf Terraform modules. This means one place to update the provider version! Notice how you can use local references to select which subscription you want to deploy to. Depending on which folder the "leaf folder" is located in, it's going to use a different subscription.

• The remote state - Here we tell Terragrunt to generate backend configuration and place it in the backend.tf file for each leaf folder.

• Input magic - Remember how we were adding locals blocks in the subscription.hcl and other non-default Terragrunt files? Essentially, we will be reading all of these local variables and merging them as inputs. So, let's say that all of your Terraform modules had an input variable of location for the Azure region. We won't have to specify location = eastus a million times with this input magic.

Almost there! 🙏🏾 We just need a few things on the Azure side.

Wait ✋🏽, some prep work

We defined a storage account for our backend and we haven't created it yet.

First, create a resource group for the Terraform backend.

• Name: rg-terragrunt-example-001

• Location: eastus

Next, create the storage account that will hold the Terraform backend. You will also need a storage account container.

• Name: stterragruntexample001

• Location: eastus

• Redunancy: LRS

• Container Name: environment-states

Also, remember that our code is creating the resource group for the virtual network? Well, if we do a terragrunt run-all plan, it will fail since no resource groups exist yet. So, let's create a "mock" resource group.

• Name: rg-terragrunt-mock-001

• Location: eastus

In reality, this "mocking" resource group will only be used when new infrastructure is being created for the first time. This is unique to Azure because, in AWS, resource groups are not required to create resources. This is also unique to Terragrunt for Azure because we can't add raw Terraform code in a Terragrunt configuration file. I guess the only way to avoid this issue is to create resource groups from within the modules themselves. But, creating resource groups within modules doesn't sound effective either since we would have less control over the tags at the resource group level.

Lastly, in that storage account, create a private container called environment-states. This is where the Terraform states will go. Don't worry; environments will be separated through sub-folders.

Ready to run it?

This part is so cool. Set the current directory of your shell to the region folder. Then run terragrunt run-all plan.

Terragrunt will find all the terragrunt.hcl files and then run terragrunt plan from each folder (It won't work from the root of the repository).

Oh my...

❯ terragrunt run-all plan
INFO[0000] The stack at /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise will be processed in the following order for command plan:
Group 1
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/global/resource_groups

Group 2
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/eastus/spoke-vnet
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/westus/spoke-vnet


Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.vnet_resource_group will be created
  + resource "azurerm_resource_group" "vnet_resource_group" {
      + id       = (known after apply)
      + location = "eastus"
      + name     = "rg-example-vnet"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + vnet_resource_group_name = "rg-example-vnet"

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_subnet.subnet[0] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.1.0/26",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "AzureBastionSubnet"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-westus-001"
    }

  # azurerm_subnet.subnet[1] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.2.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Management"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-westus-001"
    }

  # azurerm_subnet.subnet[2] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.3.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Tools"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-westus-001"
    }

  # azurerm_subnet.subnet[3] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.4.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Workloads"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-westus-001"
    }

  # azurerm_virtual_network.vnet will be created
  + resource "azurerm_virtual_network" "vnet" {
      + address_space         = [
          + "10.0.0.0/16",
        ]
      + dns_servers           = []
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "eastus"
      + name                  = "vnet-spoke-staging-westus-001"
      + resource_group_name   = "rg-terragrunt-mock-001"
      + subnet                = (known after apply)
      + tags                  = {
          + "environment" = "staging"
        }
      + vm_protection_enabled = false
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + vnet_address_space = [
      + "10.0.0.0/16",
    ]
  + vnet_id            = (known after apply)
  + vnet_location      = "eastus"
  + vnet_name          = "vnet-spoke-staging-westus-001"
  + vnet_subnets       = [
      + (known after apply),
      + (known after apply),
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_subnet.subnet[0] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.1.0/26",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "AzureBastionSubnet"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-eastus-001"
    }

  # azurerm_subnet.subnet[1] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.2.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Management"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-eastus-001"
    }

  # azurerm_subnet.subnet[2] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.3.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Tools"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-eastus-001"
    }

  # azurerm_subnet.subnet[3] will be created
  + resource "azurerm_subnet" "subnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.4.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "Workloads"
      + resource_group_name                            = "rg-terragrunt-mock-001"
      + virtual_network_name                           = "vnet-spoke-staging-eastus-001"
    }

  # azurerm_virtual_network.vnet will be created
  + resource "azurerm_virtual_network" "vnet" {
      + address_space         = [
          + "10.0.0.0/16",
        ]
      + dns_servers           = []
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "eastus"
      + name                  = "vnet-spoke-staging-eastus-001"
      + resource_group_name   = "rg-terragrunt-mock-001"
      + subnet                = (known after apply)
      + tags                  = {
          + "environment" = "staging"
        }
      + vm_protection_enabled = false
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + vnet_address_space = [
      + "10.0.0.0/16",
    ]
  + vnet_id            = (known after apply)
  + vnet_location      = "eastus"
  + vnet_name          = "vnet-spoke-staging-eastus-001"
  + vnet_subnets       = [
      + (known after apply),
      + (known after apply),
      + (known after apply),
      + (known after apply),
    ]

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

But, wait, should you use run-all all the time?

No, the Terragrunt team comments that they originally designed this capability to stand up an entire infrastructure stack from scratch. Once the infrastructure is up, they recommend using terragrunt plan and apply at the individual modules.

Alright, do it. Apply. 🤯

I love this part. Run terragrunt run-all apply -auto-approve.

❯ terragrunt run-all apply -auto-approve
INFO[0000] The stack at /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise will be processed in the following order for command apply:
Group 1
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/global/resource_groups

Group 2
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/eastus/spoke-vnet
- Module /Users/facundo/Repos/terragrunt-azure-example/vs-enterprise/staging/westus/spoke-vnet

Are you sure you want to run 'terragrunt apply' in each folder of the stack described above? (y/n) y
azurerm_resource_group.vnet_resource_group: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # azurerm_resource_group.vnet_resource_group has changed
  ~ resource "azurerm_resource_group" "vnet_resource_group" {
        id       = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet"
        name     = "rg-example-vnet"
      + tags     = {}
        # (1 unchanged attribute hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

vnet_resource_group_name = "rg-example-vnet"
azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001]
azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001]
azurerm_subnet.subnet[3]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Workloads]
azurerm_subnet.subnet[2]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Tools]
azurerm_subnet.subnet[0]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/AzureBastionSubnet]
azurerm_subnet.subnet[1]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Management]
azurerm_subnet.subnet[3]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Workloads]
azurerm_subnet.subnet[2]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Tools]
azurerm_subnet.subnet[0]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/AzureBastionSubnet]
azurerm_subnet.subnet[1]: Refreshing state... [id=/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Management]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # azurerm_subnet.subnet[0] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/AzureBastionSubnet"
        name                                           = "AzureBastionSubnet"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[1] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Management"
        name                                           = "Management"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[2] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Tools"
        name                                           = "Tools"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[3] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Workloads"
        name                                           = "Workloads"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_virtual_network.vnet has changed
  ~ resource "azurerm_virtual_network" "vnet" {
        id                      = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001"
        name                    = "vnet-spoke-staging-westus-001"
      ~ subnet                  = [
          + {
              + address_prefix = "10.1.1.0/26"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/AzureBastionSubnet"
              + name           = "AzureBastionSubnet"
              + security_group = ""
            },
          + {
              + address_prefix = "10.1.2.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Management"
              + name           = "Management"
              + security_group = ""
            },
          + {
              + address_prefix = "10.1.3.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Tools"
              + name           = "Tools"
              + security_group = ""
            },
          + {
              + address_prefix = "10.1.4.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Workloads"
              + name           = "Workloads"
              + security_group = ""
            },
        ]
        tags                    = {
            "environment" = "staging"
        }
        # (7 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

vnet_address_space = tolist([
  "10.1.0.0/16",
])
vnet_id = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001"
vnet_location = "eastus"
vnet_name = "vnet-spoke-staging-westus-001"
vnet_subnets = [
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/AzureBastionSubnet",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Management",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Tools",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-westus-001/subnets/Workloads",
]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # azurerm_subnet.subnet[0] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/AzureBastionSubnet"
        name                                           = "AzureBastionSubnet"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[1] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Management"
        name                                           = "Management"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[2] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Tools"
        name                                           = "Tools"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_subnet.subnet[3] has changed
  ~ resource "azurerm_subnet" "subnet" {
        id                                             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Workloads"
        name                                           = "Workloads"
      + service_endpoint_policy_ids                    = []
      + service_endpoints                              = []
        # (6 unchanged attributes hidden)
    }

  # azurerm_virtual_network.vnet has changed
  ~ resource "azurerm_virtual_network" "vnet" {
        id                      = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001"
        name                    = "vnet-spoke-staging-eastus-001"
      ~ subnet                  = [
          + {
              + address_prefix = "10.0.1.0/26"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/AzureBastionSubnet"
              + name           = "AzureBastionSubnet"
              + security_group = ""
            },
          + {
              + address_prefix = "10.0.2.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Management"
              + name           = "Management"
              + security_group = ""
            },
          + {
              + address_prefix = "10.0.3.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Tools"
              + name           = "Tools"
              + security_group = ""
            },
          + {
              + address_prefix = "10.0.4.0/24"
              + id             = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Workloads"
              + name           = "Workloads"
              + security_group = ""
            },
        ]
        tags                    = {
            "environment" = "staging"
        }
        # (7 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

vnet_address_space = tolist([
  "10.0.0.0/16",
])
vnet_id = "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001"
vnet_location = "eastus"
vnet_name = "vnet-spoke-staging-eastus-001"
vnet_subnets = [
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/AzureBastionSubnet",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Management",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Tools",
  "/subscriptions/<< redacted subscription id >>/resourceGroups/rg-example-vnet/providers/Microsoft.Network/virtualNetworks/vnet-spoke-staging-eastus-001/subnets/Workloads",
]

That's it! All resources were created. You'll notice that the output looks like someone iterated through all the folders and ran terraform apply.

Summary 🙏🏾

As I was learning Terragrunt, I wanted to like it. I am starting to, and I think I will end up liking it more. There were some quirks that are unique when using Azure. For example, AWS regions are king and therefore the AWS Terraform Provider has to be configured with the region. In Azure, subscriptions are king and it's wise to configure the Azure Terraform provider with the subscription ID. Also, Azure resource groups are mandatory to deploy resources and you have to make some weird decisions around when you create your resource groups and whether you pass them as inputs to your infrastructure modules. Lastly, don't forget that you might have to make a data module to query Azure AD for some Object IDs.

If you followed, this guide, I hope you found it useful. Leave a comment below, especially if you have some tips and tricks too. I would love to hear from you. 😊

Background

One of the most straightforward ways to deploy Azure is through a service principal.

az login --service-principal -u <aad app registration client id> -p <<aad app client secret>> --tenant <<aad tenant id>>
# now you can invoke az commands

The problem with this is:

• You have to store that secret in your CI/CD system safely. Sure, you could integrate with KeyVault, but your mileage may vary depending on your CI/CD system.
• Typically, client secrets expire. When creating the client secret, you can choose varying expiration timeframes, but you have to keep track of when these secrets expire. It's annoying when deployments stop working because the service principal has passed. And, if you don't choose an expiration, you're opening yourself up for a security incident.

Alternatively, you can authenticate with a service principal using a certificate.

az login --service-principal -u <aad app registration client id> -p <<certificate>> --tenant <<aad tenant id>>
# now you can invoke az commands

The problem with this is:

• You have to use self-hosted runners. And, you have to install the certificates on those runners. And, certificates do expire, ugh.

Azure Active Directory and GitHub OIDC Integration!

There's documentation on both the GitHub and the Microsoft side. I found the documentation on the Microsoft side to be a little more useful.

I used this walkthrough as described by Microsoft.

The app registration with federation

I won't re-document everything. From Azure AD, you create an app registration like you would normally do. Then instead of setting up the client secret, you set up a federated credential.

Sign in to the Azure portal. Go to App registrations and open the app you want to configure.

Go to Certificates and secrets. In the Federated credentials tab, select Add credential. The Add a credential blade opens.

In the Federated credential scenario drop-down box select GitHub actions deploying Azure resources.

Specify the Organization and Repository for your GitHub Actions workflow.

For Entity type, select Environment, Branch, Pull request, or Tag and specify the value. The values must exactly match the configuration in the GitHub workflow. For more info, read the examples.

Add a Name for the federated credential.

The Issuer, Audiences, and Subject identifier fields autopopulate based on the values you entered.

Click Add to configure the federated credential.

Here's my parameters:

• Organization - super-duper-github-oidc-pipelines-demo
• Repository - secretless-amazing-pipelines-to-azure
• Entity type - Environment
• GitHub environment name - Sandbox
• Name - sandbox_subscription

💡 What's neat about this step is that the person performing the registration does not need access to GitHub at all! Great for those enterprise scenarios where you have to do a screen share with an AAD admin.

Next, grant this new app registration Reader role to the Azure subscription that you'll be using. In reality, you will probably need a different role based on what you're trying to automate. But, this does mean that you need to have Owner role of the subscription or User Access Administrator. If you're neither of these things, you'll need someone else to help you. If you need help with this, here's the docs.

Now, to the GitHub side of things.

The GitHub side of things

Under my repository, I configured an environment called Sandbox. To add a new environment you go to Repository Settings, Environments, then add the new environment.

Next, we'll create a GitHub action that leverages the OIDC capability. To verify that we can login to Azure, we'll add a step to list all the resource groups in the subscription.

So, start with a simple workflow.

Now replace the contents with the following pieces.

The next bits of the process has been derived from the GitHub Action for Azure Login README.

First, you need to grant permissions to the action to read the OIDC ID Token from Azure AD. Add a permissions block at the workflow level (at the same indentation level as the name of the workflow).

permissions:
  id-token: write
  contents: read

And, you'll want to create a simple job that deploys to the Sandbox environment.

jobs:
  login:
    runs-on: ubuntu-latest

    environment: Sandbox

    steps:
      - uses: actions/checkout@v2

Next, you'll want a step your action to login to Azure. Notice how it doesn't use a secret. 🙃

- name: 'Az CLI login'
   uses: azure/login@v1
   with:
     client-id: <<your client id for the app registration you created above>>
     tenant-id: <<the tenant id for aad>>
     subscription-id: <<the subscription id>>

Then, we'll add a step to run a test command to verify that we can authenticate.

- name: Test Command 1
   shell: bash
   run: 
     az group list -o table

Putting it all together

name: Login to Azure

permissions:
  id-token: write
  contents: read

on:
  push:
    branches: [ main ]

jobs:
  login:
    runs-on: ubuntu-latest

    environment: Sandbox

    steps:
      - uses: actions/checkout@v2

      - name: 'Az CLI login'
        uses: azure/login@v1
        with:
          client-id: <<your client id for the app registration you created above>>
          tenant-id: <<the tenant id for aad>>
          subscription-id: <<the subscription id>>

      - name: Test Command 1
        shell: bash
        run: 
          az group list -o table

Then, notice how your mind melts with awe. 🤯

Pre-Requisites

You’ll need:

• An AKS cluster. Here’s a Gist I use to create a simple cluster.
• Helm 3
• Azure CLI
• A Custom DNS/Access to the provider (I will be using Azure DNS)

NGINX Ingress Controller with Let’s Encrypt

I was inspired by this page on the Microsoft docs. We’re going to install the NGINX ingress controller with Let’s Encrypt.

Note: To simplify things, I will use the NGINX image from Docker Hub. The Microsoft Docs walk you through importing the NGINX image into your own container registry, and… for good reason.

Install NGINX Ingress Controller

First, add the NGINX Helm Repository.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Then install the Helm chart,

helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress --create-namespace\
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux

Install Cert Manager

Now, install the cert-manager Helm chart. Notice: you will have to add the Jetstack helm repository first.

CERT_MANAGER_TAG=v1.3.1

# Label the ingress-basic namespace to disable resource validation
kubectl label namespace ingress cert-manager.io/disable-validation=true

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
  --namespace ingress \
  --version $CERT_MANAGER_TAG \
  --set installCRDs=true \
  --set nodeSelector."kubernetes\.io/os"=linux

And lastly, create a CA cluster issuer. This will tell cert-manager to issue certificates for certificate requests across the cluster.

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@mydomain.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux
EOF

Install Argo CD

Now, for the good part, let’s install Argo CD with a TLS ingress.

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

For example: argocd.yourdomain.com You can get the Public IP of the NGINX Ingress controller by:

$ kubectl get svc -n ingress
NAME                                               TYPE           CLUSTER-IP   EXTERNAL
nginx-ingress-ingress-nginx-controller             LoadBalancer   10.1.0.88    20.120.123.218   80:30101/TCP,443:31968/TCP   5m13s         443/TCP                      5m13s

Next, you’re going to deploy the Ingress rules to be able to reach ArgoCD’s UI using HTTPS. Make sure you add an A record on your DNS to the ingress controller.

Note: Change the code snippet below to include your custom domain.

cat << EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # If you encounter a redirect loop or are getting a 307 response code
    # then you need to force the nginx ingress to connect to the backend using HTTPS.
    #
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.mydomain.com
    secretName: argocd-secret # do not change, this is provided by Argo CD
EOF

Open your browser with your custom address for Argo. For example: https://argocd.mydomain.com. Note: it can take about 5 minutes for the Let’s Encrypt certificate to be assigned.

Congrats 🎉

To login, get the initial password via a Kubernetes secret. The username is admin

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Login to Argo. Then change the default password.

I hope that helped! 🎊

Pre-Requisites

• An AKS cluster using Azure CNI as the network plugin

• A virtual network with two subnets asg and aks. The asg subnet will hold the Application Gateway, and the aks subnet will hold the AKS cluster.

• A resource group, in this case rg-appg-ingress-test

• Your AKS cluster's identity will need Network Contributor role scoped to the virtual network

Deploying NGINX ingress controller with a private IP

During this section, we’ll borrow instructions from this MSFT docs page.

First, we need a private IP address that the NGINX ingress controller will accept requests from. So, choose a private IP address and verify that it’s available. In this case, the IP address I chose is 10.0.0.100.

az network vnet check-ip-address --name vnet-ingress-test -g rg-appg-ingress-test --ip-address 10.0.0.100

Create the following file named internal-ingress.yaml with the chosen IP address.

controller: 
  service: 
    loadBalancerIP: 10.0.0.100 
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

This will be used to configure the NGINX Helm chart to use the given IP address for the ingress controller.

# Create a namespace for your ingress resources 
kubectl create namespace ingress

# Add the ingress-nginx repository 
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# Use Helm to deploy an NGINX ingress controller 
helm install nginx-ingress ingress-nginx/ingress-nginx \
  --namespace ingress \
  -f internal-ingress.yaml \
  --set controller.replicaCount=2 \
  --set controller.nodeSelector."kubernetes.io/os"=linux \
  --set defaultBackend.nodeSelector."kubernetes.io/os"=linux \
  --set controller.admissionWebhooks.patch.nodeSelector."kubernetes.io/os"=linux

It may take a few minutes for the IP address to be assigned.

$ kubectl --namespace ingress get services -o wide -w nginx-ingress-ingress-nginx-controller 
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 
nginx-ingress-ingress-nginx-controller LoadBalancer 10.0.75.105 80:31026/TCP,443:30254/TCP 2m8s app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

If it's stuck in <pending>, then likely, the Ingress controller can't reserve the static IP. Therefore, you're most likely missing the Network Contributor role on the AKS cluster's identity.

You can verify this by looking at the events for the service. bash kubectl --namespace ingress get services -o wide -w nginx-ingress-ingress-nginx-controller

Deploying two test applications

Let’s create a namespace for the sample apps.

kubectl create ns ingress-test

Deploy the first sample application.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld
  namespace: ingress-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld
  template:
    metadata:
      labels:
        app: aks-helloworld
    spec:
      containers:
      - name: aks-helloworld
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld
  namespace: ingress-test
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld
EOF

Now, deploy the second test application in the same namespace.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-demo
  namespace: ingress-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-demo
  template:
    metadata:
      labels:
        app: ingress-demo
    spec:
      containers:
      - name: ingress-demo
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-demo
  namespace: ingress-test
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: ingress-demo
EOF

Okay, create the ingress definition to verify part of the ingress controller. This will create an ingress rule to map requests for that domain to the appropriate service in Kubernetes.

export DOMAIN_NAME="mysub.mydomain.com"

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  namespace: ingress-test
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: aks-helloworld
      port:
        number: 80
  rules:
  - host: $DOMAIN_NAME
    http:
      paths:
      - backend:
          service:
            name: aks-helloworld
            port:
              number: 80
        path: /hello-world-one
        pathType: Prefix
      - backend:
          service:
            name: ingress-demo
            port:
              number: 80
        path: /hello-world-two
        pathType: Prefix
EOF

Notice: how there’s an annotation of nginx.ingress.kubernetes.io/ssl-redirect: "false". This will ensure that if we ever assign a TLS certificate to the ingress definition, NGINX won't start re-routing http traffic to https. This is important for us since we are using an Application Gateway that will terminate SSL and we want this behavior to be enforced by the Application Gateway, not the ingress controller. Otherwise, it might mess up with the health probes from the Application Gateway to the Kubernetes cluster.

Testing the ingress controller from a test container

Now, let’s launch a pod to validate the configuration.

kubectl run -it --rm aks-ingress-test --image=debian --namespace ingress-test

You will be inside the container at this point. Then install curl, we’re going to use curl against our ingress controller.

apt-get update && apt-get install -y curl

Once curl is installed on the container, then let’s make sure that our ingress is working.

curl -L -H "Host: ingress.gaunacode.com" http://10.0.0.100

Notice how we’re changing the Host header to ensure our ingress route is used. We haven’t created an external DNS record or configured the application gateway yet.

If everything works well, you should see the HTML with “Welcome to Kubernetes”. That’s one of our sample applications.

root@aks-ingress-test:/# curl -L -H "Host: ingress.gaunacode.com" http://10.0.0.100

<form id="form" name="form" action="/"" method="post"> Welcome to Azure Kubernetes Service (AKS)

Preparing an Application Gateway

If you don’t have one, here’s how to create a Standard_v2 application gateway. It pre-configures some aspects like the backend. It uses the private IP of the ingress controller.

az network public-ip create -g rg-appg-ingress-test -l eastus -n pip-appg-ingress-test --sku Standard

az network application-gateway create --name azappg-appg-ingress-test -g rg-appg-ingress-test -l eastus --sku Standard_v2 --public-ip-address pip-appg-ingress-test --vnet-name vnet-ingress-test --subnet appg --servers 10.0.0.100

Configuring the Application Gateway (Manually)

Next, grab the public IP of the application gateway and create an external DNS record.

In my case,

[A record @ ingress.gaunacode.com] -> 20.72.158.202

Modify the default HTTP settings and override the hostname. This will ensure that our ingress rule is used in the NGINX ingress controller.

Verify that ingress works. I’m my case, http://ingress.gaunacode.com. HTTPS should not work yet.

Create an HTTPS listener. I am using a trial certificate for the ingress.gaunacode.com domain.

Next, create a “Rule” to tie the HTTPS listener to the backend.

Then the “Backend Targets”

Once the “Rule” is created, then the App Gateway should accept traffic from the public IP, through the HTTP listener, tied to a “backend” using the Rule and http settings. The App Gateway creates a new connection to the NGINX ingress controller through a private static IP and overriding the “hostname” so that the Ingress rule kicks in.

That’s it! Hope that helped.

I recently had this scenario where I wanted to experiment with far fetched ideas on a cluster. However, I needed to use all the existing configurations. Because I was not using GitOps or extensive re-hydration CI/CD pipelines, I was able to use Velero to restore my Kubernetes configuration onto a new 1-node “sandbox” cluster.

Install Velero on new cluster

Assuming you already configured Velero on your main cluster, like this, then start by creating a namespace.

kubectl create ns velero

Then from your existing cluster, get the velero-credentials secret holding the connection information to your Azure storage account. For example,

# Against the main cluster (cluster #1)
kubectl get secret -o yaml velero-credentials -n velero > velero-credentials.yaml

Then switch to the new cluster and place it back.

# Against the new cluster (cluster #2)
kubectl apply -f velero-credentials.yaml   
rm velero-credentials.yaml 

Then install Velero on the new cluster. I used the same Helm command I used on the main cluster (cluster #1) some while back.

# Against new cluster (cluster #2)
export AZURE_SUBSCRIPTION_ID="" # to get current sub: az account show --query "id" -o tsv
export STORAGE_RESOURCE_GROUP="" 
export STORAGE_ACCOUNT=""
export STORAGE_CONTAINER_NAME=""

helm install velero vmware-tanzu/velero --namespace velero --version 2.13.2 \
--set "initContainers[0].image=velero/velero-plugin-for-microsoft-azure:v1.1.0" \
--set "initContainers[0].imagePullPolicy=IfNotPresent" \
--set "initContainers[0].volumeMounts[0].mountPath=/target" \
--set "initContainers[0].volumeMounts[0].name=plugins" \
--set "initContainers[0].name=velero-plugin-for-azure" \
--set credentials.existingSecret='velero-credentials' \
--set configuration.provider='azure' \
--set configuration.backupStorageLocation.bucket=$STORAGE_CONTAINER_NAME \
--set configuration.backupStorageLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.backupStorageLocation.config.storageAccount=$STORAGE_ACCOUNT \
--set configuration.backupStorageLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID \
--set configuration.volumeSnapshotLocation.name='azure-eastus' \
--set configuration.volumeSnapshotLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.volumeSnapshotLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID

This also assumes that you’re using Azure as the provider/backend for Velero.

Modify Velero’s backend to read-only

Find the backup location configured. It’s probably default.

# Against new cluster (cluster #2)
$ kubectl -n velero get backupstoragelocation.velero.io
NAME      PROVIDER   BUCKET/PREFIX                          PHASE       LAST VALIDATED                  ACCESS MODE
default   azure      backups-azaks-blah-blahh-staging-001   Available   2021-01-29 12:57:49 -0800 PST   ReadWrite

Now, unfortunately, there’s no CLI option on Velero 1.5.1 to edit a backup-location. So, let’s edit the CRD instead.

kubectl -n velero edit backupstoragelocation.velero.io <NAME>

Using Vim, add/set the spec.accessMode to ReadOnly.

Then, verify that the backup-location is ReadOnly instead of ReadWrite.

# Against new cluster (cluster #2)
$ velero get backup-locations                                                                                                                                                              
NAME      PROVIDER   BUCKET/PREFIX                          PHASE       LAST VALIDATED                  ACCESS MODE
default   azure      backups-azaks-blah-blahh-staging-001   Available   2021-01-29 12:57:49 -0800 PST   ReadOnly

Try a restore

Great! Now let’s try restoring from a scheduled backup. This is assuming that you have some scheduled backups on your main cluster (cluster #1). If not, you’ll have to skip this and create a manual backup.

# Against new cluster (cluster #2)
$ velero backup get                                                                                                                                                                        
NAME                            STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
every-day-at-1-20210129010007   Completed   0        2          2021-01-28 17:00:07 -0800 PST   29d       default            <none>
every-day-at-1-20210128010006   Completed   0        2          2021-01-27 17:00:06 -0800 PST   28d       default            <none>
every-day-at-1-20210127010006   Completed   0        2          2021-01-26 17:00:06 -0800 PST   27d       default            <none>
every-day-at-1-20210126010006   Completed   0        2          2021-01-25 17:00:06 -0800 PST   26d       default            <none>
every-day-at-1-20210125010005   Completed   0        2          2021-01-24 17:00:05 -0800 PST   25d       default            <none>
every-day-at-1-20210124010005   Completed   0        2          2021-01-23 17:00:05 -0800 PST   24d       default            <none>

Pick a backup and restore it.

velero restore create --from-backup every-day-at-1-20210129010007

Velero will also give you the command to check the progress. For example:

# Against new cluster (cluster #2)
$ velero restore describe every-day-at-1-20210129010007-20210129130618
Name:         every-day-at-1-20210129010007-20210129130618
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Started:    2021-01-29 13:06:19 -0800 PST
Completed:  <n/a>

Backup:  every-day-at-1-20210129010007

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

It took only about a minute or two. For me, there were a few warnings and some errors. It tried to restore itself, therefore, there were some warnings related to that. But, most of the deployments did not work because of the container registry. In my case, I did not add the ACR integration with my new cluster so many of the images failed to pull down. Once I “attached” the ACR to AKS, the deployments self-healed.

There might be unhealthy deployments

Expect some of the Kubernetes deployments to not work. In my case, the Rancher agent was not working because one with the same name and secret was already running in my main cluster (cluster #1). Therefore, Rancher server was denying my second agent.

Also, my NGINX ingress controller was not healthy. This was because when I configured it on my main cluster, I tied it to a public static IP on Azure. The new cluster’s identity did not have permissions to modify that static IP nor did I want it to.

That’s it. Thanks for reading.

Overview

I have a 4-node (1 master, 3 workers) Kubernetes cluster hosted on 4 Raspberry Pis 4 Model B with 4 GB of RAM. The nodes are running k3s - a lightweight distribution of Kubernetes for the edge. On my cluster, I am using the inlets project to expose some test web applications to the public internet without requiring public static IPs from my ISP.

The bill of materials

Here’s the list of things I bought. It all cost me $389.42.

• 1x wall-mountable power strip
• 4x patch ethernet cables
• 4x Raspberry Pis Model B w/ 4 GB RAM
• 4x power supplies for the Raspberry Pis
• 1x Raspberry Pi cluster case
• 4x Samsung SD cards

What’s the power strip for? I ran out of plugs in my old power strip. This one has 12 outlets and I could also mount it underneath my desk to hide most of the cables.

What I did not buy? An ethernet cable to connect the Pis to my network. I already had some. Also, I did not buy a switch, I already had one too. If you need one, something like this would work.

Why not using a USB charging hub? Alex Ellis recommended using the power supply. He mentioned there are several accounts of people experiencing their charging hubs to be “browning out.”

The assembly

It took me about 1.5 - 2 hours to put everything together. I stacked the Pis on the cluster case and tediously placed the fans. The fans are highly recommended and I configured them to be in quiet mode. They’re really quiet.

Then, I also spent significant time re-doing the wiring at my desk to have more outlets. This allowed me to have the cluster on my desk.

So plan accordingly and take your time.

Flashing the SD cards

I flashed all the SD cards with Rasberry Pi OS Lite 32-bit. I used the Raspberry Pi Imager.

It was really easy. However, as each SD card was flashed, I would re-plug it. I would make the following two changes:

• Enabled SSH by creating an empty file in the /boot directory called ssh. Read more here.
• I modified the cmdline.txt file in the /boot directory. I appended the cgroup_memory=1 cgroup_enable=memory commands . This is a requirement for k3s to run on Raspberry Pis.

Shaving the Yak

After each SD card was flashed and some of the pre-requisites were taken care of, it was time to do more even more prep work.

First, I booted up the Pis and observed that the blinky lights looked healthy. I then used nmap to find the private IPs for these new Pis.

nmap -sn 192.168.1.0/24

If they were working fine, the default hostname started with raspberry.

Then, I would SSH into each Pi and change the default password. I would also change the default hostname in /etc/hostname and the matching entry in the hosts file at /etc/hosts. I would also enable iptables since it’s another pre-requisite for k3s.

sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot

Once all the nodes rebooted, they had new hostnames and new DNS labels.

If you are OCD like me, you can make the pi-master DNS label/IP be the top pi of the stack, and each of the worker Pis can increment top to bottom. (i.e. from top to bottom of the stack: pi-master, pi-worker-1, pi-worker-2, and pi-worker-3).

Lastly, I would also copy my SSH key into each Pi via ssh-copy-id.

Installing k3s

I had limited success trying to install k3s through the official instructions. It was time-consuming and I had issues with the master node coming online. Unfortunately, I don’t recall the issues and I did not have enough time to look for the root cause of the issue (unhappy kids). I started over by re-flashing the SD cards and used the k3sup project by Alex Ellis.

Once k3sup was installed, then it was really simple to create a k3s server node. A server node is the Kubernetes master.

export MASTER="[your private ip of the master pi]"
k3sup install --ip $MASTER --user pi

After installation, I would ensure that the master was up and healthy by attempting a simple kubectl command. k3sup was also nice enough to place the Kubernetes config file into my working directory.

kubectl get nodes --kubeconfig k3s.yaml

After the master node was up, I moved on to the worker nodes.

k3sup join --ip 192.168.1.217 --server-ip $MASTER --user pi
k3sup join --ip 192.168.1.213 --server-ip $MASTER --user pi
k3sup join --ip 192.168.1.221 --server-ip $MASTER --user pi

Then, verified that they registered properly.

Installing inlets for ingress with a little Azure

I wanted to deploy something on my new cluster. But, there were a few gotchas with this. Typically, in a cloud environment, you’ll probably use a LoadBalancer service to expose to public IPs for your ingress controller to allow traffic into the cluster to direct it to some applications. In a home lab set-up, unless you buy public static IPs from your ISP, then you have dynamic public IPs that change outside your control. So, if you point your custom domain to the public IPs of your home router, then your custom domain might not work in the future. This is especially true if you wanted to take your Pis into the office or a presentation for a Meetup.

So there’s another open-source project, inlets, that essentially helps you create some cheap VMs on the cloud provider of choice, create/associate some public static IPs to those VMs, then install an agent on those VMs that talks to an operator running on your k8s cluster - effectively providing you with a public static IP for your cluster, securely.

To install inlets, the easiest way and the documented way is using arkade. To be honest, I don’t quite understand the value proposition of arkade. I haven’t found a strong use-case for it in the real world.

I had to log in to Azure and create an Azure service principal with the CLI. I saved the authentication info for the service principal into a temporary file outside of source control.

az login
az ad sp create-for-rbac --sdk-auth > /tmp/az_client_credentials.json

SUBSCRIPTION_ID=$(az account show | jq '.id' -r)

Then, create a Kubernetes cluster in your Pi cluster. It will contain the contents of the service principal authentication file.

kubectl create secret generic inlets-access-key --from-file=inlets-access-key=/tmp/az_client_credentials.json

Then, I installed “inlets” through “arkade”.

arkade install inlets-operator \
 --provider azure \
 --region eastus \
 --subscription-id=$SUBSCRIPTION_ID

Notice, how I used the Azure provider. The default inlets provider is Digital Ocean.

After a few minutes, I checked the status on the “traefik” ingress controller. Traefik comes installed by default with k3s.

kubectl get svc -n kube-system

There’s a traefik service in the kube-system namespace. This LoadBalancer service is used by the ingress controller. If everything worked fine, inlets should have provided a public IP to this service. Make note of it.

Deploying the test application

Once inlets was set-up, then I moved onto deploying a test application. One caveat with Raspberry Pi k8s clusters is the CPU architecture. Most Docker images support the default architectures of amd64. Raspberry Pis need the arm64 architecture; therefore, Docker images have to be built to target that architecture.

I won’t get into how to do this, but if you want to build your own Docker image that runs on Pis, then I found this blog post useful.

If you want, you can use the test application I created. The image is public on Docker Hub. Start by creating a test-application.yaml with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realworld-deploy
  labels:
    app: realworld
spec:
  replicas: 1
  selector:
    matchLabels:
      app: realworld
  template:
    metadata:
      labels:
        app: realworld
    spec:
      containers:
      - name: realworld
        image: fgauna12/realworld-react:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: realworld
  name: realworld-svc
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: realworld
---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: "realworld-ingress"
spec:
  rules:
    - host: pi.gaunacode.com
      http:
        paths:
          - path: /
            backend:
              serviceName: realworld-svc
              servicePort: 80

You’ll need a custom domain. Use that instead of mine in the Ingress.spec.rules[0].host portion.

Next, I create an A record to point to the traefik service’s public IP.

A record on pi.gaunacode.com -> [Azure Public IP created by inlets and used by Traefik service]

Once you created the A record, you’re ready to deploy the test app.

kubectl apply -f test-application.yaml

In a few minutes, I had my app running on my cluster and available on the internet! It will be addressable at the root domain without https. In my case, http://pi.gaunacode.com.

This example app is a dockerized version of this React front-end. It’s simply a static website driven by data from an API hosted by someone else.

Hope that helped. Don’t hesitate to reach out if you need help. Happy hacking.

In this post, I’ll guide you through acquiring a free certificate, then uploading it to Kubernetes, setting up the ingress controller with a static IP, configuring DNS, then defining the ingress route for a sample application, then verifying everything works.

Pre-requisites

• You have a domain or access to one.
• You can verify the ownership of the domain or have an SSL certificate and its private key.

First, the certificate.

If you don’t have a certificate, you can either use a self-signed certificate or use a free and trial certificate from a certificate authority. If you already have a certificate, you can skip to the next section.

Here’s how you can get a free certificate from Comodo, a popular certificate authority.

Go to https://ssl.comodo.com/free-ssl-certificate.

Choose the domain for your application. In my case, I will be using https://my-app.gaunacode.com.

Sign up for an account and fill out your information. You don’t have to provide your credit card.

View the certificate and choose “Setup ssl.comodo.com SSL Certificate.”

You will be asked for a Certificate Signing Request (CSR).

Here’s how to create one with Open SSL.

# Create the private key. You will need this later.
openssl genrsa -out my-app-gaunacode.key 2048

# Create a CSR from the private key
openssl req -new -key my-app-gaunacode.key -out my-app-gaunacode.csr

# View the CSR
cat my-app-gaunacode.csr

Copy the CSR and paste it into the textbox from Comodo.

Then, Comodo will ask you to choose a method to verify the ownership. I chose email verification. If you try CNAME verification, you might have trouble with it as I did. With email verification, I had to set-up an email distro at admin@gaunacode.com using my GSuite account.

Then, I verified my domain and received an email from Sectigo (Comodo).

Once I verified, I received my SSL certificate in an email.

Setting up the ingress

Now that you have a certificate, you can get to the fun part. First, create a public static IP using the Azure CLI. Create it on the same resource group where the node pools reside so that AKS has enough permissions to modify the resource. Alternatively, you can grant Network Contributor rights to the AKS service principal and create the public IP in any resource group of your choice.

az aks show --resource-group my-resource-group --name my-cluster --query nodeResourceGroup -o tsv

az network public-ip create --resource-group MC_my_nodepool_resource_group --name my-ip-name --sku Standard --allocation-method static --query publicIp.ipAddress -o tsv

The last command will give you the static public IP address. Copy it. In my case, it was 52.146.67.3.

Now, create a namespace to place the NGINX ingress controller.

kubectl create ns ingress

Then, deploy NGINX using a Helm chart. If your cluster is RBAC enabled with Azure AD, then set rbac.create=true. Choose a name for the DNS label on the public static IP address. I chose the same name as the Azure public IP resource. Set this DNS label on the last parameter (controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name")

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install nginx-ingress ingress-nginx/ingress-nginx \
    --set rbac.create=false \
    --namespace ingress \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.service.loadBalancerIP="[your public static ip address]" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="[dns label]"

Once the ingress controller deploys, it shouldn’t take long because you can check the deployment.

kubectl --namespace ingress get services -o wide -w nginx-ingress-ingress-nginx-controller

Verify that the External IP address should be the same as the public static IP.

NAME                                     TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
nginx-ingress-ingress-nginx-controller   LoadBalancer   10.0.64.221   52.146.67.3   80:31148/TCP,443:32641/TCP   13m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

Create an A record from your domain to the public IP. For example, I am using Netlify DNS. Give it some time for the DNS change to propagate.

Next, let’s verify that ingress works with two sample applications.

Create a namespace for the sample app we will use.

kubectl create ns ingress-test

Deploy the SSL certificate on the namespace as a Kubernetes secret. Notice how it’s deployed on the same namespace as the application to be deployed.

kubectl create secret tls aks-ingress-tls \
    --namespace ingress-test \
    --key my-app-gaunacode.key \
    --cert my-app_gaunacode_com.crt

Deploy the first sample application on the ingress-test namespace.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld
  namespace: ingress-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld
  template:
    metadata:
      labels:
        app: aks-helloworld
    spec:
      containers:
      - name: aks-helloworld
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"
---
apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld
  namespace: ingress-test
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld
EOF

Now, deploy the second sample application also on the same namespace.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-demo
  namespace: ingress-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-demo
  template:
    metadata:
      labels:
        app: ingress-demo
    spec:
      containers:
      - name: ingress-demo
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "AKS Ingress Demo"
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-demo
  namespace: ingress-test
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: ingress-demo
EOF

Lastly, deploy the ingress in the same namespace as the applications.

export DOMAIN_NAME="my-app.gaunacode.com"

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  namespace: ingress-test
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  tls:
  - hosts:
    - $DOMAIN_NAME
    secretName: aks-ingress-tls
  rules:
  - host: $DOMAIN_NAME
    http:
      paths:
      - backend:
          serviceName: aks-helloworld
          servicePort: 80
        path: /hello-world-one(/|$)(.*)
      - backend:
          serviceName: ingress-demo
          servicePort: 80
        path: /hello-world-two(/|$)(.*)
      - backend:
          serviceName: aks-helloworld
          servicePort: 80
        path: /(.*)
EOF

That’s it! Open the browser and navigate to the first app. The second application can be visited by adding /hello-world-two to the path.

Why?

Mean Time To Recover (MTTR)

You might have everything automated. From the cluster creation to the applications being deployed on the cluster. But, in my experience, this automation can take a long time. For example, it took me 45 minutes for all pipelines to deploy each of ~15 microservices to a cluster.

During a disaster, recreating new infrastructure and re-deploying all components can take time. Depending on the criticality of the incident and the importance of the app, it can feel like an eternity.

By using a tool like Velero to backup all Kubernetes resources, a cluster can be quickly restored to a certain state, lessening recovery time from ~45 minutes to ~15 minutes.

Avoiding Data Loss

If you have stateful applications deployed on your clusters, you will likely use persistent volumes to store data. For example, running a database on Kubernetes. So, tools like Velero can also help backup the data in persistent volumes as well.

Velero

Velero is an open source tool that helps backup and restore Kubernetes resources. It also helps with migrating Kubernetes resources from one cluster to another. Also, it can help backup/restore data in persistent volumes.

How it will work

Velero will have a pod running on the cluster, taking backups against an Azure Storage account (Although you can use other backends). You can take ad-hoc backups or create a schedule for them. But to do so, it has to be able to authenticate against the Azure API with enough permissions to be able to write to the storage account.

Velero has a CLI and a server side component to it. Either way, you’re going to have to install the CLI.

The CLI

Here are the basic steps on how to install the CLI on MacOs, Windows, or Linux.

The Server Side

For the server side component, there’s two main methods of installation:

• The Velero CLI
• A Helm Chart

Installing Velero

I prefer to use the Helm chart so that I can install Velero from a GitOps workflow. You can see the official instructions on the Helm chart’s readme These instructions below will be a little different so that it’s easier to automate.

Velero uses an Azure Plugin to interact with Azure. To authenticate, use Service Principals for now. Usually, I prefer using Azure Active Directory Pod Identities but there’s an open issue with Managed Identities. It’s a project that allows pods to authenticate against Azure using Managed Identities. In other words, by using pod identities (managed identities), you won’t need an API secret for Velero to authenticate against Azure. Remember though, managed identities only work on Azure.

The Credentials

Start by getting the resource group where the virtual machines for your AKS cluster. Velero will need permissions to this.

AZURE_RESOURCE_GROUP=$(az aks show -n mycluster -g myresourcegroup --query "nodeResourceGroup" -o tsv)

Then, get the Azure subscription and Azure AD tenant information.

AZURE_SUBSCRIPTION_ID=$(az account list --query '[?isDefault].id' -o tsv)
AZURE_TENANT_ID=$(az account list --query '[?isDefault].tenantId' -o tsv)

Now create an Azure service principal for Velero to authenticate. I like to name the service principal with the name of the cluster so that I know which Velero identity belongs to which cluster.

Note: you can also limit the scope of the service principal to certain resource groups --scopes so that it doesn’t have access to the entire subscription.

AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac --name "mycluster-velero" --role "Contributor" --query 'password' -o tsv)

Almost there… now, get the Client Id for the service principal just created.

AZURE_CLIENT_ID=$(az ad sp list --display-name "mycluster-velero" --query '[0].appId' -o tsv)

Create a file called credentials-velero. Do not commit this to source control.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Using kubectl, upload the credentials to a kubernetes secret in a new velero namespace.

kubectl create ns velero
kubectl create secret generic velero-credentials -n velero --from-literal="cloud=$(cat ./credentials-velero)"

The Storage Account

First, create a storage account, this will store the backups.

STORAGE_ACCOUNT='stmyapptest001'
STORAGE_RESOURCE_GROUP='rg-myapp-test-001'
az storage account create -n $STORAGE_ACCOUNT -g $STORAGE_RESOURCE_GROUP

Then, create the container to place the backups. I also like to name the container name after the cluster.

STORAGE_CONTAINER_NAME='backups-mycluster'
az storage container create --account-name $STORAGE_ACCOUNT -n $STORAGE_CONTAINER_NAME

Putting it all together

Now, we install Velero using Helm and using Velero’s Azure plugin.

First, add the VMware Tanzu Helm repo.

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

Lastly, install the helm chart.

helm install velero vmware-tanzu/velero --namespace velero --version 2.13.2 \
--set "initContainers[0].image=velero/velero-plugin-for-microsoft-azure:v1.1.0" \
--set "initContainers[0].imagePullPolicy=IfNotPresent" \
--set "initContainers[0].volumeMounts[0].mountPath=/target" \
--set "initContainers[0].volumeMounts[0].name=plugins" \
--set "initContainers[0].name=velero-plugin-for-azure" \
--set credentials.existingSecret='velero-credentials' \
--set configuration.provider='azure' \
--set configuration.backupStorageLocation.bucket=$STORAGE_CONTAINER_NAME \
--set configuration.backupStorageLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.backupStorageLocation.config.storageAccount=$STORAGE_ACCOUNT \
--set configuration.backupStorageLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID \
--set configuration.volumeSnapshotLocation.name='azure-eastus' \
--set configuration.volumeSnapshotLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.volumeSnapshotLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID

You should see Velero’s pods come up.

kubectl get pods -n velero

Testing the backup

Assuming you installed Velero’s CLI, you can simply create a test backup.

velero backup create my-backup

velero backup logs my-backup

Note: you might run into this issue. If so, then you have to exclude the admission webhook configuration when creating the backup.

velero backup create my-backup --exclude-resources MutatingWebhookConfiguration.admissionregistration.k8s.io
``

If it works, you can go ahead and create a schedule.

#### Setting up the schedule

"Back up my cluster every day at 7 am"

```bash
velero schedule create every-day-at-7 --schedule "0 7 * * *"

Note: again, you might run into this issue and if so then you’ll have to exclude the webhook admission configuration.

velero schedule create every-day-at-7 --schedule "0 7 * * *" --exclude-resources MutatingWebhookConfiguration.admissionregistration.k8s.io

That’s it!

Based on rough calculations, this approach could save you roughly 46% on costs.

Edit: As suspected, Microsoft released a way to start/stop AKS clusters through the CLI, you no longer need to stop the AKS scale sets yourself.

Option A (Preferred)

Microsoft released the preview feature as an Azure CLI extension. Follow the steps on this Microsoft docs page on how to install this extension. You can come back to this page to see how to schedule a script to start/stop a cluster during off hours using Azure Pipelines.

Option B (DIY)

Stopping the cluster

Create a bash script called aks-stop.sh.

#!/bin/bash

CLUSTER_NAME=$1
RESOURCE_GROUP=$2

NODE_RESOURCE_GROUP=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query "nodeResourceGroup" -o tsv)

az vmss list -g $NODE_RESOURCE_GROUP --query "[].name" -o tsv | while read -r scale_set
do 
    echo "Shutting down scaleset in the AKS resource group. Scale Set: $scale_set"
    az vmss deallocate -g $NODE_RESOURCE_GROUP -n $scale_set
done

To test it, login to Azure (az login) and the correct subscription (az account set -s [subscription id].

Invoke the script.

CLUSTER_NAME="[your cluster name]"
RESOURCE_GROUP="[resource group name]"

source ./aks-stop.sh "$CLUSTER_NAME" "$RESOURCE_GROUP"

Once the script runs, a kubectl command will show that the nodes are not ready.

$ kubectl get nodes
NAME                       STATUS     ROLES   AGE    VERSION
aks-default-1-vmss000000   NotReady   agent   5d2h   v1.18.6
aks-default-2-vmss000001   NotReady   agent   5d2h   v1.18.6
aks-default-3-vmss000002   NotReady   agent   5d2h   v1.18.6

What does it do?

It finds the resource group where the underlying vmss are for the AKS cluster. Then it iterates through all the vmss in the resource group and deallocates them.

If you have more than one node pool, it will stop all the node pools.

Start the vmss

Create a bash script called aks-start.sh.

#!/bin/bash

CLUSTER_NAME=$1
RESOURCE_GROUP=$2

NODE_RESOURCE_GROUP=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query "nodeResourceGroup" -o tsv)

az vmss list -g $NODE_RESOURCE_GROUP --query "[].name" -o tsv | while read -r scale_set
do 
    echo "Starting scaleset in the AKS resource group. Scale Set: $scale_set"
    az vmss start -g $NODE_RESOURCE_GROUP -n $scale_set
done

Invoke the script.

CLUSTER_NAME="[your cluster name]"
RESOURCE_GROUP="[resource group name]"

source ./aks-start.sh "$CLUSTER_NAME" "$RESOURCE_GROUP"

Then you will see shortly that the nodes are ready.

$ kubectl get nodes
NAME                       STATUS   ROLES   AGE    VERSION
aks-default-1-vmss000000   Ready    agent   5d2h   v1.18.6
aks-default-2-vmss000001   Ready    agent   5d2h   v1.18.6
aks-default-3-vmss000002   Ready    agent   5d2h   v1.18.6

Scheduling for off-hours

Here’s how to “stop” your AKS cluster after-hours and then start it in the morning.

Azure Automation accounts unfortunately don’t support the Azure CLI. I find using Azure DevOps very good for this since YAML pipelines support cron-style scheduled triggers.

For example, this is how to trigger a pipeline starting at 6 pm EST. (Note: Azure Pipeline schedules have to be in UTC). For more information on the format, here’s the detailed documentation.

pr: none 
trigger: none
schedules:
- cron: "0 23 * * 1-5"
  displayName: "After-hours (11 pm UTC)"
  always: true
  branches:
    include:
    - master

Another example, to trigger pipelines at 7 am EST:

pr: none 
trigger: none
schedules:
- cron: "0 11 * * 1-5"
  displayName: "Mornings (11 am UTC)"
  always: true
  branches:
    include:
    - master

Putting it together

Here’s an example of a pipeline starting the cluster each morning at 7 am EST.

pr: none 
trigger: none
schedules:
- cron: "0 11 * * 1-5"
  displayName: "Mornings (11 am UTC)"
  always: true
  branches:
    include:
    - master

variables:
  ClusterName: [name of cluster]
  ResourceGroup: [name of resource group]

steps:
  - task: AzureCLI@2
    inputs:
      azureSubscription: '[subscription service connection]'
      scriptType: 'bash'
      scriptLocation: 'scriptPath'
      scriptPath: './aks-start.sh'
      arguments: '"$(ClusterName)" "$(ResourceGroup)"'

Here’s an example of a pipeline stopping the cluster each morning at 7 pm EST.

pr: none 
trigger: none
schedules:
- cron: "0 23 * * 1-5"
  displayName: "After-hours (11 pm UTC)"
  always: true
  branches:
    include:
    - master

variables:
  ClusterName: [name of cluster]
  ResourceGroup: [name of resource group]

steps:
  - task: AzureCLI@2
    inputs:
      azureSubscription: '[subscription service connection]'
      scriptType: 'bash'
      scriptLocation: 'scriptPath'
      scriptPath: './aks-stop.sh'
      arguments: '"$(ClusterName)" "$(ResourceGroup)"'

There’s a project called Kured. It’s a daemonset that restarts nodes when it detects new updates. It can integrate with tools like Slack to notify you when a restart happens. You are also able to specify a schedule on when a restart is allowed.

Installing Kured through Helm

There’s a couple of ways to install Kured. My preferred way is through a Helm chart.

helm repo add kured https://weaveworks.github.io/kured
helm install my-release kured/kured

But, realistically you’re going to want to customize it by setting a schedule, alerts, or notifications.

Setting a schedule

You can configure Kured to only restart through certain windows. Useful to avoid downtime during peak hours.

  /usr/bin/kured \
  --reboot-days mon,tue,wed,thu,fri \
  --start-time 5pm \
  --end-time 12am \
  --time-zone America/New_York

Through a Helm chart, this becomes:

helm install kured kured/kured -n [some namespace for kured]\
--set configuration.rebootDays="{mon,tue,wed,thu,fri}" \
--set configuration.startTime=5pm \
--set configuration.endTime=11:59pm \
--set configuration.timeZone=America/New_York

Setting up Slack notifications

A good use of the Slack username is the name of the environment. In this example, I use staging. By the way, here’s some instructions on how to set-up Slack webhooks. Unfortunately, there’s no integration with Microsoft Teams.

helm install kured kured/kured -n [some namespace for kured]\
--set configuration.slackChannel=environments \
--set configuration.slackHookUrl=https://hooks.slack.com/services/blahblah \
--set configuration.slackUsername=staging

Testing Kured

To test on AKS, you’re going to want to follow these instructions to SSH into a worker node.

You’ll have to create a file on the node to indicate to Kured to restart the node. By default, daemon sets check for restarts once an hour. But, each pod on the daemon set offsets from the start time, so you won’t necessarily see all restarts close one after another.

sudo touch /var/run/reboot-required

Kured + GitOps (Flux)

When using GitOps, the Kubernetes cluster pulls deployments from a Git repository. The Git repository is the source of truth. Therefore, you can declaratively define what Helm charts should be installed in your cluster by committing a file like this one to your manifests repo:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: kured
  namespace: kured
spec:
  releaseName: kured
  chart:
    repository: https://weaveworks.github.io/kured
    name: kured
    version: 2.2.0
  valuesFrom:
  - secretKeyRef:
      name: slack-secrets
      namespace: kured
      key: values.yaml
      optional: false

In addition, you can use sealed secrets to also define information you don’t necessarily want committed to source control. Then, you can reference this secrets from Flux’s HelmRelease resource through a secretKeyRef.

What about alerting?

In a future posts, I’ll explore how to integrate Kured with Prometheus.

In this guide, I will show how to provision a VM on Azure with the Azure DevOps deployment group agent pre-installed. When the VM comes up, it will also register against an Azure DevOps deployment group.

Background

Deployment groups allow you to deploy to on-prem servers or IaaS workloads on any cloud. It’s a group of machines that have Azure DevOps agents installed on them and they communicate outbound with Azure DevOps. You can define a release to a deployment group that will run a set of tasks. For example, you can deploy to IIS, copy files to the machine, deploy Windows Services, and other things.

Set-Up

Creating the Deployment Group

First, define a deployment group to deploy to. Pick an Azure DevOps Team Project, and go Pipelines. From pipelines, select Deployment Groups.

Create a new deployment group. I called mine test.

Creating a new PAT

Next, you’ll need a Personal Access Token (PAT).

Next, create the new token. Choose the expiration date that works for your and give it a descriptive name. Once you create it, you won't be able to retrieve the value. So, make sure you copy it or don’t close the browser tab until you finish this guide.

Terraform

Here’s some quick configuration on provisioning a simple Windows Server VM. I called my file vm.tf.

resource "azurerm_virtual_network" "virtual_network" {
  name                = var.vnet_name
  address_space       = ["10.0.0.0/16"]
  location            = var.location
  resource_group_name = azurerm_resource_group.resource_group.name
}

resource "azurerm_subnet" "main_subnet" {
  name                 = "subnet1"
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefix       = "10.0.2.0/24"
}

resource "azurerm_network_interface" "nic" {
  name                = "nsg-${var.vnet_name}"
  location            = var.location
  resource_group_name = azurerm_resource_group.resource_group.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.main_subnet.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_windows_virtual_machine" "vm" {
  name                = var.vm_name
  resource_group_name = azurerm_resource_group.resource_group.name
  location            = var.location
  size                = "Standard_F2"
  admin_username      = var.vm_admin_username
  admin_password      = var.vm_admin_password
  network_interface_ids = [
    azurerm_network_interface.nic.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2016-Datacenter"
    version   = "latest"
  }
}

Here’s the variables you’ll need to define.

variable "vm_name" {
  type = string
}

variable "vnet_name" {
  type = string
}

variable "location" {
  type = string
}

variable "vm_admin_username" {
  type = string
}

variable "vm_admin_password" {
  type = string
}

variable "azure_devops_organization" {
  type = string
}

variable "azure_devops_teamproject" {
  type = string
}

variable "azure_devops_deploymentgroup" {
  type = string
}

variable "azure_devops_agentname" {
  type = string
}

variable "azure_devops_pat" {
  type = string
}

Deployment Group Agent

In order to install the deployment group agent to this VM, we must use the Azure DevOps deployment group VM extension.

resource "azurerm_virtual_machine_extension" "vm_extension" {
  name                 = "deploymentgroup_agent"
  virtual_machine_id   = azurerm_windows_virtual_machine.vm.id
  publisher            = "Microsoft.VisualStudio.Services"
  type                 = "TeamServicesAgent"
  type_handler_version = "1.0"
  auto_upgrade_minor_version = true

  settings = <<SETTINGS
    {
        "VSTSAccountName": "${ var.azure_devops_organization }",
        "TeamProject": "${ var.azure_devops_teamproject }",
        "DeploymentGroup": "${ var.azure_devops_deploymentgroup }",
        "AgentName": "${ var.azure_devops_agentname }"
    }
SETTINGS

  protected_settings = <<PROTECTED_SETTINGS
    {
      "PATToken": "${ var.azure_devops_pat }"
    }
PROTECTED_SETTINGS
}

The settings are parameters that the VM extension needs in order to register to the proper deployment group. The protected_settings are more sensitive settings, like the PAT.

Here’s what they should be:

• VSTSAccountName (azure_devops_organization) - The name of your Azure DevOps organization. For example, if you use https://dev.azure.com/mycompany, then the name of your org is mycompany.
• TeamProject (azure_devops_teamproject) - The Azure DevOps team project you’ll be using to release from.
• DeploymentGroup (azure_devops_deploymentgroup) - The deployment group agent you created above in “Set-Up”.
• AgentName (azure_devops_agentname) - The name of the agent. It could be useful to name this after the vm.

That’s it! Once you terraform plan and apply you’ll have a new VM that has registered to Azure DevOps. Give it several minutes.

The hypothesis

I anticipate that there is anywhere from 100-500 ms latency added by simply choosing to use Azure App Service Hybrid Connections. I also anticipate that the Azure App Service vnet integration is faster than hybrid connections.

The experiment

For the experiment, I’ll test three cases. First, I’ll measure the performance of an Azure App Service hosting an API that issues requests to another API. This other API will be hosted on a VM through IIS and listening on port 80. In this case, this other API will be publicly accessible from the internet.

Using a direct connection through internet

Secondly, I’ll measure the performance of the same Azure App Service hosting the API issuing requests to the second API. This other API will remain hosted through IIS on a VM and listening on port 80. In this case, the first API will connect to the second API using hybrid connections.

Using Hybrid Connections

Third, I’ll measure the performance of the App Service connecting to the second API using Azure App Service vnet integration. The second API will not be publicly accessible and instead it will be accessible only through from within the VNet.

Using app service VNet integration