09. Lessons learned PaaS

Overview

Lessons learned building a PaaS

Project Overview

*Names have been replaced to protect the innocent.

Timeline

The way I have decided to present the project overview is to work through a timeline of what has occurred.

NOTE: Open Timeline image in browser window

With the intention that this will help to give you a better understanding about the reasoning behind the decisions that were made throughout the process.

I will briefly provide the back story for context, and then go into further depth on subsequent slides about my specific contributions that are displayed here in ‘yellow’ and why I thought this particular project is a useful example.

2019 - Customer goes to market.

Starting in 2019, a “Large Enterprise” went to market as part of a digital transformation project.

To give you an example of the scale of this project;

The Project is slated to run over 5 years
The cost is measured somewhere in the hundreds-of-millions of dollars.
The project is split across multiple work-streams, each of which is bidded on separately so could potentially be won by different organisations.
The organisation I am currently working with, during the bid process, deemed this to be too large for them alone.
They formed a “coalition” with other organizations and worked together on a joint proposal.
The bid was successful and the coalition was down-selected for delivery
The work streams were broken up within the coalition
I was working for the organisation that has the Hybrid Public and Private Cloud portion.
Now I’m no sales or accounts person, however to my knowledge the next step in the process would then be to receive specific work-order packages from the customer.
Noted that apparently it’s quite normal for large enterprises to take months for this part of the process to happen.

Early 2020 - Teams assemble.

Pre-empting incoming work orders, the organisation engages their internal ‘Cloud’ team and resources are identified with the relevant skills to work on the project.
Taking into consideration existing projects, who is on the bench etcetera.
Now because there isn’t an official work-order yet, internal billing for this project is non-existent, and any contributions are basically ‘best-effort’ in between projects or whilst on the bench.

A spanner in the works.

COVID-19
The Enterprises funding was re-allocated
However the client still has the need for this project
The organization makes the decision to continue work under the expectation of future work-orders
And the project work slowly trickles along…

Product Owner realisation.

Fast forward to late 2020
The Product Owner asks themselves the question;

If the customer asked me to demonstrate what I have today, what could I show?

The answer to which, unfortunately, was nothing.
Because the project isn’t funded by a client, and there is no concept of R&D within this particular organisation, the work occurs on a best-effort basis.
There was also no clear ‘vision’ or goal that the project is working towards because for this organization, it usually comes in the form of a work order.

To summarize, up unto this point, it’s basically been kids in a sandbox playing around with some cloud credits.

Time to get serious.

The organization values this project highly as it also works towards other strategic goals so they decide to double down on the investment and hire dedicated staff.

They go to market for;

Project Manager
Cloud Architect
Site Reliability Engineer

The Pitch: With the pitch that we’re building a new team for a greenfield project using Azure Public and Azure Stack Hub

Jan 2021 - I joined in the SRE role.

In between assisting interviewing candidates for the Cloud Architect role, my first mission is to figure out What is the outcome of the project?

So I don my Business Analyst Hat and have a workshop with the Product Owner on the ‘vision’ which I will talk about during the business objectives shortly.
Once the vision was defined and documented, the next step was to ascertain what it looks like
So I put on the Cloud Architect Hat and put together some conceptual diagrams for the Product Owner and rest of the team to review and approve so I know were all on the same page before we get too far down the road.
Meanwhile I’m also working on the Yak shaving dependencies;
- Setup of the GitHub Enterprise Server instance
- CI/CD with GitHub Actions
- Repo templates
- And assisting with the Azure Stack Development Kit build as-hoc.

Feb 2021 - Architecture and Project Governance.

From there, placing my scrum master or product manager hat I;
- Defined a basic project governance structure so we have a clear way for the project to operate.
  - For this I consulted with the organisation’s internal Agile Coach
  - The role of their Agile Coach is defining broad guidelines that all projects across the organization should try and adhere to, but the coach doesn’t specifically work on individual projects.
  - Basically, I was looking for advice on the most-effective way to run this project given the resourcing constraints and the big outcome that the Product Owner wants to achieve.
  - After this consultation, what we’ve been calling a ‘scrumban’ framework was devised where we picked out the most relevant aspects to Scrum and Kanban for this project without adding too much overhead at this stage.
- Taking that conceptual design I then broke down an initial backlog of work with the expectation that it can be expanded on and prioritised during subsequent sprint planning sessions.
For further context about the Team
- Approx 14x people in total
- 13x of which at this time are currently engaged to other projects
- In any spare time, try and get involved in this project due to the nature of what it is.
It was certainly an interesting first sprint planning sessions where I was wearing;
- project manager hat assigning tasks
- scrum master hat attempting to run my first ever scrum session.
- cloud architect hat conveying the product owners vision to the team.
- The main challenge here is asking a user to try and estimate what they commitment this sprint.
Finally, I was able to put on that SRE or Platform Engineer hat and start cutting some code…

Making things a little more challenging, I was also tasked with some work outside of this “dedicated project”. - Assisting with a couple of pre-sales bids in an Architect capacity. - Tasked with developing a “DevOps” training plan for junior staff who are part of a graduate program.

March 2021 - Project Manager hired and Political Issues.

A dedicated Project Manager was hired and I have been working with them to handover management of the project so I can dedicate more time to the build.
The Project Manager is coming from a Waterfall background so there is a bit of learning during the process.
Although there was some good Cloud Architect candidates found and recommended for hire unfortunately a hiring freeze has been placed on the project pending a work-order.

Future.

It’s not all doom and gloom for this project.

Project is in limbo.

As for the future, it won’t be known until the Enterprise gets their budget time in May.

The latest from the customer is that want to move ahead but need to secure some or all of the previous funding in order to proceed.

Business Objectives

Hybrid Cloud

Solve real customer challenges

Reusable

Target a demonstration

Partner Technology Showcase

So this is the part of the project where I wear a Business Analyst or Project Mangaer hat.

During the first workshop with the Product Owner, I asked a series of questions that initially focused on the outcomes as opposed to any specific technologies;

I started by asking questions like;

What are the goals for the project?
What has the project set out to achieve as priorities?
Who are we building it for?
- Internal or External?
- Who is the focus?
- specific to a customer or generic?
Who will own the IP once created?
- Weird internal politics with divisions
Where will the artefacts be stored?
- National Security implications

Very quickly during this process it dawned on me that;

We didn’t know what the customer actually wants without a work-order
The organisation was so used to having the customer say “this is what I want, build it please”
We needed to make some educated guesses about what a work-order might look like
And that this has really become a “showcase” project to display what is in the realms of possibility that could be re-used and sold.
Whether that be to the original client, or other clients.

That line of questioning led me down the path of;

What are the applications or workloads that should be showcased?
What are the unique selling points to the solution compared to competitors?

From the results of all this I condensed it down to a few principles and objectives for review and approval by the product owner.

NOTE: Principles are fade-in fragments

Hybrid Cloud.

Create an Azure Hybrid Cloud solution that demonstrates the ability to deploy IaC in an automated manner
Showcase the services of Azure Public alongside Azure Stack

Solve real customer challenges.

Showcase the ability to develop in a low security domain and deploy in a high security domain with an end to end demonstration across a CDS.
Governance of IaC across multiple teams in a large organization
- This is not a customer environment where you could drop in a monorepo and wish them luck.
- No concept of inner-source
- Operate on a need-to-know basis
- And strongly adhere principles of least-privilege.
Reduction in sustainment costs through Automation
- Common challenge spoken about by this customer and others
- Large operations teams come at a cost

Reusable.

Design in a way that it can be re-used across multiple potential customers
Costs money to develop up-front, but pays off over time
Future projects might leverage this work as a base.
Not re-invent the wheel for this customer, but be based on industry best-practices.

Target a demonstration.

A very important part, is that the final result needs to be something that can be shown to potential customers.
Not just repositories of code
But a presentation or customer facing showcase that be shown to the decision makers.

Partner Technology Showcase.

This particular organisation is a Professional Services shop, and they have a list of pre-approved vendors that they partner with and technologies they use.
The solution should only be comprised from vendors on this list
And within this list, “educated” guesses from existing knowledge about what the target customer wants.

Solution Architecture

CAF

So this is the part of the project where I wear the Cloud Architect Hat.

The solution is based around the Azure Cloud Adoption Framework aka “CAF”.

The spiel for the CAF sounds a lot like the AWS well-architected framework and goes a little like;

The Microsoft Cloud Adoption Framework for Azure is proven guidance that’s designed to help you create and implement the business and technology strategies necessary for your organization to succeed in the cloud. It provides best practices, documentation, and tools that cloud architects, IT professionals, and business decision makers need to successfully achieve short-term and long-term objectives.

This project is basically a codified example based on the CAF. As such the principles and strategies are already well defined so I’m going to instead focus on the aspects that make this project different, or we might be here all day…

The Architecture will;

Include multiple environments with a promotion based life-cycle
- DEV, STAGE and PROD in Azure Public Cloud
- DEV, STAGE and PROD in Azure Stack Hub
Automated using Infrastructure as Code tooling
- Which in turn, comes with other benefits;
  - Easily audited
  - Easily reversible
  - Peer reviewing of changes is a fundamental aspect to the architecture of the platform.
  - No single person (rogue or otherwise) should be able to make a change that affects the entire environment without going through the review process.
The platform needs to be extensible
- Both from the technology and governance sides
  - The platform will initially be developed by a small team during the POC
  - Then handed over to the customer that spans across multiple silo-ed teams.
Enterprise grade security
- Security must be integrated into every step
- Shift-left with DevSecOps tooling in the pipeline to try and catch as many issues before they are deployed
- Because this platform will be host to applications of national importance the security and stability is paramount to the long-term success of the project.

Architecture Overview

Vision

Methodologies and Patterns

Landing Zone

Landing Zones.

Landing Zones are an Azure specific term about making it easier to compose reproducible templates from pre-approved designs.
They are an environment that accounts for scale, security governance, networking and identity.
I will go more in depth into Landing Zones when I discuss ’levels’ in the key architecture decisions.

GitOps.

GitOps is another pattern used in the project.
The infrastructure is managed as code (or HCL) to be precise and any approved changes results (via a PR) trigger the CI pipeline which runs a series of steps before deployment.
- Lint
- Validation
- Runs a security check
- Generates and updates documentation
- Plan
- Apply
- Notify
The idea being to catch as many issues as possible before the change is applied against the environment.
In the future we plan to also add integration testing using terratest but we’re not there yet.

Immutable Infrastructure.

Another pattern being used is immutable infrastructure.
Immutable Infrastructure of course refers to servers that are never modified post-deployment. Any updates to these servers results in the entire replacement of the server.
In this project, one of the selected workloads to be demonstrated is the Atlassian products Confluence and Jira.
The deployment and changes of these are currently planned to be performed via the combination of;
- GitHub Actions CI Pipeline
- Packer to create and seal the image
- Ansible Role runs as a Packer Provisioner to configure the image before being sealed
- Terraform to orchestrate the deployment of the resulting image and handle the detachment and re-attachment of persistent storage.

Blue/Green deployments.

Another pattern being used is blue/green deployments.
A Blue Green deployment is a technique to reduce downtime by running two identical environments called Blue and Green.
- At the start, only one environment is live “Blue”
- Green environment is deployed and tested before being cutover.
In this Project the ForgeRock component has some issues with the Rolling Update deployment strategy where (according to the ForgeRock architect) different versions of the application running at the same time is known to cause issues.
As such, this is one example of this deployment pattern used within this project.

Key Architectural Decisions (Levels)

Levels

Decision: Designing and managing the source code around levels. Justification: Scaling governance from small to large teams.

Why: The benefits include;

Control the blast radius: minimise the impact to the landing zone and don’t compromise the entire environment.
Enforce a standard configuration: compose complex environments fast from pre-defined templates.
Enable autonomy: different landing zones can be developed independently at different development life cycles.
Least privilege: Each level is protected by its own identity preventing unwanted modifications.
Teams: Large organisations are often split across multiple teams. Each level (and sub-level) might be owned, managed and operated by a different team.

The neat trick here in this project’s situation is that it allows us to ;

First start small with a monorepo broken down by folders
Scale by breaking out the levels into multiple repositories managed by separate teams.

An example deployment might contain;

One “Level 0” Landing Zone (per isolation boundary)
- Infrequent changes 6 months
A couple of “Level 1” and “Level 2”.
- Change frequency of around 1 month
A few more of “Level 3”.
- Change frequency weekly
Many “Level 4” applications will exist.
- Change frequency daily

Key Architectural Decisions (Separation)

Components

Decision: Separation of config, secrets and business logic. Justification: Re-usability across environments and security domains.

Why:

Inner feedback loop: Provides quick feedback to DevOps engineers, offering the ability to test fast and iterate. - May be local or sandbox subscription (MSDN) development - Once the code change is complete, and has passed the pre-commit hooks, a pull-request is then raised for review and merger.

Secrets: Secrets for the environment

Terraform Config repositories: Storing the environment configuration separately from the logic.

Terraform Logic repositories: The core logic which is promoted through the environments once tested.

GitOps pipeline enforcing Landing Zones: Uses git as the single source of truth for the environment deployment and configuration.

The Key Point here is that there are 3 items that are fed into the CI/CD system, the output of which represents the DAG of the environment.

Logic
Config
Secrets

The config and Secrets never leave their own environment.
The logic however can be promoted and re-used

Team Enablement

Enabling peers.

How did your contributions help enable your peers and team members?

Due to the nature of these floating resources on and off the project i’ve found it very important to;

Know the individuals “area of interest”
Keep them motivated by trying to strike a balance between
- what they want
- vs
- what the project needs.

Examples;

No one want to come off a project onto the bench only to spend their time massaging a backlog for another project
- So as the backlog is not neglected, I’ve taken on this task to enable those floating resources to focus on deliverables.
They want to work on something cool in their area of interest
A Network Engineer who is a gun a Palo Alto Networks but knows nothing about Terraform
- All they want to do is play with PAN features they don’t use day to day
- But what the project needs is an architecture for how PAN will fit into the solution
A Chef Inspec SME that just wants to learn Terraform because they do Chef Inspec all day long.
- All they want to do is play with terraform
- What the project needs is Chef Inspec deployed and configured

Juniors and Seniors.

Did you apply a different strategy for junior v.s. senior members?

Senior Member:

Senior members that come on and off the project get a lot more freedom of choice in decisions on what to work on and how to contribute.
Basically I show the the backlog, the sprint goal and work with them to determine where there skills are best utilised.
For example at the moment I have a Chef Inspec SME on the bench in between projects
So I showed him where the information is available.
Created his accounts
And set him on the task of design and implementation for that component
I don’t bog that person down with governance or control or micro-managing.
When they need help, they ask.

Junior Member:

Where as we have a couple of graduates coming on and off the project
I sit down with and make no assumptions.
Start by showing them where to find the information about the project
Determine their area of interest
Show them templates for how things should be laid out
Give them a task
I don’t know if it’s just the two grads that I have are shy, but they don’t ask any questions, then I find out they have been stuck for 3 days…
I tackled this by scheduling a couple of reviews during the week to track progress
Repository templates are a life saver.

Challenges

Technical Challenges

Avoiding monolitic composition

Multi-cloud with Terraform

Avoiding monolithic composition:

The first hurdle was how to apply a design for that scales from the small team but already plans for handover to an “enterprise” size organization. This was where the levels discussed earlier come in to play.

Then there is how to apply those levels securely to the Terraform tooling.

Each level has Read/Write access to the The Terraform state at that level
Each level has Read only access to the to the level below
This is so that outputs are the only object accessible from each layer above.
No need to be able to modify anything from another Landing Zone
However, if into a Virtual Network you need to have the Virtual Network details.

But my favourite challenge is using Terraform for multi-cloud.

Terraform works on the concept of providers.
Terraform uses providers to talk to the backend Cloud APIs

Terraform

Zoom out the window to 33%.

Although Microsoft will advertise “one API” Azure Resource Manager across both Public and Private Azure deployment, there are differences, mostly in the services available.

Unlike tooling like Pulumi, Terraform uses a separate “provider” for;

Azure Public: azurerm
Azure Stack: azurestack

These providers have different syntax and required variables for each.

Now there are 3rd party tools available like Terragrunt that make this cleaner, but they come with their own requirements, dependencies and adding another layer on top.

At this time, the fix was a simple as a bash heredoc script based on an environment variable in the CI $CLOUD_ENVIRONMENT which will generate the root module and therefore follow a different code path.

This meets the requirement of one, promotable codebase across the environments with minimal dependencies and sticks with a KISS design for now.

In the future the project may come back and look at this issue but it’s not a major sticking point or doesn’t affect the demonstration in anyway.

Interpersonal Challenges

Opinionated Engineers

On-boarding

Political / Organization Challenges

On-boarding

Funding

Given a project of this nature, this is quite the battle.

On-boarding:

I’ve found the organization to expect engineers to float in and out of the project and immediately be productive.
We’re human, humans take time to on-board and absorb what the project is about and how it operates.
A lot of time is spent by a limited set of “full-time” resources dealing with on-boarding of these floating resources.
This has been a real challenge for the organization to acknowledge

Funding:

The organization is based on a model of receiving work orders from customers to complete outcomes.

As this is currently an “un-funded” project, there isn’t enough resources to give this project the attention it deserves.

I’ve approached this by enquiring about if there is anyway that R&D funding can be allocated but not holding my breath.
Looks like it will continue until a work order is received.
Trying to make the most-effective use of the available resources.

Demonstrated Skills

Utility Player

Lessons Learned

Personal Lessons

Project Lessons

Personal Lessons.

I would say I have a number personal lessons during this project so far.

Don’t assume the role your being hired for has an immediate need
Don’t expect that a clear vision already exists for a project that’s been going for over 12 months.
I certainly now know to ask if a “work-order” has been received in Professional Services organizations.

Project Lessons.

Plan for the unexpected
Know when the hire
Know what differentiates your solution and makes it unique
Dedicated R&D funding for projects that are deemed important and have valid business requirements across multiple Federal agencies.

What would you do differently if you could do it all over again?

I would have pushed harder to work on it in isolation.
I could move must faster and cheaper rather than

What was the best decision you made on the project, and why?

Spending the time to workshop the Vision.
There was actually a moment in this workshop when an Azure resource who had been off and on the project since it’s inception said, and I quote;

Oh, now I know what we’re building

What was the worst decision you made on the project, and why?

I believe this is likely still TBD.
As the project doesn’t have a firm grasp on what the customer actually wants my best “educated” guess could be way off the mark.
However, I also believe it’s still worth pursuing and if R&D budget was allocated, could be seen as a worthy investment across multiple customers with a similar set of requirements.
Likely my worst decision in hindsight will end up being an architectural decision made without the correct knowledge of the specific customer environment in this example.

But, you can’t just sit on the fence so I tried to design for the architecture to be as flexible as possible.