Cloud Adoption Patterns
6 Key Techniques
6 Key Techniques
In my work as a contract Cloud Architect, I’ve seen a good number of customers attempting to move key application workloads to the cloud from a number of different types of environments (on-premise, colo…) over the past 10+ years. Many of those customers struggle, even fail — however, some succeed, why?
How do the successful customers get there faster?
In this article I’ll describe the top six patterns to successful enterprise cloud adoption that I’ve observed in my consulting work. To start, let’s consider two categories that are of utmost importance in building solutions on the public cloud — people and technology. I observe that getting the People focus right is more often done inadequately, so I’ll start with this topic.
People
Software is built by people. Failing to honestly access the capabilities of the current technical team to build using best practice cloud patterns has driven many a project off the rails. ‘Just winging it’ is costly at best, too often, I’ve seen it result in complete project failures.
Assessment of team abilities isn’t enough though. Customers who succeed do three things in the area of People.
Build applications that solve end-user problems
Create a culture of learning for both the technical teams and the end-users
Are led by technical managers
End-Users Focus
In order to build applications that end users actually need, technical teams should be talking with those users on a regular basis. A caution is that Slack, email, and video meetings are a poor substitute for understanding user needs.
The most effective way to gain insight is by actually working together.
The best performing cloud IT teams work side-by-side with end users on a regular basis so that they can deeply understand what to build for their users.
Also, end-users need to understand not only what they will need to do to use new applications, but, importantly, why they are being asked to change the way they work. In the case of my work, bioinformatics, a simple, ‘we are literally out of file storage space on our current cluster’ does wonders for end-user education.
Culture of Learning
End-users benefit from domain-specific training materials — use their data and their processes to create tutorials. Here’s an example from a recent client. They also want to understand migration paths, i.e. how exactly will the move from current to new (cloud) environment be mapped for their work processes. Closely listening to end-user questions and concerns and then answering honestly in non-technical language goes a long way toward end-user adoption of new or updated applications.
It’s also important to verify actual use of cloud applications. I’ve had several clients that were unpleasantly surprised when I asked (and eventually got the answer) to straightforward questions such as ‘Exactly what percentage of users have actually uploaded at least one file to a cloud bucket?’
Another aspect of educating about cloud services is to explain why one service was selected over another. Included in this explanation should be making costs visible to IT teams AND end users. Simply referring teams to use the cloud cost calculator is not enough.
Explain cloud service costs by being both concrete, i.e. “preemptible VMs cost up to 90% less than standard instances” AND specific, i.e. “running our DNA analysis pipeline with 2 full samples costs $ 1.20” is key to transferring knowledge. Also show custom tags that are used for billing management.
Teams welcome job-specific learning paths (or learning ladders) which included associated, linked resources — short screencasts, concise tutorials and links to in-depth topics. Learning paths which include customized, runnable ‘Hello MyWorld’ tutorials that take 5 minutes or less to try out is a key aspect of this approach. Here’s an example (again from bioinformatics) of tool testing.
I’ll cover more about learning for technical teams in the Technology section of this article.
Technical Mangers
Cloud adoption projects often suffer from the ‘emperor’s new clothes’ syndrome. When managers aren’t involved or aren’t technical enough to actually understand details of work in progress, then that work can be inaccurately assumed to be ‘done’. This often includes a lack of thorough and proper application validation.
A clear sign of this situation is when the only person that can run the new application is the developer of that application.
Teams that have uninvolved and/or non-technical managers often experience unpleasant surprises (and long delays) when they attempt to put was assumed to be ‘working’ cloud applications into production usage. I’ve had several clients over the years in this unfortunate situation. A fast mitigation is to create what I call minimum viable documentation.
To create minimal viable documentation, start by drawing a one page sketch of the system. Next update this sketch as you work with the team to validate that the parts of the system exist and actually run when you try to use it. Here are some examples of MVDs from my work.
The best technical managers join new deployments as test end users, so that they can be part of the application verification process before new applications are made available for those end users. These managers also are part of end-user education and pitch in to reduce application installation and usability complexity.
Technology
Unfortunately, some technical teams work is directed by vague & high level directives to ‘move to the cloud’. When priorities are mixed up, that is focusing on using technology BEFORE developing skilled people, then the outcomes are rarely delivered on time and on budget. Sadly, frequently these ill-prepared teams deliver nothing.
A number of my consulting engagements over the years have been around ‘starting up’ or ‘cleaning up’ stalled cloud projects. The attitude of ‘let’s just dive right in and figure it out as we go’ rarely produces satisfactory results. To mitigate, building a culture of learning (and budgeting for it at the beginning of cloud migration) is key.
Another subtle, but equally important aspect, is considering how to validate whether or not technical learning is actually occurring. A quick and easy way to do so is to ask teams to demonstrate working applications on a regular basis.
Also, learning time is work time — how much of your technical team’s time is allocated for learning? 10% 15% 25%? Do you have learning deliverables which are measurable on the team work schedule?
The three key techniques here are the following:
Use Elastic Services
Use Browser-Based Tools
Find and Socialize Success
Elastic Services
Technical teams that are educating themselves in cloud native patterns will seek to use elastic cloud services as the basis of their compute and data storage applications. Taking the time to understand key primitives such as virtual machines and file buckets should include understanding advanced configuration. For example, teams that know how and when to use ephemeral (or spot) VM instances and how to configure bucket lifecycle policies show cloud fluency. They also get results that can significantly improve key business processes — for an example, see this case study.
Teams that work to understand and use serverless architectures are demonstrating the highest cloud fluency because this is the ultimate elastic architecture.
Serverless and data lake based architectures aren’t best fit for all use cases, but when end-users have event-based burstable scenarios, using a serverless approach results in the fastest time to market. Here’s an example from a one of my customers.
The trade-off to getting enterprise IT teams to user serverless architecture is NOT technology, it’s understanding the need to educate teams in how best to implement. To that end, I’ve done several talks and have created courses on this topic.
However, working at the wrong level of cloud abstraction is an anti-pattern. Lately, the most common example of this is technical teams that are enamored by the momentum around containers. These teams are unfortunately also often overconfident in their ability to implement performant Kubernetes clusters.
While there are certainly use cases for cloud-container architectures, adequate team education is particularly critical when implementing cloud container architecture. See this concise and amusing article (image from the article too!)
Cloud-Native Tools
A subtle, but important aspect of cloud success is how the technical and end-user teams use tools to work with cloud services and applications. Teams that use browser-based tools and cloud observability tools produce working cloud applications faster than those that rely on locally installed tools.
These tools include developer environments and CI/CD pipeline tools(all of which can and should be delivered using serverless browser-based services). Another aspect of using browser-based developer tools (IDEs, etc…) is that on-boarding new team members is much simpler (and faster!) than when a new hire has to find, download, install and configure tools and libraries on a company-issued laptop.
Examples of these types of tools and common usage include the following:
Cloud-based SSH tool to run scripts on cloud VMs
Cloud-based notebook to run ML scripts on cloud VMs
Cloud Shell terminal tool to develop cloud infrastructure scripts
Cloud-based IDE tool to develop cloud application code
Cloud services which visualize and aggregate cloud service logs
I also wrote an article about using cloud-based tools with more detailed examples.
Build on Success
I’ve observed all too frequently that large organizations frequently have siloed technical teams. This results in duplication of effort, sometimes even seeing multiple teams make the same mistakes with no focus on sharing and/or learning.
Despite this, I can usually find pockets of success within any organization that is moving toward building applications on the cloud. These are often 1–3 person teams who’ve independently taught themselves, then built a small, but useful cloud application and deployed it.
Locating, connecting and socializing cloud success from these teams to the organization and within the organization accelerates cloud migration success in my experience. It’s as simple as getting them to share what they’ve done with others.
Find the people who have developed cloud applications
Find the people who have deployed these cloud applications
Find the end users who are using these cloud applications
Also this exploration often results in an improved experience for end users, as they are using new or updated cloud applications. These applications can be created and delivered by technical teams that have had no exposure to each other’s work prior to this effort at connecting the organization’s successes.
Next Steps
Learning and using best practices is important to my work. As mentioned, one focus area is visualizing cloud systems at various levels of detail (example shown below). To that end, I’ve added a PATTERNS section into my anchor GitHub Repo learning-cloud.
Sharing these effective implementation patterns is a focus for me as I reflect on 10+ years of work in the cloud.
Expanded Info
I also did a keynote (for Agile India) on the topic of Cloud Adoption Patterns. Here’s a recording (~30 min).