Cloud Dev Environments
Stop installing stuff on your laptop!
CODEX
Stop installing stuff on your laptop!
I work with a plethora of languages, clouds and tools. To that end I find myself setting up dev environments frequently. In this article, I’ll share patterns for efficiency in the process.
1. Cloud-based Laptops
Long ago I stopped installing random stuff on my personal laptop. Rather, I have been using one or more cloud-based services as developer environments for quite awhile now. To that end, it seemed odd when a recent client actually sent me a physical laptop, rather than using a cloud-based environment so that I could get my work done for them.
Initially (as in 10 years ago), I used one or more AWS EC2 instances as my work laptop. Being able to snapshot complex environments and move them to instances with more CPUs, RAM, etc… is such a time-saver. Also, it’s nearly impossible to run out of disk space in the cloud, as adding one or more EBS volumes to an instance is simple. Of course, using AWS S3 also works for storage. Stopping and starting the VM instance when not in use saves cloud costs — it’s a no-brainer.
Despite the conveniences of cloud VMs, I still had to wait for those VMs to start up each time I used them and also to install and configure the various tools, libraries and languages that I wanted to use on the instances. And I had to locally install, configure and use the correct version of the cloud client tool (in this case the aws cli) to access my cloud-based VM.
2. Better Cloud-based Laptops
Enter GCP — their VMs start up faster than AWS, and they are cheaper in the sizes I need. All of that is good, but the killer feature was using their integrated SSH client (shown below) or their cloud shell to access the VM. Voila — nothing to install! Subsequently AWS added their own SSH client (EC2 instance connect) and also a cloud shell, but GCP did get there first.
Despite the convenience of interacting with cloud VM using a cloud shell, I still found myself installing the same stuff over and over (JDK anyone?) and also the permissions model for GCP remains idiosyncratic.
3. Enter Jupyter and SaaS
A couple of years ago I discovered Jupyter notebooks. For my work, these interactive environments serve often serve as a replacement for terminal . I love a notebook’s ability to support Markdown, multiple language runtimes AND visualization. Well, that’s all good, but who wants to install more stuff?
Apparently I am not the only one who wondered this. Google to the rescue, again. They created an endpoint (or software-as-a-service) for Jupyter notebooks. It’s called Colab and it’s wonderful. In addition to providing Jupyter Notebooks as a service, Google included the ability to use GPU or TPU resources. Did I mention that this service is FREE as well?
Now, as great as Colab is, the free service tier (there is also a Colab Pro tier which costs $ 10 / month), has some pretty significant resource and runtime limits. Ok for demos, not ok for production scenarios, so what to do…
4. PaaS Notebook VMs
Yet again Google innovates. Relatively recently, they, along with other cloud providers, created a set of offers which I would call ‘specialty VM instances’. Google was out first with their AI Platform Notebook (VM) instances (shown below). These are, in essence, GCE VMs which have a large number of tools, languages and libraries pre-installed. You select an image family when you create an instance. These image families are derived from container images that you can select and/or customize.
Looking for Python, Docker? pip? JDK? conda? → all there already!
LOVE at first sight. A VM with Python, JDK, Docker, pip, conda, TensorFlow and more? Yes, I’ll take that please. Oh and did I mention a set of these instances include example notebooks which demonstrate how to interact with key GCP services such as BigQuery, GCS and others. Of course the pattern of saving money by stopping instances (these are VMs, albeit specialty VMs) when not in use is also relevant here.
AWS jumped on the bandwagon with SageMaker Notebook instances. Despite the fact that these AWS instances take longer to start up than their GCP counterparts, using AWS notebook instances you get a HUGE number of example notebooks and also the familiar AWS IAM permission model.
5. What about an IDE?
Ok well, I’ve been going on and on about Jupyter. And yes, notebooks are great for short, simple code or script execution. However, sometimes I need to deal with gnarly legacy code. For that I want a code editor or IDE. So what’s my trick to use an IDE WITHOUT installing it?
Conceptually similar to Google Colab, IDE “endpoints” (or SaaS) have been popping up. One example is Codepen.io, another one is Repl.it. As much fun as it is to try out toy examples in any one of the 50 languages Repl.it supports (shown below using the LOLCODE language — yes, it’s real), I still find myself wanting to use a more familiar IDE for ‘hard problems. For me that’s VSCode.
6. My Code in Your Cloud Notebooks
In addition to wanting to use a cloud-based IDE there is also the challenge of getting my code into a vendor’s cloud dev environment. Do I have to upload all of my code to a VM? pull it from GitHub? I don’t want to do either of these things. What I want to do is launch a cloud-based development environment FROM GitHub. I would also like this environment to include all of the code in the Repo.
The Jupyter Team and their collaborators agree. In the area of “notebook as a better terminal”, they support an interesting tool called Binder.
The binder tool allows me to “turn a Git repo into a collection of interactive notebooks”. Using configuration files IN my repo, the binder tool associates a Repo url to an action (button), which when clicked builds and runs a Docker image (on GCP GKE) using the language requirements and including the language runtime, Jupyter and the notebooks in my repo.
7. My Code in Your Cloud IDE
So Binder is great, but how do I get a my code AND a full IDE (such as VSCode) on the cloud with minimal effort? Enter GitHub Codespaces. Currently in private beta, I got access to this beta about 6 months ago.
Once again, I was charmed. So charmed that I built a use case — teaching our TKPJava library for 8th graders (on Chromebooks, still in COVID lockdown in California) on a custom VSCode IDE instance.
Despite being in beta, the process to build a custom container which creates a dev environment with VSCode, my files and any other customizations that I specify (such as VSCode extensions) that is deployable FROM a GitHub Repo was mostly intuitive. Shown above is the .devcontainer folder where the customization files live.
I wrote a series of tips on creating / customizing GitHub Codespaces on Dev.to as we built this example out. Shown below is a quick demo (starts at 11:10).
So that’s my journey building/using cloud-based dev environment so far. What about you? What is your cloud dev env these days?