Episode #653
Why Containers for Development?

Join me for a ramble in the rain as I talk about all the reasons you should consider using containers not just for deployment, but for development as well. We’ll talk about how containers enable repeatable, portable, executably-documented development environments. Also, there’s a rainbow!

Episode Script

Hey there. So I want to talk today about devcontainers: containers for development. But , today I kind of want to just do an introduction to the topic, which means that I don’t have a lot to show on a screen. it’s really more of a high level bullet point kind of conversation.

and I also have a house full of noisy kids right now. And so I thought to liven up the bullet points. I would talk to you about dev containers while walking around my driveway.

So when you first start on a new development project on a new team, what do you usually have to do? Well, maybe they hand you a new computer, or maybe not. Either way, they probably hand you a repository or several repositories that you need to get into and then you have to do a whole bunch of setup.

And usually it’s like, the setup is documented in a combination of like a README, that’s  a little bit out of date. And then there’s, like, some references to some Wiki articles.   there’s some other Wiki articles that are really useful but that didn’t get linked into the README for whatever reason. Oh, but not that one! That one’s totally out of date.

And, and a lot of times it’s, it’s a lot of just like trial and error and asking people. And these onboarding sessions can take, in my experience, they can take days. Sometimes they can take over a week just to get to that first ability to run the tests , and start making changes to the software.

Now, imagine this, imagine you start a new programming project.  you’ve got a completely brand new computer and  you download your, git client to that computer and you installed Docker on that computer.

You clone the repos that you need for this project. You run docker-compose up, you go get some coffee, and then you come back. And you have a development environment that’s completely ready to go.  if it’s a rails project, you can run your, your bundle install. You can do your DB setup and you can run the tests and you’ve got everything in place ready for you to actually get some work done. Not just in your first day, but in the first four hours.

That’s not a dream. That’s, that’s a workflow that I’ve actually been experiencing lately. And I’ve fallen in love so much with it that these days, whenever I started a new project, the first thing that I do is I create a dev container configuration for that project if one doesn’t exist already.

and  I’m hoping to make a series of videos about this. just going through, bit by bit how to build up a dev container configuration for a typical rails app. but, right now I just want to like, kind of do an overview.

So one of the first questions. That comes up when we talk about dev containers is, well, what is a dev container versus like a deployment container? Like a lot of apps are being deployed as containers these days,  and also, CI is done with a container.

So you might already have  a container configuration for your project. So do you already have what you need? Well no. A dev container is something different. because deployment and development time have very different needs.

So for deployment, what do you want?

Well, you want, you want to minimize your file size. You want to minimize the size of the container image that that is being pulled to your servers or to your CI.

you want to, you want to minimize your memory footprint so that they’re there using the smallest instance you can get away with.

you want a small attack surface area, you know, you want as, as little installed on these virtual machines as possible so there are fewer things for potential attackers to target. You want it to be as hardened as possible. you want ports to be locked down. You don’t want extraneous server services running, you know, maybe you don’t even want it to respond to pings. you just want it to do with one job run that app service.

And so as a result, a lot of times your deployment container is going to be based on something like Debian slim, a slimmed down version of Debian or maybe Alpine Linux, which is an even more stripped down version of Linux. It doesn’t even have the usual lib C In a deployment container.

A lot of times  you want to ensure that you have observability. So you’re going to have that thing start up and it’s going to start sending events to, to event servers or log servers. As a matter of fact, not sending events, not connecting to those servers might be an error.

What else? well, or your Docker file, you probably want to organize that file into a minimum number of layers because that makes building it faster and it produces a smaller, resulting image.

And then there’s just the workflow that’s around a deployment container. you don’t want to be changing that container definition very often. Anytime you change that definition in any way, whether it’s toupgrade the operating system or upgrade some library that’s on it, or add a new utility… anytime you change it, there’s a potential for breaking it. And you don’t want to break production. So you want that process to be really controlled.

Now, let’s talk about what you want for development environment. First off, you don’t want a really slim Linux distribution. You probably want a full batteries included, comfortable development environment that like you get with something like the full Debian system or a full Ubuntu system.

you want your compiler tool chains, you know, your Make, your GCC and all that stuff. You want all kinds of development utilities.

you want ports that are open for debugging ,

you don’t want to talk into your observability servers. You don’t want to talk into your log servers. In fact, you want to suppress it’s any kind of phoning home  the app would normally do.

and,  there may be some things that you want to tweak that trade data integrity for running the tests really fast.

You might also want to include some kind of fakes on that machine, fakes for services that you use in production. For instance, with rails, you might include SQLite so that you can run the tests  on the local file system rather than having a database server.

And when it comes to modifying the Docker file, The style of modifications that you make is probably going to be different. Rather than trying to consolidate the layers, you’re probably going to want to slowly add stuff incrementally to the end of the Docker file. Which makes for a larger total image, but shorter, incremental rebuilds.

And you’re going to want to be able to change this any time you, like, you want any developer to be able to change it, add a new utility, upgrade a library, that kind of thing. and you don’t want to like live in fear of accidentally breaking production because you updated something for your development environment.

So what you can see is that all these needs are opposed to each other.  they’re pulling in two completely different directions, which is why, I’ve come to think that. You should have two completely separate definitions for your deployment containers and for your development time containers. And  don’t try to share stuff in between them. you really want them separated so that you can have separate workflows for each one.

So what goes into a dev container? Well, a devcontainer needs to do a few things. It needs to orchestrate  one or more containers. Because you’re going to have a container for your app, but you might also want to have separate containers for any services that it depends on like a database service.

It turns out in container land,  it’s a lot easier to just use separate containers for different services, rather than trying to have everything all to all running together in a single image.

You want your devcontainer configuration to include mappings from the app container into your host machine.

You want it to have file mapping so that you have your development repository mapped in, so you can edit the files locally on your machine and see the updates come through inside the container. Or edit them inside the container and see the  updates come through on your local machine.

And unlike a deployment container, you want your development container to have a rich set of development tools. You want it to have all the different tools that you normally depend on when you’re developing.

there are a lot of perks to developing with dev containers. we already talked about how it makes onboarding a lot easier. And you know, sometimes we think about onboarding as one of those things that’s painful, but you know, you only have to do it once. But in my experience, even that’s not true.

I’ve had a machine die on me, you know, and then I lost a couple of days because I was having to get the new machine up to a development worthy state. I often have multiple machines that I use for development at once. I have a really thin and light laptop that I take with me. And then I have a more powerful desktop machine. so having a dev container makes all those scenarios a lot easier and faster to get started

with Docker based development. It doesn’t matter what operating system the machine is running. Docker runs just fine on Linux, on windows, on MacOS. And then for the actual app development, you don’t have to come up with different instructions for people using different operating systems. you can just maintain the Linux version that’s running inside the container.

With Docker-based development, various projects leave no trace on the host machine. That might or might not be important to you. If you have a corporate laptop, which you’re only using to develop one project, then you know, you can do whatever kind of tweaks you need to, to that machine. But if you’re a consultant and you work on many different projects, you don’t want the changes that you make for one project breaking another project.

For instance, I can definitely remember cases where I totally forgot about the host file changes that I made to support one project’s expectations, and then something didn’t work on another project. And I spent hours debugging until I finally realized, Oh, I’ve got entries in my host file that I totally forgot about.

And it’s not just about having those kinds of reproducibility onto physical machines. It also opens up the possibility of moving your development into cloud-based collaborative development platforms like Amazon’s cloud nine, Github codespaces. If you’ve already got the container definition, it makes it a lot easier to move into services like that.

one of my favorite aspects, is it eliminates, or at least reduces the “works on my machine” issue. I’ve seen so many issues where one developer saw an issue on another developer’s machine. It worked fine, supposedly they’re configured the same, but at some point one developer got a newer version of some system library, and now it’s not working right. With Docker based development. as long as you make sure that you’re controlling the version of the operating system in that container definition, and the version of various tools that are installed, you don’t run into so many issues like that. Everybody is using the exact same environment.

another cool advantage is if I’m developing and I discover that there’s a really handy set of bash aliases that make development faster, or a tool that really helps out, I can throw that in the dev container definition. And the next time other developers pull that from the repo and rebuild their dev container, they get all that for free. I don’t have to like document it and evangelize it. They just get it for free.

A devcontainer is also a great place to experiment with updates to libraries or to the operating system version or to the language SDK version. Because it’s effectively a throwaway machine. You can try out your changes. If they break everything, you can always just rebuild your container. All those changes will be gone. If it works, you can move those changes into the devcontainer definition and propagated out to the entire team.

And last but not least: remember a few minutes ago, I was talking about that situation where you are getting started and there are all these readme files, and Wiki pages, and out of date documentation for how to configure a system for this project? Well, with a dev container definition, you have executable documentation of how to set up a machine for the project and that documentation is tested and verified every time somebody rebuilds the container.

I wonder if you can see this rainbow in the video?

So those are some of the advantages of working with a dev container. Of course, there are always drawbacks too,

in this case, well, obviously, there’s a learning curve involved. You’ve got to get up to speed on working with Docker and Docker compose.

you may see some reductions in the speed that your software runs, although this has been getting better and better lately. A lot of times you’ll see, close to native speeds.

In my experience, you’ll sometimes see some wonkiness in the file mappings between the local machine and the containers. but this too is something that has been getting better  as Docker has improved.

it’s an adjustment in workflow. You know, before Docker based development, you might install a new utility on your machine. If it works, you might tell other developers about it and document how to install it and configure it. With a dev container. You would try it out. And then if it works, you’d move it into the container definition so that it doesn’t get lost the next time you rebuild.

there are some things that might’ve just worked natively and now require a little extra configuration. For instance, like, doing native browser testing might require opening up some ports or something.

Also container layers, are large files and they will gradually grow to fill up your hard drive until you get into the habit of pruning them back every now and then get rid of the unused ones.

Personally, I’m totally sold on container based development. I think it’s improved my productivity. I think it has cut down on a lot of my common frustrations. And I think a lot of teams would benefit from it.

thanks for watching me blather on about this. I’m looking forward to making some more concrete videos about how to containerize the app that you’re developing.

Stay tuned, and happy hacking!