October 17, 2013
Never say, “oops.” Always say, “Ah, interesting.”
Human Error is unavoidable. That does not mean we shouldn’t try and fix the stupid or forgotten mistakes. Working in teams can introduce inconsistent errors when each team member has their own development environment. There are just too many variables to forget about. In the past year, Elevate has been experimenting with automating the setup of our development and production server environments. We thought it would be beneficial to outline our journey using Vagrant and Puppet.
We used to work for many years on a centralized ESXi VMware machine that ran many development virtual machines. Everything was dependent on our office infrastructure to run the machines and work as a developer. Setting up each developer became quite a chore and stress on resources of the ESXi machine. The development boxes were inconsistent and ran with a “wild west” mentally: little to no rules for server configurations and software.
Challenges We Needed to Solve
- Eliminate human error, as best we can, and keep in sync with server configurations and software.
- Automate server environments (development, staging, production) that we DO have control over.
- Emulate and automate client production environments that we DO NOT have control over.
- Provide a disposable machine that can easily be rebuilt.
- Eliminate the worry to backup HUGE virtual machines.
- Easily manage development domains/IPs that don’t conflict internally.
- Easily share a development url.
- Need an easy way to flip between client projects and not worry about backing up or destroying what machine we have.
Solving These Problems with Vagrant and Puppet
When Vagrant 1.1.x was released it added a big feature called providers (VMware provider for Vagrant). Vagrant 1.0.x required Virtual Box that came with a number of quirks related to file sharing with the virtual machine. We were more than thrilled to move over to VMware that provided years of stability and support. It also opened up a host of other Vagrant providers that allowed us to rethink our workflow. Since we use some Amazon services, the AWS vagrant provider allowed us to spin up an instance and install everything we needed very easily. All our effort was now easily portable among various cloud services.
Vagrant has solved a lot of these issues that we were trying to solve from above.
PROBLEM 1: Eliminate human error, as best we can, and keep in sync with server configurations and software.
ANSWER: With the help of Puppet, Vagrant helps provision machines to be identical. In all honesty, learning the basics of Puppet is the longest investment. There have been a number of Vagrant / Puppet specific resources that didn’t exist when we started. They offer good examples you can study to get the basics down. A few are listed in the resources below. Also don’t forget about search github for the specific type of machine you need to build.
PROBLEM 2: Automate server environments (development, staging, production) that we DO have control over.
ANSWER: You can get creative with how you want your machine to be built with Puppet or the configuration files. We use various tricks to allow the app to adapt depending on its environment.
PROBLEM 3: Emulate and automate client production environments that we DO NOT have control over.
ANSWER: Since we are building every machine the same, we can script the environment as close as possible to how the client server will be. Sometimes we just have to recreate file paths. We can even go as far as setting up a load balanced environment and have it work the same on every developers machine.
PROBLEM 4: Provide a disposable machine that can easily be rebuilt.
ANSWER: It is so comforting to know that we can assemble huge virtual machines from a handful of text files that can be versioned with GIT.
PROBLEM 5: Eliminate the worry to back up HUGE virtual machines.
ANSWER: Since it only takes a few minutes to build a machine, the worry of backing up or even exploring crazy things is gone. Want to explore a new version of PHP or something that drastically changes the operating system? No problem, just rebuild it if you need to revert.
PROBLEM 6: Easily manage development domains/IPs that didn’t conflict internally.
ANSWER: A common way to develop locally is to bind everything to localhost. When you have many sites to flip around, this can be annoying or easy to forget which site you are on. The domain naming we like to follow is to add .dev to the end of the final TLD. Check out the resource for a handy vagrant plugin to handle binding dev domains to the VMware IPs.
PROBLEM 7: Easily share a development url.
ANSWER: When it comes to assigning an IP to your development machine you have a few options.
- You can assign a name to local IP 127.0.0.1. The drawback is that you can’t run more than one machine.
- You can assign an internal IP similar to 192.x.x.x or 10.x.x.x ( or however the office is setup). The drawback is that you have to manually pick that internal IP you want with vagrant. Once you version the Vagrantfile, another developer could spin up the machine and conflict with your IP.
- You can run VMware with a virtual private network. It usually gets assigned an IP like 172.x.x.x. The is the way we run our development setup for a few reasons. The biggest reason is that VMware takes care of the IPs for you. So if you happen to run many machines, it will never conflict an IP. If you pair this with the hostmanager plugin below, it even can detect which virtual IP the machine is and bind your development name all from just running vagrant up. It brings a smile to my face.
PROBLEM 8: Need an easy way to flip between client projects and not worry about backing up or destroying what machine we have.
ANSWER: As mentioned above, with the combination of using the VWware virtual IP and the hostmanager plugin, it makes it all seamless.
Vagrant and Puppet has solved a lot of our problems. It’s not just beneficial to teams, everything discussed here works fine as a solo developer as well. There are more benefits when you involve a team. Vagrant has an interesting future. It will be interesting to see how it shapes out as it matures. If you have a 40 minutes, I would recommend taking a look at Mitchell Hashimoto talk on one of his other tools Packer.
Vagrant Resources and Tools
As the community gets larger, there seems to be more clever plugins and providers popping up. Here is a nice list from Mitchell https://github.com/
Vagrant Host Manager Plug-in
It was driving me nuts to manually add and manage the VMware machine IP to my host file. I also hate working off of http://localhost because I’m often hopping around clients and I have a short term memory. Sometimes VMware even assigns a new virtual IP if you shut the machine down or suspend it. I was using a tool Gas Mask https://code.google.com/p/gmask/ help manage the IPs, but was still a nuisance. This vagrant plugin hooks into vagrant up and writes out the the current VMWare IP to your local hosts file and then symlinks that file in the virtual machine.
If you know just the IP may have changed you can just run
This will update your host file and comment it and everything. It looks something like this:
172.16.131.227 elevatestudios.com.dev # VAGRANT ID: /Users/person/sites/
elevatestudios.com/ .vagrant/machines/default/ vmware_fusion/…
This tool helps to solve the sharing a url from a dev machine to another coworker on the same network. Since we use the vmware IP range (172.16.x.x), other people on our network can not resolve these IPs. Passing it through Ghostlab allows us to send a routable IP to a designer or account manager. We also use this tool to test our work on physical mobile devices. Otherwise we use http://www.browserstack.com/ for all our other testing.
If you need to publicly share a development machine temporarily for a client or whatever, this service works. It also is a great service if you need to test https and don’t want to go through the hassle of setting up legitimate ssls for testing.
This tool makes it idiot proof to build out a LAMP (apache) / LEMP (nginx) server using vagrant and puppet. It can serve as a good base to better understand how vagrant and puppet expect things.
Tuts+ Vagrant / Puppet Course
If you subscribe to Tuts+, they have a great overview course on the topic of vagrant and puppet.
Mitchell Hashimoto talks about vagrant and packer at the 2013 Puppet Conf. This was not covered in this article, but is certainly something worth looking into and keeping an eye on as it matures.