For years I’ve been using a Mac for development and it’s been great. But when I switched to a new Macbook Pro M1, an inconvenient truth emerged: I’ve been running development and production on different platforms. Year after year, for bash, python, Ruby, Rails, deploying to real servers, AWS EC2, Docker/Fargate, etc, everything just worked… until now!

Most everything worked fine out of the box, but a few M1 issues ended up wasting a lot of time, so I’m writing down my results.

Python/pip

Everything installed without a hitch except pip install pyodbc. The get that working, I had to Use Homebrew to install the unixodbc package (Maybe you have to do this no matter what, I can’t remember.) And then I had to run this from the command line before the pip install:

export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/2.3.9_1/lib"
export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/2.3.9_1/include"

After that it was smooth sailing. All the packages installed OK individually and via the -r option with a requirements file.

Ruby, rbenv

At the time of this post, I’m running Ruby 2.6.5 and 2.6.6 apps and both work with this solution.

I use rbenv to manage Ruby environments, and installed it using Homebrew, but none of the Ruby versions I wanted could be installed via rbenv. After a lot of digging, I found I needed to set CFLAGS like this:

CFLAGS="-Wno-error=implicit-function-declaration" rbenv install 2.6.5

Apparently the builds would have worked fine except that some “errors” should probably be just warnings. You have to turn them off so the build completes. Whatever. This makes it work and the installed Ruby versions all work fine.

bundler

It’s actually not fair to blame all the package build problems on bundler, but bundle install is where the shit hits the fan and it is where you have to fix it.

The first problem was that the mysql2 gem would not install with “ld: library not found for -lssl.” Two steps to solve this, and the first was to use Homebrew to install openssl@1.1. (Not openssl or openssl@3. Harrrumph.) Then the command line solution turned out to be this, apparently because of the far-too-typical inability to find the right openssl installation.

gem install mysql2 -v '0.5.3' -- --with-opt-dir=$(brew --prefix openssl@1.1) 

You can’t use this form when installing via bundle install, but you can use the following command so add a build setting for bundler:

bundle config --global build.mysql2 --with-opt-dir=$(brew --prefix openssl@1.1)

This saves the setting globally (or rather “user-ly”) at ~/.bundle/config. Note that omitting the –global option will save the setting at the project level, leaving you with the choice to include it in the git repo (where it will be included in any deployments, possibly screwing up Docker builds, etc) or .gitignore-ing it (so it will be gone if you clone the repo again.) No great options here, but –global seems best.

Everything else was resolved by running bundle update to get newer versions of a few gems that do not seem to be backported (yet?) for the Mac M1.

Docker/Fargate, the naive solution

Now the big problem, the real development/production platform mismatch problem: Docker. On the surface it might seem like the whole container thing would make this a non-issue, but the containers actually run on the host platform.

The default platform for Docker containers is linux/amd64, and that works with most systems including Intel Macs, Fargate deployments, etc, so it was never a worry. But the Mac M1 platform is linux/arm64 (and a pox on whomever chose such similar-looking names as amd64 and arm64. Grrr.) Using all the old code and configurations, the Docker images initially would not build at all. To get this working, I had to:

• Remove the version from the Dockerfile’s bundler command to get the latest
• Update the base linux image form buster to bullseye, since the old version did not seem to be available for arm64
• And ran “bundle config set force_ruby_platform true” from the command line to add the “ruby” platform to the Gemfile, and then bundle update to get some fresh gems.

So far so good. Now the app would build and run in Docker Desktop. But when I pushed the images to Fargate (and you saw this coming, right?) the tasks would not run because my arm64 images would not work with the Fargate amd64 architecture.

Bundler Docker and Fargate, the final solution

After careful consideration, I decided there was really no value in having a local Docker solution except for testing the production containers, so no point in building arm64 containers. With a greater understanding of the end-to-end, I implemented this:

In my Gemfile.lock, I edited the platform section to this:

PLATFORMS
  -darwin-21
  ruby
  x86_64-linux

The first entry guarantees the bundle includes versions that will work for development locally, and at least the last one provides versions that will work for the Docker amd64 build. I have no idea what the “ruby” entry does but with these platforms, the bundle install works both locally and for the Docker amd64 image.

Now the only problem is how to get the docker build to create images for amd64 even though the local machine is a different architecture. There are two ways that worked:

• Start the Dockerfile with “FROM –platform=linux/amd64 <base image name>” instead of just invoking the base image.
• Use the relatively new buildx facility to set the platform at build time.

It seems about the same, but our process worked best with the latter, so we build like this:

docker buildx build --platform linux/x86_64

(We also specify a specific Dockerfile and several tags as options, but that’s really not part of the M1 solution.)

FINALLY with these changes, we can

• Develop and run everything locally
• Deploy amd64 Docker images that run on AWS Fargate
• And AMAZINGLY we can run the same Docker locally because Docker Desktop current version apparently runs an emulator when needed. At this time the only glitch is a few apparently-innocuous error messages in the startup output.

Final notes

If you are up on all this business you would reasonably ask, “why not multi-platform Docker images.” But that sure seems like we’d still not really be running the same code in the containers locally vs Fargate, so in our environment, what’s the point?

Hopefully this will all get better. Eventually I expect most of the special cases will go away and this will all feel seamless again. But if you have been doing this for as long as I have, the current solution does not seem that bad.