Deploying Elixir / Phoenix App Using Rolling Updates

As discussed in this post we gave rolling updates a shot after trying the releases, this kind of deployment is flexible and fast, so until we really need hot updates we’ll stay there.

There are few things to note though when deploying to production (our Server runs Ubuntu 16.04).

System Service

Though we could just run MIX_ENV=prod PORT=4000 mix phoenix.server or more elaborate MIX_ENV=prod PORT=4000 elixir –detached -S mix do compile, phoenix.server we’d like to go one step further and “supervise” our main process so that it starts and restarts itself on system startup as well as in any possible “things go wrong” scenarios.

Now this one is a little trickier since we (and most examples on the net) always used Upstart services to handle such cases and Ubuntu 16.04 switched to using systemd. systemd is a cool init system, but it differs from Upstart not only syntactically but also logically – It doesn’t have much environment setup as Upstart did, so we have to get our hands a little bit dirtier.

Here is an example of systemd service that manages an elixir app, in Ubuntu 16.04 it would have to be placed in /lib/systemd/system/myservice.service

[Unit]
Description=My app daemon

[Service]
Type=simple
User=myappuser
Group=myappgroup
Restart=on-failure
Environment=MIX_ENV=prod "PORT=4000"
Environment=LANG=en_US.UTF-8

WorkingDirectory=/var/apps/my-app

ExecStartPre=/usr/local/bin/mix compile
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

After setting it up use systemctl enable myservice.service, this must only be done once so that the system creates symlinks to the service file. Without this step everything will work fine but will not be restarted on boot.

Any generic “my app” things like user, group and application path have to be replaced with real stuff. Few other things to note:

• Environment=LANG=en_US.UTF-8 this part is important, otherwise Elixir will complain about Erlang nut using utf8, you’d get something like this logged: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running “locale” in your shell). This issue has to do with systemd being “bare bones”, actually running “locale” will probably show a correct encoding which might create confusion
• WorkingDirectory=/var/apps/my-app this makes sure there is a phoenix.server command available
• ExecStart=/usr/local/bin/mix phoenix.server systemd has no path environment so we have to use full path to the mix executable. Use whereis mix to find it in your environment

After initial setup you will be able to manage your service just as any other system service:

sudo systemctl status myservice.service

sudo systemctl restart myservice.service

sudo systemctl start myservice.service

sudo systemctl stop myservice.service

systemd’s status will also show latest IO output of your app which is very helpful when debugging the service.

Logs

If you have Phoenix’ default logging to standard output, you can use journalctl to view logs, here are few examples

journalctl -u myservice.service --since today
journalctl -u myservice.service --since 09:00 --until "1 hour ago"
journalctl -u myservice.service --since "2016-11-10 12:00" --until "2016-11-10 13:00"

Nginx proxying

Though not required, using nginx as a proxy makes it easier to “play well” with other parts of the system by using port 80 and not having to run root processes, behind the scenes nginx will use root to set things up and then goes down to its normal user. Since it’s only used as a proxy there is barely an overhead.

Here is a typical config (thanks to the official Phoenix guide):

upstream phoenix {
  server 127.0.0.1:4000 max_fails=5 fail_timeout=60s;
}

server {
  server_name my-app.com;
  listen 80;

  location / {
    allow all;

    # elixir / phoenix is probably not vulnerable to httproxy but we reset the header anyway just to make sure
    proxy_set_header Proxy "";    

    # Proxy Headers
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Cluster-Client-Ip $remote_addr;

    # The Important Websocket Bits!
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_pass http://phoenix;
  }
}

One thing to note in this config aside from default forwarding: we reset the Proxy header to mitigate the (potential) proxy vulnerability.

Deployment Commands

In the end here’s whats happening during an automated rolling deployment of a Phoenix app:

1. Pull changes
2. Run mix deps.get –only prod to pull production dependencies
3. Run mix ecto.migrate to migrate the database (if used)
4. Run brunch build –production and MIX_ENV=prod mix phoenix.digest to generate production assets (if any)
5. Run MIX_ENV=prod mix compile to compile the app
6. Run sudo systemctl restart myservice.service to restart the service, it might make sence to allow this specific command to not require password (by editing sudoers using visudo)
7. Run mix test to make sure nothing is broken

The above steps can be automated and integrated in most build tools pretty easily which means in the end we get fast and secure deployment that gives us feedback if anything went wrong.