Category Archives: Programming

Using pretrained AI models in Elixir / Phoenix application

Posted on by
Reply

This year the Elixir team has made a huge progress in the AI area, it has started with NX, then came Axon with somewhat higher level API and finally Bumblebee which makes using pretrained AI models and integrating them into Phoenix app really easy (I hate it when people say that, but in this case it’s the truth – it’s easy).

This Christmas I finally got to play with it and was surprised at how well it went. Bumblebee is still very new, the documentation is “fresh”, but despite that, it works quite well. There are also few libraries that would help working with the data (Scholar, Explorer, all came this year), but the important part is that we now have the ability to use available open source models without the need to train them.

I am not sure how commercially viable it is, but it’s definitely fun to play with 🙂

Here’s the result showcasing few models integrated into a sample Phoenix application.

How To Fix Encoding Error In Postgres

Posted on by
Reply

Here is an example error when trying to create a database in Postgres:

ERROR 22023 (invalid_parameter_value) encoding "UTF8" does not match locale 
The chosen LC_CTYPE setting requires encoding "LATIN1".

Such an error might come up if OS locale was not set while installing Postgres. It is fixable by shuffling templates or by reinstalling Postgres after locale is set, I prefer the latest, cause having a fresh install just gives me more peace of mind. To fix that, we must first configure locale, on Debian/Ubuntu it would be something like that (the command will trigger an interactive prompt):

Continue reading →

GitHub Actions CI Pipeline For An Elixir / Phoenix App

Posted on by
Reply

Here is a complete action that can be put into .github/workflows/ci.yml so that tests and other checks (formatter, Credo, Dialyzer) are ran automatically on each push and each pull request:

name: CI

on: [push, pull_request]

env:
  MIX_ENV: test

jobs:
  build-and-test:
    name: Build and Tests
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-20.04]
        elixir: [1.11.2]
        otp: [23.3.4]

    services:
      postgres:
        image: postgres:12.6
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: kiz_test
        ports:
          - 5432:5432
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

    steps:
    - name: Check Out Repository
      uses: actions/checkout@v2

    - name: Prepare Application
      run: cp config/test.secret.ci.exs config/test.secret.exs

    - name: Setup Elixir
      uses: actions/setup-elixir@v1
      with:
        elixir-version: ${{ matrix.elixir }}
        otp-version: ${{ matrix.otp }}
        experimental-otp: true

    - name: Restore Dependencies, Build & Dialyzer PLTs Cache
      uses: actions/cache@v2
      with:
        path: | 
          deps
          _build
          priv/plts
        key: ${{ runner.os }}-mix-${{ matrix.elixir }}-${{ matrix.otp }}-${{ hashFiles('**/mix.lock') }}
        restore-keys: ${{ runner.os }}-mix-${{ matrix.elixir }}-${{ matrix.otp }}

    - name: Install Dependencies
      run: mix deps.get

    - name: Setup DB
      run: mix ecto.setup

    - name: Run Tests
      run: mix test

    - name: Check Formatting
      run: mix format --check-formatted

    - name: Review Code With Credo
      run: mix credo

    - name: Typecheck With Dialyzer
      run: mix dialyzer

Few things to note with that setup. Continue reading →

Elixir Releases: Switching App Version

Posted on by
Reply

Since version 1.9 Elixir’s Mix has got a new command that helps building Erlang releases. It’s usage is straightforward and well documented, but one thing that deployments using releases brings, is the possibility to fallback to any earlier version of the app that has been released, this feature did not get much love in the docs or around the Internet.

The documentation mentions following environment variable:

RELEASE_VSN – the version of the release, otherwise the latest version is used. It can be set to a custom value when invoking the release. The custom value must be an existing release version in the releases/ directory

That means when starting the app we can set that variable to any existing release version (releases directory contains all of them and can be used as reference) to run that specific version, everything is bundled, so in most cases that would even use older runtime version when needed.

If our release runs in prod environment and is named “live” then the commands from the project root might look like that: _build/prod/rel/live/bin/live start starts the default (latest) version and RELEASE_VSN=0.0.1 _build/prod/rel/live/bin/live would start the version 0.0.1 of the app (that must have been built at some point earlier) even if that’s not the last version. The ls _build/prod/rel/live/releases/ would list all the versions that are available.

One thing to keep in mind is that if you use Systemd to manage the application on the OS level and would try the same there (e.g. RELEASE_VSN=0.0.1 sudo systemctl start myapp.service) that wouldn’t work, Systemd would just ignore the variable. The reason: Systemd deals with different life-cycle stages of the operating system and the variables are loaded at different points of time so relying on them would do more harm then good.

To make a variable known to Systemd a separate systemctl command should be used, for the above example it would be sudo systemctl set-environment RELEASE_VSN=0.0.1. After running this command, the variable is set and will be seen by the services, so starting / restarting it (sudo systemctl restart myapp.service) will activate the needed version. To reset the variable and fall back to the latest version, just unset it like that: sudo systemctl unset-environment RELEASE_VSN.

Elixir Ecto: Wildcard search a Postgres JSONB field

Posted on by
Reply

This seemingly straightforward task made me do some digging. Imagine we have a table with JSONB Field and we want to search for specific field values using wildcard character to allow partial matches.

Ecto doesn’t have a JSONB support but this particular task is so common that it’s even mentioned in the documentation:

fragment("?->>? ILIKE ?", p.map, "key_name", ^some_value)

What we need is just add the wildcard characters around the value. We can’t do that directly (“?->>? ILIKE %?%”) though since it would trigger Ecto’s syntax error and the query wouldn’t compile.
Continue reading →

Streaming Data From The Database In Elixir

Posted on by
Reply

Say we have a search feature and an index_article/1 function that knows what to do with the given article – can fetch all related data, concatenate it into a tsvector field and then save in the search cache table. With it we can index any article – so far so good.

Now what if we want to rebuild the whole search index? Looping over each entry in the database and indexing it with the existing function would work, but will require loading everything in RAM which will definitely break in the long run.

Usually we would create some kind of Task that loops as long as there are unprocessed entries in the Database, takes a number of those entries, processes each of them adjusts offset etc.

Well, Elixir can do better, using Repo.stream/2 we can create a stream that we can use with the Stream module, this way we can greatly simplify the above logic.

Here is the whole function that reindexes all articles of the given type:
Continue reading →

Elixir – Testing Phoenix Flash Messages

Posted on by
3

At times in my integration tests I want to check if a user sees a flash message. It’s easy when re-rendering a template, just check the response body like that:

assert html_response(conn, 200) =~ "some message"

UPD as pointed out by Dave Lugg there is also a built in helper that can do that:

assert get_flash(conn, :info) == “some Message”
assert get_flash(conn, :info) =~ “part of some Message”

But often we set flash messages and redirect the user, in this case response body would not contain them and we’d have to check against the data that is saved on the conn struct, this is doable using Phoenix controller module:

assert(
    conn
    |> Phoenix.Controller.get_flash()
    |> Enum.any?(fn(item) -> String.contains?(elem(item, 1), "some message") end))

which works, but doesn’t look very nice and clutters the tests. So the cleaner solution would be to pack the whole thing in a test helper function and put it in test/support (anything in this directory will be compiled when running tests). Here is an example of such a helper (file test/support/helper.ex): Continue reading →

Elixir – Nested Changesets With Phoenix

Posted on by
Reply

Say we have a user registration form where the user enters their name, email etc (User schema) as well as their password (Authorization schema). Both have to be validated, both require extra actions and belong together, so if anything fails the whole thing has to roll back and give meaningful feedback.

This is doable using Ecto.Multi (or nested case with transactions in controllers, but that would be ugly). Here is an alternative to Ecto.Multi that keeps it as minimalistic as possible while still allowing for much flexibility when dealing with complex user inputs: nested forms with custom changesets (note: Ecto 2 is needed). Continue reading →