fennec_precompile

cocoa-xu/fennec_precompile

Apache-2.0

Drop-in library for `:elixir_make` for precompiling NIF binaries with Zig as the cross-compiler.

0000

View program on GitHub

FennecPrecompile

(This repo is archived, please use cc_precompiler)

Drop-in library for :elixir_make for precompiling NIF binaries with Zig as the cross-compiler.

This work is inspired by (massively copy-and-paste from) rustler_precompiled. However, this library is more focused on crosscompiling C/C++ projects using Zig as a cross-compiler whereas rustler_precompiled is focused on crosscompiling Rust projects to NIF using Rust with rustler.

Installation

If available in Hex, the package can be installed by adding fennec_precompile to your list of dependencies in mix.exs:

def deps do
  [
    {:fennec_precompile, "~> 0.3.0"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/fennec_precompile.

Usage

Setup `elixir_make`

Add :elixir_make to the compilers list and set make_precompiler to FennecPrecompile to use fennec_precompile.

Setup fennec_precompile specific config values

fennec_base_url should be set to the base URL of the precompiled binaries. fennec_nif_filename will be used to indicate the file name of the compiled shared library.

@version "0.1.0"
def project do
  [
    # ...
    version: @version,
    compilers: [:elixir_make] ++ Mix.compilers(),
    make_precompiler: FennecPrecompile,
    fennec_nif_filename: "stb_image_nif",
    fennec_base_url: "https://github.com/me/myproject/downloads/releases/v#{@version}"
    # ...
  ]
end

A table of supported environment variables, their scopes and examples can be found in the Enviroment Variable section.

Precompile NIFs

Precompiling happens when run mix elixir_make.precompile.

# optional settings to override the default cache directory
export ELIXIR_MAKE_CACHE_DIR="$(pwd)/cache"

# precompile
mix elixir_make.precompile

# it's also possible to run `mix elixir_make.precompile` with other flags 
# other flags will be passed to `:elixir_make`
mix elixir_make.precompile --my-flag

What happens when you run mix elixir_make.precompile?

CC will be set to zig cc -target "ARCH-OS-ABI"
CXX will be set to zig c++ -target "ARCH-OS-ABI"
CPP will be set to zig c++ -target "ARCH-OS-ABI"

Everything else is the same as when you run mix compile (with :elixir_make, or mix compile.elixir_make).

The following targets will be compiled by default:

macOS
- x86_64-macos
- aarch64-macos
Linux
- x86_64-linux-gnu
- x86_64-linux-musl
- aarch64-linux-gnu
- aarch64-linux-musl
- riscv64-linux-musl
Windows
- x86_64-windows-gnu

A full list of supported targets can be found using zig targets.

It's worth noting that some targets may not successfully compile on certain platforms. For example, x86_64-macos will not compile on Linux and x86_64-windows-msvc will not compile on macOS.

Specifying targets to compile

To compile for a specific target/a list of targets, set the FENNEC_PRECOMPILE_TARGETS environment variable.

# for example, to compile for aarch64-linux-musl,riscv64-linux-musl
export ELIXIR_MAKE_CACHE_DIR="$(pwd)/cache"
export FENNEC_PRECOMPILE_TARGETS="aarch64-linux-musl,riscv64-linux-musl"
mix elixir_make.precompile

Fetch Precompiled Binaries

To fetch precompiled binaries, run mix elixir_make.fetch.

# fetch all precompiled binaries
mix elixir_make.fetch --all
# fetch specific binaries
mix elixir_make.fetch --only-local

# print checksums
mix elixir_make.fetch --all --print
mix elixir_make.fetch --only-local --print

Use zig for native build

This section only relates to the behaviour of the mix compile and mix compile [--args] ... commands.

For native build, zig is not used by default for two reasons.

For users who are only interested in using the app their native host, it is not necessary to compile the app using Zig.
As this tool aim to be a drop-in replacement for :elixir_make, the default behaviour of mix compile and mix compile [--args] ... of this tool is the same as what would be expected with :elixir_make.

However, you can choose to always use zig as the compiler by setting environment variable FENNEC_PRECOMPILE_ALWAYS_USE_ZIG to true.

To be more specific, by default, the environment variables CC, CXX and CPP will not be changed by this tool when running mix compile or mix compile [--args] .... When FENNEC_PRECOMPILE_ALWAYS_USE_ZIG is true, the compiled NIF binaries (for the native host, identified as ARCH-OS-ABI) should be the same as the one generated by mix elixir_make.precompile.

For example, when running mix compile or mix compile [--args] on arm64 macOS with this option set to true, files in the _build/${MIX_ENV}/lib/my_app/priv directory should match the ones in the my_app-nif-NIF_VERSION-aarch64-macos-VERSION.tar.gz generated by mix elixir_make.precompile.

To install Zig from a package manager, please refer to the officail guide from zig, Install Zig from a Package Manager.

Where is the precompiled binaries?

The path of the cache directory is determined in the following order:

$ELIXIR_MAKE_CACHE_DIR
:filename.basedir(:user_cache, "", ...)

If the environment variable ELIXIR_MAKE_CACHE_DIR is set, the cache directory will be $ELIXIR_MAKE_CACHE_DIR. Otherwise, the cache directory will be determined by the following function:

cache_opts = if System.get_env("MIX_XDG"), do: %{os: :linux}, else: %{}
cache_dir = :filename.basedir(:user_cache, "", cache_opts)

cache_dir =
  System.get_env("ELIXIR_MAKE_CACHE_DIR", cache_dir)
  |> Path.join(sub_dir)

Environment Variable

FENNEC_PRECOMPILE_TARGETS

Only used when running mix elixir_make.precompile. This environment variable is mostly used in CI or temporarily specify the target(s) to compile.

It is a comma separated list of targets to compile. For example,
```
export FENNEC_PRECOMPILE_TARGETS="aarch64-linux-musl,riscv64-linux-musl"
mix elixir_make.precompile
```
If FENNEC_PRECOMPILE_TARGETS is not set, the fennec_precompile will then check config/config.exs to see if there is a :fennec_targets key for my_app. If there is, the value of the key will be the targets.
```
import Config

config :fennec_precompile, :config, my_app: [
  fennec_targets: ["aarch64-linux-musl", "riscv64-linux-musl"]
]
```
:fennec_targets in the project will only be used in the following cases:
```
def project do
  [
    # ...
    fennec_targets: ["aarch64-linux-musl", "riscv64-linux-musl"]
  ]
end
```
1. When :fennec_force_build is set to true. In this case, the :fennec_targets acts as a list of compatible targets in terms of the source code. For example, NIFs that are specifically written for ARM64 Linux will fail to compile for other OS or CPU architeture. If the source code is not compatible with the current node, the build will fail.
2. When :fennec_force_build is set to false. In this case, the :fennec_targets acts as a list of available targets of the precompiled binaries. If there is no match with the current node, no precompiled NIF will be downloaded and the app will fail to start.
FENNEC_PRECOMPILE_ALWAYS_USE_ZIG

Only used when running mix compile or mix compile [--args] ....

It is a boolean value. When set to true, zig will be used as the compiler instead of the default $CC, $CXX or $CPP. For more information, please refer to the section above, Use zig for native build.
```
# this is the default, equivalent to run `mix compile` with `:elixir_make`
unset FENNEC_PRECOMPILE_ALWAYS_USE_ZIG
mix compile

# this will force using zig as the compiler
export FENNEC_PRECOMPILE_ALWAYS_USE_ZIG=true
mix compile
```

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.