The InfluxDB 3 Processing engine is an embedded Python VM for running code inside the database to process and transform data. This document discusses how the processing engine is built within InfluxDB. For usage instructions, see: https://docs.influxdata.com/influxdb3/core/

See the 'Discussion' section for more information on why the processing engine is implemented the way it is.

Implementation

InfluxDB uses the PYO3 crate to build InfluxDB with an embedded python and the processing engine is enabled by default.

PYO3 will then inspect the system to find a python runtime to build and link against. The resulting influxdb3 binary will be dynamically linked to the libpython that PYO3 found during the build. Eg, on a typical Debian or Ubuntu system, if you install the following, then InfluxDB can be built against the system's python:

# build dependencies
$ sudo apt-get install build-essential pkg-config libssl-dev clang lld \
    git protobuf-compiler python3 python3-dev python3-pip

# runtime dependencies
$ sudo apt-get install python3 python3-pip python3-venv

# build
$ cargo build

The choice of python can be influenced by setting the PYTHONHOME environment variable for cargo build or creating a PYO3_CONFIG_FILE file for more specialized setups (such as 'Official builds', below). For details, see https://pyo3.rs/main/building-and-distribution.html

In order for InfluxDB to successfully use the python it was built against, the same libpython version as well as the full runtime environment of the python install (ie, its standard library) must be available to InfluxDB in a location that it can find it. Building against the system python can be a reasonable choice for users who target their builds to a specific release of an OS as InfluxDB will simply use the installed python from the system.

Official builds

To provide a consistent, robust and maintained python environment for InfluxDB that is portable across a range of operating systems, InfluxData's official InfluxDB is built against a pre-built release of python-build-standalone (a CPython standalone python distribution). For a given release of InfluxDB, official builds will use the same version of python for all install methods and operating systems.

The following operating systems and architectures are currently supported:

• Linux amd64/arm64 (tar.gz, deb and rpm)
• Darwin arm64 (tar.gz)
• Windows amd64 (zip)
• Docker (Linux amd64/arm64)

Due to constraints with python-build-standalone and statically linking, all builds are dynamically linked to python-build-standalone's libpython as well as a few OS-specific libraries. Specifically:

• Linux (seen with ldd and strings on the binary):
• python-build-standalone is linked against glibc and is compatible with glibc 2.17+
• influxdb3 is linked against libpython from python-build-standalone as well as glibc (official builds target glibc 2.23+)
• Darwin (seen with otool -L; cross-compiled with osxcross):
• python-build-standalone is linked against:
  • CoreFoundation.framework/Versions/A/CoreFoundation compatibility version 150.0.0
  • libSystem.B.dylib compatibility version 1.0.0
• influxdb3 is linked against:
  • CoreFoundation.framework/Versions/A/CoreFoundation compatibility version 150.0.0
  • IOKit.framework/Versions/A/IOKit compatibility version 1.0.0
  • libiconv.2.dylib compatibility version 7.0.0
  • libobjc.A.dylib compatibility version 1.0.0
  • libSystem.B.dylib compatibility version 1.0.0
  • Security.framework/Versions/A/Security compatibility version 1.0.0
  • SystemConfiguration.framework/Versions/A/SystemConfiguration compatibility version 1.0.0
• Windows (seen with dumpbin /HEADERS ... and dumpbin /DEPENDENTS ...):
• python-build-standalone claims Windows 8/Windows Server 2012 or newer. Specifically, it has:
  • 14.42 linker version
  • 6.00 operating system version
  • 6.00 subsystem version
• influxdb3 has:
  • 2.40 linker version
  • 4.00 operating system version
  • 5.02 subsystem version

At a high level, the build process for Official builds consists of: 1. downloading an appropriate build of python-build-standalone for the target OS and architecture from https://github.com/astral-sh/python-build-standalone/releases 2. unpacking the python-build-standalone build on disk 3. creating a pyo3 build configuration file to point to the unpacked directory and setting the PYO3_CONFIG_FILE environment variable to point to it. Eg (on Linux):

```
implementation=CPython
version=3.13
shared=true
abi3=false
lib_name=python3.13
lib_dir=/path/to/python-standalone/python/lib
executable=/path/to/python-standalone/python/bin/python3.13
pointer_width=64
build_flags=
suppress_build_script_link_lines=false
```

PYO3 will try to auto-detect the location which can work well with a system
python, but not with an unpacked `python-build-standalone`. While the
`PYO3_PYTHON` environment variable can be used to point to the unpacked
directory (eg,
`PYO3_PYTHON=/path/to/python-standalone/python/bin/python3`), this was not
sufficient. Defining the build configuration in the `PYO3_CONFIG_FILE`
correctly worked for all supported environments with our current build
process

1. run PYO3_CONFIG_FILE=/path/to/pyo3_config_file.txt cargo build to build InfluxDB
2. adjust the library search paths for Linux and Darwin so libpython can found (see 'Discussion', below)

3. create official build artifacts:

4. Linux/Darwin tar.gz contain influxdb3 and python/...

5. Linux deb and rpm contain /usr/bin/influxdb3 and /usr/lib/influxdb3/python
6. Windows zip contains influxdb3, *.dll files from python/... and python/... (see 'Discussion', below)

Licensing information for python-build-standalone as distributed by official builds of InfluxDB can found in the python/LICENSE.md.

With the above, influxdb3 can be run in the normal way. Eg, on Linux:

# unpack tarball to /here
$ tar -C /here --strip-components=1 -zxvf /path/to/build/influxdb3-<VERSION>_linux_amd64.tar.gz

# without processing engine
$ /here/influxdb3 serve ...
$ /here/influxdb3 query ...

# with processing engine (/path/to/plugins/.venv created automatically)
$ mkdir /path/to/plugins
$ /here/influxdb3 serve --plugin-dir /path/to/plugins ...        # server
$ /here/influxdb3 create database foo                            # client
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
... <plugins can use whatever is in /path/to/plugins/.venv> ...
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar'> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

# with processsing engine and alternate venv (/path/to/other-venv created automatically)
# start server to create/use other-venv
$ /here/influxdb3 serve --plugin-dir /path/to/plugins --virtual-env-location /path/to/other-venv
...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
... <plugins can use whatever is in /path/to/other-venv> ...
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar'> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

# with processsing engine and alternate pre-created venv
$ /here/python/bin/python3 -m venv /path/to/another-venv         # create another-venv
$ source /path/to/another-venv/bin/activate
(venv)$ python3 -m pip install foo
(venv)$ python3 -m pip freeze > /path/to/another-venv/requirements.txt
...
(venv)$ deactivate

# start server to use another-venv
$ /here/influxdb3 serve --plugin-dir /path/to/plugins --virtual-env-location /path/to/another-venv
... <plugins can now use whatever is in /path/to/another-venv> ...

$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client
$ /here/influxdb3 install package bar                            # client
... <plugins can now 'import bar' in /path/to/another-venv> ...
$ /here/influxdb3 test schedule_plugin -d foo testme.py          # client

Local development with python-build-standalone

Local development with python-build-standalone currently consists of:

1. download python-build-standalone and unpack it somewhere
   • get from https://github.com/astral-sh/python-build-standalone/releases
   • based on your host OS, choose one of aarch64-apple-darwin-install_only_stripped.tar.gz, aarch64-unknown-linux-gnu-install_only_stripped.tar.gz, x86_64-pc-windows-msvc-install_only_stripped.tar.gz, x86_64-unknown-linux-gnu-install_only_stripped.tar.gz

2. create pyo3_config_file.txt to match the unpacked dir and downloaded python version. Eg, if downloaded and unpacked a 3.13.x version to /tmp/python:

   $ cat ./pyo3_config_file.txt implementation=CPython version=3.13 shared=true abi3=false lib_name=python3.13 lib_dir=/tmp/python/lib executable=/tmp/python/bin/python3.13 pointer_width=64 build_flags= suppress_build_script_link_lines=false

3. build with:

   ```

   note: PYO3_CONFIG_FILE must be an absolute path

   $ PYO3_CONFIG_FILE=${PWD}/pyo3_config_file.txt cargo build --features "aws,gcp,azure,jemalloc_replacing_malloc" ```

4. Linux/OSX: patch up the binary to find libpython:

   ```

   linux

   $ patchelf --set-rpath '$ORIGIN/python/lib' ./target//influxdb3

   osx (be sure to match the libpython version with what you downloaded)

   $ install_name_tool -change '/install/lib/libpython3.13.dylib' '@executable_path/python/lib/libpython3.13.dylib' ./target//influxdb3 ```

5. Linux/OSX: put the python runtime in the expected location (XXX: may be possible at run time to see where the libpython we are using is and adjust the code to base the location of the runtime on that). Eg, if unpacked to /tmp/python:

   $ test -e ./target/<profile>/python || ln -s /tmp/python ./target/<profile>/python

6. run with:

   ``` $ mkdir -p /path/to/plugin/dir

   linux and osx (if can't find libpython or the runtime, check previous steps)

   $ ./target//influxdb3 ... --plugin-dir /path/to/plugin/dir

   windows requires moving the binary into the python-build-standalone unpack directory

   $ cp ./target//influxdb3 \path\to\python-standalone\python

   run influxdb with

   $ \path\to\python-standalone\python\influxdb3.exe ... --plugin-dir \path\to\plugin\dir ```

Discussion

Why python-build-standalone?

python-build-standalone is designed to be portable, maintained and permissively licensed. It is purpose-built for embedding and being redistributable and has a good upstream maintenance story (https://github.com/astral-sh) with lots of users and a corporate sponsor.

An alternative to using a standalone python distribution is to use the system python. While this can be a reasonable choice on systems where the python version and installation locations can be relied upon, it is not a good choice for official builds since users would have to ensure they had a python installation that met InfluxDB's requirements and because the myriad of operating systems, architectures and installed python versions would be a problem to support.

By choosing python-build-standalone, InfluxDB should deliver a consistent experience across OSes and architectures for all users as well as providing a reasonable maintenance story.

Which builds of python-build-standalone are used?

python-build-standalone provides many different builds. Official InfluxDB builds use the following python-build-standalone recommended builds:

• aarch64-apple-darwin-install_only_stripped.tar.gz
• aarch64-unknown-linux-gnu-install_only_stripped.tar.gz
• x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
• x86_64-pc-windows-msvc-install_only_stripped.tar.gz

How will InfluxData maintain the embedded interpreter?

The https://github.com/astral-sh project performs timely builds of CPython micro-releases for python-build-standalone based on the release cadence of upstream Python. InfluxData need only update the build to pull in the new micro-release for security and maintenance releases. This is done by updating the PBS_DATE and PBS_VERSION environment variables in .circleci/config.yaml. See that file and .circleci/scripts/fetch-python-standalone.bash for details.

astral-sh creates new builds for CPython minor releases as they become available from upstream Python. Updating the official builds to pull in a new minor release is straightforward, but processes for verifying builds of InfluxDB with the new python-build-standalone minor release are TBD.

References: * https://www.python.org/dev/security/ * https://mail.python.org/mailman3/lists/security-announce.python.org/ * https://mail.python.org/archives/list/security-announce@python.org/la