New nim-ffi repository

Logos
1

Background

At nwaku, we have implemented a mechanism that exposes a Waku asynchronous library, which can be integrated into other languages such as Golang, Rust, Python, or C++.

To achieve this, a separate thread is started, and the host language thread (Go, Rust, etc.) interacts with it.

This technique is starting to be used in other Nim projects, such as nim-sds. And there will be many more projects willing to follow the same approach.

However, this approach has the drawback of duplicated code being scattered across different repositories. Therefore, we need a unified solution for all Nim projects that aim to be integrated universally: nim-ffi.

nim-ffi Proposal

The new nim-ffi project will:

  1. Contain the current ffi logic, implemented within nwaku, and offer a generic solution.
  2. Abstract the complexity involved in the current threading communication.
  3. Be a separate repository that can included in any Nim project, preferably via Nimble.
  4. Define the {.ffi.} pragma (and others) that we help to expose Nim functions as a C library functions.
  5. Not introduce additional memory or CPU usage overhead.
  6. Be a generic solution for solving FFI problems, similar to how nim-chronos solves async problems in any Nim project.

The investment required to implement nim-ffi will be significant, but it will be well worth it because any Nim project will then be able to integrate seamlessly across environments. This work will likely require extensive metaprogramming efforts.

nim-ffi Development Phases

  1. Basic extraction of nwaku libwaku into the nim-ffi repository to allow nwaku to use it.
  2. nim-sds integrates nim-ffi.
  3. nim-chat-sdk uses nim-ffi.
  4. libRLN (still doesn’t exist today) integrates nim-ffi.
  5. Codex uses nim-ffi.

nim-ffi Operating Modes

We envision two primary ways of interacting with nim-ffi:

  1. JSON - simpler and is the current approach followed within nwaku.
  2. Protocol Buffers (Protobuff) - more complex but allows having a more robust API definition.

In the early phases of nim-ffi implementation, we will likely follow the JSON approach as it is easier for debugging purposes. Nevertheless, in the future, nim-ffi will likely support both formats to meet the needs of various project consumers.

6 Likes