Proposal for binding Nim libraries

Waku
1

Targeted benefits

  • Make a clear API as a contract defined by data.
    • Write API once in nim
    • Define data as protobuf definition
    • Generate in many target languages.
  • Eliminate C-ABI.
    • Each targeted language uses its types.
  • Slim the C/C++ glue between Nim library and host language.
  • Lower maintenance costs, faster support.

Problem statement

Currently, our libwaku is well defined and mature. However, due to Nim’s own memory management and garbage collection mechanism, the library code must run in its own thread.

This implies that API calls must be transferred to the nim thread before processing, while returning of data shall happen in callbacks - those actually to be processed on Nim’s thread.

For simplicity, we currently use json textual representation to transfer data back to the caller.

When defining a new library interface, the process involves a lot of manual work, it needs to use C-types for data exchange, both on the target language and the Nim library side.

Proposal

Having protobuf as a common language glue between Nim library and basically any kind of target language comes with several benefits.

  • Protobuf data definition stands for API contract that our library offers.
  • Allow to use target language’s own data types.
  • Glue code can be generated.
  • Encoded buffer can be easily transferred between threads without manual handcrafting code.

Ideal solution

1526×550 68.1 KB

The ideal solution shall:

  • Generate as much code as possible from protobuf definitions
    • This can be achieved by protoc code generator plugin
  • C/C++ glue shall be library and API agnostic
    • Write once and used for many Nim libs
  • With bi-directional data pipes between the user and a Nim library, we can detach application processing from Nim processing.
    • Synchronous or asynchronous modes can be supported.
1 Like
2

Thanks for the proposal!
We need to evaluate the pros and cons.
From my PoV:
Pros:

  • faster than JSON

Cons:

  • More difficult to debug. JSON is simpler as it is raw text that can be printed easily.
  • We are adding the “glue” component, which adds another point of failure and dependency even though is generated.

Btw , afaik , the following is already supported:

  • Allow to use target language’s own data types.

Also, notice that we cannot use any GC’ed type in the interface between the host and the Nim’s thread. The memory should be managed manually.

IMO, this is very interesting to consider but only if there are big flaws in the current approach, which I consider there aren’t.

Nevertheless, even though at first I don’t quite buy the idea, I strongly support you and ping me if need help creating a PoC for Rust, Go, or other language. I think the best way to sell it is having a plausible example.

Thanks again for it! :raised_hands:

3

I can definitely see a benefit in only having to pass a protobuf definition to the host language that can then use a generator to define types and de/serializing.

While something similar existing for JSON via JSON schema:

  • It’s cumbersome
  • Do you even use it?

The “can print JSON on log” point seems moot, most modern language enables stringification when logging, especially when it’s a generated type accompanied with protobuf de/ser.

I have locked in a PoC deliverable to further explore this path, as I expect that we, and other teams in Logos, will end maintaining more and more C bindings.