mozilla/uniffi-rs

a multi-language bindings generator for rust

UniFFI - a multi-language bindings generator for Rust

UniFFI is a toolkit for building cross-platform software components in Rust.

For the impatient, see the UniFFI user guide or the UniFFI examples.

By writing your core business logic in Rust and describing its interface in an "object model", you can use UniFFI to help you:

• Compile your Rust code into a shared library for use on different target platforms.
• Generate bindings to load and use the library from different target languages.

You can describe your object model in an interface definition file or by using proc-macros.

UniFFI is currently used extensively by Mozilla in Firefox mobile and desktop browsers; written once in Rust, auto-generated bindings allow that functionality to be called from both Kotlin (for Android apps) and Swift (for iOS apps). It also has a growing community of users shipping various cool things to many users.

UniFFI comes with support for Kotlin, Swift, Python and Ruby with 3rd party bindings available for C# and Golang. Additional foreign language bindings can be developed externally and we welcome contributions to list them here. See Third-party foreign language bindings.

User Guide

You can read more about using the tool in the UniFFI user guide.

We consider it ready for production use, but UniFFI is a long way from a 1.0 release with lots of internal work still going on. We try hard to avoid breaking simple consumers, but more advanced things might break as you upgrade over time.

Etymology and Pronunciation

ˈjuːnɪfaɪ. Pronounced to rhyme with "unify".

A portmanteau word that also puns with "unify", to signify the joining of one codebase accessed from many languages.

uni - [Latin ūni-, from ūnus, one] FFI - [Abbreviation, Foreign Function Interface]

Alternative tools

Other tools we know of which try and solve a similarly shaped problem are:

• Diplomat - see our writeup of the different approach taken by that tool
• Interoptopus

(Please open a PR if you think other tools should be listed!)

Third-party foreign language bindings

• Kotlin Multiplatform support. The repository contains Kotlin Multiplatform bindings generation for UniFFI, letting you target both JVM and Native.
• Go bindings
• C# bindings
• Dart bindings

External resources

There are a few third-party resources that make it easier to work with UniFFI:

• Plugin support for .udl files for the IDEA platform (uniffi-dl in the JetBrains marketplace). It provides syntax highlighting, code folding, code completion, reference resolution and navigation (among others features) for the UniFFI Definition Language (UDL).
• cargo swift, a cargo plugin to build a Swift Package from Rust code. It provides an init command for setting up a UniFFI crate and a package command for building a Swift package from Rust code - without the need for additional configuration or build scripts.
• Cargo NDK Gradle Plugin allows you to build Rust code using cargo-ndk, which generally makes Android library builds less painful.
• uniffi-starter is a minimal project demonstrates a wide range of UniFFI in a complete project in a compact manner. It includes a full Android library build process, an XCFramework generation script, and example Swift package structure.

(Please open a PR if you think other resources should be listed!)

Contributing

If this tool sounds interesting to you, please help us develop it! You can:

• View the contributor guidelines.
• File or work on issues here in GitHub.
• Join discussions in the #uniffi:mozilla.org room on Matrix.

Code of Conduct

This project is governed by Mozilla's Community Participation Guidelines.

uniffi-rs's Issues

Handle name collisions with internal helpers

All bindings define some internal helper methods on the records and classes they generate—things like lift, lower, liftFrom, lowerInto, lowersToSize, toFFIValue, and fromFFIValue—as well as internal types like RustBufferStream. What happens if a type declares a method in its IDL with the same name as one of those internal helpers?

This is broadly related to #2, although more specialized.

Approach one

• Blanket ban on defining methods with those names. This is probably the easiest and also most correct approach—we make the programmer think of a better name instead of trying to make it work. If we keep the set of reserved method names consistent across all our language bindings (even if they’re not necessarily consistent with the naming conventions of the target language—this is internal generated code, after all), we can keep the size of the blocklist down. Or, we can introduce language-specific blocklists. So RustBufferStream would be blocked only in Python, liftFrom, lowerInto, and lowersIntoSize only in Kotlin, and toFFIValue and fromFFIValue only in Swift.

A benefit of language-specific lists would also help us block accidental use of reserved words. For example, def is reserved in Python, but none of the others; trait can’t be an argument name in Rust, but can be used freely in others; func is reserved only in Swift, fun only in Kotlin, and so on. Of course, we could also block these globally—so you couldn’t use def, func, or trait as an argument name in any language.

Drawbacks: authors have to think of new names themselves, and it doesn’t handle extensions and subclasses (do we care?).

Approach two. Try to uniquify internal helper names. This would be more helpful for authors, but feels too magical, and has lots of corner cases and hacks. The idea is, we’d have a list of all user-defined method names, and generate a unique one for the internal helper only if there’s a collision. It keeps the generated code clean in most cases, since we’d only mangle names if we see a collision.

There are drawbacks to this, though. First, for methods, we have to rename the whole method globally, even if there’s only one collision. As soon as a type defines, say, a lower method, we have to call our internal one lower0 everywhere. It means we have to keep track of more state, making the parser more complicated. Also, each language has its own notion of what’s a collision and what’s not—in Swift, labels and return types are part of the name, so it’s perfectly OK to have lower(into writer: Writer), lower(b: Int), and lower() -> Foo; in other languages, it’s not. And it also doesn’t handle extensions and subclasses—what happens if you subclass or extend a generated type, and define a reserved name on it?

Approach three. Unconditionally append a random, short ID to internal symbols. Essentially, this means doing our own name mangling—things like Lowerable_AZBYCX1234. This makes the generated code uglier and harder to debug, and does it for all bindings, even if they don’t define colliding names. It also introduces churn, because regenerating the bindings would write a new ID (though we can make the scheme predictable—again, borrowing from how name mangling works in other languages). And it makes collisions virtually impossible.

Approach 3.1: prefix or suffix internal stuff (and reserved words!) with _. Also easy, and probably good enough!

I feel like approach one is the easiest and least magical option, but let’s discuss, and mention other approaches we think of!

┆Issue is synchronized with this Jira Task
┆Issue Number: UNIFFI-2

SYNC-1528 ⁃ Integrate rust crate for Fenix experimentation (nimbus)

┆Issue is synchronized with this Jira Epic

Add support for strings in arguments, return values, and record fields

The example code so far only really exercises support for numbers. In practice we have a lot of code that uses strings, so we need to be able to deal with those conveniently and safely.

Adding support for strings would be a good opportunity for someone new to this codebase to get involved. They would need to:

• Think up a new example for some small code that uses strings, and try writing the IDL and the corresponding rust code for such a component. (It won't work yet, but having a specific example to get working is a good place to start).
  • Try to cargo build your example component, and confirm that it doesn't compile. If it does compile despite not having support for strings, that's a separate bug...
• Decide on how Strings should be passed across the FFI layer, both as individual arguments and when part of a compount record. Implement the rust half of that via the traits in uniffi::support.
  • My suggestion (which isn't necessarily the best way!) would be to
    • Implement Lowerable by writing out a fixed-size length field, followed by the body of the string.
    • Implement Liftable by doing the reverse, reading the fixed-size length followed by the string data.
    • Implement ViaFfi using the ffi_support::FfiStr helper, which is what we use for strings in our hand-written rust component today.
• Add support for string types in [uniffi:scaffolding]. There's some placeholder stuff in there using &str and FfiStr<> but it'll probably need adjusting.
  • Try to get to a point where you can cargo build your example component and have it work successfully.
• Add support for strings in the kotlin and python bindings generators. The details here will depend on how strings are represented when crossing the FFI.
  • If you used FfiStr above, you might find that passing strings are arguments just works, because the native-code-binding layers of each language know how to pass a string object as a const char* in their FFI layers.
  • You'll need to add support for lifting and lowering strings as part of compound records; copy the approach taken for other primitive data types in these files.
  • Receiving strings as a return value and making sure they show up as native language string objects, is left as an exercise to the reader.

Rename 'master' -> 'main'

This is mostly a heads-up to @linacambridge @thomcc @jhugman that I'm going to do the s/master/main/g on this repo.

Prototype bindings for Firefox Desktop front-end (JS) consumers

We currently have bindings for Kotlin and Python, with Swift in progress. It would be wonderful to generate them for Desktop, too. There are a couple of uniquely Desktop (mostly in the jeez-this-is-clunky sense 😅) things to think about, so let's use this issue to plan out how to do this.

Our current approach is to consume the component's Rust API, and hand-roll a Rust XPCOM wrapper around it, so that JS can call it. This was a quick way to ship our first components. Unfortunately it has a few downsides:

• It makes reasoning about the system harder, since our other bindings go through the same FFI.
• We have to write the FFI ayer twice: once for mobile, and once for Desktop, with XPIDL playing the same role as the C FFI.
• As part of that, we have to reinvent serialization, exchanging objects between Rust and JS by serializing them to JSON, since vanilla XPIDL doesn't do well with complex structures. The FFI handles this for us with JSON and Protobufs.
• It's not particularly ergonomic to work with from either JS or Rust. On the Rust side, there's lots of clownshoes just to do something as common as "calling a method that returns something". On the JS side, we almost always need a .jsm wrapper because using the interface directly is so clunky.

XPIDL does have several escape hatches. Methods can return Promises, instead of having to implement their own callback interfaces. jsval and webidl types are much easier to use from JS (and C++!) An XPIDL interface that makes heavy use of these can be almost as ergonomic as a WebIDL one.

Unfortunately, Rust XPCOM can't do any of those.

• It can't call into a nostdcall, notxpcom (these use different calling conventions), or noscript method, even if it supports that method's types.
• It can't call a method with a jsval, webidl, or native type. The first two require Rust SpiderMonkey bindings, and the last is C++-only.
• It can't consume or return promises (SpiderMonkey again).
• It can't implement an interface at all that uses any of those features.

Although there are helper crates in the tree to make it easier to use Rust XPCOM, it's not as mature or well-supported as C++ or JS yet 😭

So, what else can we use to talk to Rust from JS? Chrome WebIDL, js-ctypes...and XPCOM interfaces implemented in C++. Let's look at that last one a bit closer. Since we're generating code, anyway, why not target C++ instead of Rust?

What if we made a backend that spits out an XPIDL interface and a C++ implementation? It could be async-by-default, using the C++ TaskQueue class, the same one we used for storage.sync and FxA in Rust. It can return promises, and handle threading for us automatically (ooh, threading is a whole other discussion...let's use another issue for that). It could lower jsvals to RustBuffers, pass them over the FFI on a background thread, lift the returned value into another jsval, and resolve a promise with it—going through the same flow for calling our Rust component as other FFIs.

What are the other alternatives?

• chrome-webidl. Used mostly for hot code. We're discouraged from using it heavily, since there is a binary size cost ("A WebIDL method binding can easily cost a kilobyte or more, depending on the number and types of the arguments."), and lots of security semantics that don't really apply for chrome code. Much better integration with JS; easy-ish to call from C++, depending on what you're doing.
• js-ctypes. Lotsa sharp edges, but can be used from JS directly, no need for a C++ wrapper. Can use symbols from libxul. Can also be used from a ChromeWorker (on a background thread), communicating with the main thread using message passing. Can't be used from existing Firefox C++ components—might be a concern when we Oxidize other components, but perfect for JS-only ones.

┆Issue is synchronized with this Jira Task

SYNC-1535 ⁃ Audit supported experiment context details

In this prototype we have the following fields supported in AppContext: 

https://github.com/a-s-dev/glean/pull/1/files#diff-c7271f5c37709867ac4f79e6d85240ceR27

pub app_id: Option<String>,
 pub app_version: Option<String>,
 pub locale_language: Option<String>,
 pub locale_country: Option<String>,
 pub device_manufacturer: Option<String>,
 pub device_model: Option<String>,
 pub region: Option<String>,
 pub debug_tag: Option<String>,

The fields should be audited, new fields added or removed.

┆Issue is synchronized with this Jira Task

Split `ViaFfi` trait into `IntoFfi` and `TryFromFfi`

This is based on the discussion in https://github.com/rfk/uniffi-rs/pull/22#discussion_r456967989.

This would entail the following:

• Take a look at the ViaFfi trait in https://github.com/rfk/uniffi-rs/blob/e35bbac8880b61de1ef9a73e736d4f6b7933244e/uniffi/src/support/mod.rs#L105-L122
• Split it into two traits,
  • an IntoFfi trait that requires the fn into_ffi_value(self) -> Self::Value; function
  • a TryFromFfi trait that requires the fn try_from_ffi_value(v: Self::Value) -> Result<Self>; function
• Make the changes neccesary to make it build and pass the tests, those include:
  • Modifying the macro that generates the traits for primative types: https://github.com/rfk/uniffi-rs/blob/main/uniffi/src/support/mod.rs#L124
  • Splitting each impl for ViaFfi into two separate impls, one for IntoFfi and one for TryFromFfi, you will notice that you only need to implement TryFromFfi for &str (And not IntoFfi) and only need to implement IntoFfi for String (And not TryFromFfi) - Note: The string stuff only applies if #25 merges before you do this
  • Look around for uses of ViaFfi (example: https://github.com/rfk/uniffi-rs/blob/main/uniffi/src/scaffolding.rs#L58) and replace it with the appropriate trait (for lifting you'll use TryFromFfi, and for lowering you'll use IntoFfi)
  • Do some magic until it builds and tests pass! ✨

I might have missed something, \cc @rfk I think this is a good-first-issue kind of issue, so feel free to add any details 😄

┆Issue is synchronized with this Jira Task

SYNC-1543 ⁃ Prepare AET launch blog post

Lead by Kirby and working with the Comms team, we're going to prepare a blog post to accompany the landing of AET in nightly, to help set expectations for anyone who sees a new telemetry ping and gets curious/suspicious/outraged. This is a placeholder issue to flag the fact that this work is taking place and consuming some of my bandwidth, we don't need to land anything in our repos in support of this.

┆Issue is synchronized with this Jira Task

Support passing objects as function/method arguments

It is not currently possible to pass objects as arguments to functions or methods, apart from the special-case handling of the implicit first argument in a method call.

In theory this should be easy right? You accept the u64 handle as an argument, look up the object in the appropriate handle-map, and you're off to the races!

There are significant complexities in practice. I don't think we need this functionality with any urgency but I wanted to put a bug on file for why this is harder than it might seem. Some thoughts below, please comment with others if you have them.

If the object argument needs to be mutable, then we can't just grab the object out of the handle-map, we have to use a closure that manages locking for exclusive access. If we have several object arguments, we would need to codegen a series of nested closures, which could get messy fast.

Mutable object arguments are also a deadlock hazard. Imagine a function call that takes two object arguments, like modify(obj1, obj2), and a concurrent call to modify(obj2, obj1). If we naively try to access each object argument in turn inside the generated FFI function, then the first call will lock obj1, and second call will lock obj2, and then they'll both deadlock waiting for the lock on their second argument. We might be able to mitigate that by locking object arguments in a consistent order, such as always trying to lock arguments with lower numeric handles first. But that'll make the codegen even messier.

┆Issue is synchronized with this Jira Task

What's our threading story for generated code?

Currently, our story is something like this: the Rust components expose synchronous APIs, and do blocking filesystem I/O and network consumers. Consumers wrap them in whatever async API makes sense for their needs—Kotlin Coroutines, Swift Dispatch Queues, Gecko Background Task Queues and so on.

There are several big advantages, the biggest one being keeping our components simple:

• Pushing this to consumers gives them flexibility, since they can decide how to break up work sent to a background thread.
• Integration with libraries that do blocking I/O. Rusqlite, for example (for very good reasons!) doesn't support holding a connection or transaction open across an await point.
• We don't have to worry about the kinds of async deadlocks we've seen in Desktop code, where we're waiting on an operation that's waiting on us.
• The mutex protecting the HandleMap means we can guarantee our Rust calls will be serialized. It's easy to understand what's going on conceptually.

On the flip side:

• We end up with wrappers like a-c's AsyncLoginsStorage, whose reason for existence is rewrapping the logins component's API in coroutines.
• How would this work in an async/await world? Would we ever be able to migrate our components to use it? Is that a thing we want?
• It means non-blocking code has to be rewritten in a blocking wrapper to call from Rust. See Viaduct for an example.
• Blocking I/O doesn't make sense for the Desktop bindings, since chrome JS is main thread-only. So we'll need at least one async API no matter what.
• Rust has excellent support for safe concurrent programming, and compile-time guarantees for preventing data races. This offloads that responsibility to embedding languages.

Add support for reference arguments in the rust code

Currently, every rust function exposed over the FFI is expected to take ownership of its arguments. This is most obvious in the existing "sprites" example, where the code for translate has to be declared as:

• fn translate(p: Point, v: Vector) -> Point

Which means that when the move_to method of the Sprite class calls it, it has to pass in a clone:

• self.current_position = translate(self.current_position.clone(), direction)

Making this more egonomic could be a good opportunity for somone new to the codebase to get familiar with it. The challenge would be to update the sprites example so that the translate function is declared as:

• fn translate(p: &Point, v: &Vector) -> Point

And the .clone() mentioned above can be removed.

In order to achieve this I think you would need to:

• Dig in to the code that generates the ffi wrappers that call these functions, to see where to pass the argument by reference instead of by value.
• Observe that you need some flag in the IDL to know whether to pass each argument by reference or by value, and that this will probably need to be added to the Argument struct so that it's available when you're generating the code.
• Observe that there is some placeholder code on the Argument struct that currently says "argument attributes are not yet supported", which corresponds to the extended attributes feature of WebIDL.
• Experiment with adding an extended attribute to the arguments of the translate function in sprites.idl, something like Point translate([ByRef] Point position, [ByRef] Vector direction).
  • I'll be totally honest, I don't know what the right syntax for this is in WebIDL, I'm just lead to believe it's possible by the fact that there's an attributes member on the arguments that we get back from the WebIDL parser!
  • See if you can add something that feels like a good developer experience in the IDL file, and which triggers the "argument attributes are not supported" error when you try to build the sprites example.
• Interpret the resulting argument into a flag we can set on our own Argument strut, then use that flag during code-gen to know whether to pass by reference.
• Make sure the sprites example still runs and outputs sensible result (in lieu of actual tests, which are forthcoming).

┆Issue is synchronized with this Jira Task

Boolean field in dictionary support

I want to be able to compile the following code:

dictionary Point {
  double x;
  double y;
  boolean visited;
};

Currently getting errors:

error[E0277]: the trait bound `bool: uniffi::support::Lowerable` is not satisfied
  --> /Users/eoger/uniffi-rs/target/debug/build/uniffi-example-geometry-32b57fa3d176b525/out/geometry.uniffi.rs:38:9
   |
38 |         uniffi::support::Lowerable::lower_into(&self.visited, buf);
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `uniffi::support::Lowerable` is not implemented for `bool`
   |
  ::: /Users/eoger/uniffi-rs/uniffi/src/support/mod.rs:17:22
   |
17 |     fn lower_into<B: BufMut>(&self, buf: &mut B);
   |                      ------ required by this bound in `uniffi::support::Lowerable::lower_into`

error[E0277]: the trait bound `u8: uniffi::support::Liftable` is not satisfied
  --> /Users/eoger/uniffi-rs/target/debug/build/uniffi-example-geometry-32b57fa3d176b525/out/geometry.uniffi.rs:47:22
   |
47 |             visited: <u8 as uniffi::support::Liftable>::try_lift_from(buf)?,
   |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `uniffi::support::Liftable` is not implemented for `u8`
   |
  ::: /Users/eoger/uniffi-rs/uniffi/src/support/mod.rs:68:25
   |
68 |     fn try_lift_from<B: Buf>(buf: &mut B) -> Result<Self>;
   |                         --- required by this bound in `uniffi::support::Liftable::try_lift_from`

error[E0308]: try expression alternatives have incompatible types
  --> /Users/eoger/uniffi-rs/target/debug/build/uniffi-example-geometry-32b57fa3d176b525/out/geometry.uniffi.rs:47:22
   |
47 |             visited: <u8 as uniffi::support::Liftable>::try_lift_from(buf)?,
   |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `bool`, found `u8`

┆Issue is synchronized with this Jira Task

Use panic-safety helper functions from ffi-support

Panics in the rust code are not currently handled very well, and I suspect would actually trigger undefined behaviour by propagating across the FFI boundary. We should use the support functions offered by ffi-support to ensure safe panic handling in all cases.

The most basic thing that could work:

• All ffi functions should have an ExternError out parameter
• The rust scaffolding for all standalone functions and object methods should wrap the call in call_with_output.

Object methods could instead rely on the call helpers of the HandleMap for the same effect, but I think it would be worth our while to catch and handle panics in argument conversion or other operations that might occur before we call into the HandleMap at all.

┆Issue is synchronized with this Jira Task

Static and/or multiple constructors support

In the fxa-client crate the FirefoxAccount can be instanced in two ways: either by its regular constructor or by supplying a JSON string representing a previous state. IIUC we don't have support for an alternative interface constructor right now..

┆Issue is synchronized with this Jira Task

SYNC-1526 ⁃ Nimbus: Autogenerate ffi bindings with uniffi

This is an auto generated bridge that allows the rust component to work between mobile and desktop. Using auto generated uniffi so that we can write fewer boiler plates (reference: https://github.com/rfk/uniffi-rs#what))

┆Issue is synchronized with this Jira Epic

Fix or disable python bindings

I am seeing bugs around Optional lift/lower that I'm not sure I know how to fix

┆Issue is synchronized with this Jira Task

Allow manual generation of scaffolding and bindings from IDL file

The way uniffi is currently set up, optimizes for a very specific use-case: you're building a crate and you're going to publish it as a rust component, so you want uniffi integrated at the top level of your build process (cargo build produces the .so, cargo run -- generate produces the foreign-language bindings, etc).

From some thoughtful feedback from @tarikeshaq over in https://github.com/rfk/uniffi-rs/issues/24#issuecomment-661608571, it's clear that the first set of potential real-life consumers won't be shaped like that. They're more likely to want to tentatively integrate uniffi into the build process of an existing project, and it's not obvious how to do that based on what we have right now.

A strawfox proposal, broadly inspired by how we use protobufs in app-services:

• Invest in making a uniffi-bindgen tool that can be used in a standalone fashion, for integration with other build setups. We actually already have the beginnings of this, but it's hidden away in implementation details.
• Have a command like uniffi-bindgen scaffolding path/to/api.idl that can read an IDL file and spit out the rust code that flattens it into a C-style FFI. Rather than writing a bunch of pub extern "C" functions by hand, consumers could use this tool to generate them automatically from the IDL file and then include!() them into their rust code (or perhaps import them as a sub-module?).
• Have a command like uniffi-bindgen generate --language=<language> path/to/api.idl that can read an IDL file and spit out the appropriate foreign-language bindings to consume the C-style FFI from above. Rather than writing Kotlin, Swift etc code by hand, consumers could use this tool to generate it automatically from the IDL file, and then incorporate the generated files into their package build.

That's going to feel a bit clunky as a dev experience, in much the same way that working with protobuf files feels a bit clunky for us in app-services (like the CI job whose only purpose is to check that you didn't forget to generate the bindings). But from recent conversations, I think it could set us up to deliver early value quicker, without having to solve a bunch of more general tooling challenges up-front.

Most of the codegen pieces for this already exist, but need to be exposed for easy use in this new way from the command-line.

Thoughts? /cc @eoger @jhugman @linacambridge @tarikeshaq

┆Issue is synchronized with this Jira Task

Figure out how to integrate with the megazord build system

.

┆Issue is synchronized with this Jira Task

SYNC-1538 ⁃ places_bookmarks_import_from_fennec: Unexpected error: MergeError(Error(Cycle(Guid...

Reference: https://sentry.prod.mozaws.net/operations/fenix-fennec/issues/9257398/events/cea5418189f54230b9ff26a7256ca841/

┆Issue is synchronized with this Jira Task

Nullable arguments support

This currently does not work:

namespace bobo {
  double do_something(MyType? arg);
}}

Error is

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
[1]    78220 abort      cargo run -- generate

Add support for returning and handling errors

We have lots of TODOs for adding error handling. A few things we’ll want to think about:

• How do we represent errors in WebIDL? As Result return values? Following how Firefox does it, using a Throws annotation on fallible methods? Do we want to have structured errors that can be inspected, or are error strings good enough?
• In the FFI, we’ll need to have fallible methods take an error out parameter.
• Swift and Kotlin specify fallible functions differently—Swift has the throws keyword to indicate a function is fallible, and Kotlin and Python functions can throw exceptions. We will need to teach our code generation layer to reflect error results as exceptions.

┆Issue is synchronized with this Jira Task

What should we do with this repo?

I pushed this code in a separate repo in order to be able to iterate without landing it in application-services, and it's never been my intention for it to live here for a significant amount of time. Have we done enough experimenting here to make a decision about where it should live longer term? Some options that we should consider:

1) Declare this a failed experiment, and archive/delete the repo

Not my preferred option, but I think we should be explicit that this remains on the table. If we've tried playing around with autogenerating FFI bindings and it seems like it's going to be too much complexity for too little reward, we should capture our learnings somewhere for posterity, and move on.

2) Move this into app-services, and try to use it for a new component

In its current form the code here is not ready for use in a "real" component. To meet that bar it would need, at a minimum: support for strings, basic error handling, and integration with the appservices megzording logic.

But if we think the remaining work is tractable enough and the potential for saving work is big enough, we could commit to trying this out on an upcoming component and move it over to the appservices repo for further iteration.

3) Move this into a separate mozilla-controlled repo, and try to use it for a new component

As above, but in a separate repo rather than as part of apps-services. We'd have to understand what we hope to gain from it being in a separate repo, to offset the extra workflow friction when trying to iterate on the tool and its consumers in two separate repos.

4) Decide we don't know yet, and keep poking away at it over here

"We don't know yet" is a fine position to have, but if that is indeed where we are, I'd like to figure out a plan for coming to know at some point in the future :-)

Enum field in dictionary support

I want to be able to compile this :

enum BoboType {
  "TRON",
  "ZORD",
};

dictionary TestMock {
  BoboType name_suffix_variant;
};

┆Issue is synchronized with this Jira Task

SYNC-1547 ⁃ Investigate how our minizord mashes up with Android Components

Investigate how our minizord mashes up with Android Components—given that the components are standalone, but we want the Glean and Nimbus wrappers to use the same minizord

┆Issue is synchronized with this Jira Task

Rename `Serializable` to `ViaFFIUsingByteBuffer` in the Swift bindings

..And Primitive to ViaFFI.

Even though they're not quite the same, the comment already says they're pretty analogous, and it would be nice to have that bit of symmetry—especially with #26 splitting up ViaFFI into IntoFFI and TryFromFFI.

┆Issue is synchronized with this Jira Task

Add support for sequence types in arguments, return values, and records

Sequences (or "lists" or "arrays" or "vectors" or whatever you like) are a pretty important data type that we don't currently have support for. Let's add them!

I think we can use the existing Optional type as a good template. Broadly, we'll need to:

• Add a sequence type to one of the examples. I think Tarik's "todolist" example from https://github.com/rfk/uniffi-rs/pull/25 would be a great candidate, since returning a list of todo items is an obvious extension of that functionality.
  • The WebIDL syntax for sequences is pretty straightfoward: sequence<T> for any type T.
  • There's already some skeleton code for parsing this type in uniffi/src/interface/types.rs but it might need some tweaking.
  • Try to cargo build the example and confirm that it doesn't work yet.
• Decide on a representation for sequences to cross the FFI layer.
  • Like the existing Option<T> type, it seems reasonable to me to always pass them as a bytebuffer.
  • Similar to the strings in https://github.com/rfk/uniffi-rs/pull/25, writing the number of items and then serializing each item in turn to the buffer seems like a reasonable representation in bytes.
• Implement the rust side of that representation in uniffi/src/support by implementing the Liftable and Lowerable traits for Vec<T>, and them implementing ViaFfi for Vec<T> by using them (similar to what's done for Option<T>.
• Add support for sequence types in uniffi/src/scaffolding, again following the example of option types.
• Add support for sequences in each of the Kotlin, Swift and Python bindings generators.
  • I think we mostly rely on protobufs for this in the current appservices code so you might have to just feel your way around here a bit, again using Option<T> as a guide.
• Run cargo test to confirm that your updated example code works correctly with sequence types!

┆Issue is synchronized with this Jira Task

SYNC-1527 ⁃ MetricsManager in Rust to send experiments telemetry via glean

┆Issue is synchronized with this Jira Epic

SYNC-1529 ⁃ Design plan for repo organization, CI and build products

We should decide on the basics of:

• Where will the source code live?
• How will its CI work
• How does it need to link to and relate to other libraries (especially Glean)

The discussion is happening in this Google doc

┆Issue is synchronized with this Jira Task

Add testing infrastructure for running examples as fixtures.

From #2

I'd like to be able to turn the existing example.py and example.kts files into actual tests with assertions that can be executed to make sure the generated bindings are working as expected. So far I've just been running them by hand and eyeballing the output, but that's clearly not going to scale.

Refactor templates to increase template re-use and make them easier to navigate

There are quite a few places where we're doing the same thing, but with minor differences in the header or footer.

e.g.

• unpacking arguments, calling method calls with and without a return value
• methods in enums, namespaces, interfaces, etc etc.
• iterating through properties to generate lift and lower methods.
• copyright & similar comments

We should investigate ways our templates could be split up and parameterized so working with templates easier for language experts and more readable.

1. Decorator stucts and rendering templates in place.
2. Using the path sub-attribute of the template attribute.
3. include directive. e.g. {% include "hotmess.txt" %}
4. Template inheritance.
5. Template importing.

This is likely a starting point / meta for smaller bugs rather than a specific piece all on its own.

Implement Scheduling for network requests

We already have a lot of parts included in the prototype here: https://github.com/a-s-dev/glean/pull/1/files/28bbd0e637a27777ed35b1d146e605c6b62194d9?short_path=35660eb#diff-35660eb478d9194651365436ecf6a68a

We need to implement to scheduling and refresh the experiments from the server when needed.

SYNC-1531 ⁃ Connect uniffi to experimenter component

Prototype a way to add uniffi to https://github.com/a-s-dev/glean/pull/1/files/28bbd0e637a27777ed35b1d146e605c6b62194d9?short_path=35660eb#diff-35660eb478d9194651365436ecf6a68a prototype and generate the ffi for Kotlin

┆Issue is synchronized with this Jira Task

SYNC-1532 ⁃ Implement bucketing logic

There are some bucketing specs at https://docs.google.com/document/d/1WAForAUIchVPaiZFCJO3hNQHY_7KZAjddfscTM_Lx0Y/edit#heading=h.ilipj4uaks21 

Some of it is already in the prototype, we need to add the missing bits, such as namespacing and add testing

┆Issue is synchronized with this Jira Task

SYNC-1537 ⁃ Support custom configuration in experimenter

Potentially right now Fenix testers would want to change the following parameters while testing nimbus:

[ConfigurationDetails]
Option,
Option,
Option

┆Issue is synchronized with this Jira Task

Audit Python helper size conversions

uniffi-rs/uniffi/src/bindings/python/templates/RustBufferHelper.py

Lines 43 to 44 in 3c36c7b

	def getInt(self):
	return self._unpack_from(4, ">i")

.

This looks like this corresponds to a put_u32 on the Rust size, looking at the Python docs it looks like we need to use I instead.

┆Issue is synchronized with this Jira Task

SYNC-1544 ⁃ Submit a final sync ping on logout

We have some logic in the sync ping to force submission of the ping when we notice an "idchange" (aka a user change or a change in sync storage node type), but it doesn't look like this is hooked up to submit when the user logs out. Thus we might have information about some number of completed syncs queued up in the sync ping, but never submit it because the user has since signed out.

Let's add some logic to the existing onAccountLogout handler for the sync ping, to flush any pending telemetry on logout.

---

🪲 Issue is synchronized with Bugzilla Bug 1655899

┆Issue is synchronized with this Jira Task

Handle collisions resulting from name mangling

Followup from discussion in #2

There are a number of examples where mangling of names to fit foreign naming conventions may cause collisions.

• Methods/functions.

namespace {
     fooBar();
     foo_bar();
};

• Properties in struct

dictionary MyDict {
   string fooBar;
   string foo_bar;
};

• Arguments in the same argument list.

interface MyObject {
     void myMethod(string fooBar, string foo_bar);
};

• Enum values

enum MyEnum { 
    "fooBar",
    "FOO_BAR",
};

• enum/dictionary/interfaces/types all in the same scope.

enum  FooBar {};

dictionary Foo_Bar {};

This is likely to be fixed in the same place as #10 . Bailing with a helpful error message will probably be the best we can hope for.

Thankfully, it appears we won't need to worry too hard about differing definitions of unique function signatures because rust only seems to take into account the function name.

There is a possibility of the user threading the needle here for two function with a name collision after normalization, but also a different arity.

e.g.

namespace {
    void foo();
    void Foo(uint32 i);
}

I think it would be better if we disallowed this rather than introducing more and more sophisticated methods.

┆Issue is synchronized with this Jira Task
┆Issue Number: UNIFFI-3

Report enrolled experiments to Glean

Experimenter crate needs a way to bubble up freshly enrolled experiments into Glean, this should be done using the platform layer until we find a way to use Glean from Rust

Don't hardcode Swift library names when running examples

The way we run Swift examples now is hecka janky.

If you're using a generated library in an Xcode project, you only need the bridging header (am I using the right terminology for this? It's an "umbrella header" for the module map, but I think "bridging" means something very specific, and suspect this isn't that) and the .swift source file for the component—it'll take care of the rest. But, we need to do some clever things to run the example manually.

First, we compile our component.swiftmodule and libcomponent.dylib with swiftc, and link that with the Rust FFI library (libuniffi_component.dylib). If we don't link with the FFI library, it'll complain it can't find any of the C FFI symbols when it goes to compile the Swift module. We also have to pass it a module map, which has the path to the "bridging header" declaring the FFI symbols. The libcomponent.dylib is the weird part—why do we need that? Apparently, component.swiftmodule module isn't enough; without libcomponent.dylib, running import component will work, but calling any of its functions will fail with an error that it can't find the Swift symbol.

Then, when we want to run an example script that imports the component, we call swift (not swiftc), and link the example script with the component.dylib we emitted in step 1. We don't have to link the script with the FFI library, but we do have to pass the same module map again.

SPM does this automagically under the hood. The way I figured out these incantations was doing swift build -v and swift run -v, looking at the commands, and whittling down the arguments until it worked 😆

For us, though, we'll need a better way that doesn't rely on run_swift_script hardcoding library names and module map paths. @rfk mentioned in https://github.com/rfk/uniffi-rs/pull/6#discussion_r446863815 that the component name gets passed down here, which we'll need as the base for the different library and module map paths.

SYNC-1542 ⁃ Firefox sync not syncing

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0

Actual results:

1595991934794 Sync.LogManager DEBUG Flushing file log
1595991934808 Sync.LogManager DEBUG Log cleanup threshold time: 1595127934808
1595991934826 Sync.LogManager DEBUG Done deleting files.
1595991939120 Sync.ErrorHandler DEBUG Beginning user-triggered sync.
1595991939122 Sync.Service DEBUG User-Agent: Firefox/59.0.2 (Windows NT 10.0; Win64; x64) FxSync/1.61.0.20180323154952.desktop
1595991939122 Sync.Service INFO Starting sync at 2020-07-29 09:05:39 in browser session 4IGBiemCVu6c
1595991939122 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score.
1595991939126 Sync.Status INFO Resetting Status.
1595991939126 Sync.Status DEBUG Status.service: error.sync.failed => success.status_ok
1595991939126 Sync.Status DEBUG Status.sync: success.sync => error.login.reason.network
1595991939126 Sync.Status DEBUG Status.service: success.status_ok => error.sync.failed
1595991939128 Sync.ErrorHandler ERROR Sync encountered an error: Error: Can't sync: Network is offline (resource://services-sync/stages/enginesync.js:50:13) JS Stack trace: [email protected]:50:13 < [email protected]:1106:13 < [email protected]:181:27 < [email protected]:137:22 < [email protected]:1099:12 < sync/<@service.js:1091:13 < [email protected]:107:22 < [email protected]:1080:12
1595991939128 Sync.SyncScheduler DEBUG Sync error count has exceeded 3; enforcing backoff.
1595991939128 Sync.SyncScheduler DEBUG Starting client-initiated backoff. Next sync in 7633152 ms.
1595991939128 Sync.SyncScheduler DEBUG Next sync in 7633152 ms. (why=client-backoff-schedule)
1595991939134 Sync.Service DEBUG Exception calling anonymous function: Error: Can't sync: Network is offline (resource://services-sync/stages/enginesync.js:50:13) JS Stack trace: [email protected]:50:13 < [email protected]:1106:13 < [email protected]:181:27 < [email protected]:137:22 < [email protected]:1099:12 < sync/<@service.js:1091:13 < [email protected]:107:22 < [email protected]:1080:12
1595991939136 Sync.ErrorHandler DEBUG Addons installed: 0

---

🪲 Issue is synchronized with Bugzilla Bug 1655894

┆Issue is synchronized with this Jira Task

SYNC-1546 ⁃ iOS is not mentioned in Connect Another Device pairing modal

Build
Prod [Train 1.181.2]

Affected platforms
Windows 10
iOS 13.6

Steps to reproduce

1. Login on a desktop using a Firefox Account
2. Navigate to about:preferences?action=pair#sync.
3. Observe the second paragraph displayed in “Connect Another Device” modal.

Expected results
Both Android and iOS are mentioned as being able to scan the pairing code.

Actual Results
Only Android is specified in the following paragraph: “2. Then sign in to Sync, or on Android scan the pairing code from inside the Sync settings.”

UNITO-UNDERSCORE!27!

┆Issue is synchronized with this Jira Task

Generate bindings docstrings from Rust method comments

From @mdroettboom on Slack, it sounds like a great idea .

┆Issue is synchronized with this Jira Task

Add destructors to Kotlin and Swift

The Python bindings already define __del__ destructors for classes, but Kotlin and Swift don’t yet. We’ll want to make sure we wire these up and call them correctly, to avoid leaks and double-frees.

┆Issue is synchronized with this Jira Task

SYNC-1533 ⁃ Support the `filter_expression` in Nimbus schema

Related to https://jira.mozilla.com/projects/EXP/issues/EXP-32

The schemas has a filter expression that needs to be supported in some ways. There should be some prior art for this. Need to check with the Glean / Nimbus teams before we proceed with one.

┆Issue is synchronized with this Jira Task

SYNC-1536 ⁃ First pass at supporting the Nimbus schema

One example of the known schema we have right now is at https://github.com/mozilla/nimbus-shared/blob/main/data/experiment-recipe-samples/pull-factor.json

There could be some changes to the schema in the future, but this is fairly close to what we need to support. We can do a first pass at supporting the schema above.

┆Issue is synchronized with this Jira Task

Boolean return support

I want to be able to compile the following

boolean return_true();

Currently I'm getting the error:

87 |     <u8 as uniffi::support::ViaFfi>::into_ffi_value(_retval)
   |                                                     ^^^^^^^ expected `u8`, found `bool`

SYNC-1541 ⁃ Verify clients act sanely on quota related error from server

We are in the process of implementing a new storage quota that will affect user collections.

On the client, we'd like to prevent users who have exceeded this quota from syncing. We should also log something that we can easily track for debugging purposes in the future.

See plan here related to this work.

---

🪲 Issue is synchronized with Bugzilla Bug 1655531

┆Issue is synchronized with this Jira Task

Add prototype for generating Swift bindings

Mainly as an exercise to get familiar with uniffi! 😊

Breakup scaffolding.rs templates in to separate files

We should break up the rust scaffolding templates in the same manner as #17.

SYNC-1545 ⁃ Make Nimbus a sub-crate inside the Glean repo

Create a simple crate structure inside of the Glean repo for experiments

┆Issue is synchronized with this Jira Task