demergent-labs/azle

A WebAssembly runtime for TypeScript and JavaScript on ICP

azle's Issues

consider using the word variant instead of enum

Consider getting rid of Candid type

We might be able to determine all candid types from the function signatures of Query and Update functions @dansteren the courtesy man

factorial Motoko example

Recreate the counter Motoko example found here: https://github.com/dfinity/examples/tree/master/motoko/factorial

Use the calc Motoko example as a template: https://github.com/demergent-labs/azle/tree/main/examples/motoko_examples/calc

A big reason for recreating the Motoko examples is to find limitations with Azle. Please document any limitations/bugs/errors you find in Azle in this issue thread.

js-sha256 broken

The library js-sha256 is also broken, not sure why. Might want to check once the postfix operator issue is figured out to make sure that it works

hello Motoko example

Recreate the counter Motoko example found here: https://github.com/dfinity/examples/tree/master/motoko/hello

Use the calc Motoko example as a template: https://github.com/demergent-labs/azle/tree/main/examples/motoko_examples/calc

A big reason for recreating the Motoko examples is to find limitations with Azle. Please document any limitations/bugs/errors you find in Azle in this issue thread.

Code rules

Create a code quality sheet that we can all refer to

boa issues

sha224 is returning different results in boa and chrome and node. Go through function by function to figure it out. Open issues for this on boa

Automatic deploy process

Use the TypeScript compiler to handle imports

We will not be able to rely on the ES module system at runtime, we will need to have a bundle of JavaScript to deploy into the Wasm binary...the TypeScript compiler might be the best way to do this

Consider CLA or CAA

We might need an explicitly signed transfer of copyright ownership in order for the copyright of each contributor and the project as a whole to be legally given to the AZLE token holders. The copyright code of the USA is very explicit that a written and signed transfer is required, except for operation of law.

It seems the norm in open source is for individual contributors to simply retain the copyright to their contributions, but then license those contributions under the license of the project. What I think we need for the intellectual property tokens to work is for each contributor to transfer copyright ownership of their contributions to the token holders...this should be explicit and signed (electronic signing should be fine I would think).

This might be annoying to do right now, but something like the CLA bot for GitHub could work in the short term. Eventually, when things like PrivIC or other NFTs are created, we could probably have a digital signature created and stored in the decentralized git repo that I am hoping would act as the legal written signature and transfer of ownership to the token holders.

create a type for the ic and allow it to be extended by the user

The type has been created, but why would the user need to extend it?

Look into using BigInt for numbers

We really need TryFromJsValue for this to work: boa-dev/boa#1975

Mac requirements

Put this information in the documentation.

• Rust
• Cmake
• wasm32-unknown-unknown target

calc Motoko example

I need to implement Int or i128, the calc example has highlighted this

Turn strict mode back on

Disabling strict mode for now to get around some issues, especially this: using future reserved keyword 'arguments' not allowed in strict mode

The compilation of array spread uses arguments inside of a function and because TypeScript emits strict mode code boa isn't liking that for some reason.

Warn users not to use variables that are the same as any other variables

This issue in boa needs to be addressed: boa-dev/boa#1663

The issue has manifested itself in one of my projects like this: I have a local variable named balance and a function named balance. Setting the balance variable actually sets the value of the balance function, which is not good. There may be other implications, I am not sure.

Cross-canister calls blocking issue

This needs to be resolved: dfinity/cdk-rs#174

I need to be able to block on futures, but ic_cdk::block_on does not return a value.

Consider using u128 instead of candid::Nat

candid::Nat is giving us some issues...maybe I can just use u128 according to the documentation here: https://sdk.dfinity.org/docs/candid-guide/candid-types.html#type-nat

Rust to JS serialization security

We need to analyze the serialization and deserialization of parameters and other untrusted values to and from the JS environment. See here as a starting place: serde-rs/json#537

Consider forcing signed commits

AZLE token

The AZLE token has been created. Here's the command used:

dfx deploy --network ic --argument="(\"Azle Token\", \"AZLE\", 8, 100000000000000:nat, principal \"3m4bm-h6vl5-lxosf-oabzl-tdcxf-roedu-r2llk-75qg3-tiylq-snwek-uqe\")"

Caveats to document

• Avoid hyphens in file and path names, use underscores instead
• Avoid += and other shorthand assignment operators
• Use type aliases for all function signatures and return types
• do not use string or number keys for objects
• imports do not work
• no async/await
• Apparently closures do not fully work

Security

We should have extremely high security standards. I want to see multiple independent audits from our team and outside teams. We should thoroughly research the state of the art of ECMAScript security.

• SES
• https://github.com/endojs/endo
• Sanitize all user input
• Look into Agoric

Study up on canister security:

• https://internetcomputer.org/docs/current/references/security/general-security-best-practices
• https://www.youtube.com/watch?v=PneRzDmf_Xw
• https://www.youtube.com/watch?v=6wyIhzsFbKw
• https://www.youtube.com/watch?v=36L33S_DYHY
• https://www.joachim-breitner.de/blog/788-How_to_audit_an_Internet_Computer_canister

allow void return types

echo Motoko example

Recreate the counter Motoko example found here: https://github.com/dfinity/examples/tree/master/motoko/echo

Use the calc Motoko example as a template: https://github.com/demergent-labs/azle/tree/main/examples/motoko_examples/calc

A big reason for recreating the Motoko examples is to find limitations with Azle. Please document any limitations/bugs/errors you find in Azle in this issue thread.

addition assignment does not work in boa

+= does not seem to work, look into this

Context eval

We might be able to do multiple evals in a context...if so we could just eval all of the code once, and then eval each function call after that

Waiting on cross-canister calls

• Clean up the compiler code
  • Make IC type work correctly: #39
  • Clean up file structure: #40
  • Clean up individual files: #40
  • No TypeScript errors
• Add candid compiler (generator at least is complete for now, TypeScript -> Candid compiler will probably be coming soon)
  • Candid compiler has some issues with multiple commas
• Use Rust -> Candid or Candid -> Rust compiler...I don't remember which compiler I am going to use but use it: #32
• Add imports
• Add swc support to provide hopefully all ECMAScript features
• Open boa issue for += and such operators (this may be fixed now, try running broken examples again)
• Create pull request to boa for randomness
• Add date capabilities
• Start writing examples
• Implement all synchronous APIs
• Fix error reporting, we need nice succinct errors not the huge globby mess
• Use boa upstream, because a lot of issues have actually been solved
• Use official conversions from and to JsValue instead of the hacky stringification I am doing: boa-dev/boa#1851
• spack and imports
  • Apparently spack is breaking when trying to import from node_modules, it won't parse TypeScript files and I might run into issues where it won't bundle other imports if they are from node_modules
• I am going to try out esbuild, seems like spack might not be ready for prime-time
• Create better documentation
• Start creating video series
• Fix double-quote problem
• Int, i128, i64, Nat, u128, and u64 are not fully representable in JS. We will probably need to use some kind of BigInt or string representation to get this to work...I suggest we use an automatic BigInt representation for all special number types...this might be the best way to do it
• Add float32 and float64...
• Experiment with async code since I am compiling to ES3
• Do we allow for any kind of optional types on the function parameters or return types?
• I think cross-canister calls might be possible using ic_cdk::spawn and callbacks
• Add project-wide TypeScript check

Bring up with DFINITY

• Post build scripts
• Automatically generate Candid types from Rust Code
• Automatically generate Rust code from Candid types
• Cycle benchmarking

strict mode

I have removed all strict mode annotations from the build process.

• Look into the implications of this
• If we are going to keep this choice, it might be nice to use an official esbuild or swc option to remove the strict mode annotations, instead of the string.replace that I am using
• We can turn strict mode off globally in boa, would be cleaner that what we're doing now: https://docs.rs/Boa/0.13.0/boa/context/struct.Context.html#method.set_strict_mode_off

Better IC Type

Make the IC type much better for users

Absolutely correct candid interop

Some of the number conversions are probably wonky...look into them and ensure they are perfect

counter Motoko example

Recreate the counter Motoko example found here: https://github.com/dfinity/examples/tree/master/motoko/counter

Use the calc Motoko example as a template: https://github.com/demergent-labs/azle/tree/main/examples/motoko_examples/calc

A big reason for recreating the Motoko examples is to find limitations with Azle. Please document any limitations/bugs/errors you find in Azle in this issue thread.

Prettier

Create the most basic bare-bones prettier configuration for Azle.

Consider changing ICBlob to Vec<u8>

consider making this a Vec instead, just create a Vec and allow it to have a type parameter

Object.defineProperty exports object

The TypeScript compiler currently adds an Object.defineProperty(exports line in strict mode above all other code. boa is not allowing any code to be defined above "use strict", thus exports can't be created properly. Right now I'm using a simple str::replace in Rust to add let exports = {} in front of Object.defineProperty. It would be nice to find a more elegant way to do this.

Consider adding class support (defining canister as a class)

• Rewrite all examples with class API at the bottom
• Remove old JS generation stuff
• attempt to get rid of the CanisterResult type if possible
• Make sure all tests pass
• Rewrite parsing
• Remove old API
• Make sure all tests pass
• Add tsconfig.json to documentation now that we need the experimentalDecorators field
• Make sure stable storage examples use the correct stable storage memory (they will overwrite each other if you are not careful)
• Consider all of these alternative ways to create decorator-like things: https://stackoverflow.com/questions/35949554/invoking-a-function-without-parentheses
• Explore the $ syntax
• Explore combining new with a special type that will allow new Canister(Principal) without requiring the user to use a class to define the external canister, and also create the return type for them automatically...ideally they would define the external canister in the same way as they define their local canister methods

Consider creating support for classes. I would really like to avoid encouraging object oriented design, but canisters are stateful, they have methods, and the class syntax example shown below does map very well.

class IC {
    caller: Principal;
    // memory: {
    //     balances: {
    //         [key: AccountIdentifier]: Balance | undefined;
    //     };
    //     extensions: Extension[];
    //     initialized: boolean;
    //     supply: Balance;
    // };

    get memory(): {
        balances: {
            [key: AccountIdentifier]: Balance | undefined;
        };
        extensions: Extension[];
        initialized: boolean;
        supply: Balance;
    } {
        return ic.memory;
    }
}

export class JSONIC extends IC {
    constructor() {
        super();

        this.memory.initialized = true;
        this.memory.balances = {};
        this.memory.supply = 0;
        this.memory.extensions = ['@ext/common'];
    }

    @update
    balance(request: BalanceRequest): BalanceResponse {
        const aid = getUserAID(request.user);
    
        const balance = ic.memory.balances[aid];
    
        if (balance === undefined) {
            return {
                ok: 0
            };
        }
    
        return {
            ok: balance
        };
    }
}

Cycles benchmarking

We could do some live benchmarking, but it would be really nice to have local benchmarking capabilities and estimations of cycle costs, or even exact cycle costs returned as part of the result of an update or query call. This roadmap issue seems the most promising: https://forum.dfinity.org/t/make-local-development-experience-similar-to-the-mainnet/7735

Idea to improve performance:

• Never eval to get exports, try storing it in a global variable (like ic) and getting it out of that variable later on
• Create framework for running benchmarks against Rust, Motoko, and Azle examples
• Create framework for fine-grained benchmarks against Rust, Motoko, and Azle code
• Automatically generate markdown reports that are easy to consume
• Create models that take into account all cycle costs and convert the running costs into dollars
  • Update calls per second
  • Size in bytes per update call
  • Other variables that could change
  • Order of magnitude differences between the different CDKs
• Consider how to measure the Motoko and Azle GC
• Generate markdown
• Generate CSV
• Generate USD cost comparison
• Explain methodology (explain what each test does, how many runs, averages, etc)
• empty query
• empty update
• empty init
• empty preupgrade
• empty postupgrade
• Get the average percentage and multiplier for Azle against each language
• We need to add back in the costs of everything before the function starts, but we can only do this for Azle and Rust right now: https://forum.dfinity.org/t/introducing-performance-counter-on-the-internet-computer/14027/10
• Go through all outliers and make sure things check out
• Add all percentage/multiplicator factor for each language to each other language, I think people will want that
• Rust is optimizing too many things away, need to figure out how to not let it optimize away the primitive number stack inits
• Keep two branches, one that measures just the function bodies, and one that measures everything before the function body up until the return value
• How often should we run the benchmarks? Should they run on every deploy?? Perhaps they should run on every release?
• Maybe whenever we want to run extensive benchmarks, we can create a special branch called benchmarks-- with the version or something?

Init function

init with some initial parameters

• Make sure records and variants can be found from the init function...I think we might need to generate them in the candid???

First try at IC APIs

• query calls
  • Seems this will take some compile-time code and possibly will require the user to author their canister in TypeScript, otherwise we will not know the types for the query function they are exposing
• update calls
• orthogonal persistence
  • The eval/interpreter can run in a loop. This would keep all of the memory alive for as long as the loop lived. Actually, this might not work because then update/query calls could never finish
  • At the end of an eval call, we can gather all global variables and store them in the Rust environment, then on subsequent calls we provide those global variables back to the script...this seems most promising
• Complex data types as parameters
  • Stuck on candid::Nat from a JSON string
    • I might just have to implement a custom deserializer for canid::Nat: https://users.rust-lang.org/t/need-help-with-serde-deserialize-with/18374
• Candid function parameters
• Candid function return types
• Perhaps try out getting randomness as the first cool thing to do with asynchrony and cross-canister calls...yes, the management canister should prove out the concept pretty well
• init with some initial parameters
• pre upgrade
• post upgrade
• stable memory
• cross canister calls
• time
• randomness
• optional fields in rust structs
• allow non-type aliases
• stable memory access
• cross-canister calls
• basically find everything in cdk-rs and make sure it works in Azle
• Add option types

Analyze security

Most especially the Rust parameters being transformed into JS parameters as strings needs to be analyzed. All of that serialization needs to be heavily sanitized.

Refactor directory structure, file structure, functions, etc

hello-world Motoko example

Recreate the counter Motoko example found here: https://github.com/dfinity/examples/tree/master/motoko/hello-world

Use the calc Motoko example as a template: https://github.com/demergent-labs/azle/tree/main/examples/motoko_examples/calc

A big reason for recreating the Motoko examples is to find limitations with Azle. Please document any limitations/bugs/errors you find in Azle in this issue thread.

Generate candid from TypeScript

This would get us to feature-parity with Motoko, which would be pretty nice. Though, I heard talk the team was pushing for developers to write candid files manually as a best practice...we'll see

Examples

• Counter
• Hello world
• Query
• Update
• All data types
• Just make an example for each thing you might want to do
• Heartbeat
• Init
• pre_upgrade
• post_upgrade
• simple database
• graphql
• simple_user_accounts
• primitive_types
• complex_types

Testing

Get property tests going, I think most of the testing will just be around candid and the different type translations from JS -> Rust

Optimize JS

• Minify (this might be hard, I tried minifying with esbuild and it broke)
• only repeat once
• etc

Consider new compilation path

What if we compiled from TypeScript -> Candid -> Rust?

There are already tools that compile Candid <-> Rust. Right Now I am writing a compiler from TypeScript to Rust basically...but perhaps compiling to Candid would be better, since all of the Rust implementation details would be figured out for us.

And imagine in the future with Python and other languages, as long as we can compile those type systems into Candid, then Rust will automatically be done for us.

I think this is the best path forward. The Candid <-> Rust compiler infrastructure will be maintained by DFINITY thus we can rely on that infrastructure.

We will just need TypeScript -> Candid, Python -> Candid, etc.

And I'm glossing over some details, because there will always be some compilation from the source language to Rust, but I'm mostly talking about the custom developer-defined types/data structures.

Consider swc

As we start to run into limitations with boa, let's see how far we can get with https://github.com/swc-project/swc

spack issues

We need these types of issues to be resolved so that imports work perfectly no matter where they are located and whether or not they are written in TypeScript:

• swc-project/swc#3667
• swc-project/swc#2574