circlesubi/circles-core

Common methods to interact with the Circles ecosystem

circles-core

Common methods to interact with the Circles ecosystem

API Docs | Handbook | Releases | Contributing

This library provides common methods for JavaScript clients and wallets to interact with the circles-contracts and off-chain services.

Features

• Interact with circles-contracts and off-chain services like safe-relay-service, graph and circles-api
• Register and maintain user accounts and organizations
• Create and search off-chain data like transfer descriptions, usernames and profile pictures
• Trust other users in the network and retreive trust network
• List owned Circles tokens and their current balance
• Show last activities like transfers, trusts and Safe ownership changes
• Calculate transitive transfer steps to send Circles
• Update the Version of the Safe contract to v1.3.0

Example

import CirclesCore from '@circles/core';
import Web3 from 'web3';

// Initialize web3
const web3 = new Web3();

// Initialize core with default configs when running against local `circles-docker` setup
const core = new CirclesCore(web3, {
  hubAddress: '0xCfEB869F69431e42cdB54A4F4f105C19C080A601',
  proxyFactoryAddress: '0x9b1f7F645351AF3631a656421eD2e40f2802E6c0',
  safeMasterAddress: '0x59d3631c86BbE35EF041872d502F218A39FBa150',
  apiServiceEndpoint: 'http://api.circles.local',
  pathfinderServiceEndpoint: 'http://pathfinder.circles.local',
  pathfinderType: 'server', // other option: 'cli'
  graphNodeEndpoint: 'http://graph.circles.local',
  databaseSource: 'graph',
  relayServiceEndpoint: 'http://relay.circles.local',
  subgraphName: 'circlesubi/circles-subgraph',
  fallbackHandlerAddress: '0x67B5656d60a809915323Bf2C40A8bEF15A152e3e',
});

// Create existing account from private key which owns a Safe
const account = web3.eth.accounts.privateKeyToAccount('0x...');

// Find out the address of the owned Safe
const [safeAddress] = await core.safe.getAddresses(account, {
  ownerAddress: account.address,
});

// Request Circles UBI payout
await core.token.requestUBIPayout(account, {
  safeAddress,
});

Installation

npm i @circles/core

Make sure you have all peer dependencies isomorphic-fetch and web3 installed as well. Check out the circles-docker repository for running your code locally against Circles services during development.

Development

circles-core is a JavaScript module, tested with Jest, transpiled with Babel and bundled with Rollup. Most of the tests are designed to test end-to-end against all external services and require a running circles-docker environment to work in your development setup.

# Install NodeJS dependencies
npm install

# Copy config file and edit variables according to your needs.
# When running against the default docker setup no changes are required here
cp .env.example .env

# Run e2e test suite. Make sure services are running in the background
# via `circles-docker` repository
npm run test

# Run tests automatically during development when changes have been made
npm run test:watch

# Check code formatting
npm run lint

# Build it!
npm run build

License

GNU Affero General Public License v3.0 AGPL-3.0

circles-core's Issues

Fix transitive transfer algorithm

The current graph being used to calculate the max flow is incorrect as it is not considering different tokens. There is already a fix underway but it needs to be integrated.

Tests fail sometimes after running multiple times

Describe the bug

When I run the tests (only the Token > Transitive Transactions > should send Circles to someone transitively test) several times with the docker deployment, the 3rd or 4th time the test fails with this error:

RequestError: Request failed @ http://relay.circles.local/api/v2/safes/0x514c97a4fa711D4f2B55354FEf4454F05C07D6b1/transactions/estimate/ with error 422

Therefore, I restart the docker containers and tests run well again:

make ENV=backend down
docker volume prune -f
make ENV=backend up 
make ENV=backend contracts
make ENV=backend subgraph
make ENV=backend up # sometimes circles-api-worker is down

Check if relayer funded Token deployment before requesting it

We have to wait for the relayer to fund Token deployment before we can actually do it. Right now we hardcoded the check in the client (https://github.com/CirclesUBI/circles-baby-phoenix/blob/development/src/store/onboarding/actions.js#L69) but it might make sense to move it to the core as we can automatically check the actual gas costs here.

Integration with land backend:

• use the intermediate layer
• Investigate how to query the land db (see telegram chat),
• Check how to host the copy of the db,
• Check how to continue updating DB (host land indexer),
• Document

create new method to interact with Pathfinder service

Modify the findTransitiveTransfer method to make request to the pathfinder service API instead of the circles-api.

• pass core-test locallly
• pass core-test in gitaction
• configure pathfinder endpoint for both local and prod env

fix test GH Action

Transitive transaction pathfinding hangs

This might be due to circular connections?

Migration contracts 1.1.1 to 1.3.0

• Add function that checks safe version and call the method update within gnosis safe contract to migrate contract from gnosis.sol
• Create test for migration
• Make test pass

Fetch last x activities and balance from Ethereum Node instead of Graph

Store the docker images in a build cache

If we use the docker images in Docker Hub, we could reduce further the build time by using the build cache in the GH Actions, not having to re-download all of the images every time we run the tests workflow.

Some references about best practices for using Docker Hub for CI/CD:

• https://www.docker.com/blog/best-practices-for-using-docker-hub-for-ci-cd/
• https://docs.docker.com/ci-cd/github-actions/

Allow Multisend calls

The delegate call cannot be estimated through the relayer (see https://github.com/safe-global/safe-eth-py/blob/master/gnosis/safe/safe.py#L800).

In local environment with docker it works, but not in production.
We need to enable the feature in the relayer.

As an example of how to do it, let's see the case of using Multisend to update safe version and change fallback handler.
Check tx data in the gnosis ap ui, both estimations are attempted: first from the Safe and then directly from the owner account.

Use BN.js for max-flow algorithm

The current max-flow implementation in the core uses plain Integers, which works for now but we might run into trouble when numbers get larger. We should change the algorithm to work with BN objects instead.

Move transitive transfer algorithm into own module

Wrong error message for transitive transactions when network error occurred

When the relayer throws an server error (500 e.g.) the core still returns an TransferError even though it should be an RequestError.

`typedData.js` is duplicated in `src` and `test`

The ideal would be to use the file in src in tests too

Fix Token tests

Token Tests started to fail only in GitHub Actions with this PR: #119
See the logs from https://github.com/CirclesUBI/circles-core/runs/6858140027?check_suite_focus=true:

> @circles/[email protected] test /home/runner/work/circles-core/circles-core/circles-core
> jest --runInBand
cat: edges-data/edges.json: No such file or directory
FAIL test/token.test.js (358.495 s)
  ● Token › Find transitive transfer steps › should return max flow and possible path
    TransferError: Request failed @ http://api.circles.local/api/transfers/ with error 503
      54 |     return response.data;
      55 |   } catch (error) {
    > 56 |     throw new TransferError(error.message, ErrorCodes.UNKNOWN_ERROR);
         |           ^
      57 |   }
      58 | }
      59 |
      at _callee13$ (src/token.js:56:11)
      at tryCatch (node_modules/regenerator-runtime/runtime.js:63:40)
      at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:294:22)
      at Generator.throw (node_modules/regenerator-runtime/runtime.js:119:21)
      at asyncGeneratorStep (node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
      at _throw (node_modules/@babel/runtime/helpers/asyncToGenerator.js:29:9)
          at runMicrotasks (<anonymous>)
  ● Token › Transitive Transactions › should send Circles to someone transitively
    TransferError: Request failed @ http://api.circles.local/api/transfers/ with error 503
      54 |     return response.data;
      55 |   } catch (error) {
    > 56 |     throw new TransferError(error.message, ErrorCodes.UNKNOWN_ERROR);
         |           ^
      57 |   }
      58 | }
      59 |
      at _callee13$ (src/token.js:56:11)
      at tryCatch (node_modules/regenerator-runtime/runtime.js:63:40)
      at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:294:22)
      at Generator.throw (node_modules/regenerator-runtime/runtime.js:119:21)
      at asyncGeneratorStep (node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
      at _throw (node_modules/@babel/runtime/helpers/asyncToGenerator.js:29:9)
          at runMicrotasks (<anonymous>)
  ● Token › Transitive Transactions › should fail sending Circles when data error
    Command failed: docker exec circles-api cat edges-data/edges.json
    cat: edges-data/edges.json: No such file or directory
      315 |       await Promise.resolve().then(() => {
      316 |         const edgesData = JSON.parse(
    > 317 |           execSync(
          |           ^
      318 |             `docker exec circles-api cat edges-data/edges.json`,
      319 |           ).toString(),
      320 |         );
      at test/token.test.js:317:11
          at runMicrotasks (<anonymous>)
PASS test/common.test.js
PASS test/safe.test.js (134.65 s)
PASS test/activity.test.js (89.094 s)
PASS test/trust.test.js (89.627 s)
PASS test/organization.test.js (155.172 s)
PASS test/utils.test.js (32.205 s)
PASS test/user.test.js
PASS test/lib.test.js
Test Suites: 1 failed, 8 passed, 9 total
Tests:       3 failed, 54 passed, 57 total
Snapshots:   0 total
Time:        865.09 s
Ran all test suites.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @circles/[email protected] test: `jest --runInBand`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the @circles/[email protected] test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2022-06-13T09_38_59_875Z-debug.log
Error: Process completed with exit code 1.

Fix documentation CI task

Match payment note with HUB_TRANSFER events

Editing profile utilities

That calls the new endpoints in the api

History is not intuitive

Currently when clicking the "back" button we move to a hard-coded parent root but this seems to be unintuitive for some users as they might have come from somewhere else.

Use dedicated database only for tests

1. Set up a test database environment in circles-docker
2. Use that environment when running core tests
3. Clear database after each test

Identify queries to the graph

• also check myxo, core, api

Use Safe-core-sdk to replace relayer

Fix CI: Check if Safe is funded before deploying Token

Right now our Travis CI tests fail as Token deployment is not funded yet. This can be fixed with this ticket: #16

Speed up tests by parallelizing deployments

We could parallelize the deployment of multiple Contracts (Tokens, Safes) to make our tests faster. Currently we wait for each deployment.

Decimal values get rounded when sending Circles

Allow same username with diffrent case variations if they are not taken

If my username was "miko" and I wanted to edit to "Miko", I should be able to do it as long as the username with other case variations is not taken by other account. Right now it says that the username is taken, but actually it's not used by other account, so it should be allowed (unless ther's other user with username "MIKO" or similar)

Currently
await core.utils.requestAPI({ path: ['users'], method: 'POST', data: { username, }, });
returns 409

Build intermediate layer to communicate with different databases

• Review codebase where TheGraph service is used (myxo, core, api)
• Implement abstraction layer in the core (only with the existing graph queries)
• Run core tests (we should check if it matters that the api is importing the old core lib)
• Build unit test
• Test in stg
• Build
• Document

Update tests to reflect async api worker changes

Revisit tests and make them run again / clean up API changes which occurred during launch phase.

Export getNetwork and findTransitiveTransaction methods of token module

Change max steps to a lower number

• add it to environment variables
• change the default value

Default suggestion: 30
Until we do more research

Related: CirclesUBI/circles-myxogastria#589

Frontend feature: intermediate layer for app notifications in activity log

API: CirclesUBI/circles-api#124 and CirclesUBI/circles-myxogastria#396

In the query you specify language

Provide util that returns a json object with notifications

[{
    message: <String>,
    date: <Date>,
    icon_id: <Icon-ID>,
}]

Possibly only for example 10 at a time and not the whole history

Run tests concurrently

We could improve test speed when removing the --runInBand flag of jest as it will make them run concurrently.

This can only be done when we fixed this issue in the relayer though: CirclesUBI/safe-relay-service#44 since running them at the same time leads to a "mass deployment" of Safe contracts which apparently breaks something in the relayer code 😭

Show mututal friends when returning trust network

Ignore undeployed Safes when building trust graph for transfers

Add method to manually predict Safe address

Add methods to deploy an organizational account

Change trust limits of trusts to 100%

Include users who trust users in response from isTrusted

In getTrustNetworkStatus extract trusting user address from id
In trust.js change trustConnections in isTrusted from being a count to returning information about the trust connections e.g.

[{address: 0x....., }, {address: 0x....}]

circles-core/src/trust.js

Line 59 in 7e0980b

trustConnections: 0,

circles-core/src/utils.js

Line 345 in 7e0980b

return requestGraph(graphNodeEndpoint, subgraphName, query);

This is needed for this feature in the app: Related CirclesUBI/circles-myxogastria#325

Show and filter activities only related to another user address

Move CI from Travis to GitHub action

Check if the onboarding organization process relies on the graph

Make use of Gnosis Safe `MultiSend` feature for batch transactions

This can greatly improve the loading times and UX experience as we can wrap multiple transactions into one. This is especially interesting for the organization onboarding flow where many transactions are required.

Enable update edges per user

Make available the endpoint to update edges per user, so the client can use it on its implementation

Add methods to store and read payment notes from API

See API changes here: CirclesUBI/circles-api#20

`databaseSource` in the Core constructor should be optional and have a default value

Myxogastria doesn't work with this new version v2.10.11 (error: Uncaught CoreError: "databaseSource" is missing in options). The constructor of the core requires the param databaseSource.
Solution: make the parameter optional and with a default value: graph.

Look over error handling and include errors thrown in documentation of core

https://circlesubi.github.io/circles-core/#coretokenfindtransitivetransfer

It would be helpful for projects using @circles/core package to know what error can be thrown by different functions such as
findTransitiveTransfer and updateTransitiveTransfer and transfer to better handle them. It should be clear when they occur and what they indicate.

Fix state issues with concurrently running tests

The token tests are failing with invalid nonce errors in the relayer service.

It seems that the same safe is used for multiple transactions at the same time which might cause the invalid transactions due to race conditions.

One solution could be to decouple the test state by creating a larger test network before all tests, with independent sub networks for each test.

Documentation: add response example

Improve documentation by giving examples of the response of the different methods (like in the circles-api repo)

[Challenge] How can we solve the issue with the relayer sending too many requests to the blockchain?

Research & Learn

Why questions - Which experience and behavior do we want to inspire?

Try to understand and define the pains and gains we are going to solve

Pains:

• Cancelled user onboardings are staying infinitely in the relayers scheduler which means we collect dozens of deployment jobs which never get finished. This causes many requests per minute which exceeds our Infura rate limit

Gains:

• We can refactor parts of our Relayer codebase
• We might not need to pay for Infura anymore / lower the costs
• The app does not crash because of rate limits

Brainstorm & Prototype

How questions - How do we enable such experience and incentivize such behavior?

Try to generate the full spectrum of ideas and possibilities - the more the better

• Can we solve this by not using the scheduler at all and actually ask about the current deployment / funding state manually when the user uses the application (via polling)?

Test & Feedback

What questions - Which ideas and prototypes are able to trigger the intended experience and behaviors in the best and worst ways?

Try to find all the weak and strong spots of the prototypes, the more details you find the better

Build & Implement

Who questions - Who is able to ship and deliver a full working proposal?

Try to figure out the most suited collaborators to get the job done efficiently, with the goal to get the proposal accepted by the community

Vote Proposal

Where questions - Where is this proposal leading us next?

Try to judge the proposal based on its ability to incrementally improve the original intended experiences and incentivized behaviors. Accept or reject the proposal and adjust the prioritization of your next weeks challenges and roadmap accordingly, consider to pivot radically on your current strategy