thxprotocol/monorepo

Monorepo containing THX Network applications and infrastructure for quest and reward campaigns.

THX - REST API

• NodeJS Express REST API
• Auth by THX OAuth2 OIDC server
• Job Scheduling by agenda.js
• Transaction Processing by OpenZeppelin Defender Relay
• Multisig Architecture by Safe Protocol Kit

Usage

Run in this order as apps depend on eachother and will fail if services are not fully started.

yarn
yarn contracts:serve
yarn docker:serve
yarn safe:install
yarn api:serve

Resources

Account
|-- Discord
|-- Twitter
|-- Youtube
|-- Wallet
Campaigns
|-- Analytics
|-- Collaborators
|-- Participants
|-- PointBalances
|-- Subscriptions
|-- Wallets
|-- Webhooks
Rewards
|-- Coin
|-- NFT
|-- Coupon
|-- Custom
|-- DiscordRole
Quests
|-- Daily
|-- Invite
|-- Social
|-- Custom
|-- Web3
ERC20
|-- Balances
|-- Tokens
|-- Transfers
ERC721
|-- Balances
|-- Tokens
|-- Transfers
ERC1155
|-- Balances
|-- Tokens
|-- Transfers
Webhook
|-- Daily
|-- Invite
|-- Custom
|-- Wallet

Troubleshooting

Transaction Processing

• Make sure agenda.js is started properly as this will process scheduled transactions.
• Make sure txs is running and troubleshoot specific txs issues below
• Make sure a local signer is available for hardhat or a network of choice

Safe Transaction Service

• Internal Server Error Originating from httpRequests.js in thx-web container are likely caused by a connectivity issue in your local Docker container setup. Restart docker machine and its containers to fix this.

• Bad Gateway Error Originating from txs-web this is likely caused by a missing SafeMasterCopy in the txs db. Insert with the command yarn safe:init

monorepo's Issues

Make account.email an optional field

Auth

• Make account.email field on account edit page editable
• Extend AccountService to send a confirmation email if the email address has changed
• Change the active flag into isEmailVerified and make sure regular signup still works

Dashboard

• Show new ModalRequestAccountEmailUpdate.vue with big button "Go to account page"

API

• When emails are sent, make sure the account has an email address and early return if not

Optional

• Login with Twitter and see what email is added for the account int he auth db
• Remove that email and try to login with Twitter SSO again
• If nothing breaks, create migration to remove all autogenerated Twitter and Spotify emails from the account collection

@spotify.thx.network
@twitter.thx.network

Implement Sign Typed Data V3 for signing relayed call data

Extend tests for OIDC and SSO routes

SSO is a critical part of our authentication layer, but is hard to test due to dependencies to external services. Let's see if we can mock those services and simulate a successful redirect so we can verifiy that the callbacks are functioning correctly.

As a user I want to sign in using my Metamask account

Add contract verification for tokens, facets, factory and registry

• Create API keys at polygonscan.com
• Read docs:
  https://docs.polygonscan.com/api-endpoints/contracts#verify-source-code
• Implement for the deployment of a pool (through poolFactory)
• Implement for the deployment of an erc20 (through tokenFactory)
• Implement for the deployment of an erc721 (through tokenFactory)
• Only verify contracts deployed on dev or prod

Paralellize relayed pool transactions

In the requireTransactions job we should create an arrays of transaction for every pool that has pending transactions and process them in parellel.

As a pool owner I want to create a new OAuth client

Dashboard

• In the pool -> info section remove the client_id and client_secret form-groups
• Also remove the creation of a client_credentials client from the POST /pools controller (users will have to do this intentionally now)
• Show a list of clients instead
• Per list group item show the client_id and client_secret (as ******, with a button next to it to unhide it)
• Show an "add client" button at the bottom of the list
• Open BaseModalClientCreate when clicking button
• Show input text field for clientName
• Show select field for authorization_code and client_credentials values for grantType property.
• If authorization_code is selected, show input text fields redirectUri, requestUri
• On submit request POST /clients with {grantType, redirectUri, requestUri, clientName } payload
• Request GET /clients to update the list of clients

Index and see if all transactions could be relayed

Reduce the attack vector and reduce balance API key requires.

Visually hide client secret with *** and show on click

Refactor ERC20Facet

Decompose the ERC20 facet in child facets

• RegistryFacet
• ERC20ProxyFacet
• ERC20SwapFacet
• ERC20DepositFacet

In exports/index.ts:

• DefaultPool: all facets above
• Add ERC20Facet and TopupFacet to nftPool variant

Create SDK for authorization_code grant clients

Package: @thxnetwork/sdk

Deps:

• oidc-client-ts
• @toruslabs/customauth
• typescript
• since its a browser SDK use window.fetch for doing HTTP requests

const client = new THXClient("WAGIJWGW", "WAWAGW", "https://example.com");
await client.signin();

Interface:

class THXClient {
  constructor(
    clientId: string,
    clientSecret: string,
    url: string
  ) {

  }
  signin(); // Do the oidc popup signin, getPrivateKey()
  signout(); // 
  account.get();
  erc721.tokens.get(); 
}

• Create new repository "sdk" @peterpolman Create repo before 26th of July
• Class exposed by package is THXClient
• Introduce THXClient.signin()
• On signin() use Torus util to fetch private key based on retrieved auth response
  https://github.com/thxprotocol/wallet/blob/develop/src/utils/torus.ts
• On signout() use the regular oidc-client-ts signoutRedirect() method
• On account.get() call the api GET /accounts endpoint
• On erc721.tokens.list() call the api GET /erc721/tokens
• On erc721.tokens.get(id: string) call the api GET /erc721/tokens/:id

Gas account per pool owner account

Create a gas admin account for relayed gas payments for every sub that deploys a pool. Limit to one per sub and aggregate the costs.

API Tasks

• Generate a private key for the sub that calls POST /asset_pools
• Encrypt the private key with server key and store in AssetPool.admin (string value)
• Transfer 2 MATIC from the main admin balance to pool admin account
• Create the getPoolAdmin(poolAddress, npid) helper function in network.ts
• Decrypt the gas_admin value with the secure key from the .env
• Use this decrypted private key to add the wallet and return the account object
• Extend the sendTransaction() function to use poolAdmin as from when it is available
• Every sendTransaction() function call should check the balance of the related poolAdmin
• If the balance is lower than .75 MATIC transfer 2 MATIC from the main admin account to the poolAdmin account

Implement transferToMany in the withdrawal queue

• Group transaction for 1 pool together and parellise executing in queue
• Aggregate reward amounts per beneficiary and do 1 tx for that
• Use transferToMany to transfer an array of rewards to an array of beneficiaries

Create brand store, move HTTP requests there and maintain state

Redirect user properly when claiming a reward

• Redirect user to auth when route starts with /claim and pass claim_id from url id available
• If hash is available as claim parameter also pass this to auth

Token reward URL's should be shorter

• Remove the current logic to generate a hash
• This should be the new reward claim URL:

https://dev-wallet.thx.network/claim?hash=eyJjaGFpbklkIjoxMzcsInBv....
https://dev-wallet.thx.network/claim/<claim_id>

• QR code should be generated for this URL

Improve email templates

• Use this layout template and only change colors https://github.com/leemunroe/responsive-html-email-template
• Apply layout for these emails:
• Signup confirmation template
• Password reset template
• Send login link template
• QR codes ready template (API)

POC: Implement IPFS for metadata and file storage

Use this ticket to create a proof of concept for storing NFT images and NFT metadata (json) on IPFS. I would expect IPFS storage to be a feature that should be enabled on a pool level. When it is not enabled we should just use S3 storage.

The proof of concept should generate a couple of learnings:

• What are the costs?
• Are we able to simulate/run it locally and during test runs?
• What is the performance in terms of "upload" speed?
• What third party libraries should we use to connect to IPFS?

Only request API if object is not available in local state

• Implement https://github.com/superwf/vuex-cache for caching purposes
• Do this for the vuex pools store in Dashboard first within this task

Implement contract verification for facets

Contracts

• Look into currently used deploy tooling: https://github.com/wighawag/hardhat-deploy
• See if we can configure polygonscan api for contract deployments

Prepare @thxnetwork/scss package

• Add publish config of the latests scss@master branch to the package as in modules-solidity (artifacts package)
• Publish only the scss files
• Add bootstrap dependency to the package too
• Compile and pack the package locally

Introduce claims resource

Model:

type Claim = {
    _id: string,
    poolId:string,
    erc20Id: string,
    erc721Id:string,
    rewardId: string,
}

• Change POST /rewards controller to insert a new Claim object (do this in RewardService.create)
• Introduce claim resource and router
• Introduce GET controller and respond with Claim data from db

chainId
poolAddress
tokenSymbol
rewardId
rewardAmount
rewardCondition
clientId

• Router should be guarded with rewardclaim scopes
• Extend the reward tests with checks for reward.claimId

Create SDK for erc721 data

Add contract verification for upgraded pools

Upgrade script

• Add property verifiedAt: Date & verifiedVersion: string to pool model
• Set MAX_DAILY_CONTRACT_VERIFICATIONS to 75 in .env and .env.example (and secrets.ts)
• Look into upgradeContractsToLatest script
• Run verification logic for upgraded pools until MAX_DAILY_CONTRACT_VERIFICATIONS is reached
• After verifying successfully set pool.verifiedAt to Date.now() and verifiedVersion to pool.version

Cleanup stale data in api prod database

Implement @types/thxnetwork-api for shared types (client/server)

• Create new repo types @peterpolman do this
• Evaluate type usage by both client and server a bit and see if we can optimise it
• Move the shared types into the repo (types, enums, definitions etc)
• Build a package and test if the contents are as expected

As a user I want to set a token image for ERC20 and ERC721 contracts

Dashboard

• In ERC20 Create modal add image upload field
• On submit call POST /erc20 and also attach image data to the payload
• Make sure it is shown correctly in BaseIdenticon.vue

API

• On POST /erc20 also upload logo image to public storage bucket
• Store the public URL with the erc20.logoImgUrl property
• Extend erc20 test with image data

Rename assetPool -> pool

• Rename all assetPool and AssetPool texts to pool equivalents
• Make sure that all tests still run nicely
• Rename collection too and create migration for the rename

As a user I want to download a zip file with multiple (unique) QR codes for claims on a specific reward

CardReward:

• Instead of showing the QR code for the reward object as we do now, show the QR code for the claim URL (https://wallet.thx.network/claim/<claimId>)
• When reward.claimIds > 1 there are multiple QR codes possible, in this case present a "Download QR codes" button in the UI and hide the URL input field.
• When clicking "Download" call GET /rewards/:id/claims/qrcode and see that a zip file is being downloaded.

ModalRewardCreate:

• Provide input[type="number"] to modal
• Add amount property to POST /rewards request (different than the current withdrawAmount or supply)
• Default value for amount should be 1

As a user I want to only create pools for tokens that are in the database

TIMEBOX: Research impact multichain

Let's see if we can get to a rough estimate on the work required to integrate with another EVM based chain.Best to limit to Infura supported chains so we can use their RPCs.

Support @ references in imports

Only use pools as ERC20 or ERC721 proxies and hide them in the UI

Dashboard

• Add "Deploy Pool" button to ERC20 and ERC721 card overview
• On click show modal with information about why we deploy a pool
• For ERC20 show disabled erc20 dropdown with erc20Id set
• For ERC721 show disabled erc721 dropdown with erc20Id set
• For ERC721 show erc20 dropdown with nothing set by default but user can choose
• On cancel, do nothing
• On ok, deploy pool with erc20Id or erc721Id
• Show loading state with message "This can take about 20s" to finish
• On success close the modal

API

• Adjust POST /pools controller to accept req.body.erc20Id or req.body.erc721Id and remove req.body.variant from the validation
• In the controller determine the pool variant to deploy based on the erc721Id or erc20Id
• When deploying the pool contract get the token address from the referenced erc20Id or erc721Id
• When saving the pool in the database also store the erc20Id or erc721Id for it
• Add migration to
• - iterate over all the pools and
• - based on pool.variant and pool.chainId call contract methods getERC20() or getERC721()
• - use the returned address to find the ERC20 or ERC721 object in the database
• - add the erc20._id or erc721._id in the pool.erc20Id and pool.erc721Id properties and update
• Extend GET /erc20 list controller to also return poolId in the response
• Extend GET /erc721 list controller to also return poolId in the response

Implement @thxnetwork/utils package

The package does not exist yet but it is probably good to introduce one. Quite some utils are shared amongst the API and Auth project. We should probably see if we can move those to a versioned NPM package and make them a dep of the projects.

• @peterpolman Create utils repo
• Verify duplicate utils in api, auth and client projects
• Move shared utils into utils repo
• Build package and verify contents to be as expected

Blur QR codes until hovered

As a user I want to archive ERC20, ERC721 and pool contracts in overview

API:

• Add archived boolean property to model
• Filter response of list endpoints to not show archived=true objects
• Add query string param to also list items for archived property on objects eg GET /erc721?archived=true
• Add migration to set archived to false for all objects (erc20, erc721, pool).

Dashboard:

• Add dropdowns to erc20 and erc721 cards (as for the pool card)
• Add "Archive" button to erc20, erc721 and pool cards to call PATCH operation with archived = true on the item
• Add toggle button on overview pages to do the e.g. GET /erc721?archived=true request and display all items
• When toggled again clear the state and add only the non archived items

Send signed data to patch account address

"How to verify address ownership with web3js"

• Research what kind of message needs to be signed by the user
• Before patching the account sign a message with the obtained private key for the user
• Remove address from the PATCH /account payload
• Add signedMessage to the PATCH /account payload

As a user I want to upload my own QR logo

convert image data to black and white

** Ignore this snippet for now** We will offer coloured images as a default to make life easier.

const img = document.getElementById("eeveelutions");
const canvas = document.getElementById("canvas");
const ctx = canvas.getContext("2d");

img.onload = function () {
  img.crossOrigin = "anonymous";
  ctx.drawImage(img, 0, 0);
  const imgData = ctx.getImageData(0, 0, canvas.width, canvas.height);
  for (i = 0; i < imgData.data.length; i += 4) {
    let count = imgData.data[i] + imgData.data[i + 1] + imgData.data[i + 2];
    let colour = 0;
    if (count > 383) colour = 255;

    imgData.data[i] = colour;
    imgData.data[i + 1] = colour;
    imgData.data[i + 2] = colour;
    imgData.data[i + 3] = 255;
  }
  ctx.putImageData(imgData, 0, 0);
};

Dashboard:

• Fetch brand.logoSrc from API in Reward component
• If there is a logoSrc image available, display it on the QRcode
• If not, display the default (THX) logo image.

API:

• Extend the new job for QR code generation to include brand.logoSrc if available (and default if not)
• See comment over here: https://github.com/thxprotocol/api/pull/360/files#r924374171

Migration to create a Claim for every existing Reward in the Database

• Generate the hash like it is done in the dashboard and save it in the Reward table
• Generate the claim and save the claimId in the Reward table

Create Job Processor for contract verification

• Add job/verificationProcessor.ts and steal some boilerplate code from transactionProcessor
• Configure the job to run on a daily basis in agenda.ts
• Find all pools where pool.verifiedVersion !== pool.version
• Start verifying pool contracts until MAX_DAILY_CONTRACT_VERIFICATIONS is reached
• After verifying successfully set pool.verifiedAt to Date.now() and verifiedVersion to pool.version

As a user I want to be able to create payment request for NFT metadata

Dashboard (2)

• Extend CreatePaymentRequest modal with SelectNFTMetadata dropdown (see CreateReward modal and re-use component)
• Extend POST /payments payload with optional payment.metadata property containing the metadataId

API (6)

• Validate optional metadataId in POST /payments payload
• Query for ERC721Metadata based on metadataId and push payment.accessToken onto ERC721Metadata.whitelistedAccessTokens.
• Extend PaymentService.pay() callback with query for ERC721Metadata based on payment.accessToken
• If there is a result, mint a token for that metadata for payment.sender.
• Create tests

Wallet: (1)

• If available, show title and description in initial screen of payment UI
• If available, show attributes in success screen of payment UI

DASHBOARD - As a user I want to import an external ERC20 or ERC721 contract

• In dashboard on ERC20 overview page add button "Import Token".
• Open a model BaseModalErc20Import where the user can pick from the quickswap list of existing tokens or provide its own ERC20 contract address.
• POST tokenAddress and chainId to POST /erc20/import
• Close the model

Add endpoint to download zip file with QR code images

• Extend Reward modal with amount property
• Create migration to set amount for all the already existing rewards to 1
• Extend GET /reward endpoint response with a list of claimId's (claims: string[]) when reward.amount > 1
• Add GET /rewards/:id/claims/qrcode endpoint
• In the controller query for all claims for rewardId req.params.id
• Create QR code images for all claims (and write to disk? or do this in memory)
• Zip all the individual images and remove temporary storage
• Test with 5000 - 10000 claims and see how it performs
• Probably create a job if the requests takes long and send an email with download link when done.

POC: ERC721 Royalties

• Create and deploy test NFT collection which adheres to the RoyaltyRegistry standards
• Test if the royalty distribution works as expected when trading NFTs on OpenSea and Rarible (or another of your choice)

Create SDK for erc20 data

Extend auth with claims:write and claims:read scopes and process claim_id or reward_hash correctly

Auth

• Introduce claims:write, claims:read scopes in oidc config
• Extend THX Auth client with claims:write, claims:read scopes
• Extend requested scopes in api.ts util with claims:read
• Change oidc/:uid/claim controller to process reward_hash if available
• Change oidc/:uid/claim controller to process claim_id if available and use apiClient util
• Introduce ClaimProxy in project (like BrandProxy)

Manage old reward claim url

Change PATCH /account to receive signedMessage

• Extend PATCH /account validator with check for signedMessage
• Extend PATCH /account controller to run recoverAddress() for signedMessage
• If the address result is different than the current address for the account, update it.

Fix pagination issues in wallet withdrawals view

Problem
Make sure to create 10+ withdrawals (or set the pager to a different page limit) so the pagination appears. When navigating to the next page and withdrawing withdrawals there they disappear from the list, which currently seems to "break" the pagination in terms of navigating to incorrect pages with incorrect page limits.

Queue Airtable signups to work around Airtable API rate limits

Current rate limit is 5 per second. Claims links with SSO could potentially hit this limit. Better to queue those calls.