consensys/tessera

Tessera - Enterprise Implementation of Quorum's transaction manager

tessera's Issues

Provide ability to specify whitelist of node identities permitted to join network (smart contract based permissioning)

Release workflow steps

the current process used to create 0.5.2

1. Create and switch to writable branch from master "release from-master"

mvn clean release:prepare -B

Note: The clean isn't required but was there when I did it.

mvn clean release:perform -B

After the tag was created on git hub there's a last step to create release from tag.

Issues/manual steps

1. Creating the branch from master as can't write to master without a PR
2. needed to run test suite multiple times, that might add to the 10 minute build time for the release:perform
3. Release publish step (could be fine for now)

Private key password Encryption - Add ability to change password

Strategically we will have key management tools integrated onto Tessera

Support for FUSE file system directory support

There's a legacy file system data format. Our present code tries to leverage jdbc as much as possible.

Assuming that is appetite for directory system datastore support the choice is find a jdbc driver that allows us to do this which will give us the benefit of less code changes or add specialised support by way of a FileSystemDAO or something. This will mean more code and having a less homogenous code base.

Socket - unixsocket package add '-Api' to name to be consistent across

Tessera help looks funky.

Looks like this on my mac:

 ✘ fixanoid@Felixs-MacBook-Pro  ~/Projects/tessera   master  tessera help  
usage: tessera -configfile <PATH> [-keygen <PATH>] [-pidfile <PATH>]
    --alwaysSendTo.keyBytes <BYTE[]...>                         Override
                                                                option for
                                                                alwaysSend
                                                                To.keyByte
                                                                s , type:
                                                                byte[]
 -configfile <PATH>                                             Path to
                                                                node
                                                                configurat
                                                                ion file
    --jdbc.password <STRING>                                    Override
                                                                option for
                                                                jdbc.passw
                                                                ord ,
                                                                type:
                                                                String

All of the param explanations are like this.

Add CLI override for config.json

Rename the java packages from com.github.tessera to com.quorum.tessara

Organisation change

KeyGeneratorImpl is attempting to interact with an end user

I don't know why, but when i'm executing unit tests in the cli module i end up seeing the following

System.out.println("Enter a relative or absolute path (without extension) to save the keys to");
System.out.println("or leave blank to not save to separate file:");
final String path = new Scanner(filenameStream).nextLine();

if the fileNameStream doesn't have a next line it just sits. It looks like the object should throw an exception in this case or something a CLI can use to inform the user that something is afoot.

Provide ability to have an internal cluster of Tessera nodes for heightened recoverability

To have a cluster of nodes against each quorum node to resume transaction processing even if one node in the cluster is down. All nodes in cluster will share all private/public key pairs associated with that quorum node. The connection is to listen over HTTP or gRPC with robust authentication with a load balancing layer managing the cluster

File overwrite tests don't run as a root/admin user

Tests that create files and remove the writable attribute so an exception is thrown for the test don't work as the root user (on Linux) because the root user always has write access, regardless of permissions.

Tests that use File#setWritable(false) are the ones that are failing. An alternative way to ensure the exception gets thrown should be used that works with root/admin users.

Document or Tool generate new config from TOML

Go through and identify all items in Constellation config files that is not in Tessera config and add support to all through migration script/toolset.

not in tessera config that is in constellation

Private key encryption

I have a conceptual question regarding Tessera/Constellation and private key encryption/decryption.
https://github.com/jpmorganchase/tessera/wiki/Configuration#using-existing-keys

I would like to encrypt a private key using your tools. It looks like that the private key derived from the password is directly used as the Tessera private key. Am I correct with this observation?

If so, this means that there is a strong dependency between the password, the private key and the public key. If I want to change the password for what ever reason (e.g. compromised password), this means that I can't "keep" my public key, since password => private key => public key.

Would it be feasible to add another "encryption mode" where the derived key isn't the actual private key, rather an intermediary private key to encrypt/decrypt the actual private key (like geth does it)? Are there any alternatives you would recommend instead (using a 3rd party solution) for encrypting the key with a password that can be rotated?

Improve tests for cmdline overrides

While code coverage is good, some of the more complex override test cases aren't in the suite.

Test CLI Scenerios - Private key generation

Methodology: Create tessera alias and then cd to tressera/config then run using sample files.

1. Misleading message when public key path is invalid.

tessera -configfile src/test/resources/keytests/pubPrivUsingPathsUnlocked_missingPublicKey.json --unixSocketFile /tmp/foo.ipc
Config validation issue: keys.keyData KeyData 1 : Public key path BOGUS.pub does not exist

FIXED.

2. Misleading message when private key path is invalid

tessera -configfile src/test/resources/keytests/pubPrivUsingPathsUnlocked_missingPrivateKey.json --unixSocketFile /tmp/foo.ipc
Config validation issue: keys.keyData KeyData 1 : Private key path BOGUS.key does not exist

FIXED.

3. pubPrivUsingPathsLocked case locks and provides on output
   tessera -configfile src/test/resources/keytests/pubPrivUsingPathsLocked.json --unixSocketFile /tmp/foo.ipc
   Expectation is to for cli to return some feedback, but this happy case should result in success

4. passwordsMissing.json

tessera -configfile src/test/resources/keytests/passwordsMissing.json --unixSocketFile /tmp/foo.ipc
Config validation issue: keys.keyData[0].config A locked key was provided without a password.
 Please ensure the same number of passwords are provided as there are keys and remember to include empty passwords for unlocked keys

FIXED

5. Wrong passwords

tessera -configfile src/test/resources/keytests/passwordsWrong.json --unixSocketFile /tmp/foo.ipc
Could not decrypt the private key with the provided password, please double check the passwords provided
Config validation issue: keys.keyData[0].privateKey Could not decrypt the private key with the provided password, please double check the passwords provided

FIXED

6. Invalid Unix domain socket file path isn't validated and ends up with big stacktrace on server startup

tessera -configfile src/test/resources/keytests/pubPrivUsingPathsUnlocked.json
Config validation issue: unixSocketFile Unable to create file /Users/peter/IdeaProjects/quorum-ex/examples/7nodes/qdata/c1/tm.ipc

FIXED

Provide ability to enable or disable peer-to-peer node discovery

For mysql , what is the table about tessera?

I want to use mysql to store the information about tessera. I want to kown which version of mysql I need to use and What the table structure I need to define ? Thank you.

Launching without parameters should default to help

After alias tessera=java\ -jar\ ~/Projects/tessera/tessera-app/target/tessera-app-1.0-SNAPSHOT-app.jar and launching tessera prints errors. Should default to help.

Separate api model objects from internal domain model

At the moment tessera-core reuses the api objects as its domain model rather than defining its own domain model.

Task 1. Create domain model in core and redefine transaction manager to use them
Task 2. Move com.quorum.tessera.api.model in tessera core into jax rs module. Provide convertors to copy between domain and api objects.

Implement a common set of interfaces to ensure that the read methods are defined on api and domain.

Provide ability for configuration to be updated at run time without need for restart

This should include node discovery ON/OFF and addition of peer nodes.

Dont null values/properties during marshalling.

Rather than masking configuration output to avoid printing privateKey values and/or passwords we have jaxb adapters that null values while they are being marshalled. This has broken override cli args and makes the entire config loading unpredictable and unmaintainable.

1. Add a masking marshaller to deal with these case
2. Check that the new function is always used when logging or/and printing to console

Integrate gRPC in Tessera for peer node communication

base package names

all classes in tessera core should have the base package com.quorum.tessera.core

this rule of having a module reference in the package name should be applied to all of the modules.

com.quorum.tessera.{module ref}

having a distinct namespace will mean tessera can be more easily packaged for java 9+ module-info descriptors, as modules cannot have package name clashes.

Provide ability to specify at startup list of static peer nodes to establish connection to

Database DDL to be added to Tessera repo

DDL Added:

tessera/tessera-app/src/main/resources/tesseraDDL.sql

CREATE TABLE ENCRYPTED_TRANSACTION (ID BIGINT NOT NULL, ENCODED_PAYLOAD LONGVARBINARY NOT NULL, HASH LONGVARBINARY NOT NULL UNIQUE, PRIMARY KEY (ID))
CREATE SEQUENCE ENC_TX_SEQ INCREMENT BY 50 START WITH 50

Salvage structured configuration

We lost a lot of benefit of having a structured xml means for providing configuration before we broke it swapping to a purely json and worse supporting toml file .. now we've created several strands of tech debt reactively just implementing things without sufficient challenging. And therefore risk to the success of the overall program.

The TOML config is particularly problematic as it attempt to align the private key paths, public key paths and the contents of the passwords file to create a key config. Worse still the private key contents are intact key generation options structured as son fragments.

How to add mysql to 7nodes example?

I have installed mysql on local machine
I have downloaded the jar(version 8.0.12) about mysql jdbc and saved at /home/vagrant/tessera/mysql.jar
I have run the ddl(https://github.com/jpmorganchase/tessera/blob/master/ddls/mysql-ddl.sql) of mysql

I modify the tessera-init.sh as follwoing and change the jdbc url to mysql

#!/usr/bin/env bash
echo "[*] Initialising Tessera configuration"

currentDir=$(pwd)
for i in {1..7}
do
DDIR="qdata/c$i"
mkdir -p ${DDIR}
mkdir -p qdata/logs
cp "keys/tm$i.pub" "${DDIR}/tm.pub"
cp "keys/tm$i.key" "${DDIR}/tm.key"
rm -f "${DDIR}/tm.ipc"
#change tls to "strict" to enable it (don't forget to also change http -> https)
cat < ${DDIR}/tessera-config${i}.json
{
"useWhiteList": false,
"jdbc": {
"username": "root",
"password": "123456",
"url": "jdbc:mysql//127.0.0.1:3306/quorum?useSSL=false"
},
"server": {
"port": 900${i},
"hostName": "http://localhost",
"sslConfig": {
"tls": "OFF",
"generateKeyStoreIfNotExisted": true,
"serverKeyStore": "${currentDir}/qdata/c${i}/server${i}-keystore",
"serverKeyStorePassword": "quorum",
"serverTrustStore": "${currentDir}/qdata/c${i}/server-truststore",
"serverTrustStorePassword": "quorum",
"serverTrustMode": "TOFU",
"knownClientsFile": "${currentDir}/qdata/c${i}/knownClients",
"clientKeyStore": "${currentDir}/qdata/c${i}/client${i}-keystore",
"clientKeyStorePassword": "quorum",
"clientTrustStore": "${currentDir}/qdata/c${i}/client-truststore",
"clientTrustStorePassword": "quorum",
"clientTrustMode": "TOFU",
"knownServersFile": "${currentDir}/qdata/c${i}/knownServers"
}
},
"peer": [
{
"url": "http://localhost:9001"
},
{
"url": "http://localhost:9002"
},
{
"url": "http://localhost:9003"
},
{
"url": "http://localhost:9004"
},
{
"url": "http://localhost:9005"
},
{
"url": "http://localhost:9006"
},
{
"url": "http://localhost:9007"
}
],
"keys": {
"passwords": [],
"keyData": [
{
"config": $(cat ${currentDir}/qdata/c${i}/tm.key),
"publicKey": "$(cat ${currentDir}/qdata/c${i}/tm.pub)"
}
]
},
"alwaysSendTo": [],
"unixSocketFile": "${currentDir}/qdata/c${i}/tm.ipc"
}
EOF
done

Provide ability to certify the identity of participants

GRPC acceptance tests should be expanded

The GRPC acceptance tests do some basic checks to ensure a GRPC service is running, but a full set of tests for each endpoint should be made to ensure the calls act as expected, similar to the REST acceptance tests.

Quorum examples needs versions syncing across with tessera releases

The tessera examples need maintaining and executing as part of the main project. Or as a minimum have the ability to read the current (or user defined) tessera version and run from that.

PID file arg

At the moment a pid file system property is used to define an optional pid file. Usage usage should be

nexus -pidfile /somepath/my.pid ..

rather than java -Dnexus.pid.file=/somepath/my.pid

The makes the nexus alias more portable, using the system property means the pid will be used for all invocations.

Test with different databases and provide ddls, sample config and how to setup

Tessera should be readily configurable to any DB supporting jdbc driver. Test with various DBs and provide config info

Fail during mvn install

Hello. I was trying install and compile tessera.

1. $ java -version
   java version "1.8.0_181"
   Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
   Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

2. $ mvn --version
   Apache Maven 3.3.9
   Maven home: /usr/share/maven
   Java version: 1.8.0_181, vendor: Oracle Corporation
   Java home: /usr/lib/jvm/java-8-oracle/jre
   Default locale: en_US, platform encoding: UTF-8
   OS name: "linux", version: "4.15.0-1022-azure", arch: "amd64", family: "unix"

3. Apache Tomcat/9.0.11

I cloned this repo, switch folder and tried run mvn install but got error:

I used basic config.json

What did I do wrong?

Add distinct configuration for p2p and quorum facing services

The current configuration has no clear means of separating node to node communication (party info) and the endpoints used for privacy management that quorum uses.

Provide ability to define read/write permissions at node and organisation/user level

Advertised URL is composed of server url and port

The advertised URL may be different the the port the web server is bound on, and the configuration should be separated.

KeyGeneratorFactory too tightly coupled.

com.quorum.tessera.config.keys.KeyGeneratorFactory needs use the service loader or other mechanism so the Cli and other modules that use it aren't forced to use the implementation just to run some unit tests that only need to prove that there is an interaction with the api.

Provide cli with means to save generated config

When private and public keys are generated. using -keygen an additional -output or needs defining.

The default -output should be to sys.out if not defined.. if a path is define the config should be marshalled to that and exit wit running.

Provide ability to specify blacklist of node identities not permitted to join network (via smart contract based permissioning) or interface

Typo mvn install -P kalium

In the README :
Change mvn install -Pkalium to mvn install -P kalium.

• Bonus: It would be nice to have a 'or' between jnacl and kalium for better understanding.
  Thanks for your work !

Ported Constellation bug into tessera where -filename has no value

When a user elects to generate a key from the tessera cli s/he apparently can provide an empty file name. Normal usage is :

tessera -keygen -filename somefile,someotherfile

apparently its desirable to also support

tessera -keygen -filename

In this case we've now gone to the effort of defining the filename option and then not provided any values for it, the reasonable exception would be that the user is then informed that they haven't defined the file name(s) s/he wishes to use to generate the keys.

Once we clarify that having no args for -filename is incorrect we need to remove support for it.

Support reading tls private/public key in Apache format (.pem)

Allow for tessera to be imported as java library allowing for custom storage and encryption if required.

Release to maven central

Most of the configuration has been added. We need to raise a JIRA with sonotype to get the nexus access.

https://central.sonatype.org/pages/ossrh-guide.html

Alwayssendto[] config support to be added to Tesserra

This is mainly used for backups / regulatory node etc...

Key generation from KeyDataConfig

com.github.nexus.config.adapters.KeyDataAdapter needs to be interpreted with the create pairs code in KeyManager located in the nexus-app module. The KeyManger code that does this needs to be in the config module and all of this needs to happen in the config module.

    @Override
    public KeyData unmarshal(KeyData keyData) throws Exception {

        if (keyData.hasKeys()) {
            return keyData;
        }

        //TODO: Generate key pairs from and create populated keyData
        KeyDataConfig keyDataConfig = KeyDataConfigStore.INSTANCE.pop();

        return new KeyData(keyDataConfig, null, null);
    }

The private and public keys need generating from the KeyDataConfig object and storing on the KeyData

Quorum - Tessera performance

Benchmarking & profiling. See how to improve

Create enclave deployment artefact

The encryption-api module defines enclave as the service that core uses for encryption operations and key access. Private keys should never be exposed outside of its system boundary.

One of the proposed deployment options is that enclave can be co-deployed in the same location as private keys and expose its services to tessera which is potentially deployed in a different location.

Task 1. Provide optional enclave api/transport options to communicate with tessera core.
Task 2. Provide optional configuration that instructs tessera core how to access enclave securely.

Allow contract keys to be in separate files, rather than only in the config file

Test If Vagrant will run under 4 GB for Tessera integrated 7 nodes example. Last test was with 6 GB.

Test memory profiling and update in readme