hyperledger-archives/aries-framework-go-ext

aries-framework-go-ext's Issues

enable testpackage and nlreturn linters

Implement did:jwk VDR - Read

did:jwk is a deterministic transformation of a JWK into a DID Document. Spec: https://github.com/quartzjer/did-jwk/blob/main/spec.md

Implement Read, Update (not supported) and Deactivate (not supported). Create will be handled in subsequent issue.

Ensure that the MySQL provider is not vulnerable to injection attacks

Some of the SQL queries involve inserting text. If it's something a user can define then there may be a vulnerability here.

Don't add empty anchor origin to sidetree requests

Currently sidetree client adds anchorOrigin even when it is empty. Only add anchor origin to create/recover requests if anchor origin is not empty.

Update sidetree-core

Update sidetree-core

MongoDB storage provider: Allow caller to pass in context

See this comment chain for more info: #259 (comment)

Full PostgreSQL storage provider/store implementation

The current PostgreSQL storage implementation is incomplete.

In no particular order:

• Some methods are just not implemented at all
• The SetStoreConfig method does not allow tag names to be removed from the store config. Additionally, tag names cannot be added properly - if some existing tag names plus some new ones are used in one call, then the whole operation will fail. There are many difficulties here that require consideration. With the PostgreSQL implementation, tag names are columns. If removing a tag names, do we remove the whole column? If so, the tag information gets destroyed. This isn't true in other implementations (like NoSQL ones), so there's an inconsistency here among implementations.
• Tag names are case-insensitive. For other storage provider implementations, they are case-sensitive. Either this implementation should allow them to be case-sensitive or the interface should specify that they should be case-insensitive in order to ensure consistency among implementations.
• SetStoreConfig is mandatory, but should be optional. This is tricky since we need the columns to exist before doing a Put or Query using those tags (in NoSQL implementations like MongoDB, there are no columns so this isn't a problem).
• Overwriting an existing key-value pair will not update the tags - they should be updated.
• The query method needs to support tag name + tag value queries (currently only tag name queries are supported)

Add postgres package

Update sidetree-core and orb (rename also-known-as patches)

Update sidetree-core and orb (rename also-known-as patches)

mysql: optimize the `tagMap` mechanism

The tagMap mechanism of the mysql storage implementation should be optimized. It is being stored as a composite value of a single record in the data table. Instead, the tags should be maintained in a separate 'tags' table. Queries, updates, and deletions involving the tagMap should not require SELECT prior to UPDATE/DELETE. Furthermore, it should be possible to do bulk updates/deletions to the tag map (see Batch()).

Don't base64-encode JSON data in CouchDB storage provider. Store binary data as attachments

CouchDB provider: if data is binary and large, store as a CouchDB attachment

CouchDB has something called "attachments" that could be used to store binary data in a more efficient form. We should make it so that if the binary data is greater than some size, then it gets stored as an attachment. When retrieving documents from CouchDB, we must then check to see if they have an attachment. If they do, then we return the attachment data. Otherwise, we look at the usual "value" field.

MySQL storage provider: in GetStoreConfig, should check for underlying database's existence instead of looking at in-memory stores

MongoDB storage provider: investigate using compound indexes to make querying with a sort faster

The current storage interface makes it hard to specify compound indexes cleanly. Investigate how compound indexes may benefit performance, and how they can be supported by the storage interface in a clean way.

Return error did not found for orb vdr

Dereference nil pointer

If len(didDoc.VerificationMethod) = 0, the function will return (nil, nil).

CouchDB provider - store tags as a nested object so we don't have to worry about conflicts with "payload"

The current code has an explicit check to ensure that no tag names are "payload", because that's a keyword reserved for the stored value (of the key-value pair). This is because tags are stored at the same level as "payload" - instead they could be nested one level deep.

Should all store implementations require tags to be defined in store config before allowing them to be used?

Some store implementations don't require it at all (EDV in non-ext repo), some store implementations don't require it but will benefit from a performance improvement thanks to indexing (CouchDB) and some store implementations require it and will throw an error if it's not done (probably MySQL?)

Should all store implementations enforce the store config? Even if it's not required or optional?

Edit: the EDV storage implementation will soon make use of the store config to create indexes, so it'll be in the "benefits from from indexing but not requiring it" category

Add option retry for DID not found for orb vdr

Create repo structure

The following structure is based on known proposed contributions.

Common Actions:

• All contributions must include Apache v2 License headers. This must be enforced by global checks.
• All contributions must be linted. This must be enforced by global checks.
• All contributions must have unit & bdd / integration tests. This must be enforced by global checks.
• Checks should only be run on changed (or new) modules for a particular commit.

Open Questions

• Name of parent folder. Based on a prior aries presentation, I am thinking "component".
• Do we have any common code in this repo contained within the top-level module?
• How do we handle codecov?

Structure:

• README
• LICENSE
• .github (workflow definitions)
• go.mod (empty for now)
• Makefile (global checks enforcement)
• component (or ... module | mod | addon | ext) ("plugin" would be confused for go plugins)
  • storage
    • couchdb
      • go.mod
      • go sources (or under pkg directory if needed) - example: store.go, store_test.go
      • Makefile
      • README
      • test
    • mysql | sql (does it make sense to combine several variations of sql)
      • go.mod and other files (see above)
    • postgres (see above)
      • go.mod and other files (see above)
    • mongodb
      • go.mod and other files (see above)
  • vdr
    • indy
      • go.mod and other files (see above)
    • trustbloc
      • go.mod and other files (see above)
  • crypto (future)
  • kms (future)
  • credential | cred (future)
  • protocol (future)
• scripts (common scripts - e.g., checks enforcement).
• test (common test support - if applicable).
• pkg (TBD: do we have any extension common code - most common code be in the framework repo).
• cmd (TBD: do we have any binaries generated here - initial thought is no).

Longform VDR method should not include did prefix

Longform VDR method should not include did prefix

Add vdr to create and resolve sidetree long-form (including did:ion) without network calls

MongoDB storage provider should disallow using tag names containing "<" and ">" in store configuration

The Put method already does this, but the SetStoreConfig method should as well, since those names conflict with the query syntax. Currently SetStoreConfig only checks for ":" characters.

Add AlsoKnownAs support to Sidetree VDR

Add AlsoKnownAs support to Sidetree VDR

What's the aim of this repository?

Hello, could you add to README information how this repository differ from https://github.com/hyperledger/aries-framework-go?

It took me some time digging to understood that it's probably containing extensions for aries-framework-go.

Add AlsoKnownAs support to Orb VDR

Add AlsoKnownAs support to Orb VDR

Add sidetree client

Ensure storage implementations are compatible with binary data like cbor

New CouchDB implementation - can't retrieve data stored under a key containing special URL characters

Due to a bug in the underlying Kivik library (appears to be an issue with the URL encoding, if keys containing special URL characters are used, then the document won't be retrievable later. We'll either need to wait until the Kivik library is fixed, or we'll have to come up with a workaround.

Provide performance flag for MongoDB batch method to allow unordered operations

MongoDB allows a BulkWrite call to be ordered or unordered. Right now all calls to BulkWrite are always ordered. For callers that don't need this, allowing them to specify that the call can be unordered can allow MongoDB to process the operations faster.

Update Orb revision

Update Orb revision (fix for deactivated doc verification)

CouchDB support for non-admin account

Currently, the couchdb store can't operate with a non-admin account.

I found the following issues that hold back the couchdb store:

1. The _users database is queried. This feature has been disabled in version 3.1.0 apache/couchdb#3084
2. Requested stores are directly created without testing their existence. The subsequent error message is that there's a lack of permissions https://github.com/hyperledger/aries-framework-go-ext/blob/main/component/storage/couchdb/store.go#L373

CouchDB provider - return newstorage.MultiError from Batch method

The Batch method just returns a regular error. It would be better to examine the BulkResults object returned from the BulkDocs method from the Kivik library (currently that return value is ignored) to see if we can return a MultiError which will contain errors for each operation (if any occurred)

Update storage interface.

Aries interface was changed. We need to update the storage extensions due to this change. See hyperledger-archives/aries-framework-go@b51ab9a#diff-5b2df4e72b7663d31d113c48920f911f24be00c599ce90bee6a16253d425e937R63

MongoDB tests don't work

make unit-test command runs all unit tests. Currently, MongoDB tests are not working properly.
MongoDB depends on the docker image. But Image isn’t created anywhere. Theoretically, tests should fail but they are not.
The problem is with os.Exit(0). Even if an error happens we exit with a successful code (0).
See https://github.com/hyperledger/aries-framework-go-ext/blob/main/component/storage/mongodb/mongodb_test.go#L47

What needs to be done?

• when an error happens the exit code must be greater than 0
• docker image needs to be set up (e.g dockertest is preferable)

CouchDB provider - consider adding option for preserving history in Batch calls?

In the case of a batch call where some of the operations are redundant (like a put, and then a put that overrides the previous one), those redundant operations will be stripped out. This is necessary to allow the CouchDB batch call to work correctly. CouchDB will not have any revision history of these stripped out operations, however. We may want to consider adding an option to preserve revision history (at the expense of speed most likely)

Move external databases from aries-framework-go repo

https://github.com/hyperledger/aries-framework-go/tree/master/pkg/storage

• CouchDB
• MySQL

CouchDB provider - allow context to be passed in

Currently context is always context.Background(). Allow this to be customizable.

Full MongoDB implementation of new storage interface

The MongoDB implementation needs to be updated to support new functionality in order to be used as an aries-framework-go storage provider. At a minimum, the SetStoreConfig and Query methods need to be implemented, as well as tag storage (currently the Put method throws an error if you try to store tags).

Use proper indexing in MySQL provider instead of the "Tag Map"

Use MySQL's built in indexing instead the "tag map" introduced in #74

Add options to VDR accept; update long-form logic

Currently VDR makes decision on whether to accept or reject request based on method only.
Add options to VDR Accept function so VDR can make more complex decisions (for example
long-form VDR should reject request if number of parts in DID is not at least four)

Support multi-tag query in CouchDB

Refactor sidetree client send operation request function

Refactor sidetree client send operation request function so it can be passed as an option.

CouchDB implementation of new storage interface in Aries

CouchDB implementation of https://github.com/hyperledger/aries-framework-go/blob/master/pkg/newstorage/newstorage.go

Update Orb VDR

Update Orb VDR with latest sidetree-core, sidetree VDR and aries-framework go

Don't send service priority if not set

Don't send service priority if not set

MongoDB storage provider: investigate using aggregates to allow results and total items counts to be obtained in a single find call

This might help performance for clients that always want to get total item counts. Maybe it'll reduce performance for clients that don't need it? We should investigate. The usage of aggregates/non-aggregates could be enabled by adding a new query option.

Error when running linter locally

Try running make lint on your local system, and you'll get the following error:

level=warning msg="[runner] Can't run linter goanalysis_metalinter: bodyclose: failed prerequisites: [[email protected]/hyperledger/aries-framework-go-ext/component/vdr/indy: analysis skipped: errors in package: [/opt/workspace/vdr.go:21:2: could not import github.com/hyperledger/indy-vdr/wrappers/golang/vdr (/go/pkg/mod/github.com/hyperledger/indy-vdr/wrappers/[email protected]/vdr/cb.go:32:8: could not import C (cgo preprocessing failed))]]"
level=warning msg="[runner] Can't run linter unused: buildir: failed to load package vdr: could not load export data: no export data for \"github.com/hyperledger/indy-vdr/wrappers/golang/vdr\""
level=error msg="Running error: buildir: failed to load package vdr: could not load export data: no export data for \"github.com/hyperledger/indy-vdr/wrappers/golang/vdr\""

Note that this doesn't happen in the CI.

Full MySQL implementation of new storage interface

There are additional methods that are not implemented that may be useful for someone extending aries-framework-go.

Error when running linter locally (Mac)

#42 fixed #38 on Linux, but on Mac the error still comes up.