hyperledger/aries-agent-test-harness

Aries agent test framework, with agent backchannel support

aries-agent-test-harness's Issues

Auto Connection fail on VCX and new agent support

During ACA-PY and VCX test, I found the following error.
Agents with auto connection setting show an error in the current test code.
(I found same issue on aca-py --auto-accept-invites setting.)
Maybe we need to change 0160-connection.py for auto-connection features?

./manage run -d acapy -b vcx -t @AcceptanceTest -t ~@wip

  And "Faber" and "Bob" have an existing connection  features/steps/0160-connection.py:245 1.862s
  Assertion Failed: FAILED SUB-STEP: And "Bob" receives the connection invitation
  Substep info: Traceback (most recent call last):
    File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
      match.run(runner.context)
    File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
      self.func(context, *args, **kwargs)
    File "features/steps/0160-connection.py", line 113, in step_impl
      assert expected_agent_state(invitee_url, "connection", context.connection_id_dict[invitee][context.inviter_name], "invited")
  AssertionError
  Traceback (of failed substep):
    File "/usr/local/lib/python3.7/site-packages/behave/model.py", line 1329, in run
      match.run(runner.context)
    File "/usr/local/lib/python3.7/site-packages/behave/matchers.py", line 98, in run
      self.func(context, *args, **kwargs)
    File "features/steps/0160-connection.py", line 113, in step_impl
      assert expected_agent_state(invitee_url, "connection", context.connection_id_dict[invitee][context.inviter_name], "invited")

  Captured stdout:
  From http://0.0.0.0:9030 Expected state ['invited'] but received request , with a response status of 200

And I create new branch for vcxagencynode(agent repo : https://github.com/AbsaOSS/vcxagencynode)
https://github.com/sktston/aries-agent-test-harness/tree/nodeagency
this repo includes node agency back-channel and vcx support.(but same issue on connection test)

I still have several issues(connection, change SDK to Aries-VCX, etc.), but do you need to merge in Master?

make operation a route parameter

Part of the opportunity is providing feedback on the test harness which could simplify the implementation of the Aries Framework .NET backchannel.

I think making the operation a route parameter instead of a key in the JSON body could make the implementation of this - and future - backchannels simpler. It would be more in line with API conventions where each URL generally represents a unique action. By adding the operation to the url parameter each endpoint represents a unique operation.

old: /agent/command/{topic}
new: /agent/command/{topic}/{operation}

I think this could have benefits for both the .NET backchannel and future backchannels:

• We could create a swagger definition allowing users to explore the backchannel API via the browser
• Users could import the swagger definition in tools such as Postman to interact with the API
• Backchannel setup could be made easier by generating the boilerplate code from the swagger definition using e.g. OpenAPIGenerator

It would also make the code in the .NET backchannel a bit cleaner - but that is negligible.

Test Harness Expansion to cover AIP 1.0 Protocol 0037-present-proof (Happy Path Only)

References:
0037-present-proof from the Aries Interop Profile

• Get experts in the protocol to identify and define the BDD test cases

• Determine and write any cases negative or otherwise not found by protocol experts

• Implement the test steps for the BDD Test Cases

• Define the backchannel API calls necessary to support the test functions.

• Ideally, include a documentation structure (perhaps Swagger) that will help agent builders to know how to build the backchannel controller for their agent.

• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py

Test Harness Expansion to cover AIP 1.0 Protocol 0160-connection-protocol (Exception & Negative Tests)

This Story relates to issue #7

Test Harness Expansion to cover AIP 1.0 Protocol 0037-present-proof (Exception & Negative Tests)

This Story relates to issue #11

NegativeTest ExceptionTest P3 wip NeedsReview
Scenario: Present Proof without the verifier requesting for presentation of proof

ExceptionTest P2 wip NeedsReview
Scenario: Present Proof where the prover doesn’t want to reveal the data and sends rejection

Compare Aries Agent Test Harness w/ Aries Protocol Test Suite

One thing that you should do is to look at the other solution and to decide if we are right or wrong about the Test Harness architecture vs. the Aries Protocol Test Suite architecture. You may not be able to determine one is better than the other because both have pros and cons, but you should be able to come up with questions for us on why one might be preferred over the other. What we really care about in the test suite is that it be easy to use (interactively and within a pipeline) and that tests be easy to develop. Which architecture is better given those criteria? Currently, there is a little bit of work going on with the APTS, and you will be the only person on the Test Harness, so it's not a matter of which one is further along.

• Get the Aries Protocol Test Suite running locally

• Connect with the people doing the work on APTS

use requested_attributes instead of requested_values

When the AATH client sends a proof/send-request operation to a backchannel it uses requested_values as attribute key for the requested attributes. However the Indy API's expect this to be requested_attributes.

I now see the ACA-Py backchannel transforms this parameter before sending it to ACA-Py. I know AATH is not Indy specific, but when another format than Indy will be used the whole request_presentations~attach will be different, so I think it will be easier to just use requested_attributes.

Maybe I'm missing something here?

Example:

{
   "connection_id":"<<connection_id>>",
   "presentation_proposal":{
      "@type":"did:sov:BzCbsNYhMrjHiqZDTUASHg;spec/present-proof/1.0/request-presentation",
      "comment":"This is a comment for the request for presentation.",
      "request_presentations~attach":{
         "@id":"libindy-request-presentation-0",
         "mime-type":"application/json",
         "data":{
            "requested_values":{ // --> change to `requested_attributes`
               "attr_1":{
                  "name":"attr_1",
                  "restrictions":[
                     {
                        "schema_name":"test_schema",
                        "schema_version":"1.0.0"
                     }
                  ]
               }
            }
         }
      }
   }
}

Remove dependancy on Dummy Cloud Agent and Cargo

Remove Indy-isms in Issue Credential - change steps

In an effort to remove the Indy-specific features in the test cases, I'm thinking that we should replace the current test case flow from:

    Given "Acme" has a public did                                                        # features/steps/0036-issue-credential.py:28 0.003s
    When "Acme" creates a new schema                                                     # features/steps/0036-issue-credential.py:40 0.022s
    And "Acme" creates a new credential definition                                       # features/steps/0036-issue-credential.py:59 0.019s
    Then "Acme" has an existing schema                                                   # features/steps/0036-issue-credential.py:76 0.012s
    And "Acme" has an existing credential definition                                     # features/steps/0036-issue-credential.py:89 0.016s
    Given "2" agents                                                                     # features/steps/0160-connection.py:15 0.000s
      | name | role   |
      | Acme | issuer |
      | Bob  | holder |
    And "Acme" and "Bob" have an existing connection                                     # features/steps/0160-connection.py:245 31.036s
    And "Acme" offers a credential               

To be something like the following. For Indy, the "prepares to issue a new credential type" would be create a new schema and create a new cred def, likewise for the "is ready to issue an existing credential type".

    Given "Acme" has a public did                                                        # features/steps/0036-issue-credential.py:28 0.003s
    When "Acme" prepares to issue a new credential type
    Then "Acme" is ready to issue an existing credential type
    Given "2" agents                                                                     # features/steps/0160-connection.py:15 0.000s
      | name | role   |
      | Acme | issuer |
      | Bob  | holder |
    And "Acme" and "Bob" have an existing connection                                     # features/steps/0160-connection.py:245 31.036s
    And "Acme" offers a credential               

I was also expecting that the the above would have parameters indicating what credential type (e.g. driverslicence) to create/check for. Is that not they way it works?

What do you think @nodlesh @TimoGlastra @ianco ?

Test Harness Expansion to cover AIP 1.0 Protocol 0056-service-decorator

This protocol is (will be) tested indirectly with the Connectionless tests #90 for the Present Proof Protocol.

References:
0056-service-decorator from the Aries Interop Profile

• Get experts in the protocol to identify and define the BDD test cases

• Determine and write any cases negative or otherwise not found by protocol experts

• Implement the test steps for the BDD Test Cases

• Define the backchannel API calls necessary to support the test functions.

• Ideally, include a documentation structure (perhaps Swagger) that will help agent builders to know how to build the backchannel controller for their agent.

• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py

Align expected protocol states with RFCs

The test harness often checks in with agents for their current state in a specific protocol. E.g. assert that the state for this connection is invitation. Current problem I'm facing is that the states the test harness expects are not consistent with the RFCs.

RFC -> Expects:

• invitation -> invited
• proposal-sent -> proposal_sent
• ...

The .NET framework currently only supports Invited, Negotiation and Connected states for a connection - dating back from before the states were formally defined. However Tomislav is currently thinking about whether to change this. If he can resolve his backwards compatibility concerns and this gets changed in .NET, it would be nice for the test harness to also align with the RFCs.

This would also be nice for Framework JavaScript in the future so we wouldn't have to map RFC state to AATH state.

Thoughts? I suspect this is done because of the states AcaPy returns, but as AATH is expanding beyond AcaPy I think this would be a good moment to think about changing current behaviour.

Test Harness Easily Consumable Test Results/Reports

Basic reporting is provided by native Behave.

Needs to be Machine consumable
Possibly results serialization with query capability and trend over time reports

Research:
https://behave.readthedocs.io/en/latest/formatters.html
https://docs.qameta.io/allure/#_behave

Aries Agent Test Harness determining Conformance with Aries Interop Profile 1.0 for Agent Builder

AS AN Aries Agent Builder, I WANT to run a series of tests, SO THAT I can determine interoperability with other Aries Agents based on Aries Interop Profile 1.0

Acceptance Criteria

Scenario: Agent Builder creates agent and gets conformance report
Given a ACA-Py Agent Builder has created a new agent
When they execute the Aries Agent Test Harness
Then they will get a report stating the level of conformance to the Aries Interop Profile 1.0.0

Note: The aries-rfc 0270: Interop Test Suite States "The test suite isn't a compliance tool, and it should be unopinionated about what's important and what's not.”
We agree that we are not aligning with this goal? Or are we using the term compliance in a different sense? This Test Harness determines "conformance" to a version of the AIP it is not labeling anything as compliant with a standard.

Scenario: Agent Builder creates a backchannel adaptor for their agent instance (This may need its own sub story instead of having it in the epic)
Given a ACA-Py Agent Builder
And has created a new agent
When they use the Aries Agent Test Harness
And create a backchannel adaptor based on the interface provided
Then they will be able to execute the Aries Agent Test Harness suite of tests

Scenario: Usage in Pipelines (Maybe move into its own Epic)
Scenario: Usage for Devs (Maybe move into its own Epic)
Scenario: Usage for P2P Interop (Maybe move into its own Epic)

Test Harness Expansion to Cover AIP 1.0 Protocol 0019-encryption-envelope

References:
0019-encryption-envelope from the Aries Interop Profile

Allure Security Update with Open Reports Viewer

This feature was released from Allure Docker Service version 2.13.8:
Check documentation here:
https://github.com/fescobar/allure-docker-service#make-viewer-endpoints-public
https://github.com/fescobar/allure-docker-service/releases/tag/v2.13.8
https://github.com/fescobar/allure-docker-service-ui/releases/tag/v7.0.3

This feature was released from Allure Docker Service UI version 7.0.3:
https://github.com/fescobar/allure-docker-service-ui/releases/tag/v7.0.3
Check documentation here:
https://github.com/fescobar/allure-docker-service#make-viewer-endpoints-public
https://github.com/fescobar/allure-docker-service/releases/tag/v2.13.8

In a nutshell, if security is enabled, you can use the MAKE_VIEWER_ENDPOINTS_PUBLIC flag to expose the reports publicly.
In Docker Compose

    environment:
      SECURITY_USER: "my_username"
      SECURITY_PASS: "my_password"
      SECURITY_ENABLED: 1
      MAKE_VIEWER_ENDPOINTS_PUBLIC: 1

• Update aries-agent-test-harness/aries-test-harness/docker-compose.yml security (1)
• Update aries-agent-test-harness/aries-test-harness/send-results.sh for security (1)
• Deploy security-enabled Allure Service on OpenShift (Maybe a separate instance leaving the unsecured one in place for now?)
• Test new send-results.sh with deployed allure service
• Obfuscate the username and password in send-results.sh
  (1) in Allure-Security-Public-Viewer branch in https://github.com/nodlesh/aries-agent-test-harness/tree/Allure-Security-Public-Viewer/aries-test-harness/allure

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means its critical that we work to make our content as discoverable as possible; Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

That's in, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

• If your org is not in the list below, or the table contains errors, please create an issue here.

• While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

• Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

• If your application is live, add the production URL.

Ministry Short Codes

Short Code	Organization Name
AEST	Advanced Education, Skills & Training
AGRI	Agriculture
ALC	Agriculture Land Commission
AG	Attorney General
MCF	Children & Family Development
CITZ	Citizens' Services
DBC	Destination BC
EMBC	Emergency Management BC
EAO	Environmental Assessment Office
EDUC	Education
EMPR	Energy, Mines & Petroleum Resources
ENV	Environment & Climate Change Strategy
FIN	Finance
FLNR	Forests, Lands, Natural Resource Operations & Rural Development
HLTH	Health
FLNR	Indigenous Relations & Reconciliation
JEDC	Jobs, Economic Development & Competitiveness
LBR	Labour Policy & Legislation
LDB	BC Liquor Distribution Branch
MMHA	Mental Health & Addictions
MAH	Municipal Affairs & Housing
BCPC	Pension Corporation
PSA	Public Safety & Solicitor General & Emergency B.C.
SDPR	Social Development & Poverty Reduction
TCA	Tourism, Arts & Culture
TRAN	Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.

Revocation Test Coverage for Negative and Exception Scenarios

Open API spec not updated

When working on the backchannel for .NET I added an open api spec to the repo that describes all endpoints a backchannel should implement. When looking at the test harness again I see it is not updated. I know I was the one that created it so it won't be magically updated.

I haven't looked at AATH for a while and when now implementing more backchannel operations for the AFJ backchannel it was really helpful. I've added a few endpoints that I came across when updating the AFJ backchannel (in #158), but I think it would be great if we can do a shared effort of keeping it up to date :)

OR should we take something else as reference for creating backchannels? The google sheet also doesn't seem completely up to date.

How to test with postgres as database for aca-py agent.

where to pass more arguments to run postgres as database like --wallet-storage-type postgres_storage

Error in test execution

How I solve this error ?

1 feature passed, 4 failed, 0 skipped
15 scenarios passed, 8 failed, 56 skipped
143 steps passed, 8 failed, 571 skipped, 0 undefined
Took 6m23.121s
2021-01-26 19:02:24,123 aries_cloudagent.core.dispatcher ERROR Handler error: invitation_create
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/routes.py", line 73, in invitation_create
use_public_did=use_public_did,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/manager.py", line 92, in create_invitation
multi_use=multi_use,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/connections/v1_0/manager.py", line 130, in create_invitation
raise ConnectionManagerError("Public invitations are not enabled")
aries_cloudagent.protocols.connections.v1_0.manager.ConnectionManagerError: Public invitations are not enabled
2021-01-26 19:02:24,125 aries_cloudagent.admin.server ERROR Handler error with exception: Public invitations are not enabled
2021-01-26 19:02:24,125 aiohttp.server ERROR Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 144, in ready_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 177, in debug_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aiohttp_apispec/middlewares.py", line 45, in validation_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 275, in apply_limiter
return await task
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/routes.py", line 73, in invitation_create
use_public_did=use_public_did,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/manager.py", line 92, in create_invitation
multi_use=multi_use,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/connections/v1_0/manager.py", line 130, in create_invitation
raise ConnectionManagerError("Public invitations are not enabled")
aries_cloudagent.protocols.connections.v1_0.manager.ConnectionManagerError: Public invitations are not enabled
2021-01-26 19:02:24,100 aries_cloudagent.core.dispatcher ERROR Handler error: invitation_create
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/routes.py", line 73, in invitation_create
use_public_did=use_public_did,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/manager.py", line 92, in create_invitation
multi_use=multi_use,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/connections/v1_0/manager.py", line 130, in create_invitation
raise ConnectionManagerError("Public invitations are not enabled")
aries_cloudagent.protocols.connections.v1_0.manager.ConnectionManagerError: Public invitations are not enabled
2021-01-26 19:02:24,104 aries_cloudagent.admin.server ERROR Handler error with exception: Public invitations are not enabled
2021-01-26 19:02:24,104 aiohttp.server ERROR Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 144, in ready_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 177, in debug_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aiohttp_apispec/middlewares.py", line 45, in validation_middleware
return await handler(request)
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/admin/server.py", line 275, in apply_limiter
return await task
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/routes.py", line 73, in invitation_create
use_public_did=use_public_did,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/out_of_band/v1_0/manager.py", line 92, in create_invitation
multi_use=multi_use,
File "/usr/local/lib/python3.7/site-packages/aries_cloudagent/protocols/connections/v1_0/manager.py", line 130, in create_invitation
raise ConnectionManagerError("Public invitations are not enabled")
aries_cloudagent.protocols.connections.v1_0.manager.ConnectionManagerError: Public invitations are not enabled

Test Harness Expansion to cover AIP 1.0 Protocol 0025-didcomm-transports

This Protocol is tested indirectly because all tested protocols encompass this protocol in themselves. Consider closing this unless there are tests that can be put together to address didcomm transports directly.

References:
0025-didcomm-transports from the Aries Interop Profile

• Get experts in the protocol to identify and define the BDD test cases

• Determine and write any cases negative or otherwise not found by protocol experts

• Implement the test steps for the BDD Test Cases

• Define the backchannel API calls necessary to support the test functions.

• Ideally, include a documentation structure (perhaps Swagger) that will help agent builders to know how to build the backchannel controller for their agent.

• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py

Add reporting output to the test harness

We need a useful way to report on the status of a test run, ideally including (off the top of my head):

• what was tested (agents involved)
• what tests were run (likely, what tags where run)
• what was the status of the runs (likely, for each tag, how many tests passed, failed, skipped, errored off)
• how many "crashes" occurred -- e.g. stack traces were produced

Since we are using Behave (a standard test framework) it makes sense to use a Behave-specific test output reporting tool. A quick search uncovered Allure (docs, website), which looks pretty cool. I assume there are others that should be considered as well.

This Stack Overflow question offers several ideas, including using a "behave2cucumber" tool that enables the use of an BDD Cucumber reporting tool.

While, most effort should be spent on that, Aries RFC 0270 proposes an output format that is specific to Aries protocols and roles. Some thought should be given to how the test cases and roles (e.g. Acme, Bob, Faber, Mallory) relate to Aries protocols and roles within those protocols such that an RFC 0270-style report can be generated.

Test Suite Execution/Conformance Result Public Links

AS AN Agent Builder, I WANT to know what the test suite results are for my agent based on a Aries Interop Profile version, SO THAT I can link the results in the AIP RFC where my Aries Agent is listed.

Acceptance Criteria

Scenario: Agent Builder add a link to Test Suite results for the Aires Agent
Given an Agent Builder who has created an Aries Agent
When they execute a test suite related to the AIP version
Then they will see the results of the test suite
And be provided a public link to use for future reference

Test Harness Expansion to cover AIP 1.0 Protocol 0035-report-problem

References:
0035-report-problem from the Aries Interop Profile

• Get experts in the protocol to identify and define the BDD test cases

• Determine and write any cases negative or otherwise not found by protocol experts

• Implement the test steps for the BDD Test Cases

• Define the backchannel API calls necessary to support the test functions.

• Ideally, include a documentation structure (perhaps Swagger) that will help agent builders to know how to build the backchannel controller for their agent.

• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py

Add way to build docker images with branches or versions

Using the acapy-master docker build process, it was pretty easy to temporarily update the requirements.txt file to use a branch from another fork that had been submitted as a PR, build the docker images and run the tests.

It would be nice to have a general purpose way to do that. This is easy enough if we constrain the issue to the ACA-Py use case, but it would be nice to make it general purpose. Perhaps when specifying the agent to build, being able to pass in the branch to be used, and then let the build process handle the details.

Consider options for doing that to provide a consistent interface to do that, but allowing the implementation to be backchannel/test agent specific.

correctly log ACA-Py logs to log file

In #81 I made some changes to the ACA-Py backchannel to also log the ACA-Py logs. However after using it for a while it doesn't seem to work correctly.

For context: this pr persists the logs of all containers (using docker logs) to files in the .logs folder.

However only the backchannel logs (not the ACA-Py logs) are written to their corresponding log file (e.g. .logs/acme_agent.log. As I'm pretty unfamiliar with Python I'm certain I messed something up. @ianco @nodlesh could you maybe take a look or point me in the right direction if you have the time?

As a side note: would it be possible to add a param to print the logs without color when using a non interactive shell or something? The logs are a bit hard to read with the color escaping characters:

�[0m�[?7h�[0;35mPassing payload to handler handle_connections payload is: {"accept": "manual", "created_at": "2020-08-03 13:04:39.742471Z", "connection_id": "915086c2-2515-49ee-af7c-1c6707d3c33a", "state": "invitation", "their_label": "dotnet", "initiator": "external", "invitation_key": "MRZsFpyVZ4HHCzkr4VF4sPodjgB5cNpEi8PQgFZr6gi", "routing_state": "none", "updated_at": "2020-08-03 13:04:39.742471Z", "invitation_mode": "once"}�[0m
�[0mListening to backchannel on port 9030
Listening to web_hooks on port 9033

Propagate Agent startup issues through to the backchannel startup if a problem arises

AS A Aries Test Harness User, I want to know all aca-py issues on startup, SO THAT I can properly diagnose and rectify the issue.

• Propagate aca-py startup executions though to the backchannel startup routine.

• print out either by default or as a debug message all the agents startup parameters.

Finish Connectionless Proof Test Scenario(s) to AATH

This was attempted in Aug 2020. There is code in the AATH to make this work. In submission related to this issue. It was determined that Acapy at the time did not support the connectionless scenario is played as the prover. This work has been put on hold until Acapy has this fully implemented.

The main Story for this work was #11.

Hyperledger Aries Interoperability Testing with Aries Framework Go

Overview

The AATH is a BDD-based test harness for verifying interoperability across a variety of “components under test” (CUTs)—Aries Agents and Aries Agent Frameworks. To add a new CUT, a backchannel needs to be implemented that converts test case steps (in the form of HTTP requests) into operations performed by the CUT. The backchannel and the CUT need to be combined into a Test Agent docker image. This task calls for the creation of an Aries Framework Go backchannel which supports tests compatible with that framework. In turn, this requires creating some new tests based on existing tests as Aries Framework Go uses different protocols for establishing connections and issuing/proving verifiable credentials.

Description

The Aries Agent Test Harness is an interoperability test suite for Aries components based on the Aries protocols (documented in the Aries RFC repo). The test harness defines an HTTP API and uses Docker conventions to enable the wrapping of an Aries component into a Test Agent (TA) that can participate in runs of the test suite. Each test run involves two or more TAs, so once enabled, the interoperability of a component with other Aries components can be verified through a series of test runs. Details of the Aries Agent Test Harness can be found in this repo (https://github.com/hyperledger/aries-agent-test-harness). There you can find instructions to do a run of the test suite in your browser or on your local system (assuming you have docker, git and bash).

This task requires working with the BC Gov Digital Trust Service (DTS) team and the Aries Framework Go team to deliver a key element of interoperability in the Hyperledger Aries. We have created the Aries Agent Test Harness to be a cornerstone of interoperability in the Aries and larger Trust over IP ecosystem. By enabling different components to be embedded in Test Agents (TAs), we can execute test runs with a mixture of TAs to verify the interoperability of the underlying components.

Since Aries Framework Go has a similar Admin API to Aries Cloud Agent Python, the existing backchannel for ACA-Py (written in Python) could be either modified to support both frameworks, or copied and modified to be unique for the Aries Framework Go. The best choice depends on the gap between the ACA-Py and Aries Framework Go Admin API. A side benefit of the work might be feedback to the two teams on how to align the APIs.

Because of the different protocols supported by the current frameworks supported in Aries Agent Test Harness (e.g. ACA-Py) and Aries Framework Go, some new tests, or variations on existing tests will be needed for exercising even the most basic features of the framework — establishing connections. Aries Framework Go uses the newer “Out-of-Band” and “DID Exchange” protocols to establish a connection, while Aries Cloud Agent Python uses the current standard approach — the “Connections” protocol. Some of the work will require adapting the existing tests or creating new tests that use the new protocols. This work being done now will come in handy very soon as the ACA-Py team completes work on “Out-of-Band” and “DID-Exchange” and we can test it’s interoperability with Aries Framework Go.

Beyond the basics outlined here, the work will then move into protocols supporting the use of different DID methods and ledgers, as Aries Framework Go supports ledgers other than Indy. We’ll work with the Aries Framework Go team to understand how best to do that work in the context of Aries Framework Go.

Acceptance Criteria

1. The implementation of a backchannel for Aries Framework Go that executes new tests to establish connections using the ““Out-of-Band” and “DID-Exchange” protocols.
2. Embed the backchannel and Aries Framework Go in a Test Agent docker container using the conventions required by the Aries Agent Test Harness.
3. Enable debugging of the Test Agent running in the docker container during test runs.
4. Execute the Aries Framework Go compatible tests in the test suite such that they complete successfully, or GitHub issues are raised against the flawed component.
5. Document the backchannel, including how to run, debug and extend the test suite.
6. Provide feedback on the test harness, including requests to change the harness itself to simplify the implementation of the Aries Framework Go backchannel.
7. Provide feedback to the Aries Cloud Agent Python and Aries Framework Go teams on their products and how they might be better aligned, such as making the Admin API matching between the two frameworks.

Full Test Coverage of Revocation RFC 441 (Happy Path)

Test specific versions of aca-py agent

Would be a "nice to have" to test multiple aca-py versions, for example when we release a new aca-py version we could run a backward compatibility test against a previous version.

Accessible Hosting of the Allure Docker Containers for AATH running in Build Pipeline

AS Quality Assurance & Dev Ops, I WANT to access allure reports from Aries Agent Test Harness test execution running in the build pipeline perpetually, SO THAT the Test execution engine can be sure there is a reporting service to send results to for every daily run.

Description

To get the Allure docker related files:

git clone https://github.com/nodlesh/aries-agent-test-harness.git
cd aries-agent-test-harness/aries-test-harness/allure

There are two containers to instantiate and keep running. Allure; which is the main service to serve reports to the public and handle report generation, and allure-ui; which is just an admin ui on all the services allure provides.
To instantiate those containers:

docker-compose up -d allure 
docker-compose up -d allure-ui

The docker-compose.yml will need to be updated to point allure-ui to the of the allure service (the allure container) host.

environment:
      ALLURE_DOCKER_PUBLIC_API_URL: "http://localhost:5050"

• Verify if Allure API is working. Go to -> http://host:5050/allure-docker-service/latest-report

• Verify if Allure UI is working. Go to -> http://host:5252/allure-docker-service-ui/

Let @nodlesh know the Host for the URLs above.

---

Security & Authentication

The following will need to be investigated before implementation.
The Allure UI may need to be secured. this would be done through basic authentication. Note there should be no need to secure the allure service. Access to the reports should be open.

If it is deemed necessary to secure both the UI and the Service we can use Allure internal security to do this. We would need to change the docker compose xml file in the allure folder to include

    environment:
      SECURITY_USER: "my_username"
      SECURITY_PASS: "my_password"
      SECURITY_ENABLED: 1

Allure Docker Service's security features will secure all or nothing, so the UI, the API, and the report would need a login. There is an item in their backlog that should address this by adding roll based access. fescobar/allure-docker-service#128
See https://github.com/fescobar/allure-docker-service#enable-security for more information on native Allure security.

In combination with the Allure security or if using basic auth, we will need to use GitHub Tokens and GitHub Secrets to run the send-results script form a GitHub action after a test execution.

Test Harness Test Scenario Execution Flexibility

AS A Test Suite Executioner I WANT to be able to execute suites, subsets, single, multiple test cases in a single run, SO THAT I can only run what will achieve my particular execution goals at the time

Acceptance Criteria

Scenario: User Executes an entire suite
Scenario: User executes a subset of the suite
Scenario: User executes multiple suites
Scenario: User executes a subset across multiple suites
Scenario: User executes a single test
Scenario: User executes a list of tests

The implementation may be the same as #3 using tags in the feature file may be all the flexibility we need, that along with Behaves ability to specify sets on the command line.

Docker Image for the Aries Agent Test Harness

AS A Test Harness Executioner, I WANT to just stand up a docker image with the Test Harness environment already configured, SO THAT I can increase my productivity, minimize user configuration errors, and standardize on a working environment

• remove dependancy on having the aca-py executable in aries-agent-test-harness/venv/bin/. Aca-py should be in the PATH, so there is no need to specify.

Admin API on Aca-py Issue Credential intermittently give 500 Internal server errors

Issue

The following Gets and Posts through the Admin API for Issue Credential (AIP 1.0) will intermittently throw a 500 internal server error, some Gets (for the webhooks) even need a sleep() before the call in the test harness to make the call work, even though there are sleeps and wait loops in the backchannel.

All of these examples are from the 0036-Issue-credential.py step implementation.

Store in the "Holder acknowledges the credential issue" step
(resp_status, resp_text) = agent_backchannel_POST(holder_url + "/agent/command/", "issue-credential", operation="store", id=context.holder_cred_ex_id, data=credential_id)

Get data from web hook in the "Holder Requests the credential" step
sleep(1)
(resp_status, resp_text) = agent_backchannel_GET(context.issuer_url + "/agent/response/", "issue-credential", id=context.cred_thread_id)

** Get data from web hook in the "Issuer Offers a credential" step**
sleep(1)
(resp_status2, resp_text2) = agent_backchannel_GET(context.holder_url + "/agent/response/", "issue-credential", id=context.cred_thread_id)

Sometimes removing the sleep(1) from some of the Gets will trigger a 500 on the next post. For example, not sleeping on (resp_status, resp_text) = agent_backchannel_GET(context.issuer_url + "/agent/response/", "issue-credential", id=context.cred_thread_id)
may cause a 500 on this Post, (resp_status, resp_text) = agent_backchannel_POST(issuer_url + "/agent/command/", "issue-credential", operation="issue", id=context.issuer_cred_ex_id, data=credential_preview)

Outcome

500 Internal Server Error\n\nServer got itself in trouble

Expected Outcome

A successful call or an error explaining the exact issue if its timing related or otherwise.

Workaround

Putting in the sleep(1) in the test harness seems to work for most calls, though it will occasionally still happen on Posts.

Other Information

I have not seen this happen when just running the tests through manage run. It will happen more when the agents are started and left running like while developing tests or debugging. Also note these 500 errors will not happen when stepping through code.
This does not happen with the Connection Protocol

Test Cases Affected

T001-API10-RFC0036
T002-API10-RFC0036
T003-API10-RFC0036
T004-API10-RFC0036

Severity

Medium

Business Priority

Medium

Research - Integration with APTS to use the Test AS AN Agent approach to do Negative testing

The "backchannel" is a Python API - you copy this file into your repo and edit it to support your agent:

https://github.com/ianco/aries-protocol-test-suite/blob/master/apts/src/aut.py

(Actually you copy the whole repo into your agent repo)

It's possible (although probably not trivial) to map the APTS "backchannel API" to AATH backchannel (theirs is a sub-set) and create a version of their aut.py that calls an AATH backchannel using http calls ... This may help in taking advantage of the Test IS AN Agent approach to do negative and exception testing, and maybe to run AATH tests through the APTS.

More productive would be to rewrite their aut.py to use an http interface rather than native Python calls, and change the APTS to call their backchannel using http, in which case we could converge the two and use the same backchannel for both test suites.

In summary, APTS:

• you have to copy the APTS repo into your agent repo and edit it
• not sure how you manage updates to the test coverage after you'e already copied it over
• supports only Python (?) since it uses a native Python API for the backchannel

Test Harness Expansion to cover AIP 1.0 Protocol 0160-connection-protocol (Happy Path Only)

AS AN Agent, I WANT to connect to another Agent, SO THAT we can do secure transactions together beyond this initial connection

Acceptance Criteria

Feature: Aries agent connection functions RFC 0160

Scenario: establish a connection between two agents
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection response
And "Alice" accepts the connection response
And "Bob" sends a response ping
And "Alice" receives the response ping
Then "Alice" and "Bob" have a connection

Question: The RFC states "The inviter sends a connection-response message at the end of the share phase" How is this captured here? It seems there might be some confusion here. Not sure if the Protocol Implemented needs to change or just tests?
Here is my rewrite of the scenario above based on how I see the RFC.
tags: P1 AcceptanceTest NeedsReview Automated
Scenario: establish a connection between two agents
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And "Alice" receives the connection request
And "Alice" sends a connection response
And "Bob" sends a response ping
And "Alice" receives the response ping
Then "Alice" and "Bob" have a connection

tags: P1 AcceptanceTest NeedsReview Automated
Scenario: Connection established between two agents and inviter gets a preceding message
Given we have two agents "Alice" and "Bob"
And Bob has sent a connection request to Alice
And Alice has accepted the connection request by sending a connection response
And Bob is in the state of complete
And Alice is in the state of responded
When Bob sends [message]
Then Alice is in the state of complete

Examples:
| message |
| acks |

tags: SingleUseInvite P2 ExceptionTest NeedsReview Automated
Scenario: Inviter Sends invitation for one agent second agent tries after connection
Given we have three agents "Alice", "Bob", and "Frank"
And "Alice" generated a single-use connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
And "Alice" accepts the connection request by sending a connection response
And "Alice" and "Bob" have a connection
When "Frank" sends a connection request based on the connection invitation
Then "Alice" sends a request_not_accepted error

tags: SingleUseInvite P2 ExceptionTest NeedsReview Automated
Scenario: Inviter Sends invitation for one agent second agent tries during first share phase
Given we have three agents "Alice", "Bob", and "Frank"
And "Alice" generated a single-use connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
When "Frank" sends a connection request based on the connection invitation
Then "Alice" sends a request_not_accepted error

tags: MultiUseInvite P3 DerivedFunctionalTest NeedsReview
Scenario: Inviter Sends invitation for multiple agents
Given we have three agents "Alice", "Bob", and "Frank"
And "Alice" generated a single-use connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
And "Alice" sent a connection response to "Bob"
When "Frank" sends a connection request based on the connection invitation
Then "Alice" sent a connection response to "Frank"

tags: MultiUseInvite P3 DerivedFunctionalTest NeedsReview
Scenario: Inviter Sends invitation for multiple agents completing connections for both
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And "Frank" receives the connection invitation
And "Frank" sends a connection request
And "Alice" accepts the connection request of "Bob" by sending a connection response
And "Alice" accepts the connection request of "Frank" by sending a connection response
Then "Alice" and "Bob" have a connection
And "Alice" and "Frank" have a connection

Question: Are invitations generic or specific to the intended invitee?
Answer: There can be two types single-use for one invitee or multi-use for more than one invitee. With a single-use invite it is generic at first, then at that point the invitation connection request, that invite can no longer be used.

Scenario: Inviter Sends invitations to an agent and never gets a connection request from them
Question: Does the invitee stay invited perpetually, what is the lifespan of an invite?
Answer: That's business logic. In theory (and to this point, in practice), there is no time limit on them. However, that could be logic applied by the controller. But inherently, there is no time limit from an RFC perspective.

tags: P3 NegativeTest NeedsReview
Scenario: Inviter gets connection request but invitee is offline for connection response
Given "Alice" and "Bob" do not have an existing connection
And "Alice" has generated a connection invitation
And "Bob" has received a connection invitation
And "Bob" sends a connection request
And "Bob" goes offline
And "Alice sends a connection response
When "Bob" comes online
And Bob resends a connection request
Then Alice and Bob can successfully complete the connection
Question: Connection Protocol again. What happens if one of the agents is offline during the invite or share phase? Ian mentioned that there may be 2 scenarios. 1. Agent might retry a few (3?) times. 2 There may be a "routing agent" in between. Assuming RFC for mediators and relays take up the slack here but that protocol is not included in AIP1.0. Is this another case where its up to the business to decide what happens here and not part of the AIP 1.0 Protocols RFCs?
Answer: For the mobile use case, there will be a mediator to handle the offline use case -- the sender never knows. All messages are async and no expectations are set on the time to reply at the RFC level. That would be a business question. Where it's an unexpected offline scenario, any sort of retry strategy would be up to the agent. At this point, I'm not aware of any agents that have gotten into that.

Scenario: Invitee sends connection response but inviter is offline
Scenario: Invitee sends new invitation after a preceding one
Question: Since the new invitation "supersedes" the old one, does that mean the old one will no longer work? Depends on the agent. up to implementation
**Answer:**Actually not sure you should worry about it. I think the language is clear enough that it might be up to the business. Its in the state table in the invitation column "No Change (resend or new invite that supersedes)"

tags: P3 DerivedFunctionalTest NeedsReview
Scenario: Establish a connection between two agents who already have a connection
Given we have two agents "Alice" and "Bob"
And "Alice" and "Bob" have an existing connection
When "Alice" generates a connection invitation
And they complete the connection process
Then Alice and Bob have another connection

tags: P4 DerivedFunctionalTest NeedsReview
Scenario: Establish a connection between two agents who already have a connection initiated from invitee
Given we have two agents "Alice" and "Bob"
And "Alice" and "Bob" have an existing connection
When "Bob" generates a connection invitation
And they complete the connection process
Then Alice and Bob have another connection

Question: Should I change the following 2 to an acknowledgement, trust ping isn't implemented? Maybe Move this to the ack suite?
Scenario: send a trust ping between two agents
Given "Alice" and "Bob" have an existing connection
When "Alice" sends a trust ping
Then "Bob" receives the trust ping

Maybe Move this to the ack suite?
Scenario: Send an acks between two agents that do not have a connection
Given "Alice" and "Bob" do not have an existing connection
When "Alice" sends a acks
Then "Alice" receives
Then "Bob" doesn't receive a response?

Question: Can we check the states of the Agents through the backchannel?
Answer: Yes we can.

tags: P2 AcceptanceTest NeedsReview
Scenario: Inviter sends an Invitation to Invitee
Given "Alice" and "Bob" do not have an existing connection
When "Alice" sends generates a connection Invitation
Then "Alice" is in the state of invited

tags: P2 AcceptanceTest NeedsReview
Scenario: Invitee receives an Invitation
Given "Alice" and "Bob" do not have an existing connection
And "Alice" has generated a connection invitation
When "Bob" receives the connection Invitation
Then "Bob" is in the state of invited

tags: P2 AcceptanceTest NeedsReview
Scenario: Invitee sends a connection request
Given "Alice" and "Bob" do not have an existing connection
And "Alice" has generated a connection invitation
And Bob has received a connection invitation
When "Bob" sends a connection request
Then "Bob" is in the state of requested
And "Alice" is in the state of invited

tags: P2 AcceptanceTest NeedsReview
Scenario: Inviter sends a connection response
Given "Alice" and "Bob" do not have an existing connection
And "Alice" has generated a connection invitation
And Bob has received a connection invitation
And Bob has sent a connection request
When "Alice" sends a connection response
Then "Alice" is in the state of responded
And "Bob" is in the state of complete or responded?

Error Conditions

The following tests may need to be split into Retryable and SingleTry. The states are different based on each. If retryable, the tests would need to capture how many times the agent will retry to run the test effectively.
tags: T007-API10-RFC0160 P2 ExceptionTest SingleTryOnException NeedsReview
Scenario: Establish a connection between two agents but gets a request not accepted report problem message
Given we have two agents "Alice" and "Bob"
And Bob has [problem]
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
Then Alice sends an request not accepted error
And the state of Alice is reset to Null?
And the state of Bob is reset to Null?

Examples:
| problem |
| Invalid DID Method |
| unknown endpoint protocols |
| ? |

Scenario: Retry to Establish a connection between two agents that previously had a request not accepted report problem message
Given we have two agents "Alice" and "Bob"
And a request not accepted was issued by Alice in a previous connection attempt
And Bob has fixed his [problem]
When "Bob" sends a connection request
Then they can complete the connection successfully

Examples:
| problem |
| Invalid DID Method |
| unknown endpoint protocols |
| ? |

Scenario: Establish a connection between two agents but gets a request processing error report problem message
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And Alice has a problem processing the connection request (What conditions?)
Then Alice sends a request processing error

Scenario: Establish a connection between two agents that had a request processing error report problem and Invitee retries
Given we have two agents "Alice" and "Bob"
And "Alice" has generated a connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
And Alice sent a request processing error
When Bob retries sending the connection request
Then they can complete the connection successfully

Scenario: Establish a connection between two agents but gets a response not accepted report problem message
Given we have two agents "Alice" and "Bob"
And Bob has a [problem]
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
Then Alice has sends a response not accepted report problem
And the report contains the reason for the [problem]

Examples:
| problem |
| not accepting the method of the provided DID |
| unknown endpoint protocols |
| invalid signature |
| ? |

Scenario: Retry to establish a connection between two agents that had a response not accepted report problem message
Given we have two agents "Alice" and "Bob"
And Bob has a [problem]
And "Alice" generated a connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
And Alice has sent a response not accepted report problem
When Bob corrects the [problem]
And retires the connection request
Then they can complete the connection successfully

Examples:
| problem |
| not accepting the method of the provided DID |
| unknown endpoint protocols |
| invalid signature |
| ? |

Scenario: Establish a connection between two agents but gets a response processing error report problem message
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And Alice sends the connection response
And Bob has a [problem] processing the connection response (What conditions?)
Then Bob sends a request processing error

Scenario: Retry to establish a connection between two agents that had a response processing error report problem message
Given we have two agents "Alice" and "Bob"
And "Alice" generated a connection invitation
And "Bob" received the connection invitation
And "Bob" sent a connection request
And Alice sent the connection response
And Bob had a [problem] processing the connection response (What conditions?)
And Bob sends a request processing error
When Alice retries the connection response
Then they can complete the connection successfully

Question How can the tests force an unknown error report problem?
Scenario: Establish a connection between two agents but gets a unknown error report problem message
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And Alice sends the connection response
And Bob has a [problem] processing the connection response (What problems?)
Then Bob sends an unknown error

Question Can either party retry the last communication to continue the connection process after an unknown error or does the process have to restart?
Scenario: Retry to establish a connection between two agents that had an unknown error report problem message

No errors are sent in timeout situations. If the inviter or invitee wishes to retract the messages they sent, they record so locally and return a request_not_accepted or response_not_accepted error when the other party sends a request or response .
Scenario: Established connection between two agents but Inviter retracts/terminates the connection
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Alice" retracts the invitation
When "Bob" sends a connection request
Then "Alice" sends a request_not_accepted error

Scenario: Established connection between two agents but Invitee retracts/terminates the connection
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request
And "Bob" terminates the connection
And "Alice" accepts the connection request by sending a connection response
Then "Bob" sends a response_not_accepted error

Scenario: Attempt to established connection between two agents where the Inviter has retracted the connection
Given we have two agents "Alice" and "Bob"
And Alice has [retracted the connection]
When "Bob" generates a connection invitation
Then "Alice" sends a request_not_accepted error

Examples:
| retracted the connection |
| sent a request not accepted error during a previous connection attempt |
| retracted the connection after a successful connection was made |

Scenario: Attempt to established connection between two agents where the Invitee has retracted the connection
Given we have two agents "Alice" and "Bob"
And Bob has [retracted the connection]
When "Alice" generates a connection invitation
Then "Bob" sends a request_not_accepted error

Examples:
| retracted the connection |
| sent a response not accepted error during a previous connection attempt |
| retracted the connection after a successful connection was made |

Negative Conditions

Scenario: Invitee sends unexpected message during share phase
Given we have two agents "Alice" and "Bob"
When "Alice" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends an [unexpected message]
Then "Alice sends What?
Question: can they continue the proper connection procedure after the unexpected message?

Examples:
| unexpected message |
| ack |

Scenario: Inviter sends unexpected message during share phase
Question: What happens if before Alice gets in the complete state Bob initiates a connection with an invitation in turn becoming an Inviter? Are all other previous messages and states ignored and the process starts over? It's a separate connection. Low priority.

References:
0160-connection-protocol from the Aries Interop Profile

• Get the Aries Test Harness Working

• Write All Gherkin Test Definitions

• Implement the Step Defs for the Tests

Test Harness Expansion to cover AIP 1.0 Protocol 0036-issue-credential (Happy Path Only)

AS A, I WANT, SO THAT

Acceptance Criteria

Feature: Aries agent issue credential functions RFC 0036

https://github.com/hyperledger/aries-rfcs/tree/bb42a6c35e0d5543718fb36dd099551ab192f7b0/features/0036-issue-credential

wip AcceptanceTest P1
Scenario: Issue a credential with the Holder beginning with a proposal
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
When “Bob” proposes a credential
And “Alice” offers a credential
And “Bob” requests the credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued

Question: Morning. Question on Issue Credential. In the RFC it mentions that the ack message is adopted into the protocol for confirmation. Not sure how to handle that message since currently ack is not implemented at least for the Aca-Py agent. Is it a different custom ack than the Acknowledgment protocol? Should this now be changed to trust ping or something?
Answer: ack is a specific ack for issue credential

wip AcceptanceTest P2
Scenario: Issue a credential with the Holder beginning with a proposal with negotiation
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
And “Bob” proposes a credential
And “Alice” offers a credential
When “Bob” negotiates the offer with another proposal of the credential
And “Alice” Offers the credential
And “Bob” requests the credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued

wip AcceptanceTest P1
Scenario: Issue a credential with the Issuer beginning with an offer
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
When “Alice” offers a credential
And “Bob” requests the credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued

wip AcceptanceTest P2
Scenario: Issue a credential with the Issuer beginning with an offer with negotiation
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
And “Alice” offers a credential
When “Bob” negotiates the offer with a proposal of the credential
And “Alice” Offers the credential
And “Bob” requests the credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued

wip AcceptanceTest P3
Scenario: Issue a credential with negotiation beginning from a credential request
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
When “Bob” requests the credential
And “Alice” offers credential
When “Bob” negotiates the offer with a proposal of the credential
And “Alice” offers credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued

wip AcceptanceTest P1
Scenario: Issue a credential with the Holder beginning with a request and is accepted
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
When “Bob” requests the credential
And “Alice” issues the credential
And “Bob” acknowledges the credential issue
Then “Bob” has the credential issued
Question:Should we expose the Preview Credential though the backchannel?
Answer:

Error Conditions

See Issue #35

Negative Conditions

See issue #35

References:
0036-issue-credential from the Aries Interop Profile

• Check with Ian on coverage

• Get experts in the protocol to identify and define the BDD test cases

• Determine and write any cases negative or otherwise not found by protocol experts

• Implement the test steps for the BDD Test Cases

• Define the backchannel API calls necessary to support the test functions.

• Ideally, include a documentation structure (perhaps Swagger) that will help agent builders to know how to build the backchannel controller for their agent.

• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py

Test Harness Expansion to cover AIP 1.0 Protocol 0036-issue-credential (Exception & Negative Tests)

This Story relates to issue #10

ExceptionTest wip P2
Scenario: Issue a credential with the Holder beginning with a request and is rejected

ExceptionTest wip P2
Scenario: Issuer offers a credential that the Holder is unwilling to accept
Given “2” agents
| name | role |
| Alice | Issuer |
| Bob | Holder |
And “Alice” and “Bob” have an existing connection
When “Alice” offers a credential to “Bob”
And “Bob” does not accept the offer of the credential
Then “Bob”? sends a problem-report message
And the issue credential flow is reset
And the flow is issuance-abandoned

ExceptionTest wip TestIsAnAgent P4
Scenario: Issuer server crash in the middle of the issue credential workflow
Then someone sends a problem-report message
And the issue credential flow is reset
And the flow is issuance-abandoned

ExceptionTest wip TestIsAnAgent P4
Scenario: Holder server crash in the middle of the issue credential workflow
Then someone sends a problem-report message
And the issue credential flow is reset
And the flow is issuance-abandoned

NegativeTest
Scenario: Role reversal tests

Scenario: Holders requests credential before the issuer offers a credential

Scenario: Issuer proposes a credential to Bob and Mallory attempts to request the credential

Scenario: Issuer issues a credential to Bob and Mallory attempts to acknowledge the credential

DerivedFunctionalTest P4
PaymentRequest (optional tag)
Scenario: Issuer Offers Credential with a Payment Request, Holder pays and proceeds
Scenario: Issuer Offers a Credential with a Payment Request, Holder attempts to proceed without payment

OfferExpires (optional tag)
Scenario: Issuer offers a credential with an expiration, holder proceeds within the expiration time
Scenario: Issuer offers a credential with an expirations, holder proceeds outside the expiration time

NoAck (optional tag)
Scenario: Issuer issues a credential without requiring an acknowledgement.

PresentProof (optional tag)
Scenario: Issuer issues a credential requiring the holder to present proofs

Test Harness Expansion to cover AIP 1.0 Protocol 0015-acks

References:
0015-acks from the Aries Interop Profile

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means its critical that we work to make our content as discoverable as possible; Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

That's in, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

• If your org is not in the list below, or the table contains errors, please create an issue here.

• While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

• Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

• If your application is live, add the production URL.

Ministry Short Codes

Short Code	Organization Name
AEST	Advanced Education, Skills & Training
AGRI	Agriculture
ALC	Agriculture Land Commission
AG	Attorney General
MCF	Children & Family Development
CITZ	Citizens' Services
DBC	Destination BC
EMBC	Emergency Management BC
EAO	Environmental Assessment Office
EDUC	Education
EMPR	Energy, Mines & Petroleum Resources
ENV	Environment & Climate Change Strategy
FIN	Finance
FLNR	Forests, Lands, Natural Resource Operations & Rural Development
HLTH	Health
FLNR	Indigenous Relations & Reconciliation
JEDC	Jobs, Economic Development & Competitiveness
LBR	Labour Policy & Legislation
LDB	BC Liquor Distribution Branch
MMHA	Mental Health & Addictions
MAH	Municipal Affairs & Housing
BCPC	Pension Corporation
PSA	Public Safety & Solicitor General & Emergency B.C.
SDPR	Social Development & Poverty Reduction
TCA	Tourism, Arts & Culture
TRAN	Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.

python backchannels differentiate between routes with and without trailing slash

Took me a while to figure out the requests to the python backchannels differentiate between routes with and without trailing slashes.

e.g. GET /agent/command/connection returns 404 not found
while GET /agent/command/connection/ returns 200 with all connections.

This seems like unexpected behaviour to me.

No such container: bob_agent

While I was following these steps in play-with-docker, I ran into this error:
Error: No such container: bob_agent

Add / update connection test to exercise OOB and DID Exchange protocol

Support for OOB and DID Exchange has been added to ACA-Py (PR in draft as I write this) and exists in Aries Framework Go. so we need some tests that exercise the protocols. They tests will parallel connecting using 0160 Connections, so could be variations of the same tests, or new. OOB overall has more capabilities (eclipsing the current way to do connectionless, for example), so ultimately there will be more, be more tests, but not initially.

@nodlesh @ianco -- this will be a collaborative effort, including supporting Clement in using it with Aries Go.

Reorder the output from the tests runs

Currently, the test run output is in the following order:

• Failing scenarios:
• Summary (features, scenarios and steps passing/failing)
• Unimplemented
• Cleanup:

It would be great to reorder that to:

• Unimplemented
• Failing scenarios:
• Summary (features, scenarios and steps passing/failing)
• Cleanup

That puts the most important information at bottom, and the user can scroll up to see the details if necessary. Leaving cleanup at the bottom ensures the user can see the results without waiting for all the containers to stop.

Reorganize the backchannels folder to put each agent and/or framework in their own folder

Currently all backchannels are in one folder. Put them in a folder per backchannel, optionally with multiple dockerfiles to support testing multiple versions of agents.

Plan is to have:

aries-backchannels
common
vcx
acapy - docker file and py
...

In the dockerfile, COPY the current folder and the contents of ../common into a common folder.

Check that the GHAs are running on Hyperledger organization and don't run if not

Today, if a person forks the AATH repo, they GHActions run on their fork, but they shouldn't. They can turn off the running of GHA, but they shouldn't have to do that. Please add something (not sure where) to make sure that by default the jobs ONLY run from the hyperledger organization repo of AATH.

Bonus point if there is a way that the fork can activate the running of the tests.

It may be that the tests have to run within a fork, but the tests could have a first line that detects the test is not running on Hyperledger account and exits. The status could be "Pass", and it would quietly run everyday, or "Fail" and given instructions on how to block running the tests on the fork.

An alternative to all this would be to have a second repo that has the GHAs in it, and it just uses AATH to get the tests to be run. That would mean that every fork of the AATH repo wouldn't have any GHAs and so wouldn't try to run them.

Open to ideas, but I think we need to fix this sooner than later.

Positive (Happy Path) Test Coverage of Credential Revocation

AS AN Issuer, I WANT to be able to revoke an issued credential, SO THAT holders can't continue to prove claims from the credential if it was replaced by an updated version, issued in error, or the holder did something "bad" to warrant they no longer qualify to hold said credential.

Acceptance Criteria

Scenario: Credential revoked by Issuer and Holder attempts to prove
Scenario: Credential revoked and replaced with a new updated credential, holder proves claims with the updated credential
Scenario (Neg): Proof in process while Issuer revokes credential before presentation
Scenario (Neg): Credential revoked and replaced with a new updated credential, holder proves claims with the updated credential but presents the revoked credential

References & Source Documents

0011 Credential Revocation
Revocation Registry Handling in ACA-Py
RFC 0183 Revocation Notification 1.0
Implement revocation registry support for credential issuance
ACA-Py Admin endpoint for revoking a credential
Evolve the handling of revocation registries into ACA-Py and away from the controller
Revoked credential
Question: how revocation works?

Action Items

• Get experts in the protocol and research references to identify and define the BDD test cases and document them in the Acceptance Criteria and feature file
• Implement the test steps for the BDD Test Cases
• Define the backchannel API calls necessary to support the test functions.
• Implement capabilities in the ACA-Py backchannel controller to enable execution of the tests where both agents are ACA-Py
• Document Coverage in the Test Coverage Matrix

Categorized Test Cases to AIP Verisons with Links on AIP RFC

AS AN Agent Builder, I WANT to know what test cases are for what part of the Aries Interop Profile RFCs, SO THAT I know what test cases to execute based on the implemented RFCs

Acceptance Criteria

Scenario: Agent Builder needs to know what tests to run for the AIP version they are working on
Given an Agent Builder who is looking at an AIP version
When they look at an AIP version in the AIP RFC
Then they will see a link to the relevant subset of Aries test cases in the Aires Test Suite

Scenario: Aries Test Harness Owner maintains links from AIP RFC version to relevant Test Cases
Given the Aries Test Harness Owner
When they have made a change to the Aries Test Suite
Then they will tag the test scenarios in the feature file with the appropriate RFC version
And they will update the links in the AIP RFC to the relevant Test Cases

Doubt with report generation

Hi

I'm trying to generate the reports with the command:

./send_results.sh ./allure-results http://localhost:5050 -p acapy

But the execution of the tests is not shown. Am I making a mistake?