dongmingh/v1performance

v1performance's Issues

PTE Driver nThread Bottleneck/Incompletion. Inability to Calculate Accurate TPS for Higher nThread Runs

BC Topology: 1 Org, 1 Peer, 1 Ca, 1 Ord on bcfab2 w/ 10 IFLs, 48Gb Memory
While attempting to drive the above topology to max TPS, I continously encountered an issue with driving the peer to max TPS or CPU utilization. I tried multiple combinations of user input parameters (nThreads, constFreq, devFreq) to achieve the highest TPS possible. Although, I noticed using an "nThread" value higher than 20 threads produced the error:

"stderr: (node:36570) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3129): undefined" For every thread running.

This error indicated that the threads were not able to complete and therefore not able to produce the "eventRegister" message necessary for calculating TPS. In terms of driving the SUT, using more than 20 threads would still allow for proper container statistics and sampler data collection, but calculating the TPS to go along with it was not possible. For example, using 50+ threads would drive the SUT to ~ 50% CPU utilization, about the highest I was able to achieve. But only when nThread was < 20 was I able to get an accurate TPS. There are a few workarounds to patching this problem but it is very inconvenient that the PTE driver cannot produce an accurate TPS for more strenuous runs. If there are any more questions about the issue/error I am seeing, don't hesitate to contact me on sametime or email me at [email protected], and thanks for the help!

fabric-ca commit level not in working tree

vmorris@vem-bcaas-driver-1:~/go/src/github.com/hyperledger/fabric-ca$ git reset --hard git reset --hard 29385879bc2931cce9ec833acf796129908b72fb
fatal: ambiguous argument 'git': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Please update fabric-ca commit level as appropriate. I think 4b58d5c838266c3e4898bb22ab61b66ca27d8ee4 is okay.

get error when run testcases for sdk=node userInputs/samplecc-chan1-join1.json

error: [Orderer.js]: sendDeliver - rejecting - status:FORBIDDEN
error: [Chain.js]: joinChannel - Failed Proposal. Error: Error: Invalid results returned ::FORBIDDEN
at ClientDuplexStream. (/home/swangbj/gopath/src/github.com/hyperledger/fabric-sdk-node/node_modules/fabric-client/lib/Orderer.js:197:22)
at emitOne (events.js:96:13)
at ClientDuplexStream.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)
at ClientDuplexStream.Readable.push (_stream_readable.js:134:10)
at readCallback (/home/swangbj/gopath/src/github.com/hyperledger/fabric-sdk-node/node_modules/grpc/src/node/src/client.js:245:14)
Error: Invalid results returned ::FORBIDDEN
at ClientDuplexStream. (/home/swangbj/gopath/src/github.com/hyperledger/fabric-sdk-node/node_modules/fabric-client/lib/Orderer.js:197:22)
at emitOne (events.js:96:13)
at ClientDuplexStream.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)
at ClientDuplexStream.Readable.push (_stream_readable.js:134:10)
at readCallback (/home/swangbj/gopath/src/github.com/hyperledger/fabric-sdk-node/node_modules/grpc/src/node/src/client.js:245:14)

Requirement - Request ability to utilize multiple clients(users) per thread.

• Competitors publish gee whiz marketing #s, making quotes like " ... ability to support 40,000 clients...". Could we provide multiple client id capability to compete with such statements.

Got Error "(Timeout expired while executing transaction)" when I instantiate chaincode

When I instantiate a new chaincode , I got errors like following .

And from my peer-log ,I see errors like this .

Lost Proc without error

PTE commit 3c03c12
Fabric/CA/SDK 1.0-GA

Network: There are 4 peers in 2 orgs and 1 order in sole mode.
Testing:
I use constant mode to invoke
"constantOpt": {
"recHist": "HIST",
"constFreq": "0",
"devFreq": "0"
},

"nOrderer": "1",
"nPeer": "4",
"nOrg": "2",
"nPeerPerOrg": "2",
"nProc": "8",
"nRequest": "1000",
"runDur": "10000",
"TLS": "disabled",
"channelOpt": {
    "name": "mychannel1",
    "action":  "create",
    "orgName": [
        "org1",
        "org2"
    ]
},
"listOpt": {
    "org1": ["peer1","peer2"],
    "org2": ["peer1","peer2"]
},

Problems:
I will have 16 proc to send transaction to the network. But only get less than 16 result.

info: [PTE 0 main]: stdout: info: [PTE 0 exec]: [Nid:chan:org:id=0:mychannel1:org1:1 eventRegister] completed Rcvd(sent)=1000(1000) Invoke(Move) in 34494 ms, timestamp: start 1500960720702 end 1500960755196, #event timeout: 0

Two PTE log:
result3.txt
result4.txt

Density -- KeyValueStore

Currently, KeyValueStore dir is: /tmp/hfc-test-kvs+'_'+org. Could you add network filter to differentiate different network?

Report Timed out problem when send transactions to network

1. PTE level: commit a15e57f
2. Fabric level 1.0GA

Problems:
I create/join/install/instaniate network.
Then I use runCases-constant-i.zvm to send request.

PTE report following problems:

info: [PTE 0 main]: stderr: error: [Peer.js]: sendProposal - timed out after:45000

info: [PTE 0 main]: stderr: error: [Peer.js]: sendProposal - timed out after:45000
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: REQUEST_TIMEOUT
at Timeout._onTimeout (/root/gopath/src/github.com/hyperledger/fabric-sdk-node/node_modules/fabric-client/lib/Peer.js:100:19)
at ontimeout (timers.js:365:14)
at tryOnTimeout (timers.js:237:5)

Requirement - Ability to simulate multiple levels of chaincode resource consumption

• There's currently very limited customer-representative chaincode that we conduct performance tests with. Hence, it would be valuable to at least simulate multiple levels of chaincode resource consumption (processor and memory) in our testing. Could you provide a means to vary chaincode processing and memory requirements for us to test with. Perhaps add 2 additional ccOpts cpuIntensity, memIntensity, and allow a integer setting to set the degree of intensity (or demand). Perhaps provide an example, so tester can put into perspective.

fabric commit level f9e40c3c0070303285b4249112c966969551d9056 not in working tree.

vmorris@vem-bcaas-driver-1:~/go/src/github.com/hyperledger/fabric$ git reset --hard f9e40c3c0070303285b4249112c966969551d9056
fatal: ambiguous argument 'f9e40c3c0070303285b4249112c966969551d9056': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Please update README.md with appropriate commit level. I think a97886a026970a4da3a5582ccb6411765d0717bc is okay..

Attempting to run samplecc-chan1-create.json results in Error

After testUtil.getSubmitter returns, error is:
Failed to enroll user 'admin'. TypeError: Cannot read property 'then' of null

Indeed, admin referenced on https://github.com/dongmingh/v1performance/blob/master/pte-main.js#L664 doesn't seem to exist!

Full log:

ubuntu@vem-workload-1:~/go/src/github.com/hyperledger/fabric-sdk-node/test/v1performance$ ./pte_driver.sh userInputs/runCases-chan-create.txt 
sdk type spported: sdk=node
sdk type unknown: #sdk=node
Node Array: userInputs/samplecc-chan1-create.json, userInputs/samplecc-chan1-create.json
executing 1 node requests
nNetwork: 1, tStart: 1493666711343
execution: userInputs/samplecc-chan1-create.json
no python requests
no java requests
info: Returning a new winston logger with default configurations
GOPATH:  /home/ubuntu/go
input parameters: Nid=0, uiFile=userInputs/samplecc-chan1-create.json, tStart=1493666711343
[chaincodeInstall:Nid=0] chaincode_id: sample_cc_ch1, chaincode_ver: v0, chain_id: testchainid_ch1
TLS: DISABLED
channelName: testChannel1
channelOrgName.length: 1, channelOrgName: testOrg1
svcFile;  SCFiles/config-chan1.json
[performance_main]: org=testOrg1, org Name= PeerOrg1
[createOneChannel] org= testOrg1, org name=PeerOrg1
[createOneChannel] user= admin, secret=REDACTED
[2017-05-01 15:24:57.919] [DEBUG] E2E - [FileKeyValueStore.js]: FileKeyValueStore.js - constructor
[getSubmitter] ca:  http://REDACTED:45011
Failed to enroll user 'admin'. TypeError: Cannot read property 'then' of null

eventOpt register option "None" Still Showing that its listening for Transactions

While running with eventOpt option "None" PTE still had eventRegisters listening for transactions and we were able to produce a TPS from them. If I am correct, the "None" option should not show any transactions because it should be ignoring them after sending it. Also when running with "None" option, the TPS acquired from the peerlog showed a reduction in TPS versus eventOpt options "Transaction" and "Block". The incoming TPS was fixed at 240, "Transaction"/"Block" had a TPS of 238, while "None" produced a TPS of 177. If you need access to any files please let me know at [email protected], thanks.

PTE result formation

We got different message from PTE info:
info: [PTE main]: stdout: info: [PTE exec]: [Nid:chan:org:id=0:mychannel1:org2:0 invoke_query_simple] completed 100 Invoke(Move) in 4719 ms, timestamp: start 1500268105623 end 1500268110342
info: [PTE exec]: [Nid:chan:org:id=0:mychannel1:org2:5 invoke_query_simple] completed 100 Invoke(Move) in 4726 ms, timestamp: start 1500268105617 end 1500268110343

Would you format to make a same info for result? So we can calculate the performance easy. Thanks.

Requirement - Request 'PTE Completed' message in driver output

• Could you write 'PTE Complete' message to output once all threads / sub-threads have ended for a particular run. Additionally post-measurement data collection scripts can key off this message.

Ability for PTE to handle failover

We need to test a scenario where PTE automatically sends transactions to a backup peer if a peer becomes available. I don't believe this capability is currently available. Is this something that could be implemented?

samplecc-chan1-latency-i-TLS.jsont hangs

samplecc-chan1-latency-i-TLS.json seems to just hang.

There are no messages after tStart and no timeouts; I do see that the node processes are running though the peers are idle.

Create functionality and instructions for running PTE against both HSBN and HSBN-like (zblockchain appliance) networks.

This is an enhancement and usability request.

We have a need to run PTE against HSBN and HSBN-like networks. The HSBN interfaces on Bluemix provide capability to create channels, join peers, and install chaincode, however there is also a need to be able to do this from PTE given some Fabric network configuration and credentials.

For example, the Fabric network config and credentials we current get from HSBN vNext appear as such:
https://gist.github.ibm.com/vmorris/1a73cad83c0f2dd7cd34676cbaf8db1c

Conversely, the Fabric network config and credentials we currently get from HSBN-like networks (as created by IBM-Blockchain/v1networkgen), appear as such:
https://gist.github.ibm.com/vmorris/51552c1da774b3ee6c2a4773cd598708

Density -- PID: Network relationship

PIDs start from 0 and only increase in 1 Org scope, not network scope. Could go with: Netid, OrgId, PID.