diem/client-sdk-python

Python client library for Diem

Note to readers: Silvergate Capital Corporation announced in January 2022 that it acquired intellectual property and other technology assets related to running a blockchain-based payment network from Diem, further investing in its platform and enhancing its existing stablecoin infrastructure.

---

Diem Core implements a decentralized, programmable distributed ledger which provides a financial infrastructure that can empower billions of people.

Note to Developers

• Diem Core is a prototype.
• The APIs are constantly evolving and designed to demonstrate types of functionality. Expect substantial changes before the release.
• We’ve launched a testnet that is a live demonstration of an early prototype of the Diem Blockchain software.

Quick Links

Learn About Diem

• Diem Protocol: Key Concepts
• Life of a Transaction
• JSON-RPC API Specification
• JSON-RPC SPEC

Getting Started - Try Diem Core

• My First Transaction
• Getting Started With Move

Technical Papers

• The Diem Blockchain
• Move: A Language With Programmable Resources
• State Machine Replication in the Diem Blockchain

Governance

• Diem Improvement Proposals

Blog

• Diem: The Path Forward

License

Diem Core is licensed as Apache 2.0.

client-sdk-python's Issues

[python] Support reading transactions from testnet

rough API

class Transaction:
    def events() -> list(EventHandle): ...

LibraNetwork().transactions_by_acc_seq(account, seq, include_events) -> list(Transaction) : ...
LibraNetwork().transactions(start_version, limit, include_events) -> list(Transaction): ...

JSONSignedTransaction's properties public_key and signature are of type str instead of bytes.

https://fburl.com/295ne0pu
some output digest:

@@@@@@@@@@ queried txn type  <class 'pylibra._jsonrpc_transport.JSONSignedTransaction'>
@@@@@@@@@@ submitted txn type  <class 'pylibra._native.NativeSignedTransaction'>
@@@@@@@@@@ queried key type  <class 'str'>
@@@@@@@@@@ submitted key type  <class 'bytes'>

parameterized test cases

In order to test language specific wrappers are correctly implemented. we should generate test case params(blob and all value of fields) in LIbra-dev, and use it to test cpp, rust, and python code.

Opensource entire repo

work on application to open source this repo

Documentation for apis

create nightly test to verify we work with current testnet

Add User-Agent HTTP header to requests made by the jsonrpc client

https://github.com/diem/client-sdks/blob/master/specs/json_rpc_client.md#user-agent-http-header

json-rpc clients should set the User-Agent HTTP header with sufficient information for a server to be able to identify which sdk client the request was sent from, e.g. diem-client-sdk-python / x.y.z.

Need figure out how to get sdk version for the this header from current setup.py file.

Document offchain module and it's submodules

[ ] offchain module summary
[ ] diem.offchain.action
[ ] diem.offchain.client
[ ] diem.offchain.command
[ ] diem.offchain.error
[ ] diem.offchain.http_header
[ ] diem.offchain.http_server
[ ] diem.offchain.jws
[ ] diem.offchain.payment_command
[ ] diem.offchain.payment_state
[ ] diem.offchain.state
[ ] diem.offchain.types

can the wallet sign any smart contract with a private key?

I assume this wallet can send payments. My question is:
can the wallet sign any smart contract with a private key?

Is there any javascript version for the wallet?

George Wu

Support asyncio

Async client has significant better performance and scalability.
It will become harder and harder to switch to async IO when more features are built.
We may keep current blocking client using requests package for backward compatible.

• async jsonrpc client and faucet client
• async miniwallet app
• async miniwallet client and testing suites
• dmw cli support async
• update example code to use async clients, or remove examples that are not useful
• update doc and refer to mini-wallet app for some examples.
• Delete offchain http_server.py, which is a blocking server

INVALID_SIGNATURE on My First Client code

I'm executing the code from My First Client Documentation adapting it to the new coin names (replacing libra with diem and Coin1 with XUS).

This is the code:

import time

from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from diem import stdlib, utils, diem_types, testnet, AuthKey
from diem.testnet import CHAIN_ID

CURRENCY = "XUS"

"""
submit_peer_to_peer_transaction_example demonstrates currencies transfer between 2 accounts on the diem blockchain
"""


def main():
    # connect to testnet
    client = testnet.create_client()

    # generate private key for sender account
    sender_private_key = Ed25519PrivateKey.generate()
    # generate auth key for sender account
    sender_auth_key = AuthKey.from_public_key(sender_private_key.public_key())
    print(f"Generated sender address: {utils.account_address_hex(sender_auth_key.account_address())}")
    # create sender account
    faucet = testnet.Faucet(client)
    faucet.mint(sender_auth_key.hex(), 100000000, CURRENCY)

    # get sender account
    sender_account = client.get_account(sender_auth_key.account_address())

    # generate private key for receiver account
    receiver_private_key = Ed25519PrivateKey.generate()
    # generate auth key for receiver account
    receiver_auth_key = AuthKey.from_public_key(receiver_private_key.public_key())
    print(f"Generated receiver address: {utils.account_address_hex(receiver_auth_key.account_address())}")

    # create receiver account
    faucet = testnet.Faucet(client)
    testnet.Faucet.mint(faucet, receiver_auth_key.hex(), 10000000, CURRENCY)

    # create script
    script = stdlib.encode_peer_to_peer_with_metadata_script(
        currency=utils.currency_code(CURRENCY),
        payee=receiver_auth_key.account_address(),
        amount=10000000,
        metadata=b'',  # no requirement for metadata and metadata signature
        metadata_signature=b'',
    )
    # create transaction
    raw_transaction = diem_types.RawTransaction(
        sender=sender_auth_key.account_address(),
        sequence_number=sender_account.sequence_number,
        payload=diem_types.TransactionPayload__Script(script),
        max_gas_amount=1_000_000,
        gas_unit_price=0,
        gas_currency_code=CURRENCY,
        expiration_timestamp_secs=int(time.time()) + 30,
        chain_id=CHAIN_ID,
    )
    # sign transaction
    signature = sender_private_key.sign(utils.raw_transaction_signing_msg(raw_transaction))
    public_key_bytes = utils.public_key_bytes(sender_private_key.public_key())

    signed_txn = utils.create_signed_transaction(raw_transaction, public_key_bytes, signature)
    # submit transaction
    client.submit(signed_txn)
    # wait for transaction
    client.wait_for_transaction(signed_txn)


if __name__ == "__main__":
    main()

The code is giving me a SignatureError with the following Traceback

Traceback (most recent call last):
  File "<input>", line 71, in <module>
  File "<input>", line 65, in main
  File "C:\Users\username\git\project\diem\venv\lib\site-packages\diem\jsonrpc\client.py", line 398, in submit
    return self.submit(txn.lcs_serialize().hex())
  File "C:\Users\username\git\project\diem\venv\lib\site-packages\diem\jsonrpc\client.py", line 400, in submit
    self.execute_without_retry("submit", [txn], result_parser=None, ignore_stale_response=not raise_stale_response)
  File "C:\Users\username\git\project\diem\venv\lib\site-packages\diem\jsonrpc\client.py", line 526, in execute_without_retry
    raise JsonRpcError(f"{err}")
diem.jsonrpc.client.JsonRpcError: {'code': -32001, 'message': 'Server error: VM Validation error: INVALID_SIGNATURE', 'data': {'StatusCode': 1}}

Error handling by returning status code

https://github.com/calibra/libra-client-dev/pull/9/files#r348240470

https://github.com/calibra/libra-client-dev/pull/9/files#r348240384

use GoogleTest in CPP client to write tests

Ideally with sanitizer enabled.

publish python package

publish python package to PIP so pubsub daemon can start to use it.

The testnet module does not fully support the local testnet

In the testnet module _mint_without_retry method, the url should use self._url when post

Faucet minting copied from Documentation throws TypeError

I'm building an application on Diem testnet using the Python SDK.
I'm following the "my first client" documentation on developers.diem.com and I'm trying to execute the copy-paste of the code to generate an address, minting some coin to that address and then getting the balance. This is the complete code (copy-paste of the docs + import)

from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from diem import jsonrpc, testnet, AuthKey, utils
from diem.testnet import Faucet

#connect to testnet
client = testnet.create_client()

#generate private key
private_key = Ed25519PrivateKey.generate()

#generate auth key
auth_key = AuthKey.from_public_key(private_key.public_key())
print(f"Generated address: {utils.account_address_hex(auth_key.account_address())}")

#create account
faucet = testnet.Faucet(client)
testnet.Faucet.mint(faucet, auth_key.hex(), 1340000000, "Coin1")

#get account information
account = client.get_account(auth_key.account_address())
print("Account info:")
print(account)

The code fails at the

testnet.Faucet.mint(faucet, auth_key.hex(), 1340000000, "Coin1")

With the following traceback:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\testnet.py", line 66, in mint
    self._retry.execute(lambda: self._mint_without_retry(authkey, amount, currency_code))
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 86, in execute
    raise e
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 80, in execute
    return fn()
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\testnet.py", line 66, in <lambda>
    self._retry.execute(lambda: self._mint_without_retry(authkey, amount, currency_code))
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\testnet.py", line 85, in _mint_without_retry
    self._client.wait_for_transaction(txn)
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 425, in wait_for_transaction
    return self.wait_for_transaction2(
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 460, in wait_for_transaction2
    txn = self.get_account_transaction(address, seq, True)
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 320, in get_account_transaction
    return self.execute("get_account_transaction", params, _parse_obj(lambda: rpc.Transaction()))
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 491, in execute
    return self._retry.execute(
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 80, in execute
    return fn()
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 492, in <lambda>
    lambda: self.execute_without_retry(method, params, result_parser, ignore_stale_response)
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 523, in execute_without_retry
    json = self._rs.send_request(self, request, ignore_stale_response or False)
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 98, in send_request
    return client._send_http_request(client._url, request, ignore_stale_response)
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 554, in _send_http_request
    self.update_last_known_state(
  File "C:\Users\username\git\project\venv\lib\site-packages\diem\jsonrpc\client.py", line 262, in update_last_known_state
    if curr.version > version:
TypeError: '>' not supported between instances of 'int' and 'NoneType'

Add Multisignature to Python Library

See: https://github.com/diem/diem/tree/master/specifications/crypto#multi-signatures

Add Verification to Python Library

See: https://github.com/diem/client-sdks/blob/master/specs/verifying_client.md

API to craft raw transaction

Check binary size for client library

BCS Standalone Package?

Is the serialization/deserialization available as a standalone package?

I'm writing a SUI python SDK and recent changes require BCS serialization/deserialization.

naming convention for C API

We need to have an consistent naming scheme for all struct, functions etc.

I propose

namespace libra {
struct LibraAccountResource {
}

struct LibraAccountResource libra_account_resource_create(...);

}

In C++ we have namespace , but in C we should prefix all structure and api names. Also, we should avoid mentioning internal implementation details like "LCS" if we don't want user to worry about that .

Offchain API (DIP-1) support

Basics

• Data types
• JWS encoding and decoding
• Support both DD and VASP account
• A client for off-chain API HTTP calls.
• Deserialize json object into types, looking up object type by "_ObjectType" field for dynamic type fields (request.command)

Object Data Validations

• invalid_json: decoded JWS body is not json.
• invalid_object: command request object json is not object.
• invalid_overwrite
  • Overwrite write once fields: field can be written once anytime.
  • Overwrite immutable field (original_payment_reference_id): field can be none, but immutable.
• missing_field: missing required field
• unknown_field: field is unknown for an object
• unknown_command_type: invalid/unsupported command_type
• invalid_field_value:
  • invalid enum field values
  • data type mismatch, e.g. payment.sender.status should be object, but given string.
  • payment.{actor}.metadata should be a list of string, but contains int.
  • invalid uuid:
    • reference_id
    • cid
  • invalid actor address: actor's address is not a valid DIP-5 account identifier with expected network hrp.
  • value is not a valid Diem currency code for the connected network.

Business Logic Validations

• invalid_command_producer: Payment object state can only be triggered by one specific actor
• invalid_transition: Payment object state transition: a state can be transited from limited state. For example: VASP A sends RSOFT to VASP B, VASP B should send next payment object with ABORT, or SSOFTSEND, but it sends payment object SSOFT.
  • Can't update after settled / aborted. use a specific error code?
• invalid_initial_or_prior_not_found: could not find command by reference_id for a non-initial state command.
• invalid_overwrite
  • Overwrite opponent actor's fields.
• missing_field: an optional field is required to be set for a specific state, e.g. PaymentObject requires sender's kyc_data (which is an optional field for PaymentActorObject) when sender init the PaymentObject.
• no_kyc_needed: amount is under travel rule limit.
  • Use ceil of the exchanged xdx amount for checking under the limit. As it involves float point converting, relax this validation to avoid rounding difference comparing with the Diem Move VM validation logic.
• invalid_recipient_signature: recipient_signature is invalid, validated when sender received it from receiver
• unknown_address:
  • The DIP-5 account identifier address in command object is not HTTP request sender’s address or receiver’s address. For payment object it is sender#address or receiver#address.
  • Could not find on-chain account by an DIP-5 account identifier address in command object address (this is not the case for the payment object because the sender / receiver address is either HTTP request sender’s address which is validated when you validate HTTP header X-REQUEST-SENDER-ADDRESS or the HTTP request receiver’s address which must exist otherwise it is HTTP request receiver’s internal error)
    • invalid_address is validated before validating actor address, this is not required anymore.
• [-] invalid_original_payment_reference_id: could not find object by the original_payment_reference_id, or the status of the object is invalid (e.g. requires ready for settlement, but got aborted)
  • included in the wallet example

JWS Validations

• invalid_jws: invalid JWS format (compact) or protected header
• invalid_jws_signature: JWS signature

HTTP Header Validation

• invalid_http_header:
  • X-REQUEST-SENDER-ADDRESS must be same with one of payment actor#address
  • Could not find compliance key by the address:
    • A parent VASP or DD account has no valid compliance key set.
    • It is not a dd, child or parent VASP account
• missing_http_header
  • X-REQUEST-SENDER-ADDRESS

The following validations won't be covered in SDK

• missing_http_header for X-REQUEST-ID
• invalid_uuid for X-REQUEST-ID

VASP wallet example

• Generate payment intent identifier.
• Initial payment command for a given payment intent identifier.
• Process inbound request command.
• Background tasks for processing inbound command.
• Retry on outbound command request.
• Evaluate KYC data, reject / accept.
• Soft match and clear soft match.
• Submit on-chain transaction after payment object is synced.
• conflict error:
  • [-] command object is conflict with another one by cid, likely a cid is reused for different command object.
    • cid is useful for debugging, but it seems not required to be uniq for any real reason unless we need save or query the command object by the id. current example wallet implementation does not require it, hence won't implement it.
  • command object is conflict with another processing in progress by same reference_id (as we shall lock payment object before update).
• Payment with original_payment_reference_id and other optional fields: metadata and description
• HTTP headers' validations
• unsupported-currency: payment.action.currency value is a valid Diem currency code, but it is not supported / acceptable by receiver.

[C-api] Support more transaction types

as seen here:

/// Returns a user friendly mnemonic for the transaction type if the transaction is
/// for a known, white listed, transaction.
pub fn get_transaction_name(code: &[u8]) -> String {
    if code == &ADD_VALIDATOR_TXN[..] {
        return "add_validator_transaction".to_string();
    } else if code == &PEER_TO_PEER_TXN[..] {
        return "peer_to_peer_transaction".to_string();
    } else if code == &PEER_TO_PEER_WITH_METADATA_TXN[..] {
        return "peer_to_peer_with_metadata_transaction".to_string();
    } else if code == &CREATE_ACCOUNT_TXN[..] {
        return "create_account_transaction".to_string();
    } else if code == &MINT_TXN[..] {
        return "mint_transaction".to_string();
    } else if code == &REMOVE_VALIDATOR_TXN[..] {
        return "remove_validator_transaction".to_string();
    } else if code == &ROTATE_AUTHENTICATION_KEY_TXN[..] {
        return "rotate_authentication_key_transaction".to_string();
    } else if code == &ROTATE_CONSENSUS_PUBKEY_TXN[..] {
        return "rotate_consensus_pubkey_transaction".to_string();
    }
    "<unknown transaction>".to_string()
}

currently only p2p (no metatdata) and mint transactions are supported in C

setup bors

[python] utilize new LibraAccountKey() stuff in tests and expose it as API

1. derive pubkey/address
2. generate random private key

ResourceWarning: unclosed socket from session

There's warning about resources unclosed, see screenshot below

I think it's because the session here is not released https://github.com/calibra/libra-client-dev/blob/master/python/src/pylibra/_jsonrpc_transport.py#L276

Off-chain v3 JSON-RPC API reference implementation

• content detached jws encode, decode
• JsonRpc server
• Auth by content detached JWS
• Map JsonRpc error to http error
• JsonRpc client
• get_supported_methods method
• create_payment method
• get_account_identification method
• get_payment_signature method
• approve_payment method
• get_payment_status method
• cancel_payment method
• get_certificate_fingerprints method
• ping method
• sender_initiated_diem_id_exchange
• server process request idempotency

create dev branch so we can track dev release more closely

There hasn't been an testnet release for an month , we are accumulating too much changes that we need to catchup once it is released, it is best if we could have an branch that track dev branch , so we can migrate changes easier.

Add copyright and license header to all code files

Apply the following information at top of each code file, adjust as necessary for different programming language.

# Copyright (c) The Libra Core Contributors
# SPDX-License-Identifier: Apache-2.0

[pylibra] Handle account without any state on chain properly

Currently pylibra (not sure if it's the same for other language lib) will throw error for such account, a more proper way might be return 0 balance/sequence instead of throwing exception.

Diem ID

Diem Core

• Payment Metadata: diem/diem#8335
• Create string type of 64 chars: diem/diem#8362
• JSON RPC: "diem ID domains" field under parent VASP diem/diem#8363
• JSON RPC: get_account fetches diem ID domain diem/diem#8363
• JSON RPC: add EventDataView diem/diem#8434
• Faucet: add TC script to add domain ID into faucet diem/diem#8363

Move

• Move: Add types diemIDdomain, diemIDdomainevent: diem/diem#8336
• Move: DiemAccount publishes diemIdDomain resource using create_parent_vasp_account: diem/diem#8336
• Move: TC publishes diemIdDomainEventManager resource using create_treasury_compliance_account: diem/diem#8336
• Move: TC add/remove diemIdDomain in diemIdDomains: diem/diem#8336
• Move: Event for add/remove of diemIdDomain emitted and saved to DomainEventManager: diem/diem#8336
• Move: Script in AccountAdministrationScripts to add diemIdDomain resource to existing parent vasps: diem/diem#8336

build: cargo run
tests: cargo test
cargo x test -p move-lang-functional-tests diem_id_domain.move

Offchain

• Add reference ID exchange command objects: #274
• create and send CommandRequestObject: #283
• Update transaction object with reference ID when ref exchange is successful: #309
• Start txn with diemID, need to check for diemID vs subaddress and send transaction accordingly: #309
• Use Paymentmetadata for onchain transaction: #306
• handle error cases i.e. duplicate ref Id: #309

Mini wallet

• Set up mini wallet testing framework for diem ID: #306
• Send p2p transaction using DiemID as payee: #309
• Receive p2p transaction using event puller: #309
• Fetch from TC event stream and create domain map: #309
• Test: test_reference_id_command for e2e: #309

AOS

• Implement adding domain via AOS UI

Renaming

• #322

API to sign raw transaction bytes

[python] Support deserializing events

Documentation link is broken

It seems the python sdk docs link is broken:

https://developers.diem.com/docs/python-client-sdk-docs

Build test fixture base on genesis data

Base on conversation with Sam Blackshear

0xA550C18 is an fixture address that will survive an rewipe, we should build test around that to verify we work with testnet.

here's the code that creates it: https://github.com/libra/libra/blob/master/language/vm/vm-genesis/src/lib.rs#L233

Off-chain payment state should allow transition from S_SOFT / R_SOFT to R_ABORT / S_ABORT

Actor should be able to abort at any time it can update the command.

[core] Support client mnemonic generation

1. Generating an random wallet mnemonic, with private key
2. Restore private key from mnemonic string

related logics are here
https://github.com/libra/libra/blob/master/client/libra_wallet/src/mnemonic.rs

make an libra_dev_sys rust crate

Rust client should use libra_dev_sys crate , which builds lilibra_dev.so files with C-API.

[core] Support deserializing other type of transcations

As per offline discussion, we need to have an extensible serialization mechanism that can be extended to other type of transactions , a couple of them shown here:

At least, mint, transfer and rotate_key etc should be supported, they are all user-transactions with different payload.

We can consider ignore block_metatdata and writeset txns, if it is not related to the user.