pelioniot/manifest-tool

A tool for creating and parsing update manifests

manifest-tool's Issues

Error when aborting an active campaign

When I try to ctrl-C during a manifest-tool update device, it attempts to delete a campaign, but the campaign needs to be stopped before it can be deleted. The following errors are generated:

^C[CRITICAL] 2019-02-12 17:09:56 - manifesttool.update_device - User Aborted... Cleaning up.
[INFO] 2019-02-12 17:09:56 - manifesttool.update_device - ** Deleting update campaign and manifest **
[CRITICAL] 2019-02-12 17:09:56 - manifesttool.update_device - Cleanup of campaign failed with:
(409) "Conflict" (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Content-Length': '162', 'Access-Control-Expose-Headers': 'X-Request-ID,Content-Length,Content-Disposition', 'Set-Cookie': 'TS017f188d=01c6170234c1888f130abfb28e4615196173f8134fd5bc00a456775fce2821bc566ff513e4119c74a203b52c03b876fefee1d3bdeb; Path=/', 'Strict-Transport-Security': 'max-age=10886400; includeSubDomains; preload', 'Vary': 'Origin', 'X-Request-ID': '00005c635242d20ae583edf300000002', 'Connection': 'keep-alive', 'Allow': 'GET, PUT, PATCH, DELETE, HEAD, OPTIONS', 'Date': 'Tue, 12 Feb 2019 23:09:54 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'})
HTTP response body: {"object":"error","code":409,"type":"conflict","message":"Cannot delete the campaign while in the current phase.","request_id":"00005c634c5fd20ae583edf300000000"}

Please could this be released on PyPI?

instead of:

pip install git+https://github.com/ARMmbed/manifest-tool.git

with commit hashes for version locking

the user should be able to do:

pip install mbed-manifest-tool

with numbered versions

This also decouples the choice of git code repository from the dependency resolver by using the official PyPA package repository.

manifest-tool 2.1.1 - fails to install on Ubuntu 20.04 w Python 3.8.5 (asn1ate, pyrsistent)

Freshly created Ubuntu 20.04 machine with Ubuntu 3.8.5 fails to install manifest-tool 2.1.1

jankii01@ubuntu:~$ python3 -m venv ~/pelion-venv
jankii01@ubuntu:~$ source ~/pelion-venv/bin/activate
(pelion-venv) jankii01@ubuntu:~$ python --version
Python 3.8.5
(pelion-venv) jankii01@ubuntu:~$ 
(pelion-venv) jankii01@ubuntu:~$ sudo apt install git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
git is already the newest version (1:2.25.1-1ubuntu3.1).
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
(pelion-venv) jankii01@ubuntu:~$ 
(pelion-venv) jankii01@ubuntu:~$ 
(pelion-venv) jankii01@ubuntu:~$ git --version
git version 2.25.1
(pelion-venv) jankii01@ubuntu:~$ pip install manifest-tool==2.1.1
Collecting manifest-tool==2.1.1
  Downloading manifest_tool-2.1.1-cp38-cp38-manylinux2014_x86_64.whl (325 kB)
     |████████████████████████████████| 325 kB 1.1 MB/s 
Collecting jsonschema<=3.2.0,>=2.6.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 3.3 MB/s 
Collecting pyasn1<=0.4.8,==0.3.1
  Downloading pyasn1-0.3.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 1.2 MB/s 
Collecting asn1ate<=0.6.0,>=0.5
  Downloading asn1ate-0.6.0.tar.gz (24 kB)
Collecting requests<=2.25.1,>=2.20.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 11.4 MB/s 
Collecting PyYAML<=5.4.1,>=4.2b1
  Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
     |████████████████████████████████| 662 kB 6.1 MB/s 
Collecting cryptography<=3.3.1,>=2.5
  Downloading cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
     |████████████████████████████████| 2.6 MB 5.7 MB/s 
Collecting six>=1.11.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting attrs>=17.4.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 8.6 MB/s 
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.17.3.tar.gz (106 kB)
     |████████████████████████████████| 106 kB 5.1 MB/s 
Requirement already satisfied: setuptools in ./pelion-venv/lib/python3.8/site-packages (from jsonschema<=3.2.0,>=2.6.0->manifest-tool==2.1.1) (44.0.0)
Collecting pyparsing>=2.0.0
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 9.9 MB/s 
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 9.0 MB/s 
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 6.2 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 10.8 MB/s 
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 11.4 MB/s 
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
     |████████████████████████████████| 411 kB 8.1 MB/s 
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 11.7 MB/s 
Building wheels for collected packages: asn1ate, pyrsistent
  Building wheel for asn1ate (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jankii01/pelion-venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q4ls89jh/asn1ate/setup.py'"'"'; __file__='"'"'/tmp/pip-install-q4ls89jh/asn1ate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-_crew3o_
       cwd: /tmp/pip-install-q4ls89jh/asn1ate/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for asn1ate
  Running setup.py clean for asn1ate
  Building wheel for pyrsistent (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jankii01/pelion-venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q4ls89jh/pyrsistent/setup.py'"'"'; __file__='"'"'/tmp/pip-install-q4ls89jh/pyrsistent/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-met8czmi
       cwd: /tmp/pip-install-q4ls89jh/pyrsistent/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for pyrsistent
  Running setup.py clean for pyrsistent
Failed to build asn1ate pyrsistent
Installing collected packages: six, attrs, pyrsistent, jsonschema, pyasn1, pyparsing, asn1ate, chardet, urllib3, certifi, idna, requests, PyYAML, pycparser, cffi, cryptography, manifest-tool
    Running setup.py install for pyrsistent ... done
    Running setup.py install for asn1ate ... done
Successfully installed PyYAML-5.4.1 asn1ate-0.6.0 attrs-20.3.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 cryptography-3.3.1 idna-2.10 jsonschema-3.2.0 manifest-tool-2.1.1 pyasn1-0.3.1 pycparser-2.20 pyparsing-2.4.7 pyrsistent-0.17.3 requests-2.25.1 six-1.15.0 urllib3-1.26.4

manifest tool dev init does not allow specifying vendor id

We should really allow specifying the vendor id, because Pelion Edge still has (at least briefly) the dependency on the vendor id for the gateway features. It's not possible with manifest tool v2.0 now.

https://developer.pelion.com/docs/device-management-edge/2.2/quick-start/lmp-quick-start.html

Is ECC-384 (secp384r1) key supported for creating and signing manifest?

I am trying to create a manifest and want to use the ECC-384 (secp384r1) keypair.
I have generated the private key using:

$ openssl ecparam -genkey -name secp384r1 -outform PEM -out secp384r1.priv.key.pem

However, when I try to create manifest
manifest-tool --debug create
--config my.config.yaml
--key secp384r1.priv.key.pem
--fw-version 1.2.3
--output my.manifest.bin

I am getting below error:
023-01-11 09:06:11,803 INFO Vendor-ID: fe1d22aa8d2f589db31a962349b16969
2023-01-11 09:06:11,803 INFO Class-ID: 692be9ae461c5018b855726b71a820f9
2023-01-11 09:06:11,826 ERROR int too big to convert

Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/manifesttool/mtool/mtool.py", line 142, in entry_point
CreateAction.entry_point(args, ManifestAsnCodecV3)
File "/.local/lib/python3.6/site-packages/manifesttool/mtool/actions/create.py", line 190, in entry_point
asn1_codec_class=asn1_codec
File "/.local/lib/python3.6/site-packages/manifesttool/mtool/actions/create.py", line 152, in do_create
ecdsa_helper.signature_der_to_raw(signature)
File "/.local/lib/python3.6/site-packages/manifesttool/mtool/ecdsa_helper.py", line 90, in signature_der_to_raw
return int_to_bytes(r, 32) + int_to_bytes(s, 32)
File "/usr/local/lib64/python3.6/site-packages/cryptography/utils.py", line 77, in int_to_bytes
length or (integer.bit_length() + 7) // 8 or 1, "big"
OverflowError: int too big to convert

Also., tried with Python 3.10.7 and got the same error:

2023-01-11 08:52:48,494 INFO Vendor-ID: fe1d22aa8d2f589db31a962349b16969
2023-01-11 08:52:48,494 INFO Class-ID: 692be9ae461c5018b855726b71a820f9
2023-01-11 08:52:48,532 ERROR int too big to convert
Traceback (most recent call last):
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\manifesttool\mtool\mtool.py", line 142, in entry_point
CreateAction.entry_point(args, ManifestAsnCodecV3)
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\manifesttool\mtool\actions\create.py", line 185, in entry_point
manifest_bin = cls.do_create(
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\manifesttool\mtool\actions\create.py", line 152, in do_create
ecdsa_helper.signature_der_to_raw(signature)
File "\AppData\Local\Programs\Python\Python310\lib\site-packages\manifesttool\mtool\ecdsa_helper.py", line 90, in signature_der_to_raw
return int_to_bytes(r, 32) + int_to_bytes(s, 32)
File "~\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\utils.py", line 63, in int_to_bytes
return integer.to_bytes(
OverflowError: int too big to convert

Cannot install manifest-tool 1.4.8

When attempting to install I get this error:

pip install manifest-tool==1.4.8
Collecting manifest-tool==1.4.8
  Using cached manifest-tool-1.4.8.tar.gz (64 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/daltonv/repositories/OC_CCE7102_CCE7103_ASIC_programmer/asic-programmer-mbed/mbed-os/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8_ydflp4/manifest-tool_79342852e4b34bef8794e849de49c790/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8_ydflp4/manifest-tool_79342852e4b34bef8794e849de49c790/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-e12k0ych
         cwd: /tmp/pip-install-8_ydflp4/manifest-tool_79342852e4b34bef8794e849de49c790/
    Complete output (12 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-8_ydflp4/manifest-tool_79342852e4b34bef8794e849de49c790/setup.py", line 28, in <module>
        from pip._internal.download import PipSession
    ModuleNotFoundError: No module named 'pip._internal.download'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-8_ydflp4/manifest-tool_79342852e4b34bef8794e849de49c790/setup.py", line 30, in <module>
        from pip.req import parse_requirements
    ModuleNotFoundError: No module named 'pip.req'
    ----------------------------------------

I have tried this on Windows, Linux, and python 3.10.2, 3.7.6, and 2.7.14. All run into the same problem. Is this version no longer supported?

Update Pypi homepage

https://pypi.org/project/manifest-tool/

The Pypi homepage link is stale and does not point to this repo, and needs to be updated.

v1.4.8 - Pip install fails - missing requirements.txt

When trying to install or upgrade to v1.4.8, Pip install manifest-tool fails with:
Collecting manifest-tool Using cached https://files.pythonhosted.org/packages/61/00/fb183d44ad303ef3462f04398a94c8ce09ba049000f616c630d1e83bb07c/manifest-tool-1.4.7.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-2zsjtx/manifest-tool/setup.py", line 34, in <module> reqs = [str(r.req) for r in install_reqs] File "/proj/ben/testvenv/local/lib/python2.7/site-packages/pip/_internal/req/req_file.py", line 82, in parse_requirements filename, comes_from=comes_from, session=session File "/proj/ben/testvenv/local/lib/python2.7/site-packages/pip/_internal/download.py", line 440, in get_file_content 'Could not open requirements file: %s' % str(exc) pip._internal.exceptions.InstallationError: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
I have tested this in and out of a venv using python 2.7.13. pip installing from git does work though.

using manifest-tool in Github actions

I've successfully setup a Github Actions CI/CD pipeline, that builds my firmware.
Next I want to auto upload the firmware and manifest to Pelion, so I started adding the manifest-tool to the pipeline.
I've was successful in installing the manifest-tool from source, and the tool runs.
As a side note, I'm using an older version of the manifest-tool (1.5.2) since I have that one on my desktop working for few years now, combined with python 2.7.

While on my desktop the manifest-tool runs fine on Windows and WSL(Ubuntu), I'm having trouble using the manifest-tool in a Github Workflow runner based upon ubuntu. Underneath is a glimpse on how I did define that manifest part. Note that I'm storing the signing certificate and key in Github secrets.

- name: Upload to Pelion
              env:
                  SIGN_CERT_PK: ${{ secrets.MY_ORG_MANIFEST_SIGNING_CERT_PRIVATEKEY }}
                  SIGN_CERT_HEX: ${{ secrets.MY_ORG_MANIFEST_SIGNING_CERT_HEX }}
              run: |
                  echo "Restoring Manifest Signing certificate"
                  echo $SIGN_CERT_HEX | xxd -r -ps > firmware-integrity-certificate.der
                  echo $SIGN_CERT_PK > firmware-integrity-certificate.key
                  
                  # do some stuff such as uploading firmware to Pelion and retrieving the download URL into $FWURL

                  echo "Creating manifest..."
                  manifest-tool create -k firmware-integrity-certificate.key -p firmware.bin -u $FWURL -i manifest.json -o manifest.bin

The log:

Creating manifest...
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/2.7.18/x64/bin/manifest-tool", line 30, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/2.7.18/x64/bin/manifest-tool", line 27, in main
    return clidriver.main()
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/clidriver.py", line 33, in main
    return driver.main()
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/clidriver.py", line 63, in main
    }[self.options.action](self.options) or 0
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/create.py", line 52, in main
    return create(options, manifestInput)
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/v1/create.py", line 810, in main
    manifest = create_signed_resource(options, manifestInput)
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/v1/create.py", line 761, in create_signed_resource
    signature = get_signature(options, manifestInput, enc_data = resource_encoded)
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/manifesttool/v1/create.py", line 654, in get_signature
    sk = ecdsa.SigningKey.from_pem(options.private_key.read())
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/ecdsa/keys.py", line 930, in from_pem
    valid_curve_encodings,
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/ecdsa/keys.py", line 996, in from_der
    s, empty = der.remove_sequence(s)
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/ecdsa/der.py", line 147, in remove_sequence
    raise UnexpectedDER("Empty string does not encode a sequence")
ecdsa.der.UnexpectedDER: Empty string does not encode a sequence

The error state as if the private key is empty. However when I try to print the private key the github shell prevents my from exposing the secret in every single way. I've even tried to manually tier appart the .key file and compose it again using echo but GH every times detect that I'm willing to expose some secret and hides using using **** **** ****. In the end I'm only guessing that the .key file is correctly restored, but I can't prove it. How checked it multiple times on my desktop and I see no problem there. The fact that GH is hiding the output whenever and however I'm reading the .key file makes my think the content is OK.

So now I'm guessing maybe the security shell also prevents the manifest-tool from running well. Any idea what could be wrong?

Compile error when installing with python 3.10.2

Problem

When trying to install manifest-tool with python 3.10.2 it exits the setup because gcc failed. (see error below)
python ver.: 3.10.2
pip ver.: 21.2.3
gcc ver.: 11.2.1
Installing with python 3.8 does work.

Command line error

command line error

`Installing collected packages: manifest-tool
    Running setup.py install for manifest-tool ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-yjw7bx_w/manifest-tool_f7bbb78c03ab4282b4d77d300a46b960/setup.py'"'"'; __file__='"'"'/tmp/pip-install-yjw7bx_w/manifest-tool_f7bbb78c03ab4282b4d77d300a46b960/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3i6fei9c/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/wnijenhuis/.local/include/python3.10/manifest-tool
         cwd: /tmp/pip-install-yjw7bx_w/manifest-tool_f7bbb78c03ab4282b4d77d300a46b960/
    Complete output (120 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.10
    creating build/lib.linux-x86_64-3.10/manifesttool
    copying manifesttool/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool
    creating build/lib.linux-x86_64-3.10/manifesttool/package_tool
    copying manifesttool/package_tool/package_tool.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool
    copying manifesttool/package_tool/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool
    creating build/lib.linux-x86_64-3.10/manifesttool/mtool
    copying manifesttool/mtool/payload_format.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool
    copying manifesttool/mtool/mtool.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool
    copying manifesttool/mtool/ecdsa_helper.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool
    copying manifesttool/mtool/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool
    creating build/lib.linux-x86_64-3.10/manifesttool/dev_tool
    copying manifesttool/dev_tool/dev_tool.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool
    copying manifesttool/dev_tool/defaults.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool
    copying manifesttool/dev_tool/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool
    creating build/lib.linux-x86_64-3.10/manifesttool/delta_tool
    copying manifesttool/delta_tool/delta_tool.py -> build/lib.linux-x86_64-3.10/manifesttool/delta_tool
    copying manifesttool/delta_tool/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/delta_tool
    creating build/lib.linux-x86_64-3.10/manifesttool/common
    copying manifesttool/common/common_helpers.py -> build/lib.linux-x86_64-3.10/manifesttool/common
    copying manifesttool/common/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/common
    creating build/lib.linux-x86_64-3.10/manifesttool/package_tool/package_format
    copying manifesttool/package_tool/package_format/tar_package.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/package_format
    copying manifesttool/package_tool/package_format/package_format.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/package_format
    copying manifesttool/package_tool/package_format/embedded_package.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/package_format
    copying manifesttool/package_tool/package_format/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/package_format
    creating build/lib.linux-x86_64-3.10/manifesttool/package_tool/asn1
    copying manifesttool/package_tool/asn1/package_schema.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/asn1
    copying manifesttool/package_tool/asn1/package_encoder.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/asn1
    copying manifesttool/package_tool/asn1/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/asn1
    creating build/lib.linux-x86_64-3.10/manifesttool/package_tool/actions
    copying manifesttool/package_tool/actions/parse.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/actions
    copying manifesttool/package_tool/actions/create.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/actions
    copying manifesttool/package_tool/actions/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/package_tool/actions
    creating build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1
    copying manifesttool/mtool/asn1/manifest_codec.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1
    copying manifesttool/mtool/asn1/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1
    creating build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    copying manifesttool/mtool/actions/schema.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    copying manifesttool/mtool/actions/public_key.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    copying manifesttool/mtool/actions/parse.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    copying manifesttool/mtool/actions/create.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    copying manifesttool/mtool/actions/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/actions
    creating build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v3
    copying manifesttool/mtool/asn1/v3/manifest_schema_v3.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v3
    copying manifesttool/mtool/asn1/v3/manifest_encoder_v3.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v3
    copying manifesttool/mtool/asn1/v3/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v3
    creating build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v1
    copying manifesttool/mtool/asn1/v1/manifest_schema_v1.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v1
    copying manifesttool/mtool/asn1/v1/manifest_encoder_v1.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v1
    copying manifesttool/mtool/asn1/v1/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/mtool/asn1/v1
    creating build/lib.linux-x86_64-3.10/manifesttool/dev_tool/pelion
    copying manifesttool/dev_tool/pelion/pelion.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/pelion
    copying manifesttool/dev_tool/pelion/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/pelion
    creating build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    copying manifesttool/dev_tool/actions/update.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    copying manifesttool/dev_tool/actions/init.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    copying manifesttool/dev_tool/actions/create.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    copying manifesttool/dev_tool/actions/__init__.py -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    running egg_info
    writing manifest_tool.egg-info/PKG-INFO
    writing dependency_links to manifest_tool.egg-info/dependency_links.txt
    writing entry points to manifest_tool.egg-info/entry_points.txt
    writing requirements to manifest_tool.egg-info/requires.txt
    writing top-level names to manifest_tool.egg-info/top_level.txt
    reading manifest file 'manifest_tool.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    adding license file 'LICENSE'
    writing manifest file 'manifest_tool.egg-info/SOURCES.txt'
    copying manifesttool/package_tool/package-schema.json -> build/lib.linux-x86_64-3.10/manifesttool/package_tool
    copying manifesttool/mtool/manifest-input-schema.json -> build/lib.linux-x86_64-3.10/manifesttool/mtool
    copying manifesttool/dev_tool/actions/code_template.txt -> build/lib.linux-x86_64-3.10/manifesttool/dev_tool/actions
    running build_ext
    building 'manifesttool.armbsdiff' extension
    creating build/temp.linux-x86_64-3.10
    creating build/temp.linux-x86_64-3.10/bsdiff
    gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DLZ4_MEMORY_USAGE=10 -Ibsdiff -I/usr/include/python3.10 -c bsdiff/bsdiff.c -o build/temp.linux-x86_64-3.10/bsdiff/bsdiff.o --std=c99 -O3
    gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DLZ4_MEMORY_USAGE=10 -Ibsdiff -I/usr/include/python3.10 -c bsdiff/bsdiff_helper.c -o build/temp.linux-x86_64-3.10/bsdiff/bsdiff_helper.o --std=c99 -O3
    bsdiff/bsdiff_helper.c: In function ‘do_diff’:
    bsdiff/bsdiff_helper.c:158:42: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t’ {aka ‘long int’} [-Wformat=]
      158 |             "Wrote diff file %s, size %lld. Max undeCompressBuffer frame size was %lld, max deCompressBuffer frame size was %lld.\n",
          |                                       ~~~^
          |                                          |
          |                                          long long int
          |                                       %ld
      159 |             delta_file, patch_file_size, max_frame_size,
          |                         ~~~~~~~~~~~~~~~
          |                         |
          |                         int64_t {aka long int}
    bsdiff/bsdiff_helper.c:158:86: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘int64_t’ {aka ‘long int’} [-Wformat=]
      158 |             "Wrote diff file %s, size %lld. Max undeCompressBuffer frame size was %lld, max deCompressBuffer frame size was %lld.\n",
          |                                                                                   ~~~^
          |                                                                                      |
          |                                                                                      long long int
          |                                                                                   %ld
      159 |             delta_file, patch_file_size, max_frame_size,
          |                                          ~~~~~~~~~~~~~~
          |                                          |
          |                                          int64_t {aka long int}
    bsdiff/bsdiff_helper.c:158:128: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 5 has type ‘int64_t’ {aka ‘long int’} [-Wformat=]
      158 |             "Wrote diff file %s, size %lld. Max undeCompressBuffer frame size was %lld, max deCompressBuffer frame size was %lld.\n",
          |                                                                                                                             ~~~^
          |                                                                                                                                |
          |                                                                                                                                long long int
          |                                                                                                                             %ld
      159 |             delta_file, patch_file_size, max_frame_size,
      160 |             max_deCompressBuffer_size
          |             ~~~~~~~~~~~~~~~~~~~~~~~~~
          |             |
          |             int64_t {aka long int}
    gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DLZ4_MEMORY_USAGE=10 -Ibsdiff -I/usr/include/python3.10 -c bsdiff/bsdiff_python.c -o build/temp.linux-x86_64-3.10/bsdiff/bsdiff_python.o --std=c99 -O3
    bsdiff/bsdiff_python.c:19:10: fatal error: Python.h: No such file or directory
       19 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-yjw7bx_w/manifest-tool_f7bbb78c03ab4282b4d77d300a46b960/setup.py'"'"'; __file__='"'"'/tmp/pip-install-yjw7bx_w/manifest-tool_f7bbb78c03ab4282b4d77d300a46b960/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3i6fei9c/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers`

Update campaign does not start

$ manifest-tool update device -D 0167cb2b921600000000000100100062 -p ./BUILD/K64F/GCC_ARM-DEBUG/mbed-cloud-client-example_update.bin
[WARNING] 2018-12-20 16:41:33 - manifesttool.update_device - Using mbed-cloud-client-example_update.bin-2018-12-20T16:41:33 as payload name.
[WARNING] 2018-12-20 16:41:33 - manifesttool.update_device - Using mbed-cloud-client-example_update.bin-2018-12-20T16:41:33-manifest as manifest name.
[INFO] 2018-12-20 16:41:38 - manifesttool.update_device - Created new firmware at http://objstore.mbedonprem.com/fw-catalog/mbed-cloud-client-example_update_ee7FOZu.bin
[INFO] 2018-12-20 16:41:38 - manifesttool.update_device - Created temporary manifest file at /var/folders/84/tqqdqp696j73582q658x7pkx0l_yxp/T/tmpqgnu7T/manifest
[INFO] 2018-12-20 16:41:39 - manifesttool.update_device - Created new manifest at http://objstore.mbedonprem.com/fw-catalog/manifest_JshgwXF
[INFO] 2018-12-20 16:41:39 - manifesttool.update_device - Manifest ID: 0167cc10994c0000000000010010036f
[INFO] 2018-12-20 16:41:40 - manifesttool.update_device - Campaign successfully created. Current state: 'draft'
[INFO] 2018-12-20 16:41:40 - manifesttool.update_device - Campaign successfully created. Filter result: {'id': {u'$eq': '0167cb2b921600000000000100100062'}}
[INFO] 2018-12-20 16:41:40 - manifesttool.update_device - Starting the update campign...
[INFO] 2018-12-20 16:41:40 - manifesttool.update_device - Campaign successfully started. Current state: u'scheduled'. Checking updates..
[INFO] 2018-12-20 16:41:41 - manifesttool.update_device - Current state: 'checkedmanifest'
[INFO] 2018-12-20 16:41:42 - manifesttool.update_device - Current state: 'draft'

This hangs forever.

Device does not start update.

Cannot call manifest-tool on command line when installed from pip

Description

manifest-tool is not recognized as a valid executable when installed through pip. Tested on both Windows and Linux

Steps to recreate the issue

> pip install -U manifest-tool
> manifest-tool
'manifest-tool' is not recognized as an internal or external command,
operable program or batch file.

Background

• I have confirmed my path variable is correct. It includes the path to pip packages.
  • C:\Users\xxxxxxx\AppData\Roaming\Python\Python37\Scripts
• After installation, there is a manifest-tool file present with no file extension in that location. It is a python file.
• Other packages installed with pip can be accessed with no problems. They have .exe executable files in that location that can be called directly. these include pip.exe, pyocd.exe, etc.
• I have confirmed that the system can locate manifest-tool using the windows where command, it is just not an executable.

  > where manifest-tool
  > C:\Users\xxxxxx\AppData\Roaming\Python\Python37\Scripts\manifest-tool
  

Environment

• manifest-tool 1.5.2
• Windows 10
• Python 3

Impact

Multiple users have reported this.

Workaround

Install from source.

> git clone https://github.com/armmbed/manifest-tool
> cd manifest-tool
> pip install . --user
> manifest-tool
usage: manifest-tool [-h] [-l {debug,info,warning,exception}] [--version]
                     {create,parse,verify,cert,init,sign,update} ...

pyasn1 version requirement 'pyasn1==0.3.1,<=0.4.8' causes dependency resolution problems

requirements file has

pyasn1==0.3.1,<=0.4.8

This creates dependency resolution problems and seems incorrect.
It looks like the correct line would be:

pyasn1>=0.3.1,<=0.4.8

Add support to release linux aarch64 wheels

Problem

On aarch64, ‘pip install manifest-tool’ builds the wheels from source code and then installs it. It requires the user to have a development environment installed on his system. Also, it takes some time to build the wheels than downloading and extracting the wheels from pypi.

Resolution

On aarch64, ‘pip install manifest-tool’ should download the wheels from pypi

@moshe-shahar and Team Please let me know your interest in releasing aarch64 wheels. I can help in this.

Manifest Tool - Update Device - Invalid API Key

Description of defect

When executing the manifest-tool update device command as suggested in the tutorial, an Invalid API key error occurs. This is strange because the API key should be valid as my device is successfully registered to Pelion. The .manifest_tool.json and .mbed_cloud_config.json seem to be correctly filled as well.

Target(s) affected by this defect ?

K64F + I-NUCLEO-S2868A1

Toolchain(s) (name and version) displaying this defect ?

GCC_ARM - gcc-arm-none-eabi-9-2019-q4-major

What version of Mbed-os are you using (tag or sha) ?

Mbed-OS 5.15

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli 1.10.2

How is this defect reproduced ?

The problem can be reproduced by running the manifest-tool update device on the mbed-cloud-client-example application. The error log is attached.

manifest-tool-update-device-error_log.txt