azure/azure-batch-cli-extensions

Batch extension cli commands for Azure cli v2

Microsoft Azure Batch Extensions

Azure Batch CLI extensions will be retired on 30 September 2024 https://azure.microsoft.com/updates/azure-batch-cli-extensions-will-be-retired-on-30-september-2024/

Remove the Azure Batch CLI Extensions: .. code-block:: bash

$ az extension remove -n azure-batch-cli-extensions

This project is a preview build of the Microsoft Azure command-line interface to demonstrate proposed features in Azure Batch. For further details on the Azure CLI, please check the official documentation.

The purpose of this project is to allow customers to try out proposed Batch features and provide feedback to help shape the direction of the Batch service. The features presented here may not be compatible with other Batch client SDKs and tools, nor will they necessarily be adopted into the core Batch service.

As these features are still in preview, they will be updated regularly, and refined based on customer feedback. Unfortunately this may result in occasional breaking changes, though every effort will be made to keep this to a minimum.

Features

• Input data upload to Batch linked storage accounts

  A new command to allow a user to upload a set of files directly into the storage account linked to their Azure Batch account.

• Input data references using linked storage accounts

  Input data stored in linked storage under a file group can be simply referenced by a task by using some new ResourceFile properties.

• Automatic persistence of task output files to a file group

  When declaring task output files, you can now persist outputs to a named file group, without the need to generate a container SAS URL.

• Download job outputs from directly from storage

  A new command to allow a user to download job output files from a file group in the storage account linked to their Azure Batch account.

• Pool and job templates with parameterization

  Templates allow pools and jobs to be defined in parameterized json files with a format inspired by ARM templates.

• Task factories for automatic task generation on job submission

  Task factories provide a way for a job and all its tasks to be created in one command instead of calling azure batch task create for each task. There are currently three kinds of task factory:

  • Task Collection - tasks are explicitly defined as a part of the job
  • Parametric Sweep - a set of tasks are created by substituting a range or sequence of values into a template
  • Per File - a template task is replicated for each available input file

• Split job configuration and management with reusable application templates

  Application templates provide a way to partition the details of a job into two parts. All of the details about how the job should be processed are moved into the application template, creating a reusable definition that is independent of a particular account. Application templates are parameterized to allow the processing to be customized without requiring modification of the template itself.

• Easy software installation via package managers

  Integration with existing 3rd party package managers to streamline the installation of applications. Currently the following package managers are supported:

  • Chocolatey - for Windows
  • APT - as used by some Linux distros including Ubuntu, Debian, and Fedora.
  • Yum - a package manager used by some Linux distros including Red Hat Enterprise Linux, Fedora, CentOS.

Samples

Samples for all of the preview features can be found in samples.

SDK Installation

$ pip install azure-batch-extensions

CLI Extensions Installation

In order to make use of these features, you must have the Azure CLI installed. You can find futher instructions in the official documentation and in the Azure CLI GitHub repository.

This extension package can be installed to supplement the existing Azure CLI Batch commands. It can be installed using the CLI extension tools (recommended):

$ az extension add -n azure-batch-cli-extensions

Each release of the CLI ships with a specific version of the azure-batch-cli-extensions. This is considered the official version for the CLI release. If you would like to install a different version you can specify any release listed at https://github.com/Azure/azure-batch-cli-extensions/releases/. To add a specific extension version and pin to it:

$ az extension add --source https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-x.x.x/azure_batch_cli_extensions-x.x.x-py2.py3-none-any.whl

Uninstall CLI extensions

The CLI extensions can be removed using the az extension remove command:

$ az extension remove -n azure-batch-cli-extensions

Azure Batch account requirements

In order to make use of the new features previewed here, you will need an Azure Batch account with a linked storage account. For more information on this, see Create an Azure Batch account using the Azure Portal.

Contributing

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Developer Installation

Preparing your machine

1. Install Python 3.5.x from http://python.org. Please note that the version of Python that comes preinstalled on OSX is 2.7.
2. Clone your repository and check out the master branch.
3. Create a new virtual environment “env” for Python 3.5 in the root of your clone. You can do this by running:

Windows

python -m venv <clone root>\env

OSX/Ubuntu (bash)

python –m venv <clone root>/env

4. Activate the env virtual environment by running:

Windows

<clone root>\env\scripts\activate.bat

OSX/Ubuntu (bash)

. <clone root>/env/bin/activate

5. Install the dependencies and load the command module as a local package using pip.

python scripts/dev_setup.py

azure-batch-cli-extensions's Issues

support configuration variables.

[variables('environmentSettings')[parameters('environmentName')].instanceSize]

"parameters": {
"environmentName": {
"type": "string",
"allowedValues": [
"test",
"prod"
]
}
},
"variables": {
"environmentSet": {
"test": {
"instanceSize": "Small",
"instanceCount": 1
},
"prod": {
"instanceSize": "Large",
"instanceCount": 4
}
}
},
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-templates-variables

as discussed in an email thread configuration variables don't work since the array slices the first two characters off \'nvironmentSet)[test].instanceSi\''”

Extension relies on dependency not installed with the CLI

The extension relies on this: https://pypi.org/project/azure-batch-extensions/
Leading to:

Unable to load extension 'azure-batch-cli-extensions'. Use --debug for more information.
Traceback (most recent call last):
  File "c:\users\wilx\azure-cli\src\azure-cli-core\azure\cli\core\__init__.py", line 176, in _update_command_table_from_extensions
    extension_command_table = _load_extension_command_loader(self, args, ext_mod)
  File "c:\users\wilx\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 582, in _load_extension_command_loader
    return _load_command_loader(loader, args, ext, '')
  File "c:\users\wilx\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 568, in _load_command_loader
    command_table = command_loader.load_command_table(args)
  File "C:\Users\wilx\.azure\cliextensions\azure-batch-cli-extensions\azext_batch\__init__.py", line 28, in load_command_table
    from azext_batch.commands import load_command_table
  File "C:\Users\wilx\.azure\cliextensions\azure-batch-cli-extensions\azext_batch\commands.py", line 12, in <module>
    from azure.batch_extensions import __version__ as batch_ext_version
ModuleNotFoundError: No module named 'azure.batch_extensions'

Azure storage incompatibility

I'm getting this error message from installation.
OS: MacOS 10.13.4
Shell type: bash

azure-batch-extensions 1.0.1 has requirement azure-storage<0.35,>=0.32, but you'll have azure-storage 0.36.0 which is incompatible.

pool: Unexpect element identity in template

Get error

Unexpect element identity in template

with this template

{
    "parameters": {
        "nodeCount": {
            "type": "int",
            "metadata": {
                "description": "The number of pool nodes"
            }
        },
        "poolId": {
            "type": "string",
            "metadata": {
                "description": "The pool ID "
            }
        },
        "subnetId": {
            "type": "string",
            "metadata": {
                "description": "The subnet ID pool vms go in "
            }
        }
    },
    "pool": {
        "type": "Microsoft.Batch/batchAccounts/pools",
        "apiVersion": "2016-12-01",
        "properties": {
            "id": "[parameters('poolId')]",
            "virtualMachineConfiguration": {
                "imageReference": {
                    "publisher": "microsoft-azure-batch",
                    "offer": "centos-container",
                    "sku": "7-7",
                    "version": "latest"
                },
                "nodeAgentSKUId": "batch.node.centos 7",
                "containerConfiguration": {
                    "type": "dockerCompatible",
                    "containerRegistries": [
                        {
                            "registryServer": "abceastusdevncc.azurecr.io",
                            "username": "abceastusdevncc",
                            "password": "xxxx"
                        }
                    ]
                }
            },
            "vmSize": "STANDARD_D3_V2",
            "networkConfiguration": {
                "subnetId": "[parameters('subnetId')]",
                "dynamicVNetAssignmentScope": "none",
                "publicIPAddressConfiguration": {
                    "provision": "BatchManaged"
                }
            },

            "targetDedicatedNodes": "[parameters('nodeCount')]",
            "enableAutoScale": false,
            "taskSchedulingPolicy": {
                "nodeFillType": "Pack"
            },
            "identity": {
                "type": "UserAssigned",
                "userAssignedIdentities": [
                    {
                        "resourceId": "/subscriptions/xxx/resourceGroups/abc-eastus-dev-nuc-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/abceastdevncctch001-id",
                        "clientId": "xxx",
                        "principalId": "xxx"
                    }
                ]
            }
        }
    }
}

Unable to install batch extension in azure cli docker image

Create a docker instance from
https://hub.docker.com/r/microsoft/azure-cli/

Running az extension add -n azure-batch-cli-extensions --debug fails

File encoding and line endings

Hello,

I've noticed on a couple of different occasions when using --template option on the CLI an error would be returned showing:

benji$ az batch job create --template job-template.json --account-name BATCH_NAME --account-endpoint BATCH_ENDPOINT
The behavior of this command has been altered by the following extension: azure_batch_cli_extensions
You are using an experimental feature {Job Template}.
Invalid JSON file: No JSON object could be decoded

Having investigated this it appears a couple of causes are in play:

• UNIX line endings
• File encoding

Can we have the service better interpret/handle template files?

json files that have encountered this error:
:: https://github.com/whatsondoc/azure-batch/tree/master/pool/marketplace-image (pool and job templates)
:: https://github.com/karlpodesta/azure-batch-hpc-lab (pool and job templates)

Unable to upload file using "az batch file upload" on windows

I was unable to upload file using az batch file upload. Below are the error message. What has I done wrong?

Describe the bug

Command Name
az batch file upload Extension Name: azure-batch-cli-extensions. Version: 6.0.0.

Errors:

The command failed with an unexpected error. Here is the traceback:
'NoneType' object has no attribute 'format'
Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 658, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 713, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/batch/_exception_handler.py", line 28, in batch_exception_handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 692, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 328, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "C:\Users\Yong Hui Yee\.azure\cliextensions\azure-batch-cli-extensions\azext_batch\custom.py", line 205, in upload_file
    client.file.upload(local_path, file_group, remote_path=remote_path, flatten=flatten)
  File "C:\Users\Yong Hui Yee\.azure\cliextensions\azure-batch-cli-extensions\azext\batch\operations\file_operations.py", line 70, in upload
    file_utils.upload_blob(f, file_group, file_name, self.get_storage_client(),
  File "C:\Users\Yong Hui Yee\.azure\cliextensions\azure-batch-cli-extensions\azext\batch\batch_extensions_client.py", line 187, in _storage_account
    keys = storage_client.storage_accounts.list_keys(storage_resource_group, storage_account)
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/mgmt/storage/v2021_08_01/operations/_storage_accounts_operations.py", line 1197, in list_keys
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/transport/_base.py", line 582, in format_url
AttributeError: 'NoneType' object has no attribute 'format'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Put any pre-requisite steps here...
• az batch file upload --local-path {} --file-group {} --account-name {} --account-endpoint {}

Expected Behavior

Environment Summary

Windows-10-10.0.18362-SP0
Python 3.8.9
Installer: MSI

azure-cli 2.34.1

Extensions:
azure-batch-cli-extensions 6.0.0

Dependencies:
msal 1.16.0
azure-mgmt-resource 20.0.0

Additional Context

Batch azure-batch-cli-extensions shows unknown version

While running a batch job command:
az batch job create --template sql_job_template.json --parameters job-params.json,
a series of python file errors return.
/*****************************************
Unexpect element blobSource in template
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 328, in execute
raise ex
********************************************/

progress_callback from file upload to include the file name

progress_callback currently has 2 parameters, data uploaded so far and total data to be uploaded. When calling the CLI to upload multiple files we have no way of matching the current progress to a given file. just including the file path to the callback as well is all that is needed.

edit: looking around, this callback might be the responsibility of the storage-client.

Unable to get azure batch CLI extensions working on Windows

Steps to reproduce:

1. Use a PC with latest Windows 10 installed
2. Install Azure CLI using Windows Installer
3. Open README.md and try to install the package
4. Use az extension add -n azure-batch-cli-extensions. You get:

An error occurred. Pip failed with status code 1. Use --debug for more information.

5. Install latest Python binaries. Use pip install azure-batch-extensions You get:

azure-mgmt-storage 1.5.0 has requirement msrestazure~=0.4.11, but you'll have msrestazure 0.5.0 which is incompatible. 
azure-mgmt-batch 4.1.0 has requirement msrestazure~=0.4.7, but you'll have msrestazure 0.5.0 which is incompatible.

I am completely not sure if this tooling is correctly installed after all, but I'm unable to create a job from a JSON template after I log in using az login. If I type:

az batch job create --template sample.json --account-name myaccount --account-endpoint https://myaccount.northeurope.batch.azure.com

I get:

az: error: unrecognized arguments: --template sample.json
usage: az [-h] [--verbose] [--debug] [--output {json,jsonc,table,tsv,yaml}]
          [--query JMESPATH]
          {batch} ...

Need to mention that this extension works perfectly on my PC with Ubuntu 18.04 installed.

Passing environment variables is not documented

The documentation sadly doesn't explain how to set environment variables for the containers that are created by az batch job create using e.g. a repeatTask.

By searching the Python code of the extension in this repo, I found out that environment variables can be configured as follows:

"repeatTask": {
    "commandLine": "env",
    "containerSettings": {
        "imageName": "alpine:latest"
    },
    "environmentSettings": [
        {
            "name": "VAR_1",
            "value": "someValue"
        },
        {
            "name": "VAR_2",
            "value": "anotherValue"
        }
    ]
}

It would be much easier to find out how to do this, if it was explained somewhere in the (otherwise quite detailed) docs in this repo. If it's not something specific to this extension, a link describing it would have already helped a lot, since web searches also didn't turn it up for me.

Error creating Batch Pool from CLI template

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name
az batch pool create Extension Name: azure-batch-cli-extensions. Version: 6.0.0.

Errors:

Unexpect element taskSlotsPerNode in template
Traceback (most recent call last):
python3.6/site-packages/knack/cli.py, ln 231, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 657, in execute
    raise ex
...
azure/cli/core/__init__.py, ln 807, in default_command_handler
    return op(**command_args)
cliextensions/azure-batch-cli-extensions/azext_batch/custom.py, ln 66, in create_pool
    templates.validate_json_object(json_obj, pool)
azure-batch-cli-extensions/azext/batch/_template_utils.py, ln 1203, in validate_json_object
    raise ValueError('Unexpect element {} in template'.format(item))
ValueError: Unexpect element taskSlotsPerNode in template

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Put any pre-requisite steps here...
• az batch pool create --template {}

Expected Behavior

Environment Summary

Linux-4.15.0-1113-azure-x86_64-with-debian-10.2 (Cloud Shell)
Python 3.6.10
Installer: DEB

azure-cli 2.22.0 *

Extensions:
azure-batch-cli-extensions 6.0.0
aks-preview 0.4.64
hpc-cache 0.1.0
ai-examples 0.2.5
ssh 0.1.4

Additional Context

Unable to upload files using "az batch file upload" on ubuntu

This is autogenerated. Please review and update as needed.

Hi, all. I was unable to upload file using "az batch file upload". I got the following error. Do you have solution on this issue?

Command Name
az batch file upload Extension Name: azure-batch-cli-extensions. Version: 6.0.0.

Errors:

The command failed with an unexpected error. Here is the traceback:
'NoneType' object has no attribute 'format'
Traceback (most recent call last):
  File "/opt/az/lib/python3.8/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 713, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/batch/_exception_handler.py", line 28, in batch_exception_handler
    raise ex
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/home/huiyee/.azure/cliextensions/azure-batch-cli-extensions/azext_batch/custom.py", line 205, in upload_file
    client.file.upload(local_path, file_group, remote_path=remote_path, flatten=flatten)
  File "/home/huiyee/.azure/cliextensions/azure-batch-cli-extensions/azext/batch/operations/file_operations.py", line 70, in upload
    file_utils.upload_blob(f, file_group, file_name, self.get_storage_client(),
  File "/home/huiyee/.azure/cliextensions/azure-batch-cli-extensions/azext/batch/batch_extensions_client.py", line 187, in _storage_account
    keys = storage_client.storage_accounts.list_keys(storage_resource_group, storage_account)
  File "/opt/az/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/az/lib/python3.8/site-packages/azure/mgmt/storage/v2021_08_01/operations/_storage_accounts_operations.py", line 1197, in list_keys
    request.url = self._client.format_url(request.url)
  File "/opt/az/lib/python3.8/site-packages/azure/core/pipeline/transport/_base.py", line 582, in format_url
    base = self._base_url.format(**kwargs).rstrip("/")
AttributeError: 'NoneType' object has no attribute 'format'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Put any pre-requisite steps here...
• az batch file upload --local-path {} --file-group {} --account-name {} --account-endpoint {}

Expected Behavior

Environment Summary

Linux-5.13.0-1017-azure-x86_64-with-glibc2.29, Ubuntu 20.04.3 LTS
Python 3.8.12
Installer: DEB

azure-cli 2.34.1

Extensions:
azure-batch-cli-extensions 6.0.0

Dependencies:
msal 1.16.0
azure-mgmt-resource 20.0.0

Additional Context

Small inconsistency when using escape characters in CLI templates

In the JSON template file, \" is used to escape double quotes in string values, but in the JSON file containing the values of the template parameters, double quotes must be escaped using \\\". Presumably, commands such as az batch job create --template job.json --parameters params.json cause values in params.json to be processed twice.

Task factory "task per file" fails with source "containerUrl"

Using task factory "task per file" with source defined as container URL fails with error:
AttributeError: 'SplitResult' object has no attribute 'pathname'

Here is extract from the job template:
{
(...)
"parameters":{
(...)
"inputContainerUrl":{
"type":"string",
"defaultValue":"https://tojobatch.blob.core.windows.net/fgrp-sequencer-input?st=2020-04-29T20%3A20%3A06Z&se=2020-05-30T20%3A20%3A00Z&sp=rl&sv=2018-03-28&sr=c&sig=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
},
(...)
"job":{
(...)
"taskFactory":{
"type":"taskPerFile",
"source":{
"containerUrl": "[parameters('inputContainerUrl')]"
},
(...)

The same template works ok when using source with "fileGroup" parameter.

Software level:
$ az --version
azure-cli 2.4.0 *
command-modules-nspkg 2.0.3
core 2.4.0 *
nspkg 3.0.4
telemetry 1.0.4
Extensions:
image-copy-extension 0.0.9
azure-batch-cli-extensions 5.0.1

Ignored code blocks?

It appears as though certain blocks of code are being ignored if they fall outside of a property definition, without returning any errors or warnings.

Case in point is here: https://github.com/whatsondoc/azure-batch/blob/master/pool/marketplace-image/batch-pool-MP-fixed.azuredeploy.json

...where the pool startTask doesn't get applied as the pool Properties: property is closed (inadvertently, in this case) before the startTask is defined.

Can we add a validation to track/capture such scenarios, and (at least) report them as a warning on the CLI when deploying using templates?

NoneType not parsed correctly by template parser

pool: Unexpect element taskSlotsPerNode in template

Get error:
Unexpect element taskSlotsPerNode in template

Trying to create a pool with this template:

{
    "parameters": {
        "nodeCount": {
            "type": "int",
            "metadata": {
                "description": "The number of pool nodes"
            }
        },
        "poolId": {
            "type": "string",
            "metadata": {
                "description": "The pool ID "
            }
        },
        "subnetId": {
            "type": "string",
            "metadata": {
                "description": "The subnet ID pool vms go in "
            }
        }
    },
    "pool": {
        "type": "Microsoft.Batch/batchAccounts/pools",
        "apiVersion": "2016-12-01",
        "properties": {
            "id": "[parameters('poolId')]",
            "virtualMachineConfiguration": {
                "imageReference": {
                    "publisher": "microsoft-azure-batch",
                    "offer": "centos-container",
                    "sku": "7-7",
                    "version": "latest"
                },
                "nodeAgentSKUId": "batch.node.centos 7",
                "containerConfiguration": {
                    "type": "dockerCompatible",
                    "containerRegistries": [
                        {
                            "registryServer": "abceastusdevncc.azurecr.io",
                            "username": "abceastusdevncc",
                            "password": "xxxx"
                        }
                    ]
                }
            },
            "vmSize": "STANDARD_D3_V2",
            "networkConfiguration": {
                "subnetId": "[parameters('subnetId')]",
                "dynamicVNetAssignmentScope": "none",
                "publicIPAddressConfiguration": {
                    "provision": "BatchManaged"
                }
            },

            "targetDedicatedNodes": "[parameters('nodeCount')]",
            "enableAutoScale": false,
            "taskSlotsPerNode": 4,
            "taskSchedulingPolicy": {
                "nodeFillType": "Pack"
            },
            "identity": {
                "type": "UserAssigned",
                "userAssignedIdentities": [
                    {
                        "resourceId": "/subscriptions/xxx/resourceGroups/abc-eastus-dev-nuc-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/abceastdevncctch001-id",
                        "clientId": "xxx",
                        "principalId": "xxx"
                    }
                ]
            }
        }
    }
}

azure-batch-cli-extension is not compatible with the latest batch SDK

azure-batch-cli-extension is not compatible with the latest batch SDK

Broken links in 'samples' read-me

In https://github.com/Azure/azure-batch-cli-extensions/tree/master/samples it seems a lot of the links in the 'Features used' sections are 404ing.

unable to use commonResourceFiles in a multiinstance task

When using a commonResourceFiles with a file group inside a multi-instance task as per this sample https://github.com/Azure/azure-batch-cli-extensions/blob/master/samples/mpi/job.json
I’m getting this error :
Malformed ResourceFile: Must have either 'source' or 'blobSource'

If I remove the commonResourceFiles, my job get submitted. So what is wrong here ?
xhpl-job.json.txt

Create pool from json does not run pool-task as custom user

Attempting to create a batch pool using the below json file (--json-file)
mon03support.txt

The pool comes up, however the pool-task is NOT run under user (kim0) which is the expected behavior. Instead it is run under some other _azbatch user account (which does not have sudo powers). Am I missing something in the json, or is this a bug

Thanks

pool list -o json cannot be used for pool create --json-file

Dumping an existing pool into json format using pool list -o json, cannot later be imported into pool create --json-file. The reason being, the serialized json contains nodeAgentSkuId while the correct letter-case is nodeAgentSKUId (KU capitalized now).

As you can imagine, some hair pulling was involved :)

Perhaps a unit test for that would be useful

pool json file should take ssh pub-key not priv-key

The json file that creates an azure batch pool, asks to take a user's ssh private key, while it should take a public key (servers always get public keys, and the private key doesn't leave my laptop). You can see the offending line here

https://github.com/Azure/vscode-azure-batch-tools/blob/master/schema/pool.schema.json#L626

Task factory taskPerFile storing source files with owner root for absolute paths

When trying to download files from the source container of "taskPerFile" task factory to an absolute path in the filesystem, e.g. to a data disk mount /datadisk:
"job":{
"type":"Microsoft.Batch/batchAccounts/jobs",
"properties":{
"id":"[parameters('jobName')]",
"onAllTasksComplete":"terminateJob",
"poolInfo": {
"poolId": "[parameters('poolId')]"
},
"taskFactory":{
"type":"taskPerFile",
"source":{
"fileGroup": "[parameters('inputFilegroup')]"
},
"repeatTask":{
"displayName":"Process file {fileName}",
"resourceFiles": [
{
"httpUrl": "{url}",
"filePath": "/datadisk/[parameters('jobName')]/{fileName}"
}, (…)
files (and directories in the path) are created with root owner, making it impossible for the tasks to access them:

ls -l /datadisk

total 58343588
drwxrwxrwx 7 root root 4096 Apr 9 12:49 ./
drwxr-xr-x 24 root root 4096 Apr 9 11:45 ../
drwxr-x--- 2 root root 4096 Apr 9 12:40 dummyjob/

ls -l /datadisk/dummyjob

total 58343588
drwxrwxrwx 7 root root 4096 Apr 9 12:49 ./
drwxr-xr-x 24 root root 4096 Apr 9 11:45 ../
-rwxrwx--- 1 root root 9 Apr 9 12:25 dummyfile*

'ValidationError' object has no attribute 'message'

This is autogenerated. Please review and update as needed.

Describe the bug

Tried to create a job with "outputFiles" and in the "containerUrl" I accidently didn't mention the protocol. The command failed with an unexpected error that prevented me from getting the real validation error (see attached call stack)

The only way I was able to get around it and get the real validation issue was by monkey-patching custom_errors.CreateTasksErrorException.__init__ and adding a dummy message to the error.

Command Name
az batch job create Extension Name: azure-batch-cli-extensions. Version: 6.0.0.

Errors:

CLIInternalError: The command failed with an unexpected error. Here is the traceback:
'ValidationError' object has no attribute 'message'
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-s90ugolk\knack\cli.py", line 215, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\commands\__init__.py", line 654, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\commands\__init__.py", line 717, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\commands\__init__.py", line 709, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-2g1ftftt\azure\cli\command_modules\batch\_exception_handler.py", line 30, in batch_exception_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-7f66x3i5\six.py", line 703, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\commands\__init__.py", line 688, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\commands\__init__.py", line 325, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-_1bqfhco\azure\cli\core\__init__.py", line 784, in default_command_handler
  File "C:\Users\<username>\.azure\cliextensions\azure-batch-cli-extensions\azext_batch\custom.py", line 177, in create_job
    client.job.add(job, add_option, threads=multiprocessing.cpu_count()//2)
  File "C:\Users\<username>\.azure\cliextensions\azure-batch-cli-extensions\azext\batch\operations\job_operations.py", line 218, in add
    threads)
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-uej0p7bz\azure\batch\custom\patch.py", line 282, in bulk_add_collection
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-uej0p7bz\azure\batch\custom\custom_errors.py", line 30, in __init__
AttributeError: 'ValidationError' object has no attribute 'message'
To open an issue, please run: 'az feedback'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• az batch job create --template {..."outputFiles": [ { "filePattern": "..\\stdout.txt", "destination": { "container": { "path": "stdout.txt", "containerUrl": "my.blob.without.https" } }, "uploadOptions": { "uploadCondition": "TaskCompletion" } } ]...} --parameters {}

Expected Behavior

A detailed error message pointing to the validation error

Environment Summary

Windows-10-10.0.19041-SP0
Python 3.6.8
Installer: MSI

azure-cli 2.15.0

Extensions:
azure-batch-cli-extensions 6.0.0

Additional Context

Dry run mode? (emit rendered template only)

How can i just emit the rendered template without actually submitting it? This would be useful for debugging, like add a --dry-run flag or something

Release is incompatible with CLI core version 2.0.20

Attempted to add extension to current CLI installation with:
az extension add --source https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.0.0/azure_batch_cli_extensions-2.0.0-py2.py3-none-any.whl

Error message:

The extension is not compatible with this version of the CLI.
You have CLI core version 2.0.20 and this extension requires a min of 2.0.17 and max of 2.0.18.

No source distribution available on PyPI

https://pypi.python.org/pypi/azure-batch-extensions

poolEndpointConfiguration does not create inbound_nat_pool

While using the Azure CLI v2 and the extensions with a --template and --json file to create an Azure Batch Pool, I found that the cli would report and error that poolEndpointConfiguration.inbound_nat_pool could not be none. This was because my code did not use "pool". Which is how the GUI (Portal) generates it when I perform a "az batch pool show"

        "networkConfiguration": {
            "endpointConfiguration": {
                "inboundNatPools": [

The GUI would successfully add the inbound nat rule to the load balancer.

The only way I could successfully run a "az batch pool create --template --json-file" command was if my json contained the "poolEndpointConfiguration" as below. The command would succeed, but NO nat rule would be created on the load balancer. Only the default 3 rules per compute nodes.

        "networkConfiguration": {
            "poolendpointConfiguration": {
                "inboundNatPools": [

commonEnvironmentSettings being clobbered in job templates

The command az batch job create --template job.json appears to overwrite the value of commonEnvironmentSettings unconditionally with null. If the job is instead created using az batch job create --json-file job.json, then commonEnvironmentSettings is preserved.

az batch pool create command does not recognize the "interNodeCommunication": "false" element

This is autogenerated. Please review and update as needed.

Describe the bug

I have created a json template with the pool properties in it. When I try to create a new pool with a template that contains the "interNodeCommunication": "false" element, it generates the error message below.

Command Name
az batch pool create Extension Name: azure-batch-cli-extensions. Version: 6.0.0.

Errors:

Unexpect element interNodeCommunication in template
Traceback (most recent call last):
python3.6/site-packages/knack/cli.py, ln 215, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 654, in execute
    raise ex
cli/core/commands/__init__.py, ln 718, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
...
cliextensions/azure-batch-cli-extensions/azext_batch/custom.py, ln 66, in create_pool
    templates.validate_json_object(json_obj, pool)
azure-batch-cli-extensions/azext/batch/_template_utils.py, ln 1203, in validate_json_object
    raise ValueError('Unexpect element {} in template'.format(item))
ValueError: Unexpect element interNodeCommunication in template

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.
If that element is in the template, the pool creation fails. If it is not in the template, the pool creation is successful.

• Put any pre-requisite steps here...
• az batch pool create --account-name {} --account-endpoint {} --account-key {} --template {}

Expected Behavior

Environment Summary

Linux-4.15.0-1091-azure-x86_64-with-debian-stretch-sid (Cloud Shell)
Python 3.6.10
Installer: DEB

azure-cli 2.9.0 *

Extensions:
azure-batch-cli-extensions 6.0.0

Additional Context

Please close as this is a duplicate.

• Please close, this is a duplicate.

InvalidURL error in az pool create --template

Unfortunately, az pool create -template ..json does not work with the latest az cli and az batch extensions.

I get
"Error occurred in request., InvalidURL: Failed to parse: https://"dshardbatch.westeurope.batch.azure.com"/pools?api-version=2019-08-01.10.0&timeout=30"

Fail to add job if job prepare task object was created with azure.batch.models namespace

Here is code snippet:

job_prep_task = batch.models.JobPreparationTask(command_line=command, resource_files=app_resources, wait_for_success=True, user_identity=user_id) job = batch.models.JobAddParameter( job_id, batch.models.PoolInformation(pool_id=pool_id), job_preparation_task=job_prep_task ) extclient.jobs.add(job)

Traceback (most recent call last):
File "C:/Users/dapolina/PycharmProjects/AzureBatch/AzureBatchTest.py", line 426, in
main()
File "C:/Users/dapolina/PycharmProjects/AzureBatch/AzureBatchTest.py", line 405, in main
create_job(batch_client, _JOBID, _POOLID, app_resource_files)
File "C:/Users/dapolina/PycharmProjects/AzureBatch/AzureBatchTest.py", line 229, in create_job
batch_service_client.job.add(job)
File "C:\Users\dapolina\PycharmProjects\AzureBatch\venv\lib\site-packages\azext\batch\operations\job_operations.py", line 175, in add
job.job_preparation_task = models.JobPreparationTask(**job_prep_task_parameters)
TypeError: init() got an unexpected keyword argument 'additional_properties'

Automatic insertion of cmd /c in task command lines

I've run into a couple of issues with the 'command prefixing' functions ( _add_cmd_prefix(), _is_prefixed(), etc.) used by _template_utils.py.

1. For job preparation tasks at least, if the commandLine template variable does not begin with the string cmd /c, then the CLI returns with the error 'Unknown pool OS flavor: None'. It appears that os_flavor isn't always being set, and this problem is exposed by the python functions pertaining to the command prefix.

2. I would like to use a commandLine value that doesn't invoke cmd.exe, but the command prefixing code in _template_utils.py unconditionally inserts cmd /c (and strips any other command line invocation, such as the useful cmd /s /c) . In particular, I'm trying to use the AzCopy command like so:

"%ProgramFiles(x86)%/Microsoft SDKs/Azure/AzCopy/AzCopy" /S /Source:"https://containerURL" /Dest:. /SourceSAS:"?SASToken"

This works fine as is, but I can't for the life of me get it to work when it is prepended with 'cmd /c'. The environment variable %ProgramFiles(x86)% contains spaces that cause the AzCopy path to be interpreted as 'C:\Program' rather than 'C:\Program Files (x86)\Microsoft SDKs...', but only when either the /Source or/SourceSAS values are wrapped in quotes.

A solution would be to remove all "" pairs except for the one enclosing the copy command. Unfortunately, the SourceSAS value contains '&' characters, and so must be enclosed in double quotes. The only workaround I've found, other than modifying the _add_cmd_prefix() function, is to put "%ProgramFiles(x86)%/Microsoft SDKs/Azure/AzCopy/AzCopy" into a .bat file, and execute that instead, but it's not ideal. Perhaps it isn't necessary to hardcode the cmd /c prefix into the az cli?

Update: cmd /c setlocal ENABLEDELAYEDEXPANSION & "%ProgramFiles(x86)%/Microsoft SDKs/Azure/AzCopy/AzCopy" ... fixes the problem.

create Task only for existing job

How can I just create a Task only for a pre-existing job?

Release is incompatible with CLI core version 2.0.24

2.0.22 and 2.0.23 of az-cli are good; 2.0.24 does not work any more with https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.0.1/azure_batch_cli_extensions-2.0.1-py2.py3-none-any.whl

Unable to load extension 'azure-batch-cli-extensions'. Use --debug for more information.
debug points to : ModuleNotFoundError: No module named 'azure.cli.core.help_files'

Invalid value in parameters file causes hard-to-read error

1. Create a job template with an int parameter.
2. Create a parameters file that sets this parameter to "weeble".
3. Run az batch job create with this template and parameters file.

Expected result: the command fails with a message such as "ERROR: Invalid value for : 'weeble' is not an int"

Actual result: the command fails with the following error message:

Traceback (most recent call last):
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 86, in _validate_int
    value = int(value)
ValueError: invalid literal for int() with base 10: 'weeble'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python35\lib\site-packages\azure\cli\main.py", line 36, in main
    cmd_result = APPLICATION.execute(args)
  File "D:\Python35\lib\site-packages\azure\cli\core\application.py", line 203, in execute
    result = expanded_arg.func(params)
  File "D:\Python35\lib\site-packages\azure\cli\core\commands\__init__.py", line 278, in __call__
    return self.handler(*args, **kwargs)
  File "D:\Python35\lib\site-packages\azure\cli\core\commands\__init__.py", line 473, in _execute_command
    reraise(*sys.exc_info())
  File "D:\Python35\lib\site-packages\six.py", line 686, in reraise
    raise value
  File "D:\Python35\lib\site-packages\azure\cli\core\commands\__init__.py", line 450, in _execute_command
    result = op(client, **kwargs) if client else op(**kwargs)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\custom.py", line 142, in create_job
    expanded_job_object = template_utils.expand_template(template, parameters)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 1038, in expand_template
    return _parse_template(json.dumps(template_json), template_json, parameters)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 664, in _parse_template
    updated_json += _parse_template_string(string_content, template_obj, parameters)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 613, in _parse_template_string
    parsed = _parse_arm_expression(expression, template_obj, parameters)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 575, in _parse_arm_expression
    result = _parse_arm_parameter(expression[12:-2], template_obj, parameters)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 510, in _parse_arm_parameter
    return _validate_int(user_value, param_def)
  File "D:\Python35\lib\site-packages\azure\cli\command_modules\batch_extensions\_template_utils.py", line 88, in _validate_int
    raise TypeError()
TypeError

It is very hard to locate the actual problem in all this traceback spew, and when you do it doesn't tell you which parameter was the culprit.

Ideally it should emit a message for each parameter in error.

Exit code does not distinguish successful job creation from failure

1. Create a job template with an int parameter.
2. Create a parameters file which sets that parameter to "weeble".
3. Run az batch job create with this template and parameters file.

Expected result: the command exits with a nonzero exit code, allowing a batch file, shell script or other programmatic consumer to detect failure.

Actual result: the command exits with exit code 0. In addition, consumers cannot use stderr to detect failure, because the extensions write their "experimental feature" warnings to stderr regardless of whether the tool succeeds or not.

As we can only invoke the extension features via the CLI, not by calling the SDK directly, it would be really useful to be able to clearly distinguish success and failure and report the appropriate messages for each.

Installation failure

az extension add --source https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.1.0/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl -y --debug

DEBUG: Command arguments: ['extension', 'add', '--source', 'https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.1.0/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl', '--debug', '-y']
DEBUG: Event: Cli.PreExecute []
DEBUG: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fa5f68e5b70>, <function OutputProducer.on_global_arguments at 0x7fa5f640a840>, <function CLIQuery.on_global_arguments at 0x7fa5f5fa6378>]
DEBUG: Event: CommandInvoker.OnPreCommandTableCreate [<function CLILogging.remove_logger_flags at 0x7fa5f68e5bf8>]
DEBUG: Installed command modules ['acr', 'acs', 'advisor', 'appservice', 'backup', 'batch', 'batchai', 'billing', 'cdn', 'cloud', 'cognitiveservices', 'configure', 'consumption', 'container', 'cosmosdb', 'dla', 'dls', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'interactive', 'iot', 'keyvault', 'lab', 'monitor', 'network', 'profile', 'rdbms', 'redis', 'reservations', 'resource', 'role', 'servicebus', 'servicefabric', 'sql', 'storage', 'vm']
DEBUG: Loaded module 'acr' in 0.002 seconds.
DEBUG: Loaded module 'acs' in 0.002 seconds.
DEBUG: Loaded module 'advisor' in 0.001 seconds.
DEBUG: Loaded module 'appservice' in 0.003 seconds.
DEBUG: Loaded module 'backup' in 0.003 seconds.
DEBUG: Event: CommandLoader.OnLoadCommandTable []
DEBUG: Loaded module 'batch' in 0.006 seconds.
DEBUG: Loaded module 'batchai' in 0.002 seconds.
DEBUG: Loaded module 'billing' in 0.001 seconds.
DEBUG: Event: CommandLoader.OnLoadCommandTable []
DEBUG: Loaded module 'cdn' in 0.002 seconds.
DEBUG: Loaded module 'cloud' in 0.001 seconds.
DEBUG: Loaded module 'cognitiveservices' in 0.001 seconds.
DEBUG: Loaded module 'configure' in 0.001 seconds.
DEBUG: Loaded module 'consumption' in 0.001 seconds.
DEBUG: Loaded module 'container' in 0.001 seconds.
DEBUG: Loaded module 'cosmosdb' in 0.003 seconds.
DEBUG: Loaded module 'dla' in 0.003 seconds.
DEBUG: Loaded module 'dls' in 0.002 seconds.
DEBUG: Loaded module 'eventgrid' in 0.001 seconds.
DEBUG: Loaded module 'eventhubs' in 0.002 seconds.
DEBUG: Loaded module 'extension' in 0.001 seconds.
DEBUG: Loaded module 'feedback' in 0.001 seconds.
DEBUG: Loaded module 'find' in 0.001 seconds.
DEBUG: Loaded module 'interactive' in 0.000 seconds.
DEBUG: Loaded module 'iot' in 0.003 seconds.
DEBUG: Loaded module 'keyvault' in 0.003 seconds.
DEBUG: Loaded module 'lab' in 0.002 seconds.
DEBUG: Loaded module 'monitor' in 0.003 seconds.
DEBUG: Loaded module 'network' in 0.014 seconds.
DEBUG: Loaded module 'profile' in 0.002 seconds.
DEBUG: Loaded module 'rdbms' in 0.003 seconds.
DEBUG: Loaded module 'redis' in 0.001 seconds.
DEBUG: Loaded module 'reservations' in 0.001 seconds.
DEBUG: Loaded module 'resource' in 0.003 seconds.
DEBUG: Loaded module 'role' in 0.002 seconds.
DEBUG: Loaded module 'servicebus' in 0.003 seconds.
DEBUG: Loaded module 'servicefabric' in 0.001 seconds.
DEBUG: Loaded module 'sql' in 0.003 seconds.
DEBUG: Event: CommandLoader.OnLoadCommandTable []
DEBUG: Loaded module 'storage' in 0.025 seconds.
DEBUG: Loaded module 'vm' in 0.009 seconds.
DEBUG: Loaded all modules in 0.119 seconds. (note: there's always an overhead with the first module loaded)
DEBUG: Extensions directory: '/home/ceitestuser/.azure/cliextensions'
DEBUG: Event: CommandInvoker.OnPreCommandTableTruncate []
DEBUG: Event: CommandInvoker.OnPostCommandTableCreate [<function add_id_parameters at 0x7fa5f3d4f0d0>]
DEBUG: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x7fa5f3b88488>]
DEBUG: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fa5f640a8c8>, <function CLIQuery.handle_query_parameter at 0x7fa5f5fa6400>, <function handler at 0x7fa5f3aba9d8>]
DEBUG: Extension source is url? True
DEBUG: Extensions directory: '/home/ceitestuser/.azure/cliextensions'
DEBUG: Downloading https://github.com/Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.1.0/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl to /tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl
DEBUG: urllib3.connectionpool : Starting new HTTPS connection (1): github.com
DEBUG: urllib3.connectionpool : https://github.com:443 "GET /Azure/azure-batch-cli-extensions/releases/download/azure-batch-cli-extensions-2.1.0/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl HTTP/1.1" 302 None
DEBUG: urllib3.connectionpool : Starting new HTTPS connection (1): github-production-release-asset-2e65be.s3.amazonaws.com
DEBUG: urllib3.connectionpool : https://github-production-release-asset-2e65be.s3.amazonaws.com:443 "GET /83587333/4fd5e3aa-f61c-11e7-88b7-b32a3d567e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180406%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180406T200406Z&X-Amz-Expires=300&X-Amz-Signature=61a57429c6679b4b10a328659385c7c9695c92cb352f48d758c83d1ce1484c6e&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dazure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl&response-content-type=application%2Foctet-stream HTTP/1.1" 200 12300
DEBUG: Downloaded to /tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl
DEBUG: Validating the extension /tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl
DEBUG: Extension compatibility result: is_compatible=True cli_core_version=2.0.30 min_required=2.0.24 max_required=2.1.0
DEBUG: Validation successful on /tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl
DEBUG: Linux distro check: Reading from: /etc/apt/sources.list.d/azure-cli.list
DEBUG: Linux distro check: Found in list file: xenial
DEBUG: Linux distro check: Reported by API: xenial
DEBUG: Executing pip with args: ['install', '--target', '/home/ceitestuser/.azure/cliextensions/azure-batch-cli-extensions', '/tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl']
DEBUG: Running: ['/opt/az/bin/python3', '-m', 'pip', 'install', '--target', '/home/ceitestuser/.azure/cliextensions/azure-batch-cli-extensions', '/tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']
DEBUG: Traceback (most recent call last):
File "/opt/az/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/opt/az/lib/python3.6/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/opt/az/lib/python3.6/runpy.py", line 109, in _get_module_details
import(pkg_name)
File "/opt/az/lib/python3.6/site-packages/pip/init.py", line 28, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/opt/az/lib/python3.6/site-packages/pip/vcs/mercurial.py", line 9, in
from pip.download import path_to_url
File "/opt/az/lib/python3.6/site-packages/pip/download.py", line 36, in
from pip.utils.glibc import libc_ver
File "/opt/az/lib/python3.6/site-packages/pip/utils/glibc.py", line 4, in
import ctypes
File "/opt/az/lib/python3.6/ctypes/init.py", line 7, in
from _ctypes import Union, Structure, Array
ImportError: libffi.so.5: cannot open shared object file: No such file or directory

DEBUG: Command '['/opt/az/bin/python3', '-m', 'pip', 'install', '--target', '/home/ceitestuser/.azure/cliextensions/azure-batch-cli-extensions', '/tmp/tmp_t4vvl26/azure_batch_cli_extensions-2.1.0-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']' returned non-zero exit status 1.
DEBUG: Pip failed so deleting anything we might have installed at /home/ceitestuser/.azure/cliextensions/azure-batch-cli-extensions
ERROR: An error occurred. Pip failed with status code 1. Use --debug for more information.

Ran on a linux data science vm on Azure.

Am I missing some development headers or something?

Job template wont create task if i add resource files collection

Am trying to debug an issue. I am trying to submit a job template, cut down to a very basic template to see what is causing the issue. The following job template creates the job but with no task and there is no error returned from the CLI.

        "taskFactory": {
            "type": "taskCollection",
            "tasks": [
                {
                    "id": "ffmpeg",
                    "commandLine": "cmd.exe /c echo hello",
                    "resourceFiles": [
                        {
                            "source": {
                                "fileGroup": "[parameters('inputData')]"
                            }
                        }
                    ]
                }
            ]
        }

If i remove the "resourceFiles" property, the job submits and the task is created.

        "taskFactory": {
            "type": "taskCollection",
            "tasks": [
                {
                    "id": "ffmpeg",
                    "commandLine": "cmd.exe /c echo hello"
                }
            ]
        }

Any idea what is going on here?
Andrew

az batch pool create command isn't running as expected.

Hi,
I tried running az batch pool create --template pool.json, but what I get in return is
"az: error: unrecognized arguments: --template pool.json",
Has the CLI version been updated and this article wasn't?
my pool.json file has the following:
"pool": {
"type": "Microsoft.Batch/batchAccounts/pools",
"apiVersion": "2016-12-01",
"properties": {
"id": "helloworld-pool",
"cloudServiceConfiguration": {
"osFamily": "4",
"targetOSVersion": "*"
},
"vmSize": "small",
"targetDedicatedNodes": 1
}
}
}
my az batch cli version is 3.3.0
Detailed error under --debug:
Command arguments: ['batch', 'pool', 'create', '--template', 'pool.json', '--debug']
Event: Cli.PreExecute []
Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x03F02E40>, <function OutputProducer.on_global_arguments at 0x0443EA98>, <function CLIQuery.on_global_arguments at 0x04226468>]
Event: CommandInvoker.OnPreCommandTableCreate [<function CLILogging.remove_logger_flags at 0x03F02E88>]
Installed command modules ['acr', 'acs', 'advisor', 'ams', 'appservice', 'backup', 'batch', 'batchai', 'billing', 'cdn', 'cloud', 'cognitiveservices', 'configure', 'consumption', 'container', 'cosmosdb', 'dla', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'interactive', 'iot', 'keyvault', 'lab', 'maps', 'monitor', 'network', 'policyinsights', 'profile', 'rdbms', 'redis', 'reservations', 'resource', 'role', 'servicebus', 'servicefabric', 'sql', 'storage', 'vm']
Loaded module 'acr' in 0.004 seconds.
Loaded module 'acs' in 0.004 seconds.
Loaded module 'advisor' in 0.002 seconds.
Event: CommandLoader.OnLoadCommandTable []
Loaded module 'ams' in 0.006 seconds.
Loaded module 'appservice' in 0.005 seconds.
Loaded module 'backup' in 0.003 seconds.
Event: CommandLoader.OnLoadCommandTable []
Loaded module 'batch' in 0.010 seconds.
Loaded module 'batchai' in 0.004 seconds.
Loaded module 'billing' in 0.002 seconds.
Event: CommandLoader.OnLoadCommandTable []
Loaded module 'cdn' in 0.003 seconds.
Loaded module 'cloud' in 0.002 seconds.
Loaded module 'cognitiveservices' in 0.002 seconds.
Loaded module 'configure' in 0.002 seconds.
Loaded module 'consumption' in 0.006 seconds.
Loaded module 'container' in 0.002 seconds.
Loaded module 'cosmosdb' in 0.004 seconds.
Loaded module 'dla' in 0.005 seconds.
Loaded module 'dls' in 0.004 seconds.
Loaded module 'dms' in 0.002 seconds.
Loaded module 'eventgrid' in 0.002 seconds.
Loaded module 'eventhubs' in 0.003 seconds.
Loaded module 'extension' in 0.002 seconds.
Loaded module 'feedback' in 0.001 seconds.
Loaded module 'find' in 0.001 seconds.
Loaded module 'interactive' in 0.001 seconds.
Loaded module 'iot' in 0.005 seconds.
Loaded module 'keyvault' in 0.005 seconds.
Loaded module 'lab' in 0.004 seconds.
Loaded module 'maps' in 0.002 seconds.
Loaded module 'monitor' in 0.006 seconds.
Loaded module 'network' in 0.019 seconds.
Loaded module 'policyinsights' in 0.003 seconds.
Loaded module 'profile' in 0.283 seconds.
Loaded module 'rdbms' in 0.004 seconds.
Loaded module 'redis' in 0.003 seconds.
Loaded module 'reservations' in 0.002 seconds.
Loaded module 'resource' in 0.014 seconds.
Loaded module 'role' in 0.003 seconds.
Loaded module 'servicebus' in 0.005 seconds.
Loaded module 'servicefabric' in 0.003 seconds.
Loaded module 'sql' in 0.006 seconds.
Event: CommandLoader.OnLoadCommandTable []
Loaded module 'storage' in 0.020 seconds.
Loaded module 'vm' in 0.011 seconds.
Loaded all modules in 0.477 seconds. (note: there's always an overhead with the first module loaded)
Extensions directory: 'C:\Users\malhmoud.azure\cliextensions'
Event: CommandInvoker.OnPreCommandTableTruncate []
Event: CommandLoader.OnLoadArguments []
Event: CommandInvoker.OnPostCommandTableCreate [<function add_id_parameters at 0x044060C0>, <function register_global_subscription_parameter..add_subscription_parameter at 0x044560C0>]
Event: CommandInvoker.OnCommandTableLoaded []
Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x04653DF8>]
az: error: unrecognized arguments: --template pool.json
usage: az [-h] [--verbose] [--debug] [--output {json,jsonc,table,tsv}]
[--query JMESPATH]
{batch} ...