azure-samples / azure-batch-samples Goto Github PK
View Code? Open in Web Editor NEWAzure Batch and HPC Code Samples
License: Other
Azure Batch and HPC Code Samples
License: Other
When using the latest version of Microsoft.WindowsAzure.Storage (7.1.2.0) and Microsoft.Azure.Batch (5.0.0.0), this solution does not compile.
Issue #1 - "WhenAllAsync" does not exist in the new Batch DLL:
bool timedOut = await taskStateMonitor.WhenAllAsync(tasks, TaskState.Completed, timeout);
Issue #2 - CloudBlockBlob does not have a method that matches this signature:
await blobData.UploadFromFileAsync(filePath, FileMode.Open);
As per the doc, it will wait until its members has reached a desired state at least once.
Here comes my code (I know its dirty) ---
var tasksDetail = new ODATADetailLevel(selectClause: "id,executionInfo");
var tasks = await batchClient.JobOperations.ListTasks(jobId, tasksDetail).ToListAsync();
Parallel.ForEach(tasks, (task) =>
{
if (task == null)
{
Logger.Warn("Invalid Task info");
return;
}
if (task.State == TaskState.Completed)
{
Logger.Info($"Task '{task.Id}' is already completed, skipping monitoring");
return;
}
// Wait for 'Running' state
batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();
task.RefreshAsync(new ODATADetailLevel(selectClause: "id,nodeInfo")).Wait();
Logger.Debug($"Task '{task.Id}' is running on Node '{task.ComputeNodeInformation?.ComputeNodeId}' now");
// Wait for 'Completed' state
batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Completed, timeout).Wait();
task.RefreshAsync(new ODATADetailLevel(selectClause: "id,executionInfo")).Wait();
Logger.Info($"Task '{task.Id}' is completed now, duration: {task.ExecutionInformation.EndTime - task.ExecutionInformation.StartTime:c}");
});
This code snippet works well. But when I tried to switch 'Running' and 'Completed' section (like below), it blocked while waiting for 'Running' state. IMHO, this behavior differs from what the document describes ---
the task first became 'Running' and then 'Completed', so after it reached 'Completed' state, it must have already reached 'Running' and should not be blocked at batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();
.
// Wait for 'Completed' state
batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Completed, timeout).Wait();
task.RefreshAsync(new ODATADetailLevel(selectClause: "id,executionInfo")).Wait();
Logger.Info($"Task '{task.Id}' is completed now, duration: {task.ExecutionInformation.EndTime - task.ExecutionInformation.StartTime:c}");
// Wait for 'Running' state
batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();
task.RefreshAsync(new ODATADetailLevel(selectClause: "id,nodeInfo")).Wait();
Logger.Debug($"Task '{task.Id}' is running on Node '{task.ComputeNodeInformation?.ComputeNodeId}' now");
There seems to be an issue when using the blobep switch. Running like this
blobxfer.py --storageaccountkey test storage file.mp4 --upload --blobep blob.core.chinacloudapi.cn
Throws this error
Traceback (most recent call last):
File "./blobxfer.py", line 1310, in
main()
File "./blobxfer.py", line 980, in main
blobskipdict = get_blob_listing(blob_service, args)
File "./blobxfer.py", line 664, in get_blob_listing
maxresults=_MAX_LISTBLOBS_RESULTS)
File "./blobxfer.py", line 525, in azure_request
return req(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 485, in list_blobs
response = self._perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 143, in _perform_request
resp = self._filter(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 132, in _perform_request_worker
return self._httpclient.perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 192, in perform_request
self.send_request_headers(connection, request.headers)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 171, in send_request_headers
connection.endheaders()
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 776, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1157, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known
Even tho I can resolve the storage URL on the same server that this is running on. Any help would be great.
Currently one cannot specify any details of the user account used to execute the coordination command. It would be very useful to allow this to be set to an administrator account for configuring system services (e.g. a shared filesystem for the task)
Current work around is adding a script and making it setuid root with a pool start task, but that's horrible.
As I was using SAS to upload a 1Gb tsv file to a container, it fail. Is that any problem to do the upload or i am missing something need to do?
i have some blob name like "https://xxx.blob.core.windows.net/my-container/Top%2024%20P/BV.json"
when i use blobxfer it error like this
remote file TOP 20 P/ length: 0 bytes, md5: 1B2M2Y8AsgTpgAmY7PhCfg==
Traceback (most recent call last):
File "/usr/local/bin/blobxfer", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 1245, in main
blob, blobdict[blob][0], blobdict[blob][1], False)
File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 840, in generate_xferspec_download
lmd5 = compute_md5_for_file_asbase64(localfile)
File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 711, in compute_md5_for_file_asbase64
with open(filename, 'rb') as filedesc:
IsADirectoryError: [Errno 21] Is a directory: '/var/www/html/my-container/TOP 20 P/'
I can't download a file from task node. I'am getting this exeception:
Microsoft.Azure.Batch.Common.BatchException: Operation returned an invalid status code 'Forbidden' ---> Microsoft.Azure.Batch.Protocol.Models.BatchErrorException: Operation returned an invalid status code 'Forbidden'
at Microsoft.Azure.Batch.Protocol.FileOperations.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.Protocol.BatchRequestBase2.<ExecuteRequestWithCancellationAsync>d__42.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Batch.Protocol.BatchRequestBase
2.d__40.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.Azure.Batch.Protocol.BatchRequestBase2.<ExecuteRequestAsync>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Batch.ProtocolLayer.<ProcessAndExecuteBatchRequest>d__84
1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.ProtocolLayer.d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.JobOperations.d__39.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.Models.TaskModel.d__58.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\Models\TaskModel.cs:line 384
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.Models.TaskModel.d__56.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\Models\TaskModel.cs:line 356
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.ViewModels.MainViewModel.d__215.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\ViewModels\MainViewModel.cs:line 1728
Request Information
ClientRequestId:
RequestId:6cbffd54-1fc7-48cd-a56f-4ec3a5ab9cae
HttpStatusCode:Forbidden
StatusMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
There is no statistics for jobs. They are visible for individual tasks though.
Hi,
the problems from the closed issue 131 seems to persist. Both using the BatchExplorer and the usual batch rest API, the tasks have statistics, but the jobs don't.
Hi guys,
I'm trying to use Azure Batch to run a Job once a day. The input will be three files with different sets of data, that I would like to process in parallel.
To achieve this I'm trying to use JobSchedules, but I don't know how to create one Task for each file using a JobManagerTask. I didn't find any documentation about JobSchedule that covers this either.
Do you have some examples that I can use ?
Thanks!
Hi,
I have task with 100 jobs and I want to download all the stdout.txt files.
I tried to do it using this python command :
txtfile = urllib2.urlopen('https://barilanbatchhpcuseast.eastus.batch.azure.com/jobs/Job1/tasks/task1_0_1/files/stdout.txt').read()
But i got the following error : urllib2.HTTPError: HTTP Error 400: A query parameter that's mandatory for this request is not specified
What can be the problem?
hey. is it possible to get blobxfer that when uploading a directory from my local machine, to delete remote files which are no longer local present? this way i could use blobxfer for backup purposes and make sure the remote tree is identical to my local one
I just tried to run the python sample with azure 2.0.0rc5.
I get the following error
me@camnetwork:/tmp/azure-batch-samples/Python/Batch/article_samples$ python python_tutorial_client.py
Sample start: 2016-07-29 12:07:48Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/python_tutorial_task.py to container [application]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata1.txt to container [input]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata2.txt to container [input]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata3.txt to container [input]...
Traceback (most recent call last):
File "python_tutorial_client.py", line 480, in
base_url=_BATCH_ACCOUNT_URL)
File "/tmp/sandbox/local/lib/python2.7/site-packages/azure/batch/batch_service_client.py", line 137, in init
self._serialize = Serializer(client_models)
TypeError: init() takes exactly 1 argument (2 given)
I am trying to run python_tutorial_client.py
and it says:
ModuleNotFoundError: No module named 'common.helpers'
I guess the problem is in this line:
import common.helpers # noqa
I tried installing the common
module with pip install common
, which didn't help although the library common
got installed.
It was confusing to find out that there is this library named common
, but there's also azure.common
from the Azure SDK.
Anyways, the import in the python file did not start working even after installing azure.common
and changing the non-working line to import azure.common.helpers
or from azure.common import helpers
(the last one gives ImportError: cannot import name 'helpers'
).
I am using Python 3.6.2 on Windows 10.
Receiving "AADSTS90093: User cannot consent to web app requesting user impersonation as an app permission." Is Administrative access required for AAD tenant to run this project?
it will prompt the "there is unsupport parameter" error when throw out the create_pool rest call.
The storage account key is taken in as a command line parameter only for blobxfer. Which implies that the storageaccountkey which is a secret for the account is visible in the ps aux output, for big uploads. This a security issue, other clients(e.g. the aws client for s3) get around this issue by accepting the secrets as a environment variable or a config file.
Thoughts?
cc @Dagrol
If you add a resource file to the coordination command for a multi-instance task then this is not downloaded to the task working directory. Instead, it is added to the parent of this (where the task stderr.txt etc are placed). This wouldn’t be a problem except that the working directory for coordination command execution is the task working directory. (Checked by setting my command string to be pwd && sh ../coordination.sh
).
As a work-around you can just cd ..
first, but it's quite annoying.
This is for the agent SKU ID "batch.node.centos 7", in case this is specific to that.
It seems that Batch Explorer has a bug that when trying to delete a pool nothing happens. A colleague confirmed that. Deleting the pool via Azure Portal works, and also in my own Python code using the azure-batch library.
Is there a way to get the __version__
from the Batch Python client lib? e.g. 2.0.1? If not, it would be helpful to implement __version__
in the Batch Python client lib so calling code can detect and work around SDK changes.
Hi,
i set the proxy for the blob_client (python) with
blob_client.set_proxy("REDACTED",8080)
And for the storage everything is working fine. But for the batch_client in the creation of the pool i just wait and after some time i get a timeout exception from the urllib3 library.
Using the python blobxfer, many times and with many large files (but always smaller than 190G), I receive the error "The request body is too large and exceeds the maximum permissible limit"
I can upload some large files, but is not possible for others files, doesn't matter how many retries or the number of workers.
Also, I have this problem when the local resource to upload is a folder (a folder containing one single file), but if I try to upload the same file specifying the file, the error doesn't appear.
I am using the following python packages:
pip freeze
azure==1.0.2
azure-common==1.0.0
azure-mgmt==0.20.1
azure-mgmt-common==0.20.0
azure-mgmt-compute==0.20.0
azure-mgmt-network==0.20.1
azure-mgmt-nspkg==1.0.0
azure-mgmt-resource==0.20.1
azure-mgmt-storage==0.20.0
azure-nspkg==1.0.0
azure-servicebus==0.20.1
azure-servicemanagement-legacy==0.20.1
azure-storage==0.20.2
elasticsearch==2.2.0
futures==3.0.3
python-dateutil==2.4.2
requests==2.9.1
six==1.10.0
urllib3==1.14
wheel==0.26.0
And blobxfer.py v0.9.9.5
As an example:
azure blobxfer parameters [v0.9.9.5]
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: archive/
remote resource: None
max num of workers: 4
timeout: None
storage account: ---
use SAS: False
upload as page blob: False
auto vhd->page blob: False
container: ---
blob container URI: https://---.blob.core.windows.net/---
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: True
keep mismatched MD5: False
recursive if dir: True
keep root dir on up: False
collate to: disabled
script start time: 2016-01-29 10:00:09
g--.tar.gz md5: lhD55kDeLW9uh4PXtJ7LhQ==
detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
xfer progress: [ ] 0.00% 0.00 blocks/min The request body is too large and exceeds the maximum permissible limit.
RequestBodyTooLarge
The request body is too large and exceeds the maximum permissible limit.
RequestId:a6af7e74-0001-00f6-0474-5ae0d5000000
Time:2016-01-29T09:06:25.4964043Z100000
ls -lah
-rw-r--r-- 1 root root 100G Jan 26 15:46 g--.tar.gz
create container and skip on MD5 match are both True by default.
So, when running a command like:
blobxfer storage_account new_container /path/to/dir --upload --storageaccountkey=${AZURE_STORAGE_ACCOUNT_KEY}
the upload fails because the script, first tries to retrieve the blob listings from the container. But it will raise an azure.common.AzureMissingResourceHttpError and stop the execution.
I have gotten all of the other samples to work with the exception of Python Azure Batch Sample 4. The pool gets created successfully, it is the key generation step that keeps failing with file not found. I replaced ssh-keygen with ls command only and it still failed. I think I have the latest version, maybe not?
Figured it out, please disregard,
Getting strange error when uploading
.7/dist-packages/requests/packages/urllib3/util/ssl_.py:100: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
xfer progress: [>>>>>>>>>>>> ] 40.00% 3275.82 blocks/min The value for one of the HTTP headers is not in the correct format.
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:9ab12df3-0001-0036-7f50-200b01000000
Time:2015-11-16T09:21:10.3991192Z</Message><HeaderName>x-ms-blob-content-length</HeaderName><HeaderValue>0</HeaderValue></Error>
i'm just testing uploading using blobxfer. I have no issue downloading data, but can't get the upload to work. I'm sure the SAS has write permission on the container, and I've also just used the storage account key to no effect. Here's what I'm running (foo.txt isn't empty).
$ blobxfer account container foo.txt --no-overwrite --numworkers 1 --saskey $BLOB_SAS_KEY --upload --no-createcontainer --no-computefilemd5
=======================================
azure blobxfer parameters [v0.10.0]
=======================================
platform: Darwin-15.4.0-x86_64-i386-64bit
python interpreter: CPython 2.7.11
package versions: az.common=1.1.1 az.sml=0.20.2 az.stor=0.30.0 crypt=1.3.1 req=2.9.1
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: foo.txt
include pattern: None
remote resource: None
max num of workers: 1
timeout: None
storage account: account
use SAS: True
upload as page blob: False
auto vhd->page blob: False
container: container
blob container URI: https://account.blob.core.windows.net/container
compute file MD5: False
skip on MD5 match: True
chunk size (bytes): 4194304
create container: False
keep mismatched MD5: False
recursive if dir: True
component strip on up: 1
remote delete: False
collate to: disabled
local overwrite: False
encryption mode: disabled
RSA key file: disabled
RSA key type: disabled
=======================================
script start time: 2016-05-05 18:18:45
If I kill the job, it looks to be hung-up making the ssl connection:
script start time: 2016-05-05 18:18:45
^CTraceback (most recent call last):
File "/Users/clackner/anaconda/envs/poc/bin/blobxfer", line 11, in
sys.exit(main())
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 2118, in main
blobskipdict = get_blob_listing(blob_service[0], args)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1560, in get_blob_listing
container_name=args.container, marker=marker, include=incl)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1373, in azure_request
return req(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 538, in list_blobs
requests.get, url=url, params=reqparams, timeout=self.timeout)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1373, in azure_request
return req(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, *_send_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 180, in recv
data = self.connection.recv(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1319, in recv
result = _lib.SSL_read(self._ssl, buf, bufsiz)
KeyboardInterrupt
Could you please provide more information on how to prepare these files?
What they actually contain and How did you generate these files?
Is Intel MPI a part of CentOS images?
Thanks,
lumirosl @ MSFT
As per this guide, application litware was created as demo/example, can you please release its source code for developers to learn and debug?
The Python article tutorial currently monitors tasks for completion - however does not monitor whether the nodes on the pool successfully start up - if start task fails then the tasks will never start leaving the script to poll for 20 minutes then exit without helpful feedback to the user.
First and foremost, I really like blobxfer. I've only just discovered it but it really hits a sweet spot for a handful of use cases I have right now. I came across a small issue in the README (https://github.com/Azure/azure-batch-samples/blob/master/Python/Storage/README.rst ).
Should:
blobxfer mystorageacct container0 mylocaldir --remoteresource .
Be:
blobxfer mystorageacct container0 mylocaldir --download --remoteresource .
The former wrote various temp files to be container and it ended up being a bit confusing.
Getting the following error with C# Sample in jobs.cs
CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(
new StorageCredentials(
accountSettings.StorageAccountName,
accountSettings.StorageAccountKey),
accountSettings.StorageServiceUrl,
useHttps: true);
System.ArgumentException: 'Primary and secondary location URIs in a StorageUri must point to the same resource.
Parameter name: secondaryUri'
Getting TypeError on the xrange command. numworkers appears to be set to 64 (default), maybe a problem with assigning maxworkers here?
maxworkers = min([args.numworkers, nstorageops])
Error below...
=======================================
azure blob xfer parameters
=======================================
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: libfoo.deb
remote resource: None
max num of workers: 64
timeout: None
storage account: foo
use SAS: True
upload as page blob: False
auto vhd->page blob: False
container: packages
blob container URI: https://foo.blob.core.windows.net/packages
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: False
keep mismatched MD5: False
recursive if dir: True
keep root dir on up: False
=======================================
script start time: 2015-09-04 19:50:17
foo.deb md5: ThWXPkbFTO1HAeddfZUv8A==
foo-dev_.deb->foo-dev.deb: local ThWXPkbFTO1HAeddfZUv8A== remote ThWXPkbFTO1HAeddfZUv8A== ->match, skipping upload
performing None put blocks and 0 put block lists
Traceback (most recent call last):
File "blobxfer.py", line 1406, in <module>
main()
File "blobxfer.py", line 1205, in main
for _ in xrange(maxworkers):
TypeError: an integer is required
Failure occurred communicating with
Please can you provide a Batch Explorer tool for our Operations team to use.
Thanks
If the local file has a question mark in its name the upload can stack trace:
$ blobxfer --version
0.9.9.8
$ mkdir x
$ echo x > 'x/xxx?.xxx'
$ blobxfer --storageaccountkey 'redacted' redacted container x
Traceback (most recent call last):
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 877, in run
offset, bytestoxfer, encparam, flock, filedesc)
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 1002, in putblobdata
content_md5=contentmd5, timeout=self.timeout)
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 1309, in azure_request
return req(_args, *_kwargs)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/blob/blobservice.py", line 2366, in put_block
self._perform_request(request)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/storageclient.py", line 178, in _perform_request
_storage_error_handler(ex)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/_serialization.py", line 25, in _storage_error_handler
return _general_error_handler(http_error)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/_common_error.py", line 82, in _general_error_handler
raise AzureHttpError(message, http_error.status)
AzureHttpError: An HTTP header that's mandatory for this request is not specified.
MissingRequiredHeader
An HTTP header that's mandatory for this request is not specified.
RequestId:redacted
Time:2016-01-24T17:47:06.9200167Zx-ms-blob-type
Removing the '?' makes the upload succeed.
Hello Guys
When the new person is downloading the batch explorer code and trying to run first time they are getting exceptions. because the code is trying to de-serialize the xml node and since file is empty created with zero bytes , the code was exploding.
I manually created the blank account xml file in the app data directory. C:\Users\AppData\Local\Microsoft\Azure Batch Explorer with following code. then ran again and then the app worked.
`
-
`
So please fix the code here to not only check if file exists and if it is not zero bytes then de-serialize otherwise write the xml lines first and then de-serialize.
private async Task LoadAccountsAsync()
{
string accountFileName = GetAccountFileName();
if (!File.Exists(accountFileName) OR ALSO CHECK IF FILE EXISTS AS ZERO BYTES )
{
using (var myFile = File.Create(accountFileName))
{
}
await this.SaveAccountsAsync().ConfigureAwait(false);
}
//Try first to deserialize using the new account deserialization mechanism, if that fails, fall back to the old way
bool writeBackToFile = false;
try
{
XmlSerializer x = new XmlSerializer(typeof(DefaultAccountSerializationContainer));
using (StreamReader reader = new StreamReader(accountFileName))
{
object deserializedObject = x.Deserialize(reader); //TODO: This is not technically async...
DefaultAccountSerializationContainer accountContainer = (DefaultAccountSerializationContainer)deserializedObject;
List<Account>
The C# MPI Article sample includes a .vcxproj example MSMPI application. This project is currently ignored by CI. We should investigate how to best handle it. MSBuild does support the .vcxproj format, but we need to see how easy it is to integrate with our AppVeyor CI build.
The samples and accompanying tutorial at https://docs.microsoft.com/en-us/azure/batch/batch-python-tutorial need to be updated for v3.0.0 of the Azure Batch Client for Python. For example, target_dedicated and current_dedicated on CloudPool, AddPoolParameter and PoolSpecification have been renamed to target_dedicated_nodes and current_dedicated_nodes.
It would be awesome to have these updated for the 4.0 SDK.
Thanks for all your work/help!
Stripping local directory structure or set Azure blob storage destination directory on upload as AzCopy
AzCopy /Source:
Time:2015-11-17T07:44:37.7457230Z
The specified blob or block content is invalid.
InvalidBlobOrBlock
The specified blob or block content is invalid.
RequestId:445ea4c3-0001-00ed-330b-217d34000000
Time:2015-11-17T07:44:32.1473355Z
Node shutdown through the API is inconsistent. Some instances have failed to shutdown for up to 10 times, then succeeded at shutting down. The classic and the new portal GUI's don't help either.
+ Getting virtual machines
+ Shutting down VM
info: vm shutdown command OK
info: Executing command vm shutdown
+ Getting virtual machines
+ Shutting down VM
error: InternalError : The server encountered an internal error. Please retry the request.
info: Error information has been recorded to /Users/buraky/.azure/azure.err
error: vm shutdown command failed
i have some zero byte file in my blob.
when i try to download my blob by 'blobxfer.py', it show error as below.
remote file graph/FER-F.txt length: 0 bytes, md5:
Traceback (most recent call last):
File "blobxfer.py", line 1404, in <module>
main()
File "blobxfer.py", line 1173, in main
blob, blobdict[blob][0], blobdict[blob][1], False)
File "blobxfer.py", line 820, in generate_xferspec_download
filedesc.seek(contentlength - 1)
IOError: [Errno 22] Invalid argument
It seems 'User subscription' batch have the only way to connect through AAD https://blogs.technet.microsoft.com/windowshpc/2017/03/17/azure-batch-vnet-and-custom-image-support-for-virtual-machine-pools/
How can I do it with Batch Explorer?
This check:
if args.subscriptionid is not None and args.managementcert is None: raise ValueError( 'cannot specify subscription id without a management cert')
is unnecessary for the following usage:
python $blobxfer_location/blobxfer.py --timeout 1800 --numworkers $NUM_WORKERS --subscriptionid $subscriptionid --storageaccountkey $storage_account --download --remoteresource
Receiving error when transferring blob:
Traceback (most recent call last):
File "blobxfer.py", line 1310, in
main()
File "blobxfer.py", line 980, in main
blobskipdict = get_blob_listing(blob_service, args)
File "blobxfer.py", line 664, in get_blob_listing
maxresults=_MAX_LISTBLOBS_RESULTS)
File "blobxfer.py", line 547, in azure_request
raise exc
AttributeError: 'module' object has no attribute '_parse_blob_enum_results_list'
This problem starting coincides with the latest release of the Azure SDK for Python (1.0) and the problem goes away when rolling back to 0.11.1 version.
Look at:
https://github.com/Azure/azure-storage-python/releases
Interesting that the namespace has changed from azure.storage to azure.storage.blob -- could just be that?
Thanks,
John
Hi,
Maybe I'm wrong but I think that makes no difference when the option --keeprootdir is enabled or not.
I am using a modified version of blobxfer, in my version I can select with a number the start point of the local path in order to create the virtual path on azure.
Example:
/home/user/azure/ folder1
folder2/subfolder1
With my version, selecting 3 as start point, I can create the following virtual path on azure:
https://xxx.blob.core.windows.net/container/folder1
https://xxx.blob.core.windows.net/container/folder2/subfolder1
Instead of:
https://xxx.blob.core.windows.net/container/home/user/azure/folder1
https://xxx.blob.core.windows.net/container/home/user/azure/folder2/subfolder1
I can pull a request and add this option to the flag --keeprootdir like:
--keeprootdir [STARTPOINT] (with 1 as default)
Regards,
After downloading the latest fix for the issue that was recently closed with #82, I ran into another issue:
HTTPSConnectionPool(host='sampleblobstore.blob.core.windows.net', port=443):
Max retries exceeded with url: /xxxxxxxxxxxxuyaidlajdsflaksfcc0dc9fe4-671141903_p.jpg?blockid=MDAwMDAwMDA=&comp=block
(Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x44834850>:
Failed to establish a new connection: [Errno -2] Name or service not known',))
This is a largish folder that I've been trying to upload to Azure. All the other smaller folders were uploaded fine.
My command I used:
blobxfer.py --storageaccountkey $AZURE_STORAGE_ACCESS_KEY $AZURE_STORAGE_ACCOUNT $AZURE_CONTAINER_NAME .
How do I set blob properties and metadata at file level ? and also if I want to set it for all the files under a folder ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.