microsoft/mcw-high-performance-computing

MCW High Performance Computing

This workshop is archived and is no longer being maintained. Content is read-only.

High Performance Computing

ThoughtRender provides image and video processing services to many industries -- marketing and advertising, retail, medical, and media and entertainment. You know those furniture magazines you love to look at? Most of the pictures in these magazines are generated by the HPC compute clusters at ThoughtRender. Their unique ability to tie together industry solutions into a reliable service for their customers (e.g., furniture magazine designers! 3D animated movie makers!), together with their in-house knowledge on image and video, is a quality service that sets them apart from competitors.

They currently operate their own, on-premises services (with their own on-premises HPC clusters, and other IT infrastructure), but their success has led to challenges in terms of their growth, and ability to scale to new jobs, and new industries. They are curious about cloud and think that not only could it help them scale, but help them improve costs and pass savings to customers, address the seasonality of rendering demand (by only paying for resources when actually used) and to take advantage of new technologies, such as the latest GPUs.

They believe that bursting jobs to the cloud could help them deliver bigger jobs, or regular jobs quicker for their customers (e.g., in 1 day, instead of 5 days). They intend to pilot a solution to address this.

June 2020

Target audience

• Cloud Architects
• Developers
• HPC Engineers

Abstracts

Workshop

In this workshop, you will set up and configure a scale-out media processing architecture using Azure Batch. You'll discover how to use High Performance Computing (scale out compute, embarrassingly parallel processing) techniques without having to write a lot of code, and learn how these tasks can be accomplished declaratively.

At the end of this workshop, you will have a deeper understanding of how to use the core capabilities of Azure Batch, understand how to author Custom Pool and Job templates, work with Job input and output files, author Batch auto-scale formulas, leverage Batch Explorer and the Azure Portal for management and monitoring, and use Marketplace applications to simplify common big compute tasks, such as 3D rendering.

Whiteboard design session

In this whiteboard design session, you will work with a group to design a scale-out media processing solution using High Performance Computing (HPC) techniques in Azure.

At the end of this session, you will be better able to design and recommend High Performance Computing solutions that are highly scalable and can be configured through declarative means as opposed to large amounts of complicated application code. You will also learn how to manage and monitor these solutions to ensure predictable outcomes effectively.

Hands-on lab

In this hands-on lab, you will implement High Performance Computing (HPC) workloads targeted at 3D rendering and media processing in Azure using Azure Batch and Azure Storage.

At the end of this hands-on lab, you will be better able to deploy Azure Batch and an Azure Batch Pool consisting of Linux Virtual Machines. You will be able to configure the pool of VMs to run the FFmpeg app to process videos across the VM nodes in the pool. You will be able to do this using Batch templates and files, which enable you to perform scale-out execution of command-line executables (in this case FFmpeg) without having to write any code.

Azure services and related products

• Azure Batch
• Azure Batch Explorer
• Virtual Machines
• Azure Storage
• Azure Storage Explorer
• Import/Export Service

Related references

• MCW

mcw-high-performance-computing's Issues

Error when running a job

Hi when I try to run a job (Exercice #3 Task#2)
with the the command:
sudo az batch job create --template ~/work/job.json --account-name mybatchaccount --account-endpoint mybatchdemo.westeurope.batch.azure.com

I get the following error:

Unable to deserialize to ExtendedJobParameter: Unable to deserialize {'blob_source': '{url}', 'file_path': '{fileName}', 'file_mode': None, 'source': None} into model <class 'azext.batch.models.extended_resource_file.ExtendedResourceFile'>. __init__() takes 1 positional argument but 4 were given
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1397, in _instantiate_model
    response_obj = response(**kwargs)
  File "/home/admindev/.azure/cliextensions/azure-batch-cli-extensions/azext/batch/models/extended_resource_file.py", line 46, in __init__
    super(ExtendedResourceFile, self).__init__(blob_source, file_path, file_mode)
TypeError: __init__() takes 1 positional argument but 4 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/admindev/.azure/cliextensions/azure-batch-cli-extensions/azext/batch/operations/job_operations.py", line 81, in jobparameter_from_json
    job = models.ExtendedJobParameter.from_dict(json_data)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 318, in from_dict
    return deserializer(cls.__name__, data, content_type=content_type)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1230, in __call__
    return self._deserialize(target_obj, data)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1296, in _deserialize
    value = self.deserialize_data(raw_value, attr_desc['type'])
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1458, in deserialize_data
    return self._deserialize(obj_type, data)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1296, in _deserialize
    value = self.deserialize_data(raw_value, attr_desc['type'])
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1458, in deserialize_data
    return self._deserialize(obj_type, data)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1296, in _deserialize
    value = self.deserialize_data(raw_value, attr_desc['type'])
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1445, in deserialize_data
    return self.deserialize_type[iter_type](data, data_type[1:-1])
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1476, in deserialize_iter
    return [self.deserialize_data(a, iter_type) for a in attr]
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1476, in <listcomp>
    return [self.deserialize_data(a, iter_type) for a in attr]
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1458, in deserialize_data
    return self._deserialize(obj_type, data)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1303, in _deserialize
    return self._instantiate_model(response, d_attrs, additional_properties)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 1406, in _instantiate_model
    raise DeserializationError(msg + str(err))
msrest.exceptions.DeserializationError: Unable to deserialize {'blob_source': '{url}', 'file_path': '{fileName}', 'file_mode': None, 'source': None} into model <class 'azext.batch.models.extended_resource_file.ExtendedResourceFile'>. __init__() takes 1 positional argument but 4 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/interactive/azclishell/app.py", line 659, in cli_execute
    result = invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 369, in execute
    cmd.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/batch/_exception_handler.py", line 30, in batch_exception_handler
    reraise(*sys.exc_info())
  File "/opt/az/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 346, in execute
    result = cmd(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
    result = op(**command_args)
  File "/home/admindev/.azure/cliextensions/azure-batch-cli-extensions/azext_batch/custom.py", line 144, in create_job
    job = ExtendedJobOperations.jobparameter_from_json(json_obj)
  File "/home/admindev/.azure/cliextensions/azure-batch-cli-extensions/azext/batch/operations/job_operations.py", line 86, in jobparameter_from_json
    raise ValueError("Unable to deserialize to {}: {}".format(result, exp))
ValueError: Unable to deserialize to ExtendedJobParameter: Unable to deserialize {'blob_source': '{url}', 'file_path': '{fileName}', 'file_mode': None, 'source': None} into model <class 'azext.batch.models.extended_resource_file.ExtendedResourceFile'>. __init__() takes 1 positional argument but 4 were given

HOL - step-by-step

Exercise 4, Task 3, step 3 (line 1002)
Should those instructions be indented?

HOL - unguided

Please check lines 468 - 477 are those lines showing up correctly? They look different when looking from the formatting to the document.

HOL - unguided

Please check line 248 - not sure if this is supposed to look like this or maybe there's a hard return command where there should be **

sudo az batch account login -g mcw-lab-big-compute -n **batchAccountName\

Before the HOL

I renamed the document & updated to current template but I don't know how to format a table of contents yet. Please add that section to the doc.

Exercise 2: Task 2: Stage the sample videos to process

Exercise 2: Task 2: Step 1: The video link given here is broken.

Here is the video link and the error:
https://sample-videos.com/video123/mp4/720/big_buck_bunny_720p_30mb.mp4

Kindly check this.

Thank You,
Seerat

HOL - step by step

Please take a look at the formatting on lines 748 - 757 (Exercise 3, task 1, step 2). I'm not sure it's formatting the way you want it to when I look from the code to the document.

Same with step 3 - should those be indented??

Step 6 says: This means the VM nodes in the pool are ready for some work. Continue with the next task to supply some … ?

link broken

Hi,
link https://azure.github.io/BatchLabs/ is broken.
Points to a 404 not found page

HOL - unguided

Line 252 - please verify formatting is correct.

ReadMe page

Updated to most current template. Missing intro (right under title), target audience, updated abstract & Services & Products.

With updates, please update WDS & HOL docs too.

WDS trainer & student guides

Supplemental Materials section (line 369 in trainer guide & line 252 in student guide)
I think this is supposed to be a chart but it's not showing up correctly...

The following projects may be helpful to you after completing the workshop in understanding other ways in which Azure Batch can be applied, besides media and rendering. | | | |----------|:-------------:| | Description | Links | | CycleCloud Lab | https://github.com/azurebigcompute/BigComputeLabs/tree/master/CycleCloud/ | | Do Azure Parallel R Package | https://github.com/Azure/doAzureParallel/ | | Big Compute Batch VM | https://github.com/mkiernan/bigcomputebench/ |

Ready for review and QC

The sept-2018-test-fix branch is ready for review and QC.

June 2019 - Content update

@ZoinerTejada @timahenning
Hello,
This workshop is scheduled for a June 2019 update. Please review the workshop and provide your update suggestions to be sent for SME review.

Note: This update was previously scheduled, then placed on hold. Adding notes from previous issue and closing outdated issue to avoid confusion.
Thanks, Dawnmarie

From Kyle:
Here are my suggested updates for this MCW:
Move deployment of Jump Box into Before the HOL.
Add clarification around steps to create the batch job files (pool.json and job.json), to avoid potential for errors when copying and pasting file content.

From Joel:
I am good with those suggestions. There may be new changes to the portal UI, services, and features that will need to be updated as well since the last time this MCW has been touched.

May 2019 - Scheduled content update

@ZoinerTejada @joelhulen @kylebunting
Solliance team -
This workshop is scheduled for an update in March. Please update this issue with suggested content changes. Once done, we'll assign to our SME team for review and additional feedback.

WDS trainer presentation - PPT

I made some minor formatting/punctuation changes to the PPT and uploaded it. Please make sure to download what is on GitHub to address issues listed below.

Preferred solution section:

• Under Batch Computing, questions 3 & 4 & 6-10 answers do not have slides with answers. (comparing the trainer guide doc to the PPT)
• Operationalizing & Integrating answers both questions, but then pops to question #1 of Visualization & Remote workstations (Slide 22) - but there's no slide with that as a subtitle
• Slide 23 - answers question number 2 of Visualization & Remote workstations, but the slide title is still Operationalizing & Integrating
• Questions 3 - 6 of Visualization & Remote Workstations are not answered in the deck

WDS trainer & student guides

• Data Loading - student guide asks 2 questions, trainer guide, preferred solution only answers # 1.
• Batch Computing - student guide lists 6 questions, trainer guide, preferred solution answers 10 (and the questions are different).
• Visualization & Remote Workstations - Student guide has questions 1 & 2, but trainer guide, preferred solution answers 1 - 6.

*Please make sure whatever changes you make in guides match what's in the PPT as well.

To perform this lab we require azure cli older 2.0.68 version

In this lab we are installing latest version for Azure CLI, which is not allowing to perform lab. Show below warning while running any azure batch command :
Unable to load extension 'azure-cli-batch-extension'.

Same lab perform with older CLI 2.0.68 version and able to perform the lab.

December 2018 Big Compute test-fix

This package is scheduled for a December test-fix. Please review the contents of this repo and comment on this issue with suggested updates. All suggestions must be added to this issue by 12/21/2018.