azure/msrest-for-python

The runtime library "msrest" for AutoRest generated Python clients.

AutoRest: Python Client Runtime

Disclaimer

This package is deprecated and no longer receives updates

• The authentication part of this package has been moved to azure-identity
• The serialization part of this package is now vendored inside SDKs themselves
• The other parts of this library are covered by azure-core

As such, we will no longer accept PR and fix issues on this project, and this repo will be soon archived.

Installation

To install:

$ pip install msrest

Release History

2022-06-10 Version 0.7.1

Bugfixes

• Declare correctly msrest as Python 3.6 and more only for clarity #251

2022-06-07 Version 0.7.0

Features

• Add azure-core as installation requirement #247
• Replace SerializationError and DeserializationError in msrest.exceptions with those in azure.core #247

Bugfixes

• Typing annotation in LROPoller (thanks to akx) #242

Thanks to kianmeng for typo fixes in the documentation.

2021-01-26 Version 0.6.21

Bug Fixes

• Fixes failsafe_deserialize introduced in 0.6.20 #232

2021-01-25 Version 0.6.20

Features

• Add failsafe_deserialize method to the Deserializer object. #232
• Serialize datetime, date, time, timedelta and Decimal correctly when serializing object . #224

2020-09-08 Version 0.6.19

Bugfixes

• Fix serialization of random Model object #220
• Fix serialization of unicode string in Py2 and object mode #221

2020-07-27 Version 0.6.18

Features

• Add support for attributes/text in the same XML node #218

2020-06-25 Version 0.6.17

Bugfixes

• Fix XML and discriminator #214

2020-06-09 Version 0.6.16

Bugfixes

• Fix XML parsing with namespaces and attributes #209

Features

• Add py.typed for mypy support

2020-06-04 Version 0.6.15

Bugfixes

• Fix RFC regression introduced in 0.6.14 (RFC parse date are no longer pickable) #208
• Fix XML parsing with namespaces #206

Thanks to ivanst0 for the contribution

2020-05-18 Version 0.6.14

Bugfixes

• Fix "from_dict" in some complex flattening scenario #204
• Fix RFC date parsing if machine locale is not English #201

2020-04-07 Version 0.6.13

Bugfixes

• Fix deserializer and flattening if intermediate node is None #198
• Fix validation exception message for minimum/maximum checks #199

2020-04-06 Version 0.6.12

Features

• Add "time" serializer/deserializer #196

2020-01-30 Version 0.6.11

Features

• XML mode can now be enabled even if the given Model has no XML metadata #184
• Add Kerberos Authentication #186
• Improve error message if expected type is dictionary and something else is provided #188

Bugfixes

• Fix comma separated serialization of array in query #186
• Fix validation of basic types in some complex scenario #189

Thanks to catatonicprime for the contribution

2019-09-04 Version 0.6.10

Features

• XML mode now supports OpenAPI additional properties # 174

Bugfixes

• Accept "is_xml" kwargs to force XML serialization #178
• Disable XML deserialization if received element is not an ElementTree #178
• A "null" enum deserialize as None, and not "None" anymore #173
• Fix some UTF8 encoding issue in Python 2.7 and XML mode #172

2019-07-24 Version 0.6.9

Features

• Accept extensions of JSON mimetype as valid JSON #167

2019-06-24 Version 0.6.8

BugFixes

• Impossible to serialize XML if model contains UTF8 characters on Python 2.7 #165
• Impossible to deserialize a HTTP response as XML if body contains UTF8 characters on Python 2.7 #165
• Loading a serialized configuration fails with NameError on NoOptionError #162

Thanks to cclauss for the contribution

2019-06-12 Version 0.6.7

Features

• Add DomainCredentials credentials for EventGrid

Thanks to kalyanaj for the contribution

2019-03-21 Version 0.6.6

Bugfixes

• Make 0.6.x series compatible with pyinstaller again
• sdist now includes tests

Thanks to dotlambda for the contribution

2019-03-11 Version 0.6.5

Bugfixes

• Fix list of integers serialization if div is provided #151
• Fix parsing of UTF8 with BOM #145

Thanks to eduardomourar for the contribution

2019-01-09 Version 0.6.4

Bugfixes

• Fix regression on credentials configuration if used outside of Autorest scope #135

2019-01-08 Version 0.6.3

Features

• Updated experimental async support. Requires Autorest.Python 4.0.64.

2018-11-19 Version 0.6.2

Bugfixes

• Fix circular dependency in TYPE_CHECKING mode #128

2018-10-15 Version 0.6.1

Bugfixes

• Remove unnecessary verbose "warnings" log #126

2018-10-02 Version 0.6.0

Features

• The environment variable AZURE_HTTP_USER_AGENT, if present, is now injected part of the UserAgent
• New preview msrest.universal_http module. Provide tools to generic HTTP management (sync/async, requests/aiohttp, etc.)
• New preview msrest.pipeline implementation:
  • A Pipeline is an ordered list of Policies than can process an HTTP request and response in a generic way.
  • More details in the wiki page about Pipeline: https://github.com/Azure/msrest-for-python/wiki/msrest-0.6.0---Pipeline
• Adding new attributes to Configuration instance:
  • http_logger_policy - Policy to handle HTTP logging
  • user_agent_policy - Policy to handle UserAgent
  • pipeline - The current pipeline used by the SDK client
  • async_pipeline - The current async pipeline used by the async SDK client
• Installing "msrest[async]" now installs the experimental async support. Works ONLY for Autorest.Python 4.0.63.

Breaking changes

• The HTTPDriver API introduced in 0.5.0 has been replaced by the Pipeline implementation.
• The following classes have been moved from "msrest.pipeline" to "msrest.universal_http":
  • ClientRedirectPolicy
  • ClientProxies
  • ClientConnection
• The following classes have been moved from "msrest.pipeline" to "msrest.universal_http.requests":
  • ClientRetryPolicy

Bugfixes

• Fix "long" on Python 2 if used with the "object" type #121

Thanks to robgolding for the contribution

2018-09-04 Version 0.5.5

Bugfixes

• Fix a serialization issue if additional_properties is declared, and "automatic model" syntax is used ("automatic model" being the ability to pass a dict to command and have the model auto-created) # 120

2018-07-12 Version 0.5.4

Features

• Support additionalProperties and XML

BugFixes

• Better parse empty node and not string types
• Improve "object" XML parsing

2018-07-10 Version 0.5.3

BugFixes

• Fix some XML serialization subtle scenarios

2018-07-09 Version 0.5.2

Features

• deserialize/from_dict now accepts a content-type parameter to parse XML strings

Bugfixes

• Fix some complex XML Swagger definitions.

This release likely breaks already generated XML SDKs, that needs to be regenerated with autorest.python 3.0.58

2018-06-21 Version 0.5.1

Bugfixes

• Lower Accept header overwrite logging message #110
• Fix 'object' type and XML format

Thanks to dharmab for the contribution

2018-06-12 Version 0.5.0

Disclaimer

This released is designed to be backward compatible with 0.4.x, but there is too many internal refactoring and new features to continue with 0.4.x versioning

Features

• Add XML support
• Add many type hints, and MyPY testing on CI.
• HTTP calls are made through a HTTPDriver API. Only implementation is requests for now. This driver API is not considered stable and you should pin your msrest version if you want to provide a personal implementation.

Bugfixes

• Incorrect milliseconds serialization for some datetime object #94

Deprecation

That will trigger a DeprecationWarning if an old Autorest generated code is used.

• _client.add_header is deprecated, and config.headers should be used instead
• _client.send_formdata is deprecated, and _client.put/get/delete/post + _client.send should be used instead

2018-04-30 Version 0.4.29

Bugfixes

• Improve SDKClient.__exit__ to take exc_details as optional parameters and not required #93
• refresh_session should also use the permanent HTTP session if available #91

2018-04-18 Version 0.4.28

Features

• msrest is now able to keep the "requests.Session" alive for performance. To activate this behavior:

  • Use the final Client as a context manager (requires generation with Autorest.Python 3.0.50 at least)
  • Use client.config.keep_alive = True and client.close() (requires generation with Autorest.Python 3.0.50 at least)
  • Use client.config.keep_alive = True and client._client.close() (not recommended, but available in old releases of SDK)

• All Authentication classes now define signed_session and refresh_session with an optional session parameter. To take benefits of the session improvement, a subclass of Authentication MUST add this optional parameter and use it if it's not None:

  def signed_session(self, session=None):

  session = session or requests.Session()

  # As usual from here.

2018-03-07 Version 0.4.27

Features

• Disable HTTP log by default (security), add enable_http_log to restore it #86

BugFixes

• Fix incorrect date parsing if ms precision is over 6 digits #82

2018-01-30 Version 0.4.26

Features

• Add TopicCredentials for EventGrid client

Bugfixes

• Fix minimal dependency of isodate
• Fix serialisation from dict if datetime provided

2018-01-08 Version 0.4.25

Features

• Add LROPoller class. This is a customizable LRO engine. This is the poller engine of Autorest.Python 3.0, and is not used by code generated by previous Autorest version.

2018-01-03 Version 0.4.24

Bugfixes

• Date parsing is now compliant with Autorest / Swagger 2.0 specification (less lenient)

Internal optimisation

• Call that does not return a streamable object are now executed in requests stream mode False (was True whatever the type of the call). This should reduce the number of leaked opened session and allow urllib3 to manage connection pooling more efficiently. Only clients generated with Autorest.Python >= 2.1.31 (not impacted otherwise, fully backward compatible)

2017-12-21 Version 0.4.23

Bugfixes

• Accept to deserialize enum of different type if content string match #75
• Stop failing on deserialization if enum string is unkwon. Return the string instead.

Features

• Model now accept kwargs in constructor for future kwargs models

2017-12-15 Version 0.4.22

Bugfixes

• Do not validate additional_properties #73
• Improve validation error if expected type is dict, but actual type is not #73

2017-12-14 Version 0.4.21

Bugfixes

• Fix additional_properties if Swagger was flatten #72

2017-12-13 Version 0.4.20

Features

• Add support for additional_properties
  • By default, all additional_properties are kept.
  • Additional properties are sent to the server only if it was specified in the Swagger, or if "enable_additional_properties_sending" is called on the model we want it. This is a class method that enables it for all instance of this model.

2017-11-20 Version 0.4.19

Features

• The interpretation of Swagger 2.0 "discriminator" is now lenient. This means for these two scenarios:

  • Discriminator value is missing from the received payload
  • Discriminator value is not defined in the Swagger

  Instead of failing with an exception, this now returns the base type for this "discriminator".

  Note that this is not a contradiction of the Swagger 2.0 spec, that specifies "validation SHOULD fail [...] there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."

  This cannot be configured for now and is the new default behvaior, but can be in the future if needed.

Bugfixes

• Optional formdata parameters were raising an exception (#65)
• "application/x-www-form-urlencoded" form was sent using "multipart/form-data". This causes problems if the server does not support "multipart/form-data" (#66)

2017-10-26 Version 0.4.18

Features

• Add ApiKeyCredentials class. This can be used to support OpenAPI ApiKey feature.
• Add CognitiveServicesAuthentication class. Pre-declared ApiKeyCredentials class for Cognitive Services.

2017-10-12 Version 0.4.17

Features

This make Authentication classes more consistent:

• OAuthTokenAuthentication is now a subclass of BasicTokenAuthentication (was Authentication)
• BasicTokenAuthentication has now a "set_token" methods that does nothing.

This allows test like "isintance(o, BasicTokenAuthentication)" to be guaranteed that the following attributes exists:

• token
• set_token()
• signed_session()

This means for users of "msrestazure", that they are guaranteed that all AD classes somehow inherits from "BasicTokenAuthentication"

2017-10-05 Version 0.4.16

Bugfixes

• Fix regression: accept "set<str>" as a valid "[str]" (#60)

2017-09-28 Version 0.4.15

Bugfixes

• Always log response body (#16)
• Improved exception message if error JSON is Odata v4 (#55)
• Refuse "str" as a valid "[str]" type (#41)
• Better exception handling if input from server is not JSON valid

Features

• Add Configuration.session_configuration_callback to customize the requests.Session if necessary (#52)
• Add a flag to Serializer to disable client-side-validation (#51)
• Remove "import requests" from "exceptions.py" for apps that require fast loading time (#23)

Thank you to jayden-at-arista for the contribution

2017-08-23 Version 0.4.14

Bugfixes

• Fix regression introduced in msrest 0.4.12 - dict syntax with enum modeled as string and enum used

2017-08-22 Version 0.4.13

Bugfixes

• Fix regression introduced in msrest 0.4.12 - dict syntax using isodate.Duration (#42)

2017-08-21 Version 0.4.12

Features

• Input is now more lenient
• Model have a "validate" method to check content constraints
• Model have now 4 new methods:
  • "serialize" that gives the RestAPI that will be sent
  • "as_dict" that returns a dict version of the Model. Callbacks are available.
  • "deserialize" the parses the RestAPI JSON into a Model
  • "from_dict" that parses several dict syntax into a Model. Callbacks are available.

More details and examples in the Wiki article on Github: https://github.com/Azure/msrest-for-python/wiki/msrest-0.4.12---Serialization-change

Bugfixes

• Better Enum checking (#38)

2017-06-21 Version 0.4.11

Bugfixes

• Fix incorrect dependency to "requests" 2.14.x, instead of 2.x meant in 0.4.8

2017-06-15 Version 0.4.10

Features

• Add requests hooks to configuration

2017-06-08 Version 0.4.9

Bugfixes

• Accept "null" value for paging array as an empty list and do not raise (#30)

2017-05-22 Version 0.4.8

Bugfixes

• Fix random "pool is closed" error (#29)
• Fix requests dependency to version 2.x, since version 3.x is annunced to be breaking.

2017-04-04 Version 0.4.7

BugFixes

• Refactor paging #22:

  • "next" is renamed "advance_page" and "next" returns only 1 element (Python 2 expected behavior)
  • paging objects are now real generator and support the "next()" built-in function without need for "iter()"

• Raise accurate DeserialisationError on incorrect RestAPI discriminator usage #27

• Fix discriminator usage of the base class name #27

• Remove default mutable arguments in Clients #20

• Fix object comparison in some scenarios #24

2017-03-06 Version 0.4.6

Bugfixes

• Allow Model sub-classes to be serialized if type is "object"

2017-02-13 Version 0.4.5

Bugfixes

• Fix polymorphic deserialization #11
• Fix regexp validation if '\w' is used in Python 2.7 #13
• Fix dict deserialization if keys are unicode in Python 2.7

Improvements

• Add polymorphic serialisation from dict objects
• Remove chardet and use HTTP charset declaration (fallback to utf8)

2016-09-14 Version 0.4.4

Bugfixes

• Remove paging URL validation, part of fix Azure/autorest#1420

Disclaimer

In order to get paging fixes for impacted clients, you need this package and Autorest > 0.17.0 Nightly 20160913

2016-09-01 Version 0.4.3

Bugfixes

• Better exception message (Azure/autorest#1300)

2016-08-15 Version 0.4.2

Bugfixes

• Fix serialization if "object" type contains None (Azure/autorest#1353)

2016-08-08 Version 0.4.1

Bugfixes

• Fix compatibility issues with requests 2.11.0 (Azure/autorest#1337)
• Allow url of ClientRequest to have parameters (Azure/autorest#1217)

2016-05-25 Version 0.4.0

This version has no bug fixes, but implements new features of Autorest: - Base64 url type - unixtime type - x-ms-enum modelAsString flag

Behaviour changes

• Add Platform information in UserAgent
• Needs Autorest > 0.17.0 Nightly 20160525

2016-04-26 Version 0.3.0

Bugfixes

• Read only values are no longer in __init__ or sent to the server (Azure/autorest#959)
• Useless kwarg removed

Behaviour changes

• Needs Autorest > 0.16.0 Nightly 20160426

2016-03-25 Version 0.2.0

Bugfixes

• Manage integer enum values (Azure/autorest#879)
• Add missing application/json Accept HTTP header (Azure/azure-sdk-for-python#553)

Behaviour changes

• Needs Autorest > 0.16.0 Nightly 20160324

2016-03-21 Version 0.1.3

Bugfixes

• Deserialisation of generic resource if null in JSON (Azure/azure-sdk-for-python#544)

2016-03-14 Version 0.1.2

Bugfixes

• urllib3 side effect (Azure/autorest#824)

2016-03-04 Version 0.1.1

Bugfixes

• Source package corrupted in Pypi (Azure/autorest#799)

2016-03-04 Version 0.1.0

Behavioural Changes

• Removed custom logging set up and configuration. All loggers are now children of the root logger 'msrest' with no pre-defined configurations.
• Replaced _required attribute in Model class with more extensive _validation dict.

Improvement

• Removed hierarchy scanning for attribute maps from base Model class - relies on generator to populate attribute maps according to hierarchy.
• Base class Paged now inherits from collections.Iterable.
• Data validation during serialization using custom parameters (e.g. max, min etc).
• Added ValidationError to be raised if invalid data encountered during serialization.

2016-02-29 Version 0.0.3

Bugfixes

• Source package corrupted in Pypi (Azure/autorest#718)

2016-02-19 Version 0.0.2

Bugfixes

• Fixed bug in exception logging before logger configured.

2016-02-19 Version 0.0.1

• Initial release.

msrest-for-python's Issues

Optional formData parameters crash msrest

If a parameter that is supposed to be formData is optional, we give None to requests:

files = [('Text', (None, 'cognituve services')), ('Mode', (None, None)), ('PreContextText', (None, None)), ('PostContextText', (None, None))]
data = {}

    @staticmethod
    def _encode_files(files, data):
        """Build the body for a multipart/form-data request.

            Will successfully encode files when passed as a dict or a list of
            tuples. Order is retained if data is a list of tuples but arbitrary
            if parameters are supplied as a dict.
            The tuples may be 2-tuples (filename, fileobj), 3-tuples (filename, fileobj, contentype)
            or 4-tuples (filename, fileobj, contentype, custom_headers).
            """
        if (not files):
            raise ValueError("Files must be provided.")
        elif isinstance(data, basestring):
            raise ValueError("Data must not be a string.")

        new_fields = []
        fields = to_key_val_list(data or {})
        files = to_key_val_list(files or {})

        for field, val in fields:
            if isinstance(val, basestring) or not hasattr(val, '__iter__'):
                val = [val]
            for v in val:
                if v is not None:
                    # Don't call str() on bytestrings: in Py3 it all goes wrong.
                    if not isinstance(v, bytes):
                        v = str(v)

                    new_fields.append(
                        (field.decode('utf-8') if isinstance(field, bytes) else field,
                         v.encode('utf-8') if isinstance(v, str) else v))

        for (k, v) in files:
            # support for explicit filename
            ft = None
            fh = None
            if isinstance(v, (tuple, list)):
                if len(v) == 2:
                    fn, fp = v
                elif len(v) == 3:
                    fn, fp, ft = v
                else:
                    fn, fp, ft, fh = v
            else:
                fn = guess_filename(v) or k
                fp = v

            if isinstance(fp, (str, bytes, bytearray)):
                fdata = fp
            else:
>               fdata = fp.read()
E               AttributeError: 'NoneType' object has no attribute 'read'

Support building instance from dict with polymorphic object

Currently the SDKs can accept a dict instead of a model and transform it to the right model automatically. This is not available if there is at any level a polymorphic object.

This should be possible looking at _subtype_map to identify the right instance type

FYI @annatisch @vishrutshah

Replace isodate

Following discussion with @zooba about isodate:

• Isodate is not maintained anymore for 2 years. Direct Gmail email to the author get no answer as well.
• Isodate still uses 2to3
• I got no issue adapting it without 2to3, but since the owner abandoned the project, it's useless.

The features we use of isodate are:

• Parsing / serializing of ISO8601 datetime
• Parsing / serializing of ISO8601 duration

Datetime is simple, and move from isodate is straightforward.

Duration is way more complicated, since this is relative duration (one month can't be converted to timedelta). Isodate algorithm returns timedelta instances for duration defined in days/hours/..., and a specific isodate.Duration class for ambiguous relative duration. It's dynamic based on what is parsed.

I found aniso8601 to parse duration. Aniso8601 is not using dynamic return type, there is a relative flag to decide if you want timedelta (might be converted with 1 month == 30 days), or dateutil.relativedelta to keep the precision. Aniso8601 does not serialize, but it's not the most complicated part.

Moving this dependency will also imply that users will get different object from a SDK call, moving from the couple timedelta/isodate.Duration to dateutil.relativedelta.

Considering that the issue itself (2to3 on Py3 install) is not that critical for now, and that there is no direct obvious solution, I plan to do nothing for now, but I keep this issue open for the record of what I found. Eventually we'll have to work on this.

FYI @johanste

application/x-www-form-urlencoded should not be sent as multipart/form-data

The server might not accept both (we see that in SpellCheck Cognitive Services)

Serialization error on DiskCreateOptionTypes

Hi,

Ever since commit: c0e61ac

trying to attach a managed disk using:

for idx, volume in enumerate(volumes_info):
    self._worker_obj.storage_profile.data_disks.append({
        'lun': idx,
        'name': volume["volume_name"],
        'create_option': DiskCreateOption.attach,
        'managed_disk': {
            'id': volume["volume_id"]
        }
    })
 compute_cli.virtual_machines.create_or_update(
    self._conf["resource_group"],
    self._worker_obj.name,
    self._worker_obj
).result()

fails with:

  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/azure/mgmt/compute/compute/v2016_04_30_preview/operations/virtual_machines_operations.py", line 180, in create_or_update
    body_content = self._serialize.body(parameters, 'VirtualMachine')
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 441, in body
    SerializationError, "Unable to build a model: "+str(err), err)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/exceptions.py", line 47, in raise_with_traceback
    raise error.with_traceback(exc_traceback)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 438, in body
    data = deserializer(data_type, data)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 958, in __call__
    self(local_type, value)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 958, in __call__
    self(local_type, value)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 968, in __call__
    return self.deserialize_data(data, response)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1096, in deserialize_data
    return self.deserialize_type[iter_type](data, data_type[1:-1])
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1118, in deserialize_iter
    return [self.deserialize_data(a, iter_type) for a in attr]
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1118, in <listcomp>
    return [self.deserialize_data(a, iter_type) for a in attr]
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1107, in deserialize_data
    return self(obj_type, data)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 987, in __call__
    value = self.deserialize_data(raw_value, attr_desc['type'])
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1100, in deserialize_data
    return self.deserialize_enum(data, obj_type)
  File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/msrest/serialization.py", line 1245, in deserialize_enum
    raise DeserializationError(error.format(data, enum_obj))
msrest.exceptions.SerializationError: Unable to build a model: <DiskCreateOption.attach: 'Attach'> is not valid value for enum <enum 'DiskCreateOptionTypes'>, DeserializationError: <DiskCreateOption.attach: 'Attach'> is not valid value for enum <enum 'DiskCreateOptionType
s'>

Or am I not using this correctly? The guide here: https://github.com/Azure/azure-sdk-for-python/wiki/Managed-Disk

suggests that this should work :)

Serialization issue if dict syntax and Duration used

msrest.exceptions.SerializationError: Unable to build a model: Unable to deserialize response data. Data: 3 years, 6 months, 4 days, 12:30:05, duration, TypeError: Expecting a string isodate.duration.Duration(4, 45005, 0, years=3, months=6), DeserializationError: Unable to deserialize response data. Data: 3 years, 6 months, 4 days, 12:30:05, duration, TypeError: Expecting a string isodate.duration.Duration(4, 45005, 0, years=3, months=6)

Regression introduced in 0.4.12

Deserializer will raise requests.exceptions.ConnectionError

I am currently using msrest 0.4.11 and found the Deserializer will raise this exception: requests.exceptions.ConnectionError

requests library read the raw socket to get the response body when calling response.text here.

if connection/server is not stable, it will raise ConnectionError exception from requests library

is it possible to catch this exception in Deserializer?
Because user of Azure SDK will expect the exception is always CloudError.

here is the callstack of msrest 0.4.11 version:

  File "D:\home\Python35\lib\site-packages\msrest\paging.py", line 109, in __next__
    self.advance_page()
  File "D:\home\Python35\lib\site-packages\msrest\paging.py", line 96, in advance_page
    self._derserializer(self, self._response)
  File "D:\home\Python35\lib\site-packages\msrest\serialization.py", line 733, in __call__
    data = self._unpack_content(response_data)
  File "D:\home\Python35\lib\site-packages\msrest\serialization.py", line 806, in _unpack_content
    if not raw_data.text:
  File "D:\home\Python35\lib\site-packages\requests\models.py", line 847, in text
    if not self.content:
  File "D:\home\Python35\lib\site-packages\requests\models.py", line 823, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "D:\home\Python35\lib\site-packages\requests\models.py", line 752, in generate
    raise ConnectionError(e)

thanks!

Object DeserializationError / unicode

I have modified the example image_analysis_in_stream at https://github.com/Azure-Samples/cognitive-services-python-sdk-samples/blob/master/samples/vision/computer_vision_samples.py to rather carry out OCR, i.e.

with open(os.path.join(IMAGES_FOLDER, "image.jpg"), "rb") as image_stream:
        image_response = client.recognize_printed_text_in_stream(image_stream,language='en')

This gives the following Traceback:

  File "/anaconda2/lib/python2.7/site-packages/azure/cognitiveservices/vision/computervision/computer_vision_api.py", line 962, in recognize_printed_text_in_stream
    deserialized = self._deserialize('OcrResult', response)
  File "/anaconda2/lib/python2.7/site-packages/msrest/serialization.py", line 1179, in __call__
    return self._deserialize(target_obj, data)
  File "/anaconda2/lib/python2.7/site-packages/msrest/serialization.py", line 1245, in _deserialize
    value = self.deserialize_data(raw_value, attr_desc['type'])
  File "/anaconda2/lib/python2.7/site-packages/msrest/serialization.py", line 1433, in deserialize_data
    return self._deserialize(obj_type, data)
  File "/anaconda2/lib/python2.7/site-packages/msrest/serialization.py", line 1249, in _deserialize
    raise_with_traceback(DeserializationError, msg, err)
  File "/anaconda2/lib/python2.7/site-packages/msrest/exceptions.py", line 57, in raise_with_traceback
    raise error
msrest.exceptions.DeserializationError: Unable to deserialize to object: type, AttributeError: 'unicode' object has no attribute 'get

Line 962 requires a response code of 200, so can I assume image_stream is valid?

How do I then deserialize OcrResult to JSON?

Is this an API version mismatch..?

Thanks for any speedy help.

(This is a cross-posting of Azure/azure-sdk-for-python#2769)

Importing msrest.exceptions is expensive

Importing msrest pulls in quite a few modules (indirectly), including, but not limited to, requests, in its init.py. The azure-cli is working on startup performance, and are looking at delay loading modules as much as we can.

Being able to at least use the exception classes from msrest without taking the module load hit would be a very welcome improvement.

Deprecate add_header of client and add headers in config

Solve that:
https://github.com/Azure/azure-cli/blob/7402fb2c20be2cdbcaa7bdb2eeb72b7461fbcc30/src/azure-cli-core/azure/cli/core/commands/client_factory.py#L88-L95

"aiter" warning is too verbose

I am using the Batch API, and whenever I query running tasks I see this message printed to the output:

Paging async iterator protocol is not available for CloudTaskPaged

Since I am querying multiple times in a loop, I see the message multiple times. But since I am the user of the library and not the developer, I can't do anything about it.

It would be better if the message only appeared when I tried to use the async iterator protocol (and maybe you can automatically run the synchronous iterator in that case so it still works?). Ideally, the team responsible for implementing it would be the ones to get the message and they would just fix it :)

Support env variable for UserAgent

See CLI:
https://github.com/Azure/azure-cli/blob/439e0e09702ef10f87d86b093b788c24e92169c2/src/azure-cli-core/azure/cli/core/commands/client_factory.py#L86

Should probably be done in the pipeline PR directly #106

Add __hash__ in Model

Currently defined in Ansible:

# Quick 'n dirty hash impl suitable for monkeypatching onto SDK model objects (so we can use set comparisons)
def gethash(self):
    if not getattr(self, '_cachedhash', None):
        spec = inspect.getargspec(self.__init__)
        valuetuple = tuple([getattr(self, x, None) for x in spec.args if x != 'self'])
        self._cachedhash = hash(valuetuple)
    return self._cachedhash

This use the __init__ signature, but we can use attribute map instead and don't use inspect. Or even use as_dict:

def __hash__(self):
    if not getattr(self, '_cachedhash', None):
        self._cachedhash = hash(self.as_dict())
    return self._cachedhash

Need to check that all possible internal types are ok (duration, datetime, etc.) but should be fine

FYI @nitzmahone @johanste

Related to ansible/ansible#33624 (comment)

Add _unmatched_attributes dict

Any JSON value that are not matched as an attribute should be available in a hidden attribute, at least for debug purpose, at best for roundtrip scenario.

Validation failed if string is valid integer, but minimum/maximum is used.

Example from @derekbekoe

>>> ExpressRouteCircuitPeering(peer_asn='10002').validate()
[ValidationError("Parameter 'ExpressRouteCircuitPeering.peer_asn' failed to meet validation requirement.",)]

I see no reason to refuse that, and should take a look

Add support to disable validation on the runtime side

Should be a flag in configuration to disable runtime validation.

This flag should be default to client-side-validation value from Autorest:
Azure/autorest#1583 (comment)

FYI @matthchr

msrest debug logging includes request/response body in clear text

This causes problems when the request and/or response container sensitive data that is then persisted to log files.

`requests` version tied to `2.16`

msrest is currently using requests~=2.16. This makes it hard to include the msrest derived libraries (like the azure ones) into other python packages that also use requests with other versions.

Would you consider changing it so we have requests>=2.12 in here: https://github.com/Azure/msrest-for-python/blob/master/setup.py#L50?

I can send the PR but wanted to ask about this first. In particular the package that I am looking at is https://github.com/certbot/certbot.

Make BasicTokenAuthentication the new base class of OAuthTokenAuthentication

To enable some KV scenarios releated to MSI

Improve exception string if custom exception

Example, with this:
https://github.com/Azure/azure-sdk-for-python/blob/master/azure-mgmt-billing/azure/mgmt/billing/models/error_response.py

And this answer:

{
	"error": {
		"code": "NotOptedIn",
		"message": "You are not allowed to download invoices. Please contact your account administrator ([email protected]) to turn on access in the management portal for allowing to download invoices through the API."
	}
}

We should have something better than:

D:\VEnvs\AzureCli\Lib\site-packages\azure\mgmt\billing\operations\invoices_operations.py in internal_paging(next_link, raw)
    110
    111             if response.status_code not in [200]:
--> 112                 raise models.ErrorResponseException(self._deserialize, response)
    113
    114             return response

ErrorResponseException: Operation returned an invalid status code 'Unauthorized'

We loose the message

Support for ISO8601 intervals

https://en.wikipedia.org/wiki/ISO_8601#Time_intervals

See also Azure/azure-sdk-for-python#2557 (comment)

'x-ms-failure-cause': 'gateway' in MCD 'x-ms-routing-request-id': 'WESTEUROPE' cause Response status: 404

Hi,

I would like to manage the Azure subscription related metrics and alerts from Python. I have started the discovering of capabilities of this area and I am facing a serious issue.

I work in MCD (Microsoft Cloud Deutschland).

When I create Monitor client client = MonitorManagementClient(credentials,subscription_id) it seems everything is OK. Then I want to make a request something like this:
metrics_data = client.metrics.list(resource_id, timespan="{}/{}".format(yesterday, today), interval='PT1H', metric='Percentage CPU', aggregation='Total') I get an error message:
"Operation returned an invalid status code 'Not Found'
raise models.ErrorResponseException(self._deserialize, response)"

I tried to change the authentication to Adal but the result was same. Then I turned on the debug logging to msrest (logging.getLogger('msrest').setLevel(logging.DEBUG);) and i checked the result and I saw this line:
DEBUG:msrest.http_logger: 'x-ms-routing-request-id': 'WESTEUROPE:20180111T155250Z:...'
This is strange for me in MCD. This line should be GERMANYCENTRAL or GERMANYNORTHEAST.

Then I tried my code in MCI and it worked. I turned on the debug logging to msrest there and I saw the line is totally same:
DEBUG:msrest.http_logger: 'x-ms-routing-request-id': 'WESTEUROPE:20180111T165801Z:...'

Accordiongly I guess something is wrong here. I would like to ask sour support in this situation.

Here is the code:
from azure.common.credentials import UserPassCredentials;
from azure.mgmt.monitor import MonitorManagementClient;
from azure.mgmt.monitor.models import RuleMetricDataSource;
from azure.mgmt.monitor.models import ThresholdRuleCondition;
from azure.mgmt.monitor.models import RuleEmailAction;
from msrestazure.azure_cloud import AZURE_GERMAN_CLOUD as cloud_environment;

credentials = UserPassCredentials(username=os.environ.get('AZURE_USER', input('Username: ')),password=os.environ.get('AZURE_PASSWORD', getpass.getpass('Password: ')), tenant=os.environ.get('AZURE_TENANT_ID', input('Tenant (AD Identifier): ')), cloud_environment=cloud_environment );

client = MonitorManagementClient(credentials,subscription_id);

metrics_data = client.metrics.list(resource_id, timespan="{}/{}".format(yesterday, today), interval='PT1H', metric='Percentage CPU', aggregation='Total');

Versions:
msrest==0.4.25
msrestazure==0.4.20

Thank you for your help in this matter.

Feel free to get in touch with any further questions.

Best Regards,
Tibor

Response body for operations are never logged in msrest.http_logger

Long running operations always send 'Transfer-Encoding: chunked' header and so the response bodies never get logged because of line 84 in http_logger.py.

        elif result.headers.get("transfer-encoding") == 'chunked':
            _LOGGER.debug("Body contains chunked data.")

v0.4.12 breaks mixed dict with enum if model-as-string=true

This breaks:

        async_security_rule = self.network_client.security_rules.create_or_update(
            self.group_name,
            security_group_name,
            new_security_rule_name,
            {
                    'access':azure.mgmt.network.models.SecurityRuleAccess.allow,
                    'description':'New Test security rule',
                    'destination_address_prefix':'*',
                    'destination_port_range':'123-3500',
                    'direction':azure.mgmt.network.models.SecurityRuleDirection.outbound,
                    'priority':400,
                    'protocol':azure.mgmt.network.models.SecurityRuleProtocol.tcp,
                    'source_address_prefix':'*',
                    'source_port_range':'655',
            }
        )

Should refuse a string as a valid list of string

Ends up in the portal as ['a','b','c'] if we use Model('abc') instead of Model(['abc']). Should fail, accepting a string for a list of string is likely an error and not a feature.

See Azure/azure-sdk-for-python#1376 (comment)

Serialize VirtualMachine object exception

Hi,

when serializing a virtual machine object I get an exception, in particular when it tries to serialize the Enum objects such as CachingTypes for the attribute storage_profile of the class azure.mgmt.compute.models.virtual_machine.VirtualMachine.

It tries to serialize using the attribute _attribute_map, not present in the enumeration objects.

def list_all(self):
         try:
             response = self.ComputeManagementClient(<credentials>, <subscription_id>).virtual_machines.list_all()
         except Exception as exc:
             raise Exception(str(exc))
         l = [item for item in response]
         dependencies = dict()
         ...
         serializer = Serializer(classes=dependencies)
         print(serializer.body(l[0], 'VirtualMachine'))

Here the stacktrace i get

Traceback (most recent call last):
...
File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 298, in body
    return self._serialize(data, data_type, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 241, in _serialize
    target_obj, data_type, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 427, in serialize_data
    return self._serialize(data, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 263, in _serialize
    orig_attr, attr_type, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 427, in serialize_data
    return self._serialize(data, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 263, in _serialize
    orig_attr, attr_type, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 427, in serialize_data
    return self._serialize(data, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 263, in _serialize
    orig_attr, attr_type, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 427, in serialize_data
    return self._serialize(data, **kwargs)
  File \"/usr/lib/python2.7/site-packages/msrest/serialization.py\", line 282, in _serialize
    raise_with_traceback(SerializationError, msg, err)
  File \"/usr/lib/python2.7/site-packages/msrest/exceptions.py\", line 50, in raise_with_traceback
    raise error
msrest.exceptions.SerializationError: Attribute None in object CachingTypes cannot be serialized.
CachingTypes.read_write, AttributeError: 'CachingTypes' object has no attribute '_attribute_map'

Am I doing something wrong?

Support additionalProperties in msrest

ImportError: cannot import name ClientRetryPolicy

I have an ansible based project which uses ansible Azure RM module. It used to work fine until I installed pythn, azure>=4.0 and ansible[azure] etc on a new dev Ubuntu machine. I see that there was a change where ClientRetryPolicy was moved to universal_http/requests . Can you please help track this issue.

ansible:~/../$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import msrest
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/msrest/__init__.py", line 27, in <module>
    from .configuration import Configuration
  File "/usr/local/lib/python2.7/dist-packages/msrest/configuration.py", line 39, in <module>
    from .pipeline import (
ImportError: cannot import name ClientRetryPolicy

PyPI releases show "Invalid Date"

See https://pypi.org/project/msrest/#history.

Refresh_token doesn't seem to be working

From Azure/azure-sdk-for-python#2467 created by @prburgu

It seems refresh_token is not working correct with latest azure sdk. If I pin, msrest and msrestazure to following version then it is working correctly.

msrestazure==0.4.26
msrest==0.4.27

Here is the error with latest azure sdk.

Traceback (most recent call last):
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/service_client.py", line 275, in send
    **kwargs)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/sessions.py", line 494, in request
    prep = self.prepare_request(req)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/sessions.py", line 437, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/models.py", line 309, in prepare
    self.prepare_auth(auth, url)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/models.py", line 540, in prepare_auth
    r = auth(self)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests_oauthlib/oauth2_auth.py", line 35, in __call__
    http_method=r.method, body=r.body, headers=r.headers)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 193, in add_token
    raise TokenExpiredError()
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/prburgu/infra/acmcloud/cli.py", line 93, in <module>
    result = func(*func_args)
  File "/Users/prburgu/infra/acmcloud/accloud/subcommands/resources.py", line 73, in setup_docdb
    return setup_docdb(self, service_name)
  File "/Users/prburgu/infra/acmcloud/accloud/subcommands/utils/docdb_utils.py", line 15, in setup_docdb
    client.azure.resource_groups.create_or_update(client.config.resource_group_name, client.config.location)
  File "/Users/prburgu/infra/acmcloud/accloud/acazure/acazure_client.py", line 75, in resource_groups
    return ResourceGroupOperations(self)
  File "/Users/prburgu/infra/acmcloud/accloud/acazure/acazure_base.py", line 82, in __init__
    self._cache_all_resource_info()
  File "/Users/prburgu/infra/acmcloud/accloud/acazure/acazure_base.py", line 456, in _cache_all_resource_info
    for resource in self.resource_client().resources.list():
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/paging.py", line 121, in __next__
    self.advance_page()
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/paging.py", line 107, in advance_page
    self._response = self._get_next(self.next_link)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/azure/mgmt/resource/resources/v2017_05_10/operations/resources_operations.py", line 361, in internal_paging
    request, header_parameters, **operation_config)
  File "/Users/prburgu/infra/acmcloud/accloud/acazure/acazure_clients.py", line 225, in send
    response = super(ACServiceClient, self).send(request, headers, content, **config)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/service_client.py", line 280, in send
    raise_with_traceback(TokenExpiredError, msg, err)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/exceptions.py", line 45, in raise_with_traceback
    raise error.with_traceback(exc_traceback)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/msrest/service_client.py", line 275, in send
    **kwargs)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/sessions.py", line 494, in request
    prep = self.prepare_request(req)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/sessions.py", line 437, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/models.py", line 309, in prepare
    self.prepare_auth(auth, url)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests/models.py", line 540, in prepare_auth
    r = auth(self)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/requests_oauthlib/oauth2_auth.py", line 35, in __call__
    http_method=r.method, body=r.body, headers=r.headers)
  File "/Users/prburgu/infra/acmcloud/miniconda_macos/lib/python3.5/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 193, in add_token
    raise TokenExpiredError()
msrest.exceptions.TokenExpiredError: Token expired or is invalid., TokenExpiredError: (token_expired)

Error when pushing app to Azure

I created a chatbot with Python SDK. When I want to push it to Azure, following this tutorial, the installation or requirements is failing:
Cloning https://github.com/Azure/msrest-for-python.git to d:\local\temp\pip-5vs3vmb1-build remote: Running setup.py (path:D:\local\Temp\pip-5vs3vmb1-build\setup.py) egg_info for package from git+https://github.com/Azure/msrest-for-python.git remote: error in msrest setup command: Invalid environment marker: python_version<'3.4' remote: Complete output from command python setup.py egg_info: remote: error in msrest setup command: Invalid environment marker: python_version<'3.4'

AADTokenCredentials doesn't properly refresh the token

It looks like the ServiceClient can't properly refresh the token if given AADTokenCredentials. It tries to do a refresh by passing in session to the refresh_session method and expects it to transform the existing session, but it looks like AADTokenCredentials will instead return a new session.

Caller:
https://github.com/Azure/msrest-for-python/blob/master/msrest/service_client.py#L351

Callee:
https://github.com/Azure/msrestazure-for-python/blob/master/msrestazure/azure_active_directory.py#L263

Since the method accepts a parameter it never raises a TypeError and the fallback method of getting a session returned will never happen unless the user tries the whole thing again.

I originally thought it was an Azure SDK issue Azure/azure-sdk-for-python#2779 but it looks like it belongs either in here or with https://github.com/Azure/msrestazure-for-python.

I think the issue might be with what is expected from the refresh_session, since the implementation in msrestazure could be ambiguous.

A problem with this lib is causing the `az aks list` command to fail

You can see more details in this slack thread:
https://kubernetes.slack.com/archives/C5HJXTT9Q/p1535546773000100

I think it’s related to api geo-location and trafficmanager failure. I guess the api server returns a paginated list with a http://acs-eastus.trafficmanager.net address as the next page base url. Even if your list is fitting on a single page, that url is tried.

3 strange things are going on here, 1. The base url is changed from https://management.azure.com to http://acs-eastus.trafficmanager.net
2. The protocol is now http
3. This should be the last page. So why a next_link?

"Accept header absent and forced to application/json" log spam

#102 introduced code which ensured the "Accept" header is always set to "application/json". However, it also introduced a WARNING level log that causes a lot of spam in applications using the Azure SDK.

This makes it really had to read application logs; here's an example of an Azure deployment in Jenkins:

Support returning a field as "raw"

Just a given field, to be returned as it was in the JSON with no transformation.

See Azure/azure-sdk-for-python#2557 (comment)

Minor typos in python source documentation

msrest-for-python/msrest/polling/poller.py

Line 192 in ddbac95

:param int timeout: Perion of time to wait for the long running

The line above says: :param int timeout: Perion of time to wait for the long running...

It should say: :param int timeout: Period, in seconds, of time to wait for the long running...

minor issue, but somewhat visible because it pops up in IDE when one is looking into how to use methods.

Regexp validate fails if meta-character like '\w' is used in Python 2.7

Example in Python 2.7:

>>> re.match('^[-\\w\\._\\(\\)]+$', u'éééé') # No match
>>> re.match('^[-\\w\\._\\(\\)]+$', u'éééé', re.U)
<_sre.SRE_Match object at 0x03935F70>

In Python 3.x both works

Need to add the re.UNICODE flag to all re calls in msrest

Requests do not share HTTP(S) connection pool

Hey there,

It looks like msrest (as used by the Azure SDK for Python) does not automatically share a requests context between requests.

I see traces of some code that would allow passing in a requests Session (e.g. here and maybe here too?), but I have no idea how to use those... I'm just creating a ComputeManagementClient :)

The reason why this came up is that I profiled a script that basically just does cmc.virtual_machines.list() about 7 times and it spends half of what it does (29.6 + 17.3 = 46.9%) just connecting to places it could have already have a connection to 😞

Function	Calls	Own Time	%
_ssl._SSLSocket.read	20	2156	39.3%
_ssl._SSLSocket.do_handshake	11	1626	29.6%
_socket.socket.connect	11	949	17.3%
_ssl._SSLContext.load_verify_locations	11	84	1.5%

So... how do I get my ComputeManagementClient to keep that requests session, or better yet, how do I get all of my Client objects to share one session (if that's safe)?

ansible azure storage DeserializationError

ansible version 2.4.2.0
pip freeze
adal==0.4.7
ansible==2.4.2.0
applicationinsights==0.11.1
argcomplete==1.9.3
asn1crypto==0.24.0
azure-cli-core==2.0.23
azure-cli-nspkg==3.0.1
azure-common==1.1.8
azure-mgmt-batch==4.1.0
azure-mgmt-compute==2.1.0
azure-mgmt-containerservice==1.0.0
azure-mgmt-dns==1.2.0
azure-mgmt-keyvault==0.40.0
azure-mgmt-network==1.7.1
azure-mgmt-nspkg==2.0.0
azure-mgmt-resource==1.2.2
azure-mgmt-sql==0.7.1
azure-mgmt-storage==1.5.0
azure-mgmt-web==0.32.0
azure-nspkg==2.0.0
azure-storage==0.35.1
bcrypt==3.1.4
certifi==2017.11.5
cffi==1.11.2
chardet==3.0.4
colorama==0.3.9
cryptography==2.1.4
enum34==1.1.6
humanfriendly==4.4.1
idna==2.6
ipaddress==1.0.19
isodate==0.6.0
Jinja2==2.10
jmespath==0.9.3
keyring==10.5.1
MarkupSafe==1.0
monotonic==1.4
msrest==0.4.22
msrestazure==0.4.19
oauthlib==2.0.6
packaging==16.8
paramiko==2.4.0
pyasn1==0.4.2
pycparser==2.18
pycrypto==2.6.1
Pygments==2.2.0
PyJWT==1.5.3
PyNaCl==1.2.1
pyOpenSSL==17.5.0
pyparsing==2.2.0
python-dateutil==2.6.1
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
SecretStorage==2.3.1
six==1.11.0
tabulate==0.7.7
urllib3==1.22

when I run the playbook:

• name: Create Azure VM
  hosts: localhost
  connection: local
  tasks:
  • name: Create storage account
    azure_rm_storageaccount:
    resource_group: awx
    name: testaccount2018
    type: Standard_LRS
  • name: Create VM with defaults
    azure_rm_virtualmachine:
    resource_group: awx
    name: testvm10
    admin_username: ubuntu
    admin_password: Password123456
    image:
    offer: UbuntuServer
    publisher: Canonical
    sku: '16.04-LTS'
    version: latest

I get following deserialization error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: msrest.exceptions.SerializationError: Unable to build a model: <SkuName.standard_lrs: 'Standard_LRS'> is not valid value for enum <enum 'SkuName'>, DeserializationError: <SkuName.standard_lrs: 'Standard_LRS'> is not valid value for enum <enum 'SkuName'>
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):
  File "/tmp/ansible_DUBaJU/ansible_module_azure_rm_storageaccount.py", line 462, in <module>
    main()
  File "/tmp/ansible_DUBaJU/ansible_module_azure_rm_storageaccount.py", line 459, in main
    AzureRMStorageAccount()
  File "/tmp/ansible_DUBaJU/ansible_module_azure_rm_storageaccount.py", line 204, in __init__
    supports_check_mode=True)
  File "/tmp/ansible_DUBaJU/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py", line 285, in __init__
  File "/tmp/ansible_DUBaJU/ansible_module_azure_rm_storageaccount.py", line 240, in exec_module
    self.results['state'] = self.create_account()
  File "/tmp/ansible_DUBaJU/ansible_module_azure_rm_storageaccount.py", line 416, in create_account
    poller = self.storage_client.storage_accounts.create(self.resource_group, self.name, parameters)
  File "/usr/local/lib/python2.7/dist-packages/azure/mgmt/storage/v2017_06_01/operations/storage_accounts_operations.py", line 159, in create
    body_content = self._serialize.body(parameters, 'StorageAccountCreateParameters')
  File "/usr/local/lib/python2.7/dist-packages/msrest/serialization.py", line 466, in body
    SerializationError, "Unable to build a model: "+str(err), err)
  File "/usr/local/lib/python2.7/dist-packages/msrest/exceptions.py", line 48, in raise_with_traceback
    raise error
msrest.exceptions.SerializationError: Unable to build a model: <SkuName.standard_lrs: 'Standard_LRS'> is not valid value for enum <enum 'SkuName'>, DeserializationError: <SkuName.standard_lrs: 'Standard_LRS'> is not valid value for enum <enum 'SkuName'>
", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}"

I know Standard_LRS is a valid storage type, just not sure why python code is complaing..

If a discriminator value is None or not present, msrest should instantiate the base class

Currently it does not, so when a "None" discriminator value comes, I get:

Unable to deserialize {'ssl_enforcement': None, 'storage_mb': 1024, 'administrator_login': 'cloudsa', 'create_mode': None, 'version': '5.6'} into model <class 'azure.mgmt.rdbms.mysql.models.server_properties.ServerProperties'>. __init__() got an unexpected keyword argument 'create_mode'

create_mode being the discriminator variable.

Blocking for RDBMS. Still be be confirmed by @amarzavery, but pretty sure it's the right behavior.

And if the discriminator value is wrong, raise a better message.

Verify LRO PATCH as per our discussion

Long running PATCH operation with initial status code as 200 and with azure-asyncoperation or location header in the response should be supported by the runtime.

[Debian/Ubuntu] ImportError: No module named packages.urllib3

If a user has the Debian version of requests, they may run into the following error.

  File "/home/ls/.local/lib/python2.7/site-packages/msrest/__init__.py", line 27, in <module>
    from .configuration import Configuration
  File "/home/ls/.local/lib/python2.7/site-packages/msrest/configuration.py", line 39, in <module>
    from .pipeline import (
  File "/home/ls/.local/lib/python2.7/site-packages/msrest/pipeline.py", line 36, in <module>
    from requests.packages.urllib3 import Retry
ImportError: No module named packages.urllib3

Observations:

• The version of requests available on Debian does not have the requests.packages sub-package.

• The file list for Debian python-requests does not include requests.packages.urllib3 - https://packages.debian.org/sid/all/python-requests/filelist

• In Debian, urllib3 is shipped separately as you can see in the dependencies through the link above.

• Here's the file list for urllib3 - https://packages.debian.org/sid/all/python-urllib3/filelist

Not 100% sure of the following but I suspect that this may be a cause.
Even if I do have the pip version of requests (which includes `requests.packages....), I may still run into this error if I also have the Debian package installed.
Based on how Python loads modules from PATH, the Debian package in /dist-packages might get loaded instead of the pip package in /site-packages.

$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print '\n'.join(sys.path)

/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/home/debekoe/.local/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages

Is the solution to try the import requests.packages.urllib3, if it fails, fall back to urllib3....?

Reference links:

• How to fix ImportError: No module named packages.urllib3?
• No module named for requests
• mk-fg/python-onedrive#13 (comment)

Default argument value is mutable

Hi,

Just want to confirm, I noticed there are some methods in ServiceClient contain a mutable default argument, for instance:
`def get(self, url=None, params={}):
"""Create a GET request object.

    :param str url: The request URL.
    :param dict params: Request URL parameters.
    """
    request = self._request(url, params)
    request.method = 'GET'
    return request`

And the default argument params is changed inself._request(url, params). Is that as design? I just think it's a little wired. Thanks!

msrest.exceptions.ValidationError.messages should be private?

See the doc here:
http://msrest.readthedocs.io/en/latest/msrest.html#msrest.exceptions.ValidationError

it looks bad, and is probably not intended for the user.

Send _unmatched_attributes

To enable roundtrip scenario, unmatched attribute can be resent (in a get/update scenario).

This should optional

serialize_enum should not accept the wrong type

Example:

        test_obj = type("TestEnumObj", (Model,), {"_attribute_map":None})
        test_obj._attribute_map = {
            "abc":{"key":"ABC", "type":"TestEnum"}
        }
        class TestEnum(Enum):
            val = "Value"

        class TestEnum2(Enum):
            val2 = "Value2"

        t = test_obj
        t.abc = TestEnum2.val2

        serializer = Serializer({
            'TestEnum': TestEnum,
            'TestEnum2': TestEnum2
        })

        serializer._serialize(t) # Should fail

Currently there is no type check, it just looks like an enum and we are happy. We should check if it's the correct enum, or least that the value is consistent.

Currently, the server will fail with probably a more complexe error message to handle that we can do

FYI @johanste

Unittests of paging

Currently no unit tests of msrest/paging in this repo (everything on Autorest side).

We should have unit tests here, because we always need full testing. Moreover, it took me a while to understand a bug in #5 because of that.

Autorest test mocks should not be in regular code here

Some code used to connect to the Autorest test server are inside the regular code here:
In the pipeline.py file for example, ClientHTTPConnectionPool or the _test_pipeline. This code should be on the Autorest repo, on the test files themself.

This solves part of the problem of #3

typo in doc comments for HttpOperationError

specificed -> specified

Import error when importing msrest with typing.TYPE_CHECKING = True

Install msrest:

(msrest) Adam@MacBook-Pro ~/Desktop> pip install --upgrade msrest
...
(msrest) Adam@MacBook-Pro ~/Desktop> pip freeze
certifi==2018.10.15
chardet==3.0.4
idna==2.7
isodate==0.6.0
msrest==0.6.1
oauthlib==2.1.0
requests==2.20.1
requests-oauthlib==1.0.0
six==1.11.0
urllib3==1.24.1
(msrest) Adam@MacBook-Pro ~/Desktop> python --version
Python 3.7.0

Attempt to import msrest with typing.TYPE_CHECKING = True:

(msrest) Adam@MacBook-Pro ~/Desktop> python -c 'import typing; typing.TYPE_CHECKING = True; import msrest'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/__init__.py", line 28, in <module>
    from .configuration import Configuration
  File "/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/configuration.py", line 37, in <module>
    from .exceptions import raise_with_traceback
  File "/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/exceptions.py", line 34, in <module>
    from msrest.serialization import Deserializer
  File "/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/serialization.py", line 46, in <module>
    from .pipeline.universal import RawDeserializer
  File "/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/pipeline/__init__.py", line 54, in <module>
    from ..exceptions import ClientRequestError, raise_with_traceback
ImportError: cannot import name 'ClientRequestError' from 'msrest.exceptions' (/Users/Adam/.virtualenvs/msrest/lib/python3.7/site-packages/msrest/exceptions.py)

My use case is that I am trying to generate documentation for code which uses msrest.
I am using https://github.com/agronholm/sphinx-autodoc-typehints.
This sets typing.TYPE_CHECKING = True.

The most recent version of msrest for which this works is 0.4.29.