azure/autorest.python

Extension for AutoRest (https://github.com/Azure/autorest) that generates Python code

Python Generator Packages

Requirements

• Node 16+
• Python 3.7+

Develop

This project uses pnpm workspaces to manage multiple packages.

pnpm install

2. Build

pnpm build

3. Build TS in watch mode

pnpm watch

4. Clean

pnpm clean

Release a new version

Semi-auto:

Branch publish/auto-release should be automatically updated with the latest changelog. Give it 5min after merging a PR or check the status of the Release action

Then go to https://github.com/Azure/autorest.python/pull/new/publish/auto-release and create this PR.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

autorest.python's Issues

XML tests

Support XML tests

Blocked by #136

Arrays tests

Tests about:

• BodyArray

#161 should be adressed first

Complex models tests

Support:

• BodyComplex

Only the part that is not about polymorphism

#161 should be adressed first

Parameters for credential should be "credential" and not "credentials"

@johanste FYI

Model parameter name in python duplicates "self" based on field name in swagger spec

With a swagger spec that identifies a model with a field named "self", the autorest generator for python results in this error on import:

SyntaxError: duplicate argument 'self' in function definition

These should ideally be renamed as something like _self

Support test get_mbcs

Having a pytest run of this test.

Polymorphism

The fish part of BodyComplex

Support basic_putValid

Having a pytest run of this test

Header response not available in YAML

https://github.com/Azure/autorest.modelerfour/issues/34

Paging tests

Support

• Paging
• CustomUrlPaging

Parameter grouping

Waiting on modelerfour to provide it to us:
https://github.com/Azure/autorest.modelerfour/issues/10

Improve exception generation

Message from @iscai-msft :
error generating HTTP, AzureHead, Paging, AzureSpecials etc (getting schema error when it looks in exception list) -> ask about how the excpetions list has been reformatted

Fix Python case

Some Python naming convention are not yet translater correctly to new autorest:
some param names aren't properly converted to python case (they're still capital) (RequiredOptional)

Swagger comments with \U, \N generate python errors

We have an API that has a description property containing the string:
"Value\Units"

When this description is converted to python, it generates the following string:
"""Value\Units"""

When evaluated, this gives the error:
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 5-6: truncated \UXXXXXXXX escape

I couldn't tell for sure, but I didn't see anything in the OpenAPI definition that precluded the use of \ or \U in a description of an API.
https://swagger.io/specification/
It says:

Where common markdown syntax links to here.

LRO tests

Support LRO

Make --keep-version-file actually interesting

Autorest is supposed to have an option --keep-version-file that act as below:

• If this is the first generation, just as usual creates a version.py file
• If a file already exists, keep it as-is and don't touch it

Problem here:

• I'm not actually sure it works in the first place, it was never tested
• It exists in the "vanilla" generator, but was never ported to "azure" generator

This issue is about making sure it works as expected, and once it works push it to "azure" as well.

Enum with different cases generates invalid code

Example:

class Foo(str, Enum):
   error = "ERROR"
   error = "Error"

Validation code for enum is missing.

It seems when the a property is defined as enum (not x-ms-enum), then no validation code is generated for it. Neither the possible/allowed strings are available as an array or object in the generated code, nor does the validation logic in msrest/serialization.py have something for enum. Is this expected behavior?

Support basic_getvalid

Having pytest run of this test

xref link broken for msrestazure.azure_active_directory param

The generated docstring for the msrestazure.azure_active_directory parameter results in an invalid xref link in the reference docs for each library's management client.

Source: GeneratorSettingsPya.cs

Example from ContainerInstanceManagement:

Resultant generated code: container_instance_management_client.py

Cc: @dendeli-msft @rloutlaw

Using not ARM generator, credentials doc says None instead of a link to msrest

class SpellCheckAPIConfiguration(Configuration):
    """Configuration for SpellCheckAPI
    Note that all parameters used to create this instance are saved as instance
    attributes.
    :param credentials: Subscription credentials which uniquely identify
     client subscription.
    :type credentials: None
    :param str base_url: Service URL
    """

    def __init__(
            self, credentials, base_url=None):

        if credentials is None:
            raise ValueError("Parameter 'credentials' must not be None.")
        if not base_url:
            base_url = 'https://api.cognitive.microsoft.com/bing/v7.0'

Ref: Azure/azure-sdk-for-python#2863

Dictionnary tests

Should support

• BodyDictionary

#161 should be adressed first

Primitive tests

Support tests about Primitive:

• BodyBoolean
• BodyByte
• BodyDate
• BodyDateTime
• BodyDateTimeRfc1123
• BodyDuration
• BodyInteger
• BodyNumber
• BodyString
• Header
• Http
• Url
• UrlMultiCollectionFormat

FATAL: Error: Unrecognized schema type:'integer' / format: Int32 (LRO)

Crash of modelerfour https://github.com/Azure/autorest.modelerfour/issues/29

Full type hints in models_py3 and aio folder

Update the code generator to have the full annotations on models_py3 and aio folder. Since this is Python 3 files, direct annotations can be used. Some of them are done already.

So roughly this means:

• Compute the list of necessary typing import (List? Optional? Dict?, Any?) and import just that
• Annotate correctly the code
• Be sure that if a class is just needed to type annotation and not direct usage, import it under TYPE_CHECKING global constant

#82 needs to be done before to measure progress and status

Polymorphism

We don't the necessary information yet to conclude on this one, blocked by https://github.com/Azure/autorest.modelerfour/issues/12

Automatic generation of ApiVersion as a query param even when not in Swagger

Not sure if bug or feature request but in my Swagger file I didn't specify any:
"ApiVersionParameter": { "name": "api-version", "in": "query", "required": true, "type": "string", "description": "Client Api Version." }

But in my generated code I still get:
query_parameters['api-version'] = self._serialize.query("self.api_version", self.api_version, 'str')
In every operation, but I don't want it.

I'm using "@microsoft.azure/autorest.python": "^4.0.67", "autorest": "^3.0.5165",

Constant

Old autorest will generate "constant" if there was an enum with one unique value. We need to decide what to do once the modelerfour discussion cross language has been done:
https://github.com/Azure/autorest.modelerfour/issues/27

MISSING_DESCRIPTION discussion

I understand that adding MISSING_DESCRIPTION is nice for development, created this issue to be sure we don't forget to address it before we assume we're done.
https://github.com/Azure/autorest.modelerfour/issues/30

Support test put_mbcs

Having a pytest run of this test

Paging operation doesn't pass kwargs to pipeline

Making unable to pass "enable_loggin" or things like this:

autorest.python/test/azure/Expected/AcceptanceTests/Paging/paging/operations/_paging_operations.py

Line 82 in 5b27658

pipeline_response = self._client._pipeline.run(request)

[Epic] Generate operations

Operations generation didn't really start yet

Generate type hints for Python code

I.e. https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code

Test infrastructure for Autorest v3

Work on https://github.com/Azure/autorest.python/tree/autorestv3

Short version: port to Python these two calls:

• gulp regenerate
• gulp test

Suggestion is to use "invoke": http://www.pyinvoke.org/

gulp regenerate:

Python version in pyinvoke of this gulp script:
https://github.com/Azure/autorest.python/blob/azure-core/.gulp/regeneration.iced

Assuming the testserver has been installed (npm install), this reads the Swaggers and call our current autorest with it to generate in the folders test/vanilla/Expected/AcceptanceTests and test/azure/Expected/AcceptanceTests

gulp test

Python version in pyinvoke of this part script:

autorest.python/.gulp/gulpfile.iced

Lines 23 to 27 in 1d943d2

	# Run language-specific tests:
	task 'test', "", [], (done) ->
	await execute "tox", { cwd: './test/vanilla/' }, defer code, stderr, stdout
	await execute "tox", { cwd: './test/azure/' }, defer code, stderr, stdout
	done();

IOW, run tox on both vanilla and azure folder

Enum tests

Support tests about:

• ExtensibleEnums

#161 should be adressed first

File tests

Support BodyFile

Blocked by #134

Introduce mypy in the tests

This issue is about introducing mypy as part of the tox run, for both vanilla and azure. This means:

• tox should install mypy as a dev dependency
• tox should run mypy before doing the unittests

mypy should be run on both the generated code and the unittests (both Expected and AcceptanceTests folder)

Build should be clean with no errors. This might require update of annotations in msrest and msrestazure to solve some of them.
https://github.com/Azure/msrest-for-python/
https://github.com/Azure/msrestazure-for-python/

Model flattening

Figure out what to do with model flattening, what part is modelerfour and what part is not:
https://github.com/Azure/autorest.modelerfour/issues/14
https://github.com/Azure/autorest.modelerfour/issues/13

Paged AutoRest models do not have client_request_id

https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/batch/azure-batch/azure/batch/models/_paged_models.py

Non-major. Ran into this today when attempting to track down customer request by id.

No XML information in the model

See https://github.com/Azure/autorest.modelerfour/issues/26

CustomUrl tests

Support:

• CustomBaseUri
• CustomBaseUriMoreOptions

Property name cannot be used as an identifier

Message from @iscai-msft:

• error generating Report, Validation, ModelFlattening, AzureReport (getting a property name cannot be used as an identifier error)

Can't deserialize header that are enums

https://github.com/Azure/autorest.modelerfour/issues/35

Autorest should support a mixed JSON/XML client

Example: method 1 takes XML, and method takes JSON.
https://docs.microsoft.com/en-us/rest/api/storageservices/create-permission

Simplify model generation

The current file structure of generated SDKs creates a "models" folder that contains:

• One file per paging result
• One file for all enums
• One file per model generic Python 2 / 3
• One file per model specific to Python 3 (which is loaded to replace the previous one if Python 3)

This creates a LOT of files, example with Network:
https://github.com/Azure/azure-sdk-for-python/tree/master/azure-mgmt-network/azure/mgmt/network/v2018_10_01/models

We realized recently that all these tiny files have a massive impacts on performance:

• It makes the wheel 80% bigger than needed
• It creates a lot of contention on the I/O (writing small files)
• Pre-compilation of the package by pip is extremely long
• It increases the likeliness of long path on Windows issue

Not only performance, but also documentation since it provides two ways to import the same file:

• import client.models.MyClass
  or
• import client.models.my_class.MyClass

And actually on Python 3, these are not the same, since the first one includes the automatic choice between Python 3 or Python 2 file (where the second one is Python 2 specific). This creates issues and bugs in the CLI already.

We need to refactor this folder with less files, and fixing the doubling syntax at the same time. This is a breaking changes, and release should upgrade the major version of the resulting package.

I made a prototype of this in Python. This script should be called on generated Autorest folder (loaded in dev mode in a venv) that post-process and patch the files manually. The script is here:
https://gist.github.com/lmazuel/dad0ac92c733b27f6d81ea57a47625ce#file-patch_models_v2-py

The result is:

• One file for all paging results
• One file for all enums
• One file for all models generic Python 2 / 3
• One file for all models specific to Python 3 (which is loaded to replace the previous one if Python 3)

I released one package already with this patched folder, which can be used as reference:
https://github.com/Azure/azure-sdk-for-python/tree/master/azure-cognitiveservices-vision-contentmoderator/azure/cognitiveservices/vision/contentmoderator/models

CLI already releases using this script to post-process all SDKs (this means CLI doesn't ship the raw SDK but a patched version). We don't have any issues and then assume the prototype works as expected.

The target of this issue is to track backporting the logic of the Python script into Autorest by default.

The tricky part is the inheritance part. Because this new version will declare all classes, but they should be declared in inheritance order, so if B subclass A, A is defined before B in the Python file. The Python script here is doing it using the magic attribute mro, and this logic has to be translated in Autorest core model (which is less intuitive).
Once this part is done, then it's mostly changing the output file of models and is not complicated.

FATAL: Error: Not able to process media type multipart/form-data at this moment. (BodyFormData)

Crash of modelerfour on BodyForm data https://github.com/Azure/autorest.modelerfour/issues/28

Fix enum comments

For enum service we generated this:

not_applicable = "NotApplicable"  #: User either hasn't started configuring their template 
or they haven't started the configuration process.

instead of

not_applicable = "NotApplicable"  #: User either hasn't started configuring their template or they haven't started the configuration process.

See https://github.com/Azure/azure-sdk-for-python/pull/4366/files

Reminder

> linux-x64 restart

Create ivar doc for operations models alias

See Azure/azure-sdk-for-python#5172

Models regeneration for Autorest v3

Work on https://github.com/Azure/autorest.python/tree/autorestv3

Short version:
Make the template structure to generate the models folder ONLY (no operation yet at this point).

Dependency on #114 first to regenerate easily