azure/autorest.cli

Autorest plugin to Azure CLI command modules and other tooling

See documentation here

use-extension:
  "@microsoft.azure/autorest.modeler": "2.3.45" # keep in sync with package.json's dev dependency in order to have meaningful tests

pipeline:
    cli/imodeler1:
        input: openapi-document/identity
        output-artifact: code-model-v1
        scope: cli
    cli/commonmarker:
        input: imodeler1
        output-artifact: code-model-v1
    cli/cm/transform:
        input: commonmarker
        output-artifact: code-model-v1
    cli/cm/emitter:
        input: transform
        scope: scope-cm/emitter
    cli/generate:
        plugin: cli
        input: cm/transform
        output-artifact: source-file-cli
    cli/transform:
        input: generate
        output-artifact: source-file-cli
        scope: scope-transform-string
    cli/emitter:
        input: transform
        scope: scope-cli/emitter

scope-cli/emitter:
  input-artifact: source-file-cli
  output-uri-expr: $key

output-artifact:
- source-file-cli

#``` yaml #use-extension:

"cli": "$(this-folder)"

#```

autorest.cli's Issues

Enable --cli.debugger option

[MM]Porperty Type Error in api.yaml

Property Name	Original Type	Should Be
'excludedKbNumbers'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'includedKbNumbers'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'excludedPackageNameMasks'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array  (item_type: Api::Type::String)
'includedPackageNameMasks'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'nonAzureComputerNames'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'locations'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'weekDays'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'monthDays'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::Integer)
'includedPackageNameMasks'	!ruby/object:Api::Azure::Type::ResourceReference	!ruby/object:Api::Type::Array (item_type: Api::Azure::Type::ResourceReference)
'scope'	!ruby/object:Api::Azure::Type::ResourceReference	!ruby/object:Api::Type::Array  (item_type: Api::Azure::Type::ResourceReference)
'azureQueries'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array (item_type: !ruby/object:Api::Type::NestedObject)
'nonAzureQueries'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::Type::NestedObject)
'advancedSchedule'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::Type::NestedObject)
'monthlyOccurrences'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::T

[MM] NestedObject flatten

If a NestedObject only contains one field, the reviewers from terraform would require us to flatten the structure. For example, on behalf of cli, we have an api.yml saying

      - !ruby/object:Api::Type::NestedObject
        name: 'identity'
        description: ''
        required: false
        azure_sdk_references: ['/identity']
        properties:
          - !ruby/object:Api::Type::Enum
            name: 'type'
            description: 'The type of Managed Identity used by the DiskEncryptionSet. Only SystemAssigned is supported.'
            required: false
            azure_sdk_references: ['/identity/type']
            values:
              - :SystemAssigned
            default_value: :SystemAssigned

This should be replaced by

      - !ruby/object:Api::Type::Enum
        name: 'identityType'
        description: 'The type of Managed Identity used by the DiskEncryptionSet. Only SystemAssigned is supported.'
        required: false
        azure_sdk_references: ['/identity/type']
        values:
          - :SystemAssigned
        default_value: :SystemAssigned

This issue was initially filed to MM here

[MM]go_type_name is not correct in api.yaml

Use the SoftwareUpdateConfiguration resource api.yaml file here as an example.

Property	Original Name	Should Be
'/'	softwareUpdateConfiguration	SoftwareUpdateConfiguration
'/properties'	softwareUpdateConfigurationProperties	SoftwareUpdateConfigurationProperties
'/properties/updateConfiguration'	updateConfiguration	UpdateConfiguration
'/properties/tasks'	softwareUpdateConfigurationTasks	SoftwareUpdateConfigurationTasks
'/properties/tasks/preTask'	taskProperties	TaskProperties

[MM] Missing fields in `properties` section

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

In the generated magic-module-input files computedisk.yaml, we have some field missing in the properties section. To be specific, for struct KeyVaultAndSecretReference (here), one of the field SourceVault is not included in properties section, thus those fields will not show in the terraform schema.

[MM] issues encountered when generating SpringCloud terraform codes

using file: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/appplatform/resource-manager/Microsoft.AppPlatform/preview/2019-05-01-preview/appplatform.json

issue 1

it generates

client := meta.(*ArmClient).servicesClient

however, it should like this

client := meta.(*ArmClient).AppPlatform.ServicesClient

issue 2

'serviceName': !ruby/object:Api::Azure::SDKTypeDefinition::StringObject
            id_portion: Spring
            go_variable_name: name

attribute serviceName and name generates two attribute name in terraform codes

issue 3

attribute config_server_properties should be flattened

issue 4

generation code Uri is not consistent with go sdk model attribute URI

issue 5

generation code

resource := appplatform.ServiceResource{
        ClusterResourceProperties: ....
    }

however, in go sdk model, the attribute is Properties but not ClusterResourceProperties

The link in MANUAL.md has staled

The link in MANUAL.md https://github.com/zikalino/cli-hatchery has staled, please update.
Thanks!

Autogenerated code for Peering integration test has some issues

I followed the doc here to generate python sdk integration test for peering service. And there are some issues:
1.property peering_service_location needs to be peeringServiceLocation.
2.PEERING_SERVICE_NAME and PREFIX_NAME is not generated from example, instead there is a generic SERVICE_NAME but not used.
3.The parameter flatten feature is considered in generated test code. When there is <= 2 properties and x-ms-client-flatten set true, the paramters inside the properties will be flattened, instead of passing

{
    "properties": {
          "prefix": "192.168.1.0/24"
     }
 }

we should pass in prefix="192.168.1.0/24".
4.result_create = azure_operation_poller.result() is used for all responses, but the response from PeeringService doesn't have attribute 'result'.
5.some of the values in example should be valid ones like peeringServiceLocation to make the example work, the examples in peering swagger just have fake values that don't work. (This should be fixed in swagger example)

Issue with renaming xxx_name to name

This code needs to be cleaned up:

autorest.cli/Common/MapFlattener.ts

Line 46 in f958360

// OPTIONS ARE NOT SORTED CORRECTLY

[MM]It doesn't generate the required properties for separated update function

It doesn't generate the required properties for separated update function in api.yml and threw below error message while running MM command. MM just generates partially properties like below snapshot. After investigated, MM failed in file resource.erb of MM since it is requesting a property "/" under update function but it doesn't exist in api.yml so that it failed. So the root cause is that autorest.cli doesn't generate the property "/" and other required properties in api.yml. So please have a look and fix it. Thanks.

Resource in azure rest api specs: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/hanaonazure/resource-manager/readme.md

It just generated porperties "resourceGroupName" and "sapMonitorName". It should also generate other properties like "/", "/location", "/properties", "/hanaSubnet" and so on.

File resource.erb of MM:

Error message:

It should generate the properties like create function. Please see below snapshot.

SDK reference for tags in MM input

Because Python SDK has its own flattening logic, thus according the update operation definition, this property should be copied to tags parameter (the key defined in azure_sdk_definition is tags) in Python SDK, at the same time, it is the Tags structure field in Go SDK (the key defined in azure_sdk_definition is /tags). We would like to include both cases in azure_sdk_references.

    Suggested change
    
      
    
  


  
    
      
              azure_sdk_references: ['/tags']
    
      
        
                azure_sdk_references: ['tags', '/tags']
      
  


          
            
            

              

              
              
              
                
                  Commit suggestion
                  
                
              
              
                Add suggestion to batch

https://github.com/VSChina/magic-modules/pull/5/files#r271985781

[MM] suboptions are not included in readonly part

resource: automationagentregistrationinformation
Microsoft.Automation/stable/2015-10-31/dscNode.json

The 'keys' property does not has suboptions in readonly part.

• !ruby/object:Api::Type::NestedObject
  name: 'keys'
  description: 'Gets or sets the agent registration keys.'
  output: true
  azure_sdk_references: ['/keys']
  properties:

Add "properties" into MM input files

as described here

https://github.com/VSChina/magic-modules/pull/5/files#r274632280

[MM] `[unknown-primary[object]]` appears in `properties`

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

In the generated magic-module-input files computevirtualmachine, some properties are generated as:

            - !ruby/object:Api::Azure::Type::[unknown-primary[object]]
              name: 'settings'
              description: 'Json formatted public settings for the extension.'
              required: false
              azure_sdk_references: ['/resources/settings', '/resources/properties/settings']
            - !ruby/object:Api::Azure::Type::[unknown-primary[object]]
              name: 'protectedSettings'
              description: 'The extension can contain either protectedSettings or protectedSettingsFromKeyVault or no protected settings at all.'
              required: false
              azure_sdk_references: ['/resources/protectedSettings', '/resources/properties/protectedSettings']

[MM] Missing properties

addonsoperation | package-2017-05

--- !ruby/object:Api::Product
name: Azure Operation Management
versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: NotUsedInAzure
scopes:
  - NotUsedInAzure
objects:
  - !ruby/object:Api::Resource
    name: Operation
    api_name: Addons
    base_url: NotUsedInAzure

    azure_sdk_definition: !ruby/object:Api::Azure::SDKDefinition
      provider_name: Microsoft.Addons
      go_client_namespace: addons
      go_client: operationsClient
      python_client_namespace: azure.mgmt.addons
      python_client: AzureAddonsResourceProviderClient.operations
      list_by_subscription: !ruby/object:Api::Azure::SDKOperationDefinition
        go_func_name: List
        python_func_name: list
        request:

    description: |
      Manage Azure Operation instance.
    properties:
      # read only values included only in response

`softwareupdateconfiguration` api.yaml fix

Here are the blocking issues:

• Escape single quote ' here, here.
• 'Api::Azure::Type::Integer' should be Api::Type::Integer.
• Remove prefix.
• Nested Object should contain sub-properties here.
• resource_type_name: 'Virtual Machine' is required for this Api::Azure::Type::ResourceReference. And resource_type_name: Subscription is required for this ResourceReference.
• Rename list to list_by_parent.

Here are the non-blocking issues:

• Remove name because it is duplicated with name.
• Please remove input: true for pure output attributes.
• It seems that the type of 'nextRunOffsetMinutes' should be Api::Azure::SDKTypeDefinition::FloatObject instead of IntegerObject from its definition in Go SDK. Can you double-check about that?

Flatten storage account id

Flatten storageAccountId in batch account module:

https://github.com/VSChina/magic-modules/pull/5/files#r274635784

[MM] Autorest.cli generates incorrect api.yml for sub resource Account of NetApp

I am trying to generate api.yml for sub resource "Account" of NetApp. After generated api.yml with command "docker run --rm -i -t -v C:/terraform/projects/azure-rest-api-specs:/azure-rest-api-specs -v C:/terraform/generated:/generated zikalino/autorest.cli /azure-rest-api-specs/specification/netapp/resource-manager/readme.md", I found the api.yml missed many necessary fields like "properties.activeDirectories" and etc. This is the API doc https://docs.microsoft.com/zh-cn/rest/api/netapp/accounts/createorupdate of new resource "NetAppAccounts". Attached the input file "readme.md" and output file "api.yml".

api.zip
readme.zip

[MM] Missing properties in `azure_sdk_references`

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

In the generated magic-module-input files computediskencryptionset, we have two sdk_reference
/properties/previousKeys/sourceVault and /properties/previousKeys/sourceVault/id missing.

Ketyvault reference URL nor required and input only

https://github.com/VSChina/magic-modules/pull/5/files#r271989273

Consuming readme.cli.md

make sure that readme.cli.md is read along readme.md

Create official container

Currently we use zikalino/azure-sdk-tools container for both Python SDK and autorest.cli tools.
This should be made official.

`frontdoor` api.yaml fix

The issues below are block issues.

• Remove prefix.
• Escape the single quote ' in description here, here, here and here.
• Integer types in properties should be Api::Type::Integer instead of Api::Azure::Type::Integer.
• Nested Object should contain sub-properties. For example: routeConfiguration.
• Rename 'list' to list_by_subscription.

Here are the issues not blocking us.

• Remove name because it is duplicated with name.
• Remove 'input: true' from pure output attributes.

[MM]Options missing in Module

Some option is missing in ModuleOptions

For resource 'Reservation' \azure-rest-api-specs\specification\reservations\resource-manager\readme.md,
the parameter option 'reservationId' is not included in ModuleOptions in CliModule
method:Reservation_Get

[MM] Empty properties for NestedObject

actiongroup | package-2019-06-01-only

- !ruby/object:Api::Type::Array
        name: 'smsReceivers'
        description: 'The list of SMS receivers that are part of this action group.'
        output: true
        azure_sdk_references: ['/smsReceivers', '/properties/smsReceivers']
        item_type: !ruby/object:Api::Type::NestedObject
          properties:
      - !ruby/object:Api::Type::Array
        name: 'webhookReceivers'
        description: 'The list of webhook receivers that are part of this action group.'
        output: true
        azure_sdk_references: ['/webhookReceivers', '/properties/webhookReceivers']
        item_type: !ruby/object:Api::Type::NestedObject
          properties:

[MM] integration naming-conversion rules in GO-SDK to autorest.cli terraform

Several issues in softwareupdateconfiguration

1. read is missing. Here is the API definition: https://docs.microsoft.com/en-us/rest/api/automation/softwareupdateconfigurations/getbyname.
2. Type of duration is missing: https://github.com/Azure/magic-module-specs/blob/master/softwareupdateconfiguration/api.yaml#L91, https://github.com/Azure/magic-module-specs/blob/master/softwareupdateconfiguration/api.yaml#L166. I would say to use !ruby/object:Api::Azure::SDKTypeDefinition::ISO8601DurationObject.
3. If the type is float: https://github.com/Azure/magic-module-specs/blob/master/softwareupdateconfiguration/api.yaml#L212, just using !ruby/object:Api::Azure::SDKTypeDefinition::FloatObject. If it is interger, then using one of the integer types specified here: https://github.com/Azure/magic-module-specs/blob/master/RESOURCE_SPEC.md#sdk-operation-request.
4. Use !ruby/object:Api::Azure::SDKTypeDefinition::ISO8601DateTimeObject for date time: https://github.com/Azure/magic-module-specs/blob/master/softwareupdateconfiguration/api.yaml#L209.

For the corresponding types defined in properties, please use !ruby/object:Api::Azure::Type::ISO8601Duration, !ruby/object:Api::Azure::Type::ISO8601DateTime and the number types specified here: https://github.com/Azure/magic-module-specs/blob/master/RESOURCE_SPEC.md#devops-user-interface-definition.

[MM]Integer Type For Go SDK in api.yaml

Property	Original Azure SDK Type	Should Be
'/properties/scheduleInfo/interval'	Azure::SDKTypeDefinition::IntegerObject	Azure::SDKTypeDefinition::Integer32Object
'/properties/scheduleInfo/advancedSchedule /monthlyOccurrences/occurrence'	Azure::SDKTypeDefinition::IntegerObject	Azure::SDK

Setup issues for Generating Python SDK Integration Test on ignite-fixes branch

I followed the doc here to generate python sdk test. And there are some issues when setup:

1. The dependency package autorest-extension-base is somehow not found and need to be mannually installed.
2. node-yaml and js-yaml is also needed for the tool to run.

Several issues in automation job resource

1. The type of parameters is missing here: https://github.com/Azure/magic-module-specs/blob/master/automationjob/api.yaml#L41. According to the website, we should use !ruby/object:Api::Azure::SDKTypeDefinition::StringMapObject.
2. Missing type as well: https://github.com/Azure/magic-module-specs/blob/master/automationjob/api.yaml#L130. We should use !ruby/object:Api::Type::KeyValuePairs.
3. Output attribute id is missing in properties.
4. To control the job (start, stop, suspend), please also include a job_status attribute in properties just like other Ansible schema did (virtual machine).

[MM] support enum array

support enum array in autorest.cli

`batchapplication` api.yaml fix

Only the prefix, Properties and list are blocking issues, others are not.

• Remove prefix
• Rename list to list_by_parent
• Remove input: true from a pure output attribute (for example id)
• In Go SDK, Properties in Create, Update and Read are named as ApplicationProperties.

[MM] duplicated entries in definitions

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

The generated computevirtualmachine/api.yaml contains two entries with same name list_by_subscription:

      list_by_subscription: !ruby/object:Api::Azure::SDKOperationDefinition
        go_func_name: ListAll
        python_func_name: list_all
        request:
      list_by_subscription: !ruby/object:Api::Azure::SDKOperationDefinition
        go_func_name: ListByLocation
        python_func_name: list_by_location
        request:
          '/location': !ruby/object:Api::Azure::SDKTypeDefinition::StringObject
            go_field_name: Location
            python_field_name: location

[MM] No properties body specified for resource.

addonssupportplantype | package-2017-05

description: |
      Manage Azure SupportPlanType instance.
    properties:
      - !ruby/object:Api::Type::String
        name: 'providerName'
        description: 'The support plan type. For now the only valid type is "canonical".'
        required: true
        input: true
        azure_sdk_references: ['providerName']
      - !ruby/object:Api::Type::String
        name: 'name'
        description: 'The Canonical support plan type.'
        required: true
        input: true
        azure_sdk_references: ['planTypeName']
      # read only values included only in response
      - !ruby/object:Api::Type::String
        name: 'id'
        description: 'The id of the ARM resource, e.g. "/subscriptions/{id}/providers/Microsoft.Addons/supportProvider/{supportProviderName}/supportPlanTypes/{planTypeName}".'
        output: true
        azure_sdk_references: ['/id']
      - !ruby/object:Api::Type::String
        name: 'name'
        description: 'The name of the Canonical support plan, i.e. "essential", "standard" or "advanced".'
        output: true
        azure_sdk_references: ['/name']
      - !ruby/object:Api::Type::String
        name: 'type'
        description: 'Microsoft.Addons/supportProvider'
        output: true
        azure_sdk_references: ['/type']
      - !ruby/object:Api::Type::Enum
        name: 'provisioningState'
        description: 'The provisioning state of the resource.'
        output: true
        azure_sdk_references: ['/provisioningState', '/properties/provisioningState']
        values:
          - :Succeeded
          - :Failed
          - :Cancelled
          - :Purchasing
          - :Downgrading
          - :Cancelling
          - :Upgrading

[MM] Inconsistency in property types

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

Struct VirtualMachineProperties, its field AvailabilitySet and VirtualMachineScaleSet both are of type *SubResource, but they are generated as different types in computevirtualmachine/api.yaml:

      - !ruby/object:Api::Type::NestedObject
        name: 'availabilitySet'
        description: 'Specifies information about the availability set that the virtual machine should be assigned to. Virtual machines specified in the same availability set are allocated to different nodes to maximize availability. For more information about availability sets, see [Manage the availability of virtual machines](https://docs.microsoft.com/azure/virtual-machines/virtual-machines-windows-manage-availability?toc=%2fazure%2fvirtual-machines%2fwindows%2ftoc.json). <br><br> For more information on Azure planned maintenance, see [Planned maintenance for virtual machines in Azure](https://docs.microsoft.com/azure/virtual-machines/virtual-machines-windows-planned-maintenance?toc=%2fazure%2fvirtual-machines%2fwindows%2ftoc.json) <br><br> Currently, a VM can only be added to availability set at creation time. An existing VM cannot be added to an availability set. <br><br>This property cannot exist along with a non-null properties.virtualMachineScaleSet reference.'
        required: false
        azure_sdk_references: ['/availabilitySet', '/properties/availabilitySet']
        properties:
          - !ruby/object:Api::Azure::Type::ResourceReference
            name: 'id'
            description: 'Resource Id'
            required: false
            sample_value: /subscriptions/{{ subscription_id }}/resourceGroups/{{ resource_group }}/providers/Microsoft.Compute/availabilitySets/{{ name }}
            resource_type_name: TBD
            azure_sdk_references: ['/availabilitySet/id', '/properties/availabilitySet/id']
      - !ruby/object:Api::Type::NestedObject
        name: 'virtualMachineScaleSet'
        description: 'Specifies information about the virtual machine scale set that the virtual machine should be assigned to. Virtual machines specified in the same virtual machine scale set are allocated to different nodes to maximize availability. Currently, a VM can only be added to virtual machine scale set at creation time. An existing VM cannot be added to a virtual machine scale set. <br><br>This property cannot exist along with a non-null properties.availabilitySet reference. <br><br>Minimum api‐version: 2019‐03‐01'
        required: false
        azure_sdk_references: ['/virtualMachineScaleSet', '/properties/virtualMachineScaleSet']
        properties:
          - !ruby/object:Api::Type::String
            name: 'id'
            description: 'Resource Id'
            required: false
            azure_sdk_references: ['/virtualMachineScaleSet/id', '/properties/virtualMachineScaleSet/id']

One of them is String, the other is ResourceReference.
Since these kind of fields usually need to be flattened, String type is more preferred.

Properly support arrays in the UX (!ruby/object:Api::Type::Array)

After studying recent version of the spec here:

autorest.devops shall use
!ruby/object:Api::Type::Array: an array of any type.
with
!ruby/object:Api::Type::NestedObject: a structure containing sub-properties.

for arrays.

Previously I assumed that MM somehow magically maps the UX structure to SDKTypeDefinition::ComplexArrayObject without specifically defining arrays

[MM] Autorest.cli fails to generate api.yaml for AppServiceCertificateOrders

I'm using this json file generating api.yaml but get below warning and nothing generated for magic-modules.

WARNING: Cannot read property 'x-ms-examples' of undefined -- "TypeError: Cannot read property 'x-ms-examples' of undefined\n at new ExampleProcessor (/autorest.cli/dist/ExampleProcessor.js:17:62)\n at Object. (/autorest.cli/dist/index.js:77:36)\n at Generator.next ()\n at fulfilled (/autorest.cli/dist/index.js:4:58)"

Seems it's because the rest-api-specs missing some examples for cli but I expect autorest.cli can generate something for magic-modules even with an incomplete rest-api-specs.

Several issues in batchapplication

Here is the current api.yaml: https://github.com/Azure/magic-module-specs/blob/master/batchapplication/api.yaml.

I found several issues:

• The ID (output property) is missing in properties
• Go variable name should use camelCase instead of PascalCase (it is different from go_field_name and go_type_name, they should use PascalCase): https://github.com/Azure/magic-module-specs/blob/master/batchapplication/api.yaml#L33

[MM] Generated go-enum values cannot contains hyphens or underscores

I am generating api.yaml file using autorest.cli with input file: azure-rest-api-specs\specification\compute\resource-manager\readme.md

Tag in use

package-2019-07.

Cli section added in readme.md

cli:
  cli-name: compute
  azure-arm: true
  license-header: MICROSOFT_MIT_NO_VERSION
  payload-flattening-threshold: 2
  namespace: azure.mgmt.compute
  package-name: azure-mgmt-compute
  clear-output-folder: false
  debug: true
  disable-azure-cli: true

Problem encountered

In the generated magic-module-input files computevirtualmachine, the generated values of go enum variable can contain hyphens or underscores or both. Golang will give warnings about snake cased variable names, so in the generation of Go SDK, hyphens and underscores in enum variables will be removed. For instance, in the generated api.yaml, in the representation of an enum variable

- !ruby/object:Api::Type::Enum
  name: 'storageAccountType'
  description: 'Specifies the storage account type for the managed disk. NOTE: UltraSSD_LRS can only be used with data disks, it cannot be used with OS Disk.'
  required: false
  sample_value: Standard_LRS
  azure_sdk_references: ['/storageProfile/osDisk/managedDisk/storageAccountType', '/properties/storageProfile/osDisk/managedDisk/storageAccountType']
  values:
    - :Standard_LRS
    - :Premium_LRS
    - :StandardSSD_LRS
    - :UltraSSD_LRS
  default_value: :Standard_LRS

All these underscores should be removed. In the definition of vmSize, there are more enum variables contained hyphens and underscores.

- !ruby/object:Api::Type::Enum
  name: 'vmSize'
  description: 'Specifies the size of the virtual machine. For more information about virtual machine sizes, see [Sizes for virtual machines](https://docs.microsoft.com/azure/virtual-machines/virtual-machines-windows-sizes?toc=%2fazure%2fvirtual-machines%2fwindows%2ftoc.json). <br><br> The available VM sizes depend on region and availability set. For a list of available sizes use these APIs:  <br><br> [List all available virtual machine sizes in an availability set](https://docs.microsoft.com/rest/api/compute/availabilitysets/listavailablesizes) <br><br> [List all available virtual machine sizes in a region](https://docs.microsoft.com/rest/api/compute/virtualmachinesizes/list) <br><br> [List all available virtual machine sizes for resizing](https://docs.microsoft.com/rest/api/compute/virtualmachines/listavailablesizes)'
  required: false
  sample_value: Standard_D2_v2
  azure_sdk_references: ['/hardwareProfile/vmSize', '/properties/hardwareProfile/vmSize']
  values:

Properties missing in several resources

From the documentation, there are a lot of properties (request body) in API Management Diagnostic: https://docs.microsoft.com/en-us/rest/api/apimanagement/2019-01-01/diagnostic/createorupdate#request-body.

But the generated file only contains very few: https://github.com/Azure/magic-module-specs/blob/master/products/azure_rm_apimanagementdiagnostic/api.yaml#L50.

Maybe I am looking at the wrong file?

[MM]go_enum_const_prefix is not correct in api.yaml

Use the SoftwareUpdateConfiguration resource api.yaml file here as an example.

Property	Original Name	Should Be
'/properties/updateConfiguration/ windows/includedUpdateClassifications'	NONE	WindowsUpdateClasses

[MM] Common errors in api.yaml

Use the SoftwareUpdateConfiguration resource api.yaml file here as an example.

1. [go_type_name]

Property	Original Name	Should Be
'/'	softwareUpdateConfiguration	SoftwareUpdateConfiguration
'/properties'	softwareUpdateConfigurationProperties	SoftwareUpdateConfigurationProperties
'/properties/updateConfiguration'	updateConfiguration	UpdateConfiguration
'/properties/tasks'	softwareUpdateConfigurationTasks	SoftwareUpdateConfigurationTasks
'/properties/tasks/preTask'	taskProperties	TaskProperties

2. [go_field_name]

Property	Original Name	Should Be
'/properties'	Properties	SoftwareUpdateConfigurationProperties

3. [go_enum_const_prefix]

Property	Original Name	Should Be
'/properties/updateConfiguration/ windows/includedUpdateClassifications'	NONE	WindowsUpdateClasses

4. Porperty Type Error

Property Name	Original Type	Should Be
'excludedKbNumbers'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'includedKbNumbers'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'excludedPackageNameMasks'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array  (item_type: Api::Type::String)
'includedPackageNameMasks'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'nonAzureComputerNames'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'locations'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'weekDays'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::String)
'monthDays'	!ruby/object:Api::Type::String	!ruby/object:Api::Type::Array (item_type: Api::Type::Integer)
'includedPackageNameMasks'	!ruby/object:Api::Azure::Type::ResourceReference	!ruby/object:Api::Type::Array (item_type: Api::Azure::Type::ResourceReference)
'scope'	!ruby/object:Api::Azure::Type::ResourceReference	!ruby/object:Api::Type::Array  (item_type: Api::Azure::Type::ResourceReference)
'azureQueries'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array (item_type: !ruby/object:Api::Type::NestedObject)
'nonAzureQueries'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::Type::NestedObject)
'advancedSchedule'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::Type::NestedObject)
'monthlyOccurrences'	!ruby/object:Api::Type::NestedObject	!ruby/object:Api::Type::Array  (item_type: !ruby/object:Api::Type::NestedObject)

5. Integer Type For Go SDK

Property	Original Azure SDK Type	Should Be
'/properties/scheduleInfo/interval'	Azure::SDKTypeDefinition::IntegerObject	Azure::SDKTypeDefinition::Integer32Object
'/properties/scheduleInfo/advancedSchedule /monthlyOccurrences/occurrence'	Azure::SDKTypeDefinition::IntegerObject	Azure::SDKTypeDefinition::Integer32Object

6. Input and Output Should Not Be Both Set to True

Involved Property	'id'

7. Repeated Property

Involved Property	'name'

8. Unnecessary Property

Involved Property	'provisioningState'   ,  'type'

9. Request Header Should Be Declared Behind the Properties

Involved Property	'clientRequestId'

10. Enum Type Marked 'required: true' Should Not Have 'default_value'

Involved Property	'operatingSystem'

 

Upgrade autorest-extension-base to the lastest version

`list` operations design

I noticed we have a list operation in batchapplication. I would like to name it as list_by_parent based on the following assumptions:

1. A top level (or maybe sub-resource as well) resource could be listed by resource group, we have list_by_resource_group support.
2. A sub-resource could be listed by its parent resource (e.g. batchaccount is batchapplication's parent), I would like to introduce list_by_parent to api.yaml for each sub-resource.

Correct me if you know there are other cases for this scenario.

If ALL sub-resources are not able to be listed by resource group, actually we might also combine those two into one list_by_parent, because we can treat resource_group as the parent of all top-level resources.

Create/Enable DevOps CI/CD pipeline

This task should include:

• create new pipeline in DevOps
• create npm package
• run tests
• publishing a package as @autorest/cli

Resource names for MM Input

In Terraform, we will also set the name back to user configuration files after "Get" operation is finished. The second /name matches one of the response item defined under 'read' section, which instructs MM to copy that value back to user state. Typically, all resources names is made of two items: ['{WhatEverPropertyName}', '/name']

    Suggested change
    
      
    
  


  
    
      
              azure_sdk_references: ['accountName']
    
      
        
                azure_sdk_references: ['accountName', '/name']
      
  


          
            
            

              

              
              
              
                
                  Commit suggestion
                  
                
              
              
                Add suggestion to batch

https://github.com/VSChina/magic-modules/pull/5/files#r271983294

Enumeration and input set to true

To list all available enumeration candidates, MM uses values, and each value is prepended by a : (colon).
We can also use default_value in MM to specify the default value, it make this property optional instead of required.
It seems that this property can not be updated in portal, so maybe we should also add input: true to it.

    Suggested change
    
      
    
  


  
    
      
                  required: true
    
      
        
                    input: true
      
      
        
                    values:
      
      
        
                      - :BatchService
      
      
        
                      - :UserSubscription
      
      
        
                    default_value: :BatchService
      
  


          
            
            

              

              
              
              
                
                  Commit suggestion
                  
                
              
              
                Add suggestion to batch

https://github.com/VSChina/magic-modules/pull/5/files#r271988231

[MM]property value is not correct in api.yaml

1. Input and Output Should Not Be Both Set to True

Involved Property	'id'

2. Repeated Property

Involved Property	'name'

3. Unnecessary Property

Involved Property	'provisioningState'   ,  'type'

4. Request Header Should Be Declared Behind the Properties

Involved Property	'clientRequestId'

5. Enum Type Marked 'required: true' Should Not Have 'default_value'

Involved Property	'operatingSystem'

6. Enum Type Marked 'output: true' should Not Have 'default_value'

Involved Property	'provisioningState'

[MM]go_field_name is not correct in api.yaml

Use the SoftwareUpdateConfiguration resource api.yaml file here as an example.

Property	Original Name	Should Be
'/properties'	Properties	SoftwareUpdateConfigurationProperties

[MM]The api.yml shouldn't be generated to package folder for SAP Monitor

The api.yml shouldn't be generated to package folder like "package-2017-11" for SAP Monitor. It should be generated to resource folder like "hanaonazuresapmonitor".

Command:
C:\terraform\projects\autorest.cli>autorest --cli --use=C:/terraform/projects/autorest.cli --mm --output-folder=C:/terraform/generated/abc C:\terraform\projects\azure-rest-api-specs\specification\hanaonazure\resource-manager\readme.md

Added below command to readme.md for hanaibazure: