smithy-lang/smithy-kotlin

Smithy code generator for Kotlin (in development)

Smithy Kotlin

Smithy code generators for Kotlin.

Getting Started

• Get an introduction to Smithy
• Follow Smithy's quickstart guide
• Apply the Smithy Gradle plugin to your project and start generating Kotlin code!

Development

Module Structure

• codegen - module(s) for generating Kotlin code from Smithy models

  • protocol-tests - module for generating Smithy protocol tests
  • smithy-aws-kotlin-codegen - module containing AWS-specific codegen, will eventually be refactored to aws-sdk-kotlin
  • smithy-kotlin-codegen - primary codegen module
  • smithy-kotlin-codegen-testutils - utilities for testing generated code (shared with aws-sdk-kotlin)

• runtime - library code used by generated clients and servers to perform SDK functions

  • auth - authentication and signing related modules
  • crt-util - utilities for using the AWS Common Runtime (CRT)
  • observability - contains various telemetry provider implementations
  • protocol - protocol support (including HTTP, application level protocols, test support, etc)
  • runtime-core - contains core functionality used by all clients, servers, or other runtime modules
  • serde - serialization/deserialization modules
  • smithy-client - runtime support for generated service clients
  • smithy-test - runtime support for generated tests (e.g. smithy protocol tests)
  • testing - internal testing utilities for the runtime

• tests - test and benchmark module(s)

  • benchmarks - benchmarks for runtime
  • codegen - codegen integration tests for various features (e.g. testing waiters, paginators, etc)
  • compile - compile tests for generated code
  • integration - tests for different versions of our dependencies to ensure compatibility

Feedback

You can provide feedback or report a bug by submitting an issue. This is the preferred mechanism to give feedback so that other users can engage in the conversation, +1 issues, etc.

Contributing

If you are interested in contributing to Smithy Kotlin, please take a look at CONTRIBUTING.

License

This project is licensed under the Apache-2.0 License.

Security

See CONTRIBUTING for more information.

smithy-kotlin's Issues

Fix handling of unboxed primitives in serialization

Unboxed primitives (which have a default value) are currently not handled/considered by the SerializeStructGenerator.

It should be as simple as converting the shape to a symbol and checking our extension method Symbol.isBoxed() (since we stash whether a type is boxed or not into the symbol property bag when we create the symbol)

internal id: 174905362

Fix comment escaping

In some places comment text causes a comment to become untemrinated, resulting in the rest of the file being parsed as a comment.

Example: service: Apigatewayv2 class CreateApiRequest

/**
     * <p>A CORS configuration. Supported only for HTTP APIs. See <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-cors.html">Configuring CORS</a> for more information.</p>
     */
    val corsConfiguration: Cors? = builder.corsConfiguration
    /**
     * <p>This property is part of quick create. It specifies the credentials required for the integration, if any. For a Lambda integration, three options are available. To specify an IAM Role for API Gateway to assume, use the role's Amazon Resource Name (ARN). To require that the caller's identity be passed through from the request, specify arn:aws:iam::*:user/*. To use resource-based permissions on supported AWS services, specify null. Currently, this property is not used for HTTP integrations. Supported only for HTTP APIs.</p>
     */
    val credentialsArn: String? = builder.credentialsArn
    /**
     * <p>The description of the API.</p>
     */
    val description: String? = builder.description

internal id: 176971894

Change expression start character from $ to # for KotlinWriter

Refactor KotlinWriter (CodeWriter) to use a different "expression start" symbol. By default smithy code writer uses $ as the start of an expression but that requires us escaping it in every string. Change it to #.

Will make it so we go from writing:

writer.write("val foo: \$T", symbol)

to:

writer.write("val foo: #T", symbol)

internal id: 177020506

Fix when content-type gets set

At one point we added application/json content type to all outgoing requests because it was dictated by protocol tests. It seems this was actually erroneous in the protocol test(s):

smithy-lang/smithy#692

We should look at backing this out and only setting content-type in the serializer headers block.

Also fix aws json middleware here

Ask Aaron if unsure

internal id: 176662766

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Interpolated string encoding error in url path builder

The iotjobsdataplane service has a input type[1] which specifies a URI containing the $ symbol. This is generating an invalid string expression in the serializer.

        builder.url {
            path = "/things/${input.thingName}/jobs/$next"
        }

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/iotjobsdataplane.2017-09-29.json#L600

internal id: 176972455

AWS Rest XML Error

Generate exceptions that inherit from AwsServiceException instead of ServiceException

Implement serde pipeline middleware to connect modeled errors to HTTP responses.

internal id: 174040294

Setup GH repo for beta package publishing

Setup a new (private) repository for publishing current packages to using GH packages.

Using a new repository (rather than e.g. aws-sdk-kotlin) serves a few purposes:

1. Published packages cannot be deleted once released, this prevents us from cluttering our existing repositories and people consuming them accidentally
2. We can have a central place for all three of our repositories to publish to (it wouldn't make sense to publish smithy-kotlin to aws-sdk-kotlin for example)
3. We can carefully control permissions to just this single repository instead of potentially multiple (e.g. smithy-kotlin, aws-sdk-kotlin, and aws-crt-kotlin).
4. We can delete it when finished if we want to (we may want to keep it around for e.g. future betas though).

https://docs.github.com/en/packages/guides/configuring-gradle-for-use-with-github-packages

internal id: 177077195

import name collisions

firehose defines a model type Deserializer. This conflicts with all references to the Kotlin SDK type of the same name.

import aws.sdk.kotlin.firehose.model.Deserializer

...
    suspend fun deserialize(deserializer: Deserializer): InputFormatConfiguration {
...

https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/firehose.2015-08-04.json#L2127

internal id: 177018423

Interpolated string encoding error in toString of model type

The aws service customerprofiles generates a type ListProfileObjectsItem which includes a member called Object[1]. This breaks the codegen toString() implementation for the type.

    override fun toString() = buildString {
        append("ListProfileObjectsItem(")
        append("object=$object,")
        append("objectTypeName=$objectTypeName,")
        append("profileObjectUniqueKey=$profileObjectUniqueKey)")
    }

In other places in the class the name is correctly escaped, example:

    /**
     * <p>A JSON representation of a ProfileObject that belongs to a profile.</p>
     */
    val `object`: String? = builder.`object`

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/customerprofiles.2020-08-15.json#L2164

internal id: 176972683

Implement stateless serde

Refactor codegen to produce stateless serialize/deserialize implementations.

operation Foo {
    input: FooInput,
    output: FooOutput
}
structure FooInput {
    nested: Nested,
    ...
}
structure Nested { ... }

structure FooOutput { ... }

Operations get a filename with the operation name + Serde. Serializers and deserializers both get placed there.

// operation serde
// file: ...{service}.transform.FooOperationSerde.kt

suspend fun serializeFooOperation(...)
suspend fun deserializeFooOperation(...)

Document only payloads get a filename like {type}Serde.kt and serializers/deserializers both get placed there

// document types
// file: ...{service}.transform.NestedSerde.kt
suspend fun serializeNestedDocument(...)

suspend fun deserializeNestedDocument(...)

We may still need a helper type since suspend isn't compatible with SAM interfaces. (Operation serializers/deserializers need to implement HttpSerialize<T> and HttpDeserialize<T> respectively.

e.g.

val op = SdkHttpOperation.build<FooInput, FooOutput> {
    serializer = HttpSerializer(::serializeFooOperation)
    deserializer = HttpDeserializer(::deserializeFooOperation)
}

where HttpSerializer/Deserializer are new runtime types that implement HttpSerialize/Deserialize by delegating to the lambda/function passed.

internal id: 177151831

Implement generation of synthetic members to handle model evolution.

Based on design task: /story/show/174760723

Acceptance criteria:

1. Disabled integration tests in the model evolution test class pass.

internal id: 176446022

Update/Test AWS models once bug fixes finished

Acceptance criteria: once the bugs associated w/ label bug-codegen are complete, update the AWS models to latest, re-run tests and ensure no regressions. If regressions found, create issues for bugs.

internal id: 177126367

HttpStream.close() segfault

There is a unit test failing sporadically for crt-kotlin JVM bindings. Either manifests as an exception in the JNI layer detecting that the stream is null OR results in segfault.

internal id: 175850550

Passing undeserializable content as string to Union.SdkUnknown

This task is to determine if it adds sufficient customer value to string-ify the unknown field in a given Union in the case the field is unrecognized. Currently we discard the value and return a singleton SdkUnkown to specify that the response was unrecogized. In the case of enums, the string value of the enum is returned as property of a similar SdkUnknown type. Because Union members may prepresent any structure, this approach will not work. Given these contraints, it's unclear if providing the stringified form to customers has any value.

If it's determined that this heuristic is desired, another ticket can be created to implement the feature.

Related to /story/show/176773703

internal id: 177025237

kgwh test

test body

deserialization of unions doesn't consume the end object token

We have very basic logic:

     try {

        val scanRequest = ScanInput {
            tableName = tableNameVal
        }

        val response = ddb.scan(scanRequest)

scan throws:

Exception in thread "main" software.aws.clientrt.serde.DeserializerStateException: expected class software.aws.clientrt.serde.json.JsonToken$BeginObject (Kotlin reflection is not available); found class software.aws.clientrt.serde.json.JsonToken$Name (Kotlin reflection is not available)
at software.aws.clientrt.serde.json.JsonDeserializer.deserializeMap(JsonDeserializer.kt:224)
at software.aws.clientrt.serde.DeserializerKt.deserializeMap(Deserializer.kt:195)
at aws.sdk.kotlin.dynamodb.transform.ScanDeserializer$deserialize$2$1.invokeSuspend(ScanDeserializer.kt:48)
at aws.sdk.kotlin.dynamodb.transform.ScanDeserializer$deserialize$2$1.invoke(ScanDeserializer.kt)
at software.aws.clientrt.serde.DeserializerKt.deserializeList(Deserializer.kt:191)
at aws.sdk.kotlin.dynamodb.transform.ScanDeserializer$deserialize$2.invokeSuspend(ScanDeserializer.kt:45)
at aws.sdk.kotlin.dynamodb.transform.ScanDeserializer$deserialize$2.invoke(ScanDeserializer.kt)
at software.aws.clientrt.serde.DeserializerKt.deserializeStruct(Deserializer.kt:186)
at aws.sdk.kotlin.dynamodb.transform.ScanDeserializer.deserialize(ScanDeserializer.kt:39)
at software.aws.clientrt.http.feature.HttpSerde$install$2.invokeSuspend(HttpSerde.kt:75)
at software.aws.clientrt.http.feature.HttpSerde$install$2.invoke(HttpSerde.kt)
at io.ktor.util.pipeline.SuspendF

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Sparse list codegen incorrectly breaks loop which causes null to not be added

    This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

    *internal id*: 176812586                

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Ref Arch - Design Interceptors

• /IsPjAGykeDeG/Kotlin-Extensibility-Considerations
• /f8HpA5G89lQs/Survey-of-AWS-SDK-Client-Configuration-and-MiddlewareCustomization

Finalize design for middleware and how customers can/will implement their own middleware.

Design how operation overrides will be implemented (if at all).

internal id: 176720015

Document Shape - Kotlin

Initial design of the document shape is already written in smithy-kotlin.

See also Smithy's Simple Shapes section about document.

Fix encode path for signing

/archives/C017Z3N2CTD/p1614199323025400
/archives/C017Z3N2CTD/p1614199323025600

tl;dr : is a valid pchar according to RFC3986 but apparently needs to be percent encoded for signing purposes...

Acceptance criteria:

• Run invoke operation on a lambda using the arn of the lambda instead of a simple name:

suspend fun invokeFunction(client: LambdaClient, name: String) {
    val req = InvocationRequest {
        functionName = name
        payload = """
            {"foo": "bar"}
        """.trimIndent().encodeToByteArray()
    }
    val resp = client.invoke(req)
    println(resp)
}

fun main() = runBlocking {
    val client = LambdaClient { region = "us-east-2" }

    System.setProperty("aws.crt.log.level", "Debug")
    try {

        invokeFunction(client, "arn:aws:lambda:us-east-2:459419245142:function:test-kotlin-sdk")
    }catch (ex: AwsServiceException) {
        val resp = ex.protocolResponse as HttpResponse
        println(resp)
        println(resp.body.readAll()?.decodeToString())
        println(ex)
    }

    client.close()
    Unit
}

internal id: 177102170

Member name collision

In AWS service serverlessapplicationrepository there is an exception type BadRequestException which codegens a member errorCode which shadows the built-in AwsServiceException member of the same name.

class BadRequestException private constructor(builder: BuilderImpl) : AwsServiceException() {
    /**
     * <p>400</p>
     */
    val errorCode: String? = builder.errorCode
...

internal id: 176972501

Implement XML diff utils

internal id: 176716288

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Alternative Middleware Design

internal id: 177145291

Generated serde is wrong for some union edge cases

• generated serializer for list of union values is wrong. It should be col0 being serialized not input.value (and type name should be el0 here as well)

                is AttributeValue.L -> {
                    listField(L_DESCRIPTOR) {
                        for (col0 in input.value) {
                            serializeSdkSerializable(AttributeValueSerializer(input.value))
                        }
                    }
                }

• deserialization of lists/maps for union values is wrong.
  The type generated for the deserialize method of a union is T? not T so the inference below for a regular (non-sparse) map has type AttributeValue? instead of the expected AttributeValue.

                M_DESCRIPTOR.index -> value =
                    deserializer.deserializeMap(M_DESCRIPTOR) {
                        val map0 = mutableMapOf<String, AttributeValue>()
                        while (hasNextEntry()) {
                            val k0 = key()
                            val v0 = if (nextHasValue()) { AttributeValueDeserializer().deserialize(deserializer) } else { deserializeNull(); continue } // wrong, inferred type is AttributeValue? but v0 is expected to be AttributeValue
                            map0[k0] = v0
                        }
                        AttributeValue.M(map0)
                    }

This is set here and only applies to unions. The reason I believe is that forward compat dictates that we may receive a union variant we don't know about and must return null as a possibility but I'm not positive.

Acceptance criteria:

• Generate dynamodb and see that it compiles.

internal id: 176751942

Add an unknown variant to unions

We should generate an SdkUnknown or similar variant for union shapes the same way we do for enums.

This would also allow us to generate the deserialize implementation for unions as deserialize(): T instead of deserialize(): T? which would bring struct and union deserialization closer in terms of interfaces.

See: #176751942 which is directly related to this

internal id: 176773703

Handle Operational inputs with http bindings removed - Kotlin

We need to handle the edge case in which operational inputs can be reused not as an operational input which means http bindings don't apply and in that case if it was nested inside a body, all members would be encoded.

internal id: 174760745

Rename model types with same name as builtin types

Split off from #176972905

After talking with the Smithy team and internally we agreed we should rename top level model shapes that have the same name as a builtin Kotlin type (e.g. Unit, String, etc).

So if a model has a type like this: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/connect.2017-08-08.json#L9717

we should rename it by prefixing it with the service name (formatted sdkid?).

This DOES NOT apply for enum and union variants though because they are namespaced within the type they belong to (both are generate as a sealed class). Only top level shapes need renamed.

internal id: 177022993

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Migrate kotlin issues to github

Moving active work modeled in pivotal to github for better collaboration.

Acceptance critera: all active or incomplete tasks and important details from pivotal are available in smithy-kotlin repo issues.

internal id: 177192761

Http and stand-alone deserializer generated w/ same classname

the kinesis service specifies an exception type KMSAccessDeniedException[1]. Codegen produces two classes in the same file with the same name:

class KMSAccessDeniedExceptionDeserializer : HttpDeserialize {
...
}

class KMSAccessDeniedExceptionDeserializer {
...
}

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/kinesis.2013-12-02.json#L1141

internal id: 176989575

field and object field descriptors not generated

In the service transfer, there is a type aws.sdk.kotlin.transfer.transform.ThrottlingExceptionDeserializer [1] which after codegen does not have it's single field or object descriptor generated in a companion object:

class ThrottlingExceptionDeserializer : HttpDeserialize {

    override suspend fun deserialize(response: HttpResponse, provider: DeserializationProvider): ThrottlingException {
        val builder = ThrottlingException.dslBuilder()

        val payload = response.body.readAll()
        if (payload != null) {
            val deserializer = provider(payload)
            deserializer.deserializeStruct(OBJ_DESCRIPTOR) {
                loop@while (true) {
                    when (findNextFieldIndex()) {
                        RETRYAFTERSECONDS_DESCRIPTOR.index -> builder.retryAfterSeconds = deserializeString()
                        null -> break@loop
                        else -> skipValue()
                    }
                }
            }
        }
        return builder.build()
    }
}

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/transfer.2018-11-05.json#L2155

internal id: 176989755

Header deserializer nullable type mismatch

In the AWS service polly there is a deserializer aws.sdk.kotlin.polly.transform.SynthesizeSpeechDeserializer which populates a value from a header[1], however the deserializer expression return type is Int? but Int is required.

class SynthesizeSpeechDeserializer : HttpDeserialize {

    override suspend fun deserialize(response: HttpResponse, provider: DeserializationProvider): SynthesizeSpeechOutput {
        val builder = SynthesizeSpeechOutput.dslBuilder()

        builder.contentType = response.headers["Content-Type"]
        builder.requestCharacters = response.headers["x-amzn-RequestCharacters"]?.toInt()

        builder.audioStream = response.body.toByteStream()

        return builder.build()
    }
}

It seems that a default is required...but just a guess.

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/polly.2016-06-10.json#L1531

internal id: 176972546

Generated type collision with union members

The groundstation service defines a union that names it's members with the same name as the types the map to[1]. This causes the codegend union to have a constructor parameter of itself:

sealed class ConfigTypeData {
    /**
     * <p>Information about how AWS Ground Station should configure an antenna for downlink during a contact.</p>
     */
    data class AntennaDownlinkConfig(val value: AntennaDownlinkConfig) : ConfigTypeData()
    /**
     * <p>Information about how AWS Ground Station should configure an antenna for downlink demod decode during a contact.</p>
     */
    data class AntennaDownlinkDemodDecodeConfig(val value: AntennaDownlinkDemodDecodeConfig) : ConfigTypeData()
    /**
     * <p>Information about how AWS Ground Station should configure an antenna for uplink during a contact.</p>
     */
    data class AntennaUplinkConfig(val value: AntennaUplinkConfig) : ConfigTypeData()
    /**
     * <p>Information about the dataflow endpoint <code>Config</code>.</p>
     */
    data class DataflowEndpointConfig(val value: DataflowEndpointConfig) : ConfigTypeData()
    /**
     * <p>Object that determines whether tracking should be used during a contact executed with this <code>Config</code> in the mission profile. </p>
     */
    data class TrackingConfig(val value: TrackingConfig) : ConfigTypeData()
    /**
     * <p>Information about an uplink echo <code>Config</code>.</p>
     *          <p>Parameters from the <code>AntennaUplinkConfig</code>, corresponding to the specified <code>AntennaUplinkConfigArn</code>, are used when this <code>UplinkEchoConfig</code> is used in a contact.</p>
     */
    data class UplinkEchoConfig(val value: UplinkEchoConfig) : ConfigTypeData()
}

This actually compiles w/ warnings but the de/serializers fail to compile because of the type mismatch. One solution to this would be to add the namespace to the constructor parameters to disambiguate between the sealed class members and the structrues they are meant to contain. eg:

sealed class ConfigTypeData {
    /**
     * <p>Information about how AWS Ground Station should configure an antenna for downlink during a contact.</p>
     */
    data class AntennaDownlinkConfig(val value: aws.sdk.kotlin.groundstation.model.AntennaDownlinkConfig) : ConfigTypeData()
...

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/groundstation.2019-05-23.json#L292

internal id: 176973066

Type name collisions between Kotlin and modeled types

In the aws service connect, there are members named Unit[1]. This causes a compiler error in the builder code as the generated type is being assumed instead of the Kotlin type for Unit.

Example:

fun copy(block: DslBuilder.() -> Unit = {}): AssociateInstanceStorageConfigRequest = BuilderImpl(this).apply(block).build()

Updating the source files to use kotlin.Unit resolves the compiler errors.

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/connect.2017-08-08.json#L4135

internal id: 176972905

Add an unknown variant to unions

We should generate an SdkUnknown or similar variant for union shapes the same way we do for enums.

This would also allow us to generate the deserialize implementation for unions as deserialize(): T instead of deserialize(): T? which would bring struct and union deserialization closer in terms of interfaces.

See: #176751942 which is directly related to this

internal id: 176773703

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Union type member is same name as Union

Similar to /story/show/176973066 but somewhat different, aws service route53domains generates the type aws.sdk.kotlin.route53domains.model.Transferable which is a union, which contains a member of the same name of the union type:

sealed class Transferable {
   ...
    object Transferable : Transferable() {
        override val value: kotlin.String = "TRANSFERABLE"
        override fun toString(): kotlin.String = value
    }

internal id: 176987920

field descriptor erroneously generated for struct member bound to header

in service migrationhubconfig there is a type aws.sdk.kotlin.migrationhubconfig.transform.ThrottlingExceptionDeserializer [1] which has a field generated in the deserize that is bound to a header. The descriptor itself is not generated (correct) but the deserializer code incorrectly generates a when clause for this field anyway:

class ThrottlingExceptionDeserializer : HttpDeserialize {

    companion object {
        private val MESSAGE_DESCRIPTOR = SdkFieldDescriptor("Message", SerialKind.String)
        private val OBJ_DESCRIPTOR = SdkObjectDescriptor.build() {
            field(MESSAGE_DESCRIPTOR)
        }
    }

    override suspend fun deserialize(response: HttpResponse, provider: DeserializationProvider): ThrottlingException {
        val builder = ThrottlingException.dslBuilder()

        val payload = response.body.readAll()
        if (payload != null) {
            val deserializer = provider(payload)
            deserializer.deserializeStruct(OBJ_DESCRIPTOR) {
                loop@while (true) {
                    when (findNextFieldIndex()) {
                        MESSAGE_DESCRIPTOR.index -> builder.message = deserializeString()
                        RETRYAFTERSECONDS_DESCRIPTOR.index -> builder.retryAfterSeconds = deserializeInt()
                        null -> break@loop
                        else -> skipValue()
                    }
                }
            }
        }
        return builder.build()
    }
}

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/migrationhubconfig.2019-06-30.json#L457

internal id: 176989233

Invalid characters in package and import declarations codegen

In the AWS service rds-data, packange and import declarations are generated w/ the rds-data name including hyphen. This fails to compile.

aws.sdk.kotlin.rds.RdsDataClient Example:

// Code generated by smithy-kotlin-codegen. DO NOT EDIT!

package aws.sdk.kotlin.rds-data

import aws.sdk.kotlin.rds-data.model.*

internal id: 176972723

Automatic pagination

Community Note

1. Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
2. Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
3. If you are interested in working on this issue, please leave a comment

Describe the feature

Provide API's that support automatic pagination of operations marked with the paginated trait

Proposed Solution

See the proposed design doc

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

AWS service keyword collision in modeled type toString/hashCode/equals

In the aws service codeartifact the CopyPackageVersionRequest defines a member package [1]. When codegened we do not escape the literal and so the compilation fails.

    override fun hashCode(): Int {
        var result = package?.hashCode() ?: 0
        result = 31 * result + (allowOverwrite?.hashCode() ?: 0)
...

    override fun toString() = buildString {
        append("CopyPackageVersionsRequest(")
        append("package=$package,")

    override fun equals(other: Any?): Boolean {
        if (this === other) return true
        if (javaClass != other?.javaClass) return false

        other as CopyPackageVersionsRequest

        if (package != other.package) return false

1: https://github.com/awslabs/aws-sdk-kotlin/blob/88a469e20503e7763439789a31621a8d17c3b72c/codegen/sdk/aws-models/codeartifact.2018-09-22.json#L469

internal id: 176983238

Sparse list codegen incorrectly breaks loop which causes null to not be added

This is the codegen produced from a sparse list currently:

deserializer.deserializeList(SPARSESTRUCTLIST_DESCRIPTOR) {
                                val col0 = mutableListOf<Greeting?>()
                                while (hasNextElement()) {
                                    val el0 = if (nextHasValue()) { GreetingDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                                    col0.add(el0)
                                }
                                col0
                            }

The continue statement is incorrect, it is causing the nulls to not be added to the list.

internal id: 176812586

Codegen incorrectly generating nested members at top level

In rds-data, the type aws.sdk.kotlin.rdsdata.transform.FieldDeserializer does not generate the full set of field descriptors:

class FieldDeserializer {

    companion object {
        private val ARRAYVALUE_DESCRIPTOR = SdkFieldDescriptor("arrayValue", SerialKind.Struct)
        private val BLOBVALUE_DESCRIPTOR = SdkFieldDescriptor("blobValue", SerialKind.Blob)
        private val BOOLEANVALUE_DESCRIPTOR = SdkFieldDescriptor("booleanValue", SerialKind.Boolean)
        private val DOUBLEVALUE_DESCRIPTOR = SdkFieldDescriptor("doubleValue", SerialKind.Double)
        private val ISNULL_DESCRIPTOR = SdkFieldDescriptor("isNull", SerialKind.Boolean)
        private val LONGVALUE_DESCRIPTOR = SdkFieldDescriptor("longValue", SerialKind.Long)
        private val STRINGVALUE_DESCRIPTOR = SdkFieldDescriptor("stringValue", SerialKind.String)
        private val OBJ_DESCRIPTOR = SdkObjectDescriptor.build {
            field(ARRAYVALUE_DESCRIPTOR)
            field(BLOBVALUE_DESCRIPTOR)
            field(BOOLEANVALUE_DESCRIPTOR)
            field(DOUBLEVALUE_DESCRIPTOR)
            field(ISNULL_DESCRIPTOR)
            field(LONGVALUE_DESCRIPTOR)
            field(STRINGVALUE_DESCRIPTOR)
        }
    }

    suspend fun deserialize(deserializer: Deserializer): Field {
        var value: Field? = null
        deserializer.deserializeStruct(OBJ_DESCRIPTOR) {
            when(findNextFieldIndex()) {
                ARRAYVALUES_DESCRIPTOR.index -> value =
                    deserializer.deserializeList(ARRAYVALUES_DESCRIPTOR) {
                        val col0 = mutableListOf<ArrayValue>()
                        while (hasNextElement()) {
                            val el0 = if (nextHasValue()) { ArrayValueDeserializer().deserialize(deserializer) } else { deserializeNull(); continue }
                            col0.add(el0)
                        }
                        ArrayValue.ArrayValues(col0)
                    }
                BOOLEANVALUES_DESCRIPTOR.index -> value =
                    deserializer.deserializeList(BOOLEANVALUES_DESCRIPTOR) {
                        val col0 = mutableListOf<Boolean>()
                        while (hasNextElement()) {
                            val el0 = if (nextHasValue()) { deserializeBoolean() } else { deserializeNull(); continue }
                            col0.add(el0)
                        }
                        ArrayValue.BooleanValues(col0)
                    }
                DOUBLEVALUES_DESCRIPTOR.index -> value =
                    deserializer.deserializeList(DOUBLEVALUES_DESCRIPTOR) {
                        val col0 = mutableListOf<Double>()
                        while (hasNextElement()) {
                            val el0 = if (nextHasValue()) { deserializeDouble() } else { deserializeNull(); continue }
                            col0.add(el0)
                        }
                        ArrayValue.DoubleValues(col0)
                    }
                LONGVALUES_DESCRIPTOR.index -> value =
                    deserializer.deserializeList(LONGVALUES_DESCRIPTOR) {
                        val col0 = mutableListOf<Long>()
                        while (hasNextElement()) {
                            val el0 = if (nextHasValue()) { deserializeLong() } else { deserializeNull(); continue }
                            col0.add(el0)
                        }
                        ArrayValue.LongValues(col0)
                    }
                STRINGVALUES_DESCRIPTOR.index -> value =
                    deserializer.deserializeList(STRINGVALUES_DESCRIPTOR) {
                        val col0 = mutableListOf<String>()
                        while (hasNextElement()) {
                            val el0 = if (nextHasValue()) { deserializeString() } else { deserializeNull(); continue }
                            col0.add(el0)
                        }
                        ArrayValue.StringValues(col0)
                    }
                BLOBVALUE_DESCRIPTOR.index -> value = Field.BlobValue(deserializeString().decodeBase64Bytes())
                BOOLEANVALUE_DESCRIPTOR.index -> value = Field.BooleanValue(deserializeBoolean())
                DOUBLEVALUE_DESCRIPTOR.index -> value = Field.DoubleValue(deserializeDouble())
                ISNULL_DESCRIPTOR.index -> value = Field.IsNull(deserializeBoolean())
                LONGVALUE_DESCRIPTOR.index -> value = Field.LongValue(deserializeLong())
                STRINGVALUE_DESCRIPTOR.index -> value = Field.StringValue(deserializeString())
                else -> value = Field.SdkUnknown.also { skipValue() }
            }
        }
        return value ?: throw DeserializationException("Deserialized value unexpectedly null: Field")
    }
}

internal id: 177126780