Type annotations builder for boto3 compatible with VSCode, PyCharm, Emacs, Sublime Text, pyright and mypy.
Home Page: https://youtype.github.io/mypy_boto3_builder/
License: MIT License
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
Describe the bug
I can't figure out how to annotate some code using SQS
To Reproduce
Steps to reproduce the behavior:
from typing import List
import mypy_boto3_sqs as SQS
import boto3
def main() -> None:
# using the client everything is OK
client: SQS.Client = boto3.client("s3")
response = client.receive_message(QueueUrl="queue_url")
messages2: List[SQS.type_defs.MessageTypeDef] = response["Messages"]
# SQS.ServiceResource.Queue or SQS.ServiceResource.Message are not valid
# types according to mypy
resource: SQS.ServiceResource = boto3.resource("sqs")
queue = resource.Queue("queue_url")
messages: List[SQS.ServiceResource.Message] = queue.receive_messages()
if __name__ == "__main__":
main()mypy output:
Function "mypy_boto3_sqs.service_resource.SQSServiceResource.Message" is not valid as a type
I need to annotate these results since they will be returned by some functions.
Describe the bug
Return type from the filter function is not annotated but in 1.12.2.0 it was annotated as mypy_boto3_s3.service_resource.BucketObjectsCollection.
To Reproduce
Steps to reproduce the behavior:
import boto3
resource = boto3.session.Session(region_name='eu-central-1').resource(
service_name='s3',
aws_access_key_id='xxx',
aws_secret_access_key='xxx'
)
bucket = resource.Bucket('bucket-name')
reveal_type(bucket.objects.filter(Prefix='prefix'))note: Revealed type is '<nothing>'
Expected mypy-output
note: Revealed type is 'mypy_boto3_s3.service_resource.BucketObjectsCollection'
Additional context
OS: Ubuntu 18.04
boto3 installation method: pip
boto3 version: 1.12.16
Describe the bug
This is a follow up on #19 It fixed the output shape, but now the input shape doesn't accept bytes, only IO[bytes].
I had a look at the botocore json definitions and there's no distinction there between PutObjectRequest and GetObjectOutput unfortunately. I believe a fix would require to check against shape.name and pass it down to _parse_shape.
This way, we could special case some operations.
That feels hackish and brittle though, so I wonder if you have any other idea?
To Reproduce
Steps to reproduce the behavior:
import boto3
from mypy_boto3 import s3
client: s3.S3Client = boto3.client("s3")
client.put_object(Bucket="my-bucket", Body=b"Hellow", Key="my-key")error: Argument "Body" to "put_object" of "S3Client" has incompatible type "bytes"; expected "IO[bytes]
Expected mypy-output
All good. Body should have the type Union[bytes, IO[bytes]]
Additional context
boto3-stubs version 1.13.24.2
Describe the bug
When installing the stubs (in my case specifically just the EC2 ones without any other S3 services), Pycharm grinds to a halt and is unusable. I've left it running for an hour to ensure there's no indexing/recaching occurring and the issues persists.
This is most likely a Pycharm issue but I thought I would post here anyway in case someone else has run into it.
Thanks so much for putting this package together!
To Reproduce
Steps to reproduce the behavior:
pip install 'boto3-stubs[ec2]')Additional context
OS X, boto3-stubs
Describe the bug
It seems that the InvocationResponseTypeDef ['Payload'] has the type bytes which triggers the type checker.
incompatible type "bytes"; expected "SupportsRead[Union[str, bytes]]" return json.load(invoke_response['Payload'])
To Reproduce
Steps to reproduce the behavior:
import boto3
lambda_payload = {"test" : 123}
invoke_response: InvocationResponseTypeDef = boto3.client('lambda')invoke(FunctionName="some-lambda",
Payload=lambda_payload.encode('utf-8'))`
response_payload = json.load(invoke_response['Payload])
incompatible type "bytes"; expected "SupportsRead[Union[str, bytes]]"
return json.load(invoke_response['Payload'])
Expected mypy-output
...
Additional context
`pip3.8 install boto3-stubs[essential]==1.16.2.0
The type definition defines the Payload as bytes, the boto3 documentation describes it as StreamingBody()
InvocationResponseTypeDef = TypedDict(
"InvocationResponseTypeDef",
{
"StatusCode": int,
"FunctionError": str,
"LogResult": str,
"Payload": bytes,
"ExecutedVersion": str,
},
total=False,
)
From what I understood reading other PRs it seems to me that the Payload should be of type IO[bytes]
InvocationResponseTypeDef = TypedDict(
"InvocationResponseTypeDef",
{
"StatusCode": int,
"FunctionError": str,
"LogResult": str,
"Payload": IO[bytes],
"ExecutedVersion": str,
},
total=False,
)
Types are incorrect for optional types. This causes confusion and typing errors.
For example, in mypy_boto3_batch.client:
def submit_job(
self,
jobName: str,
jobQueue: str,
jobDefinition: str,
arrayProperties: ArrayPropertiesTypeDef = None,
dependsOn: List["JobDependencyTypeDef"] = None,
parameters: Dict[str, str] = None,
containerOverrides: "ContainerOverridesTypeDef" = None,
nodeOverrides: NodeOverridesTypeDef = None,
retryStrategy: "RetryStrategyTypeDef" = None,
timeout: "JobTimeoutTypeDef" = None,
) -> SubmitJobResponseTypeDef:
The nodeOverrides should have type Optional[NodeOverridesTypeDef].
This matters, because passing an empty dictionary ({}) is different than passing in None. The former causes an error if we specify container overrides (non-empty dict) and an empty dict for node overrides, versus no error if the latter is None.
I'd be happy to submit a PR to this repo, or it may be the case boto3 needs to be updated, but a simple search couldn't find submit_job in this repo or that one. For now I can ask the typer to ignore, but that defeats type annotations.
To Reproduce
Steps to reproduce the behavior:
python -m mypy_boto3import boto3
import mypy_boto3_dynamodb as dynamodb
client: dynamodb.DynamoDBClient = boto3.client("dynamodb") 4:35 - error: No overloads for "boto3.client("dynamodb")" match parameters
Argument types: (Literal['dynamodb']) (reportGeneralTypeIssues)
Expected pyright-output
No error
Describe the bug
The return type of mypy_boto3_sqs.service_resource.Queue.receive_messages() is actually List[mypy_boto3_sqs.service_resource.Message] and not ReceiveMessageResultTypeDef
To Reproduce
Steps to reproduce the behavior:
import boto3
from mypy_boto3_sqs import SQSServiceResource
service_resource: SQSServiceResource = boto3.resource("sqs")
queue = service_resource.Queue("blarghle")
message_list = queue.receive_messages()
for message in message_list:
message.delete()sqs_service_resource.py:9: error: "str" has no attribute "delete"
Found 1 error in 1 file (checked 1 source file)
Expected mypy-output
No Errors Found (checked 1 source file)
Additional context
Mac, boto3-stubs installation method, boto3 version 1.12.12
Describe the bug
When mypy is run with implicit_reexport=False (either on the command line or config), it fails to find "client" and "resource" exports from boto3 module.
https://mypy.readthedocs.io/en/stable/config_file.html#miscellaneous-strictness-flags
To Reproduce
Steps to reproduce the behavior:
import boto3
client = boto3.client("lambda")test.py:3: error: Module has no attribute "client"
Expected mypy-output
Success: no issues found in 1 source file
Additional context
mypy 0.770
boto3 1.14.18
boto3-stubs 1.14.17.0
It would be nice to install for all services with something like:
python -m pip install 'boto3-stubs[all]'like there is for essential:
```bash
python -m pip install 'boto3-stubs[essential]'
Describe the bug
In EC2 (and probably other components) tags are described as List[TagTypeDef], the boto3 can also return null (None), when there are no tags, this causes mypy to not detect bugs of not checking to it.
To make things easier for developers, it would be great to provide a single type for that list, so there's less typing and also can be easily changed if the type changes, for example:
TagsTypeDef = Optional[List[TagTypeDef]]version: 1.14.51.0
Describe the bug
Starting today, attempting to upgrade via pip install --user --upgrade boto3-stubs[all] fails due to the mypy-boto3-redshift-data package
To Reproduce
pip install --user --upgrade boto3-stubs[all]
Output:
ERROR: Could not find a version that satisfies the requirement mypy-boto3-redshift-data==1.14.58.0; extra == "all" (from boto3-stubs[all]) (from versions: none)
ERROR: No matching distribution found for mypy-boto3-redshift-data==1.14.58.0; extra == "all" (from boto3-stubs[all])
Expected output
successful installation of upgraded packages
Additional context
OS: Arch Linux / Ubuntu / Opensuse
Install Method: pip
Arch Linux is my daily driver OS where I first noticed, as a test I attempted on a few other machines (both of which were fresh installs, ie pip install --user boto3-stubs[all]) and they failed with same errors
Thanks for looking into this.
THIS WORKS: As client is declared globally, auto-complete for s3_client., and hover on get_bucket_policy shows stub info and boto3 URL for API call.
s3_client = boto3.client("s3")
def get_bucket_policy(bucket):
response = s3_client.get_bucket_policy(Bucket=bucket)
return response
if __name__ == "__main__":
bucket = "example-bucket"
response = get_bucket_policy(bucket)
print(f"{response=}")THIS DOES NOT WORK: As client is not declared globally, and instead passed to the function, auto-complete for s3_client., and hover on get_bucket_policy shows Unknown
def get_bucket_policy(s3_client, bucket):
response = s3_client.get_bucket_policy(Bucket=bucket)
return response
if __name__ == "__main__":
s3_client = boto3.client("s3")
bucket = "example-bucket"
response = get_bucket_policy(s3_client, bucket)
print(f"{response=}")Describe the bug
i am using boto3-stubs(1.16.9.0) on Pycharm as soon as i install this package and try to use some shortcuts on pycharm does not respond ,when i check the task manager i see cpu being utilized 100 by the pycharm .
if i uninstall the boto3-stubs everything works well .i was earlier using the 1.13.19.0 for boto3 and its stubs and everything working quite well until my renovate updated it to latest version.
To Reproduce
Steps to reproduce the behavior:
Additional context
Your OS, boto3-stubs installation method, boto3 version, etc.
windows 10
pip install boto3-stubs[cloudfront,dynamodb,kms,lambda,ses,sqs,s3,ssm,secretsmanager,glue,ses,sns,stepfunctions,redshift,cognito,apigateway,athena,ec2,batch]==1.16.9.0
Hello @vemel
First of all, thank you for the great work on this.
Unfortunately boto and botocore don't have builtin async support, but there is aiobotocore. What do you think about supporting it as well inside mypy_boto3_builder or would you prefer a separate repository?
Is there a reason you didn't do something like the following to avoid specifying the service when calling .client?
from typing import Generic, TypeVar
T = TypeVar("T")
class S3Client():
def put_bucket(self):
pass
class RDSClient():
def describe_db_instances(self):
pass
class AWSService(Generic[T]):
pass
S3: AWSService[S3Client] = "s3"
RDS: AWSService[RDSClient] = "rds"
def client(service: AWSService[T]) -> T:
pass
client(S3).put_bucket()
client(RDS).describe_db_instances()
I'm having a hard time understanding if autocompelete of boto3 clients is supported by this library-- is this possible or am I misunderstanding something? Narrowing the question further, is autocomplete through pyright supported?
My current use case is using Emacs with a pyright server. After setting it up, I'm getting type and return object key/attribute checking, which has been wonderful. But I'm not getting any autocomplete for boto3 clients:
ec2 = boto3.client('ec2')
ec2.
### ^[autocomplete should come up here]Is there anything I can configure to enable this feature? Thanks.
It seems that the return type of obj.get()["Body"] doesn't allow the caller to invoke the read method, instead causing mypy to complain with: Item "bytes" of "Union[bytes, IO[Any]]" has no attribute "read".
A simple example to demonstrate this behavior:
import boto3
from mypy_boto3_s3 import S3ServiceResource
from mypy_boto3_s3.service_resource import Object
class S3Client:
def get_key(self, key: str, bucket_name: str) -> Object:
res: S3ServiceResource = boto3.resource("s3", aws_access_key_id="...", aws_secret_access_key="...")
obj: Object = res.Bucket(name=bucket_name).Object(key=key)
obj.load()
return obj
def read_key(self, bucket_name: str, key: str) -> str:
obj = self.get_key(key, bucket_name)
body = obj.get()["Body"]
return body.read().decode("utf-8")I'm unsure if this is intended and if so how I would go about calling the read method?
The type stub generated for Volume.create_tags (in service_resource.py), expects a Resources argument:
class Volume(Boto3ServiceResource):
...
def create_tags(
self, Resources: List[Any], Tags: List[TagTypeDef], DryRun: bool = None
) -> _Tag:
"""
[Volume.create_tags documentation](https://boto3.amazonaws.com/v1/documentation/api/1.14.46/reference/services/ec2.html#EC2.Volume.create_tags)
"""
However, EC2.Volume.create_tags doesn't require the Resources argument.
boto3-stubs 1.14.46.0
To use boto3-stubs with pyright/pylance, I do the following:
pip install 'boto3-stubs[ec2]'typings/ as .pyi files so pyright picks them up:for f in __init__ client paginator service_resource waiter type_defs; do \
cp $VIRTUAL_ENV/lib/python*/site-packages/mypy_boto3_ec2/$f.py typings/mypy_boto3_ec2/$f.pyi; done
Given this sample:
from typing import Any, Dict, List, Optional, TypeVar, Union
import boto3
from mypy_boto3_ec2 import Client
from mypy_boto3_ec2.type_defs import FilterTypeDef
def describe(config: Dict[str, Any], name: Optional[str] = None) -> List[Dict[str, Any]]:
"""List EC2 instances in the region."""
ec2_client:Client = boto3.client("ec2", region_name=config["region"])
filters: List[FilterTypeDef] = [] if name is None else [{"Name": "tag:Name", "Values": [name]}]
response = ec2_client.describe_instances(Filters=filters)
instances = [
{
"State": i["State"]["Name"],
"Name": first_or_else([t["Value"] for t in i.get("Tags", []) if t["Key"] == "Name"], None),
"Type": i["InstanceType"],
"DnsName": i["PublicDnsName"] if i.get("PublicDnsName", None) != "" else i["PrivateDnsName"],
"LaunchTime": i["LaunchTime"],
"ImageId": i["ImageId"],
"InstanceId": i["InstanceId"],
}
for r in response["Reservations"]
for i in r["Instances"]
]
return sorted(instances, key=lambda i: i["State"] + str(i["Name"]))
E = TypeVar("E")
T = TypeVar("T")
def first_or_else(li: List[E], default: T) -> Union[E, T]:
return li[0] if len(li) > 0 else defaultWhen I run pyright it takes ~20 seconds to type check.
If I then removetype_defs.pyi:
rm typings/mypy_boto3_ec2/type_defs.pyi
This drops to 1 second, which is what I would normally expect.
Using:
Describe the bug
rds.RDSClient.generate_db_auth_token()'s type signature returns None, when the actual function returns str
To Reproduce
Steps to reproduce the behavior:
import boto3
from mypy_boto3 import rds
r: rds.RDSClient = boto3.client("rds")
auth_token = r.generate_db_auth_token("host", 5432, "user")$ mypy test.py
test.py:6: error: "generate_db_auth_token" of "RDSClient" does not return a value
Found 1 error in 1 file (checked 1 source file)
Expected mypy-output
$ mypy test.py
Success: no issues found in 1 source file
Additional context
$ pip freeze | grep boto
boto==2.49.0
boto3==1.12.28
boto3-stubs==1.12.28.0
botocore==1.15.28
mypy-boto3==1.12.28.0
mypy-boto3-rds==1.12.28.0
this is on macOS 10.15.3, running in a virtualenv.
per boto3 documentation this function is supposed to return a signed URL.
this is the generated type definition:
def generate_db_auth_token(
self, DBHostname: str, Port: int, DBUsername: str, Region: str = None
) -> None:
"""
[Client.generate_db_auth_token documentation](https://boto3.amazonaws.com/v1/documentation/api/1.12.28/reference/services/rds.html#RDS.Client.generate_db_auth_token)
"""
Describe the bug
I'm using Docker-compose and my Python interpreter is using my compose file.
Takes forever for PyCharm to update its screen with the type annotations. For example, filling in parameters still shows that I'm missing the exact same parameter I just filled in.
Note I didn't build the type annotations and put it into an output file. I don't know how using my current configuration:
To Reproduce
Create a sample project with boto3 and use Docker compose to run the program (using a Dockerfile running Python 3.8).
Go to settings, Python Interpreter, and add an interpreter. Choose docker-compose. Select your compose file, then select the service. PyCharm should now be indexing your installed requirements.
The problem is I feel like I need to somehow build the type annotations. Not sure how I would do that. But here's what I got:
A sample file:
import boto3
from botocore.exceptions import ClientError
from mypy_boto3_s3.client import S3Client
def generate_presigned_url(key: str) -> str:
client: S3Client = boto3.client("s3")
try:
response = client.generate_presigned_url(
"get_object",
Params={"Bucket": "bucket name", "Key": key},
ExpiresIn=expires,
)
except ClientError as e:
return None
return responseAdditional context
IDE: PyCharm 2020.2.1
OS: MacOS
After writing this issue, removing the stubs import, and removing the type hints, after opening the file back up after closing it, PyCharm is overheating my laptop and still shows the same errors.
Describe the bug
Argument KeyConditionExpression of dynamo Table.query method has type str but should allow ConditionBase (I use Between to define a range for a sorting key)
To Reproduce
Steps to reproduce the behavior:
import boto3
key_exp = Key("partition_key").eq("pk") & Key("time").between(888888, 999999)
resource: DynamoDBServiceResource = boto3.resource("dynamodb")
resource.table.query(
KeyConditionExpression=key_exp, Limit=limit
)src/infrastructure/dynamodb.py:104: error: Argument "KeyConditionExpression" to "query" of "Table" has incompatible type "And"; expected "Optional[str]" [arg-type]
Expected mypy-output
No error expected
Additional context
boto3 1.14.43
Describe the bug
DynamoDB ServiceResource Table method put_item does not properly Type Check the "Item" dictionary. It seems to want a dictionary with values similar to what dynamodb.client.put_item requires, so {"S": str}|{"N": str}|{"M": <...>}), but the AWS examples and documentation for the table resource are clear that the acceptable types for the dictionary values should be 'string'|123|Binary(b'bytes')|True|None|set(['string'])|set([123])|set([Binary(b'bytes')])|[]|{}
I have read the disclaimer that "(blame botocore docs if types are incorrect)" on the pypi page, but without a direct link I wasn't able to immediately verify whether the information is correct or not wherever this project pulls it from. If this is an issue there, please kindly point me in the right direction to re-raise the issue in a more appropriate place.
To Reproduce
Steps to reproduce the behavior:
mypy reproduce.py --show-error-codesReproduce.py
import boto3
from mypy_boto3.dynamodb import DynamoDBServiceResource
dynamo_resource: DynamoDBServiceResource = boto3.resource("dynamodb")
table = dynamo_resource.Table("WHATEVER")
table.put_item(
Item={
'username': 'janedoe',
'first_name': 'Jane',
'last_name': 'Doe',
'age': 25,
'account_type': 'standard_user',
}
)reproduce.py:8: error: Dict entry 0 has incompatible type "str": "str"; expected "str": "AttributeValueTypeDef" [dict-item]
reproduce.py:9: error: Dict entry 1 has incompatible type "str": "str"; expected "str": "AttributeValueTypeDef" [dict-item]
reproduce.py:10: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "AttributeValueTypeDef" [dict-item]
reproduce.py:11: error: Dict entry 3 has incompatible type "str": "int"; expected "str": "AttributeValueTypeDef" [dict-item]
reproduce.py:12: error: Dict entry 4 has incompatible type "str": "str"; expected "str": "AttributeValueTypeDef" [dict-item]
Found 5 errors in 1 file (checked 1 source file)
(actual type is not exactly ItemAttribute, once I get my home environment set up I'll edit in actual type, but it is expecting something based on TypedDict rather than allowing str|int|float|<etc.> directly)
Expected mypy-output
Success: no issues found in 1 source file
Additional context
Mac OS,
boto3-stubs was installed with pipenv (pipenv install boto3-stubs["essential,sts"]),
boto3 version == 1.11.16
boto3-stubs version == 1.11.16.0
excellent work, thanks for shepherding this further.
Describe the bug
Probably related to #42, looks like the signature used for Image.create_tags() is from the client, and not from the image resource.
ec2: EC2ServiceResource = boto3.resource("ec2")
image = ec2.Image(ami)
image.create_tags(Tags=tags)version: 1.14.49.0
Describe the bug
According the boto3 documentation, a VpcPeeringConnection has 'references' to accepter_vpc and requester_vpc. But these references aren't being generated so MyPy complains.
To Reproduce
Steps to reproduce the behavior:
import boto3
ec2 = boto3.resource("ec2", region_name="eu-north-1")
vpc = ec2.Vpc("foo")
vpc_peer = vpc.request_vpc_peering_connection(PeerVpcId="bar")
vpc_peer.accepter_vpc # mypy error
vpc_peer.requester_vpc # mypy errorbug.py:6: error: "VpcPeeringConnection" has no attribute "accepter_vpc"; maybe "accepter_vpc_info"?
vpc_peer.accepter_vpc
^
bug.py:7: error: "VpcPeeringConnection" has no attribute "requester_vpc"; maybe "requester_vpc_info"?
vpc_peer.requester_vpc
^
Found 2 errors in 1 file (checked 1 source file)
Expected mypy-output
Success: no issues found in 1 source file
Additional context
Your OS, boto3-stubs installation method, boto3 version, etc.
OS: macOS and Linux (Alpine)
Installation method: pip install boto3-stubs[ec2]
boto3 1.13.6
boto3-stubs 1.13.6.0
mypy 0.770
mypy-boto3 1.13.6.0
mypy-boto3-ec2 1.13.6.0
Describe the bug
It's not clear from the latest 3.x release notes if this is intended, but it seems mypy_boto3_[service].client modules have been removed. The boto3-stubs package still recommends importing it like this.
To Reproduce
Steps to reproduce the behavior:
from mypy_boto3_glue.client import GlueClient from mypy_boto3_glue.client import GlueClient
ModuleNotFoundError: No module named 'mypy_boto3_glue.client'
Expected mypy-output
No error I guess
Additional context
MacOS or Linux, tested on Py 3.7 and 3.8.
To Reproduce
Steps to reproduce the behavior:
import boto3
import mypy_boto3_dynamodb as dynamodb
client: dynamodb.DynamoDBClient = boto3.client("dynamodb") 2:8 - error: Stub file not found for "mypy_boto3_dynamodb" (reportMissingTypeStubs)
4:9 - error: Type of "DynamoDBClient" is unknown (reportUnknownMemberType)
4:41 - error: "client" is not a known member of module (reportGeneralTypeIssues)
4:35 - error: Type of "client" is unknown (reportUnknownMemberType)
4:1 - error: Type of "client" is unknown (reportUnknownVariableType)
Describe the bug
Docs of mypy-boto3-rds_data says to run mypy_boto3 but outputs the following exception:
Traceback (most recent call last):
File "/Venvs/Cloud/bin/mypy_boto3", line 11, in <module>
load_entry_point('mypy-boto3==1.13.6.0', 'console_scripts', 'mypy_boto3')()
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3/main.py", line 131, in main
build_package_stubs(submodule, logger)
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3/main.py", line 311, in build_package_stubs
submodule_path / "__init__.py", _get_proxy_contents(submodule.module_name), logger,
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3/main.py", line 170, in _get_proxy_contents
service_module = importlib.import_module(module_name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3_rds_data/__init__.py", line 17, in <module>
from mypy_boto3_rds_data.client import RDSDataServiceClient, RDSDataServiceClient as Client
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3_rds_data/client.py", line 17, in <module>
from mypy_boto3_rds_data.type_defs import (
File "/Venvs/Cloud/lib/python3.8/site-packages/mypy_boto3_rds_data/type_defs.py", line 89, in <module>
"StructValueTypeDef", {"attributes": List[ValueTypeDef]}, total=False
NameError: name 'ValueTypeDef' is not defined
Lines 88-90 should be transformed to a lazy typing:
StructValueTypeDef = TypedDict(
- "StructValueTypeDef", {"attributes": List[ValueTypeDef]}, total=False
+ "StructValueTypeDef", {"attributes": List["ValueTypeDef"]}, total=False
)To Reproduce
Steps to reproduce the behavior:
mypy_boto3Additional context
> uname -rs
Darwin 19.4.0
> pip freeze | grep mypy
mypy-boto3==1.13.6.0
mypy-boto3-builder==1.0.9
mypy-boto3-cloudformation==1.13.6.0
mypy-boto3-dynamodb==1.13.6.0
mypy-boto3-ec2==1.13.6.0
mypy-boto3-lambda==1.13.6.0
mypy-boto3-rds==1.13.6.0
mypy-boto3-rds-data==1.13.6.0
mypy-boto3-s3==1.13.6.0
mypy-boto3-sqs==1.13.6.0Describe the bug
mypy_boto3_s3.S3ServiceResource.buckets should support __iter__ (and the return type of __iter__ should be mypy_boto3_s3.service_resource.Bucket
I can't seem to reproduce this with the mypy_boto3_s3.service_resource.Bucket.objects collection, although they seem to be defined identically, so I'm not sure why __iter__ seems to be supported for one but not the other.
To Reproduce
Steps to reproduce the behavior:
import boto3
from mypy_boto3_s3 import S3ServiceResource
service_resource: S3ServiceResource = boto3.resource("s3")
buckets = service_resource.buckets.all()
for bucket in buckets:
reveal_type(bucket)s3_service_resource.py:8: error: "Type[ServiceResourceBucketsCollection]" has no attribute "__iter__" (not iterable)
s3_service_resource.py:9: note: Revealed type is 'Any'
Found 1 error in 1 file (checked 1 source file)
Expected mypy-output
No Errors (checked 1 source file)
Additional context
Mac OS, pipenv installation, boto3 version 1.12.12, etc.
After updating to boto3-stubs version 1.12.48 (this matches the boto3 version I'm using) I continue to see this error:
Module has no attribute "client"
A simple test case that consistently reproduces the problem:
import boto3
sts_client = boto3.client("sts")I didn't open a bug because I'm not sure if this is not something I'm doing wrong with the installation or usage, or that the problem isn't with PyCharm.
The error message:
Expected type 'SSMClient', got 'BaseClient' instead
The code that triggers it:
import boto3
import mypy_boto3_ssm as ssm
client: ssm.SSMClient = boto3.client("ssm") # the message is for the 'client' variableThe package versions:
boto3==1.14.37.0
boto3-stubs==1.14.37.0
PyCharm 2020.2
If this should not occur, I will investigate further to know if the problem is with PyCharm's typechecks.
It seems that pages() for a given resource collection is annotated with a return type of Iterator[ResourceType]. The documentation at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/collections.html#boto3.resources.collection.CollectionManager.pages says that it returns a generator which yields iterables of the given resources, somthing like Generator[Iterable[resource], None, None] I think
The documentation seems to contradict itself when it says:
Return type
list(ServiceResource)
EDIT:
The boto3 codebase seems to be yielding Lists: https://github.com/boto/boto3/blob/0cc6042615fd44c6822bd5be5a4019d0901e5dd2/boto3/resources/collection.py#L177
Describe the bug
Not all type_defs are able to be imported. This prevents typing of functions that make use of partial responses from boto3 e.g. taking a subdict from the full response's dict. It'd be nice to access all the TypeDefs so say when accessing SNS I could grab the SubscriptionTypeDef dict (currently can't be imported) from the ListSubscriptionsByTopicResponseTypeDef (able to be imported just fine) dict.
To Reproduce
import mypy_boto3_sns as sns
from boto3 import client as boto_client
the_region = "FIXME"
aws_key = "FIXME"
aws_secret = "FIXME"
the_topic_arn = "FIXME"
client = boto_client(
"sns", region_name=the_region, aws_access_key_id=aws_key, aws_secret_access_key=aws_secret
)
def list_subs() -> List[sns.type_defs.SubscriptionTypeDef]:
response = client.list_subscriptions_by_topic(TopicArn=the_topic_arn)
return response.get("Subscriptions", [])
...Additional context
Your OS, boto3-stubs installation method, boto3 version, etc.
Ubuntu 20.04, poetry add --dev boto3-stubs, boto3 == "1.14.28", boto3-stubs = {extras = ["batch", "essential", "sns", "ssm"], version = "1.14.28"}
Hello and thanks for providing these very useful stubs!
Describe the bug
The return type of STS.Client.generate_presigned_url is set to None while it should be str
From mypy_boto3_sts/client.py:
def generate_presigned_url(
self,
ClientMethod: str,
Params: Dict[str, Any] = None,
ExpiresIn: int = 3600,
HttpMethod: str = None,
) -> None:
"""
[Client.generate_presigned_url documentation](https://boto3.amazonaws.com/v1/documentation/api/1.12.6/reference/services/sts.html#STS.Client.generate_presigned_url)From the generate_presigned_url documentation
Returns
The presigned url
To Reproduce
Steps to reproduce the behavior:
pip download 'boto3-stubs[sts]'tar xf mypy-boto3-sts-1.12.6.0.tar.gzgrep -A 10 "def generate_presigned_url" mypy-boto3-sts-1.12.6.0/mypy_boto3_sts/client.pySample Python code:
import boto3
print(boto3.__version__) # '1.9.205'
c = boto3.client('sts')
assert type(c.generate_presigned_url('get_caller_identity')) is strI get the error message
error: Incompatible types (expression has type "List[Dict[str, str]]", TypedDict item "Dimensions" has type "List[DimensionTypeDef]")
Is there any way to get more information what the TypedDict Dimensions contains? My dictionaries have the following keys:
What else is expected?
Describe the bug
When writing a lambda handler function, I would like to have type annotations for the input parameter types (event and context).
Additional context
This might not be part of boto3, so that could be the reason for not being able to find it.
I would welcome suggestions as well, not necessarily implementation of it.
Describe the bug
A clear and concise description of what the bug is.
I'm trying to use this for rds-data, but getting this error when running the pip install:
boto3-stubs 1.12.33 does not provide the extra 'mypy-boto3-rds-data'
To Reproduce
Steps to reproduce the behavior:
Additional context
Your OS, boto3-stubs installation method, boto3 version, etc.
macOS Catalina
boto3==1.12.33
Describe the bug
Module has no attribute "client"
To Reproduce
Steps to reproduce the behavior:
import boto3
s3_client = boto3.client("s3")
...$ python -m mypy src/app.py
src/app.py:8: error: Skipping analyzing 'botocore.exceptions': found module but no type hints or library stubs
src/app.py:8: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
src/app.py:12: error: Module 'mypy_boto3_sts' has no attribute 'ClientAssumeRoleResponseCredentialsTypeDef'
src/app.py:20: error: Module has no attribute "client"
src/app.py:30: error: Incompatible types in assignment (expression has type "Optional[ClientAssumeRoleResponseCredentialsTypeDef]", variable has type "Dict[Any, Any]")
src/app.py:34: error: Module has no attribute "client"
Found 5 errors in 1 file (checked 1 source file)
Expected mypy-output
...
Additional context
Your OS, boto3-stubs installation method, boto3 version, etc.
$ python -m pip show boto3-stubs
Name: boto3-stubs
Version: 1.12.33.0
Summary: Type annotations for boto3 1.12.33, generated by mypy-boto3-buider 1.0.6
$ python -m pip show boto3
Name: boto3
Version: 1.12.19
Summary: The AWS SDK for Python
I also noticed this issue vemel/mypy_boto3#12 but I am still having problems. I honestly think this may be a problem on my end but wanted to open an issue to be sure.
Describe the bug
Every response contains a ResponseMetadata key in addition to the response-specific data. This is not added to the response types, leading to mypy errors when attempting to access it.
To Reproduce
Steps to reproduce the behavior:
boto3-stubs['lambda']import boto3
import json
client = boto3.client('lambda')
response = client.invoke(
FunctionName='arn:aws:lambda:fake',
InvocationType='Event',
Payload=json.dumps({'message': 'hi there'}).encode('utf-8'),
)
print(response['ResponseMetadata']['RequestId'])test.py:10: error: TypedDict "InvocationResponseTypeDef" has no key 'ResponseMetadata'
Found 1 error in 1 file (checked 1 source file)
Expected mypy-output
Success: no issues found in 1 source file
Describe the bug
I believe the type annotations of the various "Exceptions" classes are incorrect: E.g.
class Exceptions:
BucketAlreadyExists: Boto3ClientError
BucketAlreadyOwnedByYou: Boto3ClientError
tells Mypy that exceptions.BucketAlreadyExists is an instance of BotoClientError, when in reality it is a subclass. Wouldn't
class Exceptions:
BucketAlreadyExists: Type[Boto3ClientError]
BucketAlreadyOwnedByYou: Type[Boto3ClientError]
be correct?
For some reason, mypy only notices this error in the presence of stub files for botocore.exceptions (see the following steps).
To Reproduce
Install boto3-stubs[s3]
Write a script test.py:
import boto3
from mypy_boto3 import s3
client: s3.Client = boto3.client('s3')
try:
client.list_objects(Bucket='my-test-bucket-7f327deb-2584-4754-99ba-c01f52225fa7')
except client.exceptions.NoSuchBucket:
print("Bucket not found")
__init__.pyi and a file exceptions.pyi with the following content:from typing import Dict, Any
class ClientError(Exception):
def __init__(self, error_response: Dict[str, Any], operation_name: str): ...
[mypy]
mypy_path=stubs
mypy test.py now gives this error:test.py:7: error: Exception type must be derived from BaseException
Found 1 error in 1 file (checked 1 source file)
Additional context
mypy 0.761
boto3-stubs 1.12.27.0
After following the instructions with the project new version, the boto3 client type annotations are not recognised, and rather each boto3 client type according to PyCharm is BaseClient, which doesn't contain the client specific methods.
I think this is caused by the fact that output direct of mypy_boto3_builder is not a format compatible with python's import system. After running python -m mypy_boto3_builder typings -s s3, I'd get the following directory structure:
typings
├── boto3_stubs_package
│ ├── README.md
│ ├── boto3-stubs
│ └── setup.py
├── master_package
│ ├── README.md
│ ├── mypy_boto3
│ └── setup.py
└── mypy_boto3_s3_package
├── README.md
├── mypy_boto3_s3
└── setup.py
The problem here PyCharm is looking for package ending with -stubs but it only sees, boto3_stubs_package, master_package and mypy_boto3_s3_package. boto3-stubs also requires mypy_boto3_s3 but again python interpreter doesn't see it. If we move out the directories to one level higher then everything starts working. So this is what I what did:
typings
├── boto3-stubs
├── mypy_boto3
├── mypy_boto3_s3
Of course this is just one way to solve the problem.
To Reproduce
Steps to reproduce the behavior:
pip install mypy_boto3_builder boto3
python -m mypy_boto3_builder typings -s s3
In pycharm add the typings directory to the libraries path.
Open the following code sample in pycharm:
import boto3
s3 = boto3.client('s3')
...** actual type**
BaseClient
Expected type
S3Client
Additional context
MacOS v10.15.7
mypy-boto3-builder==4.3.1
boto3==1.16.43
PyCharm Community 2020.2.5
To Reproduce
Steps to reproduce the behavior:
>>> import mypy_boto3_sts
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ryan/.local/share/virtualenvs/aspiredu-sa7rbeeh/lib/python3.8/site-packages/mypy_boto3_sts/__init__.py", line 19, in <module>
from mypy_boto3_sts.client import STSClient
File "/home/ryan/.local/share/virtualenvs/aspiredu-sa7rbeeh/lib/python3.8/site-packages/mypy_boto3_sts/client.py", line 18, in <module>
from mypy_boto3_sts.type_defs import (
File "/home/ryan/.local/share/virtualenvs/aspiredu-sa7rbeeh/lib/python3.8/site-packages/mypy_boto3_sts/type_defs.py", line 14, in <module>
if sys.version_info >= (3, 8):
NameError: name 'sys' is not definedVersion info:
"mypy-boto3-sts": {
"hashes": [
"sha256:7e83ba0699346ca92f54cdb4084d33ee100bb8248d109f919615aa153589dbb9",
"sha256:97fa61a2c8d1d1526722c8b8f3bb208172c6fa062dc314d16bd7df31a4e2fce5"
],
"version": "==1.15.12.1"
},
Setup to reproduce the behavior:
pip install 'boto3-stubs[s3]==1.14.30'
pip install 'boto3==1.14.30'
python -m mypy_boto3
Run pyright on the following code sample:
import boto3
s3_resource = boto3.resource('s3', region_name='us-east-1')pyright output:
No configuration file found.
stubPath /private/tmp/t/typings is not a valid directory.
Assuming Python platform Darwin
Searching for source files
Found 1 source file
/private/tmp/t/test.py
3:21 - error: "resource" is not a known member of module (reportGeneralTypeIssues)
1 error, 0 warnings
Completed in 0.576sec
Expected output
No type errors.
NB: When only boto3 is installed (ie: without stubs) there are no type errors.
pyright 1.1.58
look into better visual studio code integration, without requiring explicit type hinting. Maybe now with pylance extension, where stubs can be stored in a directory, and used from there, maybe that will help. so we don't have to add type hints to get auto-complete/hover working. See here: https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance
First I create a python 3.8 conda environment and install pyfgaws:
❯ conda create -y -n debug-py38 python=3.8
❯ conda activate debug-py38
❯ python -m pip install pyfgawsNext I trie to import the stub for batch:
❯ python
>>> import mypy_boto3.batch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mypy_boto3'So RTFM:
However, if you use any other way or notice that services stubs do not work, you can build services index manually.
# Use this command when you add or remove service packages python -m mypy_boto3
That works:
❯ python -m mypy_boto3
mypy_boto3: INFO Discovered Batch service stubs in mypy-boto3-batch
mypy_boto3: INFO Discovered CloudWatchLogs service stubs in mypy-boto3-logs
mypy_boto3: INFO You can now use Batch and CloudWatchLogs type annotations.
❯ python
>>> import mypy_boto3.batch
>>> But the manual also says:
This package generates a few source files depending on services that you installed. Generation is done by a post-install script, so as long as you use pip, pipfile or poetry everything should be done automatically.
I am using pip to install the package (pyfgaws), and that package is built using poetry with the correct extras specified. So why do I need to manually index, and/or why isn't the "post-install script" handling this for me automatically.
Thank-you in advance for any help you could offer.
Describe the bug
The EC2Client stubs are missing the attribute meta.
To Reproduce
Steps to reproduce the behavior:
import boto3
ec2_client = boto3.client("ec2")
print(ec2_client.meta.region_name)test.py:5: error: "EC2Client" has no attribute "meta"
Found 1 error in 1 file (checked 1 source file)
Expected mypy-output
No type error.
Additional context
boto3-stubs==1.14.46.0
It seems this typedef is not in the place I expected it to be, how can I go about easily finding it and others?
src/app.py:30: error: Incompatible types in assignment (expression has type "Optional[ClientAssumeRoleResponseCredentialsTypeDef]", variable has type "Dict[Any, Any]")
...
src/app.py:12: error: Module 'mypy_boto3_sts' has no attribute 'ClientAssumeRoleResponseCredentialsTypeDef'
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.