aws/aws-iot-device-sdk-python

SDK for connecting to AWS IoT from a device using Python.

New Version Available

A new AWS IoT Device SDK is [now available](https://github.com/awslabs/aws-iot-device-sdk-python-v2). It is a complete rework, built to improve reliability, performance, and security. We invite your feedback!

This SDK will no longer receive feature updates, but will receive security updates.

AWS IoT Device SDK for Python

The AWS IoT Device SDK for Python allows developers to write Python script to use their devices to access the AWS IoT platform through MQTT or MQTT over the WebSocket protocol. By connecting their devices to AWS IoT, users can securely work with the message broker, rules, and the device shadow (sometimes referred to as a thing shadow) provided by AWS IoT and with other AWS services like AWS Lambda, Amazon Kinesis, Amazon S3, and more.

• Overview
• Installation
• Use the SDK
• Key Features
• Examples
• API Documentation
• License
• Support

---

Overview

This document provides instructions for installing and configuring the AWS IoT Device SDK for Python. It includes examples demonstrating the use of the SDK APIs.

MQTT Connections

The SDK is built on top of a modified Paho MQTT Python client library. Developers can choose from two types of connections to connect to AWS IoT:

• MQTT (over TLS 1.2) with X.509 certificate-based mutual authentication.
• MQTT over the WebSocket protocol with AWS Signature Version 4 authentication.
• MQTT (over TLS 1.2) with X.509 certificate-based mutual authentication with TLS ALPN extension.

For MQTT over TLS (port 8883 and port 443), a valid certificate and a private key are required for authentication. For MQTT over the WebSocket protocol (port 443), a valid AWS Identity and Access Management (IAM) access key ID and secret access key pair are required for authentication.

Device Shadow

A device shadow, or thing shadow, is a JSON document that is used to store and retrieve current state information for a thing (device, app, and so on). A shadow can be created and maintained for each thing or device so that its state can be get and set regardless of whether the thing or device is connected to the Internet. The SDK implements the protocol for applications to retrieve, update, and delete shadow documents. The SDK allows operations on shadow documents of single or multiple shadow instances in one MQTT connection. The SDK also allows the use of the same connection for shadow operations and non-shadow, simple MQTT operations.

Installation

Minimum Requirements

• Python 2.7+ or Python 3.3+ for X.509 certificate-based mutual authentication via port 8883 and MQTT over WebSocket protocol with AWS Signature Version 4 authentication
• Python 2.7.10+ or Python 3.5+ for X.509 certificate-based mutual authentication via port 443

• OpenSSL version 1.0.1+ (TLS version 1.2) compiled with the Python executable for X.509 certificate-based mutual authentication

  To check your version of OpenSSL, use the following command in a Python interpreter:

  >>> import ssl
  >>> ssl.OPENSSL_VERSION

Install from pip

pip install AWSIoTPythonSDK

Build from source

git clone https://github.com/aws/aws-iot-device-sdk-python.git
cd aws-iot-device-sdk-python
python setup.py install

Download the zip file

The SDK zip file is available here. Unzip the package and install the SDK like this:

python setup.py install

Use the SDK

Collection of Metrics

Beginning with Release v1.3.0 of the SDK, AWS collects usage metrics indicating which language and version of the SDK is being used. This feature is enabled by default and allows us to prioritize our resources towards addressing issues faster in SDKs that see the most and is an important data point. However, we do understand that not all customers would want to report this data. In that case, the sending of usage metrics can be easily disabled by the user using the corresponding API:

# AWS IoT MQTT Client
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.enableMetricsCollection()
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.disableMetricsCollection()
# AWS IoT MQTT Shadow Client
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTShadowClient.enableMetricsCollection()
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTShadowClient.disableMetricsCollection()

Credentials

The SDK supports two types of credentials that correspond to the two connection types:

• X.509 certificate

  For the certificate-based mutual authentication connection type. Download the AWS IoT root CA. Use the AWS IoT console to create and download the certificate and private key. You must specify the location of these files when you initialize the client.

• IAM credentials

  For the Websocket with Signature Version 4 authentication type. You will need IAM credentials: an access key ID, a secret access key, and an optional session token. You must also download the AWS IoT root CA. You can specify the IAM credentials by:

  • Passing method parameters

    The SDK will first call the following method to check if there is any input for a custom IAM credentials configuration:

    # AWS IoT MQTT Client
    AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.configureIAMCredentials(obtainedAccessKeyID, obtainedSecretAccessKey, obtainedSessionToken)        
    # AWS IoT MQTT Shadow Client
    AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTShadowClient.configureIAMCredentials(obtainedAccessKeyID, obtainedSecretAccessKey, obtainedSessionToken)

    Note: We do not recommend hard-coding credentials in a custom script. You can use Amazon Cognito Identity or another credential provider.

  • Exporting environment variables

    If there is no custom configuration through method calls, the SDK will then check these environment variables for credentials:

    AWS_ACCESS_KEY_ID

    The access key for your AWS account.

    AWS_SECRET_ACCESS_KEY

    The secret key for your AWS account.

    AWS_SESSION_TOKEN

    The session key for your AWS account. This is required only when you are using temporary credentials. For more information, see here.

    You can set your IAM credentials as environment variables by using the preconfigured names. For Unix systems, you can do the following:

    export AWS_ACCESS_KEY_ID=<your aws access key id string>
    export AWS_SECRET_ACCESS_KEY=<your aws secret access key string>
    export AWS_SESSION_TOKEN=<your aws session token string>

    For Windows, open Control Panel and choose System. In Advanced system settings choose Environment Variables and then configure the required environment variables.

  • Configuring shared credentials file

    If there are no such environment variables specified, the SDK will check the default section for a shared credentials file (in Unix, ~/.aws/credentials and in Windows, %UserProfile%\.aws\credentials) as follows:

    [default]
    aws_access_key_id=foo
    aws_secret_access_key=bar
    aws_session_token=baz

    You can use the AWS CLI to configure the shared credentials file <http://aws.amazon.com/cli/>`__:

    aws configure

AWSIoTMQTTClient

This is the client class used for plain MQTT communication with AWS IoT. You can initialize and configure the client like this:

# Import SDK packages
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient

# For certificate based connection
myMQTTClient = AWSIoTMQTTClient("myClientID")
# For Websocket connection
# myMQTTClient = AWSIoTMQTTClient("myClientID", useWebsocket=True)
# Configurations
# For TLS mutual authentication
myMQTTClient.configureEndpoint("YOUR.ENDPOINT", 8883)
# For Websocket
# myMQTTClient.configureEndpoint("YOUR.ENDPOINT", 443)
# For TLS mutual authentication with TLS ALPN extension
# myMQTTClient.configureEndpoint("YOUR.ENDPOINT", 443)
myMQTTClient.configureCredentials("YOUR/ROOT/CA/PATH", "PRIVATE/KEY/PATH", "CERTIFICATE/PATH")
# For Websocket, we only need to configure the root CA
# myMQTTClient.configureCredentials("YOUR/ROOT/CA/PATH")
myMQTTClient.configureOfflinePublishQueueing(-1)  # Infinite offline Publish queueing
myMQTTClient.configureDrainingFrequency(2)  # Draining: 2 Hz
myMQTTClient.configureConnectDisconnectTimeout(10)  # 10 sec
myMQTTClient.configureMQTTOperationTimeout(5)  # 5 sec
...

For basic MQTT operations, your script will look like this:

...
myMQTTClient.connect()
myMQTTClient.publish("myTopic", "myPayload", 0)
myMQTTClient.subscribe("myTopic", 1, customCallback)
myMQTTClient.unsubscribe("myTopic")
myMQTTClient.disconnect()
...

AWSIoTShadowClient

This is the client class used for device shadow operations with AWS IoT. You can initialize and configure the client like this:

from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTShadowClient

# For certificate based connection
myShadowClient = AWSIoTMQTTShadowClient("myClientID")
# For Websocket connection
# myMQTTClient = AWSIoTMQTTClient("myClientID", useWebsocket=True)
# Configurations
# For TLS mutual authentication
myShadowClient.configureEndpoint("YOUR.ENDPOINT", 8883)
# For Websocket
# myShadowClient.configureEndpoint("YOUR.ENDPOINT", 443)
# For TLS mutual authentication with TLS ALPN extension
# myShadowClient.configureEndpoint("YOUR.ENDPOINT", 443)
myShadowClient.configureCredentials("YOUR/ROOT/CA/PATH", "PRIVATE/KEY/PATH", "CERTIFICATE/PATH")
# For Websocket, we only need to configure the root CA
# myShadowClient.configureCredentials("YOUR/ROOT/CA/PATH")
myShadowClient.configureConnectDisconnectTimeout(10)  # 10 sec
myShadowClient.configureMQTTOperationTimeout(5)  # 5 sec
...

For shadow operations, your script will look like this:

...
myShadowClient.connect()
# Create a device shadow instance using persistent subscription
myDeviceShadow = myShadowClient.createShadowHandlerWithName("Bot", True)
# Shadow operations
myDeviceShadow.shadowGet(customCallback, 5)
myDeviceShadow.shadowUpdate(myJSONPayload, customCallback, 5)
myDeviceShadow.shadowDelete(customCallback, 5)
myDeviceShadow.shadowRegisterDeltaCallback(customCallback)
myDeviceShadow.shadowUnregisterDeltaCallback()
...

You can also retrieve the MQTTClient(MQTT connection) to perform plain MQTT operations along with shadow operations:

myMQTTClient = myShadowClient.getMQTTConnection()
myMQTTClient.publish("plainMQTTTopic", "Payload", 1)

AWSIoTMQTTThingJobsClient __________________

This is the client class used for jobs operations with AWS IoT. See docs here: https://docs.aws.amazon.com/iot/latest/developerguide/iot-jobs.html You can initialize and configure the client like this:

from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTThingJobsClient

# For certificate based connection
myJobsClient = AWSIoTMQTTThingJobsClient("myClientID", "myThingName")
# For Websocket connection
# myJobsClient = AWSIoTMQTTThingJobsClient("myClientID", "myThingName", useWebsocket=True)
# Configurations
# For TLS mutual authentication
myJobsClient.configureEndpoint("YOUR.ENDPOINT", 8883)
# For Websocket
# myJobsClient.configureEndpoint("YOUR.ENDPOINT", 443)
myJobsClient.configureCredentials("YOUR/ROOT/CA/PATH", "PRIVATE/KEY/PATH", "CERTIFICATE/PATH")
# For Websocket, we only need to configure the root CA
# myJobsClient.configureCredentials("YOUR/ROOT/CA/PATH")
myJobsClient.configureConnectDisconnectTimeout(10)  # 10 sec
myJobsClient.configureMQTTOperationTimeout(5)  # 5 sec
...

For job operations, your script will look like this:

...
myJobsClient.connect()
# Create a subsciption for $notify-next topic
myJobsClient.createJobSubscription(notifyNextCallback, jobExecutionTopicType.JOB_NOTIFY_NEXT_TOPIC)
# Create a subscription for update-job-execution accepted response topic
myJobsClient.createJobSubscription(updateSuccessfulCallback, jobExecutionTopicType.JOB_UPDATE_TOPIC, jobExecutionTopicReplyType.JOB_ACCEPTED_REPLY_TYPE, '+')
# Send a message to start the next pending job (if any)
myJobsClient.sendJobsStartNext(statusDetailsDict)
# Send a message to update a successfully completed job
myJobsClient.sendJobsUpdate(jobId, jobExecutionStatus.JOB_EXECUTION_SUCCEEDED, statusDetailsDict)
...

You can also retrieve the MQTTClient(MQTT connection) to perform plain MQTT operations along with shadow operations:

myMQTTClient = myJobsClient.getMQTTConnection()
myMQTTClient.publish("plainMQTTTopic", "Payload", 1)

DiscoveryInfoProvider

This is the client class for device discovery process with AWS IoT Greengrass. You can initialize and configure the client like this:

from AWSIoTPythonSDK.core.greengrass.discovery.providers import DiscoveryInfoProvider

discoveryInfoProvider = DiscoveryInfoProvider()
discoveryInfoProvider.configureEndpoint("YOUR.IOT.ENDPOINT")
discoveryInfoProvider.configureCredentials("YOUR/ROOT/CA/PATH", "CERTIFICATE/PATH", "PRIVATE/KEY/PATH")
discoveryInfoProvider.configureTimeout(10)  # 10 sec

To perform the discovery process for a Greengrass Aware Device (GGAD) that belongs to a deployed group, your script should look like this:

discoveryInfo = discoveryInfoProvider.discover("myGGADThingName")
# I know nothing about the group/core I want to connect to. I want to iterate through all cores and find out.
coreList = discoveryInfo.getAllCores()
groupIdCAList = discoveryInfo.getAllCas()  # list([(groupId, ca), ...])
# I know nothing about the group/core I want to connect to. I want to iterate through all groups and find out.
groupList = discoveryInfo.getAllGroups()
# I know exactly which group, which core and which connectivity info I need to connect.
connectivityInfo = discoveryInfo.toObjectAtGroupLevel()["YOUR_GROUP_ID"]
                                .getCoreConnectivityInfo("YOUR_CORE_THING_ARN")
                                .getConnectivityInfo("YOUR_CONNECTIVITY_ID")
# Connecting logic follows...
...

For more information about discovery information access at group/core/connectivity info set level, please refer to the API documentation for AWSIoTPythonSDK.core.greengrass.discovery.models, Greengrass Discovery documentation or Greengrass overall documentation.

Synchronous APIs and Asynchronous APIs

Beginning with Release v1.2.0, SDK provides asynchronous APIs and enforces synchronous API behaviors for MQTT operations, which includes: - connect/connectAsync - disconnect/disconnectAsync - publish/publishAsync - subscribe/subscribeAsync - unsubscribe/unsubscribeAsync

- Asynchronous APIs Asynchronous APIs translate the invocation into MQTT packet and forward it to the underneath connection to be sent out. They return immediately once packets are out for delivery, regardless of whether the corresponding ACKs, if any, have been received. Users can specify their own callbacks for ACK/message (server side PUBLISH) processing for each individual request. These callbacks will be sequentially dispatched and invoked upon the arrival of ACK/message (server side PUBLISH) packets.

- Synchronous APIs Synchronous API behaviors are enforced by registering blocking ACK callbacks on top of the asynchronous APIs. Synchronous APIs wait on their corresponding ACK packets, if there is any, before the invocation returns. For example, a synchronous QoS1 publish call will wait until it gets its PUBACK back. A synchronous subscribe call will wait until it gets its SUBACK back. Users can configure operation time out for synchronous APIs to stop the waiting.

Since callbacks are sequentially dispatched and invoked, calling synchronous APIs within callbacks will deadlock the user application. If users are inclined to utilize the asynchronous mode and perform MQTT operations within callbacks, asynchronous APIs should be used. For more details, please check out the provided samples at samples/basicPubSub/basicPubSub_APICallInCallback.py

Key Features

Progressive Reconnect Back Off

When a non-client-side disconnect occurs, the SDK will reconnect automatically. The following APIs are provided for configuration:

# AWS IoT MQTT Client
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.configureAutoReconnectBackoffTime(baseReconnectQuietTimeSecond, maxReconnectQuietTimeSecond, stableConnectionTimeSecond)
# AWS IoT MQTT Shadow Client
AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTShadowClient.configureAutoReconnectBackoffTime(baseReconnectQuietTimeSecond, maxReconnectQuietTimeSecond, stableConnectionTimeSecond)

The auto-reconnect occurs with a progressive backoff, which follows this mechanism for reconnect backoff time calculation:

t^current = min(2ⁿ t^base, t^max)

where t^current is the current reconnect backoff time, t^base is the base reconnect backoff time, t^max is the maximum reconnect backoff time.

The reconnect backoff time will be doubled on disconnect and reconnect attempt until it reaches the preconfigured maximum reconnect backoff time. After the connection is stable for over the stableConnectionTime, the reconnect backoff time will be reset to the baseReconnectQuietTime.

If no configureAutoReconnectBackoffTime is called, the following default configuration for backoff timing will be performed on initialization:

baseReconnectQuietTimeSecond = 1
maxReconnectQuietTimeSecond = 32
stableConnectionTimeSecond = 20

Offline Requests Queueing with Draining

If the client is temporarily offline and disconnected due to network failure, publish/subscribe/unsubscribe requests will be added to an internal queue until the number of queued-up requests reaches the size limit of the queue. This functionality is for plain MQTT operations. Shadow client contains time-sensitive data and is therefore not supported.

The following API is provided for configuration:

AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.configureOfflinePublishQueueing(queueSize, dropBehavior)

After the queue is full, offline publish/subscribe/unsubscribe requests will be discarded or replaced according to the configuration of the drop behavior:

# Drop the oldest request in the queue
AWSIoTPythonSDK.MQTTLib.DROP_OLDEST = 0
# Drop the newest request in the queue
AWSIoTPythonSDK.MQTTLib.DROP_NEWEST = 1

Let's say we configure the size of offlinePublishQueue to 5 and we have 7 incoming offline publish requests.

In a DROP_OLDEST configuration:

myClient.configureOfflinePublishQueueing(5, AWSIoTPythonSDK.MQTTLib.DROP_OLDEST);

The internal queue should be like this when the queue is just full:

HEAD ['pub_req1', 'pub_req2', 'pub_req3', 'pub_req4', 'pub_req5']

When the 6th and the 7th publish requests are made offline, the internal queue will be like this:

HEAD ['pub_req3', 'pub_req4', 'pub_req5', 'pub_req6', 'pub_req7']

Because the queue is already full, the oldest requests pub_req1 and pub_req2 are discarded.

In a DROP_NEWEST configuration:

myClient.configureOfflinePublishQueueing(5, AWSIoTPythonSDK.MQTTLib.DROP_NEWEST);

The internal queue should be like this when the queue is just full:

HEAD ['pub_req1', 'pub_req2', 'pub_req3', 'pub_req4', 'pub_req5']

When the 6th and the 7th publish requests are made offline, the internal queue will be like this:

HEAD ['pub_req1', 'pub_req2', 'pub_req3', 'pub_req4', 'pub_req5']

Because the queue is already full, the newest requests pub_req6 and pub_req7 are discarded.

When the client is back online, connected, and resubscribed to all topics it has previously subscribed to, the draining starts. All requests in the offline request queue will be resent at the configured draining rate:

AWSIoTPythonSDK.MQTTLib.AWSIoTMQTTClient.configureDrainingFrequency(frequencyInHz)

If no configOfflinePublishQueue or configureDrainingFrequency is called, the following default configuration for offline request queueing and draining will be performed on the initialization:

offlinePublishQueueSize = 20
dropBehavior = DROP_NEWEST
drainingFrequency = 2Hz

Before the draining process is complete, any new publish/subscribe/unsubscribe request within this time period will be added to the queue. Therefore, the draining rate should be higher than the normal request rate to avoid an endless draining process after reconnect.

The disconnect event is detected based on PINGRESP MQTT packet loss. Offline request queueing will not be triggered until the disconnect event is detected. Configuring a shorter keep-alive interval allows the client to detect disconnects more quickly. Any QoS0 publish, subscribe and unsubscribe requests issued after the network failure and before the detection of the PINGRESP loss will be lost.

Persistent/Non-Persistent Subscription

Device shadow operations are built on top of the publish/subscribe model for the MQTT protocol, which provides an asynchronous request/response workflow. Shadow operations (Get, Update, Delete) are sent as requests to AWS IoT. The registered callback will be executed after a response is returned. In order to receive responses, the client must subscribe to the corresponding shadow response topics. After the responses are received, the client might want to unsubscribe from these response topics to avoid getting unrelated responses for charges for other requests not issued by this client.

The SDK provides a persistent/non-persistent subscription selection on the initialization of a device shadow. Developers can choose the type of subscription workflow they want to follow.

For a non-persistent subscription, you will need to create a device shadow like this:

nonPersistentSubShadow = myShadowClient.createShadowHandlerWithName("NonPersistentSubShadow", False)

In this case, the request to subscribe to accepted/rejected topics will be sent on each shadow operation. After a response is returned, accepted/rejected topics will be unsubscribed to avoid getting unrelated responses.

For a persistent subscription, you will need to create a device shadow like this:

persistentSubShadow = myShadowClient.createShadowHandlerWithName("PersistentSubShadow", True)

In this case, the request to subscribe to the corresponding accepted/rejected topics will be sent on the first shadow operation. For example, on the first call of shadowGet API, the following topics will be subscribed to on the first Get request:

$aws/things/PersistentSubShadow/shadow/get/accepted
$aws/things/PersistentSubShadow/shadow/get/rejected

Because it is a persistent subscription, no unsubscribe requests will be sent when a response is returned. The SDK client is always listening on accepted/rejected topics.

In all SDK examples, PersistentSubscription is used in consideration of its better performance.

SSL Ciphers Setup

If custom SSL Ciphers are required for the client, they can be set when configuring the client before starting the connection.

To setup specific SSL Ciphers:

myAWSIoTMQTTClient.configureCredentials(rootCAPath, privateKeyPath, certificatePath, Ciphers="AES128-SHA256")

Examples

BasicPubSub

This example demonstrates a simple MQTT publish/subscribe using AWS IoT. It first subscribes to a topic and registers a callback to print new messages and then publishes to the same topic in a loop. New messages are printed upon receipt, indicating the callback function has been called.

Instructions

Run the example like this:

# Certificate based mutual authentication
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -w
# Customize client id and topic
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -t <topic>
# Customize the message
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -t <topic> -M <message>
# Customize the port number
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>
# change the run mode to subscribe or publish only (see python basicPubSub.py -h for the available options)
python basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -m <mode>

Source

The example is available in samples/basicPubSub/.

BasicPubSub with Amazon Cognito Session Token

This example demonstrates a simple MQTT publish/subscribe using an Amazon Cognito Identity session token. It uses the AWS IoT Device SDK for Python and the AWS SDK for Python (boto3). It first makes a request to Amazon Cognito to retrieve the access ID, the access key, and the session token for temporary authentication. It then uses these credentials to connect to AWS IoT and communicate data/messages using MQTT over Websocket, just like the BasicPubSub example.

Instructions

To run the example, you will need your Amazon Cognito identity pool ID and allow unauthenticated identities to connect. Make sure that the policy attached to the unauthenticated role has permissions to access the required AWS IoT APIs. For more information about Amazon Cognito, see here.

Run the example like this:

python basicPubSub_CognitoSTS.py -e <endpoint> -r <rootCAFilePath> -C <CognitoIdentityPoolID>
# Customize client id and topic
python basicPubsub_CognitoSTS.py -e <endpoint> -r <rootCAFilePath> -C <CognitoIdentityPoolID> -id <clientId> -t <topic>

Source

The example is available in samples/basicPubSub/.

BasicPubSub Asynchronous version

This example demonstrates a simple MQTT publish/subscribe with asynchronous APIs using AWS IoT. It first registers general notification callbacks for CONNACK reception, disconnect reception and message arrival. It then registers ACK callbacks for subscribe and publish requests to print out received ack packet ids. It subscribes to a topic with no specific callback and then publishes to the same topic in a loop. New messages are printed upon reception by the general message arrival callback, indicating the callback function has been called. New ack packet ids are printed upon reception of PUBACK and SUBACK through ACK callbacks registered with asynchronous API calls, indicating that the the client received ACKs for the corresponding asynchronous API calls.

Instructions

Run the example like this:

# Certificate based mutual authentication
python basicPubSubAsync.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python basicPubSubAsync.py -e <endpoint> -r <rootCAFilePath> -w
# Customize client id and topic
python basicPubSubAsync.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -t <topic>
# Customize the port number
python basicPubSubAsync.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>

Source

The example is available in samples/basicPubSub/.

BasicPubSub with API invocation in callback ___________

This example demonstrates the usage of asynchronous APIs within callbacks. It first connects to AWS IoT and subscribes to 2 topics with the corresponding message callbacks registered. One message callback contains client asynchronous API invocation that republishes the received message from <topic> to <topic>/republish. The other message callback simply prints out the received message. It then publishes messages to <topic> in an infinite loop. For every message received from <topic>, it will be republished to <topic>/republish and be printed out as configured in the simple print-out message callback. New ack packet ids are printed upon reception of PUBACK and SUBACK through ACK callbacks registered with asynchronous API calls, indicating that the the client received ACKs for the corresponding asynchronous API calls.

Instructions

Run the example like this:

# Certificate based mutual authentication
python basicPubSub_APICallInCallback.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python basicPubSub_APICallInCallback.py -e <endpoint> -r <rootCAFilePath> -w
# Customize client id and topic
python basicPubSub_APICallInCallback.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -t <topic>
# Customize the port number
python basicPubSub_APICallInCallback.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>

Source

The example is available in samples/basicPubSub/.

BasicShadow

This example demonstrates the use of basic shadow operations (update/delta). It has two scripts, basicShadowUpdater.py and basicShadowDeltaListener.py. The example shows how an shadow update request triggers delta events.

basicShadowUpdater.py performs a shadow update in a loop to continuously modify the desired state of the shadow by changing the value of the integer attribute.

basicShadowDeltaListener.py subscribes to the delta topic of the same shadow and receives delta messages when there is a difference between the desired and reported states.

Because only the desired state is being updated by basicShadowUpdater, a series of delta messages that correspond to the shadow update requests should be received in basicShadowDeltaListener.

Instructions

Run the example like this:

First, start the basicShadowDeltaListener:

# Certificate-based mutual authentication
python basicShadowDeltaListener.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python basicShadowDeltaListener.py -e <endpoint> -r <rootCAFilePath> -w
# Customize the port number
python basicShadowDeltaListener.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>

Then, start the basicShadowUpdater:

# Certificate-based mutual authentication
python basicShadowUpdater.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python basicShadowUpdater.py -e <endpoint> -r <rootCAFilePath> -w
# Customize the port number
python basicShadowUpdater.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>

After the basicShadowUpdater starts sending shadow update requests, you should be able to see corresponding delta messages in the basicShadowDeltaListener output.

Source

The example is available in samples/basicShadow/.

ThingShadowEcho

This example demonstrates how a device communicates with AWS IoT, syncing data into the device shadow in the cloud and receiving commands from another app. Whenever there is a new command from the app side to change the desired state of the device, the device receives this request and applies the change by publishing it as the reported state. By registering a delta callback function, users will be able to see this incoming message and notice the syncing of the state.

Instructions

Run the example like this:

# Certificate based mutual authentication
python ThingShadowEcho.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>
# MQTT over WebSocket
python ThingShadowEcho.py -e <endpoint> -r <rootCAFilePath> -w
# Customize client Id and thing name
python ThingShadowEcho.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -n <thingName>
# Customize the port number
python ThingShadowEcho.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -p <portNumber>

Now use the AWS IoT console or other MQTT client to update the shadow desired state only. You should be able to see the reported state is updated to match the changes you just made in desired state.

Source

The example is available in samples/ThingShadowEcho/.

JobsSample

This example demonstrates how a device communicates with AWS IoT while also taking advantage of AWS IoT Jobs functionality. It shows how to subscribe to Jobs topics in order to recieve Job documents on your device. It also shows how to process those Jobs so that you can see in the AWS IoT console which of your devices have received and processed which Jobs. See the AWS IoT Device Management documentation here for more information on creating and deploying Jobs to your fleet of devices to facilitate management tasks such deploying software updates and running diagnostics.

Instructions

First use the AWS IoT console to create and deploy Jobs to your fleet of devices.

Then run the example like this:

# Certificate based mutual authentication
python jobsSample.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -n <thingName>
# MQTT over WebSocket
python jobsSample.py -e <endpoint> -r <rootCAFilePath> -w -n <thingName>
# Customize client Id and thing name
python jobsSample.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -id <clientId> -n <thingName>
# Customize the port number
python jobsSample.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -n <thingName> -p <portNumber>

Source

The example is available in samples/jobs/.

BasicDiscovery

This example demonstrates how to perform a discovery process from a Greengrass Aware Device (GGAD) to obtain the required connectivity/identity information to connect to the Greengrass Core (GGC) deployed within the same group. It uses the discovery information provider to invoke discover call for a certain GGAD with its thing name. After it gets back a success response, it picks up the first GGC and the first set of identity information (CA) for the first group, persists it locally and iterates through all connectivity info sets for this GGC to establish a MQTT connection to the designated GGC. It then publishes messages to the topic, which, on the GGC side, is configured to route the messages back to the same GGAD. Therefore, it receives the published messages and invokes the corresponding message callbacks.

Note that in order to get the sample up and running correctly, you need:

1. Have a successfully deployed Greengrass group.
2. Use the certificate and private key that have been deployed with the group for the GGAD to perform discovery process.
3. The subscription records for that deployed group should contain a route that routes messages from the targeted GGAD to itself via a dedicated MQTT topic.
4. The deployed GGAD thing name, the deployed GGAD certificate/private key and the dedicated MQTT topic should be used as the inputs for this sample.

Run the sample like this:

python basicDiscovery.py -e <endpoint> -r <IoTRootCAFilePath> -c <certFilePath> -k <privateKeyFilePath> -n <GGADThingName> -t <RoutingTopic>

If the group, GGC, GGAD and group subscription/routes are set up correctly, you should be able to see the sample running on your GGAD, receiving messages that get published to GGC by itself.

API Documentation

You can find the API documentation for the SDK here.

License

This SDK is distributed under the Apache License, Version 2.0, see LICENSE.txt and NOTICE.txt for more information.

Support

If you have technical questions about the AWS IoT Device SDK, use the AWS IoT Forum. For any other questions about AWS IoT, contact AWS Support.

aws-iot-device-sdk-python's Issues

Document or give better errors for SSL issues

Hi,
I am trying to do a simple publish, but the connect() function gives me

[12/Jul/2016 12:49:39] ERROR  [SSL] PEM lib (_ssl.c:2581)

My guess is this is related to not using the proper OpenSSL, but it looks good.

Running on a Mac, with Python 2.7.11, and SSL Version OpenSSL 1.0.2h 3 May 2016

I have another equivalent node.js script and that works, so I know the certificates and AWS is ok. Any idea?

The script is:

    myMQTTClient = AWSIoTMQTTClient('0000-10000000')  # Name of my Thing on AWS IoT console
    # For TLS mutual authentication
    myMQTTClient.configureEndpoint('XXXXXXXXXXXXX.iot.us-east-1.amazonaws.com', 8883)
    myMQTTClient.configureCredentials(
        './certs/root-CA.crt',
        './certs/0000-10000000-certificate.pem.crt',
        './certs/0000-10000000-private.pem.key'
    )
    myMQTTClient.configureOfflinePublishQueueing(-1)  # Infinite offline Publish queueing
    myMQTTClient.configureDrainingFrequency(2)  # Draining: 2 Hz
    myMQTTClient.configureConnectDisconnectTimeout(10)  # 10 sec
    myMQTTClient.configureMQTTOperationTimeout(5)  # 5 sec

    try:
        myMQTTClient.connect()
    except Exception as e:
        logger.error(str(e))
        sys.exit(1)

Also, the following POHO script works

awshost = "XXXXXXXXXXX.iot.us-east-1.amazonaws.com"
awsport = 8883
caPath = "./certs/root-CA.crt"
certPath = "./certs/0000-10000000-certificate.pem.crt"
keyPath = "./certs/0000-10000000-private.pem.key"

mqttc.tls_set(caPath,
              certfile=certPath,
              keyfile=keyPath,
              cert_reqs=ssl.CERT_REQUIRED,
              tls_version=ssl.PROTOCOL_TLSv1_2,
              ciphers=None)

mqttc.connect(awshost, awsport, keepalive=60)

thread loops on _packet_read

Almost same as issue(#24), but the difference is that the problem happens not exactly when 24 hours after the process started, maybe 1 hour or sometimes couple hours. And when it's happended, it is unable to reconnect to the AWS IoT broker because the disconnect message in packet-queue has no chance to be send. Now i don't know how it happens. Application was ran on CentOS6.5 and python version is 2.7.8.

Python Paho Mqtt error cannot connect to thing on AWS IoT

I am facing problems, when trying to connect to AWS IoT via python. My code worked previously so there isn't anything I can change there, I suspect there is something on AWS side probably ? As you can see at the last line I am having SSL problems.
Traceback (most recent call last): File "flint.py", line 44, in <module> mqttc.connect("a2yye602yte99c.iot.us-west-2.amazonaws.com", port=8883) #AWS IoT service hostname and portno File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 686, in connect return self.reconnect() File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 821, in reconnect ciphers=self._tls_ciphers) File "/usr/lib/python2.7/ssl.py", line 891, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 566, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:581)

Also yesterday when the error popped in for the first time, there was some notification on AWS IoT console :

createShadowHandlerWithName returns NoneType.

 def defineIOTButtonShadow():
        print ("Begin shadow client init")
        myShadowClient = AWSIoTMQTTShadowClient("basicPubSub")
        myShadowClient.configureEndpoint(endpoint, 8883)
        myShadowClient.configureCredentials("root-CA.crt", "ArunavLaptop.private.key", "ArunavLaptop.cert.pem")
        myShadowClient.connect()
        time.sleep(1)
        myDeviceShadow = myShadowClient.createShadowHandlerWithName("Bot", True)
        time.sleep(1)
        print ("End shadow client init")

After this myDeviceShadow.shadowGet(cycleThroughDashBoardsCallback, 5) breaks because myDeviceShadow is NoneType.

Library does not support Official Specs

Cannot implement "Detecting a Thing is Connected" from the official documentation
here due to missing on_connect and on_reconnect callback.

infinite loop

Hi SDK team,
I have a python application that runs inside a container. This application communicates with iot to exchange information. The following is the configuration inherent in the library.

self.myAWSIoTMQTTClient = AWSIoTMQTTClient (thron_key, useWebsocket = True)
self.myAWSIoTMQTTClient.configureEndpoint (host, 443)
self.myAWSIoTMQTTClient.configureCredentials (rootCAPath)

self.myAWSIoTMQTTClient.configureAutoReconnectBackoffTime (1, 128, 20)
self.myAWSIoTMQTTClient.configureOfflinePublishQueueing (-1) 
self.myAWSIoTMQTTClient.configureDrainingFrequency (2) 
self.myAWSIoTMQTTClient.configureConnectDisconnectTimeout (10) 
self.myAWSIoTMQTTClient.configureMQTTOperationTimeout (5) 

iot.myAWSIoTMQTTClient.subscribe ( "$ aws / things / thing_name / shadow / update / delta", 1, function_that_makes_things)

function_that_makes_things (client, userdata, message):
iot.myAWSIoTMQTTClient.publish ( "$ aws / things / thing_name / shadow / update", json.dumps (payload, separators = ( ',', ':')), 1)

The application works for hours without any problem, do not consume virtually no resources. But sometimes it freezes, saturate the CPU and unresponsive receives messages from iot. Interrupting the application I get this message:

Exception in thread Thread-1 (Most Likely raised During shutdown interpreter):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 754, in run
  File "/usr/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 2345, in _thread_main
  File "/usr/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 1317, in loop_forever
  File "/usr/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 862, looped
  File "/usr/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 1124, in loop_read
  File "/usr/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 1464, in _packet_read
<Type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

It seems that the problem is within the PAHO library. This problem seems to have been solved by a bugfix in version 1.1 [https://github.com/eclipse/paho.mqtt.python/blob/master/ChangeLog.txt]
The place seems to be the version 1.0, are no plans to upgrade to 1.2 or 1.1? Or am I using the library in the wrong way?

ThingShadowEcho doesn't echo shadow changes made in the console

I'm having difficulty getting the ThingShadowEcho sample to work.

The logging output shows a successful connection to my endpoint and successfully subscribing to the topic delta, but it doesn't echo back changes to the shadow that I make via the AWS IoT console.

If the MQTT topic for my updates to my thing shadow is $aws/things/Pi_Py_SDK/shadow/update, what should I change this following line to in order to connect to this shadow and be able to receive shadow updates made via the console?

Bot = myAWSIoTMQTTShadowClient.createShadowHandlerWithName("Bot", True)

Hope you can help?

AWS IoT HTTP POST Request in Python?

Is there a procedure to send HTTP POST message without hard-coding endpoint and topic.
I am able to give the message as a variable but not AWS endpoint and Topic name.

This code is working absolutely fine:

import requests

caPath = "aws-iot-rootCA.crt"
certPath = "cert.pem.crt"
keyPath = "privkey.pem.crt"

parameters = (
    ('qos', '1'),
)
payload= """{
  "message": "Hello"
}"""

r = requests.post('https://***********.us-west-2.amazonaws.com:8443/topics/TopicName',
    params=parameters,,data=payload,
    cert=(certPath,keyPath,caPath))

But how to give topic name and AWS endpoint as variables?

How to subscribe to multiple topics in aws iot?

I have tried :
myAWSIoTMQTTClient.subscribe(['Topic1,'Topic2','Topic3'], 1, customCallback)
myAWSIoTMQTTClient.subscribe('Topic1,'Topic2','Topic3', 1, customCallback)

None of them seem to work.
I am able to successfully subscribe to 1 topic but not multiple topics.

Excepts if callback methods are part of class

If callback methods are referenced from inside of a class (using self) the SDK excepts with the message Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-6, started daemon -1267731360)>>.

issue running basicPubSub.py

Hi,

I am trying to ran basicPubSub.py from Amzon EC2 machine, after giving path for root CA, Cerificates & private key file path while running file i get below error:

2016-11-19 23:58:52,719 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Client init.
2016-11-19 23:58:52,719 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: basicPubSub
2016-11-19 23:58:52,719 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQTTv3.1.1
2016-11-19 23:58:52,719 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Paho MQTT Client callbacks.
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore init.
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile from: /home/ec2-user/certificates/VeriSign-Class+3-Public-Primary-Certification-Authority-G5.pem
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key from: /home/ec2-user/certificates/3b7c2ac37b-private.pem.key
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert from: /home/ec2-user/certificates/3b7c2ac37b-certificate.pem.crt
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: baseReconnectTime = 1 sec
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: maximumReconnectTime = 32 sec
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: minimumConnectTime = 20 sec
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: queueSize = -1
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: dropBehavior = Drop Newest
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for draining interval: 0.5 sec
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum connect/disconnect timeout to be 10 second.
2016-11-19 23:58:52,720 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum MQTT operation timeout to be 5 second
2016-11-19 23:58:52,721 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
Traceback (most recent call last):
File "basicPubSub.py", line 133, in
myAWSIoTMQTTClient.connect()
File "/usr/local/lib/python2.7/site-packages/AWSIoTPythonSDK/MQTTLib.py", line 403, in connect
return self._mqttCore.connect(keepAliveIntervalSecond)
File "/usr/local/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 290, in connect
self._pahoClient.connect(self._host, self._port, keepAliveInterval) # Throw exception...
File "/usr/local/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
return self.reconnect()
File "/usr/local/lib/python2.7/site-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 777, in reconnect
sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
File "/usr/lib64/python2.7/socket.py", line 557, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Please suggest. Thanks

Pub/Sub in 2 different Accounts.

I was able to successfully implement the basicPubSub example but I see that in the example, you are publishing from the same AWS Account and Subscribing with the same account.
How should I give my Policy statement if I would be Publishing messages from Account A and subscribing messages from Account B?
So, basically I want to know what is the change I need to perform to Pub/Sub across 2 different accounts.

Thanks
Bharath

Client disconnect after 24 hours without raising exception

When using the client with web socket, the mqtt client and shadow stop receiving updates from the service (broker or shadow updates).
I know that the web socket closes after 24 hours as listed in the aws-iot limitation.
However, the client is not raising any exception nor is it exposing any socket disconnect callback to handle this case.

publishQueueDisabledException

Hi again, I've just tried starting up my code that was working after I fixed my issue in issue 4, but it's now throwing this:

Traceback (most recent call last):
  File "./awstemp.py", line 38, in <module>
    myDeviceShadow.shadowUpdate(tempreading, None, 5)
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/shadow/deviceShadow.py", line 374, in shadowUpdate
    self._shadowManagerHandler.basicShadowPublish(self._shadowName, "update", JSONPayloadWithToken)
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/shadow/shadowManager.py", line 65, in basicShadowPublish
    self._mqttCoreHandler.publish(currentShadowAction.getTopicGeneral(), srcPayload, 0, False)
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 350, in publish
    raise publishQueueDisabledException()
core.exception.AWSIoTExceptions.publishQueueDisabledException

Is this something I have done? my certificate is "ACTIVE".. shadow status says it's in sync (although 4 days since an update). I'm not sure what I may have done to break this.

This is in the ap-southeast-2 region.

Thanks,
xelfer

Can't use same string for Client ID and shadowName while using shadowRegisterDeltaCallback

While I running the sample of ThingShadowEcho, I had set "Raspberry" for both Client ID and shadowName like following:
`

myAWSIoTMQTTShadowClient = AWSIoTMQTTShadowClient("Raspberry")

Bot = myAWSIoTMQTTShadowClient.createShadowHandlerWithName("Raspberry", True)

`
Then the customShadowCallback_Delta function will not works. But for the sample of basicShadow is works for me.

This SDK vs Boto3

I'm just wondering about a general issue, maybe I'm missing something obvious.

I can send mqtt publish events using boto3. What are the advantages of this SDK vs using Boto3?

From the readme I understand there are advanced options available using this SDK which may be not present in Boto3. And on the other hand I suspect this SDK cannot manage/create certificates or create things like boto3 can.

Also I'm wondering if this SDK gets some long term, active development?

Confused By Method Naming & Docs

It took me quite some time and digging through an already closed issue to figure out how to register a specific "Thing" shadow to the device shadow client (and maybe I still don't understand).

From what I do understand, I needed to put my "Thing" name here, ThingShadowEcho.py#L149, in place of 'Bot', because the shadow name IS the "Thing" name.

I propose that the function be named something like createShadowHandlerForThing() instead of createShadowHandlerWithName(), as it more explicitly implies you are creating a handler for a device (thing), instead of creating, and seemingly arbitrarily naming, a handler for shadows.

That or update the docs to be a little more clear? I didn't see anywhere in the AWS console that indicated an explicit 'Shadow Name', which is why I think I got a bit confused here.

CERTIFICATE_VERIFY_FAILED on Own CA

For my solution I want to use X.509 device certificates, as illustrated in http://docs.aws.amazon.com/iot/latest/developerguide/device-certs-your-own.html

When I'm trying to connect I get the following error:
SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)')

I'm running:
certifi (2017.1.23)
cryptography (1.7.2)
AWSIoTPythonSDK (1.1.1)
boto3 (1.4.4)
botocore (1.5.11)

On python 2.7.13 with OpenSSL 1.0.2j. I got X.509 working with via configure a device on the IOT section. But I want to generate the certificates.

The Root CA is Generated via:

openssl genrsa -out rootCA.key 2048
root_subject="/C=NL/ST=City/L=City/O=Organisation/OU=Part/CN=Part"
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem -subj "$root_subject"

#Registration Code
aws_registration_code=$(aws iot get-registration-code)
registration_code=$(python -c "from sys import argv; import json;\
script, data = argv;\
data_dict = json.loads(data);\
print data_dict['registrationCode']" "$aws_registration_code")
echo "registration code: $registration_code"

#Key Pair
openssl genrsa -out verificationCert.key 2048
#CSR
csr_subject="/C=NL/ST=City/L=City/O=Organisation/OU=Part/CN=$registration_code"
echo $csr_subject
openssl req -new -key verificationCert.key -out verificationCert.csr -subj "$csr_subject"
#private key verification cert 
openssl x509 -req -in verificationCert.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out verificationCert.pem -days 500 -sha256

#register with AWS
reg_response=$(aws iot register-ca-certificate --ca-certificate file://rootCA.pem --verification-cert file://verificationCert.pem)
cert_id=$(python -c "from sys import argv; import json; script, data = argv; item = json.loads(data); print item['certificateId']" "$reg_response")
aws iot update-ca-certificate --certificate-id $cert_id --new-status ACTIVE

The Device CA:

def create_certificate_on_disk(self):
        ''' Check if Certificate folder & certificates are available otherwise generate '''
        #check folder
        try:
            os.makedirs(self.device_cert_folder)
        except OSError:
            pass

        if not os.path.isfile(self.device_pem):
            #device key
            call(["openssl", "genrsa", "-out", self.device_key, "2048"])
            #device csr
            device_subject = "/C=NL/ST=City/L=City/O=Organisation/OU=Part/CN=%s" % self.identifier
            call(["openssl", "req", "-new", "-key", self.device_key, "-out", self.device_csr, "-subj", device_subject])
            #device pem
            call(["openssl", "x509", "-req", "-in", self.device_csr, "-CA", self.root_pem, "-CAkey", "root_ca/rootCA.key", "-CAcreateserial", "-out", self.device_pem, "-days", "500", "-sha256"])
        else:
            raise Exception("pem already created")

The device cert is then registered and set Active.

The debug I get is:

2017-02-15 16:54:21,461 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Client init.
2017-02-15 16:54:21,461 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: basicPubSub
2017-02-15 16:54:21,461 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQTTv3.1.1
2017-02-15 16:54:21,461 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Paho MQTT Client callbacks.
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore init.
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile from: root_ca/rootCA.pem
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key from: device_certificates/deviceCert.key
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert from: device_certificates/deviceCert.pem
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: baseReconnectTime = 1 sec
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: maximumReconnectTime = 32 sec
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: minimumConnectTime = 20 sec
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: queueSize = -1
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: dropBehavior = Drop Newest
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for draining interval: 0.5 sec
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum connect/disconnect timeout to be 10 second.
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum MQTT operation timeout to be 5 second
2017-02-15 16:54:21,462 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
2017-02-15 16:54:21,544 - AWSIoTPythonSDK.core - INFO - iOT Server error: SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)')

I'm a missing a step?

on_publish() callback

Hi,

on_publish() callback is not available in mqttCore.py though it is available in underlying paho client. Why is that ?

Is there any way so that I can access underlying paho client's on_publish ?

Thanks,
Venkat

PB with library and Raspberry Pi

Hello

Your BasicPubSub.py sample file does not work with Raspberry Pi (latest Raspbian version and components), while it works under Windows 10 / Python 2 & 3.
I got a timeout during the mutual authentication (see the traces from another user in the support thread https://forums.aws.amazon.com/thread.jspa?threadID=236865&tstart=0 ).
Has anyone ever succeeded in running this library from a RPI platform ?

Register callback on Connect

I want to be able to register a callback when connection is established.
For example to send an update on the reported state on startup.

It seems this is possible with the node js sdk like demonstrated in the webinar.

I saw that the mqtt paho lib allow a registration of a callback.

import paho.mqtt.client as mqtt
def on_connect(mqttc, obj, flags, rc):
[...]
mqttc = mqtt.Client(client_id="mqtt-test")

mqttc.on_connect = on_connect

Failing basic start with connect_device_package on Raspbian

Hello,

I'm using a Raspberry Pi 3 Rev B - it's Internet appears to work just fine as a I can ping different servers, run updates, etc.

When I tried to run start.sh, I get the following problem:

Runing pub/sub sample application...
2017-04-17 18:55:27,030 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Client init.
2017-04-17 18:55:27,031 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: basicPubSub
2017-04-17 18:55:27,031 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQTTv3.1.1
2017-04-17 18:55:27,031 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Paho MQTT Client callbacks.
2017-04-17 18:55:27,032 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore init.
2017-04-17 18:55:27,032 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile from: root-CA.crt
2017-04-17 18:55:27,032 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key from: Louis-TestPi.private.key
2017-04-17 18:55:27,033 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert from: Louis-TestPi.cert.pem
2017-04-17 18:55:27,033 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: baseReconnectTime = 1 sec
2017-04-17 18:55:27,033 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: maximumReconnectTime = 32 sec
2017-04-17 18:55:27,033 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: minimumConnectTime = 20 sec
2017-04-17 18:55:27,034 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: queueSize = -1
2017-04-17 18:55:27,034 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: dropBehavior = Drop Newest
2017-04-17 18:55:27,034 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for draining interval: 0.5 sec
2017-04-17 18:55:27,035 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum connect/disconnect timeout to be 10 second.
2017-04-17 18:55:27,035 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum MQTT operation timeout to be 5 second
2017-04-17 18:55:27,036 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
Traceback (most recent call last):
  File "aws-iot-device-sdk-python/samples/basicPubSub/basicPubSub.py", line 133, in <module>
    myAWSIoTMQTTClient.connect()
  File "/usr/local/lib/python3.4/dist-packages/AWSIoTPythonSDK/MQTTLib.py", line 403, in connect
    return self._mqttCore.connect(keepAliveIntervalSecond)
  File "/usr/local/lib/python3.4/dist-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 290, in connect
    self._pahoClient.connect(self._host, self._port, keepAliveInterval)  # Throw exception...
  File "/usr/local/lib/python3.4/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
    return self.reconnect()
  File "/usr/local/lib/python3.4/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 798, in reconnect
    ciphers=self._tls_ciphers)
  File "/usr/lib/python3.4/ssl.py", line 887, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python3.4/ssl.py", line 510, in __init__
    self._context.load_verify_locations(ca_certs)
ssl.SSLError: unknown error (_ssl.c:2736)

Any idea what's going on? I tried in Python 2.7 and have the exact same error message.

publishQueueDisabledException

Hi,

Why can't I disable offlinePublishQueue ?

When I disable it I'm getting publishQueueDisabledException

Thanks,
Venkat

Hangs on connect

I can connect with the node.js sdk using the same certs and endpoint. However, with this SDK it hangs on .connect().

Code:

# For certificate based connection
myShadowClient = AWSIoTMQTTShadowClient("RWRaspberryPi")
# For Websocket connection
# myMQTTClient = AWSIoTMQTTClient("myClientID", useWebsocket=True)
# Configurations
# For TLS mutual authentication
myShadowClient.configureEndpoint("a3pzb0u64rd1gp.iot.us-east-1.amazonaws.com", 8883)
# For Websocket
# myShadowClient.configureEndpoint("YOUR.ENDPOINT", 443)
myShadowClient.configureCredentials("/Users/richmond.watkins/Documents/HomeSecurity/certs/root-ca.crt", "/Users/richmond.watkins/Documents/HomeSecurity/certs/9e95daa236-private.pem.key", "/Users/richmond.watkins/Documents/HomeSecurity/certs/9e95daa236-certificate.pem.crt")
# For Websocket, we only need to configure the root CA
# myShadowClient.configureCredentials("YOUR/ROOT/CA/PATH")
myShadowClient.configureConnectDisconnectTimeout(10)  # 10 sec
myShadowClient.configureMQTTOperationTimeout(5)  # 5 sec


myShadowClient.connect()
# Create a device shadow instance using persistent subscription
myDeviceShadow = myShadowClient.createShadowHandlerWithName("RWRaspberryPi", True)
# Shadow operations
myDeviceShadow.shadowGet(customCallback, 5)
myDeviceShadow.shadowUpdate(myJSONPayload, customCallback, 5)
myDeviceShadow.shadowDelete(customCallback, 5)
myDeviceShadow.shadowRegisterDeltaCallback(customCallback)
myDeviceShadow.shadowUnregisterDeltaCallback()

"No Access Key/KeyID Error"

Hi

I'm getting a raise ValueError("No Access Key/KeyID Error") ValueError: No Access Key/KeyID Error
but i have my key configured by aws configure.

No handlers could be found for logger "core.util.sigV4Core"
Traceback (most recent call last):
File "./awshome.py", line 53, in
iot = createIoT()
File "./awshome.py", line 43, in createIoT
iot.connect()
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/MQTTLib.py", line 710, in connect
return self._AWSIoTMQTTClient.connect(keepAliveIntervalSecond)
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/MQTTLib.py", line 355, in connect
return self._mqttCore.connect(keepAliveIntervalSecond)
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 282, in connect
self._pahoClient.connect(self._host, self._port, keepAliveInterval) # Throw exception...
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
return self.reconnect()
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 788, in reconnect
self._ssl = wssCore.securedWebsocketCore(rawSSL, self._host, self._port, self._AWSAccessKeyIDCustomConfig, self._AWSSecretAccessKeyCustomConfig, self._AWSSessionTokenCustomConfig) # Overeride the _ssl socket
File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/paho/securedWebsocket/securedWebsocketCore.py", line 170, in init
raise ValueError("No Access Key/KeyID Error")
ValueError: No Access Key/KeyID Error

aws configure
AWS Access Key ID [*************_OVGA]:
AWS Secret Access Key [_*************Rzja]:
Default region name [us-east-1]:
Default output format [None]:

Waiting Infinitely at Connection type TLSv1.2 Mutual Authentication

Hi please find the log here http://pastebin.com/974DuwFF

I'm trying MQTT in basicPubSub.py

I'm using this certificate provided in AWS documentation as rootCA
https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem

It is waiting infinitely at this point:
AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication

AWSIoTMQTTClient.connect() Occasionally hangs forever

I am working on an python app that gets stuck somewhere in the MQTT.Connect() Method, but only after a while. it might work 1000 times than hang. I decided to create a simple test by changing basicPubSub.py to do something similar and it still hangs. Here is what i've done to basicPubSub:

Everything above the While loop is the same (except my creds).

while True:
    print "waiting"
    time.sleep(3)

    data = { "xg": 2.5, "xalarm": 3, "linkqdbm": -25} 
    JSONPayload = dumps(data)  #json.dumps

    myAWSIoTMQTTClient.connect() #this will hang eventually and never throw an exception or continue or update the logger
    myAWSIoTMQTTClient.publish("Sensor/", JSONPayload, 0)  #this was orgonally 1; We probably want 1 for final app but I dont think it matters here
    myAWSIoTMQTTClient.disconnect()

Here is what the output looks like... and what I get when I keyboard interrupt the hang. I waited like an hour before doing so.

waiting
2017-04-13 19:30:11,718 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
2017-04-13 19:30:12,208 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Connect result code 0
2017-04-13 19:30:12,211 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connected to AWS IoT.
2017-04-13 19:30:12,211 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Connect time consumption: 70.0ms.
2017-04-13 19:30:12,212 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Try to put a publish request 1998 in the TCP stack.
2017-04-13 19:30:12,212 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Publish request 1998 succeeded.
2017-04-13 19:30:12,213 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Disconnect result code 0
2017-04-13 19:30:12,223 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Disconnected.
2017-04-13 19:30:12,224 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Disconnect time consumption: 10.0ms.
waiting
2017-04-13 19:30:15,227 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication



^CTraceback (most recent call last):
  File "TestScripts/basicPubSub.py", line 108, in <module>
    myAWSIoTMQTTClient.connect()
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/MQTTLib.py", line 403, in connect
    return self._mqttCore.connect(keepAliveIntervalSecond)
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 290, in connect
    self._pahoClient.connect(self._host, self._port, keepAliveInterval)  # Throw exception...
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
    return self.reconnect()
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/protocol/paho/client.py", line 798, in reconnect
    ciphers=self._tls_ciphers)
  File "/usr/lib/python2.7/ssl.py", line 891, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
KeyboardInterrupt

I am using Python 2.7.9, OpenSSL 1.0.1t, and AWSIoTPythonSDK 1.1.1
Any thoughts on how to overcome this?

basicPubSub.py hangs forever at TLSv1.2 Mutual Authentication on RPi3

I'm trying to get AWSIoTPythonSDK 1.1.0 working on a Raspberry Pi 3 running Raspbian Jessie. It appears to have compatible versions of Python and OpenSSL installed, but I am getting an infinite hang on "TLSv1.2 Mutual Authentication" when I try to run basicPubSub.py.

It doesn't give much else in the way of output and I'd be very interested in suggestions to try and get it working.

I have created the public/private keypair and downloaded the root cert as described in the docs, and moved those to the RPi, along with the latest (commit 41ead94) SDK. I am running Python 2.7.9 and ssl.OPENSSL_VERSION reports OpenSSL 1.0.1t 3 May 2016, which ought to work per the README.

The command I'm running is (where MYENDPOINT and MYCERT are redactions, but represent actual values on my system):

python /etc/aws/python-sdk/samples/basicPubSub/basicPubSub.py  \
 -e MYENDPOINT.iot.us-east-1.amazonaws.com \   
 -r /etc/aws/deviceSDK/certs/rootCA.pem.crt \   
 -c /etc/aws/deviceSDK/certs/MYCERT-certificate.pem.crt \   
 -k /etc/aws/deviceSDK/certs/MYCERT-private.pem.key

I've checked all 3 file paths to make sure they're accessible. I have permissions set to 600 on the private key, in case that was causing OpenSSL to choke (didn't seem to matter). I've tried running as the usual "rpi" user and as root, just in case. The output never seems to vary, and can be viewed in this gist; it will hang with that showing until I cause a KeyboardInterrupt.

I've verified that using the same keyfiles and endpoint, I can use the Embedded C SDK to publish and subscribe from the same device, so I know that it is definitely not a network issue. The problem seems to be definitely related to the Python SDK.

Any suggestions of what to try? I would really like to use the Python SDK.

self._shadowSubscribeCallbackTable[srcActionName]("REQUEST TIME OUT", "timeout", srcToken)

I'm using your SDK to send some sensor data to my IoT device shadow. It usually runs fine but the past day or two I've received this error when I re-attach my screen session:

payload: { "state" : { "reported": { "temp": "19.5", "humid": "55.3" } } }
payload: { "state" : { "reported": { "temp": "19.4", "humid": "55.2" } } }
payload: { "state" : { "reported": { "temp": "19.5", "humid": "55.3" } } }
payload: { "state" : { "reported": { "temp": "19.5", "humid": "55.3" } } }
Exception in thread Thread-6818:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 1082, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/shadow/deviceShadow.py", line 203, in _timerHandler
    self._shadowSubscribeCallbackTable[srcActionName]("REQUEST TIME OUT", "timeout", srcToken)
TypeError: 'str' object is not callable

Any ideas what might be going on here?

My full source is here: https://github.com/xelfer/nickshouse/blob/master/awstemp.py

Thanks,
xelfer

Request and response strategy for AWS IOT MQTT?

Does anyone have example or info on how to do request and response strategy / RPC with AWS IOT over MQTT? I would love to use AWS IOT system but I need to send a command, wait for a process to finish and get the results. It looks like I need a request and response strategy or RPC over MQTT.

connectTimeoutException happened when running basicPubSub.py

Trying to run the sample basicPubSub.py with -r , -c ,-k , -id option.

connection timeout exception happened.

INFO- ConnectionType: TLSv1.2 Mutual Autentication
mqttCore -DEBUG - Disconnect result code 1
mqttCore - ERROR- connect timeout.

Let me know any pointers what could have went wrong?

Failing basic start with connect_device_package on Windows 10

Sorry to post another issue, but I'm failing at the exact same point on Windows 10.

From Powershell, I run start.ps1

Running pub/sub sample application...
2017-04-17 20:02:41,837 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Cl
2017-04-17 20:02:41,838 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: bas
2017-04-17 20:02:41,844 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQT
2017-04-17 20:02:41,845 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Pah
2017-04-17 20:02:41,849 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore ini
2017-04-17 20:02:41,850 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile
2017-04-17 20:02:41,851 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key fro
2017-04-17 20:02:41,853 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert fr
2017-04-17 20:02:41,860 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,861 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,865 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,865 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,866 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,867 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setti
2017-04-17 20:02:41,868 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum
2017-04-17 20:02:41,877 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum
2017-04-17 20:02:41,880 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection ty
Traceback (most recent call last):
  File "aws-iot-device-sdk-python\samples\basicPubSub\basicPubSub.py", line 133, in <mo
    myAWSIoTMQTTClient.connect()
  File "C:\Users\Louis Thiery\AppData\Local\Programs\Python\Python36-32\lib\site-packag
    return self._mqttCore.connect(keepAliveIntervalSecond)
  File "C:\Users\Louis Thiery\AppData\Local\Programs\Python\Python36-32\lib\site-packag
    self._pahoClient.connect(self._host, self._port, keepAliveInterval)  # Throw except
  File "C:\Users\Louis Thiery\AppData\Local\Programs\Python\Python36-32\lib\site-packag
    return self.reconnect()
  File "C:\Users\Louis Thiery\AppData\Local\Programs\Python\Python36-32\lib\site-packag
    self._sock.setblocking(0)
OSError: [WinError 10038] An operation was attempted on something that is not a socket```

When I Google the issue, it seems to be that the socket is closed too soon... but if so, this is a problem with the source of this project.

Best,
Louis

Expose LWT methods in AWS client objects

Paho client supports LWT, but they're not directly available in the AWS clients. I have to dig into private objects to get at it, like this:

AWSIoTMQTTShadowClient.getMQTTConnectio()._mqttCore._pahoClient.will_set

This should be directly supported, probably in AWSIoTMQTTClient

AWS/aws-iot-device-sdk-python

When trying to run a sample start script to connect a thing to AWS IoT platform I'm getting this import error:

Runing pub/sub sample application...
Traceback (most recent call last):
File "aws-iot-device-sdk-python/samples/basicPubSub/basicPubSub.py", line 18, in
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
ImportError: No module named AWSIoTPythonSDK.MQTTLib

I've attached the start script.

Any help would be greatly appreciated.
start.sh.zip

Python 3.6 socket error

Hi SDK team,
I upgrade Python to version 3.6 and got following traceback:

2017-01-18 18:33:20,806 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Paho MQTT Client init.
2017-01-18 18:33:20,807 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - ClientID: ThingShadowEcho
2017-01-18 18:33:20,808 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Protocol: MQTTv3.1.1
2017-01-18 18:33:20,808 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Register Paho MQTT Client callbacks.
2017-01-18 18:33:20,809 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - mqttCore init.
2017-01-18 18:33:20,810 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: queueSize = 0
2017-01-18 18:33:20,811 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for publish queueing: dropBehavior = Drop Newest
2017-01-18 18:33:20,812 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for draining interval: 0.1 sec
2017-01-18 18:33:20,812 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load CAFile from: VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem
2017-01-18 18:33:20,813 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Key from: my-private.pem.key
2017-01-18 18:33:20,814 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Load Cert from: my-certificate.pem.crt
2017-01-18 18:33:20,814 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: baseReconnectTime = 1 sec
2017-01-18 18:33:20,815 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: maximumReconnectTime = 32 sec
2017-01-18 18:33:20,816 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Custom setting for backoff timing: minimumConnectTime = 20 sec
2017-01-18 18:33:20,816 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum connect/disconnect timeout to be 10 second.
2017-01-18 18:33:20,817 - AWSIoTPythonSDK.core.protocol.mqttCore - DEBUG - Set maximum MQTT operation timeout to be 5 second
2017-01-18 18:33:20,819 - AWSIoTPythonSDK.core.protocol.mqttCore - INFO - Connection type: TLSv1.2 Mutual Authentication
Traceback (most recent call last):
  File "ThingShadowEcho.py", line 142, in <module>
    myAWSIoTMQTTShadowClient.connect()
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/MQTTLib.py", line 808, in connect
    return self._AWSIoTMQTTClient.connect(keepAliveIntervalSecond)
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/MQTTLib.py", line 403, in connect
    return self._mqttCore.connect(keepAliveIntervalSecond)
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/mqttCore.py", line 290, in connect
    self._pahoClient.connect(self._host, self._port, keepAliveInterval)  # Throw exception...
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/paho/client.py", line 655, in connect
    return self.reconnect()
  File "/home/alarm/code/aws-iot-device-sdk-python/AWSIoTPythonSDK/core/protocol/paho/client.py", line 809, in reconnect
    self._sock.setblocking(0)
OSError: [Errno 9] Bad file descriptor

The error is repeated on all samples from this repository. It seems that the problem is within the PAHO library. This problem seems to have been solved by a bugfix (issue).
As I understand in other Python 3 versions this code does not produce exception, but ssl-socket still in blocking mode.

Is there a limit on number of topics to be created in AWS IoT?

Hi,
I have used AWS IoT and have tried to connect it to my printer and it works fine, I want to scale it up-to 75 million printers with 1 topic for each printer.

I wanted to know is there a limit for number of Topics to be created per account. I did not find information regarding these in AWS IoT Service Limits documentation.

It's mentioned that you can scale up-to billions of devices in AWS IoT, I wanted to what are the best practices for that? (Didn't find any documentation regarding this)

So, for my particular case , I would like to know ho do I start to scale from 1 device to 75 million devices.

ImportError with Python 3

Hi SDK team,

I saw that 1.1.0 was released and upgraded from 1.0.1. However, an from ... import statement that was working under Python 3.5.1 is now throwing an ImportError. I've create a new virtualenv via pyenv and pip installed just the AWSIoTPythonSDK. When I try to do the from import:

$ python
Python 3.5.1 (default, Feb 16 2016, 16:56:48) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[redacted]/.pyenv/versions/3.5.1/envs/iot_test/lib/python3.5/site-packages/AWSIoTPythonSDK/MQTTLib.py", line 18, in <module>
    import core.protocol.mqttCore as mqttCore
ImportError: No module named 'core'
>>> 

It looks like the import statements in core/protocol/mqttCore have been changed from relative imports to absolute imports starting with the package root (AWSIoTPythonSDK).

Performing the same test with a clean 2.7.10 virtualenv operates as expected:

$ pip install AWSIoTPythonSDK
Collecting AWSIoTPythonSDK
  Using cached AWSIoTPythonSDK-1.1.0.tar.gz
Building wheels for collected packages: AWSIoTPythonSDK
  Running setup.py bdist_wheel for AWSIoTPythonSDK
  Stored in directory: /[redacted]/Library/Caches/pip/wheels/65/4c/8d/70428fd5ea7d7e201fc2b5e3f83ea57dfdc450304c064f4ea1
Successfully built AWSIoTPythonSDK
Installing collected packages: AWSIoTPythonSDK
Successfully installed AWSIoTPythonSDK-1.1.0
$ python
Python 2.7.10 (default, Dec  3 2015, 15:37:46) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
>>> 

Is there something I'm missing from an environment aspect in 3.5.1?

AWS IoT Reliability test?

I am doing a reliability test to evaluate my AWS IoT Client's reconnection time, once it gets disconnected from internet.
I connected my laptop with my Mobile Data (Hotspot) and I would like to turn on/off the Data.
So,my wifi status changes from:

1. Connected to
2. Connected but no internet.

I want to know is there a procedure to know this information.(WiFi connection Status).
So, I can log all the information and know the actual re-connection time my client takes.
(I am using Windows -10)

Publish Payload size of 128 KB

Hi,

In http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html, it is given that

"The payload for every PUBLISH message is limited to 128 KB. The AWS IoT service will reject messages larger than this size."

Will this SDK handle this ?

If message size is greater than 128 KB will it be split and published by the SDK ?

Cant publish stored data in memory when internet connection comes

When the internet connection get lost, aws iot sdk store objects in memory.However aws iot sdk does not publish stored data to mqtt topic when internet connection available.

2016-09-30 19:58:48,507 - core.protocol.mqttCore - INFO - Connected to AWS IoT.
2016-09-30 19:58:48,508 - core.protocol.mqttCore - DEBUG - Connect time consumption: 260.0ms.
2016-09-30 19:58:48,508 - core.protocol.mqttCore - DEBUG - Connect result code 0
2016-09-30 19:58:50,511 - core.protocol.mqttCore - DEBUG - Try to put a publish request 1 in the TCP stack.
2016-09-30 19:58:50,511 - core.protocol.mqttCore - DEBUG - Publish request 1 succeeded.
2016-09-30 19:59:08,509 - core.util.progressiveBackoffCore - DEBUG - stableConnection: Resetting the backoff time to: 1 sec.
2016-09-30 19:59:50,516 - core.protocol.mqttCore - DEBUG - Try to put a publish request 2 in the TCP stack.
2016-09-30 19:59:50,516 - core.protocol.mqttCore - DEBUG - Publish request 2 succeeded.
2016-09-30 20:00:50,571 - core.protocol.mqttCore - DEBUG - Try to put a publish request 3 in the TCP stack.
2016-09-30 20:00:50,571 - core.protocol.mqttCore - DEBUG - Publish request 3 succeeded.
2016-09-30 20:00:51,574 - core.protocol.mqttCore - DEBUG - Disconnect result code 1
2016-09-30 20:00:51,574 - core.protocol.mqttCore - DEBUG - Disconnect result code 1
2016-09-30 20:00:51,575 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 1 sec.
2016-09-30 20:00:52,577 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 2 sec.
2016-09-30 20:00:54,580 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 4 sec.
2016-09-30 20:00:58,585 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 8 sec.
2016-09-30 20:01:06,595 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 16 sec.
2016-09-30 20:01:22,613 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 32 sec.
2016-09-30 20:01:50,631 - core.protocol.mqttCore - INFO - Offline publish request detected.
2016-09-30 20:01:50,631 - core.protocol.mqttCore - INFO - Drainging is still on-going.
2016-09-30 20:01:50,631 - core.protocol.mqttCore - INFO - Try queueing up this request...
2016-09-30 20:01:50,632 - core.util.offlinePublishQueue - DEBUG - append: Add new element: <core.protocol.mqttCore._publishRequest object at 0x7fa6695f2e10>
2016-09-30 20:01:54,632 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 32 sec.
2016-09-30 20:02:26,647 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 32 sec.
2016-09-30 20:02:50,657 - core.protocol.mqttCore - INFO - Offline publish request detected.
2016-09-30 20:02:50,657 - core.protocol.mqttCore - INFO - Drainging is still on-going.
2016-09-30 20:02:50,658 - core.protocol.mqttCore - INFO - Try queueing up this request...
2016-09-30 20:02:50,658 - core.util.offlinePublishQueue - DEBUG - append: Add new element: <core.protocol.mqttCore._publishRequest object at 0x7fa669603da0>
2016-09-30 20:02:58,652 - core.util.progressiveBackoffCore - DEBUG - backOff: current backoff time is: 32 sec.
2016-09-30 20:03:32,253 - core.protocol.mqttCore - DEBUG - Connect result code 0
2016-09-30 20:03:50,698 - core.protocol.mqttCore - DEBUG - Try to put a publish request 6 in the TCP stack.
2016-09-30 20:03:50,699 - core.protocol.mqttCore - DEBUG - Publish request 6 succeeded.
2016-09-30 20:03:52,255 - core.util.progressiveBackoffCore - DEBUG - stableConnection: Resetting the backoff time to: 1 sec.

Declaring File Path for authentication

I am using Windows 10, I have configured my Access key and Secret ID using AWS Command line.
I have tried to run the basic PubSub example but I get :
Missing '-c' or '--cert'
Missing '-k' or '--key'

The way I am giving input in my command line is:
python basicPubSub.py -e a1vg4xwraxdufo.iot.us-west-2.amazonaws.com -r VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem -c a987d73d1a-certificate.pem.crt -k 987d73d1a-private.pem.key
(and also as)
python basicPubSub.py -e a1vg4xwraxdufo.iot.us-west-2.amazonaws.com -r VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem -c C:\Users\bharath\Desktop\HP\Project\Code\AWS IoT\Python\aws-iot-device-sdk-python\samples\basicPubSub\a987d73d1a-certificate.pem.crt -k C:\Users\bharath\Desktop\HP\Project\Code\AWS IoT\Python\aws-iot-device-sdk-python\samples\basicPubSub\a987d73d1a-private.pem.key

I know both these methods are wrong.
What is the correct procedure to give the arguments?

why initial back off time to start with should be less than the stableConnectionTime.

As mentioned in the AWS IoT documentation, what is the specific reason initial back off time be less than the stableConnectionTime?

configureAutoReconnectBackoffTime(baseReconnectQuietTimeSecond, maxReconnectQuietTimeSecond, stableConnectionTimeSecond)
#Description
Used to configure the auto-reconnect backoff timing. Should be called before connect.
Syntax
# Configure the auto-reconnect backoff to start with 1 second and use 128 seconds as a maximum back off time.
# Connection over 20 seconds is considered stable and will reset the back off time back to its base.
myAWSIoTMQTTClient.configureAutoReconnectBackoffTime(1, 128, 20)
Parameters
baseReconnectQuietTimeSecond - The initial back off time to start with, in seconds. Should be less than the stableConnectionTime.
maxReconnectQuietTimeSecond - The maximum back off time, in seconds.
stableConnectionTimeSecond - The number of seconds for a connection to last to be considered as stable. Back off time will be reset to base once the connection is stable.
Returns
None

https://s3.amazonaws.com/aws-iot-device-sdk-python-docs/sphinx/html/generated/AWSIoTPythonSDK.MQTTLib.html

Deadlock on Offline Shadow Update

Preconditions:

1. Two Device shadows are associated (as in a gateway scenerio), but no update has been sent so the update/accepted and update/rejected topics are not yet subscribed to.

Process to Re-Create:

1. Remove connectivity from the library
2. Attempt to update one of the shadows, receive a subscribeTimoutException
3. Attempt to update the second shadow

Result

The call to update the second shadow never returns. The offending call is here:

def basicShadowSubscribe(self, srcShadowName, srcShadowAction, srcCallback):
        self._shadowSubUnsubOperationLock.acquire()
        currentShadowAction = _shadowAction(srcShadowName, srcShadowAction)
        if currentShadowAction.isDelta:
            self._mqttCoreHandler.subscribe(currentShadowAction.getTopicDelta(), 0, srcCallback)
        else:
            self._mqttCoreHandler.subscribe(currentShadowAction.getTopicAccept(), 0, srcCallback)
            self._mqttCoreHandler.subscribe(currentShadowAction.getTopicReject(), 0, srcCallback)
        time.sleep(2)
        self._shadowSubUnsubOperationLock.release()

On line 68 a lock is acquired. Notice that on lines 71, 73, and 74, a subscribeTimeoutException may be raised (as happened in the second step to recreate), jumping past the lock release call on line 76. Any cases of locks being acquired/released should be using context managers or exception handling to ensure that the lock is released if an operation fails.

Problem connecting AWS IoT

Traceback (most recent call last):
File "./publisher.py", line 85, in <module>
client.connect(config.HOST,config.PORT)
File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 686, in connect
return self.reconnect()
File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 821, in reconnect
ciphers=self._tls_ciphers)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()

ssl.SSLError: [Errno 1] _ssl.c:504: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Hey, I am using paho.mqtt library to connect, which is also core for this wrapper.
But, I am not able to get past above mentioned error.
I thought the problem is with TLS version.
Python 2.7 has ssl.PROTOCOL_TLSv1 = 3 max version.

However, you are using ssl.PROTOCOL_SSLv23 and it supposed to work fine.

Here, are the details I supplied
client id: gateway
ca_certs: '/root/aws_iot/keys/ca.pem.cert'
certfile: '/root/aws_iot/keys/aws.pem.cert'
keyfile: '/root/aws_iot/keys/private.pem.key'
host: ***********.iot.*******.amazonaws.com
port: 8883

Could you help me??

Paho client Vs this SDK

Hi,

I'm curious to know, apart from Progressive reconnect back-off and offline publishing what is the advantage of using this client over native paho ?

Can I directly perform shadow operations using paho ?

ThermostatSimulator device

HI, i have problem running the ThermoStatSimulator.py. it provides an invalid syntax error on line 7.

And on my serial monitor the setup monitor just hang on the following.

However, i have shadow updates on the AWS IOT console. i believe the connection to the AWS IOT console is successful.

thanks in advance for your help.

How is this SDK made?

1. Thanks for throwing this up on github, you guys rock.
2. It's not a big deal but is this SDK auto-generated? If so do you have a "contrib" guide / policy?
3. Do we have a testing suite?

There's a lot of non-idiomatic python in here and before I go willy-nilly trying to fix a bunch of stuff I wanted to understand how this was developed so we don't waste our time. I'm not a python zealot on whitespace and CamelCasing or anything like that but I do enjoy making things work well and fast.

ClientToken is limited to 64 bytes

I cannot find documentations about the limits of client token (e.g. http://docs.aws.amazon.com/iot/latest/developerguide/iot-limits.html, http://docs.aws.amazon.com/iot/latest/developerguide/thing-shadow-document.html#client-token ). However, based on my try-and-error with AWS IoT Console, the length of client token seems to be limited to 64 bytes.

Below code generating client token does not check the length of client token. So, shadow request sometimes causes errors, when client id and/or shadow name are too long.

https://github.com/aws/aws-iot-device-sdk-python/blob/master/AWSIoTPythonSDK/core/shadow/deviceShadow.py#L32-L35

progressiveBackoffCore throwing 'NoneType' object is not callable

Receiving the following traceback:

Exception in thread Thread-8 (most likely raised during interpreter shutdown):Exception in thread Thread-4 (most likely raised during interpreter shutdown):
Traceback (most recent call last):

Traceback (most recent call last):  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 1082, in run
  File "/usr/lib/python2.7/threading.py", line 1082, in run

  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/util/progressiveBackoffCore.py", line 85, in _connectionStableThenResetBackoffTime  File "/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK/core/util/progressiveBackoffCore.py", line 85, in _connectionStableThenResetBackoffTime

<type 'exceptions.TypeError'>: 'NoneType' object is not callable<type 'exceptions.TypeError'>: 'NoneType' object is not callable

when publishing to AWS IoT from a raspberry pi running Raspian Jessie. The publish works but traceback happens on teardown. This error is not triggered when testing on my Mac.

environment:

$ pip list
argparse (1.2.1)
AWSIoTPythonSDK (1.0.1)
boto3 (1.4.0)
botocore (1.4.46)
chardet (2.3.0)
colorama (0.3.2)
docutils (0.12)
futures (3.0.5)
gpiozero (1.2.0)
html5lib (0.999)
jmespath (0.9.0)
meld3 (1.0.0)
ndg-httpsclient (0.3.2)
numpy (1.8.2)
picamera (1.12)
pip (1.5.6)
pyasn1 (0.1.7)
pyOpenSSL (0.13.1)
python-apt (0.9.3.12)
python-dateutil (2.5.3)
PyYAML (3.11)
requests (2.4.3)
RPi.GPIO (0.6.2)
s3transfer (0.1.1)
setuptools (25.1.6)
six (1.8.0)
supervisor (3.0)
urllib3 (1.9.1)
wheel (0.24.0)
wiringpi (2.32.1)
wiringpi2 (2.32.3)
wsgiref (0.1.2)

$ uname -a
Linux pi12 4.4.13+ #894 Mon Jun 13 12:43:26 BST 2016 armv6l GNU/Linux

$ python --version
Python 2.7.9

AWSIoTPythonSDK - UserDefinedException Handling

Hi,

I've been trying publishing messages to my local Mosquitto MQTT broker via Paho client and also to AWS IoT using your SDK. Sometimes, even though internet connectivity is available messages don't get published. After doing some reading, I found that loop_start() and loop_stop() of paho can help to over come this. I tried in my native paho client and also in AWS SDK. Before publishing I'll do loop_start() and after publishing I'll be doing loop_stop(). It's good so far and messages are published.

Is it a good practice ?

Would like to hear your inputs on this.

Thanks,
Venkat