snowch/ibm-analytics-engine-python

Python library for IBM Analytics Engine - https://console.bluemix.net/docs/services/AnalyticsEngine/index.html

---

Overview

This project is a python library (SDK) for working with IBM Analytics Engine (IAE).

Documentation

See example notebook

Support

Please create an issue for questions, bugs and feature requests.

Contributing

See DEVELOPING.md

Pull requests are welcome!

ibm-analytics-engine-python's Issues

Change docker bash example to python

Change the bash example to python using the docker python api

Provide an API method to list all clusters that I have access to across all my spaces

import pprint
pp = pprint.PrettyPrinter(indent=4)

for org in cf.orgs_and_spaces():
    for space in org['spaces']:
        print('ORG {} | SPACE {}'.format(org['name'], space['name']))
        print()

        clusters = iae.clusters(space_guid=space['guid'])
        if len(clusters) > 0:
            for cluster in clusters:
                pp.pprint(cluster)
                print()

Code Improvements

• Rename IAE -> IAE_Service_Instance_Operations(cf)

• Create:

  • IAE_Spark_Livy_Operations(vcap)
  • IAE_Spark_Ssh_Operations(vcap)

• CF API calls, don't store whole json, store only required items:

    self.info = self._get_info()
    ...
    def _get_info(self):
        url = '{}/v2/info'.format(self.api_endpoint)
        response = self._request(url=url, http_method='get', description='_get_info', create_auth_headers=False)
        return response.json()

to

    def _get_info(self):
        url = '{}/v2/info'.format(self.api_endpoint)
        response = self._request(url=url, http_method='get', description='_get_info', create_auth_headers=False)
        return response.json()['authorization_endpoint'] + '/oauth/token'

• simplify IAE responses, users should not have to see cloud foundry details by default

if status == 'succeeded':
   credentials_json = iae.create_credentials(cluster_instance_id)
   vcap_json = credentials_json['entity']['credentials']
   print('>> VCAP:\n' + json.dumps(vcap_json, indent=4, separators=(',', ': ')))

Should be:

if status == 'succeeded':
   vcap_json = iae.create_credentials(cluster_instance_id)
   print('>> VCAP:\n' + json.dumps(vcap_json, indent=4, separators=(',', ': ')))

Provide API method to set SPARK_DIST_CLASSPATH

Steps to configure:

• using Ambari API, configuration settings can be retrieved
• grab spark2-env properties
• update SPARK_DIST_CLASSPATH (see below)
• save spark2-env properties
• restart spark2

---

Example code for working with ambari api:

from future.standard_library import install_aliases
install_aliases()
from urllib.parse import urlparse

import json
vcap = json.load(open('./vcap.json'))

USER         = vcap['cluster']['user']
PASSWORD     = vcap['cluster']['password']
AMBARI_URL   = vcap['cluster']['service_endpoints']['ambari_console']
CLUSTER_ID   = vcap['cluster']['cluster_id']
CLUSTER_NAME = 'AnalyticsEngine'

url = urlparse(AMBARI_URL)

HOST = url.hostname
PORT = url.port
PROTOCOL = url.scheme

and

    echo 'Downloading configs.py ambari script'
    curl https://raw.githubusercontent.com/apache/ambari/9e93c476ddd8d4397f550062fd1645ac5422ed2e/ambari-server/src/main/resources/scripts/configs.py > configs.py

    echo 'Getting the latest spark2-env configuration'
    # grab the latest spark2-env configuration file
    ./configs.py -u ${AMBARI_USER} -p ${AMBARI_PASSWORD} -n ${CLUSTER_NAME} -s https --port ${AMBARI_PORT} -l ${AMBARI_HOST} -a get -c spark2-env -f spark2-env-content.json

    echo 'Current spark2-env configuartion:'
    cat spark2-env-content.json

    # append the spark-avro folder to the SPARK_DIST_CLASSPATH
    echo 'export SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:/home/clsadmin/spark-avro/*' >> spark2-env-content.json

    # save the changes back 
    ./configs.py -u ${AMBARI_USER} -p ${AMBARI_PASSWORD} -n ${CLUSTER_NAME} -s https --port ${AMBARI_PORT} -l ${AMBARI_HOST} -a set -c spark2-env -f spark2-env-content.json

    echo 'Uploaded new spark2-env configuartion:'
    cat spark2-env-content.json

    echo "stop and Start Services"
    curl -k -v --user $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By: ambari" -i -X PUT -d '{"RequestInfo": {"context": "Stop All Services via REST"}, "ServiceInfo": {"state":"INSTALLED"}}' https://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services

    curl -k -v --user $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By: ambari" -i -X PUT -d '{"RequestInfo": {"context": "Start All Services via REST"}, "ServiceInfo":{"state":"STARTED"}}' https://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services

# https://git.ng.bluemix.net/chris.snow/iae-spark-package-customization-example/raw/master/bootstrap/verify_ambari_service_status.py

---

Example file based approach for extracting env var definitions:

In [18]: with open('spark-env.sh', 'r') as f:
    ...:     for line in shlex.split(f.read()):
    ...:         var, eq, value = line.partition('=')
    ...:         if var == 'SPARK_DIST_CLASSPATH' and eq:
    ...:             print(var, value)
    ...:
SPARK_DIST_CLASSPATH /home/common/lib/scala/spark2/*:/home/common/lib/scala/common/*:/home/common/lib/dataconnectorCloudant/*:/home/common/lib/dataconnectorStocator/*:/home/common/lib/dataconnectorDb2/*:/home/common/lib/dataconnectorIdax/*
SPARK_DIST_CLASSPATH $SPARK_DIST_CLASSPATH:/some/otherdir/*\

Provide function to get namenode hostname

Ambari Request example:

GET https://xxxx.bi.services.us-south.bluemix.net:9443/api/v1/clusters/AnalyticsEngine/services/HDFS/components/NAMENODE?fields=host_components

Ambari Response example:

{
  "href" : "https://xxxx.bi.services.us-south.bluemix.net:9443/api/v1/clusters/AnalyticsEngine/services/HDFS/components/NAMENODE?fields=host_components",
  "ServiceComponentInfo" : {
    "cluster_name" : "AnalyticsEngine",
    "component_name" : "NAMENODE",
    "service_name" : "HDFS"
  },
  "host_components" : [
    {
      "href" : "https://xxxx.bi.services.us-south.bluemix.net:9443/api/v1/clusters/AnalyticsEngine/hosts/xxxx.bi.services.us-south.bluemix.net/host_components/NAMENODE",
      "HostRoles" : {
        "cluster_name" : "AnalyticsEngine",
        "component_name" : "NAMENODE",
        "host_name" : "xxxx.bi.services.us-south.bluemix.net"
      }
    }
  ]
}

Python example

    # urllib example:
    try:
      url = '{0}://{1}:{2}{3}'.format(protocol, host, port, api_url)
      admin_auth = base64.encodestring('%s:%s' % (login, password)).replace('\n', '')
      request = urllib2.Request(url)
      request.add_header('Authorization', 'Basic %s' % admin_auth)
      request.add_header('X-Requested-By', 'ambari')
      request.add_data(request_body)
      request.get_method = lambda: request_type

      print(url, request_body, login, password)

      response = urllib2.urlopen(request)
      response_body = response.read()
      print(response_body)
    except Exception as exc:
      raise Exception('Problem with accessing api. Reason: {0}'.format(exc))

    name_node = response['host_components']['hostname']

cf.spaces_guid(...) returning incorrect guid

Feature request: support Livy and SparkSubmitSsh

• create a vcap class to wrap vcap.json
• create new method IAE#get_credential_vcap(cluster_instance_id) : vcap if more than one credentials on cluster, return first
• create new class Livy to encapsulate livy operations
• create new class SparkSubmitSsh to encapsulate spark submit operations

class Vcap:
    def __init__(self, json_string=None, vcap_json_filename=None):
        ...

class IAE:
    def get_credential_vcap(self, cluster_instance_id):
        ...
       vcap_json = ... cf call to get credentials
       return Vcap.loads(vcap_json)

...

class SparkSubmitSsh:
    def __init__(self, vcap):
        ...

    # See yarn examples: https://github.com/snowch/demo_2710/blob/6d87fa8e94fc8256ac287ceb915a72d73e1a9211/Step%2010%20-%20Export%20Spark%20Streaming%20model%20to%20BigI.ipynb

    def submit_yarn ...  return yarn_application_id
    def yarn_status( self, yarn_application_id ) ...
    def yarn_logs( self, yarn_application_id ) ...
    def yarn_kill( self, yarn_application_id ) ...

Example 1 - if you don’t have the vcap_json

cf = CloudFoundryAPI(api_key_filename=cf_api_key_filename)
iae = IAE(cf_client=cf)
vcap = iae.get_credential_vcap( ‘cluster_12345’ )
spark_submit = spark_submit_ssh.SparkSubmitSsh(vcap)
yarn_app_id = spark_submit.submit_yarn( ... )

Example 2 - if you have the vcap_json

vcap = Vcap( ‘./vcap.json’ )
spark_submit = spark_submit_ssh.SparkSubmitSsh(vcap)
yarn_app_id = spark_submit.submit_yarn( ... )

Support Jython

Travis info https://www.topbug.net/blog/2012/05/27/use-travis-ci-with-jython/

Add cloud foundry username and password authentication option

Use zest releaser for releases

https://pypi.python.org/pypi/zest.releaser