Giter Site home page Giter Site logo

osdupy's Introduction

osdupy

A simple python client for the OSDU data platform.

Contents

Clients

Choose the client that best meets your needs. The same methods are all supported for each.

SimpleOsduClient

BYOT: Bring your own token. Great for backend service or business logic that supplements a front-end application.

This client assumes you are obtaining a token yourself (e.g. via your application's login form or otheer mechanism. With this SimpleOsduClient, you simply provide that token. With this simplicity, you are also then respnsible for reefreeshing the token as needed and re-instantiating the client with the new token.

AwsOsduServicePrincipalClient

Requires: boto3==1.15.*

Good for batch tasks that don't have an interactive front-end. Token management is handled with the boto3 library directly through the Cognito service. You have to supply additional arguments for this.

For OSDU on AWS, this client is usually simpler than the AwsOsduClient as long as you have IAM credentials to access the necessary resources. You only need to provide the OSDU resource_prefix, region, and profile.

AwsOsduClient

Requires: boto3==1.15.*

Good for batch tasks that don't have an interactive front-end. Token management is handled with the boto3 library directly through the Cognito service. You have to supply additional arguments for this.

For OSDU on AWS, this client is useful in the case where you may want to perform actions as a specific OSDU user rather than as the ServicePrinicpal.

Currently supported methods

  • search
    • query
    • query_with_paging
  • storage
    • query_all_kinds
    • get_record
    • get_records
    • get_all_record_versions
    • get_record_version
    • store_records
    • delete_record
    • purge_record
  • dataset
    • get_dataset_registry
    • get_dataset_registries
    • get_storage_instructions
    • register_dataset
    • get_retrieval_instructions
  • entitlements
    • get_groups
    • get_group_members
    • add_group_member
    • delete_group_member
    • create_group

Installation

pip install osdupy

Tests

Run unit tests

python -m unittest -v tests.unit

Run integration tests

python -m unittest -v tests.integration

Usage

Instantiating the SimpleOsduClient

If environment variable OSDU_API_URL is set, then it does not need to be passed as an argument. Otherwise it must be passed as keyword argument.

from osdu.client import SimpleOsduClient

data_partition = 'osdu'
token = 'token-received-from-front-end-app'

# With env var `OSDU_API_URL` set in current environment.
osdu_client = SimpleOsduClient(data_partition, token)

# Without env var set.
api_url = 'https://your.api.base_url.com'
osdu_client = SimpleOsduClient(data_partition, token, api_url=api_url)

Instantiating the AwsServicePrincipalOsduClient

from osdu.client import AwsOsduClient

data_partition = 'osdu'
resource_prefix = 'osdur3mX'

osdu_client = AwsServicePrincipalOsduClient(
    data_partition,
    resource_prefix,
    profile=os.environ['AWS_PROFILE'],
    region=os.environ['AWS_DEFAULT_REGION']
)

Instantiating the AwsOsduClient

The only required argument is data_partition. If your environment variables (below) have been set, then client can be instantiated with only data_partition as an argument. Environment variables:

  1. OSDU_API_URL
  2. OSDU_CLIENT_ID
  3. OSDU_USER
  4. OSDU_PASSWORD
  5. AWS_PROFILE
  6. AWS_SECRETHASH
from osdu.client import AwsOsduClient

data_partition = 'osdu'

osdu_client = AwsOsduClient(data_partition)

If you have not set the above environment variales—or you have only set some—then you will need to pass any undefined as args when instantiating the client.

from getpass import getpass
from osdu.client import AwsOsduClient

api_url = 'https://your.api.url.com'  # Must be base URL only
client_id = 'YOURCLIENTID'
user = '[email protected]'
password = getpass()
data_partition = 'osdu'
profile = 'osdu-dev'

message = user + client_id
dig = hmac.new(client_secret.encode('UTF-8'), msg=message.encode('UTF-8'),
               digestmod=hashlib.sha256).digest()
secretHash = base64.b64encode(dig).decode()



osdu_client = AwsOsduClient(data_partition,
    api_url=api_url,
    client_id=client_id,
    user=user,
    password=password,
    secret_hash=secretHash,
    profile=profile)

Automatically re-authorizing the client

Each client will automatically attempt to re-authorize when its access token expires. In order for this re-authorization to succeed, you will need to supply the client with additional parameters (either through environment variables or in their consructor):

Simple Client:

  1. OSDU_CLIENTWITHSECRET_ID
  2. OSDU_CLIENTWITHSECRET_SECRET
  3. REFRESH_TOKEN
  4. REFRESH_URL

AWS Client:

  1. OSDU_PASSWORD (in the environment variables, or somewhere else it can persist securely)

Service Principal:

N/A--this client can re-authorize with just the variables needed for it to instantiate

Using the client

Below are just a few usage examples. See integration tests for more comprehensive usage examples.

Search for records by query

query = {
    "kind": f"osdu:wks:*:*"
}
result = osdu_client.search.query(query)
# { results: [ {...}, .... ], totalCount: ##### }

Search with paging

For result sets larger than 1,000 records, use the query with paging method.

page_size = 100 # Number of records per page (1-1000)
query = {
    "kind": f"osdu:wks:*:*",
    "limit": page_size
}
result = osdu_client.search.query_with_paging(query)

# Iterate over the pages to do something with the results.
for page, total_count in result:
    for record in page:
        # Do stuff with record...

Get a record

record_id = 'opendes:doc:123456789'
result = osdu_client.storage.get_record(record_id)
# { 'id': 'opendes:doc:123456789', 'kind': ..., 'data': {...}, 'acl': {...}, .... }

Upsert records

new_or_updated_record = './record-123.json'
with open(new_or_updated_record, 'r') as _file:
    record = json.load(_file)

result = osdu_client.storage.store_records([record])

List groupmembership for the current user

result = osdu_client.entitlements.get_groups()
# {
#  "desId": "[email protected]",
#  "groups": [
#    {
#      "description": "Datalake Plugin-Manager users",
#      "email": "[email protected]",
#      "name": "service.plugin.user"
#    },
#    {
#      "description": "Datalake csv-parser admins",
#      "email": "[email protected]",
#      "name": "service.csv-parser.admin"
#    },
#    #...
#       {
#     "description": "The viewer of the datalake csv-parser service",
#     "email": "[email protected]",
#     "name": "service.csv-parser.viewer"
#   }
# ],
# "memberEmail": "[email protected]"
# }

List membership of a particular group

result = osdu_client.entitlements.get_group_members('[email protected]')
#{
#  "members": [
#    {
#      "email": "[email protected]",
#      "role": "OWNER"
#    },
#    {
#      "email": "[email protected]",
#      "role": "OWNER"
#    },
#    {
#      "email": "[email protected]",
#      "role": "OWNER"
#    }
#  ]
#}

Add a user to a particular group

Add a user ([email protected]) to groups to give entitlement to search for and retrieve data.

query =  {
     "email": "[email protected]",
     #OWNER or MEMBER
     "role": "MEMBER",
 }
result = osdu_client.entitlements.add_group_member('[email protected]',query)
query =  {
     "email": "[email protected]",
     #OWNER or MEMBER
     "role": "OWNER",
 }
result = osdu_client.entitlements.add_group_member('[email protected]',query)

Delete user from a particular group

Remove a user ([email protected]) from a group.

query =  {
     "email": "[email protected]",
     #OWNER or MEMBER
     "role": "MEMBER",
 }
result = osdu_client.entitlements.delete_group_member('[email protected]',query)
query =  {
     "email": "[email protected]",
     #OWNER or MEMBER
     "role": "OWNER",
 }
result = osdu_client.entitlements.delete_group_member('[email protected]',query)

osdupy's People

Contributors

puremcc avatar johnnyreichmanpariveda avatar pbradshawusc avatar visrealm avatar eternelpanic avatar jplorch avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.