omerb01/pywren-ibm-cloud

PyWren for IBM Cloud Functions and IBM Cloud Object Storage

Lithops framework is a rebranded PyWren-IBM framework. Lithops goals are massively scaling the execution of Python code and its dependencies on serverless computing platforms and monitoring the results. Lithops delivers the user’s code into the serverless platform without requiring knowledge of how functions are invoked and run.

Lithops provides great value for the variety of uses cases, like processing data in object storage, running embarrassingly parallel compute jobs (e.g. Monte-Carlo simulations), enriching data with additional attributes and many more. In extending Lithops to work with object storage, we also added a partition discovery component that allows Lithops to process large amounts of data stored in the object storage. See changelog for more details.

Lithops and IBM Cloud

Lithops is officially supported for IBM Cloud and Red Hat OpenShift. Students or academic stuff are welcome to follow IBM Academic Initiative, a special program that allows free trial of IBM Cloud for Academic institutions. This program is provided for students and faculty staff members, and allow up to 12 months of free usage. You can register your university email and get a free of charge account.

Lithops for broad scope

Lithops provides an extensible backend architecture (compute, storage) that is designed to work with different Cloud providers and on-premise backends (Knative, OpenShift). You can code in Lithops and run it unmodified in IBM Cloud, AWS, Azure, Google or Alibaba. Technical support for clouds different than IBM are delegated to best efforts from the open source community.

Quick Start

1. Install Lithops from the PyPi repository:

   $ pip install lithops

2. Follow these instructions to configure your compute and storage backends

3. Test Lithops by simply running the next command:

    $ lithops test

   or by running the next code:

   import lithops
   
   def hello(name):
       return 'Hello {}!'.format(name)
   
   fexec = lithops.function_executor()
   fexec.call_async(hello, 'World')
   print(fexec.get_result())

Additional information and examples

• Examples: You can find various examples in examples folder
• Lithops API and Exectors: Lithops API and Executors
• Functions: Lithops functions and parameters
• Runtimes: Building and managing Lithops runtimes to run the functions
• Data processing: Using Lithops to process data from IBM Cloud Object Storage and public URLs
• Notebooks: Lithops on IBM Watson Studio and Jupyter notebooks

How to contribute code

Follow guidance how to contribute code to the project

Additional resources

• Decoding dark molecular matter in spatial metabolomics with IBM Cloud Functions
• Your easy move to serverless computing and radically simplified data processing Strata Data Conference, NY 2019
  • See video of Lithops usage here and the example of Monte Carlo here
• Ants, serverless computing, and simplified data processing
• Speed up data pre-processing with PyWren in deep learning
• Predicting the future with Monte Carlo simulations over IBM Cloud Functions
• Process large data sets at massive scale with PyWren over IBM Cloud Functions
• Industrial project in Technion on Lithops
• Serverless data analytics in the IBM Cloud - Proceedings of the 19th International Middleware Conference (Industry)

Acknowledgements

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184.