Giter Site home page Giter Site logo

algorithmia-rust's Introduction

Algorithmia Rust Client Library

A rust client library for the Algorithmia API.

Documentation

Build Status Crates.io

Getting started

The Algorithmia Rust client is published to crates.io. Add algorithmia = "2.0.0" to the dependencies in your Cargo.toml and run cargo install.

Instantiate an Algorithmia client using your API key:

use algorithmia::Algorithmia;

let client = Algorithmia::client("YOUR_API_KEY")?;

Now you're ready to call algorithms.

Calling algorithms

This client provides an Algorithm type (generally created by client.algo(..)) which provides methods for calling an algorithm hosted on the Algorithmia platform. The following examples of calling algorithms are organized by type of input/output which vary between algorithms.

Note: a single algorithm may have different input and output types, or accept multiple types of input, so consult the algorithm's description for usage examples specific to that algorithm.

Text input/output

Call an algorithm with text input by simply passing &str into Algorithm::pipe. If the algorithm output is text, call the as_string method on the response.

let algo = client.algo("algo://demo/Hello/0.1.1");
let response = algo.pipe("HAL 9000")?;
println!("{}", response.into_string()?);

JSON input/output

Call an algorithm with JSON input calling pipe with a reference to a type that implements serde::Serialize. If the algorithm output is JSON, you can call decode to deserialize the resonse into a type that implements serde::Deserialize.

This includes many primitive types, tuples, Vec, and other collection types from the standard library:

let algo = client.algo("algo://WebPredict/ListAnagrams/0.1.0");
let response = algo.pipe(vec!["transformer", "terraforms", "retransform"])?;
let output: Vec<String> = response.decode()?;
// -> ["transformer", "retransform"] as Vec<String>

Implementing Serialize or Deserialize for your custom types is generally as easy as adding a derive annotation.

#[derive(Deserialize, Serialize)]
struct MyStruct {
    some_field: String,
    other_field: u32,
}
// now you can call `pipe` with `&MyStruct` or `decode` into `MyStruct`

With serde_json, you can also sue the json! macro or implement custom serialization/deserialization. See serde.rs for more details on using serde.

Alternatively, you can work with raw JSON strings:

let response = algo.pipe_json(r#"["transformer", "terraforms", "retransform"]"#)?;
let output = response.to_json()?;
// -> "[\"transformer\", \"retransform\"]"

Binary input/output

Call an algorithm with binary input by calling the pipe method with a slice of bytes (&[u8]). If the algorithm response is binary data, then call the as_bytes method on the response to obtain a byte vector (Vec<u8>).

let mut input = Vec::new();
File::open("/path/to/bender.jpg").read_to_end(&mut input);
let response = client.algo("opencv/SmartThumbnail/0.1").pipe(&input)?;
let output = response.as_bytes()?;
// -> Vec<u8>

Error handling

True to the nature of explicit error handling in rust, the pipe and response parsing methods all return Result-wrapped types:

let algo = client.algo("algo://demo/Hello/0.1.1");
match algo.pipe(&[]) {
    Ok(response) => { /* success */ },
    Err(err) => println!("error calling demo/Hello: {}", err),
}
// -> error calling demo/Hello: apply() functions do not match input data

Note: this crate makes use of the error-chain crate, so for many error variants, there may be additional errors chained to err.source().

Request options

The client exposes options that can configure algorithm requests via a builder pattern. This includes support for changing the timeout or indicating that the API should include stdout in the response.

let mut algo = client.algo("algo://demo/Hello/0.1.1");
let algo = algo.timeout(10).stdout(true);
let response = algo.pipe(input)?;
if let Some(ref stdout) = response.metadata.stdout {
    println!("{}", stdout);
}

Note: stdout(true) is ignored if you do not have access to the algorithm source.

Managing data

The Algorithmia Rust client also provides a way to manage both Algorithmia hosted data and data from Dropbox or S3 accounts that you've connected to you Algorithmia account.

This client provides a DataFile type (generally created by client.file(uri)) and a DataDir type (generally created by client.dir(uri)) that provide methods for managing your data.

Create directories

Create directories by instantiating a DataDir object and calling create() with a DataAcl:

let robots = client.dir("data://.my/robots");
robots.create(DataAcl::default())

let robots = client.dir("dropbox://robots");
robots.create(DataAcl::default())

Upload files to a directory

Upload files by calling put on a DataFile object, or by calling put_file on a DataDir object.

let robots = client.dir("data://.my/robots");

// Upload local file
robots.put_file("/path/to/Optimus_Prime.png");
// Write a text file
robots.child::<DataFile>("Optimus_Prime.txt").put("Leader of the Autobots");
// Write a binary file
robots.child::<DataFile>("Optimus_Prime.key").put(b"transform");

Download contents of file

Download files by calling get on a DataFile object which returns a Result-wrapped DataResponse that implements Read:

// Download and locally save file
let mut t800_png_reader = client.file("data://.my/robots/T-800.png").get()?;
let mut t800_png = File::create("/path/to/save/t800.png")?;
std::io::copy(&mut t800_png_reader, &mut t800_png);

// Get the file's contents as a string
let mut t800_text_reader = robots.file("data://.my/robots/T-800.txt").get()?;
let mut t800_text = String::new();
t800_text_reader.read_to_string(&mut t800_text);

// Get the file's contents as a byte array
let mut t800_png_reader = robots.file("data://.my/robots/T-800.png").get()?;
let mut t800_bytes = Vec::new();
t800_png_reader.read_to_end(&mut t800_bytes);

Delete files and directories

Delete files and directories by calling delete on their respective DataFile or DataDir object. DataDirectories take a force parameter that indicates whether the directory should be deleted if it contains files or other directories.

client.file("data://.my/robots/C-3PO.txt").delete();
client.dir("data://.my/robots").delete(false);

List directory contents

Iterate over the contents of a directory using the iterator returned by calling list on a DataDir object:

let my_robots = client.dir("data://.my/robots");
for entry in my_robots.list() {
    match entry {
        Ok(DirEntry::Dir(dir)) => println!("Directory {}", dir.to_data_uri()),
        Ok(DirEntry::File(file)) => println!("File {}", file.to_data_uri()),
        Err(err) => println!("Error listing my robots: {}", err),
    }
}

Examples

For examples that use this client, see:

Build & Test

This project is built and tested with cargo:

cargo build
cargo test
cargo doc --no-deps

algorithmia-rust's People

Contributors

anowell avatar bytekast avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

anowell

algorithmia-rust's Issues

file get operation intermittently hangs forever

When downloading a file using the client.file(..).get();... io::copy(..) operation, it occasionally fails to complete downloading the file in what seems like a broken pipe failure mode.

The output from .get() is a DataFile, which looks like a stream object, and the .get() is supposed to return a Result<DataFile>, however, no exception is ever thrown when said suspected broken pipe occurs, instead it hangs for indefinitely while keeping the httpClient connection open until manually killed.

https://gist.github.com/zeryx/3cdbe5c8b0d3da30041a1b0348f4cb83

note: The error is intermittent, and won't occur on every run; it seems to have a causal relationship with cpu load, which might hint towards the underlying cause.

There might be a more refined minimal way of defining a reproducible output, but I haven't reached that yet.
As you can see, the bizarre workaround is to split the client.file(..).get() operation into a two line client.file() and file.get().

The only hypothetical cause is a bug in the underlying http-client library reqwest, however that seems farfetched.

DataFile should have .put_file() method like DataDir.

The current rust client has a useful .put_file() for the DataDir:

client.dir(some_remote_dir).put_file(some_local_file);

Which is super useful! however, when you have a particular remote filename that's different from the local file name, this the method isn't implemented:

client.file(some_remote_filename).put_file(some_local_file);

I propose that DataFile should have this method exposed to make rust life less painful!

Codex - rust client calling wrong api endpoint

when trying to download a file on a codex installation, I discovered that it was throwing the following exception:

couldn't download file from url: data://zeryx/binaries/ffmpeg-static.tar.gz 
401 Unauthorized

when talking with @pmcq it was discovered that on the api logs it was reported as:

 {"start_date":"2016-11-03T21:22:11.578Z","start_time":1478208131578,"end_time":1478208131581,"method":"GET","uri":"//v1/data/zeryx/bin/ffmpeg-static.tar.gz","status_code":401,"request_time":3,"ip":"156.34.17.89","user_agent":"algorithmia-rust/1.3.0 (Rust unknown","auth_user":"zeryx"}

2016-11-03T21:10:03.987 [application-akka.actor.default-dispatcher-48982] INFO  application : data_v0 GET error, no auth - path: zeryx/binaries/ffmpeg-static.tar.gz error: Authorizat
ionRequired(missing auth) request_headers: ArrayBuffer((User-Agent,algorithmia-rust/1.3.0 (Rust unknown), (Authorization,Simple <snip>), (Host,api.codexonazure.
com))

it looks like it's trying to call the data_v0 endpoint which is invalid on codex, breaking the rust client for all codex installations.

DataDir's put_file unwraps a local file exist check

line 362 of dir.rs contains this line:
let mut file = File::open(path_ref).unwrap();
if path_ref doesn't refer to an existing file path, the unwrap fails. I propose this is replaced with:

let mut file try!(File::open(path_ref);

where try!() is a macro that is able to parse an io::Result()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.