palantir/giraffe

Gracefully Integrated Remote Access For Files and Execution

Giraffe

Gracefully Integrated Remote Access For Files and Execution

A long neck to reach far-away places

• Home
• Documentation

Overview

Giraffe is a Java library that integrates local and remote file system access with command execution behind a common, familiar API. It combines new classes for command execution with remote implementations of the java.nio.file API introduced in Java 7.

SshHostAccessor ssh = SshHostAccessor.forPassword("example.com", "giraffe", "l0ngN3ck");

try (HostControlSystem hcs = ssh.open()) {
    Path logs = hcs.getPath("server/logs");
    Files.copy(logs.resolve("access.log"), Paths.get("log/example-access.log"));

    Command archive = hcs.getCommand("server/bin/archive.sh", "--format=zip", "logs");
    Commands.execute(archive);
}

Get Giraffe

Giraffe is available from Maven Central.

With Gradle:

repositories {
    mavenCentral()
}

dependencies {
    compile 'com.palantir.giraffe:giraffe-ssh:0.10.1'
    // or 'com.palantir.giraffe:giraffe-core:0.10.1' for local features only
}

Why Giraffe?

Why did we write Giraffe and why might you use it?

While working on deployment and test tools we found many situations where we wanted to write code that worked easily on both the local host and remote hosts using SSH. This required at least three different APIs with different abstractions:

1. Native Java functionality (with third-party utilities) for local files
2. Native Java functionality or commons-exec for local command execution
3. An SSH library (sshj, jsch, ganymed-ssh2) for remote file manipulation and command execution

This led to duplicated abstraction layers in our projects and complicated code that had to know what type of host it was targeting.

With Giraffe, a single library is required and there are only two APIs: native Java functionality for files and an intentionally similar API for command execution.

Alternatives

The closest equivalent to Giraffe is XebiaLabs's Overthere. In our view, Giraffe has two major benefits when compared to Overthere:

1. It uses the standard java.nio.file API introduced in Java 7
2. It's offered under the Apache 2.0 license instead of the GPLv2

That said, Overthere supports more protocols and supports Windows, which may make it more appropriate for your use case.

Support

In general, any release of Giraffe is supported until a newer version is released. Users should update to newer versions as soon as possible.

Occasionally, we continue to provide bug fixes and support for the previous major version series after a new major version release. These releases are listed below. We support old releases for at most 6 months after a new major version release.

No supported old releases at this time

Development

Giraffe builds with Gradle and is configured to use Eclipse as an IDE:

$ ./gradlew eclipse     # generate Eclipse projects
$ ./gradlew build       # compile libraries and run tests

See ./gradlew tasks for more options.

giraffe's Issues

Audit execution system Javadocs

Make sure the Javadocs are complete, correct, and consistent.

404 in the latest javadoc

I found some 404 in the latest javadoc of version 0.10.
https://palantir.github.io/giraffe/docs/0.10.0/api/index.html

If you click PasswordSshCredential from the left down pane, you get 404.
https://palantir.github.io/giraffe/docs/0.10.0/api/com/palantir/giraffe/ssh/PasswordSshCredential.html

Many links are also not working.
Please check it out.

Throw ExecutionSystemClosedException when system is closed

Currently, a closed execution system throws some SSHJ error when it tries to execute on a closed connection. Make the execute() method throw ClosedExecutionSystemException to make this more obvious.

Ability to obtain stdout from ShellConversation

From Dan:

To do interactive execution, I executeAsync a command with gives me a CommandFuture. I pass this into a ShellConversation to handle interactive execution (awesome!). However when the process it complete, I would like to have the stdout to obtain the information I need / verify things ran as I intended.

When I get the stdout stream from the original CommandFuture, it is empty because the interactive execution has consumed it.

Is there a way I can obtain the stdout output from the ShellConversation or something?

Migrated from GIRAFFE-17.

Audit file system Javadocs

Make sure the Javadocs are complete, correct, and consistent.

UnsupportedOperationException when copying files from SMB on Windows

When doing this:

Path source = Paths.get("\\\\filer.domain.local\\folder");
Path target = Paths.get("c:\\tmp");
MoreFiles.copyRecursive(source, target);

I get this exception:

java.lang.UnsupportedOperationException
    at sun.nio.fs.WindowsFileSystemProvider.readAttributes(WindowsFileSystemProvider.java:192)
    at java.nio.file.Files.readAttributes(Files.java:1686)
    at java.nio.file.Files.getPosixFilePermissions(Files.java:1953)
    at com.palantir.giraffe.file.MoreFiles$CopyVisitor.preVisitDirectory(MoreFiles.java:285)
    at com.palantir.giraffe.file.MoreFiles$CopyVisitor.preVisitDirectory(MoreFiles.java:269)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:192)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
    at java.nio.file.Files.walkFileTree(Files.java:2602)
    at java.nio.file.Files.walkFileTree(Files.java:2635)
    at com.palantir.giraffe.file.MoreFiles.copyRecursive(MoreFiles.java:266)

Fix doclint warning and errors

After switching to JDK 8, doclint revealed a bunch of problems with the docs. I currently have this linting disabled in gradle/javadoc.gradle, but we should actually fix all the warnings and leave doclint on instead.

Using -n for same-host SSH operations is not POSIX compliant

giraffe/ssh/src/main/java/com/palantir/giraffe/ssh/internal/SshSameHostFileHelper.java

Line 33 in 0318239

args.add("-n");

giraffe/ssh/src/main/java/com/palantir/giraffe/ssh/internal/SshSameHostFileHelper.java

Line 63 in 0318239

args.add("-n");

The -n flag (to prevent overwriting existing files) is not part of the POSIX specification for the cp or mv commands. Giraffe guarantees that it will work with any POSIX-compliant shell, so this is a violation of the contract. That said, I'm not sure how to fix this and it hasn't caused problems for any real users yet. I just ran into while updating the CI tests to an Alpine container.

Improve error message when commands timeout

Instead of throwing a simple TimeoutException, we should throw a CommandTimeoutException or something similar that includes information similar to CommandException, mainly the command and the output so far, which will help with debugging.

This affects Commands#waitFor and possible other methods in Commands.

Add a check to determine if the JCE jars are available

This would be helpful for people for debugging their setup.

A warning that they don't have the JCE unlimited strength jars might be enough.

Upgrade JGit version used by Gradle

When trying to release 0.9.0, I saw the JGit issue where it thinks the repository is dirty, but running git status shows a clean state (and also fixes the state seen by JGit).

I believe this is fixed in newer versions of JGit, but I'm not sure if there's a newer release of the flexversion plugin. We might need to force upgrade JGit or switch to a different versioning method.

Improve contract of copyRecursive

copyRecursive works but does not take options and requires a specific set of conditions for the destination. Ideally this will merge directories by default, failing if any files would be overwritten. It can then take the StandardCopyOption.REPLACE_EXISTING option to overwrite files. We need to make sure that this behavior can work consistently between the file tree implementation, cp, and scp.

Add basic Path finding utilities

Maybe something like (or as methods in MoreFiles):

public final class PathFinder {

public static Path find(String relativePath) throws IOException {
    return find(Paths.get(""), relativePath);
}

public static Path find(Path basePath, String relativePath) throws IOException {
    Path currPath = basePath.toAbsolutePath().normalize();
    while (currPath.getParent() != null && Files.notExists(currPath.resolve(relativePath))) {
        currPath = currPath.getParent();
    }
    if (Files.notExists(currPath.resolve(relativePath))) {
        throw new IOException("Could not locate " + relativePath); 
    }
    return currPath.resolve(relativePath);
}

}

Introduce new utility MoreFiles#listRecursiveFilesInDirectory

We want a utility method for recursively listing the files in a directory, returning a flattened list of Paths. I'm not sure what kind of contract we want to use for the list: I would default to appending file Paths to the list as we recurisvely traverse the DirectoryStream unless anyone has a preference.

Add utility for streaming command output to logger

The current method looks something like this:

CommandFuture future = Commands.executeAsync(cmd);
ResponseProvider<Runnable> responses = UnorderedResponseProvider.<Runnable>builder()
        .addRegex(".*", new Function<String, Runnable> () {
            @Override
            public Runnable apply(final String input) {
                return new Runnable() {
                    @Override
                    public void run() {
                        logger.info(input);
                    }
                };
            }
        })
        .build();
CommandOutputTrigger outputTrigger = new CommandOutputTrigger(responses, future);
outputTrigger.start();
Commands.waitFor(future);

Since this is a relatively common pattern, there should be a utility that simplifies this.

MoreFiles.copyRecursive loops infinitely until failure when copying a directory into one of its subdirectories

Repro:
Path a = Paths.get("a") // a is an existing directory with contents
Path b = Paths.get("a/b") // b does not exist in a
MoreFiles.copyRecursive(a, b)

This loops infinitely until failure because the copy generates more content to copy. It also passes both precondition checks in MoreFiles.copyRecursive because "a/b" does not exist AND "a" does exist.

Add ExecutionSystem contexts

There are many situations when you want all commands associated with a specific execution system to use the same default working directory, environment, prefix (sudo -u user, bash -c), or other properties. Ideally, this will be a provider-type interface that can compute values dynamically (e.g. refresh the environment by executing bash -l -c env). This leads to some complications with recursive execution, but they should be solvable.

This might relate to a larger issue of better exposing the various options that can be passed to systems when they are constructed.

Improve test coverage

Check test coverage numbers to see what could be easily tested but isn't currently.

Also, refactor the coverage system into a Gradle file/plugin that takes a threshold as the single parameter. This will make it easier to disable the more advanced coverage metrics in favor of the basic line/statement/branch coverage.

MoreFiles#listDirectory should accept globs/patterns

It would be helpful to have a version of this method that passes a pattern or glob into Files.newDirectoryStream.

Improve SystemConverter

The current contract requires that users close converted systems. Unfortunately, this means they need to know if the source system was remote or local, since local systems cannot be closed.

Since these are designed to be views of the source system, derived systems should automatically close when the base system closes; users should not have to manually close them, although closing them should do no harm.

SystemConverter should also be able to upgrade a single system into a HostControlSystem. Since HostControlSystems are always closeable, this may be another way to fix the issue: keep the current contract, but only support system upgrades, not conversions.

The way to set StrictHostKeyChecking

Thank you very much for the useful software.

Sorry it's a question rather than an issue.
I'm wondering how to specify some ssh configurations like setting no to StrictHostKeyChecking.
Could you direct me to any resource or document that I can refer ?

Best regards,
Hiro

Add HostAccessor

We need a HostControlSystem provider that can work for local and remote hosts. Basically, make RemoteHostAccessor more generic by removing the credentials.

Also consider getting HostControlSystem instances via a factory method like the other systems. We should try to preserve the discoverability that exists with HostControlSystems#getDefault and HostControlSystems#openRemote.

Fix close() race conditions on SSH systems

Currently, closed checks are susceptible to a check-then-act race condition. While this isn't too bad (when they actually go to use the connection, there will be an error), it would be better to throw the correct Closed*SystemException. We can also close the system while operations are in progress, which is a more serious error.

Make close either block until all pending operations are complete or queue the closure, but avoid closing the connection until all pending operations are complete.

Size attribute is incorrect for empty remote files

SshPosixFileAttributes#size() return -1 if getSize() returns 0. This is obviously incorrect for empty files and is a holdover from a previous implementation when getSize() could return null if the size was not available.

Large command output can cause integer overflow

Repro: Commands.execute(Commands.get("yes"));

The problem is when SharedByteArrayStream attempts to resize and buffer.length * multiplier overflows.

Caused by: java.io.IOException: exception while copying streams 
    at com.palantir.giraffe.internal.CommandFutureTask$ExceptionListener.onException(CommandFutureTask.java:127)
    at com.palantir.giraffe.internal.ProcessStreamHandler$1.onFailure(ProcessStreamHandler.java:106)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
    at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
    at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
    at java.util.concurrent.FutureTask.setException(FutureTask.java:247)
    at java.util.concurrent.FutureTask.run(FutureTask.java:267)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NegativeArraySizeException
    at com.palantir.giraffe.internal.SharedByteArrayStream.resize(SharedByteArrayStream.java:241)
    at com.palantir.giraffe.internal.SharedByteArrayStream.access$1100(SharedByteArrayStream.java:25)
    at com.palantir.giraffe.internal.SharedByteArrayStream$SharedOutputStream.write(SharedByteArrayStream.java:153)
    at com.palantir.giraffe.internal.StreamCopier.call(StreamCopier.java:36)
    at com.palantir.giraffe.internal.StreamCopier.call(StreamCopier.java:8)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    ... 3 more

Migrated from GIRAFFE-21

Remove BaseSsh* classes

These are left over from an abstraction layer that is no longer relevant and probably won't be useful in the future.

Include troubleshooting information in the documentation.

Currently, this area is lacking.

Discuss MoreFiles.copyLarge in Effective Giraffe

MoreFiles.copyLarge and similar methods can be much faster than alternatives in Files, especially for remote-to-local and local-to-remote operations. These should be mentioned in the docs, since they are too hard to find by just browsing the APIs.

Convert builds to Circle 2

CircleCI is disabling 1.0 style builds at the end of August 2018. Update the project to use Circle 2.0 configuration before then so it can continue building.

Probably want a 3 job workflow: build and integration run in parallel, and then a publish job runs on tags and publishes to Bintray and GitHub releases.

Keberos principal set incorrectly?

In at least one environment, we had to specify the principal name both via the Giraffe API and via sun.security.krb5.principal property.

Expose all Krb5LoginModule parameters

Currently, we are limited by what we can do for Kerberos. This would give people the flexibility to do what they need for Kerberos

Configurable retry count for SSH authentication

i'm using giraffe for password-based SSH auth
and i want to try to connect 3 times before giving up
is there a way to do this nicely?
(specifically, i'm trying to make the try-3-times part clean)

This will probably an environment option in the map passed to newFileSystem / newExecutionSystem, which means that that map needs to be more obvious for users. Consider this when redesigning how to get HostControlSystems in #15.

Rename SSH credentials

From @ajlake: SSH credentials all extend AbstractSshCredential but then take common names like PasswordCredential that might conflict with other systems or user classes. Rename these concrete credentials to PasswordSshCredential or SshPasswordCredential.