gregwebs/shelly.hs

Haskell shell scripting

Shelly

Shelly provides a single module for convenient systems programming in Haskell.

• Shelly is aimed at convenience and getting things done rather than being a demonstration of elegance.
• It has detailed and useful error messages.
• It maintains its own environment, making it thread-safe.
• It has low memory usage: It has
  • run_ and other underscore variants that do not return stdout,
  • runFoldLines to run a fold operation over each line rather than loading all of stdout into memory,
  • runHandle and runHandles for complete control over handles.

The focus of this library on convenience combined with good error messages should make shelly approachable for newer users of Haskell.

More shelly packages

The shelly-extra package has some additional functionality that requires additional dependencies, currently including a convenient concurrency/futures implementation.

Examples

• A small deployment script
• Yesod development installer
• cabal-meta, a haskell install tool
• antigen-hs, a zsh plugin manager

Blog Posts

• Shelly automation with Literate Haskell

Testimonials

• A beginning Haskeller does automation

Help

• google group for Haskell shell scripting

Alternatives

Haskell shell scripting libraries

• HSH: A good alternative if you want to mixup usage of String and ByteString rather than just use Text.
• HsShellScript: Has extensive low-level shell capabilities.
• shell-conduit: Efficient streaming via conduits. Makes some portability sacrifices by
  • encouraging one to just use the shell instead of cross-platform Haskell code, and
  • encouraging one to use a convenience function that searches the PATH at compile-time.
• shell-monad: Compile Haskell code down to shell script. This is a different approach from all the rest of the libraries. Writing your script is not as user-friendly as the other Haskell libraries, but it nicely solves the deployment issue.
• shh: Shell-like syntax with native piping. Can be used from GHCi as an interactive shell replacement.
• turtle: In some sense a redesign of Shelly designed for beginner-friendliness.

HSH, HsShellScript and shh (unlike Shelly currently) implement very efficient mechanisms for piping/redirecting in the system. turtle, like Shelly offers folding as a way to efficiently deal with a stream.

None of the alternatives to Shelly offer command tracing. For some this is an absolutely critical feature, particularly given that Haskell does not yet offer up stack traces.

Haskell file-finding supplements

• find-conduit: Uses conduits, similar speed to GNU find.
• FileManip: Uses Lazy IO.

Shelly's finders load all files into memory. This is simpler to use if you control the filesystem structure and know the system is bounded in size. However, if the filesystem structure is unbounded it consumes unbounded memory.

Shell commands with richer input/output

Shelly does not change the nature of shell scripting (text in, text out). If you want something more revolutionary you might try these:

• PowerShell is probably the best known.
• A Haskell project using typed JSON.
• RecordStream using untyped JSON.

Usage

Shelly's main goal is ease of use. There should be a primitive for every shell operation you need so you can easily build abstractions, so there are many of the usual file and environment operations.

There are 2 main entry points for running arbitrary commands: run and cmd. They take a FilePath as their first argument. run takes a [Text] as its second argument. cmd takes a variadic number of arguments, and they can be either Text or FilePath.

Fun Example: shows an infectious script: it uploads itself to a server and runs itself over ssh. Of course, the development machine may need to be exactly the same OS as the server.

I recommend using the boilerplate at the top of this example in your projects. This includes setting line buffering if you are dealing with text and not binary data.

    {-# LANGUAGE OverloadedStrings #-}
    {-# LANGUAGE ExtendedDefaultRules #-}
    {-# OPTIONS_GHC -fno-warn-type-defaults #-}

    import Shelly
    import System.IO
    import Data.Text as T
    default (T.Text)

    main :: IO ()
    main =  do
      hSetBuffering stdout LineBuffering
      shelly $ verbosely $ do
        host <- run "uname" ["-n"]
        if T.stripEnd host == "local-machine"
          then do d <- cmd "date"
                  c <- escaping False $ cmd "git" "log -1 | head -1 | awk '{print $2}'"
                  appendfile "log/deploy.log" $ T.intercalate " - " [T.stripEnd d, c]
                  uploads "my-server:/remote/path/" ["deploy"]
                  sshPairs_ "my-server" [("cd", ["/remote/path"]), ("./deploy", [])]
          else do
                cmd "./script/angel"

    -- same path on remote host
    -- will create directories
    uploads :: Text -> [Text] -> Sh ()
    uploads remote locals = rsync $ ["--relative"] ++ locals ++ [remote]

    rsync :: [Text] -> Sh ()
    rsync args = run_ "rsync" $ ["--delete", "-avz", "--no-g"] ++ args

Variadic arguments to cmd

Yes, as seen above you can write variadic functions in Haskell quite easily, you just can't compose them as easily. I find cmd to be more convenient, but I often use run and command variants when I am building up abstractions. Building up abstractions with cmd will require type signatures.

    -- easy signature, but only allows one argument
    let cabal = cmd "cabal" :: Text -> Sh Text

    -- more complex signature that allows partial application of cmd
    let cabal = cmd "cabal" :: Shelly.ShellCmd result => result

Escaping

By default, all commands are shell escaped. If you want the shell to interpret special characters such as *, just use escaping False $ do ....

Using Text and FilePath together

Shelly's usage of Text means you may need to convert between Text and FilePath sometimes. This should be infrequent though because:

• cmd will convert FilePath to Text.
• The </> and <.> combinators convert Text into a FilePath automatically.

Manual conversion is done through toTextIgnore or toTextWarn.

Thread-safe working directory and relative paths

Command cd does not change the process working directory (essentially a global variable), but instead changes the shelly state (which is thread safe). All of the Shelly API takes this into account, internally shelly converts all paths to absolute paths. You can turn a relative path into an absolute with absPath or canonic or you can make a path relative to the Shelly working directory with relPath.

Good error messages

Haskell's #1 weakness for IO code is a lack of stack traces. Shelly gives you something different: detailed logging. In most cases this should be more useful than a stack trace. Shelly keeps a log of API usage and saves it to a .shelly directory on failure. If you use shellyNoDir, the log will instead be printed to stderr. This is in addition to the verbosely settings that will print out commands and their output as the program is running. Shelly's own error messages are detailed and in some cases it will catch Haskell exceptions and re-throw them with better messages.

If you make your own primitive functions that do not use the existing Shelly API, you can create a wrapper in the Sh monad that use trace or tag to log what they are doing. You can turn tracing off (not generally recommended) by setting tracing False.

shelly.hs's Issues

[Question] MonadCatch instance to catch lastStdErr: worthwhile?

Hi Greg,

long time no chat.

At the moment we define an instance of MonadCatch this way:

https://github.com/yesodweb/Shelly.hs/blob/master/src/Shelly/Base.hs#L85

In production though, what I find useful sometime is to report back not only the exception but also the lastStderr, to give the user even more diagnostics.

Do you think that semantically it might be worthwhile to add this inside the MonadCatch definition, or we should probably leave this out for the user to decide what to do with the stderr case by case?

Bye,
Alfredo

Build failure with exceptions-0.6

There's a new version of the exceptions library on Hackage:
https://hackage.haskell.org/package/exceptions-0.6

As there is no upper bound specified for the version of exceptions in shelly.cabal, the newest version will be used by default. This leads to the following errors during cabal build (GHC 7.8.2):

Building shelly-1.5.3...
Preprocessing library shelly-1.5.3...
[1 of 5] Compiling Shelly.Base      ( src/Shelly/Base.hs, dist/build/Shelly/Base.o )

src/Shelly/Base.hs:84:3:
    ‘mask’ is not a (visible) method of class ‘Catch.MonadCatch’

src/Shelly/Base.hs:86:3:
    ‘uninterruptibleMask’ is not a (visible) method of class ‘Catch.MonadCatch’

Building with exceptions-0.5 still works fine and I'm not sure what change is causing this issue: looking at the source it seems that these class methods should be exported in 0.6 as well.

`rm_f` deletes destination file for symlink.

rm_f deletes destination file for symlink.

Instead of deleting the mere file representing the symbolic link, the symbolic link gets resolved into the path it's pointing at (presumably the absPath in absPath f >>= liftIO . removeFile) and the fully resolved path is then removed.

If this is the intended behaviour, it should be documented. It is not the normal behaviour of rm -f which would just delete the link.

Shelly dies silently when used inside inotify event handler

Brief description

• ghc 7.6.3
• shelly 1.5.4.1
• linux 3.12-1 x86_64

I have written a small application as a exercise to show changes made to audit
logs. Basically the program uses inotify to check when
/var/log/audit/audit.log changes and runs a handler which uses shelly to run
the ausearch tool to extract recent audits from that log.

What happens is that when ausearch is run using run nothing happens and all
commands after are not executed.

I have tried:

• Running shelly as verbose
• Echo result from the run function
• Echo stderr from the lastStderr function

Whatever I do, nothing happens after the ausearch command is executed.

I have included what output I'm expecting, the output I'm getting inside the
inotify handler and the output I'm getting inside main loop.

At the bottom of this issue there's an example that reproduce the error complete with
cabal file and source.

I'm new to shelly, inotify and systems programming so I have probably
made a mistake in how I should use these libraries.

Thanks in advance!

Expected output

This is what I see if I run the command directly in bash:

rzetterberg@mycomputer:~/armorwatcher$ sudo ausearch --start recent -r
type=USER_START msg=audit(XXX): pid=X uid=X auid=X ses=X  msg='op=PAM:session_open acct="X" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/X res=success'

.... All the other audits .....

I'm using sudo because the audit log has these permissions:

rzetterberg@mycomputer:~$ sudo ls -la /var/log/audit/audit.log
-rw------- 1 root root 2803865 Jun 15 17:31 /var/log/audit/audit.log

Actual output

This is how I'm executing my haskell program:

rzetterberg@mycomputer:~/armorwatcher$ sudo dist/build/armorwatcher-app/armorwatcher-app

This is what I see when the command is executed via shelly inside the
inotify handler:

Audit changed, searching log
ausearch --start recent -r

This is what I see if I run the same shelly snippet in my main loop:

Audit changed, searching log
ausearch --start recent -r
armorwatcher-app: 
Ran commands: 
echo 'Audit changed, searching log'
echo 'ausearch --start recent -r'
ausearch --start recent -r
which ausearch

Exception: error running: ausearch --start recent -r
exit status: 1
stderr: 

Minimal working example

This example was run using GHC 7.6.3 inside a cabal sandbox. Here are
Main.hs and cabal file used to produce the sandbox and building the
application.

cabal file

name:                armorwatcher
version:             0.1.0.0
license-file:        LICENSE
author:              Richard Zetterberg
maintainer:          [email protected]
category:            System
build-type:          Simple
cabal-version:       >=1.18

executable armorwatcher-app
    main-is:         Main.hs

    ghc-options:     
        -Werror -Wall -threaded

    build-depends:
        base >=4.6 && <4.7,
        hinotify >=0.3 && <0.4,
        shelly >=1.5 && <1.6,
        unix >=2.7 && <2.8

    default-language:
        Haskell2010

Main.hs

{-# LANGUAGE OverloadedStrings #-}

module Main where

import           Control.Monad (forever, void)
import           Control.Concurrent (threadDelay)
import           System.INotify (INotify, EventVariety(..), Event(..), WatchDescriptor)
import qualified System.INotify as INotify
import           System.Posix.Signals (Handler(..), sigINT)
import qualified System.Posix.Signals as Signals
import           Shelly

------------------------------------------------------------------------

main :: IO ()
main = do
    let watchee = "/var/log/audit/audit.log"

    putStrLn $ "Setting up watching of " ++ watchee

    inst       <- INotify.initINotify
    descriptor <- INotify.addWatch inst [Modify] watchee auditChanged

    let handler = Catch $ interruptApp inst descriptor

    void $ Signals.installHandler sigINT handler Nothing

    putStrLn "Setup complete, waiting for changes"

    forever $ threadDelay maxBound

auditChanged :: Event -> IO ()
auditChanged (Modified _ _) = shelly $ verbosely $ do
    echo "Audit changed, searching log"
    audits <- run "ausearch" ["--start", "recent", "-r"] 
    echo "Log searched"
    echo audits
auditChanged _ = return ()

interruptApp :: INotify -> WatchDescriptor -> IO ()
interruptApp inst descriptor = do
    INotify.removeWatch descriptor
    INotify.killINotify inst

    putStrLn "Watching removed, quitting!"

Test fails with exit code 127

One Mac OS X 10.6 with ghc v 7.0.4:

test $ ghc --make drain.hs
[1 of 1] Compiling Main             ( drain.hs, drain.o )
Linking drain ...
test $ ./drain
drain: error running test/drain.sh []: exit status 127:

One Ubuntu 10.04 with ghc v 7.4.1:

test# runhaskell drain.hs
drain.hs: error running test/drain.sh []: exit status 127:

Why can't FoldCallBack do IO?

That is, why not:

type FoldCallback a = (a, Text) -> IO a

At first glance, it looks like this would be ok based on where callbacks are used:

https://github.com/yesodweb/Shelly.hs/blob/0151031103a85e16129c8e10ed611ba933d594fc/Shelly.hs#L229

The context is that I want to launch several backgrounded jobs and monitor their output as it arrives (not forcing the promises / waiting till the complete). Ultimately I would probably like to add something new to Shelly.Background. But as a quick fix, as long as I can do IO from FoldCallback, then I can communicate the individual lines of output myself (e.g. on a Chan).

Alternatively, is there a reason that there isn't a lazy-IO variant of run that returns immediately with Lazy Text?

`appendToPath` appends to the end of $PATH

I am not sure whether this is a bug or the intended behavior. If I understand correctly, the system search $PATH from left to right. In that case, wouldn't it be better to give priority to the user added path, by putting it at the beginning of $PATH, instead of the system's default paths?

More effecient piping with large data

Having an issue with figuring out how to use handles. Also wanted to verify handles are the correct solution rather than say runFoldLines or the other alternative I can't recall at the moment.

Surely examples of how to do the two snippets below in Shelly would be useful to many people:

zcat large.mysql.gz | mysql largedb

cat bigfile | grep things

I guess those two are pretty much the same, though I think runFoldLines might be a better alternative for the second.

{-# LANGUAGE ExtendedDefaultRules #-}
{-# LANGUAGE OverloadedStrings    #-}
{-# OPTIONS_GHC -fno-warn-type-defaults #-}
import           Data.Monoid
import           Data.Text   as T
import           Shelly
default (T.Text)

main = shelly $ do
  -- I'm trying to do "zcat large.mysql.gz | mysql largedb" basically
  -- minimal reproduction:
  run "echo" ["a very large file"] -|- run_ "grep" ["file"] -- predictably runs out of memory
  -- runHandle "echo" ["test"] $ \h -> run "grep" ["test"] -- thought this was the equivalent

change argument order for finders

It's more idiomatic in haskell to write a predicate first and then a
source of the items. So let's change argument order in findWhen
and other functions. It becomes:

findWhen :: (FilePath -> Sh Bool) -> FilePath -> Sh [FilePath]

findFold :: (a -> FilePath -> Sh a) -> a -> FilePath -> Sh a

findDirFilter :: (FilePath -> Sh Bool) -> FilePath -> Sh [FilePath]

In findFold arguments are in sync with standard foldr.

Proposal: new function mkdirTree

It's actually already in the Shelly. It's slipped into the last upgrade. It was not my intention. I should have divided this pull request in two parts. But it's very nice function. Shell is all about strings, but FP is all about List's and Tree's. A new function

mkdirTree :: Tree FilePath -> Sh ()

creates an hierarchy of directories. I'm learning Scala, and Scala projects have very deeply nested structure. But with mkdirTree it's very easy to create a new project. Here is an example:

exec name artifact = mkdirTree $
    name # [
        "src" # [
            "main" # [
                "resources" # [],
                "scala" # leaves [artifact], 
                "java"  # leaves [artifact] 
            ],
            "test" # [
                "resources" # [],
                "scala" # leaves [artifact],
                "java" # leaves [artifact]
            ]
        ]
    ]
    where (#) = Node
          leaves = map (# []) 

FilePath can contain a chain of nodes. So artifact can be "org/code/my"

shelly 1.3.0.1 is returning output in reverse order

For many months now I've been using my "hours" script to report how many hours I have worked during the current month. In that code I have the following:

    activeTimelog <- run "org2tc" [T.pack (file opts)]
    ...
    setStdin activeTimelog

In 0.15 this worked fine, but with the latest 1.3 I've discovered that the lines of activeTimelog are now in reverse order.

What's odd is that I cannot reproduce this in a smaller case, so I'm wondering if you could take a peek at my code and see if there's anything I'm doing obviously wrong:

https://github.com/jwiegley/hours/blob/master/Main.hs#L142

rm_rf "" deletes current directory and all contents

When you give rm_rf an empty path, it deletes the present directory and all contents. This is highly unintuitive, and results in horrors when you are piping a stream of filenames that may be empty. I believe the correct behavior should be a no op.

The problem seems to be the result of converting "" to an absolute path which is the present folder. So the solution would be to leave the paths as relative or just handle the empty string specially...

Release GHC 7.10 compatible version of shelly-1.5.x

There's currently no recent shelly version that is compatible w/ GHC 7.10

However, shelly from Git is compilable w/ GHC 7.10.1RC:

$ cabal build
Building shelly-1.5.6...
Preprocessing library shelly-1.5.6...
[1 of 5] Compiling Shelly.Base      ( src/Shelly/Base.hs, dist/build/Shelly/Base.o )

src/Shelly/Base.hs:40:1: Warning:
    The import of ‘liftM’ from module ‘Control.Monad’ is redundant

src/Shelly/Base.hs:43:1: Warning:
    The import of ‘Applicative’
    from module ‘Control.Applicative’ is redundant

src/Shelly/Base.hs:50:1: Warning:
    The import of ‘Data.Monoid’ is redundant
      except perhaps to import instances from ‘Data.Monoid’
    To import instances alone, use: import Data.Monoid()
[2 of 5] Compiling Shelly.Find      ( src/Shelly/Find.hs, dist/build/Shelly/Find.o )

src/Shelly/Find.hs:14:1: Warning:
    The import of ‘Data.Monoid’ is redundant
      except perhaps to import instances from ‘Data.Monoid’
    To import instances alone, use: import Data.Monoid()

src/Shelly/Find.hs:67:5: Warning:
    This binding for ‘traverse’ shadows the existing binding
      imported from ‘Prelude’ at src/Shelly/Find.hs:11:1-32
      (and originally defined in ‘Data.Traversable’)
[3 of 5] Compiling Shelly           ( src/Shelly.hs, dist/build/Shelly.o )

src/Shelly.hs:124:1: Warning:
    The import of ‘Data.Monoid’ is redundant
      except perhaps to import instances from ‘Data.Monoid’
    To import instances alone, use: import Data.Monoid()
[4 of 5] Compiling Shelly.Lifted    ( src/Shelly/Lifted.hs, dist/build/Shelly/Lifted.o )

src/Shelly/Lifted.hs:103:1: Warning:
    The import of ‘Data.Monoid’ is redundant
      except perhaps to import instances from ‘Data.Monoid’
    To import instances alone, use: import Data.Monoid()

src/Shelly/Lifted.hs:116:1: Warning:
    Module ‘Control.Monad.Trans.Error’ is deprecated:
      Use Control.Monad.Trans.Except instead

src/Shelly/Lifted.hs:141:11: Warning:
    In the use of type constructor or class ‘Error’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:141:43: Warning:
    In the use of type constructor or class ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:142:16: Warning:
    In the use of data constructor ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:241:29: Warning:
    In the use of type constructor or class ‘Error’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:242:29: Warning:
    In the use of type constructor or class ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:243:18: Warning:
    In the use of type constructor or class ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:245:9: Warning:
    In the use of data constructor ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:246:41: Warning:
    In the use of ‘runErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"

src/Shelly/Lifted.hs:247:31: Warning:
    In the use of data constructor ‘ErrorT’
    (imported from Control.Monad.Trans.Error):
    Deprecated: "Use Control.Monad.Trans.Except instead"
[5 of 5] Compiling Shelly.Pipe      ( src/Shelly/Pipe.hs, dist/build/Shelly/Pipe.o )
In-place registering shelly-1.5.6...

Efficient writing of stdout to a file.

The other Haskell shell libs do connect file descriptors directly.

User want to run code like this without eating up RAM:

shelly $ run "cat" ["myfile"] >>= writefile "outfile"

To implement this right now you can run a shell command directly right now. Is this approach lacking for you because you will have to append strings together?

shelly $ escaping False $ run "cat myfile > outfile" []

Another approach besides directly hooking up file descriptors would be to provide a convenience function over runFoldLines possibly including something that makes a Conduit.

`which` fails for Read-Only file systems

I stumbled on a problem with Shelly on a super-computing system running, what I think is SUSE linux, but heavily customized. Basically which fails to find anything (i.e., echo), which is due to the following GHC bug:
https://ghc.haskell.org/trac/ghc/ticket/8741

Apparently when the filesystem is "Read Only" getPermissions breaks. It seems this will be fixed in ghc 7.8, but not backported to 7.6.x (according to link above). I don't know if this is fixable from your end, or what to do about it really. I was considering compiling 7.8 RC2 and using it from my ubuntu system (I compile statically linked binaries to use on the system).

Missing lower bound on MTL

Shelly does not specify a lower bound for mtl, but with only mtl-1.1 installed (which comes with ghc 7.4) compilation fails with

No instance for (Applicative (ReaderT (IORef State) IO))
arising from the 'deriving' clause of a data type declaration
Possible fix:
add an instance declaration for
(Applicative (ReaderT (IORef State) IO))

Missing environment variable

Hello,

After update to 0.13.0.1 getenv (I don't understand the need to deprecate fo a equivalent get_env_text) returns the name of environment variable instead of its value

-- | Fetch the current value of an environment variable. Both empty and
-- non-existent variables give empty string as a result.
get_env_text :: Text -> Sh Text
get_env_text k = getenv_def k ""

I think this should be

get_env_text k = getenv_def "" k

...

-- | Fetch the current value of an environment variable. Both empty and
-- non-existent variables give the default Text value as a result
getenv_def :: Text -> Text -> Sh Text
getenv_def d = get_env >=> return . fromMaybe d

Thanks,
Fabio

Ambiguous occurrence `catch'

Trying to install shelly i get:

Downloading shelly-1.5.3.1...
Configuring shelly-1.5.3.1...
Building shelly-1.5.3.1...
Preprocessing library shelly-1.5.3.1...
[1 of 5] Compiling Shelly.Base      ( src/Shelly/Base.hs, dist/build/Shelly/Base.o )
[2 of 5] Compiling Shelly.Find      ( src/Shelly/Find.hs, dist/build/Shelly/Find.o )
[3 of 5] Compiling Shelly           ( src/Shelly.hs, dist/build/Shelly.o )
[4 of 5] Compiling Shelly.Lifted    ( src/Shelly/Lifted.hs, dist/build/Shelly/Lifted.o )

src/Shelly/Lifted.hs:364:12:
    Ambiguous occurrence `catch'
    It could refer to either `Prelude.catch',
                             imported from `Prelude' at src/Shelly/Lifted.hs:100:1-34
                             (and originally defined in `System.IO.Error')
                          or `Control.Exception.Lifted.catch',
                             imported from `Control.Exception.Lifted' at src/Shelly/Lifted.hs:107:1-31

src/Shelly/Lifted.hs:581:12:
    Ambiguous occurrence `catch'
    It could refer to either `Prelude.catch',
                             imported from `Prelude' at src/Shelly/Lifted.hs:100:1-34
                             (and originally defined in `System.IO.Error')
                          or `Control.Exception.Lifted.catch',
                             imported from `Control.Exception.Lifted' at src/Shelly/Lifted.hs:107:1-31
Updating documentation index /home/arash/.cabal/share/doc/index.html
cabal: Error: some packages failed to install:
shelly-1.5.3.1 failed during the building phase. The exception was:
ExitFailure 1

$ cabal --version    
cabal-install version 0.14.0
using version 1.14.0 of the Cabal library 

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.4.1

$ uname --all
Linux arash-spotify-T440s 3.11.0-20-generic #35~precise1-Ubuntu SMP Fri May 2 21:32:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Is ghc >= 7.6 mandatory for shelly?

command1 and command1_ change the order of the arguments

In particular, I was creating a command that required 1 argument for using tar -

untar :: T.Text -> Sh ()
untar a = command1_ "tar" ["-xaf"] a []

When I tried to use this with untar "archive.tar.gz" it would fail when Shelly would try to run tar archive.tar.gz -xaf, which doesn't work because tar requires arguments in a specific order. There are other commands in unixland that require the arguments to be in a specific order as well. I would think that command1 and command1_ would respect the order the arguments were passed to them.

It's pretty easy to find in the source where this is happening, and it seem rather explicit. I could certainly fix it and send over a pull request, but I wanted to see if there was some reason why it was this way, or if it should stay like that for some reason.

"Shellish" behaviour on Mac OS X 10.8 and GHC 7.6.3

Hello guys,

in the past (I'm the guy of the "Testimonials" link) I've used Shelly without problems (Linux and GHC 7.4, I'm pretty sure), but now on Mac OS X 10.8 and GHC 7.6.3 I'm experiencing the old Shellish behaviour, namely the output is visible only after the command has been completed. To demostrate this (at least on my particular environment) I've whipped up this small program:

{-# LANGUAGE OverloadedStrings #-}
import Shelly
import Data.Text as T
import Data.Monoid

cabal :: [Text] -> Sh Text
cabal = run "cabal"

sandboxCreation :: Sh Text
sandboxCreation = undefined

main :: IO ()
main = shellyNoDir . verbosely $ do
    echo $ toTextArg "This will be printed immediately"
    initResult <- cabal ["update"]
    sandboxCreation
    echo "Finished! Now let's hack some Haskell!"

What happens is that the echo is printed immediately, whilst the program remain "stuck" on cabal update for a while, and then it print in "burst":

Downloading the latest package list from hackage.haskell.org
Main.hs:
Ran commands:
echo 'This will be printed immediately'
echo 'cabal update'
cabal update

Exception: Prelude.undefined

My first thought was that something changed codewise since I've used Shelly, or it's the platform/ghc version to do the difference.
Any idea?

Thanks!
A.

A module to expose all commands in PATH

I've been experimenting with this approach, using template-haskell.

It allows you to just write git ["status"] or date [] or rm ["file"]. For me this changes scripts from "this is a bit tedious" to "this feels more bash-convenient".

Would you be interested in adding such a thing to shelly? It takes some seconds to compile the module (maybe 10) that uses generateBinaries, and makes a rather large haddock output, so you might want to make it a flag or something.

API that returns error code

Although I certainly see how lastStderr can be useful, I find it a bit weird not to have one runner comand to rule them all:

(out, err, retCode) <- runGeneric cmd [arg1, arg2] stdin

If you are interested in all of them, this might be convenient.

Thread indefinitely blocked in an MVar in "runHandle"

Hi Greg,

I hope you are well! I'm opening this because I have spotted a potentially nasty bug we have been trying to track down in production for several days, and finally I arrived at the bottom of it. Short story: we have a programs which runs perfectly fine on my Mac OS X machine, and its tests passes both on Mac and on our production CentOS box. But when we were going to run the system in production, the system was meant to call "ffmpeg" to perform some video transcoding, but we were instead presented with the dreadful "Thread indefinitely blocked in an MVar". After trying to isolate the problem I have tried to focus on this function, at the core of the "run/run_" machinery:

runHandle exe args withHandle =
  runHandles exe args [] $ \_ outH errH -> do
    errVar <- liftIO $ do
      errVar' <- newEmptyMVar
      _ <- forkIO $ transferLinesAndCombine errH stderr >>= putMVar errVar'
      return errVar'

    res <- withHandle outH
    errs <- liftIO $ takeMVar errVar

    modify $ \state' -> state' { sStderr = errs }
    return res

what immediately struck me was the forkIO line, which dangerously reminded me of one of the MVar's pitfall described here:

http://blog.omega-prime.co.uk/?p=92
http://blog.ezyang.com/2011/07/blockedindefinitelyonmvar/
http://chimera.labs.oreilly.com/books/1230000000929/ch08.html#sec_conc-mvar-errors

Basically if transferLinesAndCombine throws an uncaught exception, the entire io action dies and the RTS detects the MVar doesn't have enough memory references attached to it, thus can't be never filled (the next putMVar will never be run), thus the exception. I have tried to modify the code this way (ugly and experimental pof, of course!):

runHandle exe args withHandle =
  runHandles exe args [] $ \_ outH errH -> do
    appendfile "/var/www/html/log.txt" "runHandle 1"
    errVar <- liftIO $ do
      errVar' <- newEmptyMVar
      _ <- forkIO $ do
        res <- transferLinesAndCombine errH stderr `catch` \(e :: SomeException) -> do
               return (T.pack . show $ e)
        TIO.appendFile "/var/www/html/log.txt" res
        putMVar errVar' res

      return errVar'

    res <- withHandle outH
    appendfile "/var/www/html/log.txt" "runHandle 2"
    errs <- liftIO $ takeMVar errVar
    appendfile "/var/www/html/log.txt" "runHandle 3"

    modify $ \state' -> state' { sStderr = errs }
    appendfile "/var/www/html/log.txt" "runHandle 4"
    return res

and indeed now, capturing ALL the exceptions, we are able to complete the take/put MVar dance. This is the content of my log, including the exception being thrown:

runHandle 1
runHandle 2
fd:11: hGetLine: invalid argument (invalid byte sequence)
runHandle 3
runHandle 4

So in this particular case it seems that the problem is related to Text.IO's hGetLine, even though I can't explain why this is not caught by yours catchIOErrors (I really think it should be an IOError). Reading here:

http://comments.gmane.org/gmane.comp.lang.haskell.cafe/103547

it might even suggest the problem might be with machine locale's (that might explain why was working on my Mac but not on an EC2 CentOS), but nevertheless I still think it's a bug the fact this uncaught exception can hang the entire program. What you reckon?

Thanks again for this fantastic library, is getting day after day an important piece of our dev stack :)

Alfredo

Path and extension manipulation

I've been happily using Shelly for a bit now, great project!

I'm missing a few FilePath related things from System.FilePath ( http://hackage.haskell.org/packages/archive/filepath/1.3.0.1/doc/html/System-FilePath-Posix.html#g:2 ) such as dropExtension and dropFileName. Would you be okay with adding these for Shelly's FilePath?

I can make a PR out of it if you think it's a good idea.

Cheers!

More general types for exit commands

Shelly's exit commands should be polymorphic, rather than returning Sh (). I've found myself writing code like this to get around it, which is unfortunate:

errorExit "foo" >> undefined

bug in relative path handling in find

ghci started from a location that contains a `tmp/cabal-dev' subdir:

# ghci -XOverloadedStrings
Prelude> import Shelly
Prelude Shelly> shelly $ cd "tmp" >> find "cabal-dev"
...
Exception: cabal-dev/cabal.config: getSymbolicLinkStatus: does not exist (No such file or directory)

How to set the PATH variable?

I gathered from the future plans that there is a problem with the way $PATH is handled. I'd like to know if the problem only concerns shelly which implementation or if appendToPath should be avoided completely?

I tried the following code:

absPath ( "dist"</>"build"</>"bnfc") >>= appendToPath
which "bnfc" >>= inspect
cmd "which" "bnfc"
cmd "env"
cmd "bnfc"

and the result is puzzling me: Shelly.which returns Nothing (which, if I understood correctly is expected) but although the path in cmd "env" is corect and cmd "which" "bnfc" print the expected result, cmd "bnfc" still returns exit status 127 (command does not exists)

Is there a workaround for this? I tried cmd "export" ... but then it can't find the export command.

Spurious newline being added

This line and the one before it shows that a newline is intentionally added to output. However, this creates for some weird conditions.

For example, these should not be the same output:

λ> S.shelly $ S.silently $ do { v <- S.run "/usr/bin/echo" ["-n", "hello"]; S.liftIO $ print v }
"hello\n"

λ> S.shelly $ S.silently $ do { v <- S.run "/usr/bin/echo" ["hello"]; S.liftIO $ print v }
"hello\n"

This is also hard to work around because if a newline is explicitly added to the output, it isn't just appended to the one Shelly adds, it replaces it:

λ> S.shelly $ S.silently $ do { v <- S.run "/usr/bin/echo" ["-n", "-e", "hello\\n"]; S.liftIO $ print v }
"hello\n"

...so I can't just always strip the last newline, because it won't always be one that Shelly added.

Can we somehow conditionalize Shelly adding the last newline, much like we do for Shelly printing its output?

Why might `shelly undefined` segfault?

Hey there,

I am debugging a very, very strange error I've come across in IHaskell. Namely, running the following code:

import Shelly
shelly undefined

causes (I think) a segfault.

This happens with any version of Shelly == 1.5.*.

Does shelly undefined do anything unsafe that might cause a segfault? IHaskell does a lot of bizarre messing with standard handles such as stdin and stdout, so there's a chance that some invariant Shelly relies upon could be broken. This seems to happen only on Ubuntu, I cannot reproduce it on my Mac. The user who reported it said it started with GHC 7.8.3 but I have not tested it on Ubuntu with earlier GHC versions.

Any ideas?

Cannot access an inherited pipe

I don't understand why this error is happening. The following code seems to trigger it:

{-# LANGUAGE OverloadedStrings #-}
import Shelly

main = shellyNoDir $
  runHandles "bash" ["test.sh"] handles doNothing
  where handles = [InHandle Inherit, OutHandle Inherit, ErrorHandle Inherit]
        doNothing _ _ _ = return ""

with test.sh:

#!/bin/bash
echo hi

This seems like a bug?

Incorrect documentation for writefile and appendfile

The documentation for writefile and appendfile claims that they use lazy Text, but according to Hackage, they use strict text (which is a change from prior versions of Shelly). The documentation should be fixed.

Potential problem with process 1.2

process-1.2 added a new field "delegate_ctlc" which breaks Shelly. Example excerpt:

Exception: src/Shelly.hs:(325,72)-(336,9): Missing field in record construction System.Process.Internals.delegate_ctlc

The solution would be:

a) Add an upper bound in the Cabal manifest
b) Do the #if min trick as you already did for the "group" field :)

Definition for handle_sh is wrong

At the moment, handle_sh is an exact clone of catch_sh. I haven't fixed it since with some of my upcoming changes, you can just import Control.Exception.Lifted and use handle.

Find commands build up large list

FileManip uses lazy IO to avoid this. This is certainly an attractive option for the short term. In the long term I want to consider using strict streaming.

Support colorized output

Sorry if I'm missing something (extremely new to Haskell), but I can't figure out how to have a shell command print color. For example:

main = escaping False $ do
  run_ "rspec" ["--color"]

I can use rawSystem which does output in color, but I lose the ability to use pipes (afaik). I'd also like to keep everything using shelly. Anything I'm missing or is it straightforward to add color support?

How to interpret stdout as utf8?

If I run something like

runGit d "cat-file" ["commit", ref ]

but the environment has a non-utf8 locale set such as LANG=C then I get

Exception: fd:5: hGetLine: invalid argument (invalid byte sequence)

What's the recommended way to run a command which is known to always output utf8 to its stdout/stderr, regardless of any LANG-setting?

Build failure with mtl 2.2

[1 of 4] Compiling Shelly.Base ( src/Shelly/Base.hs, dist/build/Shelly/Base.o )

src/Shelly/Base.hs:48:43:
Module Control.Monad.Reader' does not exportrunReaderT'
Failed to install shelly-1.5.2

Allow traversal of symbolic links with find*

I'm writing a utility that needs to descend into all directories under a directory given by the user. It shouldn't matter whether these are symbolic links or real directory. However, with Shelly's findFold, I can see no way to descend into symbolic links to directories.

Export ShellCommand type class

To write this:

cabal = cmd "cabal"

or similar functions, cabal must be given a type signature. Because ShellCommand is currently not exported, this is impossible. Is it possible to export the ShellCommand type class?

No way to run a command with a terminal

Sometimes one may desire to provide a command with a terminal, for example so as to provide a shell to the user, or to have colorized command output.

Apparently this requires jumping through PTY-related hoops which Shelly doesn't do yet.

`inspect' crashes when used with `cmd'

The following code crashes with an excpetion (*** Exception: Exception: No Way! Shelly did not see this coming. Please report this error.)

{-# LANGUAGE OverloadedStrings, ExtendedDefaultRules #-}
import Shelly

main = shelly $ do
  inspect =<< cmd "echo"

I'm sure it's not a good idea to right such code, but I suppose you would like to know that it happend anyway :)

Unknown exception on cat .zshrc

In ghci:

shelly $ run "cat" [".zshrc"]

Output:

...
bindkey "\e[5~" history-beginning-search-backward-end # PageUp
#k# search history forward for entry beginning with typed text
bindkey "\e[6~" history-beginning-search-forward-end  # PageDown

# bindkey -s '^l' "|less\n"             # ctrl-L pipes to less
# bindkey -s '^b' " &\n"                # ctrl-B runs it in the background

# insert unicode character

*** Exception: 
log of commands saved to: /home/niklas/.shelly/1.txt
Exception: error running: cat .zshrc
exit status: 13
stderr: 
Prelude Shelly> 

/home/niklas/.shelly/1.txt is empty.

This is the .zshrc.

shelly-1.5.3 - ‘mask’ is not a (visible) method of class ‘Catch.MonadCatch’

cabal install shelly-1.5.3
Resolving dependencies...
Configuring shelly-1.5.3...
Building shelly-1.5.3...
Failed to install shelly-1.5.3
Build log ( /Users/vladimir/.cabal/logs/shelly-1.5.3.log ):
Configuring shelly-1.5.3...
Building shelly-1.5.3...
Preprocessing library shelly-1.5.3...
[1 of 5] Compiling Shelly.Base      ( src/Shelly/Base.hs, dist/build/Shelly/Base.o )

src/Shelly/Base.hs:84:3:
    ‘mask’ is not a (visible) method of class ‘Catch.MonadCatch’

src/Shelly/Base.hs:86:3:
    ‘uninterruptibleMask’ is not a (visible) method of class ‘Catch.MonadCatch’
cabal: Error: some packages failed to install:
shelly-1.5.3 failed during the building phase. The exception was:
ExitFailure 1

I use OS X 10.9.4

➜ ghc --numeric-version
7.8.3
➜ cabal --version
cabal-install version 1.22.0.1

I don't have any problems with 1.6.1.2, but another package try to install 1.5.3.
Any idea how to install it?

Passthrough exceptions

How can I prevent shelly from modifying exceptions?

shelly busy-waits for background tasks

using shelly (shelly-extra installed via cabal) to background-script some command-line tasks, i figured out that the haskell process occupies one CPU, if there are less tasks than the maximum number of allowed jobs.

not sure, how/if this can be resolved or if this is a bug in the interpreter (runghc) or in shelly. however this is quite a showstopper for me, as the load of my tasks is not equally distributed and i'm wasting quite some CPU cycles there

Be able to run "expanding" commands

Hi there,

I'm trying Shelly and it's great. Anyway, as regards to this code:

main = shelly $ verbosely $ do
    run_ "ls" ["*.cpp"]

I was expecting this command to list every cpp source file in the current dir, but it seems that the command is submitted as it is, therefore without "expanding" it, like any linux shell does.

What am I missing?

Regards,
Alfredo

Should 'silently' also mute stderr output?

Currently if you do something like:

{-# LANGUAGE OverloadedStrings #-}
module Foo where

import qualified Shelly as S

doSomething = S.shelly $ S.silently $ do
  S.run_ "ruby" ["-e", "$stderr.puts 'hello world'"]

main :: IO ()
main = doSomething

you get hello world printed to the terminal (as stderr).

Because of S.silently, I would expect there to not be any output out all. So I am wondering if S.silently should mute stderr as well (I could try to send a patch if this is desired), or if there is another way to mute stderr.

In my actual usecase, I am handling stderr by using something like err <- S.lastStderr, so I don't need/want it printed to the terminal.

Build failure on Cygwin / mingw with shelly-1.3.0.3

Downloading shelly-1.3.0.3...
Configuring shelly-1.3.0.3...
Building shelly-1.3.0.3...
Preprocessing library shelly-1.3.0.3...
[1 of 4] Compiling Shelly.Base      ( src\Shelly\Base.hs, dist\build\Shelly\Base.o )
[2 of 4] Compiling Shelly.Find      ( src\Shelly\Find.hs, dist\build\Shelly\Find.o )
[3 of 4] Compiling Shelly           ( src\Shelly.hs, dist\build\Shelly.o )

src\Shelly.hs:538:5: parse error on input `case'
Failed to install shelly-1.3.0.3

cp_r tries to create a strange path

I am not sure what line 932 in Shelly.hs does.
I tried to run cp_r template cps-map-fold, where both of these are directories. The template directory exists, but cps-map-fold does not. I got the following error message:

cp -r ./template cps-map-fold
mkdir /home/mattox/class/cs440/activities/cps-map-fold/template

Exception: /home/mattox/class/cs440/activities/cps-map-fold/template: createDirectory: does not exist (No such file or directory)

I traced the line to this one, in the cp_r function:

let toDir = if filename to == fromName then to else to FP.</> fromName

I'm not sure what was intended here, but my example worked when I changed it to

let toDir = to

Is it for copying into a directory that already exists with the same name?

If I get more time to look at this, I will try to suggest a real fix, but I thought it better to mention it now.

last command not always recorded in the log

not sure yet if this is only an issue for run

shelly $ echo "hello" >> run "grep" ["foo", "-"]

This will fail (non-zero exit code from grep) and the log will only have the echo.