amitt001/delegator.py

Subprocesses for Humans 2.0.

Delegator.py — Subprocesses for Humans 2.0

Delegator.py is a simple library for dealing with subprocesses, inspired by both envoy and pexpect (in fact, it depends on it!).

This module features two main functions delegator.run() and delegator.chain(). One runs commands, blocking or non-blocking, and the other runs a chain of commands, separated by the standard unix pipe operator: |.

Basic Usage

Basic run functionality:

>>> c = delegator.run('ls')
>>> print c.out
README.rst   delegator.py

>>> c = delegator.run('long-running-process', block=False)
>>> c.pid
35199
>>> c.block()
>>> c.return_code
0

Commands can be passed in as lists as well (e.g. ['ls', '-lrt']), for parameterization.

Basic chain functionality:

# Can also be called with ([['fortune'], ['cowsay']]).
# or, delegator.run('fortune').pipe('cowsay')

>>> c = delegator.chain('fortune | cowsay')
>>> print c.out
  _______________________________________
 / Our swords shall play the orators for \
 | us.                                   |
 |                                       |
 \ -- Christopher Marlowe                /
  ---------------------------------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||

Expect functionality is built-in too, on non-blocking commands:

>>> c.expect('Password:')
>>> c.send('PASSWORD')
>>> c.block()

Other functions:

>>> c.kill()
>>> c.send('SIGTERM', signal=True)

# Only available when block=True, otherwise, use c.out.
>>> c.err
''

# Direct access to pipes.
>>> c.std_err
<open file '<fdopen>', mode 'rU' at 0x10a5351e0>

# Adjust environment variables for the command (existing will be overwritten).
>>> c = delegator.chain('env | grep NEWENV', env={'NEWENV': 'FOO_BAR'})
>>> c.out
NEWENV=FOO_BAR

Installation

$ pip install delegator.py

✨🍰✨

delegator.py's Issues

Do not set PYTHONUNBUFFERED globally

delegator sets the PYTHONUNBUFFERED environment variable globally. It should modify the environment only for the subprocesses it creates.

This breaks other subprocess calls that do not expect it, e.g. pipenv shell on Windows.

See https://github.com/kennethreitz/pipenv/issues/435

feature_request(encoding): force UTF-8 for Windows users

1. Summary

In my opinion, UTF-8 should be the default encoding for Windows users. If developers of delegator.py don't want to do this, it would be nice to have an option for UTF-8.

2. Argumentation

I don't understand, why needs these lines for Windows users.

if sys.platform == 'win32':
    default_encoding = locale.getdefaultlocale()[1]
    if default_encoding is not None:
        encoding = default_encoding

More about locale.getdefaultlocale(). I have English Windows and Russian language for non-unicode programs. Python interpreter output:

>>> import locale
>>> locale.getdefaultlocale()
('en_US', 'cp1251')

I change my language for non-Unicode programs to English (United States) as recommend in this answer:

I restart Windows → I open Python Interpreter again:

>>> import locale
>>> locale.getdefaultlocale()
('en_US', 'cp1252')

Not UTF-8 again. The example shows that locale.getdefaultlocale() can't set UTF-8 for Windows.

3. Example

• SashaDelegator.py (in UTF-8):

import delegator

print(delegator.run('echo Саша Богиня!').out)

I run in console python SashaDelegator.py

4. Expected behavior

Саша Богиня!

5. Actual behavior

???? ??????!

This hack also doesn't help.

import _locale
import delegator

_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])

print(delegator.run('echo Саша Богиня!').out)

I get the same output.

And if I use hacks, I have problems with code quality tools.

6. Environment

• Windows 10 Enterprise LTSB 64-bit EN
• Python 3.7.0
• delegator.py 0.1.0
• PYTHONIOENCODING utf-8

Thanks.

`AttributeError` for `.chain` on Python3.6

Thanks for delegator.py; this library looks extremely helpful.

In playing around with the examples in the Readme, I ran into an AttributeError:

Jupyter console 5.2.0

Python 3.6.3 (default, Oct  4 2017, 06:09:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.



In [1]: import delegator

In [2]: c = delegator.chain('fortune | cowsay')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-4380c68b9225> in <module>()
----> 1 c = delegator.chain('fortune | cowsay')

/usr/local/lib/python3.6/site-packages/delegator.py in chain(command, timeout)
    237
    238 def chain(command, timeout=TIMEOUT):
--> 239     commands = _expand_args(command)
    240     data = None
    241

/usr/local/lib/python3.6/site-packages/delegator.py in _expand_args(command)
    225
    226         while True:
--> 227             token = splitter.get_token()
    228             if token:
    229                 command.append(token)

/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shlex.py in get_token(self)
    103             return tok
    104         # No pushback.  Get a token.
--> 105         raw = self.read_token()
    106         # Handle inclusions
    107         if self.source is not None:

/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shlex.py in read_token(self)
    134                 nextchar = self._pushback_chars.pop()
    135             else:
--> 136                 nextchar = self.instream.read(1)
    137             if nextchar == '\n':
    138                 self.lineno += 1

AttributeError: 'bytes' object has no attribute 'read'

Other examples seem to work fine; this seems to be a problem in chain only.

List-style commands ignore all but [0] element

I would except cmd2 to work and cmd1 to fail.

Both run -- cmd1 running the whole command line, and cmd2 just its first list element (no "-ltr").

cmd1 = 'ls -ltr'
cmd2 = ['ls', '-ltr']

print delegator.run(cmd1).out
total 148
-rw-r--r-- 1 chris 1414874935 128653 Aug 22 14:12 dump.txt
-rw-r--r-- 1 chris 1414874935 224 Aug 22 14:32 9655bd19-f3ab-4db0-b3bd-c9fa7f895b78
-rw-r--r-- 1 chris 1414874935 224 Aug 22 14:32 2c9165d7-5a14-4c09-b23f-7c21d2923466
...

print delegator.run(cmd2).out
dump.txt
9655bd19-f3ab-4db0-b3bd-c9fa7f895b78
2c9165d7-5a14-4c09-b23f-7c21d2923466
...

delegator v0.10.0 in Python 2.7.15

Monitoring progress of long-running commands

The README seems to imply that it is possible to use c.std_out to monitor the progress of long-running commands, but there is no documentation or other guidance as to how one might actually do that. Adding even the briefest of examples to the README would be extraordinarily helpful.

Does anyone have any suggestions as to how one might monitor the progress of Delegator-spawned long-running commands?

Cannot pass variables to delegator.run

.
.
.

for host,db,user in it.izip(db_host,db_name,db_user):
archive = db + "pg_dump" + timestamp + ".sql.gz"
location = path + archive

dump = delegator.run("pg_dump", "-h", host, "-U", user, db)
print dump.out
...
Instead of getting the output of my dump, I get this error:

    dump = delegator.run("pg_dump", "-h", host, "-U", user, db)
  File "/Users/richard/python-play/lib/python2.7/site-packages/delegator.py", line 317, in run
    c.run(block=block, binary=binary, cwd=cwd, env=env)
  File "/Users/richard/python-play/lib/python2.7/site-packages/delegator.py", line 185, in run
    popen_kwargs["env"].update(env)
ValueError: dictionary update sequence element #0 has length 1; 2 is required

how to pipe string to rest of commands

Hi! I would like to simplify my usage of subprocess and so i found your work. It would seem that it does what is advertised to do (just trying to say that is an amazing project) but documentation is a bit lacking ... i do not find a way to pipe a string to a custom chain (something like echo stdout_from_other_process | chain_of_shell_commands_that_will_run_in_my_pseudo_terminal). Could someone give me a hint about it? :)
Thank you!!

return code returns None when block=False

i can get return code when i run a command with block=True but i get None when i use block=False even it finishes his job

error when using chain()

Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux

import delegator
c = delegator.chain('fortune | cowsay')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.5/dist-packages/delegator.py", line 197, in chain
commands = _expand_args(command)
File "/usr/local/lib/python3.5/dist-packages/delegator.py", line 178, in _expand_args
if isinstance(command, (str, unicode)):
NameError: name 'unicode' is not defined

Reading `err` fails if nothing written to `stderr` in 0.0.9

0.0.9, Python 2.7 on Mac, err property is broken:

import delegator

# If you have something written to stderr, all is good
p1 = delegator.run('>&2 echo "error1"')
print p1.err

# This raises ValueError:
p2 = delegator.run('echo "error2"')
print p2.err

Output:

error1

Traceback (most recent call last):
  File "..../scratch_7.py", line 8, in <module>
    print p2.err
  File "..../.venv/lib/python2.7/site-packages/delegator.py", line 106, in err
    self.__err = self.std_err.read()
ValueError: I/O operation on closed file

This is ok in 0.0.8

command with long output hangs forever

How to reproduce:

>>> import delegator
>>> delegator.run("sh -c 'for((i=1; i<=100000; i++)); do echo $i; done'").out

this will hang forever

EMR cluster "ModuleNotFoundError: No module named 'delegator'"

I can't get my script to recognize that delegator.py has been installed. I'm running conda on the EMR cluster. This are the lines from my bootstrap.sh file:

MINICONDA_PACKAGE=miniconda.tgz
tar zxf ${MINICONDA_PACKAGE}
/opt/miniconda/bin/pip install delegator.py

But when I run my script which starts with
import delegator

Using this command to run the script
PYSPARK_PYTHON=/opt/miniconda/bin/python python3 from_athena.py

I get the error described in the subject line.

Any ideas on how to get it to inderstand that it is already installed?

Some strange code in the Command.return_code method

Hello! Thank you for the fantastic job. This is a your return_code method implementation:

    @property
    def return_code(self):
        if isinstance(self.subprocess, subprocess.Popen):
            return self.subprocess.returncode
        else:
            return self.subprocess.returncode
            raise RuntimeError('return codes can only be used for blocking commands.')

I think that the isinstance condition is not needed in this situation. And RuntimeError will never be raised.
Maybe did you mean something else?

daemon.py "SyntaxError: invalid token." "os.umask(022)"

I have a file test_delegator.py:

import delegator

print('hello')
delegator.run('ls')

If I run python -m mypackage.cli in a python3.6 venv, I get this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/peter/myapp/mypackage/test_delegator.py", line 1, in <module>
    import delegator
  File "/home/peter/myapp/venv-myapp/lib/python3.6/site-packages/delegator.py", line 6, in <module>
    import daemon
  File "/home/peter/myapp/venv-myapp/lib/python3.6/site-packages/daemon.py", line 70
    os.umask(022)   # Don't allow others to write
               ^
SyntaxError: invalid token

Let me know if you need more info.

Update:

It looks like the deamon package is not Python3 compatible:

$ python3.6                         

Python 3.6.0 (default, Dec 24 2016, 06:35:36) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.umask(022)
  File "<stdin>", line 1
    os.umask(022)
               ^
SyntaxError: invalid token
>>> 

Compared to python2.7

$ python2.7

Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.umask(022)
18
>>> 

python 3.6 unicode error

I user python 3.6

delegator.py", line 197, in chain
    commands = _expand_args(command)
  File "delegator.py", line 178, in _expand_args
    if isinstance(command, (str, unicode)):
NameError: name 'unicode' is not defined

Can I run a command in shell?

It looks like not support delegator.run("my_command", shell=True).

return_code accesses non-existent subprocess attribute

I tried to get c.return_code which in turn calls subprocess.returncode. Got:
AttributeError: 'PopenSpawn' object has no attribute 'returncode'

Looking at the call I saw that the pExpect module actually has the following:

subprocess.exitstatus

hang on chain if pipe nothing to awk

delegator.chain("printf '' | awk '{print $1}'") would hang, while
delegator.run("printf '' | awk '{print $1}'") wouldn't.
It seems that the command would block in pexpect：

^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-3-fc2320a6bab7> in <module>
----> 1 delegator.chain("printf '' | awk '{print $1}'")

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/delegator.py in chain(command, timeout, cwd, env)
    308             c.subprocess.sendeof()
    309
--> 310         data = c.out
    311
    312     return c

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/delegator.py in out(self)
    126             self.__out = self.std_out.read()
    127         else:
--> 128             self.__out = self._pexpect_out
    129
    130         return self.__out

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/delegator.py in _pexpect_out(self)
    114             result += self.subprocess.after
    115
--> 116         result += self.subprocess.read()
    117         return result
    118

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pexpect/spawnbase.py in read(self, size)
    439         if size < 0:
    440             # delimiter default is EOF
--> 441             self.expect(self.delimiter)
    442             return self.before
    443

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pexpect/spawnbase.py in expect(self, pattern, timeout, searchwindowsize, async_, **kw)
    339         compiled_pattern_list = self.compile_pattern_list(pattern)
    340         return self.expect_list(compiled_pattern_list,
--> 341                 timeout, searchwindowsize, async_)
    342
    343     def expect_list(self, pattern_list, timeout=-1, searchwindowsize=-1,

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pexpect/spawnbase.py in expect_list(self, pattern_list, timeout, searchwindowsize, async_, **kw)
    367             return expect_async(exp, timeout)
    368         else:
--> 369             return exp.expect_loop(timeout)
    370
    371     def expect_exact(self, pattern_list, timeout=-1, searchwindowsize=-1,

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pexpect/expect.py in expect_loop(self, timeout)
    111                 incoming = spawn.read_nonblocking(spawn.maxread, timeout)
    112                 if self.spawn.delayafterread is not None:
--> 113                     time.sleep(self.spawn.delayafterread)
    114                 if timeout is not None:
    115                     timeout = end_time - time.time()

KeyboardInterrupt:

My delegator version is 0.1.1

pipenv check fails inside python3-alpine

Originally posted as pypa/pipenv#2176
But @uranusjr suggested that this is a delegator issue: pypa/pipenv#2176 (comment)

$ python -m pipenv.help output

Pipenv version: '11.10.1'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

• 2.6: /usr/bin/python2.6

• 2.6: /usr/bin/python2.6

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 3.6: /usr/local/bin/python3.6

• 2.7.14: /usr/local/bin/python

• 2.7.14: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.14: /usr/local/bin/python2

• 2.7.14: /usr/local/bin/python2

• 3.6.5: /usr/local/bin/python3

• 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '15.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST '
                     '2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• TMPDIR
• TERM_PROGRAM_VERSION
• Apple_PubSub_Socket_Render
• LANG
• TERM_PROGRAM
• XPC_SERVICE_NAME
• XPC_FLAGS
• TERM_SESSION_ID
• SSH_AUTH_SOCK
• TERM
• __CF_USER_TEXT_ENCODING
• SHELL
• HOME
• LOGNAME
• USER
• PATH
• SHLVL
• PWD
• OLDPWD
• ZSH
• PAGER
• LESS
• LC_CTYPE
• LSCOLORS
• VIRTUAL_ENV_DISABLE_PROMPT
• LS_COLORS
• CLICOLOR
• GREP_COLOR
• EDITOR
• GPG_TTY
• GPG_AGENT_INFO
• PYTHONIOENCODING
• HOMEBREW_NO_ANALYTICS
• MANPAGER
• PIPENV_VENV_IN_PROJECT
• PIPENV_HIDE_EMOJIS
• PIPENV_COLORBLIND
• PIPENV_NOSPIN
• NVM_DIR
• NVM_CD_FLAGS
• PYENV_SHELL
• NODE_REPL_HISTORY
• NODE_REPL_MODE
• ERL_AFLAGS
• FZF_DEFAULT_COMMAND
• FZF_CTRL_T_COMMAND
• SOBOLE_DEFAULT_USER
• LC_ALL
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: true
• PIPENV_HIDE_EMOJIS: true
• PIPENV_COLORBLIND: true
• PIPENV_NOSPIN: true

Debug–specific environment variables:

• PATH: /Users/sobolev/.pyenv/shims:/Users/sobolev/.local/bin:/usr/local/opt/python2/libexec/bin:/usr/local/opt/gpg-agent/bin:/usr/local/sbin:/usr/local/share/npm/bin:/usr/local/opt/php56/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/opt/fzf/bin
• SHELL: /usr/local/bin/zsh
• EDITOR: /usr/local/bin/nano
• LANG: ru_RU.UTF-8
• PWD: /Users/sobolev

---

---

Expected result

I expect that pipenv check will work inside docker the same as locally.

Actual result

Locally:

» pipenv check
Checking PEP 508 requirements…
Passed!
Checking installed package safety…
All good!

Inside docker:

System check identified no issues (0 silenced).
No changes detected
Checking PEP 508 requirements…
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 678, in check
    three=three, python=python, system=system, unused=unused, args=args
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 2337, in do_check
    results = simplejson.loads(c.out)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/delegator.py", line 97, in out
    self.__out = self.std_out.read()
ValueError: I/O operation on closed file.

Steps to replicate

Dockerfile:

FROM python:3.6.5-alpine3.7

LABEL maintainer="[email protected]"
LABEL vendor="wemake.services"

ARG DJANGO_ENV=production
ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100 \
  PIPENV_COLORBLIND=true \
  PIPENV_NOSPIN=true


# System deps:

RUN apk update \
    && apk --no-cache add \
      bash \
      build-base \
      curl \
      gcc \
      gettext \
      git \
      libffi-dev \
      linux-headers \
      musl-dev \
      postgresql-dev \
      tini


# Creating folders, and files for a project:

WORKDIR /code
COPY . /code

# This is a special case. We need to run this script as entry point:
COPY ./docker/django/entrypoint.sh /docker-entrypoint.sh


# Project initialization:

RUN chmod +x "/docker-entrypoint.sh" \
  && pip install pipenv \
  && pipenv install $(test "$DJANGO_ENV" == production || echo "--dev") --deploy --system --ignore-pipfile

ENTRYPOINT ["/sbin/tini", "--", "/docker-entrypoint.sh"]

And run pipenv check.

Or download this template: https://github.com/wemake-services/wemake-django-template
And run: docker-compose run --rm web sh ./docker/ci.sh

Non-blocking processes leak file handles on wait calls

For whatever reason, starting in python 3.6, non-blocking subprocess calls now raise ResourceWarnings whenever they are garbage collected without having their open file handles explicitly closed (i.e. STDOUT, STDERR, STDIN). This can cause test failures, so they should be closed on block calls.

Another release?

Can we get another release? 0.1.1 went out in Sep 17 2018, but since then there have been some bug fixes that I'd like to use

module 'delegator' has no attribute 'chain'

import delegator

c = delegator.chain('ls | grep a')
c.out

---

AttributeError Traceback (most recent call last)
in
1 import delegator
----> 2 c = delegator.chain('ls | grep a')
AttributeError: module 'delegator' has no attribute 'chain'

module 'delegator' has no attribute 'run'

I'm using Python 3.6.3, I install delegator 0.0.3 using pip:

python3 -m pip install arrow delegator
Collecting arrow
  Downloading https://mirrors.aliyun.com/pypi/packages/54/db/76459c4dd3561bbe682619a5c576ff30c42e37c2e01900ed30a501957150/arrow-0.10.0.tar.gz (86kB)
    100% |████████████████████████████████| 92kB 9.3MB/s
Collecting delegator
  Downloading https://mirrors.aliyun.com/pypi/packages/89/7f/a5103ebd2cf4e3ba10dbea3646ca12793a1040da654c98be125237363f33/delegator-0.0.3.tar.gz
Collecting python-dateutil (from arrow)
  Downloading https://mirrors.aliyun.com/pypi/packages/4b/0d/7ed381ab4fe80b8ebf34411d14f253e1cf3e56e2820ffa1d8844b23859a2/python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 21.9MB/s
Collecting six>=1.5 (from python-dateutil->arrow)
  Downloading https://mirrors.aliyun.com/pypi/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, arrow, delegator
  Running setup.py install for arrow ... done
  Running setup.py install for delegator ... done
Successfully installed arrow-0.10.0 delegator-0.0.3 python-dateutil-2.6.1 six-1.11.0 

but installed delegator module has no attribute:

>>> import delegator
>>> type(delegator)
<class 'module'>
>>> delegator.run('ls')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    delegator.run('ls')
AttributeError: module 'delegator' has no attribute 'run'
>>> dir(delegator)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

I wonder if there is problem of my Python version or delagator?

Give command timeout

hi, is possible to give timeout to command?

equal to subprocess (python 3.3+)

from subprocess import STDOUT, check_output

output = check_output(cmd, stderr=STDOUT, timeout=seconds)

expect functionality only works when subprocess is flushing

This annoys the hell out of me, because Python doesn't flush stdout when printing by default.

Currently, interactive Python scripts must be run with -u to work correctly.

what is the difference for capturing "dir" and "python -V" commands?

I'm expermenting with delegator.py on Windows, trying to capture process output (whatever was printed to console).

Apparently, I can capture output of a console command, like this:

import delegator
c = delegator.run('dir c:') 
assert c.out.startswith(' Volume in drive C is')

However, when trying to capture output of python -V command, I get empty string while somehting like 'Python 3.6.0' is expected:

d = delegator.run('python -V') 
# expected 'Python 3.6.0'
assert d.out == ''

Problems using both binary STDIN and STDOUT

Dear Kenneth,

Introduction

First things first: While staying for a while with the elder sister of Delegator.py, envoy, we salute you for giving this another attempt to finally regain sanity on Python subprocess handling. We are really looking forward to the outcome, cheers!

After successfully using it for a basic program called formailx.py, we just tried to apply Delegator.py to a slightly more advanced scenario.

Problem

We want to pipe binary data to the STDIN channel of a process and read its binary output from the STDOUT channel.

It is about image conversion using ImageMagick, so let's assume the command we want to execute is

convert - png:-

and we are trying to run Delegator.py like

with open(tiff_file, 'rb') as tiff:
    proc = delegator.run(command)
    proc.send(tiff.read())
    png = proc.out

Following the documentation, we also tried various combinations of the block and binary parameters and attempted to call proc.block() after proc.send(), but everything without success yet.

Status quo

We either get RuntimeError: send can only be used on non-blocking commands. as an answer or the program stalls completely when accessing proc.out .

On the other hand, everything works fine when using the pure subprocess module, see run_command in "util.py" from Patent2net/P2N#26.

Help?

Can we humbly ask you to take a look into this?
Thanks in advance for your efforts!

With kind regards,
Andreas.

UnicodeEncodeError

Hi, I am trying to get standard out from running /usr/bin/stat with the delegator and I am met with
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 8: ordinal not in range(128)

I did c.out.encode('utf-8') and see this:

```File: \xe2\x80\x98/\xe2\x80\x99\n Size: 4096 \tBlocks: 8 IO Block: 4096 director .... ``

I further found the following line is responsible for the error:
result += self.subprocess.read().decode('utf-8')

Thank you.

reading std_out of non blocking commands

When I try:

c = delegator.run(['sleep', ' 10'], block=False)
c.out

The call blocks for 10 secs then produces "" as expected. However if I try:

c = delegator.run(['sleep', ' 10'], block=False)
c.std_out.read()

I get UnsupportedOperation: not readable

Is this expected? I had expected to be able to iterate over c.std_out to monitor progress.

Error getting subprocess error with non-blocking mode

Accessing command.err a second time causes an exception

In [1]: import delegator

In [2]: command = delegator.run('echo a', block=False)

In [3]: command.err
Out[3]: 'a\n'

In [4]: command.err
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-5e568e65e1e2> in <module>()
----> 1 command.err

~/.virtualenvs/service-mailer/lib/python3.6/site-packages/delegator.py in err(self)
    114             return self.__err
    115         else:
--> 116             return self._pexpect_out
    117
    118     @property

~/.virtualenvs/service-mailer/lib/python3.6/site-packages/delegator.py in _pexpect_out(self)
     82             result += self.subprocess.before
     83         if self.subprocess.after:
---> 84             result += self.subprocess.after
     85
     86         result += self.subprocess.read()

TypeError: must be str, not type

The cause is that self.subprocess.after becomes <class 'pexpect.exceptions.EOF'> after the first command.err

Issues with "non-blocking" commands

Not really sure how to title this, since I think my issue is just that some of the methods shown in the documentation just don't work, or perhaps it's not clearly explained what is meant by a 'non-blocking' command and I'm not using this correctly:

>>> import delegator
>>> c = delegator.run('sleep infinity', block=False)
>>> c.terminate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/delegator.py", line 176, in terminate
    self.subprocess.terminate()
AttributeError: 'PopenSpawn' object has no attribute 'terminate'
>>> c.send('SIGTERM', signal=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/delegator.py", line 173, in send
    self.subprocess.send_signal(s)
AttributeError: 'PopenSpawn' object has no attribute 'send_signal'

When block=False, self.subprocess becomes a pexpect.popen_spawn.PopenSpawn, which doesn't implement many methods from subprocess.Popen, including send_signal, terminate, and probably others.

/usr/bin/find /non/existent/path | /usr/bin/wc -l => c.out = 2 (should be 0)

/usr/bin/find /non/existent/path | /usr/bin/wc -l
linux command line return: 0 (correct)
delegator c.out = 2 (incorrect)

As per the above, if I run this on the linux command line, I get 0 (correct).
But when I pass this through delegator, I get 2.

path = '/non/existent/path'
the_command = f'/usr/bin/find {path} | /usr/bin/wc -l'
c = delegator.chain(the_command)
output = (c.out).rstrip()
print(output)
2

and for a path that exists, that returns 1900 on the linux command line, delegator returns 1901.

Unable to send binary data to a process' stdin

I've been trying to send binary data to a process' stdin with delegator but it blows up with a TypeError as follows:

In [1]: import delegator

In [2]: cmd = delegator.run("pngquant -", block=False, binary=True)

In [3]: cmd.send(b"")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-e2d15877a492> in <module>()
----> 1 cmd.send(b"")

~/.virtualenvs/paddy-test-gGhWdBGA/lib/python3.6/site-packages/delegator.py in send(self, s, end, signal)
    184                 return self.subprocess.communicate(s + end)
    185             else:
--> 186                 return self.subprocess.send(s + end)
    187         else:
    188             self.subprocess.send_signal(s)

TypeError: can't concat str to bytes

For now I've been able to work around the problem with:

    cmd.subprocess.send(image_bytes + b"\n")

Version 0.0.6 not in master branch & setup.py still says version='0.0.5'

In the "releases" page there is version 0.0.6, but it's missing from the master branch, it only exists as a tag. And the setup.py of the 0.0.6 tarball still says version='0.0.5'.

Permissions error when installing without sudo

Ubuntu 16.10, Python 3.5, pip3
$ pip3 install delegator.py
...
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/delegator.py'

I was under the impression that installing using pip with sudo is problematic and best avoided.

CWD option(s) to set the current working directory for command?

Goodday,

I don't see that delegator allows me to specify the directory that the command(s) must executed in, thus I was wondering how to set that in delegator?

I have the case/need where I'll be running from a Flask application with a far removed (directory wise) Ansible scripts that needs to be executed.

Add optional arguments to redirect stdout, stderr

Popen has optional arguments stdout, stderr to redirect stdout, stderr. Please, make it possible to pass them through delegator.run function.

Conflict between delegator and delegator.py

if I have both delegator and delegaror.py installed python will pick the first one and I'll get an error message saying AttributeError: module 'delegator' has no attribute 'run', is there a way to ensure the right module is being used when both of these packages are installed?

delegator.chain() adds an extra newline into pipes

Expected results

$ seq 4 | awk '{ print $0 " test"; }'
1 test
2 test
3 test
4 test

Actual results

$ python test.py 
1 test
2 test
3 test
4 test
 test

test.py

#!/usr/bin/env python
import delegator
c = delegator.chain("seq 4 | awk '{ print $0 \" test\"; }'")
print(c.out)

If I substitute delegator.run() for delegator.chain() it works correctly. Tested with v0.1.0, seems to happen in both python2 and 3

Only first argument is used

I just did pip3 install delegator.py and I cannot get multiple arguments to work. As an example:

import delegator

# as a single argument works fine
delegator.run(['echo hello']).out
# outputs: 'hello\n'

# as multiple arguments seems to only use the first one
delegator.run(['echo', 'hello']).out
# outputs: '\n', expected output: 'hello\n'

Memory leaking with delegator.chain()?

I get a memory leak when using certain commands with the chain function. I boiled it down to this simple test case

import delegator

def run_shell_chain(cmd):
	return delegator.chain(cmd).out.strip("\n")

for i in range(50000):
	output = run_shell_chain("iostat -m -y 1 1")
	print(output)

If I watch ps, I can see the memory footprint increase indefinitely.

I cannot reproduce it with a simple piped hello world command. (echo 'hello world' | sed 's/hello/goodbye/')

System info

>>> pkg_resources.get_distribution("delegator.py").version
'0.1.1'

$ python3 -V
Python 3.6.8

 ~$ uname -a
4.18.0-25-generic #26~18.04.1-Ubuntu SMP Thu Jun 27 07:28:31 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

different behavior between block/non-blocking command

Python 3.6.5 (default, Apr 30 2018, 17:28:45) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import delegator
>>> delegator.run(["ls", "-ltr"])
<Command ['ls', '-ltr']>
>>> print(_.out)
3c38c29667ff92da48efe096a4d02c33-{87A94AB0-E370-4cde-98D3-ACC110C5967D}
dumps
firefox_wgjak47
gameoverlayui.log
gameoverlayui.log.last
mozilla_wgjak470
pulse-PKdhtXMmr18n
qipc_sharedmemory_vWeJlNKttqulwtsFWhvgjgMWyIsoUBcfPzegfIa531e59614117c87604f08fdce98ed6992f41dc2
qipc_systemsem_vWeJlNKttqulwtsFWhvgjgMWyIsoUBcfPzegfIa531e59614117c87604f08fdce98ed6992f41dc2
sddm-:0-admhzy
sddm-authf2cc6c54-ef27-41fb-aba4-4dd0983682e8
shadowsocks-qt5
ssh-YGJKpV2GglPo
steam_chrome_shmem_uid1000_spid8462
systemd-private-114fd641e43147049112b5c15f993361-mysqld.service-sFGecH
systemd-private-114fd641e43147049112b5c15f993361-rtkit-daemon.service-XWxi6M
systemd-private-114fd641e43147049112b5c15f993361-systemd-timesyncd.service-B0qmOs
systemd-private-114fd641e43147049112b5c15f993361-upower.service-4UBov1
Temp-932171ac-0b62-481b-8b6e-2f5e6d6e17e7
Temp-93e13b39-61d7-4834-a29f-e5e7c37902ab
vWeJlNK33ttqulwt_sFWhvgjgMWyIsoUBcfPze5g6fI=
xauth-1000-_0

>>> delegator.run(["ls", "-ltr"], block=False)
<Command ['ls', '-ltr']>
>>> _.block()
>>> print(_.out)
总用量 12
drwx------ 3 root    root     60 2月  21 18:39 systemd-private-114fd641e43147049112b5c15f993361-systemd-timesyncd.service-B0qmOs
drwx------ 2 root    root     40 2月  21 18:39 pulse-PKdhtXMmr18n
drwx------ 3 root    root     60 2月  21 18:39 systemd-private-114fd641e43147049112b5c15f993361-mysqld.service-sFGecH
srwxr-xr-x 1 root    root      0 2月  21 18:39 sddm-authf2cc6c54-ef27-41fb-aba4-4dd0983682e8
srwx------ 1 sddm    sddm      0 2月  21 18:39 sddm-:0-admhzy
drwx------ 2 wgjak47 wgjak47  60 2月  21 18:39 ssh-YGJKpV2GglPo
-rw------- 1 wgjak47 wgjak47 177 2月  21 18:39 xauth-1000-_0
drwx------ 3 root    root     60 2月  21 18:39 systemd-private-114fd641e43147049112b5c15f993361-upower.service-4UBov1
drwx------ 3 root    root     60 2月  21 18:39 systemd-private-114fd641e43147049112b5c15f993361-rtkit-daemon.service-XWxi6M
srwxrwxrwx 1 wgjak47 wgjak47   0 2月  21 18:39 shadowsocks-qt5
drwx------ 2 wgjak47 wgjak47  40 2月  21 18:39 Temp-932171ac-0b62-481b-8b6e-2f5e6d6e17e7
srwx------ 1 wgjak47 wgjak47   0 2月  21 18:39 vWeJlNK33ttqulwt_sFWhvgjgMWyIsoUBcfPze5g6fI=
-rw-r----- 1 wgjak47 wgjak47   0 2月  21 18:39 qipc_systemsem_vWeJlNKttqulwtsFWhvgjgMWyIsoUBcfPzegfIa531e59614117c87604f08fdce98ed6992f41dc2
-rw-r----- 1 wgjak47 wgjak47   0 2月  21 18:39 qipc_sharedmemory_vWeJlNKttqulwtsFWhvgjgMWyIsoUBcfPzegfIa531e59614117c87604f08fdce98ed6992f41dc2
srwxr-xr-x 1 wgjak47 wgjak47   0 2月  21 18:41 steam_chrome_shmem_uid1000_spid8462
drwxrwxrwx 2 wgjak47 wgjak47 100 2月  21 19:15 dumps
-rw-r--r-- 1 wgjak47 wgjak47 712 2月  21 19:22 gameoverlayui.log.last
-rw-r--r-- 1 wgjak47 wgjak47 637 2月  21 19:49 gameoverlayui.log
drwx------ 2 wgjak47 wgjak47  40 2月  21 19:59 Temp-93e13b39-61d7-4834-a29f-e5e7c37902ab
srwxr-xr-x 1 wgjak47 wgjak47   0 2月  21 20:21 3c38c29667ff92da48efe096a4d02c33-{87A94AB0-E370-4cde-98D3-ACC110C5967D}
drwx------ 2 wgjak47 wgjak47  40 2月  21 21:19 mozilla_wgjak470
drwx------ 2 wgjak47 wgjak47  60 2月  21 21:43 firefox_wgjak47

>>> 

Look like the same problem in #58

I guess the non block mode use pexpect and block mode use subprocess directly...

And when use subprocess with shell=True, the list[1:] will be ignored....

The packages version:

delegator.py==0.1.1
  - pexpect [required: >=4.1.0, installed: 4.6.0]
    - ptyprocess [required: >=0.5, installed: 0.6.0]

Documentation incorrectly suggests that you can pass a list of args for something you want to run

Because of the "shell": True argument below, passing in a list will not work as expected, which I had to find out the hard way

@property
def _default_popen_kwargs(self):
    return {
        "env": os.environ.copy(),
        "stdin": subprocess.PIPE,
        "stdout": subprocess.PIPE,
        "stderr": subprocess.PIPE,
        "shell": True,
        "universal_newlines": True,
        "bufsize": 0,
    }

module 'delegator' has no attribute 'run'

I'm on windows, installed delegator with pip and try running it with code below:

import delegator 
c = delegator.run("python -v")

getting error:

  File "C:/Users/EP/Documents/GitHub/todo-core/test_guz.py", line 8, in <module>
    c = delegator.run("python -v")

AttributeError: module 'delegator' has no attribute 'run'

what is possibly wrong?

pexpect.exceptions.TIMEOUT

I'm executing a process that's running for a few minutes before printing some output. After a few seconds I got:

  File "/usr/lib/python2.7/site-packages/delegator.py", line 240, in chain
    data = c.out
  File "/usr/lib/python2.7/site-packages/delegator.py", line 91, in out
    self.__out = self._pexpect_out
  File "/usr/lib/python2.7/site-packages/delegator.py", line 79, in _pexpect_out
    result += self.subprocess.read()
  File "/usr/lib/python2.7/site-packages/pexpect/spawnbase.py", line 413, in read
    self.expect(self.delimiter)
  File "/usr/lib/python2.7/site-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/lib/python2.7/site-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/lib/python2.7/site-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/lib/python2.7/site-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x1bdb510>
searcher: searcher_re:
    0: EOF
<pexpect.popen_spawn.PopenSpawn object at 0x1bdb510>
searcher: searcher_re:
    0: EOF

There doesn't seem to be a way to control the timeout, is there?