pexpect/pexpect

A Python module for controlling interactive programs in a pseudo-terminal

Pexpect is a Pure Python Expect-like module

Pexpect makes Python a better tool for controlling other applications.

Pexpect is a pure Python module for spawning child applications; controlling them; and responding to expected patterns in their output. Pexpect works like Don Libes' Expect. Pexpect allows your script to spawn a child application and control it as if a human were typing commands.

Pexpect can be used for automating interactive applications such as ssh, ftp, passwd, telnet, etc. It can be used to automate setup scripts for duplicating software package installations on different servers. It can be used for automated software testing. Pexpect is in the spirit of Don Libes' Expect, but Pexpect is pure Python.

The main features of Pexpect require the pty module in the Python standard library, which is only available on Unix-like systems. Some features—waiting for patterns from file descriptors or subprocesses—are also available on Windows.

If you want to work with the development version of the source code then please read the DEVELOPERS.rst document in the root of the source code tree.

Free, open source, and all that good stuff.

You can install Pexpect using pip:

pip install pexpect

Docs on ReadTheDocs

PEXPECT LICENSE:

http://opensource.org/licenses/isc-license.txt

Copyright (c) 2013-2016, Pexpect development team
Copyright (c) 2012, Noah Spurrier <[email protected]>

PERMISSION TO USE, COPY, MODIFY, AND/OR DISTRIBUTE THIS SOFTWARE FOR ANY
PURPOSE WITH OR WITHOUT FEE IS HEREBY GRANTED, PROVIDED THAT THE ABOVE
COPYRIGHT NOTICE AND THIS PERMISSION NOTICE APPEAR IN ALL COPIES.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

This license is approved by the OSI and FSF as GPL-compatible.

pexpect's Issues

In ANSI, it would be nice if unwanted numbers weren't left on the stack

Since an "\x1b[...m" ANSI escape sequence has a variable count of numbers, I assume that if an FSM handler was set up for that sequence, it would need to assume that fsm.memory[0] is the screen object (as usual) and that all the subsequent entries on the stack are the attribute numbers.

However, in 345eb58, when ANSI sequences with numbers are fed in, i.e. "\x1b[...m", "\x1b[?..h" and "\x1b[?..l", the numbers are left on the FSM stack. This means you can't tell which numbers relate to the current escape sequence and which are left over from previous sequences.

It would be nice, then, if handlers were provided for those sequences that just set fsm.memory = fsm.memory[0], i.e. discarded the numbers, so that anyone that wants to try updating or subclassing ANSI to process any of those sequences doesn't find other random values on the stack.

Please correct me if my understanding is wrong. Otherwise, let me know and I might try to submit a pull request for this.

using .after

Going through the docs, am I correct in saying that child.before("DONE") is everything before the variable and child.after("DONE") just matches the variable/text?

I connect to a monitoring device and the response's ending text is never the same. Is there a way to record everything after a certain text until a EOF or TIMEOUT?

spawn with timeout=None fails on expect

From an IPython test:

  File "/home/takluyver/.virtualenvs/ipy-3/lib/python3.3/site-packages/ipython-2.0.0_dev-py3.3.egg/IPython/lib/irunner.py", line 207, in run_source
    prompt_idx = c.expect_list(prompts)
  File "./pexpect/__init__.py", line 1435, in expect_list
    timeout, searchwindowsize)
  File "./pexpect/__init__.py", line 1501, in expect_loop
    if timeout < 0 and timeout is not None:
TypeError: unorderable types: NoneType() < int()

Windows support

There are at least two ports of pexpect to Windows APIs:

• Chris Gorecki's wexpect
• Geert Jansen's winpexpect

We could look at integrating this support into pexpect. Both Chris and Geert have said they'd be willing to help out with some initial work, but I think this would need someone committed to maintaining it and fixing the inevitable problems. I don't really use Windows myself, so I can't give it much support.

change meaning of timeout argument of pexpect.spawn

The definition of a timeout of -1 being equivalent to the default value is quite confusing and unintuitive. It would be more elegant to provide a default timeout in a constant in pexpect and let -1 indicate to wait for ever. The default value of timeout argument of pexpect.spawn should stay the same in the meaning, i.e. point to pexpect.TIMEOUT_DEFAULT.

ANSI doesn't handle more than two numbers in \x1b[...m properly

In 345eb58, when an instance of ANSI is passed an "\x1b[...m" sequence to set attributes, if there are more than two semicolon-separated numbers before the "m", it doesn't work properly.

First, in the 'NUMBER_X' FSM state, string.digits is not accepted as causing a transition back to the 'NUMBER_X' state, so if there is more than one digit in the third number, the second digit is logged as unexpected. I believe that a sequence like "\x1b[0;32;45m" to reset the attributes to their defaults and then set both the foreground and background is probably fairly common.

I think that in the line:

self.state.add_transition_list (string.digits, 'SEMICOLON_X', None, 'NUMBER_X')

it would be useful to replace None with DoStartNumber so that the number can be pushed onto the stack in case someone wants to use it, and then a new transition should be added:

self.state.add_transition_list (string.digits, 'NUMBER_X', DoBuildNumber, 'NUMBER_X')

to deal with the second and subsequent digits.

Further, ';' is not accepted in the 'NUMBER_X' state either, so a fourth number cannot be provided. I don't know how common four numbers would be, but I could imagine perhaps using "\x1b[0;1;32;45m" to reset attributes, then set both background, foreground and bold.

The code includes two instances of this line:

self.state.add_transition (';', 'NUMBER_2', None, 'SEMICOLON_X')

I think the second was meant to be accepted in the 'NUMBER_X' state.

pxssh.auto_prompt_reset attribute doesn't do anything

This came up in an SO question. The docs say that setting the auto_prompt_reset attribute to False on a pxssh instance should prevent it from trying to change the prompt. But it's never actually used - auto_prompt_reset is passed as a parameter to login() instead.

I think we should just correct the docs and remove the attribute.

purge, improve, and document better scripts in examples/

Not a blocker in 3.0. Looking at the pexpect package as if I'm was previously unaware of how to use it and where to begin, the examples/ folder strikes me as terrible quality.

This is where the code should be most clean, terse, clever, and useful. It should not appear as a graveyard of semi-utilitarian throw-away scripts where version control is performed by commenting out lines, as it is now :-)

Individual nit-picking:

• bd_client.py: took a long while for me to figure out to pass /tmp/mysock to work with bd_serv.py
• bd_serv.py: unused code parse_host_connect_string(), in testing, it does seem to ssh to the remote host, but the actual send/recv of commands doesn't seem to operate correctly ?
• chess[123].py -- how about just one? (that works).
• topip.py:
• df.py:
• uptime.py
• fix_cvs_files: poor use of expect, this is better done with subprocess
  (uptime contains a potentially useful regexp, example, though? Still, subprocess ...)
• monitor.py:
• sshls.py:
• astat.py: would be better served using paramikko.
• ssh_tunnel.py: this is what 'autossh' is for.
• python.py: terrible! It prints an escape sequence backwards, ouch.
• rippy.py: yuck.
• ssh_session.py: no other script uses this, doesn't serve much as an example on its own... sorry esr?

Keepers. These are actually semi-useful as examples of using pexpect ;-)
Still, they should be updated to use 'argparse' and code cleanup for pep8 and pythonic.

• hive.py
• passmass.py
• script.py

Ideas for new scripts:
AI that plays simple bsd games, which are very available on most platforms, and are in prompt/answer format:

easy: 'quiz multiplication answer', 'arithmetic'.
harder (Can use VT100): boggle

API to execute code in child process before execv?

I just came across an issue where the parent process was ignoring SIGINT using signal.signal(SIGINT, SIG_IGN), but I wanted the child to be interruptible. The UNIX model has child processes inherit their parents signal handlers on a fork, and while function handlers are reset by exec*, ignore handlers are not. For now, I'm temporarily changing the signal handler, spawning, and changing it back, but this doesn't seem ideal.

I'm thinking of introducing a new optional parameter to spawn, which would be None (by default) or a callable taking no arguments. If supplied, it would be called in the child process just before we call execv/execvpe.

This presumably won't work with our upcoming Windows support, because you don't start processes on Windows by forking. I think it's reasonable to raise NotImplementedError in that case.

I cannot grab some output on Travis but can on my machine

Hi,
First, thanks for pexpect. Great module.
Things work great on my machine (except that maybe I'm facing the «Truncated output just before child exits» issue, not sure yet), but things are all different on Travis.

Here on a build report you can read :

AssertionError: Devait afficher "Votre nom ?"
tout le monde!
"Salut tout le monde!", mais on a eu .

which is some French that says :

AssertionError: Should have displayed "Your name ?"
world!
"Hello world!", but got .

The problem is that there is nothing between «got » and the final dot, which is not the case on my machine.

The pexpect code is in some steps for Lettuce here:
https://github.com/Microalg/Microalg/blob/master/features/cli.py#L31

The assertion that triggers the error is at the last lines of this file:
https://github.com/Microalg/Microalg/blob/master/features/cli.py#L70

The purpose of the test is to run an interpreter, which will read a «Hello world?» program, which will prompt the user for is name then display the message.

As your project is tested on Travis, I thought maybe you could guess what the problem was.

Wait for multiple processes at the same time

I currently use code similar to the following to test on android.

I want to convert to using python and pexpect, but I'm unsure of how to accomplish something like the following, and have it be just as simple.

Note that spawn is used to start espresso, and a separate process is spawned for logging.

 expect {  <-- multiprocess event loop
     -i $adb_espresso_spawn_id -re ".+\r\n" {  <--- espresso process
         exp_continue
         }
     -i $adb_logcat_spawn_id -re ".+\r\n" {   <-- logcat process
         exp_continue
         }
     -i $adb_espresso_spawn_id eof {   <-- espresso process
         puts "quitting espresso, eof found."
         }
     -i $adb_espresso_spawn_id timeout {   <-- espresso process
         puts "Espresso timeout."
         }
     -i $adb_logcat_spawn_id timeout {   <-- logcat process
         puts "logcat timeout."
         }
     }

Any ideas on how to do something like this?

I've seen multithreaded (multiprocess?) stuff that could spawn 100 threads (more or less), but not something like this. I think the author of pexpect said that pexpect should handle 90% of what expect does, but I'm unsure if the other 10% is something like this.

pexpect Cannot be used in conjunction with other components

I hope pexpect can increase the binding interfaces with other modules
scene:
I login to a remote server through the SSH component, execute a command requires sudo permission。

Use of select.select() should be replaced with select.poll()

pexpect (the original version, this version or its pexpect-u ancestor) all use select.select() exclusively. This means that they cannot be used in processes with a ton of file descriptors open due to the hard limit on the underlying OS select() call.

An update to use select.poll() instead would be wonderful.

Strange EOF/TIMEOUT behavior on BSD systems

on OSX, the following script:

#!/usr/bin/env python
import pexpect, sys
while True:
    child = pexpect.spawn('ls -l /dev', timeout=10)
    child.expect(pexpect.EOF)
    sys.stderr.write('.')
    sys.stderr.flush()

Will, occasionally (about 1 out of 20 for me), delay for 10 full seconds.

Some test cases take a very long time to complete, intermittently, on OSX, for this reason.

Just about the time a TIMEOUT should occur, an EOF is detected instead. It is understood there is some bug in the .isalive() check and EOF and TIMEOUT behavior along the chain towards read_nonblocking() on BSD systems, it just not well understood.

Tab characters are not expanded by screen/ANSI

Neither the screen class, nor its ANSI subclass, expands tab characters. Instead, the tab character (\x09) is literally stored in a character cell. If this occurs and the screen contents are then retrieved and output to a normal terminal, the tab characters will be expanded at that point, but if the pretty() method is used, the right-hand border will not be aligned properly due to the expansion of the tabs.

The screen class provides some methods for dealing with setting tab stop positions, but they are not implemented or called.

It would be nice if, at the very least, tab characters caused the cursor to be moved to the standard/default tab stops.

This is fairly low priority for me, I might not get around to fixing it.

Process stops logging after last `expect`

If I spawn a new child process and expect something, I get logging output. However, once this expect has been hit, logging stops. The child process has not been killed, or closed, and it continues to run in the background and I imagined the logging should continue asynchronously, if you will.

Is this by design, or a bug? If the former, is there any way to get logging to continue while the process exists (i.e. redirect the child process' stdout to the console)?

Some (unfortunately long) sample code to demonstrate this:

#!/usr/bin/env bash
#
# Generate text periodically

while [ 1 ]; do
    echo "This is some text" | tee -a output.log
    sleep 1
done

#!/usr/bin/env python
#
# Call 'generate.sh' using pexpect

import pexpect
import os
import sys
import time

SHELL_CMD = ['/bin/bash', '-c']
ROOT_DIR = os.path.dirname(os.path.realpath(__file__))
SCRIPT_CMD = os.path.join(ROOT_DIR, 'generate.sh')

def main():
    cmd = SHELL_CMD + ['"%s"' % SCRIPT_CMD]
    cmd = ' '.join(cmd)

    child = pexpect.spawn(cmd, timeout=10)
    child.logfile_read = sys.stdout

    for i in range(0, 5):
        print('iteration %d' % i)
        try:
            child.expect(['This is some text'])
        except Exception as exc:
            print('error occured')
            raise exc

    time.sleep(1000)

main()

The child continues to execute. Here's the stdout upon killing after ~10 seconds:

$ ./run.py
iteration 0
This is some text
iteration 1
This is some text
iteration 2
This is some text
iteration 3
This is some text
iteration 4
This is some text
^CTraceback (most recent call last):
  File "./run.py", line 31, in <module>
    main()
  File "./run.py", line 29, in main
    time.sleep(1000)
KeyboardInterrupt

The contents of output.log after the same (notice how many This is some text lines there are == process still running):

This is some text
This is some text
This is some text
This is some text
This is some text
This is some text
This is some text
This is some text
This is some text
This is some text

pexpect on Solaris via cron (/dev/tty issue)

I am using the pexpect (version 3.1, installed via pip) on OmniOS (Solaris fork), and it works fine on the interactive shell. But when I try to use it via cron it has issues with /dev/tty. I thought this was an issue for Solaris but had been fixed?
Specifically, the error is:

OSError: [Errno 6] No such device or address: '/dev/tty'

Rest of the trace:

p=pexpect.spawn('ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no %s@%s' % ('root', host), timeout=60)
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 485, in __init__
    self._spawn(command, args)
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 607, in _spawn
    self.pid, self.child_fd = self.__fork_pty()
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 668, in __fork_pty
    self.__pty_make_controlling_tty(child_fd)
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 722, in __pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] No such device or address: '/dev/tty'
Traceback (most recent call last):
  File "/root/nologify.py", line 25, in <module>
    p.expect('\r\n.+]# ')
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 1418, in expect
    timeout, searchwindowsize)
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 1433, in expect_list
    timeout, searchwindowsize)
  File "/usr/lib/python2.6/site-packages/pexpect/__init__.py", line 1521, in expect_loop
    raise EOF(str(err) + '\n' + str(self))
pexpect.EOF: End of File (EOF). Very slow platform.
<pexpect.spawn object at 0x8181d6c>
version: 3.1
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-oUserKnownHostsFile=/dev/null', '-oStrictHostKeyChecking=no', 'root@host1']
searcher: <pexpect.searcher_re object at 0x8181dec>
buffer (last 100 chars): ''

timeout on run vs. spawn

I'm not entirely sure if this is a bug or just something in need of clarification in the documentation. The run method docs states that it will run a given command to completion and takes an argument timeout=-1. however run calls _run which checks to see if timeout is -1 and if it is it calls spawn with the deafult timeout of 30. This can cause a long running command to terminate at 30 seconds instead of "to completion" as the docs imply.

This raises an interesting question to me at least as to why run default arg timeout is -1 and not just 30 like spawn and then you can just pass it through to spawn instead of checking for a strange -1 flag.

test_readlines consistently fails on a Mac

Reproduced on Python 2.7 and 3.3:

======================================================================
FAIL: test_readlines (__main__.TestCaseMisc)
Note that on some slow or heavily loaded systems that the lines
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_misc.py", line 126, in test_readlines
    "readlines() did not work. page=%r" % (page,))
AssertionError: readlines() did not work. page=''

----------------------------------------------------------------------

Can't convert 'bytes' object to str implicitly

In pxssh.py:284 you can see line

raise ExceptionPxssh ('could not set shell prompt\n'+self.before)

This gives

TypeError: Can't convert 'bytes' object to str implicitly

since self.before is a 'bytes' object.

stty size reports incorrect size for pseudo-tty unless launched from ipython

Consider the following script:

import pexpect
import os
from time import sleep

child = pexpect.spawnu(u'bash', ['--norc', '--noprofile'], encoding='utf-8', env={u'TERM': u'vt100'})
child.setwinsize(40, 1024)
sleep(1)
child.send('/bin/stty size\n')
sleep(1)
print (child.read_nonblocking(size=1000))

It will print

bash-4.2$ /bin/stty size
24 80
bash-4.2$

if launched using python test.py or with execfile from python interactive console, but

bash-4.2$ /bin/stty size
40 1024
bash-4.2$ 

if launched with execfile from ipython. Using pexpect version 3.3, python-2.7.6 or 3.3.3. Same results if setting TERM to xterm or not setting it at all (except for the fact that ipython prints nothing in the latter case, need to switch shell to zsh).

Outer terminal is either konsole-256colors (with konsole 4.12.5) or screen-256colors (with tmux 1.9a), though I do not think that should matter.

TIOCSWINSZ hack produces ioctl sign-extension warning in FreeBSD system log

There's a hack in pexpect/__init__.py to work around some buggy platforms:

    # Check for buggy platforms. Some Python versions on some platforms     
    # (notably OSF1 Alpha and RedHat 7.1) truncate the value for            
    # termios.TIOCSWINSZ. It is not clear why this happens.                 
    # These platforms don't seem to handle the signed int very well;        
    # yet other platforms like OpenBSD have a large negative value for      
    # TIOCSWINSZ and they don't have a truncate problem.                    
    # Newer versions of Linux have totally different values for TIOCSWINSZ. 
    # Note that this fix is a hack.                                         
    TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)               
    if TIOCSWINSZ == 2148037735L: # L is not required in Python >= 2.2.    
        TIOCSWINSZ = -2146929561 # Same bits, but with sign.               

-2146929561 is 0xffffffff80087467 and passing this sign-extended value rather than the desired 0x80087467 produces a warning message in the system console on FreeBSD:

Feb 5 17:19:11 feynman kernel: WARNING pid 11323 (python2.7): ioctl sign-extension ioctl ffffffff80087467

This has shown up in a few places, e.g.:
http://www.freebsd.org/cgi/query-pr.cgi?pr=152770
http://llvm.org/bugs/show_bug.cgi?id=18749

interact() missing coverage for (OSError, EIO), reading after child exit

Per pull #23 regarding line # ~1645 of init.py:

What troubles me: the variable "data" is re-used, for both stdin of controlling process, and stdout of child pty, within a loop. In the case of an I/O Error reading from child process, an implicit pass occurs, and any of three conditions then occur:

1. The value of 'data' is continued from the prior loop; which, if no stdin was read, is the previously read data from the same pty -- therefor the output of the program is duplicated.
2. The value of 'data' is continued from the prior loop; which, if stdin was read, data that was previously already handled as keyboard input is now handled as keyboard output.
3. If data has not yet been defined, and an I/O Error occurs on the first assignment of it, (the first loop), then a NameError exception is raised: name 'data' is not defined.

All said -- an IO Error occurs when reading from a child pty that has exited. What we're looking to test, then, is a very brief moment of time where the child process has not yet exited (self.isalive() returns True), has data to read (fd returned in select() call), but then exits before it is read by the parent process (causing an IOError exception).

We're also looking at some very specific operating systems: In Linux, OSError raises to signal that an EOF occurred. BSD on the other hand would return an empty bytestring.

I'm going to re-name the title of this and mark it as a minor bug, as something that needs test coverage -- and fixing, if necessary. It may take some tinkering to write correct coverage for.

how to send the logfile.read to both stdout and a file

hi,

Could you give some hints on how to direct the logfile.read to both stdout and a file at the same time ?

Thanks

Terminal bells when sending long lines

import sys
# Set up the prompt to be sane
import pexpect
c = pexpect.spawn('/bin/sh')
c.sendline('export PS1="PROMPT"">"')  # odd quoting to not hit when expecting
c.expect('PROMPT>')

# Generate an onput
import base64
chars = 'a' * 3070
chars = base64.standard_b64encode(chars)
print chars

# Sent the input to base64
c.sendline('base64 --decode')
c.expect('\r\n')
batch = int(sys.argv[1])
if batch:
        batchsize = 100
        for l in range(0, len(chars), batchsize):
                c.sendline(chars[l:l + batchsize])
else:
        c.send(chars)
        c.sendline()

# Sync prompt
c.sendline()
c.sendline()
c.expect('\r\n\r\n')

# Finish up
print '>>>>>' + repr(c.before)
c.sendeof()
c.expect('PROMPT>')
print '>>>>>' + repr(c.before)

If you run this with an argument of 0 (i.e. batching disabled) you'll see that the last thing printed is an error from base64 - "base64: invalid input". This is because the characters start being replaces by the bell character after the 4095th (base64 has increased the number of chars from 3070 to 4096).
If you run it with an argument of 1 (to enable batching 100 chars at a time) it works fine. If, in batching mode, you replace sendline with a send then sendeof it also works.

I believe this is due to pipe buffer sizes. It doesn't happen if I paste the same input directly to my terminal, presumably because it sends the buffer automatically when it gets too big.

I think this is just a pexpect gotcha (not a bug), so it might be nice to mention it on the common problems page.

$ ulimit -a | grep pipe
pipe size            (512 bytes, -p) 8
$ python xx.py 0
YWFhYWFhYWFhYWF[...]FhYWFhYWFhYQ==
>>>>>'YWFhYWFhYWFhYWF[...]WFhYWFhYQ=\x07'
>>>>>'\r\nbase64: invalid input\r\n'
$ python xx.py 1
YWFhYWFhYWFhYWF[...]YWFhYWFhYWFhYQ==
>>>>>'YWFhYWFhYWFhY[...]YWFhYWFhYQ=='
>>>>>'\r\n'

readline+expect not working on Mac

I'm not sure what's the cause for this but after trying in different ways I found it actually works like a charm in Ubuntu, so I'm suspecting it has something to do with running it from a Mac.

This is the code I'm running:

def _inhouse(self, path):
        if (self.system == self.DEBIAN):
            self.s.sendline('dpkg -S %s'%path)
            index = self.s.expect(['dpkg-query', pexpect.EOF, pexpect.TIMEOUT],1)
            if (index == 0):
                return True

When run from Ubuntu it identifies correctly the files that do not belong to any package. When run from Mac, it returns TIMEOUT all the time. I checked readline+prompt+before, and I'm guessing something's wrong with buffering and/or stderr.

I can provide further details if required.

s.decode() should not be used -- codecs.IncrementalDecoder should be used instead

While developing pexpect tests for spawnu that drive sessions containing utf-8, a UnicodeDecodeError was thrown: because the read() call on the master pty fd is a stream, the bytes received may be an incomplete multibyte sequence.

That is, it may discover the utf-8 start byte, but not (yet) the subsequent finishing bytes. This occurred when driving the vim editor in the posted branch "more-utf8-tests" (a work in progress, but published for your review)

Incidentally this is how my own unicode-aware fork works, I'll provide a pull request for your review and reference this issue shortly.

Traceback (most recent call last):
  File "tests/test_unicode.py", line 172, in test_utf8_driven_vim
    p.expect(u'Σὲ γνωρίζω ἀπὸ τὴν κόψη', timeout=1)
  File "/Users/dingo1/Code/pexpect/pexpect/__init__.py", line 1420, in expect
    timeout, searchwindowsize)
  File "/Users/dingo1/Code/pexpect/pexpect/__init__.py", line 1435, in expect_list
    timeout, searchwindowsize)
  File "/Users/dingo1/Code/pexpect/pexpect/__init__.py", line 1504, in expect_loop
    c = self.read_nonblocking(self.maxread, timeout)
  File "/Users/dingo1/Code/pexpect/pexpect/__init__.py", line 927, in read_nonblocking
    s = self._coerce_read_string(s)
  File "/Users/dingo1/Code/pexpect/pexpect/__init__.py", line 1766, in _coerce_read_string
    return s.decode(self.encoding, self.errors)
  File "/Users/dingo1/Code/pexpect/ENV27/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 1023: unexpected end of data

fdexpect should have a unicode variant, like pexpect.spawnu

fdexpect.fdspawn is based on pexpect.spawn, which accepts bytes rather than unicode. fdexpect should provide a unicode variant, like/based on pexpect.spawnu.

Tests fail when locale encoding is ASCII

On Debian package build machines, the locale encoding is ASCII. We can override this with the PYTHONIOENCODING environment variable.

======================================================================
ERROR: test_interact_unicode (test_interact.InteractTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 907, in read_nonblocking
    s = os.read(self.child_fd, size)
OSError: [Errno 5] Input/output error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 1494, in expect_loop
    c = self.read_nonblocking(self.maxread, timeout)
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 911, in read_nonblocking
    raise EOF('End Of File (EOF). Exception style platform.')
pexpect.EOF: End Of File (EOF). Exception style platform.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/buildd/pexpect-3.0rc2/tests/test_interact.py", line 54, in test_interact_unicode
    p.expect ('<out>ther\xe9')
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 1410, in expect
    timeout, searchwindowsize)
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 1425, in expect_list
    timeout, searchwindowsize)
  File "/tmp/buildd/pexpect-3.0rc2/pexpect/__init__.py", line 1513, in expect_loop
    raise EOF(str(err) + '\n' + str(self))
pexpect.EOF: End Of File (EOF). Exception style platform.
<pexpect.spawnu object at 0x2b7c63786b50>
version: 3.0rc2
command: /usr/bin/python3.3
args: ['/usr/bin/python3.3', 'interact_unicode.py']
searcher: <pexpect.searcher_re object at 0x2b7c63c4a2d0>
buffer (last 100 chars): ''
before (last 100 chars): "odeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)\r\nMr. Py\xfeon\r\n"
after: <class 'pexpect.EOF'>
match: None
match_index: None
exitstatus: None
flag_eof: True
pid: 7261
child_fd: 12
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/buildd/pexpect-3.0rc2/tests/test_interact.py", line 62, in test_interact_unicode
    print(p.before)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 11: ordinal not in range(128)

----------------------------------------------------------------------

PEP-8 & refactoring

It looks like ye're doing a lot of work on this project after some years with little maintenance. What's your opinions on purely refactoring changes (i.e. breaking up __init__.py)? Assuming unit tests still work, is this worth it?

Exception when non-ASCII character appears in invalid ANSI escape sequence

In 345eb58 (with some changes, including changes to support Unicode), when some serial interface line noise caused garbage, non-ASCII characters to be received by my code, and they happened to be in the middle of an ANSI escape sequence, and this data was passed to an instance of ANSI, an exception occurred because an attempt was made to log the invalid character, and the encoding for the log file was ASCII:

Traceback (most recent call last):
  File "pexpect-test.py", line 714, in <module>
    term.process(c)
  File "pexpect.git/pexpect/ANSI.py", line 285, in process
    self.state.process(c)
  File "pexpect.git/pexpect/FSM.py", line 241, in process
    self.action (self)
  File "pexpect.git/pexpect/ANSI.py", line 179, in DoLog
    fout.write (fsm.input_symbol + ',' + fsm.current_state + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2562' in position 0: ordinal not in range(128)

I think there are actually a number of other issues with DoLog():

• the log file name "log" is fairly generic;
• the log messages are very cryptic unless you read the code; and
• it would be nice if the caller was able to control the logging in some way, e.g. I might like to write the messages (if they were more intelligible) to my script's own log file.

It's certainly better than nothing, and I'm not sure what a better solution would look like myself. Perhaps using Python's logging module would help; I know almost nothing about it, but the summary says "your application log can include your own messages integrated with messages from third-party modules", so presumably it makes this easy somehow.

select.select() cannot handle more than 1024 open files

I have been using pexpect for many months with no problem. However, I recently this line:

1680 return select.select(iwtd, owtd, ewtd, timeout)

Has been creating a few problems. The select system call cannot handle more than 1024 file descriptors. On the other hand, the select.poll() system call does not have such limitations.

I understand that not all platforms support poll() but it would be nice to incorporate this system call as an option.

Docs don't adequately describe use of "match" attribute

When a pattern is specified in an expect method, and a match occurs, the attribute 'match' is set to an RE match object (type is _sre.SRE_Match). There are interesting operations that can be done using the match object. It has attribute string, and methods group, groups, groupdict, start, end, span, and expand.
However, if there was a match on pexpect.EOF or pexpect.TIMEOUT, the type of the "match" attribute is not _sre.SRE_Match!
Also note that if method expect_exact is used, the result has type str (string) and not type _sre.SRE_Match!
It appears that attribute 'before' is always of the type 'str'.

Start bash with no customisation in replwrap

Wrapping the bash REPL can fail if the user customises their prompt in ~/.bashrc, because it has to recognise the initial prompt before it can change it.

We can start bash with no customisation by passing the --noprofile and --norc flags. The downside is that any customisation the user might want, like aliases, will also be missing. I think that's the lesser of two evils - better that it can start, even if it's missing some customisations. The user can configure it after starting, by doing e.g. b.run_command('source ~/.bash_aliases').

Clean-up tests: granular. coverage.

with py.test, we could use simple asserts again, but get detailed failure reports.
py.test also does a real stand-up job of coverage integration.
each test case should be very brief: given, exercise, verify. Never more than 4-5 lines total.

No need for all this unittest fancy footwork.
Especially the third argument of printing str(child) etc.

Importing after reassigning sys.stdout fails on Python 3

With an error like this:

  File "/usr/lib/python3/dist-packages/pexpect/__init__.py", line 276, in <module>
    class spawn(object):
  File "/usr/lib/python3/dist-packages/pexpect/__init__.py", line 286, in spawn
    write_to_stdout = sys.stdout.buffer.write
AttributeError: '_io.StringIO' object has no attribute 'buffer'

The aim was to have a standard way to write bytes to stdout, but if stdout is reassigned to a StringIO, which expects unicode, it's not clear what we should do about that.

pxssh - "Could not synchronize with original prompt"

Hey,

I have tested your pxssh example (http://pexpect.readthedocs.org/en/latest/api/pxssh.html). When I tried to connect to the SSH Server it says "pxssh failed on login. could not synchronize with original promt.". I tried several ways to solve it.
An interessting thing: After removing .ssh/known_hosts the login was successfull. A second login gave me the error again. Any ideas?

Thanks!

INSTALL doc refers to pexpect.py

The INSTALL document mentions that it is possible to install pexpect by copying the pexpect.py file:

If you do not have root access or if you do not wish to install Pexpect so that is available to any script then you can just copy the pexpect.py file to same directory as your script.

However, I couldn't find pexpect.py anywhere in the source tree.

pexpect 3.3 runs failed when use multiprocessing

I use pexpect (version 3.3) and multiprocessing in python (version 2.6) code like below:

import pexpect, multiprocessing

def login(h, u, p):
  ssh_new_key = ssh_change_key = 'abc'
  ssh = pexpect.spawn('ssh %s@%s' % (u, h))
  index = ssh.expect([ssh_change_key, ssh_new_key, 'password:'], timeout=5)

def worker():
    login('1.2.3.4', 'abc', 'abc')

if __name__ == '__main__':
  e_process = multiprocessing.Process(target=worker, args=[])
  e_process.start()
  e_process.join()

but runs failed, with this traceback:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "test.py", line 8, in worker
    login('1.2.3.4', 'abc', 'abc')
  File "test.py", line 4, in login
    ssh = pexpect.spawn('ssh %s@%s' % (u, h))
  File "/usr/lib/python2.6/site-packages/pexpect-3.3-py2.6.egg/pexpect/__init__.py", line 493, in __init__
    fd = sys.__stdin__.fileno()
ValueError: I/O operation on closed file

however pexpect 3.3 runs well in a none multiprocessing environment
And I test it on pexpect 2.4 (also with python 2.6), it runs well too.

use tox

wish list: use tox. the command 'tox' would execute all tests for all variants of python.

wheel archive

http://pythonwheels.com/

pexpect is shamed for "no wheel archives uploaded (yet!)."

being pure python, it shouldn't be hard at all to create a wheel archive -- why not help make this page go green where we can :-)

Hitting ^D in an 'interact()' causes endless printing of 'exit'

Run the following code:

import fdpexpect
import pty
import os
import sys

def proc_start(cmd_list):
        (child_pid, fd) = pty.fork()
        if child_pid == 0:
                # The first item of the list in the second argument is the name
                # of the new program
                try:
                        os.execvp(cmd_list[0], cmd_list)
                except OSError:
                        print "Failed to exec"
                        sys.exit(1)
        else:
                return fd

fd = proc_start(['/bin/bash'])
c = fdpexpect.fdspawn(fd)
c.interact()

You get a bash session, it works, you can do things.
Now, instead of hitting ^], hit ^D - observe the endless printing of 'exit' and your now locked-up terminal.

Going backwards through pypi versions, I can see this has definitely been broken since 3.0, but was working (it raised an exception instead of printing the same output forever) in 2.4

Automated builds & tests on other OS's

Currently we have Travis CI, which is great, but its linux-only. There are a lot of edge cases here for non-linux OS's. The modern ones in use would be the *BSD's and Solaris. We can probably ignore things such as Irix ..

Something long-lived that we can share credentials on would be best, I'm not sure computer(s) in my home would be very long-lived, might have to pay for hosting, joyent (opensolaris) and maybe amazon aws or similar for a bsd. I can probably afford the money and time for these later this year.

merge docs/old_changes.html and history.rst

need to identify the version release number(s) for the versions referenced by old_changes.html and put this into history.rst. Things like isAlive -> isalive seem important for upgraders.

Provide startDiscardingOutput() method

Sorry to hijack the issues for what might be more of a support request. I tried searching for a mailing list, google group or IRC channel but couldn't find one.

I am trying to use pexpect to spawn a node.js http server process and wait until it outputs a string that indicates it is ready to receive requests. After this I am attempting to use subprocess.Popen to run another command for running a suite of end-to-end tests.

I can set the spawnedProcess.logfile to sys.stdout and see that the startup seems to proceed without problems but when I start the end to end process and call .wait() on it, it seems like the node.js http server I started earlier has become unresponsive. Any idea what I am doing wrong.

The script is rather short so I am including it here:

#!/usr/bin/env python

import atexit
import sys
import os
import pexpect
import subprocess

nodejs = None

def shutdown_servers():
    print "Shutting down servers"
    global nodejs
    if nodejs:
        nodejs.sendcontrol('c')

atexit.register(shutdown_servers)
print "Starting nodejs server"
os.environ["NODE_ENV"] = 'testing'
nodejs = pexpect.spawn('node server.js -p 9001', env=os.environ)
nodejs.logfile_read = sys.stdout
try:
    nodejs.expect("server started on")
except pexpect.EOF:
    print "EOF When waiting for nodejs to listen to connections"
    print "Output:"
    print nodejs.before
    sys.exit(1)

command = './node_modules/.bin/protractor protractor.conf.js'

protractorProcess = subprocess.Popen(command, env=os.environ, shell=True)
protractorProcess.wait()

sys.exit(protractorProcess.returncode)

If I run the server manually in one terminal tab and then run the protractor command in another terminal everything works fine.

sort out migration from sourceforge.net

I observed only now the move of the project but I fully support it. I didn't had any time for pexpect but maybe it would help to do some cleanup on sf.net so people will not be redirected to the wrong location.

Can I remove the bugs of SF? Or other sections from:
https://sourceforge.net/p/pexpect/admin/tools?limit=200&page=0

Do something with psh module

pexpect.psh is an odd little bit of code that was added after the last 2.x release (before we took over maintenance). We left it around, but I've never documented it or advertised it, and I put a note in the docstring that it's provisional.

It appears to be designed to wrap an ssh connection, exposing a number of commands as methods on the psh instance, so you could do something like this:

p = pxssh.pxssh()
p.login(...)
s = psh.psh(p)
s.cd('foo')
files = s.ls()

It has only very minimal tests, and I think the approach I've taken with the new replwrap module is more flexible. Do we:

• Remove it straight away? It was never documented or promoted, so people shouldn't be relying on it.
• Deprecate it? There might be a few people out there using it despite the lack of documentation.
• Improve, test and document it? I'm willing to be convinced that it's useful.

ANSI doesn't allow nonprintable characters

In 345eb58, in write_ch():

if ch not in string.printable:
    fout = open ('log', 'a')
    fout.write ('Nonprint: ' + str(ord(ch)) + '\n')
    fout.close()
    return

I would like non-printable characters to be accepted, since I am dealing with data streams that include CP437 line drawing characters.

Obviously it's easy to comment out the above lines, but I think it might make more sense for non-printable characters to be accepted, and to leave it up to the caller to decide what to do with any that they find on the virtual "screen" as they see fit. Filtering them out, as is currently done, means that other characters do not appear in their correct locations on the screen, unless this is meant to be a way to filter out line noise, in which case you don't want the cursor to be moved when they are filtered out?

ANSI and screen don't support Unicode

It would be nice if the ANSI and screen packages supported Unicode strings, or there were additional packages/classes that did. I only had to change str() to unicode() and add u in front of the quoted strings (and perhaps even some of that wasn't necessary) in screen.pretty() and put_abs() in 345eb58 to get this working quickly for me, but I didn't test all methods.

If you could suggest your preferred solution, I might be able to implement it and submit a pull request.

use py.test

wish list: use py.test as test runner.