Provide startDiscardingOutput() method

[pexpect]

Provide startDiscardingOutput() method

[jquast]

because nodejs is not .expect()'ing anything, its stdout is not being read -- nodejs server is doing its own write(stdout, "log messsage") somewhere and blocking. Example,

>>> jot = spawnu('jot 100000 1')
>>> jot.expect(re.compile('\r\n100\r\n'))
0
>>> jot.before
u'1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n8\r\n9\r\n10\r\n11\r\n12\r\n13\r\n14\r\n15\r\n16\r\n17\r\n18\r\n19\r\n20\r\n21\r\n22\r\n23\r\n24\r\n25\r\n26\r\n27\r\n28\r\n29\r\n30\r\n31\r\n32\r\n33\r\n34\r\n35\r\n36\r\n37\r\n38\r\n39\r\n40\r\n41\r\n42\r\n43\r\n44\r\n45\r\n46\r\n47\r\n48\r\n49\r\n50\r\n51\r\n52\r\n53\r\n54\r\n55\r\n56\r\n57\r\n58\r\n59\r\n60\r\n61\r\n62\r\n63\r\n64\r\n65\r\n66\r\n67\r\n68\r\n69\r\n70\r\n71\r\n72\r\n73\r\n74\r\n75\r\n76\r\n77\r\n78\r\n79\r\n80\r\n81\r\n82\r\n83\r\n84\r\n85\r\n86\r\n87\r\n88\r\n89\r\n90\r\n91\r\n92\r\n93\r\n94\r\n95\r\n96\r\n97\r\n98\r\n99'
>>>
[1]+  Stopped                 python
~/Code/pexpect (master u+2-2 origin/master)$ ps -p `pgrep jot`
  PID TTY           TIME CMD
52584 ttys013    0:00.01 /usr/bin/jot 100000 1

This program counts to 100,000 -- but I've only expected up to 100, so its blocking right now.
This program will exit after it reaches 100,000 but its going to halt forever.

You should do something (like, untested):

while nodejs.isalive() or protractorProcess.isalive():
    nodejs.read_nonblocking(size=100)
    protractorProcess.read_nonblocking(size=100)
if not nodejs.isalive():
    print('nodejs server crashed?!')
else:
    shutdown_servers()

[sgronblo]

Isn't there some way to just let a spawned process start running freely?

[takluyver]

You can run a thread that pulls output from the process and discards it. If
you control the node code, you could have it redirect its output to
/dev/null on some signal. I don't know of a better way to achieve that from
the parent process.

[sgronblo]

I see. Would it be difficult to add some method like spawnedProcess.startDiscardingOutput() that would do this automatically?

[takluyver]

I think something like that should be possible. I'll work out what api
makes most sense, because it's also possible to pull and store output
rather than discarding it.

[jquast]

I would be very interested to see how .startDiscardingOutput() could return immediately, yet cause all subsequent output to be discarded, I don't think it is possible. If running it forever is desired, just call child.expect(pexpect.EOF, timeout=None). Call it from a managed thread and your main program can go on to do whatever it does.

The problem with the .startDiscardingOutput() is that something has poll for program output, even if it wants to discard it. If its not the caller than there's nobody else -- there are no background threads or asyncio tasks implemented in pexpect.

[sgronblo]

But child.expect is a blocking call right? What I wanted to do was just run a process up until a certain point and then let it start running without caring about its output and run another process that interacts with the first one.

Is my understanding correct that pexpect's spawned processes are stopped unless you expect something from them? Wouldn't it be possible to just connect a process's stdin to /dev/null or a file of your choosing and restart it? No thread doing expecting required?

[jquast]

well, considering the issue, all i could think of is how I would do it with bash -- and then how I could implement it in pexpect. I could only think of one way, really, and I figured what the heck, pexpect can drive a terminal just as well as a human, so why don't you just do this instead?

from pexpect import spawnu, EOF
# first start bach
jot = spawnu('bash')
# then start the process that will run for long time
jot.sendline('jot 100000 1')
# Look for a marker (here, 100 surrounded by space/newlines)
jot.expect(re.compile('\s220\s'))
# pause process (SIGSTOP)
jot.sendcontrol('z')
# wait for bash to return to prompt
jot.expect(u'Stopped')
# put it in the background, disown process ownership, exit
jot.sendline('bg; disown; exit')
# wait for bash to exit
jot.expect(EOF)
# assert it exited cleanly
assert jot.isalive() == False
assert jot.exitstatus == 0

[takluyver]

startDiscardingOutput() could spin out a thread, which would just repeatedly read output. We have a StreamCapturer class in IPython to do something very similar. This has the added benefit that you could store the output for later inspection, rather than discarding it.

This SO answer indicates that you can't directly change another process's stdout after it's started.

[takluyver]

I also want to provide asyncio support in pexpect to allow waiting for more than one thing at a time. I need to work out the right way to achieve that, though.