Help me with a bytes decoding problem in python 3

[J]

I have a bit of a problem I hope you could help me sort out with some
code I'm porting from 2.x to 3.

I have a program with a wrapper for Popen that is called to run
certain linux shell commands and gather system info. Essentially the
code looks something like this:


process = subprocess.Popen(command_str,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
result = process.communicate()

stdout,stderr = result

message = ['Output:\n'
'- returncode:\n{0}'.format(self.process.returncode)]
if stdout:
if type(stdout) is bytes:
stdout = stdout.decode()
message.append('- stdout:\n{0}'.format(stdout))
if stderr:
if type(stderr) is bytes:
stderr = stderr.decode()
message.append('- stderr:\n{0}'.format(stderr))
logging.debug('\n'.join(message))

So what essentially happens is that command_str is passed to this
function and is a string like this:

'lspci | grep Network'
or
'xinput list'

using the first command string, I get output like this:

In [12]: command = 'lspci |grep Network'
In [13]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [14]: result = process.communicate()
In [15]: stdout,stderr = result
In [16]: result
Out[16]:
(b'00:19.0 Ethernet controller: Intel Corporation 82577LC Gigabit
Network Connection (rev 06)\n07:00.0 Network controller: Intel
Corporation Ultimate N WiFi Link 5300\n',
b'')

Which is a bytes object that is very easily converted to text by the
decode() call prior to sending "message" to the logger.

However, the program seems to now be hanging up when running the
second command_str because it's actually returning bytecode inside the
bytes object:

In [18]: command = 'xinput list'
In [19]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [20]: result = process.communicate()
In [21]: stdout,stderr = result
In [22]: result
Out[22]:
(b'\xe2\x8e\xa1 Virtual core pointer
\tid=2\t[master pointer (3)]\n\xe2\x8e\x9c \xe2\x86\xb3 Virtual
core XTEST pointer \tid=4\t[slave pointer
(2)]\n\xe2\x8e\x9c \xe2\x86\xb3 Logitech Unifying Device. Wireless
PID:1017\tid=9\t[slave pointer (2)]\n\xe2\x8e\x9c \xe2\x86\xb3
SynPS/2 Synaptics TouchPad \tid=12\t[slave pointer
(2)]\n\xe2\x8e\x9c \xe2\x86\xb3 MCE IR Keyboard/Mouse (ite-cir)
\tid=14\t[slave pointer (2)]\n\xe2\x8e\xa3 Virtual core keyboard
\tid=3\t[master keyboard (2)]\n \xe2\x86\xb3
Virtual core XTEST keyboard \tid=5\t[slave keyboard
(3)]\n \xe2\x86\xb3 Power Button
\tid=6\t[slave keyboard (3)]\n \xe2\x86\xb3 Video Bus
\tid=7\t[slave keyboard (3)]\n \xe2\x86\xb3
Laptop_Integrated_Webcam_2M \tid=8\t[slave keyboard
(3)]\n \xe2\x86\xb3 HID 413c:8157
\tid=10\t[slave keyboard (3)]\n \xe2\x86\xb3 AT Translated Set 2
keyboard \tid=11\t[slave keyboard (3)]\n \xe2\x86\xb3
Dell WMI hotkeys \tid=13\t[slave keyboard
(3)]\n \xe2\x86\xb3 ITE8708 CIR transceiver
\tid=15\t[slave keyboard (3)]\n',
b'')

Unfortunately, what's happening here is all that extra formatting
stuff that xinput injects is also converted when doing bytes.decode()
on result[0].


In [24]: result[0].decode()
Out[24]: '⎡ Virtual core pointer \tid=2\t[master
pointer (3)]\n⎜ ↳ Virtual core XTEST pointer
\tid=4\t[slave pointer (2)]\n⎜ ↳ Logitech Unifying Device.
Wireless PID:1017\tid=9\t[slave pointer (2)]\n⎜ ↳ SynPS/2
Synaptics TouchPad \tid=12\t[slave pointer (2)]\n⎜↳
MCE IR Keyboard/Mouse (ite-cir) \tid=14\t[slave pointer
(2)]\n⎣ Virtual core keyboard \tid=3\t[master
keyboard (2)]\n ↳ Virtual core XTEST keyboard
\tid=5\t[slave keyboard (3)]\n ↳ Power Button
\tid=6\t[slave keyboard (3)]\n ↳ Video Bus
\tid=7\t[slave keyboard (3)]\n ↳
Laptop_Integrated_Webcam_2M \tid=8\t[slave keyboard
(3)]\n ↳ HID 413c:8157 \tid=10\t[slave
keyboard (3)]\n ↳ AT Translated Set 2 keyboard
\tid=11\t[slave keyboard (3)]\n ↳ Dell WMI hotkeys
\tid=13\t[slave keyboard (3)]\n ↳ ITE8708 CIR transceiver
\tid=15\t[slave keyboard (3)]\n'

And so I believe that when that decoded string is being pushed to the
logger, the logger is choking, causing the program to just hang
waiting on something... not sure which.

Though when doing this manually via ipython3, logger seems to handle
it just fine:

In [37]: message = ['Output:\n'
'- returncode:\n{0}'.format(process.returncode)]

In [38]: message.append('- stdout:\n{0}'.format(stdout.decode()))

In [39]: logging.debug('\n'.join(message))
DEBUG:root:Output:
- returncode:
0
- stdout:
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ Logitech Unifying Device. Wireless PID:1017 id=9 [slave
pointer (2)]
⎜ ↳ SynPS/2 Synaptics TouchPad id=12 [slave pointer (2)]
⎜ ↳ MCE IR Keyboard/Mouse (ite-cir) id=14 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Laptop_Integrated_Webcam_2M id=8 [slave keyboard (3)]
↳ HID 413c:8157 id=10 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=11 [slave keyboard (3)]
↳ Dell WMI hotkeys id=13 [slave keyboard (3)]
↳ ITE8708 CIR transceiver id=15 [slave keyboard (3)]

outside of ipython3, I'm running this and looking at the log file the
actual program generates and it always hangs up on the output from
'xinput list'.

Interestingly though, it gathers this info both before and after the
reboot action this script performs...

I tried attaching the actual program, but apparently it's too big