Communicate between Python and Node.js

[Manish]

I've been tasked to write a module that sends data from Django to a Node.jsserver running on the same machine. Some magic happens in node and I recv the results back, which are then rendered using Django templates.

At first I thought to use the requests library to GET/POST data to node, but I googled around and it seems lots of people think TCP sockets are the way to go. I tried implementing my own using several examples I have found online. It *kind of* works. It seems like I get blocked while trying to receive data back in the recv() loop. I never reach the end. I'm not an expert in sockets/networking, but maybe I'm not wrong in guessing it is because of the non-blocking nature of Node.js ?

A Stackoverflow post helped a little more in figuring things out (though I'm not sure if I'm correct here). Right now, I'm failing during connect() - I get "Operation now in progress".

So my question is, how can I get recv() to work properly so that data is seamlessly passed back and forth between my Python script and the node server.. Am I taking the right approach? Is there any better way?

Relevant scripts:
1) http://bpaste.net/show/NI2z9RhbT3HVtLVWUKuq/
2) http://bpaste.net/show/YlulEZBTDE5KS5ZvSyET/

Thanks!

 

[Chris Angelico]

At first I thought to use the requests library to GET/POST data to node, but I googled around and it seems lots of people think TCP sockets are the way to go. I tried implementing my own using several examples I have found online. It *kind of* works. It seems like I get blocked while trying to receive data back in the recv() loop. I never reach the end. I'm not an expertin sockets/networking, but maybe I'm not wrong in guessing it is because of the non-blocking nature of Node.js ?

Do you need to use non-blocking sockets here? I think, from a quick
skim of your code, that you'd do better with a blocking socket. Tip:
Any time you have a sleep(1) call in your code, look to see if it's
doing the wrong thing. In this case, I'm pretty sure it is. Sleeping
for a second and then trying to read from a nonblocking socket seems
like a messy way to just read until you have what you want.

There's another thread happening at the moment about networking in
Python. You may find it of interest.

ChrisA

 

[Steveskok]

Python: read, process and write​

First up, let’s create a simple Python process that reads from a named pipe A, processes the data, and then writes it to named pipe B (for simplicity, the process_msg() function here returns the read data). The script first creates the named pipe A using os.mkfifo() command.

import os
import select

IPC_FIFO_NAME_A = "pipe_a"
IPC_FIFO_NAME_B = "pipe_b"

def get_message(fifo):
'''Read n bytes from pipe. Note: n=24 is an example'''
return os.read(fifo, 24)

def process_msg(msg):
'''Process message read from pipe'''
return msg

if __name__ == "__main__":
os.mkfifo(IPC_FIFO_NAME_A) # Create Pipe A

try:
fifo_a = os.open(IPC_FIFO_NAME_A, os.O_RDONLY | os.O_NONBLOCK) # pipe is opened as read only and in a non-blocking mode
print('Pipe A ready')

while True:
try:
fifo_b = os.open(IPC_FIFO_NAME_B, os.O_WRONLY)
print("Pipe B ready")
break
except:
# Wait until Pipe B has been initialized
pass

try:
poll = select.poll()
poll.register(fifo_a, select.POLLIN)

try:
while True:
if (fifo_a, select.POLLIN) in poll.poll(1000): # Poll every 1 sec
msg = get_message(fifo_a) # Read from Pipe A
msg = process_msg(msg) # Process Message
os.write(fifo_b, msg) # Write to Pipe B

print('----- Received from JS -----')
print(" " + msg.decode("utf-8"))
finally:
poll.unregister(fifo_a)
finally:
os.close(fifo_a)
finally:
os.remove(IPC_FIFO_NAME_A)
os.remove(IPC_FIFO_NAME_B)

Node.js: write and read​

Next, let’s write a simple Node.js program that writes into the pipe. There are a few packages that do this named-pipe, fifo-js. However, the NodeJS offers functions that make it convenient to set up IPC with pipes. The script below writes data every 1 second into Pipe A (send data), and reads from Pipe B (receive processed data from Python). For simplicity, the data here is the current time as a string.

const fs = require('fs');
const { spawn, fork } = require('child_process');

const path_a = 'pipe_a';
const path_b = 'pipe_b';
let fifo_b = spawn('mkfifo', [path_b]); // Create Pipe B

fifo_b.on('exit', function(status) {
console.log('Created Pipe B');

const fd = fs.openSync(path_b, 'r+');
let fifoRs = fs.createReadStream(null, { fd });
let fifoWs = fs.createWriteStream(path_a);

console.log('Ready to write')

setInterval(() => {
console.log('----- Send packet -----');
fifoWs.write(`${new Date().toISOString()}`);
}, 1000); // Write data at 1 second interval

fifoRs.on('data', data => {

now_time = new Date();
sent_time = new Date(data.toString());
latency = (now_time - sent_time);

console.log('----- Received packet -----');
console.log(' Date : ' + data.toString());
console.log(' Latency: ' + latency.toString() + ' ms');
});
});