Category: Python

Posted on

Piping numpy arrays to other processes in python

To pipe data from one process to another as a stream in python we need to pickle the object and pass it to the pipe stream.
In this example I’ve used Numpy arrays but this could be applied to any object that can be pickled in Python.
This took far too long to get working and I could find little information online on how to put it all together so here it is.
This code is Python 3 only, I’ve only run this on a Mac.

I’ve used binary as the stream rather than text purley becuase of effiencies. Numpy arrays can get huge! This means readline() is not
going to work. Instead, I send a single control byte , 1, for data and 0 for stop. This could be extended to include other control operations.
I then send the length of the data as a 8 byte int, followed by the data itself.

simpleSend.py

import numpy as np
import pickle
import sys
import io
import time

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    print(control_data.decode('latin-1'),end='',flush=True)  #convert this to a byte array
    print(dlen.decode('latin-1'), end='', flush=True)   #encode the data and write it
    print(dataStr.decode('latin-1'), end='', flush=True)  # end='' will remove that extra \r\n

def send_stop():
    print(control_stop.decode('latin-1'), end='', flush=True) 

#set the stdout such that it prints in latin-1,   sys.stdout.detach() is a binary stream
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')

for p in range(10):
    arr=np.ones((5000,500))*p  #generate some data
    send_data(arr)
    #the sleep is purely for testing and can be removed, ie does the reader fall over after a long delay
    time.sleep(.1)
send_stop()

simpleReceiver.py

import numpy as np
import sys
import pickle

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

while True:
    data=sys.stdin.buffer.read(1)   #read the control byte
    if data==control_data:
        data=sys.stdin.buffer.read(8)  #read the data length
        dlen=int.from_bytes(data, byteorder='big')
        print('data lenght %d'%dlen)        
        data=sys.stdin.buffer.read(dlen) #read the data        
        npd=pickle.loads(data)  #unpickle
        print(npd.shape)
        print(npd.max())
    elif data==control_stop:
        print('stopped')
        break
    else:
        print('Oh no')

to run this
python simpleSend.py | python simpleReceiver.py

If we want to use Python’s subprocess module to start simpleReceiver.py we basically need to write to the STDIN instead of print

import numpy as np
import pickle
import sys
import subprocess as sp

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr,buff):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    mp.stdin.write(control_data)
    mp.stdin.write(dlen)
    mp.stdin.write(dataStr)
    mp.stdin.flush() #not sure this needed
     
def send_stop(mp):
    mp.stdin.write(control_stop)
    mp.stdin.flush()
     
try:
    mp = sp.Popen("python3 simpleReceiver.py",  shell = True,stdin=sp.PIPE)   
except sp.CalledProcessError as err:
    print('ERROR:', err)
    sys.exit(-1)

for p in range(10):
    arr=np.ones((5000,5000))*p  #generate some data
    send_data(arr,mp)
send_stop(mp)

With such a large array 5000×5000 this takes sometime. Running it through the python profiler indicates about 75% of the time is taken by pickle.dumps and most of the rest of the remaining 25% is taken by the write operation. Numpy’s own method gives a speed increase. Replacing dataStr=pickle.dumps(arr) with dataStr=arr.tobytes() and npd=pickle.loads(data) with npd=np.frombuffer(data) more than halves the time taken but lose the shape and dtype information. This would have to be sent along with the data.

view raw

numpyPipe.md

hosted with ❤ by GitHub

Posted on

How to write lossless video in Python

OpenCV does a reasonable job of reading videos from file or webcams. It’s simple and mostly works. When it comes to writing videos,
it however leaves a lot to be desired. There is little control over the codecs and it is almost impossible to know which codecs are installed. It also wants to know things like the frame size at intailisation. This isn’t always a problem, but if you don’t know it yet it means you have to set up the video writer inside your main processing loop.

To make something as cross-platform compatible as possible it would be nice to use FFmpeg. There are a few python wrappers around, but as far as I can tell they are mainly used for transcoding type applications. One solution is run FFmpeg as a subprocess and set its input to accept a pipe. Then every video frame is passed through the pipe. You write this yourself, in fact it’s only a few lines of code. However, the scikit-video package will do this for us, with some nice boilerplate to make life easier.

The steps are:

  1. install FFmpeg — if you running on Linux use your system’s package manager if it’s not already installed. If you’re unlucky enough to be using Windows you need to download the zip file from here, and add the bin directory to your system’s path.
  2. install scikit-video –I tried installing scikit-video via pip on my Anaconda distro but the version was too old. Instead, I cloned the github version and installed that. Instructions are provided on github.

Below is a simple example the grabs from your webcam and records lossless video.

#test recording of video
import cv2
import skvideo.io


capture=cv2.VideoCapture(0) #open the default webcam
outputfile = "test.mp4"   #our output filename
writer = skvideo.io.FFmpegWriter(outputfile, outputdict={
  '-vcodec': 'libx264',  #use the h.264 codec
  '-crf': '0',           #set the constant rate factor to 0, which is lossless
  '-preset':'veryslow'   #the slower the better compression, in princple, try 
                         #other options see https://trac.ffmpeg.org/wiki/Encode/H.264
}) 
while True:
    ret,frame=capture.read()
    if ret==False:
        print("Bad frame")
        break
    cv2.imshow('display',frame)
    writer.writeFrame(frame[:,:,::-1])  #write the frame as RGB not BGR
    ret=cv2.waitKey(10)
    if ret==27: #esc
        break

writer.close() #close the writer
capture.release()
cv2.destroyAllWindows()

view raw

pythonVideoCapture.md

hosted with ❤ by GitHub

Posted on

Python for Matlab users

Python has a number of benefits over Matlab, and a number of research groups are making the switch. I’m not going to do a complete tutorial here on how to programme in python for Matlab users since there are plenty on the web. Instead I wanted to make a few note on the practical aspects such as IDEs and how to use python.

I’m running python on Mint linux but most of this should be applicable to Windows and Macs.

  • There are several types of python implementation such as CPython, Anaconda, Python(x,y). CPython is the original and probably the best to start with since it probably is the one that comes with your Linux distro. Normal Python compiles into bytecode, other implementations can compile into C (Cython, not to be confused with CPython), .NET (IronPython) among others. Unless you have good reason stick with CPython.
  • There are two slightly incompatible version of Python: 2.x and 3.x.  Which one to use, depends on who you ask. I’ve picked 3 since it’s the future. You’ll probably need to know the differences at some point since there’s a lot 2.x around.
  • Python vs iPython. Python scripts can be run from the command line, eg. python script.py  or you can use the interactive shell by simply running python. ipython is a souped up version of the interactive shell and has additional functionality. It it more like the Matlab command prompt and is the one to use when writing scripts.
  • IDEs. There are a lot of IDEs for python. I don’t think any of them are as good as the Matlab IDE just yet. Spyder is probably the closest.
  • Instead of toolboxes, Python has packages. These can be installed a number of ways. If you are running Linux your distro will have most of the more common ones that can be installed via apt-get or its equivalent. These packages will be installed system wide and probably be slightly out of date but possibly safer. Another method is to use the Python package index PyPi and uses the program pip to install them. pip will have a larger, more up to date database of packages. It also has the ability to restrict the download to your user space only or use a completely self-contained environment virtualenv to run your scripts in. Great if you need incompatible versions for different projects or don’t want to screw up other projects. I would recommend looking into this.
  • When installing using apt-get, you’ll need suffix package names with a 3 to get the python 3 version, otherwise it will install version 2. It’s the same with spyder, install spyder3. Some packages work with 2 and 3 so there’s only one version to add to the confusion.

Plotting and images

  • A graphically plotting library matplotlib is provided and is similar to Matlab
  • In ipython plots maybe inline (i.e displayed in the console). If you don’t want this run
    %matplotlib qt and  %matplotlib inline to return to inline.

Matrices and computer vision

  • Matlab’s raison d’être is matrix manipulation and but by itself python’s support is limited. Conveniently there is a package that however supports this called numpy which is part of scipy and that provides a huge scientific library. Sympy can be used for symbolic maths manipulation. It’s the numpy library that makes python such useful replacement for Matlab.
  • As far as replacing the image processing and computer vision toolboxs, there is a range of options. Opencv has a python wrapper, which conveniently uses numpy arrays to hold images. You’ll need opencv 3 to get python 3 support. Scikit-image has a huge collection of routines. They both use numpy arrays so you can use both at once although you will have to covert the data types. Another option is Pillow which is forked from the defunct PIL library.