Displaying Opencv images in PyQT

The code for this tutorial is here

Opencv provides are useful, but limited, method of building a GUI. A much more complete system could be acheived using pyqt. The question is, how do we display images. There are quite a few possible routes but perhaps the easiest is to use QLabel since it has a setPixmap function. Below is some code that creates two labels. It then creates a grey pixmap and displays it one of the labels. code: staticLabel1.py

from PyQt5.QtWidgets import QWidget, QApplication, QLabel, QVBoxLayout
from PyQt5.QtGui import QPixmap, QColor
import sys


class App(QWidget):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Qt static label demo")
        width = 640
        height = 480
        # create the label that holds the image
        self.image_label = QLabel(self)
        # create a text label
        self.textLabel = QLabel('Demo')

        # create a vertical box layout and add the two labels
        vbox = QVBoxLayout()
        vbox.addWidget(self.image_label)
        vbox.addWidget(self.textLabel)
        # set the vbox layout as the widgets layout
        self.setLayout(vbox)
        # create a grey pixmap
        grey = QPixmap(width, height)
        grey.fill(QColor('darkGray'))
        # set the image image to the grey pixmap
        self.image_label.setPixmap(grey)


if __name__ == "__main__":
    app = QApplication(sys.argv)
    a = App()
    a.show()
    sys.exit(app.exec_())

It should look like this:

Ok, so how do with display an image. Well we could us Qt to load the image directly, but I want to do this with opencv so it can be integrated into a computer vision app. To do this load the image with cv2.imread and then we convert this to a QPixmap and rescale. Here is an example

 def convert_cv_qt(self, cv_img):
        """Convert from an opencv image to QPixmap"""
        rgb_image = cv2.cvtColor(cv_img, cv2.COLOR_BGR2RGB)
        h, w, ch = rgb_image.shape
        bytes_per_line = ch * w
        convert_to_Qt_format = QtGui.QImage(rgb_image.data, w, h, bytes_per_line, QtGui.QImage.Format_RGB888)
        p = convert_to_Qt_format.scaled(self.disply_width, self.display_height, Qt.KeepAspectRatio)
        return QPixmap.fromImage(p)

This is part of a class that defines self.disply_width and self.display_height. The full code is here (staticLabel2.ppy). The result should look like

Video

So can we now use this to display video? We could open the webcam and update our image every frame. liveLabel1.py attempts to do this - but it does not work. The problem is the video capture loop is blocking the processing. This means the message system that Qt uses to do things like draw the widgets never gets called. We need to fix this by capturing the webcam in a seperate thread. Here's the code:

from PyQt5 import QtGui
from PyQt5.QtWidgets import QWidget, QApplication, QLabel, QVBoxLayout
from PyQt5.QtGui import QPixmap
import sys
import cv2
from PyQt5.QtCore import pyqtSignal, pyqtSlot, Qt, QThread
import numpy as np


class VideoThread(QThread):
    change_pixmap_signal = pyqtSignal(np.ndarray)

    def run(self):
        # capture from web cam
        cap = cv2.VideoCapture(0)
        while True:
            ret, cv_img = cap.read()
            if ret:
                self.change_pixmap_signal.emit(cv_img)


class App(QWidget):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Qt live label demo")
        self.disply_width = 640
        self.display_height = 480
        # create the label that holds the image
        self.image_label = QLabel(self)
        self.image_label.resize(self.disply_width, self.display_height)
        # create a text label
        self.textLabel = QLabel('Webcam')

        # create a vertical box layout and add the two labels
        vbox = QVBoxLayout()
        vbox.addWidget(self.image_label)
        vbox.addWidget(self.textLabel)
        # set the vbox layout as the widgets layout
        self.setLayout(vbox)

        # create the video capture thread
        self.thread = VideoThread()
        # connect its signal to the update_image slot
        self.thread.change_pixmap_signal.connect(self.update_image)
        # start the thread
        self.thread.start()



    @pyqtSlot(np.ndarray)
    def update_image(self, cv_img):
        """Updates the image_label with a new opencv image"""
        qt_img = self.convert_cv_qt(cv_img)
        self.image_label.setPixmap(qt_img)
    
    def convert_cv_qt(self, cv_img):
        """Convert from an opencv image to QPixmap"""
        rgb_image = cv2.cvtColor(cv_img, cv2.COLOR_BGR2RGB)
        h, w, ch = rgb_image.shape
        bytes_per_line = ch * w
        convert_to_Qt_format = QtGui.QImage(rgb_image.data, w, h, bytes_per_line, QtGui.QImage.Format_RGB888)
        p = convert_to_Qt_format.scaled(self.disply_width, self.display_height, Qt.KeepAspectRatio)
        return QPixmap.fromImage(p)
    
if __name__=="__main__":
    app = QApplication(sys.argv)
    a = App()
    a.show()
    sys.exit(app.exec_())

Note, we send a numpy array using PyQt's signal slot mechanism to ensure thread saftey. This should now display live images for your webcam. There might, however, be a error created when you close the app. This is because the capture device has not been shutdown correctly. (On my mac this does not happen, but on my Linux machine it does). To solve this we need to override App's closeEvent function and single to the thread to shut down.

from PyQt5 import QtGui
from PyQt5.QtWidgets import QWidget, QApplication, QLabel, QVBoxLayout
from PyQt5.QtGui import QPixmap
import sys
import cv2
from PyQt5.QtCore import pyqtSignal, pyqtSlot, Qt, QThread
import numpy as np


class VideoThread(QThread):
    change_pixmap_signal = pyqtSignal(np.ndarray)

    def __init__(self):
        super().__init__()
        self._run_flag = True

    def run(self):
        # capture from web cam
        cap = cv2.VideoCapture(0)
        while self._run_flag:
            ret, cv_img = cap.read()
            if ret:
                self.change_pixmap_signal.emit(cv_img)
        # shut down capture system
        cap.release()

    def stop(self):
        """Sets run flag to False and waits for thread to finish"""
        self._run_flag = False
        self.wait()


class App(QWidget):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Qt live label demo")
        self.disply_width = 640
        self.display_height = 480
        # create the label that holds the image
        self.image_label = QLabel(self)
        self.image_label.resize(self.disply_width, self.display_height)
        # create a text label
        self.textLabel = QLabel('Webcam')

        # create a vertical box layout and add the two labels
        vbox = QVBoxLayout()
        vbox.addWidget(self.image_label)
        vbox.addWidget(self.textLabel)
        # set the vbox layout as the widgets layout
        self.setLayout(vbox)

        # create the video capture thread
        self.thread = VideoThread()
        # connect its signal to the update_image slot
        self.thread.change_pixmap_signal.connect(self.update_image)
        # start the thread
        self.thread.start()

    def closeEvent(self, event):
        self.thread.stop()
        event.accept()



    @pyqtSlot(np.ndarray)
    def update_image(self, cv_img):
        """Updates the image_label with a new opencv image"""
        qt_img = self.convert_cv_qt(cv_img)
        self.image_label.setPixmap(qt_img)
    
    def convert_cv_qt(self, cv_img):
        """Convert from an opencv image to QPixmap"""
        rgb_image = cv2.cvtColor(cv_img, cv2.COLOR_BGR2RGB)
        h, w, ch = rgb_image.shape
        bytes_per_line = ch * w
        convert_to_Qt_format = QtGui.QImage(rgb_image.data, w, h, bytes_per_line, QtGui.QImage.Format_RGB888)
        p = convert_to_Qt_format.scaled(self.disply_width, self.display_height, Qt.KeepAspectRatio)
        return QPixmap.fromImage(p)
    
if __name__=="__main__":
    app = QApplication(sys.argv)
    a = App()
    a.show()
    sys.exit(app.exec_())

view raw opencv_qt_label.md hosted with ❤ by GitHub

March 16, 2018

Piping numpy arrays to other processes in python

To pipe data from one process to another as a stream in python we need to pickle the object and pass it to the pipe stream.
In this example I’ve used Numpy arrays but this could be applied to any object that can be pickled in Python.
This took far too long to get working and I could find little information online on how to put it all together so here it is.
This code is Python 3 only, I’ve only run this on a Mac.

I’ve used binary as the stream rather than text purley becuase of effiencies. Numpy arrays can get huge! This means readline() is not
going to work. Instead, I send a single control byte , 1, for data and 0 for stop. This could be extended to include other control operations.
I then send the length of the data as a 8 byte int, followed by the data itself.

simpleSend.py

import numpy as np
import pickle
import sys
import io
import time

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    print(control_data.decode('latin-1'),end='',flush=True)  #convert this to a byte array
    print(dlen.decode('latin-1'), end='', flush=True)   #encode the data and write it
    print(dataStr.decode('latin-1'), end='', flush=True)  # end='' will remove that extra \r\n

def send_stop():
    print(control_stop.decode('latin-1'), end='', flush=True) 

#set the stdout such that it prints in latin-1,   sys.stdout.detach() is a binary stream
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')

for p in range(10):
    arr=np.ones((5000,500))*p  #generate some data
    send_data(arr)
    #the sleep is purely for testing and can be removed, ie does the reader fall over after a long delay
    time.sleep(.1)
send_stop()

simpleReceiver.py

import numpy as np
import sys
import pickle

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

while True:
    data=sys.stdin.buffer.read(1)   #read the control byte
    if data==control_data:
        data=sys.stdin.buffer.read(8)  #read the data length
        dlen=int.from_bytes(data, byteorder='big')
        print('data lenght %d'%dlen)        
        data=sys.stdin.buffer.read(dlen) #read the data        
        npd=pickle.loads(data)  #unpickle
        print(npd.shape)
        print(npd.max())
    elif data==control_stop:
        print('stopped')
        break
    else:
        print('Oh no')

to run this
python simpleSend.py | python simpleReceiver.py

If we want to use Python’s subprocess module to start simpleReceiver.py we basically need to write to the STDIN instead of print

import numpy as np
import pickle
import sys
import subprocess as sp

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr,buff):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    mp.stdin.write(control_data)
    mp.stdin.write(dlen)
    mp.stdin.write(dataStr)
    mp.stdin.flush() #not sure this needed
     
def send_stop(mp):
    mp.stdin.write(control_stop)
    mp.stdin.flush()
     
try:
    mp = sp.Popen("python3 simpleReceiver.py",  shell = True,stdin=sp.PIPE)   
except sp.CalledProcessError as err:
    print('ERROR:', err)
    sys.exit(-1)

for p in range(10):
    arr=np.ones((5000,5000))*p  #generate some data
    send_data(arr,mp)
send_stop(mp)

With such a large array 5000×5000 this takes sometime. Running it through the python profiler indicates about 75% of the time is taken by pickle.dumps and most of the rest of the remaining 25% is taken by the write operation. Numpy’s own method gives a speed increase. Replacing dataStr=pickle.dumps(arr) with dataStr=arr.tobytes() and npd=pickle.loads(data) with npd=np.frombuffer(data) more than halves the time taken but lose the shape and dtype information. This would have to be sent along with the data.

view raw

numpyPipe.md

hosted with ❤ by GitHub

December 1, 2017December 1, 2017

How to write lossless video in Python

OpenCV does a reasonable job of reading videos from file or webcams. It’s simple and mostly works. When it comes to writing videos,
it however leaves a lot to be desired. There is little control over the codecs and it is almost impossible to know which codecs are installed. It also wants to know things like the frame size at intailisation. This isn’t always a problem, but if you don’t know it yet it means you have to set up the video writer inside your main processing loop.

To make something as cross-platform compatible as possible it would be nice to use FFmpeg. There are a few python wrappers around, but as far as I can tell they are mainly used for transcoding type applications. One solution is run FFmpeg as a subprocess and set its input to accept a pipe. Then every video frame is passed through the pipe. You write this yourself, in fact it’s only a few lines of code. However, the scikit-video package will do this for us, with some nice boilerplate to make life easier.

The steps are:

install FFmpeg — if you running on Linux use your system’s package manager if it’s not already installed. If you’re unlucky enough to be using Windows you need to download the zip file from here, and add the bin directory to your system’s path.
install scikit-video –I tried installing scikit-video via pip on my Anaconda distro but the version was too old. Instead, I cloned the github version and installed that. Instructions are provided on github.

Below is a simple example the grabs from your webcam and records lossless video.

#test recording of video
import cv2
import skvideo.io


capture=cv2.VideoCapture(0) #open the default webcam
outputfile = "test.mp4"   #our output filename
writer = skvideo.io.FFmpegWriter(outputfile, outputdict={
  '-vcodec': 'libx264',  #use the h.264 codec
  '-crf': '0',           #set the constant rate factor to 0, which is lossless
  '-preset':'veryslow'   #the slower the better compression, in princple, try 
                         #other options see https://trac.ffmpeg.org/wiki/Encode/H.264
}) 
while True:
    ret,frame=capture.read()
    if ret==False:
        print("Bad frame")
        break
    cv2.imshow('display',frame)
    writer.writeFrame(frame[:,:,::-1])  #write the frame as RGB not BGR
    ret=cv2.waitKey(10)
    if ret==27: #esc
        break

writer.close() #close the writer
capture.release()
cv2.destroyAllWindows()

view raw

pythonVideoCapture.md

hosted with ❤ by GitHub

December 10, 2016December 10, 2016

Python for Matlab users

Python has a number of benefits over Matlab, and a number of research groups are making the switch. I’m not going to do a complete tutorial here on how to programme in python for Matlab users since there are plenty on the web. Instead I wanted to make a few note on the practical aspects such as IDEs and how to use python.

I’m running python on Mint linux but most of this should be applicable to Windows and Macs.

There are several types of python implementation such as CPython, Anaconda, Python(x,y). CPython is the original and probably the best to start with since it probably is the one that comes with your Linux distro. Normal Python compiles into bytecode, other implementations can compile into C (Cython, not to be confused with CPython), .NET (IronPython) among others. Unless you have good reason stick with CPython.
There are two slightly incompatible version of Python: 2.x and 3.x. Which one to use, depends on who you ask. I’ve picked 3 since it’s the future. You’ll probably need to know the differences at some point since there’s a lot 2.x around.
Python vs iPython. Python scripts can be run from the command line, eg. python script.py or you can use the interactive shell by simply running python. ipython is a souped up version of the interactive shell and has additional functionality. It it more like the Matlab command prompt and is the one to use when writing scripts.
IDEs. There are a lot of IDEs for python. I don’t think any of them are as good as the Matlab IDE just yet. Spyder is probably the closest.
Instead of toolboxes, Python has packages. These can be installed a number of ways. If you are running Linux your distro will have most of the more common ones that can be installed via apt-get or its equivalent. These packages will be installed system wide and probably be slightly out of date but possibly safer. Another method is to use the Python package index PyPi and uses the program pip to install them. pip will have a larger, more up to date database of packages. It also has the ability to restrict the download to your user space only or use a completely self-contained environment virtualenv to run your scripts in. Great if you need incompatible versions for different projects or don’t want to screw up other projects. I would recommend looking into this.
When installing using apt-get, you’ll need suffix package names with a 3 to get the python 3 version, otherwise it will install version 2. It’s the same with spyder, install spyder3. Some packages work with 2 and 3 so there’s only one version to add to the confusion.

Plotting and images

A graphically plotting library matplotlib is provided and is similar to Matlab
In ipython plots maybe inline (i.e displayed in the console). If you don’t want this run
%matplotlib qt and %matplotlib inline to return to inline.

Matrices and computer vision

Matlab’s raison d’être is matrix manipulation and but by itself python’s support is limited. Conveniently there is a package that however supports this called numpy which is part of scipy and that provides a huge scientific library. Sympy can be used for symbolic maths manipulation. It’s the numpy library that makes python such useful replacement for Matlab.
As far as replacing the image processing and computer vision toolboxs, there is a range of options. Opencv has a python wrapper, which conveniently uses numpy arrays to hold images. You’ll need opencv 3 to get python 3 support. Scikit-image has a huge collection of routines. They both use numpy arrays so you can use both at once although you will have to covert the data types. Another option is Pillow which is forked from the defunct PIL library.