Optics, Object Tracking and Identification
Here the robot is given a text prompt (bottom left). This is a description of the target shown in the top right. The robot then navigates the building autonomously (left images, showing two experiments), mapping as it goes until it finds the object. An example of zero-shot learning.
This example is running in a simulated environment
An example with a real robot.
The preprint of Juzhou Chen paper is here:
Han, X., Oishi, N., Tian, Y. et al. ETTrack: enhanced temporal motion predictor for multi-object tracking. Appl Intell 55, 33 (2025). https://doi.org/10.1007/s10489-024-05866-4
Our new paper on tracking multiple people using a transformer to learn complex motion patterns.
But will it work…..
Source: NEC unveils facial recognition system for 2020 Tokyo Olympics – The Verge
List all devices on windows using ffmpeg
ffmpeg -list_devices true -f dshow -i dummy
and on Linux
v4l2-ctl –list-devices
To get the device capabilites
ffmpeg -f dshow -list_options true -i video="Mobius"
where “Mobius” is the name of the camera.
On the Mac use
ffmpeg -f avfoundation -list_devices true -i ""OpenCV does a reasonable job of reading videos from file or webcams. It’s simple and mostly works. When it comes to writing videos,
it however leaves a lot to be desired. There is little control over the codecs and it is almost impossible to know which codecs are installed. It also wants to know things like the frame size at intailisation. This isn’t always a problem, but if you don’t know it yet it means you have to set up the video writer inside your main processing loop.
To make something as cross-platform compatible as possible it would be nice to use FFmpeg. There are a few python wrappers around, but as far as I can tell they are mainly used for transcoding type applications. One solution is run FFmpeg as a subprocess and set its input to accept a pipe. Then every video frame is passed through the pipe. You write this yourself, in fact it’s only a few lines of code. However, the scikit-video package will do this for us, with some nice boilerplate to make life easier.
The steps are:
Below is a simple example the grabs from your webcam and records lossless video.
#test recording of video
import cv2
import skvideo.io
capture=cv2.VideoCapture(0) #open the default webcam
outputfile = "test.mp4" #our output filename
writer = skvideo.io.FFmpegWriter(outputfile, outputdict={
'-vcodec': 'libx264', #use the h.264 codec
'-crf': '0', #set the constant rate factor to 0, which is lossless
'-preset':'veryslow' #the slower the better compression, in princple, try
#other options see https://trac.ffmpeg.org/wiki/Encode/H.264
})
while True:
ret,frame=capture.read()
if ret==False:
print("Bad frame")
break
cv2.imshow('display',frame)
writer.writeFrame(frame[:,:,::-1]) #write the frame as RGB not BGR
ret=cv2.waitKey(10)
if ret==27: #esc
break
writer.close() #close the writer
capture.release()
cv2.destroyAllWindows()
Some useful Computer Vision links
Source: jbhuang0604/awesome-computer-vision: A curated list of awesome computer vision resources
A really useful list of key papers using Deep learning in computer vision
Source: kjw0612/awesome-deep-vision: A curated list of deep learning resources for computer vision
I use axis IP camera a lot for capturing images and video. The image quality is great and they are highly customisable. I use a P1344 camera and it supports still images, MJPEG and H264. The still image are fine for capturing a one off, but too slow for video work. For this I need either MJPEG or H264. Both have there pros and cons.
MJPEG is existentially a sequence of JPEG images. It’s easy to use and the quality is good, depending on the compression settings. The downside is the bitrate over the network is larger than H264.
H264 is a lossy video compression format that has become ubiquitous on the internet these days for compressed video and blue-ray videos.
The camera can be controlled using the url. To get a H264 stream (using VLC in this case, but ffplay works perfectly well), at the command prompt type
vlc rtsp://192.168.0.103:554/axis-media/media.amp
changing the IP address to that of your camera. You can also add in the camera’s username and password using:
vlc "rtsp://192.168.0.103:554/axis-media/media.amp?user=XXX&password=XXXX"
The image resolution can be changed with
vlc rtsp://192.168.0.103:554/axis-media/media.amp?resolution=640x480
The resolution is camera dependent. There are a bunch of different settings, such as bit rate, compression, you can apply see the AXIS VAPIX documentation for the whole list. An easy way to do this is by using the cameras settings page to create a Stream Profile. There are a number built in and you can select them like so
vlc rtsp://192.168.0.103:554/axis-media/media.amp?streamprofile=Quality
Still images can be captured, by using
http://192.168.0.103/axis-cgi/jpg/image.cgi?resolution=320x240&compression=25
here I’ve selected the resolution and compression factor. You can grab the image by placing the above url into a browser.
Looking at the above shows subregions of example images captured at full resolution using JPG. We can see that there is significant compression artifacts in the image even at low compression ratios. Setting the compression ratio less than 40 appears to have little effect on image quality.
H264 streams appear to be similarly affected. The bitmap image shows some improvement however the data rate to transmit this is considerably larger.
Note that the RMS errors are calculated from the JPEG image with a compression factor of 0.
ffplay "http://192.168.0.103/axis-cgi/mjpg/video.cgi?resolution=640x480&fps=15"
uses 60% of one core of my Odroid XU4 and 17% on my 2.7 GHz iMac
ffplay "rtsp://192.168.0.103:554/axis-media/media.amp?resolution=640x480&fps=15"
uses 88% on the Odroid and 14% on the iMac.