Posted on Sun 23 June 2019
In Общие вопросы.
tags: video streaming low-latency robot up board
Creating a mobile robot makes you to invent something to see what your robot is seeing. It's good if your robot is an autonomous robot, but what if it's just a remotely operated platform or it allows both to work as an autonomous and as a remotely operated vehicle.
To operate the robot remotely the latency of the video should be as small as possible. To achieve that you have to use some tricks. Here are mines ;)
I tried different approaches (cvlc, ffmpeg, gstreamer), and GStreamer worked good for me. May be I missed something about the others?
All this article is about Aaeon Up Board, which is my favorite robotic board now, but the pipeline is a combination of tricks collected from Raspberry Pi blogs and forums, so it can be suiteable for RPi too, though I didn't check it works with it.
Installation of the GStreamer:
apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
OpenCV Ubuntu installation:
sudo apt-get install python-opencv
Before the start do not forget to give the access to the camera:
sudo chmod 777 /dev/video0
Successfull launch (minimal latency, software encoding) on the server (ie, robot):
gst-launch-1.0 -v v4l2src ! video/x-raw,format=YUY2,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.0.105 port=5000
As we want to see what's happening with the robot as well as we want the robot to work autonomously, we have to send the videostream both to the operators computer and to the local OpenCV script. So we need the following command.
Streaming both to a remote computer and to the localhost for OpenCV (minimal latency, hardware encoding). Important! perform the previous command first!:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! "image/jpeg,width=800,height=600,framerate=30/1" ! rtpjpegpay ! tee name=t ! queue max-size-time=10000000 ! udpsink sync=false host=192.168.0.105 port=5000 t. ! queue max-size-time=10000000 ! udpsink sync=false host=127.0.0.1 port=5000
As we want to see what's the robot seeing, we have to perform the following command on the local (operators) computer (Obviously it wouldn't work on a headless system)
gst-launch-1.0 -v udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegdec ! xvimagesink sync=0
To get an access to the stream from OpenCV:
import cv2
stream = cv2.VideoCapture("rtp://127.0.0.1:5000")
ret,frame = stream.read()
Please note, you will get a bunch of error messages. Do not worry! It doesn't affect the other elements, and the pipeline would work as expected.
to save the image you got:
cv2.imwrite('image.png',frame)
The file image.png would contain the image you get from your robots camera.
If you are interested what can be used for videostreaming, you may have a look at this:
CVLC option:
sudo chmod 777 /dev/video0
cvlc v4l2:///dev/video0:chroma=mjpg:width=640:height=480:fps=15 –live-caching 0 --sout '#standard{access=http,mux=mpjpeg,dst=:8080}' -vvv
You can watch the stream using the VLC palyer (open URL in it: http://
ffmpeg option:
sudo add-apt-repository ppa:jonathonf/ffmpeg-4
sudo apt-get update
sudo apt install ffmpeg
sudo chmod 777 /dev/video0
ffmpeg -f v4l2 -framerate 25 -video_size 800x600 -i /dev/video0 out.mkv
It would save the videostream to the file out.mkv. When I tried to stream it, I constantly got the error message, so I gave in and used GStreamer.
Getting started with images How to process VLC UDP stream over OpenCV ffmpeg wiki: webcam Howto build a cheap livestream with Raspberry Pi + Gstreamer Play webcam using gstreamer Implementing GStreamer Webcam(USB & Internal) Streaming[Mac & C++ & CLion] Installing GStreamer on Linux Live Webcam Streaming using VLC on the Command Line Streaming video with Raspi, VLC and Logitech C270 Streaming MJPG from webcam to RTSP using VLC Access IP Camera in Python OpenCV HD FPV на Raspberry Pi HD FPV на Raspberry Pi. Работа над ошибками Gstreamer basic real time streaming tutorial