Alexander Sayapin Teacher's site

Low-latency videostreaming

Posted on Sun 23 June 2019

In Общие вопросы.

tags: video streaming low-latency robot up board


Creating a mobile robot makes you to invent something to see what your robot is seeing. It's good if your robot is an autonomous robot, but what if it's just a remotely operated platform or it allows both to work as an autonomous and as a remotely operated vehicle.

To operate the robot remotely the latency of the video should be as small as possible. To achieve that you have to use some tricks. Here are mines ;)

I tried different approaches (cvlc, ffmpeg, gstreamer), and GStreamer worked good for me. May be I missed something about the others?

All this article is about Aaeon Up Board, which is my favorite robotic board now, but the pipeline is a combination of tricks collected from Raspberry Pi blogs and forums, so it can be suiteable for RPi too, though I didn't check it works with it.

Prerequisites

Installation of the GStreamer:

apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

OpenCV Ubuntu installation:

sudo apt-get install python-opencv

Launching the streaming

Before the start do not forget to give the access to the camera:

sudo chmod 777 /dev/video0

Successfull launch (minimal latency, software encoding) on the server (ie, robot):

gst-launch-1.0 -v v4l2src ! video/x-raw,format=YUY2,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.0.105 port=5000

As we want to see what's happening with the robot as well as we want the robot to work autonomously, we have to send the videostream both to the operators computer and to the local OpenCV script. So we need the following command.

Streaming both to a remote computer and to the localhost for OpenCV (minimal latency, hardware encoding). Important! perform the previous command first!:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! "image/jpeg,width=800,height=600,framerate=30/1" ! rtpjpegpay ! tee name=t ! queue max-size-time=10000000 ! udpsink sync=false host=192.168.0.105 port=5000 t. ! queue max-size-time=10000000 ! udpsink sync=false host=127.0.0.1 port=5000

See on the local ubuntu computer

As we want to see what's the robot seeing, we have to perform the following command on the local (operators) computer (Obviously it wouldn't work on a headless system)

gst-launch-1.0 -v udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegdec ! xvimagesink sync=0

Accessing the videostream from OpenCV

To get an access to the stream from OpenCV:

import cv2
stream = cv2.VideoCapture("rtp://127.0.0.1:5000")
ret,frame = stream.read()

Please note, you will get a bunch of error messages. Do not worry! It doesn't affect the other elements, and the pipeline would work as expected.

to save the image you got:

cv2.imwrite('image.png',frame)

The file image.png would contain the image you get from your robots camera.

Other options

If you are interested what can be used for videostreaming, you may have a look at this:

CVLC option:

sudo chmod 777 /dev/video0
cvlc v4l2:///dev/video0:chroma=mjpg:width=640:height=480:fps=15 –live-caching 0 --sout '#standard{access=http,mux=mpjpeg,dst=:8080}' -vvv

You can watch the stream using the VLC palyer (open URL in it: http://:8080)

ffmpeg option:

sudo add-apt-repository ppa:jonathonf/ffmpeg-4
sudo apt-get update
sudo apt install ffmpeg
sudo chmod 777 /dev/video0
ffmpeg -f v4l2 -framerate 25 -video_size 800x600 -i /dev/video0 out.mkv

It would save the videostream to the file out.mkv. When I tried to stream it, I constantly got the error message, so I gave in and used GStreamer.

Used Sources

Getting started with images How to process VLC UDP stream over OpenCV ffmpeg wiki: webcam Howto build a cheap livestream with Raspberry Pi + Gstreamer Play webcam using gstreamer Implementing GStreamer Webcam(USB & Internal) Streaming[Mac & C++ & CLion] Installing GStreamer on Linux Live Webcam Streaming using VLC on the Command Line Streaming video with Raspi, VLC and Logitech C270 Streaming MJPG from webcam to RTSP using VLC Access IP Camera in Python OpenCV HD FPV на Raspberry Pi HD FPV на Raspberry Pi. Работа над ошибками Gstreamer basic real time streaming tutorial


tags

алфавит (1) asp.net (1) бгд (22) бисв (22) бкб (22) бме (22) бпэ (23) бпэз (3) бпэзу (1) бпм (13) бпм объявления (7) certbot (1) cheatsheet (1) checkinstall (1) csv (1) дискретная математика (21) экзамен (1) embedded rust (2) english (1) формальные грамматики (1) gdb (2) язык (1) jupyter (1) критерии (2) курсовая работа (2) lighttpd (2) low-latency (1) machine learning (3) make (1) make install (1) markdown (1) машинное обучение (1) математическая логика (1) Математические основы кмпьютерной графики (1) Математические основы компьютерного моделирования (1) Методы оптимизации (9) методы оптмимизации (1) методы принятия решений (1) миа (1) мии (7) мик (7) мим (2) мип (8) мит (34) миу (8) миз (7) ml (1) mono (1) natural language processing (1) nlp (1) nucleo (2) объявления (28) оформление (2) openocd (2) openpgp (1) pandas (1) pgp (1) подтверждение вывода (1) programming (3) python (3) robot (1) robotics (2) setup (6) шпаргалка (1) smartcard (1) ssh (1) ssl (1) STM32 (2) streaming (1) строка (1) тб (21) teaching (1) teaching statement (1) теоретические основы цифровой обработки изображений (1) тест (1) учебник (1) up board (1) video (1) вкр (2) xls (1)