What: Our objective
The basic purpose of this project is to read in a video feed / frames of Smash Ultimate, parse them and extract meaningful information. Of course the planned features go well beyond this, but we're going to architecture our solution around this core problem. The tricky part is we want to do this with a live stream of the game, coming through a capture card. We'll make the program generic enough that it can take in VODs too, but we'll test against a live stream as that's one of the most likely usecases. To do this, a few tools immediately spring to mind:
- OpenCV: Probably the most well known computer vision library. We're going to use the python bindings.
- OBS: To stream the game.
- ClonerAlliance 4KP: The best linux compatible capture card.
OBS and the Capture Card
Setting this up is extremely easy, just plug your Switch's HDMI into the
port of the capture card then attach the included HDMI cable into the
port and your monitor. Finally plug in the USB cable into the C port on the
device and any USB3.0 port on your computer. Connecting to OBS should be easy,
but we have a major problem. We want OBS to read the stream from the switch
Inkling reads it at the same time. Capture devices can only be opened by
one process at a time so the simple setup (adding a new Video4Linux2 source to
OBS) is out of the question.
Getting around the one process limit is not trivial. In order to do so, we're
going to have to rebroadcast the capture card's stream, then listen to the
stream with OBS and
Inkling. How can we do this? The incredibly versatile
ffmpeg of course! Following the streaming
guide on the wiki only gets you
so far. A careful reading shows we can connect to the card and read its stream
ffmpeg -f v4l2 -i /dev/video0. How do we get audio? First run
arecord -l and find your card in the list. Listen to it with
ffmpeg -f alsa -i hw:# replacing
# with the card number. We have ffmpeg listening to the
device, now we need to broadcast it somewhere!
The simplest way to do this is using UDP... problem is you can only connect with one device at a time when running on localhost. I tried to get it running on another IP or another interface but couldn't figure it out, so I sought other answers.
RTSP looked promising, I just needed to find a server!
rtsp-server is the only one I could
ffserver which has been removed in the latest ffmpeg and
deprecated for awhile in earlier versions. The perl server worked ok, although
install and running was confusing. Once I finally had it setup, quality was
terrible and the stream was incredibly delayed. This won't work either.
ffserver, I searched for alternatives. A
question provided me the quickest answer:
SRS. I couldn't figure out how to install
mkvserver so I moved on to SRS. Installing SRS was a breeze... just use the
docker run --rm -p 1935:1935 -p 1985:1985 -p 8080:8080 ossrs/srs:3 &
How: Finally Connecting
Once the SRS server is running, connecting with
ffmpeg is very simple.
ffmpeg -f alsa -thread_queue_size 128 -i hw:0 -f \ video4linux2 -i /dev/video0 -b:v 15M -qp 0 \ -framerate 60 -s hd720 -f flv -muxdelay 0.1 \ rtmp://127.0.0.1/live/livestream"
The first two lines of this command should look familiar, I've just merged the
audio and video listening commands from above. Next I set the bitrate to 15MB,
set quality to max with
-qp 0 then I set the framerate, output quality and
format. The final line is the URL to serve the stream at. SRS handles setting
this up but it can be customized. Once
you run this command we should be good to go!
Open up OBS, add a new source device under the "Media Source" section and set
rtmp://127.0.0.1/live/livestream. The stream should now be visible
More info on this is in the next part of the series, but if you want to test your connection you can run this script. A window with the stream should be displayed. Close it with Ctrl+C.
import cv2 cap = cv2.VideoCapture('rtmp://127.0.0.1/live/livestream',cv2.CAP_FFMPEG) while True: _, frame = cap.read() cv2.imshow('frame', frame) if cv2.waitKey(1) & 0xFF == ord('q'): break