What: Our objective

The basic purpose of this project is to read in a video feed / frames of Smash Ultimate, parse them and extract meaningful information. Of course the planned features go well beyond this, but we're going to architecture our solution around this core problem. The tricky part is we want to do this with a live stream of the game, coming through a capture card. We'll make the program generic enough that it can take in VODs too, but we'll test against a live stream as that's one of the most likely usecases. To do this, a few tools immediately spring to mind:

  • OpenCV: Probably the most well known computer vision library. We're going to use the python bindings.
  • OBS: To stream the game.
  • ClonerAlliance 4KP: The best linux compatible capture card.

OBS and the Capture Card

Setting this up is extremely easy, just plug your Switch's HDMI into the input port of the capture card then attach the included HDMI cable into the output port and your monitor. Finally plug in the USB cable into the C port on the device and any USB3.0 port on your computer. Connecting to OBS should be easy, but we have a major problem. We want OBS to read the stream from the switch while Inkling reads it at the same time. Capture devices can only be opened by one process at a time so the simple setup (adding a new Video4Linux2 source to OBS) is out of the question.

Why: Rebroadcasting

Getting around the one process limit is not trivial. In order to do so, we're going to have to rebroadcast the capture card's stream, then listen to the stream with OBS and Inkling. How can we do this? The incredibly versatile ffmpeg of course! Following the streaming guide on the wiki only gets you so far. A careful reading shows we can connect to the card and read its stream by using ffmpeg -f v4l2 -i /dev/video0. How do we get audio? First run arecord -l and find your card in the list. Listen to it with ffmpeg -f alsa -i hw:# replacing # with the card number. We have ffmpeg listening to the device, now we need to broadcast it somewhere!

UDP

The simplest way to do this is using UDP... problem is you can only connect with one device at a time when running on localhost. I tried to get it running on another IP or another interface but couldn't figure it out, so I sought other answers.

RTSP

RTSP looked promising, I just needed to find a server! rtsp-server is the only one I could find besides ffserver which has been removed in the latest ffmpeg and deprecated for awhile in earlier versions. The perl server worked ok, although install and running was confusing. Once I finally had it setup, quality was terrible and the stream was incredibly delayed. This won't work either.

RTMP

After discovering ffserver, I searched for alternatives. A stackoverflow question provided me the quickest answer: mkvserver_mk2 and SRS. I couldn't figure out how to install mkvserver so I moved on to SRS. Installing SRS was a breeze... just use the docker container!

docker run --rm -p 1935:1935 -p 1985:1985 -p 8080:8080 ossrs/srs:3 &

How: Finally Connecting

Once the SRS server is running, connecting with ffmpeg is very simple.

ffmpeg -f alsa -thread_queue_size 128 -i hw:0 -f \
 video4linux2 -i /dev/video0 -b:v 15M -qp 0 \
 -framerate 60 -s hd720 -f flv -muxdelay 0.1 \
 rtmp://127.0.0.1/live/livestream"

The first two lines of this command should look familiar, I've just merged the audio and video listening commands from above. Next I set the bitrate to 15MB, set quality to max with -qp 0 then I set the framerate, output quality and format. The final line is the URL to serve the stream at. SRS handles setting this up but it can be customized. Once you run this command we should be good to go!

OBS

Open up OBS, add a new source device under the "Media Source" section and set the URL: rtmp://127.0.0.1/live/livestream. The stream should now be visible

OpenCV

More info on this is in the next part of the series, but if you want to test your connection you can run this script. A window with the stream should be displayed. Close it with Ctrl+C.

import cv2

cap = cv2.VideoCapture('rtmp://127.0.0.1/live/livestream',cv2.CAP_FFMPEG)
while True:
    _, frame = cap.read()
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break