{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Video Codec Unit (VCU) Demo Example: CAMERA->ENCODE->DECODE ->DISPLAY " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction\n", "\n", "Video Codec Unit (VCU) in ZynqMP SOC is capable of encoding and decoding AVC/HEVC simultaneously in real time. \n", "\n", "This notebook example captures raw video and audio(optional) data from USB Camera connected to board, does encoding and decoding to AVC/HEVC using VCU, does encoding and decoding of an audio using software Gstreamer element and renders output on DP/HDMI Display. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Implementation Details\n", "\n", "\"Drawing\"\n", "\n", "### Board Setup \n", "1. Connect the 4k display with board using DP/HDMI.\n", "2. Connect Ethernet cable.\n", "3. Connect serial cable to monitor logs on serial console.\n", "4. Connect USB camera(preferably Logitech HD camera, C920) with board.\n", "5. If Board is connected to private network, then export proxy settings in /home/root/.bashrc file as below, \n", " - create/open a bashrc file using \"vi ~/.bashrc\" \n", " - Insert below line to bashrc file\n", " - export http_proxy=\"< private network proxy address >\"\n", " - export https_proxy=\"< private network proxy address >\"\n", " - Save and close bashrc file.\n", "6. Determine Audio input output device names based on requirements. Please refer Determine AUDIO Device Names section.\n", "\n", "\n", "\n", "### Determine Audio Device Names\n", "The audio device name of audio source(Input device) and playback device(output device) need to be determined using arecord and aplay utilities installed on platform.\n", "\n", "***Audio Input***\n", "\n", "**ALSA sound device names for capture devices**\n", "- Run below command to get ALSA sound device names for capture devices\n", "\n", " root@zcu106-zynqmp:~#arecord -l\n", " \n", " It shows list of Audio Capture Hardware Devices. For e.g\n", " - card 1: C920 [HD Pro Webcam C920], device 0: USB Audio [USB Audio]\n", " - Subdevices: 1/1\n", " - Subdevice #0: subdevice #0\n", "\n", "Here card number of capture device is 1 and device id is 0. Hence \" **hw:1,0** \" to be passed as auido input device.\n", "\n", "**Pulse sound device names for capture devices**\n", "- Run below command to get PULSE sound device names for capture devices\n", "\n", " root@zcu106-zynqmp:~#pactl list short sources\n", " \n", " It shows list of Audio Capture Hardware Devices. For e.g\n", " - 0 alsa_input.usb-046d_HD_Pro_Webcam_C920_758B5BFF-02.analog-stereo ...\n", " \n", "Here \"**alsa_input.usb-046d_HD_Pro_Webcam_C920_758B5BFF-02.analog-stereo**\" is the name of audio capture device. Hence it can be passed as auido input device.\n", "\n", "***Audio Output***\n", "\n", "**ALSA sound device names for playback devices:**\n", "- Run below command to get ALSA playback device names for output devices\n", " \n", " root@zcu106-zynqmp:~#aplay -l\n", " \n", " It shows list of Playback Hardware Devices. For e.g\n", " - card 0: monitor [DisplayPort monitor], device 0: (null) xilinx-dp-snd-codec-dai-0 []\n", " - Subdevices: 1/1\n", " - Subdevice #0: subdevice #0\n", " - card 0: monitor [DisplayPort monitor], device 1: (null) xilinx-dp-snd-codec-dai-1 []\n", " - Subdevices: 1/1\n", " - Subdevice #0: subdevice #0\n", "\n", "Here card number \"0\" is being used for playback device for display port channel 0 and device id is 0, so \"**hw:0,0**\" Hence it can be passed as auido output device.\n", "\n", "**PulseAudio sound device names playback devices**\n", "- Run below command to get PULSE playback device names for output devices\n", "\n", " root@zcu106-zynqmp:~#pactl list short sinks\n", " \n", " It shows list of Playback Hardware Devices. For e.g\n", " - 0 alsa_output.platform-fd4a0000.zynqmp-display_zynqmp_dp_snd_card.analog-stereo ...\n", " \n", "Here \"**alsa_output.platform-fd4a0000.zynqmp-display_zynqmp_dp_snd_card.analog-stereo**\" is the name of audio playback device. Hence it can be passed as auido output device.\n", "\n", "\n", "\n", "### USB Camera Capabilities\n", "Resolutions for this example need to set based on USB Camera Capabilities\n", "- Capabilities can be found by executing below command on board\n", "\n", " root@zcu106-zynqmp:~#\"v4l2-ctl -d < dev-id > --list-formats-ext\".\n", " \n", " < dev-id >:- It can be found using dmesg logs. Mostly it would be like \"/dev/video0\"\n", " \n", " \n", "- V4lutils if not installed in the pre-built image, need to install using dnf or rebuild petalinux image including v4lutils" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.display import HTML\n", "\n", "HTML('''\n", "
''')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Run the Demo" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from ipywidgets import interact\n", "import ipywidgets as widgets\n", "from common import common_vcu_demo_camera_encode_decode_display\n", "import os\n", "from ipywidgets import HBox, VBox, Text, Layout" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Video" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c6ea4699ea594433b7d8e785e39bdbe2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Text(value='', description='Camera Dev Id:', placeholder='\"/dev/video1\"', style=DescriptionStyle(description_w…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "video_capture_device=widgets.Text(value='',\n", " placeholder='\"/dev/video1\"',\n", " description='Camera Dev Id:',\n", " style={'description_width': 'initial'},\n", " #layout=Layout(width='35%', height='30px'), \n", " disabled=False)\n", "video_capture_device" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3387a81bdaed4b4fbe14b076b3e9f438", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(RadioButtons(description='Codec Type:', options=('avc', 'hevc'), value='avc'), RadioButtons(des…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "codec_type=widgets.RadioButtons(\n", " options=['avc', 'hevc'],\n", " description='Codec Type:',\n", " disabled=False)\n", "video_size=widgets.RadioButtons(\n", " options=['640x480', '1280x720', '1920x1080', '3840x2160'],\n", " description='Resolution:',\n", " description_tooltip='To select the values, please refer USB Camera Capabilities section',\n", " disabled=False)\n", "HBox([codec_type, video_size])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ac5ebcea65f643e992a7961334efc302", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(interactive(children=(RadioButtons(description='Video Sink:', options=('kmssink', 'fakevideosin…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "video_sink={'kmssink':['DP', 'HDMI'], 'fakevideosink':['none']}\n", "\n", "def print_video_sink(VideoSink):\n", " pass\n", "\n", "def select_video_sink(VideoCodec):\n", " display_type.options = video_sink[VideoCodec]\n", "\n", "sink_name = widgets.RadioButtons(options=sorted(video_sink.keys(), key=lambda k: len(video_sink[k]), reverse=True), description='Video Sink:')\n", "\n", "init = sink_name.value\n", "\n", "display_type = widgets.RadioButtons(options=video_sink[init], description='Display:')\n", "j = widgets.interactive(print_video_sink, VideoSink=display_type)\n", "i = widgets.interactive(select_video_sink, VideoCodec=sink_name)\n", "\n", "HBox([i, j])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Audio" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ddda649665a44c6789fa03fdf9f8a9d1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(Text(value='', description='Input Dev:', description_tooltip='To select the values, please refe…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "device_id=Text(value='',\n", " placeholder='(optional) \"hw:1\"',\n", " description='Input Dev:',\n", " description_tooltip='To select the values, please refer Determine Audio Device Names section',\n", " disabled=False)\n", "audio_output=Text(value='',\n", " placeholder='(optional) \"hw:0\"',\n", " description='Output Dev:',\n", " style={'flex container': 'cross-end'},\n", " description_tooltip='To select the values, please refer Determine Audio Device Names section',\n", " disabled=False)\n", "display(HBox([device_id, audio_output]))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2ae085fcdfc34f56bb1658c7d0dfb4e4", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(interactive(children=(RadioButtons(description='Audio Codec:', options=('none', 'aac', 'vorbis'…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "audio_sink={'none':['none'], 'aac':['auto','alsasink','pulsesink'],'vorbis':['auto','alsasink','pulsesink']}\n", "audio_src={'none':['none'], 'aac':['auto','alsasrc','pulsesrc'],'vorbis':['auto','alsasrc','pulsesrc']}\n", "\n", "def print_audio_sink(AudioSink):\n", " pass\n", " \n", "def print_audio_src(AudioSrc):\n", " pass\n", "\n", "def select_audio_sink(AudioCodec):\n", " audio_sinkW.options = audio_sink[AudioCodec]\n", " audio_srcW.options = audio_src[AudioCodec]\n", "\n", "audio_codecW = widgets.RadioButtons(options=sorted(audio_sink.keys(), key=lambda k: len(audio_sink[k])), description='Audio Codec:')\n", "\n", "init = audio_codecW.value\n", "\n", "audio_sinkW = widgets.RadioButtons(options=audio_sink[init], description='Audio Sink:')\n", "audio_srcW = widgets.RadioButtons(options=audio_src[init], description='Audio Src:')\n", "j = widgets.interactive(print_audio_sink, AudioSink=audio_sinkW)\n", "k = widgets.interactive(print_audio_src, AudioSrc=audio_srcW)\n", "i = widgets.interactive(select_audio_sink, AudioCodec=audio_codecW)\n", "\n", "HBox([i, k, j])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Advanced options:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "aba5470d2c3c443e80ce5e0f7c82df4a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(Text(value='', description='Bit Rate(Kbps):', placeholder='(optional) 1000, 20000', style=Descr…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "frame_rate=widgets.Text(value='',\n", " placeholder='(optional) 15, 30, 60',\n", " description='Frame Rate:',\n", " disabled=False)\n", "bit_rate=widgets.Text(value='',\n", " placeholder='(optional) 1000, 20000',\n", " description='Bit Rate(Kbps):',\n", " style={'description_width': 'initial'},\n", " disabled=False)\n", "no_of_frames=Text(value='',\n", " placeholder='(optional) 1000, 2000',\n", " description=r'

Frame Nos:

',\n", " #layout=Layout(width='33%', height='30px'),\n", " disabled=False)\n", "display(HBox([bit_rate, frame_rate, no_of_frames]))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0a6479e32955479c8f04bc01176bc93c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(Dropdown(description='Entropy Buffers Nos:', index=3, options=('2', '3', '4', '5', '6', '7', '8…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "entropy_buffers=widgets.Dropdown(\n", " options=['2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15'],\n", " value='5',\n", " description='Entropy Buffers Nos:',\n", " style={'description_width': 'initial'},\n", " disabled=False,)\n", "show_fps=widgets.Checkbox(\n", " value=False,\n", " description='show-fps',\n", " disabled=False)\n", "HBox([entropy_buffers, show_fps])" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from IPython.display import clear_output\n", "from IPython.display import Javascript\n", "\n", "def run_all(ev):\n", " display(Javascript('IPython.notebook.execute_cells_below()'))\n", "\n", "def clear_op(event):\n", " clear_output(wait=True)\n", " return\n", "\n", "button1 = widgets.Button(\n", " description='Clear Output',\n", " style= {'button_color':'lightgreen'},\n", " #style= {'button_color':'lightgreen', 'description_width': 'initial'},\n", " layout={'width': '300px'}\n", ")\n", "button2 = widgets.Button(\n", " description='',\n", " style= {'button_color':'white'},\n", " #style= {'button_color':'lightgreen', 'description_width': 'initial'},\n", " layout={'width': '38px'}\n", ")\n", "button1.on_click(run_all)\n", "button1.on_click(clear_op)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "36e64d33e0e94d71a854d30c6428efb3", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(Button(description='click to start vcu-camera-encode-decode-display demo', layout=Layout(width=…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def start_demo(event):\n", " #clear_output(wait=True)\n", " arg = [];\n", " arg = common_vcu_demo_camera_encode_decode_display.cmd_line_args_generator(device_id.value, video_capture_device.value, video_size.value, codec_type.value, audio_codecW.value, display_type.value, frame_rate.value, sink_name.value, no_of_frames.value, bit_rate.value, entropy_buffers.value, show_fps.value, audio_srcW.value, audio_output.value, audio_sinkW.value);\n", " #!sh vcu-demo-camera-encode-decode-display.sh $arg > logs.txt 2>&1\n", " !sh vcu-demo-camera-encode-decode-display.sh $arg\n", " return\n", "\n", "button = widgets.Button(\n", " description='click to start vcu-camera-encode-decode-display demo',\n", " style= {'button_color':'lightgreen'},\n", " #style= {'button_color':'lightgreen', 'description_width': 'initial'},\n", " layout={'width': '350px'}\n", ")\n", "button.on_click(start_demo)\n", "HBox([button, button2, button1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# References\n", "[1] https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842546/Xilinx+Video+Codec+Unit\n", "\n", "[2] https://www.xilinx.com/support.html#documentation (Refer to PG252)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.6" } }, "nbformat": 4, "nbformat_minor": 2 }