{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenAMP examples\n",
    "1. [Introduction](#introduction)\n",
    "1. [Implementation details](#AstartR)\n",
    "    1. [Setup](#AstartR_setup)\n",
    "    1. [Load Kernel modules](#kernel_modules)\n",
    "    1. [Add firmware name](#firmware_name)\n",
    "1. [Demo: Echo Test](#echo_test)\n",
    "1. [Demo: Matrix Multiplication](#matrix_mult)\n",
    "1. [References](#References)\n",
    "\n",
    "## Introduction <a name=\"introduction\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"pics/ampLinuxBMrtos.png\" align=\"center\" alt=\"Drawing\" style=\"width: 500px; height: 400px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The latest Xilinx SoC products combine a set of heterogeneous hardware designs into one powerful and flexible platform that includes Arm Cortex-Ax, Cortex-R5 and Xilinx MicroBlaze processors. The OpenAMP project enables a distributed software architecture across this asymmetric multiprocessing platform (AMP)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Implementation details <a name=\"AstartR\"></a>\n",
    "This notebook example shows how the Cortex-A Application Processing Unit (APU) can launch an application on the Cortex-R Realtime Processing Unit (RPU). The APU subsystem running Linux is the designated master responsible for managing the life cycle of the RPU. The APU uses the remoteproc framework of OpenAMP to load, start, and stop the RPU application. RPU applications must be written in accordance with the OpenAMP application requirements. See Libmetal and OpenAMP User Guide (UG1186)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"pics/apu-start-rpu.png\" align=\"center\" alt=\"Drawing\" style=\"width: 800px; height: 200px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup <a name=\"AstartR_setup\"></a>\n",
    "We are using the PetaLinux pre-built images in these examples. For the pre-built kernel images, device tree binary or blobs (e.g.: system.dtb), root file system archives and other files, please find the `images` directory in your PetaLinux project. For example: `pre-built/linux/images`.\n",
    "\n",
    "Aside from the master OS (e.g.: Linux) support, the demos require two executables:\n",
    "1. RPU application or firmware is Cortex-R binary, used as an offloading server\n",
    "1. APU application is a Linux executable, e.g. Cortex-Ax binary - client\n",
    "\n",
    "| RPU binary in `/lib/firmware`   | APU client executable |\n",
    "| --- | --- |\n",
    "| `image_echo_test`       | `/usr/bin/echo_test`    |\n",
    "| `image_matrix_multiply` | `/usr/bin/mat_mul_demo` |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! date; uname -a; id; pwd; ls -l /lib/firmware/i* /usr/bin/{echo_test,mat_mul_demo,proxy_app}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Kernel modules <a name=\"kernel_modules\"></a>\n",
    "The demos use the remoteproc Linux kernel module to load firmware and start, stop the RPU. We also load the RPMsg module set, including VirtIO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! modprobe virtio_ring\n",
    "! modprobe zynqmp_r5_remoteproc\n",
    "! modprobe virtio_rpmsg_bus\n",
    "! modprobe rpmsg_char\n",
    "! modprobe virtio\n",
    "! lsmod | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add firmware name <a name=\"firmware_name\"></a>\n",
    "The sysfs filesystem is enabled in the pre-built kernel and it makes it easy to use the remoteproc driver from the Linux shell.\n",
    "1. Use the sysfs entry `/sys/class/remoteproc/remoteproc0/state` to stop RPU if it was running.\n",
    "1. Add the name of the RPU firmware via `/sys/class/remoteproc/remoteproc0/firmware`. Use only the <b>basename</b> of the firmware from the `/lib/firmware` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! grep .      /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'\n",
    "! [[ \"offline\" != $(</sys/class/remoteproc/remoteproc0/state) ]] && echo stop > /sys/class/remoteproc/remoteproc0/state\n",
    "! echo image_echo_test >/sys/class/remoteproc/remoteproc0/firmware\n",
    "! echo === before and after ===; grep . /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "***\n",
    "## Demo: Echo Test <a name=\"echo_test\"></a>\n",
    "At this point the APU is the master running Linux and the RPU is the remote in standby or powered down state. To start executing the firmware recorded in `/sys/class/remoteproc/remoteproc0/firmware` we write the word `start` to `/sys/class/remoteproc/remoteproc0/state`. This triggers the following sequence:\n",
    "1. The Linux kernel on the master loads the RPU's firmware into memory based on configuration in the firmware.\n",
    "1. The master starts the RPU and waits for it to initialize.\n",
    "1. The master is notified when initialization is complete and the RPU is running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! grep .       /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'\n",
    "! echo start > /sys/class/remoteproc/remoteproc0/state\n",
    "! echo === before and after ===; grep . /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'; /bin/dmesg | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The `dmesg` output should have a line indicating that the RPU is running, e.g.:\n",
    "`remote processor r5@0 is now up`\n",
    "\n",
    "After the previous step the RPU is running the echo server /lib/firmware/image_echo_test. The echo client is a Linux Cortex-Ax binary /usr/bin/echo_test. It sends a number of payloads from the APU master to the remote RPU and verifies that they match the replies from the echo server on the RPU. Both the client and the server use the Linux kernel RPMsg module to send and receive data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! /usr/bin/echo_test > /tmp/echo_test.out\n",
    "! head /tmp/echo_test.out; echo === skipping; tail /tmp/echo_test.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Demo: Matrix Multiplication <a name=\"matrix_mult\"></a>\n",
    "In this demo the remote on the RPU runs a simple matrix multiplication server. The algorithm is a direct implementation of the matrix multiplication definition which has $\\Theta \\left( n^{3} \\right)$ complexity to multiply $n \\times n$ matrices.\n",
    "\n",
    "The RPU setup steps:\n",
    "1. Stop the RPU if it was running by writing `stop` to `/sys/class/remoteproc/remoteproc0/state`\n",
    "1. Tell the remoteproc the name of the RPU firmware, i.e. `image_matrix_multiply`\n",
    "1. Start the RPU using remoteproc via sysfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! grep .      /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'\n",
    "! [[ \"offline\" != $(</sys/class/remoteproc/remoteproc0/state) ]] && echo stop > /sys/class/remoteproc/remoteproc0/state\n",
    "! echo image_matrix_multiply >/sys/class/remoteproc/remoteproc0/firmware\n",
    "! echo start                 >/sys/class/remoteproc/remoteproc0/state\n",
    "! echo === before and after ===; grep . /sys/class/remoteproc/remoteproc0/{state,firmware} |tr : '\\t'; /bin/dmesg | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Run matrix multiplication client\n",
    "After the previous step the RPU is running the matrix multiplication server `/lib/firmware/image_matrix_multiply`. The client is a Linux Cortex-Ax binary `/usr/bin/mat_mul_demo`. It generates two matrices and sends them to the RPU. The server on the RPU calculates the results and sends it back to the APU client. The client prints the result to its `sdtout`. Both the client and the server use Linux kernel RPMsg module to send and receive data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "! /usr/bin/mat_mul_demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## References  <a name=\"References\"></a>\n",
    "1. [Xilinx OpenAMP Wiki](https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841718/OpenAMP)\n",
    "1. [Xilinx User Guide: UG1186 - Libmetal and OpenAMP](http://www.xilinx.com/support/documentation-navigation/documentation-keyword-search.html?searchKeywords=OpenAMP%20UG1186)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}