For iteration 1 refer Edge Inference of Image Super Resolution Deep Learning Models
- Vitis 2020.2 (includes Vivado and Vivado HLS)
- Vitis AI v1.3
- Petalinux 2020.2
- Docker
- Development Environment with RAM: 32GB Disk Space: 500GB or higher
- Nvidia GPU with compute capability > 6.0 with sufficient memory (>4GB) - GeForce 1050 Ti is used in this case
- HD and UHD Monitors ( Acer KA220HQ is used) and BARCO ( for displaying results)
- Zynq Ultrascale board ( ZCU102 is used)
- Ethernet Cables for connectivity between systems (can be used to display on host system if there is no monitor available)
- Knowledge on FPGA and Integrating IPs in Vivado
- Knowledge on Computer Architecture and Embedded Systems
- Linux Operating system - Petalinux/Yocto
- Experience in building Deep learning models
- Programming Languages: Cpp - 17 and Python - 3.8+
- Frameworks/libraries: Opencv,Numpy,Tensorflow and Pytorch
This documentation describes the state of the repo and is divided into four sections
- Training
- Quantization
- Hardware Development
- Application Development
For How-To's and project flows and other features follow Vitis AI User Documentation and Vitis-AI Github
- Version v1.4
- Trained in house to achievce PSNR: 28.33 and SSIM: 0.9296 wandb
- SRHW.ipynb - Training Code
- pytorchtools.py - Used for Early stopping. Credits: Bjarte
- Reduce the number of model layer and use strategies like knowledge distillation /GAN based methods to imporve accuracy
- Split Images into patches and Super Resolve Patches, use multithreading to make processes parallel at software level. This imporves latency
- Version 1.15
- Trained In-house and achieved PSNR:26.476 and SSIM:0.8882. wandb
- Trained as an alternative for Pytorch in Vitis-AI version 1.2 since pytorch compilation did not support DPU for Edge
- SRHW_Tensorflow.ipynb - Training code
- local wandb files with best model. Access model (tf_ckpt/model_best.h5) and wandb from above link
- Will be deprecated for future releases and hence becomes stale
- Note: Checkpoint1_1.pt is Pytorch v1.1 version of checkpoint.pt - used in Vitis AI v1.2
- Calibration Dataset - In this directory, create a new directory calib and save it under the directory
- Vitis-AI version v1.3
- Developed a quantization script which is backward compatible with v1.2
- xir based Quanitization and Compilation which supports only VART API
- torch_quantize.py - quantizer script
- models.py - module containing models for Quantizer
- datasets.py - module containing dataset loader for use in Quantizer script
- quantize_result- output files of torch_quantize.py
- compilation_result - output
- Works only for SRHW now ; need to update to support any model and any input/output resolution of interest
- use Vitis-AI optimiser to reduce latency
- Vitis-AI version v1.2
- Used as an alterative for pytorch as pytorch compilation didnot support DPU for Edge in vitis-ai 1.2
- used for models converted from torch and also models developed ground up in tensorflow
- supports n2cube api only
- keras2tf.py - converts keras (.h5) model to tf checkpoints
- tf_calib.py - image loader function for quantizer
- tf_freeze_graph.py - freezes tf ckpts and produces .pb model file
- torch2tf.py - converts pytorch .pt model to tensorflow freezed graph - produced consistent model but could not be quantized due to Memory limitations
- tf_ckpt- .h5 model
- tf_freeze - tf_checkpoints - output of keras2tf and output of tf_freeze_graph.py
- tf_quantization_result - outputs of quantization
- tf_compilation output - outputs of compilation
- conversion output - outputs of torch2tf.py
- will be deprecated for fututre versions and hence becomes stale; also the metrics are poor compared to pytorch
- Developed Hardware Platofrms using Vitis for HDMI and DP media pipeline
- HDMI: used HDMI FRAMEBUFFER Design as the base, developed a Vitis Platform with it and integrated DPU with Custom Configuration. Visit here for more details . Status: Bringup Complete, Pipeline set - 10 min blankout due to x11 config
- DP: Developed a Vitis Platform using Zynq MPSoC base platform and created custom image to support DRM. Status: Bringup Complete, Pipeline setup failed
- Resorted to xilinx stock image for DPU Acceleration
- Platform and petalinux project files are not included as they take up more space . Follow DPU-TRD and Vitis Custom Embedded Platform Creation Example on ZCU104 for more details
- To create HDMI media pipeline in 2020.2 version and Integrate DPU
- Supports models compiled using tensorflow 1.15 compiler
- Appplcation reads an image, DPU processes it and saves it to the disk
- Supports v1.3 N2CUBE API
- srhw_dpu.cpp - application
- srhw - binary
- build.sh - compilation script
- most likely to become stale as future versions tend to support vart api
- can be used if version 2019.2 to 2020.2 is used
- As it gives greater flexibility at hardware level - this api can be used to optimize the design
- Video read and multi-DPU support
- supports models compiled using xir - pytorch and tf2.0+
- Appilcation uses multithreading to read, process and display video frames
- backward compatible with vitis-ai 1.2 and will support future versions
- srhw_vart.cpp - application
- srhw_vart - binary
- build_vart.sh - compilation script
- As x11 donot offer flexibility, DRM pipeline needs to be created
- Create a generalised application and convert application to library for imporving modularity so that other applications can invoke the feature
- switching between pass thorugh and SR feature
- coordinate dataflows with optimised data structures
- improve code readability
- modify build script to compile fiels given as parameters
- test applications to read and display video input
- test.cpp - v4l2 capture applcation Credits: Ananth
- test_naive.cpp -naive implemetation of video file capture and display
- test_opencv.cpp - multithreaded implemetation of video file capture and display
- build.sh - compilation script
- Modify to read both images and video files
- forked DRM display functions from xilinx
- drmhdmi.hpp - drm display functions
- invoke functions in applications for displaying via DRM Pipelines
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent