eos21/acoustic_camera

An app for a real time acoustic localization device coupled with a camera for an audio/visual overlay

Acoustic Camera System

• Real Time Passive Acoustic Phase Array with Visual Display

Overview

FPGA

• 4x4 microphone array driven synchronously by an Altera Cyclone V FPGA
• SoC running linux with PhaseArray command line interface (CLI)
• The array has 4x2 PCBs with two microphones per PCB operating as a stereo pair
• 2D delay and sum beamforming algorithm implemented on the same FPGA fabric
• Beamforms for 22.5° DoA steps over a 90° FoV per-direction in real-time (5x5 output)

PapaPi

• Raspberry Pi running busterOS
• RaspPi Cam attached
• Device might be changed for better processing power: camera, acoustic overlay, DL implementation

AcousticPi

• Raspberry Pi running busterOS
• Runs AcousticCam App

Network Info

• PapaPi - Access Point for Network | IP: 10.0.0.1
• AcousticPi - IP: 10.0.0.13
• FPGA - Ethernet over USB | IP: 192.168.80.1

Acoustic Camera

• How the info from the mics are combined with the camera feed

Acoustic Camera Application

• The app that gives user ways to interact with the possible functionality
• Gets acoustic overlaid video from PapaPi for display

Libraries

• OpenCV (cv2)
• Numpy
• UDP Socket
• CustomTKinter

Hardware

Links:

• Raspberry Pi 4 Purchase Page
• SD Card Purchase Page
• Screen Purchase Page
• Keyboard/Mouse Purchase Page
• Power Supply Purchase Page
• Pi Camera V2

Connect Pi Hardware

• Follow screen instructions for proper connection

Raspberry Pi Setup

• RaspPi Set Up Guide

Flash SD Card

• Matrix Voice only works with Buster OS
• Link to img zip file Link
• Used Raspberry Pi imager to image the SD card with this .img file
• Settings
  • set username / password
  • turn ssh on
  • enter WIFI credentials (optional: if using screen, you can do this later)

Set Up

• Go through on screen instructions
• Change Interface Settings
  • Home Icon / Preferences / Raspberry Pi Configuration
    • Enable everything
• Reboot as needed

SSH into Pi

• Open terminal on pi (ctr + alt + t)

# Need IP4 address
ifconfig
# Look for IP4 192.168.0.111 or inet 192.168.0.111
#If you set up a hostname, you can use it instead of the IP address

# Need root user name (should be green): pi is default
pi@raspberrypi:~ $
# Type this command with your info replaced: 
ssh -X [email protected]
# if using hostname
# ssh -X [email protected]
# ssh-keygen -f "/Users/KevMcK/.ssh/known_hosts" -R "192.168.0.147"
# -X is to show display on remote screen

# Update everything
sudo apt update
sudo apt full-upgrade
sudo apt autoremove
sudo apt clean
sudo reboot

Project Directory

git clone [email protected]:kevmck451/acoustic_camera.git

Virtual Environment & Install Libraries

# cd into folder
cd acoustic_camera/
# create virtual env with system packages
python3 -m venv --system-site-packages camera_venv
# activate environ
source camera_venv/bin/activate
# add it to your gitignore
nano .gitignore # add camera_venv/ 
# install libraries
pip install -r requirement.txt

python -c "import picamera"

Run Application

# If starting app through SSH, set remote display for app
export DISPLAY=:0
# Inside the acoustic_camera folder run:
python3 -m app.app_main

if window size is larger than screen, adjust the camera height and width

Add Desktop Icon

• create bash script to start VE and run program

nano ~/start_acoustic_camera.sh

• copy and paste into file

#!/bin/bash

# Change directory to the correct folder
cd /home/pi/Desktop/acoustic_camera

# Activate the virtual environment
source camera_venv/bin/activate

# Run the Python application
python3 -m app.app_main

• change script permissions to executable

chmod +x ~/start_acoustic_camera.sh

• create desktop app entry

nano ~/Desktop/AcousticCameraLauncher.desktop

• copy and paste into file

[Desktop Entry]
Version=1.0
Type=Application
Name=Acoustic Camera App
Comment=Start Acoustic Camera App
Exec=/home/pi/start_acoustic_camera.sh
Icon=/home/pi/Desktop/acoustic_camera/docs/papa_icon.png
Terminal=false
Categories=Utility;

• change script permissions to executable

chmod +x ~/Desktop/AcousticCameraLauncher.desktop

• When icon is first double clicked, hit the execute button on the far left