alisalevin/object_detection_aid

Object Detection Aid (ODA)

ODA: How it works

This is an object detection aid for the visually impaired that consists of a white cane enhanced with a microcomputer and camera and an iOS application. As you use this white cane throughout your daily routine, you can use its object detection capabilities to receive helpful information about your environment! Specifically, the cane’s camera and microcomputer detect objects in front of you via machine learning, and then send that information via Bluetooth to your smartphone, where it can be relayed to you via audio feedback. See demo.

Initial Setup

1. Download the ODA app on your smartphone device.
2. Make sure that your Bluetooth is turned on.
3. Turn the Nano on by plugging the cord into the Nano’s battery, and accept its request to pair with your smartphone.
4. Restart the Nano.

Everyday Use

1. Make sure your Bluetooth is turned on, open the app, and turn on the Nano by plugging the cord into the Nano’s battery. The smartphone and Nano will automatically pair.
2. Once the Nano is fully set up (this should take about 1 minute), it will begin sending detected object information to the smartphone.
3. To receive audio feedback on detected objects, swipe right anywhere on your screen. To turn off audio feedback, swipe left anywhere on your screen.
4. To stop the Bluetooth process entirely, quit the application or turn off the Nano by unplugging the cord from the battery.

Charging the Nano

Remove the battery from the white cane and charge it with its micro-USB. When fully charged, reattach it to its velcro patch on the cane.

Hardware Dependencies

1. NVIDIA Jetson Nano Developer Kit
2. Raspberry Pi V2 Camera
3. Portable Battery (This is the one we used)
4. Bluetooth module for the Nano

Software Dependencies

1. iOS Swift

2. XCode

3. Tensor Flow Object Detection API

   The network we're using is ssd-inception-v2 which has 91 object classes

4. BlueZ

5. DBus

Usage

Manual Compilation

Jetson Nano

Once the Jetson Nano has been set up, open the terminal and clone this repo.

git clone https://github.com/rahul-mitra13/Object_Detection_Aid

Change directory to the gatt-server directory.

cd /Object_Detection_Aid/gatt-server

Run gatt-server.py.

python gatt-server.py

iOS Smartphone

Clone this repo as above. Under the ObjectDetectionAid directory, open ObjectDetectionAid.xcodeproj with XCode. Turn smartphone's bluetooth on and build this project.

Compilation on Startup

Jetson Nano

Once the jetson nano is set up, open .bashrc.

vim .bashrc

Add the following line at the end of the .bashrc.

python ~/Object_Detection_Aid/gatt-server/gatt-server.py

Add gnome-terminal to Nano's startup applications.

iOS Smartphone

Same as the case for manual compilation.