tomasz-lewicki/ai-thermometer

Fever screening with IR & RGB cameras and Deep CNNs

AI Thermometer

Contactless temperature mesurement using IR & RGB cameras and Deep CNN facial detection.

Normal temperature

Elevated temperature

What's new in v0.4

• New facial detector (handles partial occlusion, e.g. with a mask)
• Added IR camera drift compensation with an external blackbody

Hardware

To build this project you will need:

#	Part	link	Price (USD)
1	Jetson Nano Dev Kit	link	99
3	FLIR Lepton 3.5 IR Camera	link	199
4	GroupGets Purethermal2 Module	link	99
2	Raspberry Pi Camera Module V2.1	link	25
5	Noctua cooling fan	link	14
6	3D printed enclosure	https://a360.co/3nmDML3	-
total			436

Enclosure (live model available from https://a360.co/3nmDML3 ):

Quickstart

1. Flash Nvidia Jetson Nano with the latest Nvidia JetPack. Update package manager

sudo apt update && sudo apt upgrade

2. Build OpenCV >4.4. It is required for CUDA DNN support.

wget https://raw.githubusercontent.com/mdegans/nano_build_opencv/master/build_opencv.sh
chmod +x build_opencv.sh
./build_opencv.sh

3. Build pytorch (instructions here)

4. Download AI Thermometer

git clone --recursive https://github.com/tomek-l/ai-thermometer
cd ai-thermometer

5. Run AI Thermometer

python3 main.py

6. Optionally, 3D print & assemble the enclosure from https://a360.co/3nmDML3

Limitations

IR and RGB camera alignment.

The current way of calculating the correspondence between IR and RGB cameras is not ideal. Factors, such as the non-rigid mount of the sensor on the Raspberry Pi CMV2.1 don't help. I'm actively working on calibration code that takes into account the intrinsic parameters of both cameras, which should allow for obtaining a pixel-level correspondence between the imags.

FAQ/common issues:

1. uvc_open error -3

Reason: Your current user does not have r/w access to the PureThermal USB device.

For a quick and dirty fix you can do do:

echo 'SUBSYSTEM=="usb",  ENV{DEVTYPE}=="usb_device", GROUP="plugdev", MODE="0664"' | sudo tee /etc/udev/rules.d/10-libuvc.rules 
sudo udevadm trigger

(this gives camera access to all users)

For a better, fine-grained permission setup, create a new group, give permissions to that group only and your user to the newly created group. You can read more here.

2. Illegal instruction (core dumped)

Reason: The provided libuvc.so is compiled for AArch64. It is a custom version of libuvc that supports Y16 video format. If you're using different architecture, you will need to build the library from source:

git clone https://github.com/groupgets/libuvc
cd libuvc
mkdir build
cd build
cmake ..
make
cp libuvc.so ~/ai-thermometer/ir/libuvc_wrapper

3. using sudo or getting permission denied

You don't need sudo to run the code in this repository 🙂

Previous versions

Version 0.2: Haar Cascade + FLIR Lepton 3.5 temp. measurement (~10FPS, CPU only)

Watch full video: https://www.youtube.com/watch?v=j9eo9Cs8J8I

Version 0.1: SSD (COCO) + FLIR Lepton 3.5 temp. measurement (~12FPS w/ GPU accel.)

Watch full video: https://www.youtube.com/watch?v=i2XMtshdjn8

ai-thermometer's Issues

Hi there, how do I change both images like 180 degress?

Hi there, how do I change both images like 180 degrees?

Originally posted by @jjferreira in #39

Version

Hey,

Thanks for this project, I've had great fun working on this, an area I am really new too. A quick question which may be worth updating in the notes, which version is this? It appears to be v.01 with the simple box detection? Is that correct? I'm really interested in the eye detection model 👍

Many thanks,

David

Flip Images 180 Degress

Hi there, how do I change both images like 180 degress?

Add an option for calibirating readings with a blackbody reference

AI Thermometer should have an option for online calibration with an external blackbody reference for more accurate readings.
My general idea to implement this is as follows:

1. The callibration can be realized with a physical blackbody that is placed at a fixed position at the scene.
2. The blackbody is set to a known temperature close to human body ~37C/~98.6F and is used to calibrate the IR camera's drift.

I plan to add this in V0.5 of the software.
Any suggestions on how to realize that feature are welcome :)

-

Incorrect Temperature values

It always shows human body temperature around 32-34 degree Celsius but actual temperature is always around 36-37 degree. How can I get accurate values? Are you using any calibration of Lepton 3.5 camera?

Low Memory Warning

Hi Guys, I’m getting this Error after executing the Main.py.

I believe it’s a low Memory Error as I’m using Jetson Nano 2GB.

Can anyone advise, how I can Resolve this with out changing to 4GB Nano Kit.

Kind Regards,

(NOT AN ISSUE) Neopixel to indicate "fever" in a detection

I am trying to use Neopixels on this project,. Recently Adafruit coded some libraries for the Neopixels that are useful in the Jetson Nano using CircuitPython. It works pretty nice...

I want to make the Neopixel stay ON with green colour when the Jetson Nano be in fever screening mode and when a 37.5 C temperature is detected, the Neopixel changes the colour to red for example... I need some help to incorporate it to the Tomasz project please.

Thank you.

Here is a simple tutorial to make Neopixels work with the Jetson Nano:

Install Pip3 if not already installed:
sudo apt-get install python3-pip

Reboot

Install Adafruit-Blinka 5.8.0:
pip3 install Adafruit-Blinka

Enable SPI on Jetson Nano:
sudo /opt/nvidia/jetson-io/jetson-io.py

• Select Configure 40-pin expansion header at the bottom.

• Select spi1 (19, 21, 23, 24, 26) and then select Back

• Finally select Save and reboot to reconfigure pins. This will create a config file and reboot the Jetson Nano.

After the Nano boots up again, verify you have the I2C devices with the command:
ls /dev/i2c* /dev/spi*

You can test to see what I2C addresses are connected by running sudo i2cdetect -r -y 0 (on pins 27/28) or sudo i2cdetect -r -y 1 (on pins 3/5)

Then, we need to set user permissions. In order to use the Jetson GPIO Library, the correct user permissions/groups must be set first. Start by creating a new gpio user group:

sudo groupadd -f -r gpio
sudo usermod -a -G gpio

Clone the Repo and copy the rules:

cd ~
git clone https://github.com/NVIDIA/jetson-gpio.git
sudo cp ~/jetson-gpio/lib/python/Jetson/GPIO/99-gpio.rules /etc/udev/rules.d

This section was originally written for releases before JetPack 4.3. For JetPack 4.3, the udev rules are in: /lib/udev/rulesd/60-jetson-gpio-common.rules. This should mean that you don’t have to copy the file anymore with JetPack 4.3 and above.

Also, it appears as if the default is for users to be included in the gpio group. You can check this by:
$ groups

Reboot

Update just in case if a library is missing:
sudo pip3 install --upgrade adafruit_blinka

Install CircuitPython NeoPixel_SPI library;:

pip3 install adafruit-circuitpython-neopixel-spi

Reboot again and that´s all!!!

Connect "Din" of your Neopixel to the SPI1_MOSI pin of your Jetson Nano(pin 19 of the expansion header), Gnd to Gnd and 5v to 5v.

try this example:

import board
import neopixel_spi

pixels = neopixel_spi.NeoPixel_SPI(board.SPI(), 10)
pixels.fill(0xff0000)

If an error occurs
Check you have installed this dependencies:

Adafruit CircuitPython BusDevice
pip3 install adafruit-circuitpython-busdevice

Pypixelbuf
pip3 install adafruit-circuitpython-pypixelbuf

Install nano (just in case the nano editor is not installed)
sudo apt-get install nano

You can check the Hex colors to use whatever colour you want by using this web page:
https://www.color-hex.com/

Another thermal sensor

I think it’s better if we can use the InfiRay Tiny1-B Micro LWIR Thermal Module sensor instead of the FLIR LEPTON 3.5

The Tiny1-B sensor have a biological temperature measurement accuracy of ±0.5°C and the FLIR Lepton have a measurement accuracy of ±5°C

The sensor is cheap and we can avoid the use of a black box in order to constantly calibrate the sensor. It have radiometric capabilities.

In that way we can have a very useful instrument without the accuracy limitation of the FLIR sensors.

https://www.infiray.com/products/tiny1-b-micro-lwir-thermal-module.html

Regards

.STL or .STEP needed to print enclosure..how to export?

does anyone have a .STEP or .STL? I need to send a file to a 3rd party to print the enclosure.
thank you!

I want to ask

I use PI 4 and when it runs there is an error /home/pi/ai-thermometer/ir/libuvc_wrapper/libuvc.so: wrong ELF class: ELFCLASS64,how to fix? thx dude

Wrong viewing mode on the FLIR-Lepton

I copied your code and am running it on the developer kit. Unfortunately the FLIR Lepton shows the pictures only in red and grey like in your video with the red channel. How can i change the output of the FLIR Lepton to a range from blue to red like in your example?

where is requirements.txt

need requirements.txt

USB camera issue

I am using a simple logitech usb camera with jetson xavier but I am getting a uvc_find_error error but the camera is getting detected and activated. Everything is working well with simple codes.

Running with web cam and IR cam, not accurate temperature reading

Hi,

First of all, thanks for sharing the source code for this project, very helpful. I have been trying to run this code with a web cam instead of using the raspberry pi cam. I have it working with display and face detection, the only issue I am running into is that the temperature readings are not accurate. Would you have any idea what could be causing this behavior?

How to get started

Hi @tomek-l ! I hope you could help me.

I am a graduating college student. I have a thesis about face mask detection and body temperature detection.. I am currently working on my model for the mask detection, however I really do not know how to get started with the temperature detection. I also have the same idea which is to detect the inner eye canthus for temperature measurement, (using haar as well?). I am also planning to buy the same materials. But I really don't know how the code works or how I will calibrate the cameras. I am still a beginner in machine learning with python because I studied on my own. I hope you could help me, I really want to learn. This is the only project I found helpful. Thank you so much! 😅

where can i get thermal from ur file

i want to get a cv2.putext and thermal(degree) information ur file..
but i cannot find it
please help me ;)

Calibrating Lepton 3.5 for Outdoor vs Indoor

Has anyone run into calibration issues when using this outdoors - particularly in winter? The system works perfectly inside with ambient temperatures of 66-72F. However, trying to use it outdoors <45F produces really drastic readings on the facial detections - typically over 1000F!

I've tried leaving the system outside but am still having the same problems. I'm a track & field coach and am trying to set this up to screen athletes so I need it to work outside. Any suggestions would be greatly appreciated.

Thanks!

How to execute FFC

hi @tomek-l is there a way to execute a FFC?, currently executing the code but the weather is fairly warm and is not working i need to execute an FFC.

thank you

bash: ENV{DEVTYPE}=="usb_device,: command not found

Thank you again for all the help!!!

Can you explain me how you fix it please? When applying the "dirty solution" I get:

bash: ENV{DEVTYPE}=="usb_device,: command not found

What I am doing wrong?

Best regards!!!

Question about face detection

Hello, thank you very much for your codes.

Just want to know if the face detection is possible if a person is using a mask. Thank you in advance!

Webcam+Lepton3.5 using pure thermal mini

Hello @tomek-l. First of all, brilliant work on the code.

I am using webcam and lepton 3.5 through Purethermal mini(connected using USB) on windows platform. I am just trying this code in my laptop. so, i believe there wont be a use for gstreamer. I would like to know how to use this code on windows. I tried writing my own program, but the temperature readings are not accurate. In lepton 3.5/2.5 documentation, it has been mentioned that the temperature value will be stored only when the t-linear mode is enabled. In your code, how have you done that? Also please let me know how to use your code on windows. what are the changes that I need to make? please let me know.

Also I am unable to view the image using matplotlib.

Thanks.

Camera Issue

can you help me with this erroor

Torchvision issue fix

Make sure you put a "v" in front of your torchvision export.
export BUILD_VERSION=0.x.0 # where 0.x.0 is the torchvision version

E.g. export BUILD_VERSION=v0.9.0

UVC error

Hey Tomasz

Thanks for putting this out for people its a really cool project bro. I'm having trouble finding the video; I get the following error. Any idea trouble where I messed up? I'm thinking it may be how cameras are lined up. I 3d printed the case you gave but I noticed its slightly different than what you showed in the blog post. I attached a picture of how it look on my end was hoping if you thought that was maybe the problem. Also I 3d printed an Iron Man Helmet as a case I think you would appreciate. I attached that as well.

Thanks again,
Raza

/ai-thermometer$ python3 main.py
Loading weights from file...
Weights loaded!
Running first net inference...
Detector initialized!
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3264 H = 2464
seconds to Run = 0
Frame Rate = 21.000000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
[ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
uvc_open error -3

ai.thermometer on Jetson Nano 2Gb

Hello,

Do you know if the AI.thermometer can work in the new Jetson Nano 2Gb or I need to modify the code?

Thank you and best regards

uvc_open error {res}

I try running pt2_example.py But script gives uvc_open_error . I tried your answer for those who have similar problem . But that didnt work . Please can you help ?

How do you add Fahrenheit to your output?

How to work without hardware component

Hello
I want to work on the AI ​​Thermometer project and I want it to work on Ubuntu system, but I do not have any of the hardware components, and if it can run on Raspberry Pi 3, I am working on it for the graduation project
could you please help me

Couldn't get correct temperature

Hi, I'm using Lepton 3.5 and purethermal 2, but my result temperature is around 100 (it should be close to 37,right?), and when I use the notebook(https://github.com/tomek-l/ai-thermometer/blob/master/docs/notebooks/Getting%20Started%20with%20Cameras.ipynb), the raw data for each pixel is around 8000 and after Kelvin to Celsius the result is around -195.
Do you have any idea about this? Thanks.

Adding support for SPI lepton board: let me know what you think!

I'm leaning towards using SPI directly (rather than USB) for interfacing Lepton 3.5 in a future hardware revision.

• It would bring down the cost of the device (the GroupGets Purethermal board costs ~100USD).
• It would mitigate the need for bulky USB connectors inside the case.
• The transition should be relatively easy. The pylepton package could be used to interface lepton module.

If you're using AI Thermometer and have any thoughts on this change, feel free to comment below.

Camera connection?

thanks for sharing us your work.
I understood that you use NVidia Nano Dev Kit with 2 cameras: Visible and Lepton 3.5.
Could you please share me information which visible camera you use and how to connect all camera with Nano? via MIPI-CS2 or SPI?
Thanks

Syntax error

Hello, need some advice please.. why I am getting syntax error when executing Capture Images.ipynb example?

Thank you and Regards!!!

!gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=1 ! video/x-raw,format=GRAY16_LE ! videoconvert ! pngenc ! filesink location="$save_path"

Version 0.2: Haar Cascade + FLIR Lepton 3.5 temp. measurement (~10FPS, CPU only)

Hello, I am a student studying artificial intelligence. Thanks to the code you posted, I was able to learn a lot. Can you upload the code for "Version 0.2: Haar Cascade + FLIR Lepton 3.5 temp. measurement (~10FPS, CPU only)"? I think it will be very helpful for me who is currently studying Haar-based recognition code. I'd appreciate it if you could post it.

ok, where do you get opencv 4.4 for the jetson nano?

I have 4.1 from jetsonhacks.

Add support for multiple facial detectors

A lot of people are asking about the previous detectors (like Haar Cascades). It would be valuable if we could change different detectors in a plug-and-play fashion.

Where can I find the 'Cheat'

Hello, can you link the pre-made SD card Image?

Also what ports is the Adafruit Lightstick plugged/soldered too?

Error occur in ai-thermometer!! Please Guide me to resolve this problem

File "main.py", line 13, in
from rgb import RGBThread
File "/home/thermal/ai-thermometer/rgb/init.py", line 1, in
from .rgb_thread import *
File "/home/thermal/ai-thermometer/rgb/rgb_thread.py", line 8, in
from .retinaface import RetinaFaceDetector
File "/home/thermal/ai-thermometer/rgb/retinaface/init.py", line 1, in
from .detector import *
File "/home/thermal/ai-thermometer/rgb/retinaface/detector.py", line 10, in
from .models.retinaface import RetinaFace
File "/home/thermal/ai-thermometer/rgb/retinaface/models/retinaface.py", line 5, in
import torchvision.models.detection.backbone_utils as backbone_utils
ModuleNotFoundError: No module named 'torchvision.models.detection'

Storage on Jetson Nano Full

Hello,
Thanks for all the help so far, I have an issue whereby the storage on my Jetson Nano fills up after a few hours. I have no value in saving the frames. I only use this system for a live feed on a monitor.
Is there a way I can stop it from saving frames?
Thanks

Can't load an ONNX model in SSD Detector

I want to change the model used in SSD Detector with my custom trained model which is an onnx and it is an facemask detector. Is it possible to use my model to detect facemask and the temperature of the person? I have modified the detector.py in SSD folder into this to load the my model :
parent_dir_pth = os.path.dirname(os.path.abspath(file))
onnx_model_pth = (
parent_dir_pth + "/caffe/ssd-mobilenet.onnx"
)
self._net = cv2.dnn.readNetFromONNX(onnx_model_pth)

but I get this error

Loading SSD weights from file...
Traceback (most recent call last):
File "main.py", line 220, in
rgb_thread = RGBThread(model=FACE_DET_MODEL)
File "/home/g1-facemask/ai-thermometer/rgb/rgb_thread.py", line 18, in init
self._detector = SsdDetector()
File "/home/g1-facemask/ai-thermometer/rgb/ssd/detector.py", line 18, in init
self._net = cv2.dnn.readNetFromONNX(onnx_model_pth)
cv2.error: OpenCV(4.4.0) /tmp/build_opencv/opencv/modules/dnn/src/dnn.cpp:604: error: (-2:Unspecified error) Can't create layer "433" of type "Exp" in function 'getLayerInstance'

I am using OpenCV4.4.0 and latest version of pytorch
Is there a thing that I can do?
Should I retrain my model that gets caffemodel and prototxt like the default? If yes, is there any tutorial to do that kind of training? @tomek-l

Waiting for RGB frames

I'm getting a message Waiting for RGB frames any idea what I am missing?

Estimated / selected altsetting bandwith : 18 / 642.
INFO: Using $DISPLAY from environment, not from config
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/nano-termo/ai-thermometer/rgb/rgb_thread.py", line 38, in run
frame_small = cv2.resize(self._frame, (400, 300))
cv2.error: OpenCV(4.4.0) /tmp/build_opencv/opencv/modules/imgproc/src/resize.cpp:3929: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

[enhancement] Improve consistency of face detection when wearing a mask and on moving target

Hi,

Just wanted to propose and to see if there is anyway to improve consistency and accuracy of the face detection when people are wearing masks or when the target is moving. Maybe training the model with people wearing masks and not wearing masks.

Let me know what you think and if it is doable.

Nick

Program/camera feed over http

Hello, I've been wondering if it is possible to access the feed the program provides(thermal and normal camera, along with the temperature overlay) so i can stream it over http. For the streaming I usually use VLC's component. My main issue is that I cannot find a way to access the feed. Is it saved locally somewhere? Am I missing something? Thanks for your time

Interface for deployment

Hi,

Thanks so much for your work this is really interesting.

I was wondering what kind of interface have you used for actual deployment. I saw you were working on a web interface, does that work? For us to put on a screen near the camera display?

Would be nice to have like an audio alert which we could play maybe through the web interface if someone is detected above the desired range.

Also curious if you're using the flir 3.5 as well, do you need any calibration? My detection seems to be a little bit off

Thanks again

I am implementing ai thermometer with your kind explanation.

I am implementing ai thermometer with your kind explanation.

Now I am building the environment of Raspberry pi camera v2 & lepton2.5 & PureThermal2.

But nothing has happened since Uvc_open error -3 appeared after python3 main.py.

Can I know the solution?

Of course, the individual operation of each sensor was confirmed.

thank you.

Version 0.1: SSD (COCO) + FLIR Lepton 3.5 temp. measurement (~12FPS w/ GPU accel.)

hi @tomek-l

First of all, i wanna thank you for your codes. it really helps the communities at my place in this pandemic situation. however, these codes could only measure temperature with face straight to the camera and no mask used. i wonder if you have codes version 0.1 with SSD detection. I would appreciate it if you want to upload it.

Capture of pixel temperature with Lepton 3.5 camera

Hi Tomasz,
Thank you very much for your answer concerning the script you have developped for temperature capture with Lepton 3.5.

Not being experienced in software development, I send you below a few questions:

• How is captured the 16 bits flow of the Lepton camera ? Does the Python script command the camera (in line 47 to 67 or 218) ? Or do we have to send separate commands through the CCI channel ?

• Which data contains the libuvc and how is it fed ?

• Which command stops the flow ?

• Have you an example of the temperature array (both link in your mail points to the script)

• Which configuration is needed to run the script ? Is Lepton 3.5 with PureThermal2 connected via USB to a PC with windows 10 acceptable ?

• Which software do we have to load (Python, openCV...) ?

Sorry to take your time by basic questions, but this could help the project a lot.

White box on RGB Camera out of sync with IR Camera

Hello,
I have copied your project directly, using the same casing. After running the program, I can see that the frame of reference is off. Is there a fix for this?

Using an USB camera

Hi there how can make this code run using a usb camera, instead of the camera suggested in your article.

Which file is needed to only display temp on jetson by lepton3.5

I just wanna display the video of the highest temperature on IR thermal image, what file pack I need to download and which module I need to download on offcial lepton website?