aiden200/2d3mf

Code and models for the paper "2D3MF: Deepfake Detection using Multi Modal Middle Fusion"

Hi there 👋, I'm Aiden

• Here is my website!
• 🔭 I’m currently a: Graduate Student at the University of Southern California!
• 🌱 I’m currently learning: ML research - CV, Multimodal LLMs, Recommendation Systems.
• 💬 Ask me about: Multimodal LLMs!
• 📧 Email: [email protected]
• Linkedin: Aiden Chang
• 🖥️ Medium: aidenchang
• ⚡ Fun fact: I've lived in three different countries & was ranked 47th in the nation for freestyle skiing!

Toolset

2d3mf's Issues

Naming convensions/checkpoints for models

Pre-train Resnet for emotion detection

The task is to perform pre-training on a ResNet model on a simple emotion detection classification task. The input to the ResNet should be MFCCs computed using 1second of audio with a sampling rate of sr=44100Hz and n_mfcc=10 i.e. you can uselibrosa.feature.mfcc(y=y_1sec_audio, sr=44100, n_mfcc=10).

The network can be trained with audioclips from the RAVDESS dataset. The labels to predict should be 8:
01 = neutral, 02 = calm, 03 = happy, 04 = sad, 05 = angry, 06 = fearful, 07 = disgust, 08 = surprised

This pre-trained network will then be used as a feature extractor within our 2D3MF pipeline.

Combining datasets on inference

Grid Search

To test with autoML, implement Grid Search

General Feature extraction

Having trouble getting Models to be loaded in before performing feature extraction.

Middle Fusion Ablations

We want to try out different middle fusion structures. Please implement the following ablations for middle fusions using the files
model/multi_modal_middle_fusion.py, model/transformer_blocks.py, and model/classifier.py.

Unlike the diagram, the values are currently multiplied at the end. Here are the ablations:

Unify requirements.txt

Currently we have two requirements.txt files. This was done as part of the development process. Now that the repo is stable and the major chunk development is finished we should unify the requirements into a single file

General Preprocessing

We need a general preprocessing pipeline that crops, tags the videos as real(-0) or fake(-1). Then generates the .npy files based off of a backbone. Args: data dir, backbone type (small, base, large).

Please specify a required structure for input. Example:

**Data dir**
   Video
       Real
       Fake
   Audio
       Real
       Fake

efficientface import in preprocess broken

Hidden Dimension Change

Under model/classifier.py, self.hidden_layers needs to be able to change dynamically without throwing an error. It is currently set to 128.

Classes that are effected from self.hidden_layers:
AudioCNNPool, VideoCnnPool, AttentionBlock

Extract base marlin features