wuyiming6526/hard

Harmony-Rhythm Disentanglement audio remixer plugin

HARD

HArmony-Rhythm Disentanglement audio remixer plugin.

• AudioUnit plugin
• Tested on MacOS with arm64 / x86_64 architecture CPU.

This repository is a submission to the Neural Audio Plugin Competition (https://www.theaudioprogrammer.com/neural-audio).

---

How to install

1. Download HARD-AUplugin.zip from here.
2. Unzip the zip file and copy HARD.component to your AU plugin installation path. Typically the installation path is /Users/[Your Username]/Library/Audio/Plug-Ins/Components/ or /Library/Audio/Plug-Ins/Components/.

---

How to use

**Note: Make sure your DAW is running at sample rate 44.1kHz.

1. Create two audio tracks in your DAW and load a music audio clip into each track,
2. Synchronize the two audio clips using the audio time-stretching feature in your DAW,
3. Insert the HARD plugin to one of the audio tracks (If "Apple cannot check app for malicious software" notification shows up, manually allow the plugin from System Settings -> Security & Privacy, then restart the DAW)
4. Send the output of the other audio track to the sidechain channel of the HARD plugin,
5. Start playback and enjoy!

---

You can control audio generation by moving the sliders.

• Harmony / Rhythm: Adjust how much the generated audio contains the harmonic / rhythmic content of the audio from the source / sidechain channel. Moving the slider to the right side gives more weight to the sidechain channel
• Synchronise the Harmony and Rhythm slider by toggling the "sync" button
• Source Gain / Sidechain Gain: Adjust the level of the audio that is input to the neural network model. Use these sliders if there is a large difference in audio level between the source and sidechain inputs

---

How to build

This repository contains the entire XCode project.

1. Clone the repository using the following command:

git clone --recursive https://github.com/WuYiming6526/HARD.git

2. Download the model file from https://www.dropbox.com/s/ndd7rrrljjccqfh/morpher.onnx?dl=0. Copy the downloaded file morpher.onnx to the root directory.
3. Open the XCode project at Builds/MacOSX/HARD.xcodeproj
4. Build the project

The built AU plugin file is automatically copied to /Users/[Your Username]/Library/Audio/Plug-Ins/Components/ when the build process is finished. If your DAW cannot find HARD, move the plugin to the other installation path.

How it works

I trained a VAE that encodes an audio spectrogram into disentangled latent features representing the harmonic (=pitch-related) and rhythmic (=pitch-invariant) content of the input audio. The decoder then generates an audio spectrogram from the latent features. By interpolating the latent features using the horizontal sliders, you can separately change the harmonic and rhythmic content of the generated audio.

hard's Issues

"HARD'' shows up in my plugin manager on logic but "crashes validation" and can't be used

Please help, I'd love to try this plugin. Full plugin scan crash report below.

---

---

validating Audio Unit HARD by AlphaTheta:

AU Validation Tool
Version: 1.7.0 
Copyright 2003-2019, Apple Inc. All Rights Reserved.
Specify -h (-help) for command options

---

VALIDATING AUDIO UNIT: 'aufx' - 'Cbl9' - 'Manu'

Manufacturer String: AlphaTheta
AudioUnit Name: HARD
Component Version: 1.0.0 (0x10000)

• • PASS

---

TESTING OPEN TIMES:
COLD:
Time to open AudioUnit: 1282.850 ms
WARM:
Time to open AudioUnit: 1211.291 ms
FIRST TIME:
Time for initialization: 0.052 ms

• • PASS

---

VERIFYING DEFAULT SCOPE FORMATS:
Input Scope Bus Configuration:
Default Bus Count:2
Bus Name: Input
Format Bus 0: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Bus Name: Sidechain
Format Bus 1: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Bus 0, Has Channel Layouts: 0x640001 0x650002 0x660002 0x670002 0x6A0002
Default Layout (bus 0):
Tag=0x650002, Num Chan Descs=0

Output Scope Bus Configuration:
Default Bus Count:1
Bus Name: Output
Default Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Has Channel Layouts: 0x640001 0x650002 0x660002 0x670002 0x6A0002
Default Layout:
Tag=0x650002, Num Chan Descs=0

• • PASS

---

VERIFYING REQUIRED PROPERTIES:

• • PASS

---

VERIFYING RECOMMENDED PROPERTIES:
VERIFYING PROPERTY: Latency
PASS
VERIFYING PROPERTY: Tail Time
PASS
VERIFYING PROPERTY: Bypass Effect
PASS

• • PASS

---

VERIFYING OPTIONAL PROPERTIES:

• • PASS

---

VERIFYING SPECIAL PROPERTIES:

VERIFYING CUSTOM UI
Cocoa Views Available: 1
JUCE_AUCocoaViewClass_fa91a0fb95f092cc
PASS

HAS FACTORY PRESETS

VERIFYING CLASS INFO
PASS

TESTING HOST CALLBACKS
PASS

• • PASS

---

PUBLISHED PARAMETER INFO:

# # 5 Global Scope Parameters:

• • PASS

---

FORMAT TESTS:

Reported Channel Capabilities (explicit):
[1, 1] [2, 2]

Input/Output Channel Handling:
1-1 1-2 1-4 1-5 1-6 1-7 1-8 2-2 2-4 2-5 2-6 2-7 2-8 4-4 4-5 5-5 6-6 7-7 8-8
X X

# AudioChannelLayouts (5), Input Scope:

ChannelLayout is Writable: T
The Unit publishes the following Channel Layouts:
0x640001, 0x650002, 0x660002, 0x670002, 0x6A0002,

Is Audio Channel Layout Available:
Mono Stereo Binau. AU_4 Ambi. AU_5 AU_5_0 AU_6 AU_6_0 AU_7_0 AU_7_0F AU_8 AU_5_1 AU_6_1 AU_7_1 AU_7_1F
X X X

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x640001, Success - did not reset format with mis-matched layout

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x650002, Successsful

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x6A0002, Successsful

# AudioChannelLayouts (5), Output Scope:

ChannelLayout is Writable: T
The Unit publishes the following Channel Layouts:
0x640001, 0x650002, 0x660002, 0x670002, 0x6A0002,

Is Audio Channel Layout Available:
Mono Stereo Binau. AU_4 Ambi. AU_5 AU_5_0 AU_6 AU_6_0 AU_7_0 AU_7_0F AU_8 AU_5_1 AU_6_1 AU_7_1 AU_7_1F
X X X

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x640001, Success - did not reset format with mis-matched layout

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x650002, Successsful

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x6A0002, Successsful

• • PASS

---

RENDER TESTS:

Input Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Output Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Render Test at 512 frames
Slicing Render Test at 64 frames
PASS

Render Test at 64 frames, sample rate: 22050 Hz
Render Test at 137 frames, sample rate: 96000 Hz
Render Test at 4096 frames, sample rate: 48000 Hz
Render Test at 4096 frames, sample rate: 192000 Hz
Render Test at 4096 frames, sample rate: 11025 Hz
Render Test at 512 frames, sample rate: 44100 Hz
PASS

1 Channel Test:
Render Test at 512 frames

validation result: crashed validation

Rookie Question - sorry!

Hi! I'm not seeing the HARD-AUplugin.zip available anywhere for download - has it been taken down? If there's a direct link or any easy pathway to find and download it, I'd appreciate it so much. Thank you!!

HARD Crashed Validation in Logic Pro

Logic Pro 10.7.2
Model Name: Mac Pro
Model Identifier: MacPro7,1
Enclosure: Tower
Processor Name: 8-Core Intel Xeon W
Processor Speed: 3.5 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 1 MB
L3 Cache: 16.5 MB
Hyper-Threading Technology: Enabled
Memory: 32 GB

validating Audio Unit HARD by AlphaTheta:

AU Validation Tool
Version: 1.8.0 
Copyright 2003-2019, Apple Inc. All Rights Reserved.
Specify -h (-help) for command options

---

VALIDATING AUDIO UNIT: 'aufx' - 'Cbl9' - 'Manu'

Manufacturer String: AlphaTheta
AudioUnit Name: HARD
Component Version: 1.0.0 (0x10000)

• • PASS

---

TESTING OPEN TIMES:
COLD:
Time to open AudioUnit: 1474.569 ms
WARM:
Time to open AudioUnit: 1250.952 ms
FIRST TIME:
Time for initialization: 0.046 ms

• • PASS

---

VERIFYING DEFAULT SCOPE FORMATS:
Input Scope Bus Configuration:
Default Bus Count:2
Bus Name: Input
Format Bus 0: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Bus Name: Sidechain
Format Bus 1: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Bus 0, Has Channel Layouts: 0x640001 0x650002 0x660002 0x670002 0x6A0002
Default Layout (bus 0):
Tag=0x650002, Num Chan Descs=0

Output Scope Bus Configuration:
Default Bus Count:1
Bus Name: Output
Default Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Has Channel Layouts: 0x640001 0x650002 0x660002 0x670002 0x6A0002
Default Layout:
Tag=0x650002, Num Chan Descs=0

• • PASS

---

VERIFYING REQUIRED PROPERTIES:

• • PASS

---

VERIFYING RECOMMENDED PROPERTIES:
VERIFYING PROPERTY: Latency
PASS
VERIFYING PROPERTY: Tail Time
PASS
VERIFYING PROPERTY: Bypass Effect
PASS

• • PASS

---

VERIFYING OPTIONAL PROPERTIES:

• • PASS

---

VERIFYING SPECIAL PROPERTIES:

VERIFYING CUSTOM UI
Cocoa Views Available: 1
JUCE_AUCocoaViewClass_e056c561e2ea6379
PASS

HAS FACTORY PRESETS

VERIFYING CLASS INFO
PASS

TESTING HOST CALLBACKS
PASS

• • PASS

---

PUBLISHED PARAMETER INFO:

# # 5 Global Scope Parameters:

• • PASS

---

FORMAT TESTS:

Reported Channel Capabilities (explicit):
[1, 1] [2, 2]

Input/Output Channel Handling:
1-1 1-2 1-4 1-5 1-6 1-7 1-8 2-2 2-4 2-5 2-6 2-7 2-8 4-4 4-5 5-5 6-6 7-7 8-8
X X

# AudioChannelLayouts (5), Input Scope:

ChannelLayout is Writable: T
The Unit publishes the following Channel Layouts:
0x640001, 0x650002, 0x660002, 0x670002, 0x6A0002,

Is Audio Channel Layout Available:
Mono Stereo Binau. AU_4 Ambi. AU_5 AU_5_0 AU_6 AU_6_0 AU_7_0 AU_7_0F AU_8 AU_5_1 AU_6_1 AU_7_1 AU_7_1F
X X X

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x640001, Success - did not reset format with mis-matched layout

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x650002, Successsful

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x6A0002, Successsful

# AudioChannelLayouts (5), Output Scope:

ChannelLayout is Writable: T
The Unit publishes the following Channel Layouts:
0x640001, 0x650002, 0x660002, 0x670002, 0x6A0002,

Is Audio Channel Layout Available:
Mono Stereo Binau. AU_4 Ambi. AU_5 AU_5_0 AU_6 AU_6_0 AU_7_0 AU_7_0F AU_8 AU_5_1 AU_6_1 AU_7_1 AU_7_1F
X X X

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x640001, Success - did not reset format with mis-matched layout

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x650002, Successsful

Current Format:AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Current Format Tag = 0x650002, New Format Tag = 0x6A0002, Successsful

• • PASS

---

RENDER TESTS:

Input Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Output Format: AudioStreamBasicDescription: 2 ch, 44100 Hz, 'lpcm' (0x00000029) 32-bit little-endian float, deinterleaved
Render Test at 512 frames
Slicing Render Test at 64 frames
PASS

Render Test at 64 frames, sample rate: 22050 Hz
Render Test at 137 frames, sample rate: 96000 Hz
Render Test at 4096 frames, sample rate: 48000 Hz
Render Test at 4096 frames, sample rate: 192000 Hz
Render Test at 4096 frames, sample rate: 11025 Hz
Render Test at 512 frames, sample rate: 44100 Hz
PASS

1 Channel Test:
Render Test at 512 frames

validation result: crashed validation

Buildin' for Windows

Try to build on Windows 11 after add latest onnxruntime 1.18.1 (not 1.13.1) and changin' options on Juce (i add also asio support)
but i have 19 warnings and 50 errors, as these:
Error C2146 syntax error: ')' missing before identifier 'and' (PluginProcessor.cpp 187)
Error C3861 'and': identifier missing (PluginProcessor.cpp 187)
Error C2146 syntax error: ';' missing before identifier 'and' (in PluginProcessor.cpp 187)
Error C3861 'and': identifier missing (in PluginProcessor.cpp 187)
Error C2059 syntax error : ')' missing (in PluginProcessor.cpp 187)
Error C2059 syntax error: 'const' (PluginProcessor.cpp 189)
Error C2660 'ONNXMorpherInferenceThread::requestInference': the function not accept 5 arguments (PluginProcessor.cpp 194 )
Error C2447 '{': missing function header. Obsolete formal list? (PluginProcessor.cpp 198 )
Error C4430 type identifier missing, int will be used. Note: default-int not more supported in C++ (PluginProcessor.cpp 210)
Error C2065 'rhythmParameter': identifier not declared (PluginProcessor.cpp 210)
Error C4430 missing identifier of type int will be used. Note: default-int not more supported in C++ (PluginProcessor.cpp 211)
Error C2065 'harmonyParameter': identifier not declared ( ... /PluginProcessor.cpp 211)
Error C2059 syntax error : '}' (PluginProcessor.cpp 212 )
Error C2143 syntax error : ';' missing before PluginProcessor.cpp 212
Error C2143 syntax: ';' missing before '{' (in PluginProcessor.cpp 216)
Error C2447 '{': missing function header. Obsolete formal list? (in ../PluginProcessor.cpp 216)
Error C2440 '': impossible to convert from 'initializer list' to 'Ort::Session' (in ONNXInferenceThread.cpp 21 )
Error C2146 syntax: ')' missing before identifir 'and' ONNXInferenceThread.cpp 49
Error C3861 'and': header missing ... ONNXInferenceThread.cpp 49
Error C2059 syntax error: ')' ONNXInferenceThread.cpp 49
Error C2059 syntax error : 'return' ONNXInferenceThread.cpp 49
Error C2059 : 'return' ONNXInferenceThread.cpp 52
Error C2143 syntax error: ';' missing ONNXInferenceThread.cpp 52
Error C2143 syntax error: ';' missing before '{' ONNXInferenceThread.cpp 55
Error C2447 '{':missing function header. Obsolete formal list? ONNXInferenceThread.cpp 55
Error C2146 error syntax: ')' missing before identifier 'or' ONNXInferenceThread.cpp 82
Error C3861 'or': identifier missing ONNXInferenceThread.cpp 82
and others ...
any chance to a workin' code for Windows?

Running iMac, M1 2021: No sidechain window

I was able to get the plugin to load in Logic once I changed security settings, however, I expected to see a sidechain window, but there isn't one.

Trouble Installing

Hi! First time github user here - I'm running 10.15.7 on my macbook pro and can't seem to get the component showing up in the plugin directories of any of my DAW's.

I've copied the HARD component onto both directories of the Library but still no luck.

Any advice or recommendations on troubleshooting this?

is this exclusive to MacOS?

hello thank you so much for sharing this amazing creation!! i discovered your plugin on twitter and i am absolutely adoring it, but i am wondering if this is currently limited to mac users only? i have downloaded the zip file and attempted to load it onto my daw by creating a file path in several VST plugin locations with no success. :( let me know if there is a way for windows users to still have this experience! thank you again!!!! :)

Ableton crash

Ableton keeps crashing whilst I'm using the plugin, sometimes it does its thing but then crashes after a few minutes of working with it.

"Plug-In could not be checked for malicious intent"

Hey there! I can't seem to find a way to enable the use of the HARD plug in. I tried system preferences-> security & privacy. I put the contents of the .zip into my plug-ins->components folder, but Logic just keeps telling me it is incompatible.

Any help would be greatly appreciated, and thank you so much for what you are doing!

Ableton crashes when opening the plugin on M1

Any idea how to solve the issue? Thanks!