mrousavy/colorwaver

🎨 An app to detect color palettes in the real world - powered by VisionCamera

Colorwaver

An app to detect colorwaves (swatches/palettes) in the real world - powered by VisionCamera and Reanimated.

I wrote this app in less than a day, a speed simply not matched by native app development. Because it's written in React Native (TypeScript/JavaScript), it runs both on iOS and Android, but performance critical parts (e.g. the image processing algorithm or the animations) are backed by native Objective-C/Swift/Java code.

See this Tweet for more information.

Try it!

Download the repository and run the following commands to try Colorwaver for yourself:

iOS

yarn try-ios

Android

yarn try-android

Project structure

This is a bare React Native project, created with create-react-native-app.

---

• 📁 src: Contains the actual TypeScript + React (-Native) front-end for the Colorwaver App.

  • 📄 src/getColorPalette.ts: Exposes the native iOS/Android frame processor plugin as a JS function with TypeScript types. This function has to be called from a frame processor. ('worklet')

  • 📄 src/Router.tsx: The main component that gets registered by index.js. This acts as a main navigator, navigating either to the Splash Page (Permissions) or the App Page (Main App) depending on whether the user has granted permissions or not.

  • 📄 src/Splash.tsx: The first Splash page to ask the user for Camera permission.

  • 📄 src/App.tsx: Contains the actual source code for the App's front-end.

    • VisionCamera is used to get a Camera device and display a Camera component. Also, a frame processor (a function that gets called for every frame the camera "sees") gets attached, which calls the native iOS/Android frame processor plugin.
    • Reanimated is used to smoothly animate between color changes.

    Because VisionCamera also uses the Worklet API, the entire process between receiving a camera frame and actually displaying the palette's colors does not use the React-JS Thread at all. The frame processing runs on a separate Thread from VisionCamera, which then dispatches the animations on the Reanimated UI Thread. This is why the App runs as smooth as a native iOS or Android app.

  • 📄 src/useAnimatedColor.ts: A helper function to animate color changes with SharedValues.

---

• 📁 ios: Contains the basic skeleton for a React Native iOS app, plus the native getColorPalette(...) Frame Processor Plugin.

  • 📄 ios/PaletteFrameProcessorPlugin.m: Declares the Swift frame processor plugin "getColorPalette(...)".

  • 📄 ios/PaletteFrameProcessorPlugin.swift: Contains the actual Swift code for the native iOS frame processor plugin "getColorPalette(...)".

    This uses the CoreImage API to convert the CMSampleBuffer to a UIImage, and then uses the UIImageColors library to build the color palette.

    VisionCamera's frame processor API is used to expose this function as a frame processor plugin.

  • 📄 ios/Colorwaver-Bridging-Header.h: A Bridging Header to import Objective-C headers into Swift.

  • 📄 ios/Podfile: Adds the UIImageColors library.

  • 📄 ios/UIColor+hexString.swift: An extension for UIColor to convert UIColor instances to strings. This is required because React Native handles colors as strings.

---

• 📁 android: Contains the basic skeleton for a React Native Android app, plus the native getColorPalette(...) Frame Processor Plugin.

  • 📄 android/app/build.gradle: The gradle build file for the Android project. The following third-party dependencies are installed:

    • androidx.camera: camera-core
    • androidx.palette: palette

  • 📁 android/app/src/main/java/com/colorwaver/utils: Contains two files copied from android/camera-samples to convert ImageProxy instances to Bitmaps. (YuvToRgbConverter.kt)

  • 📁 android/app/src/main/java/com/colorwaver: Contains the actual Java source code of the Project.

  • 📄 android/app/src/main/java/com/colorwaver/MainApplication.java: Sets up react-native-reanimated.

  • 📄 android/app/src/main/java/com/colorwaver/MainActivity.java: Installs the PaletteFrameProcessorPlugin frame processor plugin inside of the onCreate method.

  • 📄 android/app/src/main/java/com/colorwaver/PaletteFrameProcessorPlugin.java: Contains the actual Java code for the native Android frame processor plugin "getColorPalette(...)".

    This uses the YuvToRgbConverter to convert the ImageProxy to a Bitmap, and then passes that to the Palette API from AndroidX to build the color palette.

    VisionCamera's frame processor API is used to expose this function as a frame processor plugin.

---

• 📄 babel.config.js: Adds the native frame processor plugin getColorPalette (called __getColorPalette) to Reanimated's global list.

Credits

• @Simek for the original idea
• react-native-reanimated for allowing JS code to be dispatched to another Runtime (Worklets)
• react-native-vision-camera for allowing you to run JS code on a realtime feed of Camera Frames (Frame Processors)
• The Palette API from AndroidX for getting the Color Palette on Android
• UIImageColors for getting the Color Palette on iOS
• You guys, for downloading, giving 5 stars, and pushing Colorwaver up in the Charts.

colorwaver's Issues

"Attempting to assign to readonly property" when using color interpolation

My animated color hook:

export function useAnimatedColor(
  color: Reanimated.SharedValue<string>,
): Readonly<Reanimated.SharedValue<string | number>> {
  const animation = useSharedValue(0);
  const colorFrom = useSharedValue(DEFAULT_COLOR);
  const colorTo = useSharedValue(color.value);

  useAnimatedReaction(
    () => color.value,
    (newColor, prevColor) => {
      animation.value = 0;
      colorFrom.value = prevColor ?? DEFAULT_COLOR;
      colorTo.value = newColor;
      animation.value = withTiming(1, {
        duration: 150,
        easing: Easing.linear,
      });
      console.log(`Animating from ${prevColor} -> ${newColor}`);
    },
  );

  // TODO: Using colorFrom and colorTo in here raises "Attempting to assign to readonly property" error...
  return useDerivedValue(() =>
    interpolateColor(animation.value, [0, 1], [colorFrom.value, colorTo.value]),
  );
}

is causing this error:

See software-mansion/react-native-reanimated#2329 for more details.

Right now, I've hotfixed this by using a custom patch for react-native-reanimated that completely removes the RGB cache. This now runs correctly, but ideally I want to fix this and make use of the cache for performance reasons.

Possible to share the plugin as npm package?

Hi

Is it possible to share PaletteFrameProcessorPlugin as a npm package, please? :)

great job, plz provide more example

Hi, i just read your comment from this topic: react-native-camera/react-native-camera#2942 (comment).

I think it's really good, I want to make some function like add effects into face while webrtc calling, can u provide some example? Thanks so much

Errors - while running the react-native run-android

Hi @mrousavy ,

This is truly amazing work!!

I was trying the instructions to run it for my android device, on a windows machine, got the below error:

Was hoping if you too had any similar issue:

FAILURE: Build failed with an exception.

• Where:
  Build file 'C:\Usersxxxxxvision_camera\Colorwaver\android\app\build.gradle' line: 162

• What went wrong:
  A problem occurred evaluating project ':app'.

Could not get unknown property 'release' for SigningConfig container of type org.gradle.api.internal.FactoryNamedDomainObjectContainer.

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

BUILD FAILED in 2s

error Failed to install the app. Make sure you have the Android development environment set up: https://reactnative.dev/docs/environment-setup.
Error: Command failed: gradlew.bat app:installDebug -PreactNativeDevServerPort=8081

FAILURE: Build failed with an exception.

• Where:
  Build file 'C:\Users\xxxxx\Colorwaver\android\app\build.gradle' line: 162

• What went wrong:
  A problem occurred evaluating project ':app'.

Could not get unknown property 'release' for SigningConfig container of type org.gradle.api.internal.FactoryNamedDomainObjectContainer.

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

"Tried to clone a value backed by a different Runtime" on Hermes/iOS

When enabling Hermes on iOS, ShareableValue::adapt crashes with the error

Tried to clone a value backed by a different Runtime

This is a SIGABRT and unfortunately really hard to debug. So I have no idea what's causing this.

Error when generating using npx react-native run-android --variant=release

I get this error when trying to compile using the command:
npx react-native run-android --variant=release

In file included from ../../../../src/main/cpp/JSIJNIConversion.cpp:25:
../../../../src/main/cpp/java-bindings/JHashMap.h:16:8: error: redefinition of 'JHashMap'
struct JHashMap : JavaClass<JHashMap<K, V>, JMap<K, V>> {
^
../../../../build/fbjni-0.3.0-headers.jar\fbjni/detail/Iterator.h:170:8: note: previous definition is here
struct JHashMap : JavaClass<JHashMap<K, V>, JMap<K, V>> {
^
1 error generated.

Gradle-build error while try-android

When I run react-native run-android this error pops up. My phone is connected through adb.

This is an amazing app and I would love to try it. Plz help me.

Thanks for sharing.