Creating Frame Processor Plugins
Creating a Frame Processor Plugin for Android​
The Frame Processor Plugin API is built to be as extensible as possible, which allows you to create custom Frame Processor Plugins. In this guide we will create a custom Face Detector Plugin which can be used from JS.
Android Frame Processor Plugins can be written in either Java, Kotlin or C++ (JNI).
Mostly automatic setup​
npx vision-camera-plugin-builder@latest android
The CLI will ask you for the path to project's Android Manifest file, name of the plugin (e.g. FaceDetectorFrameProcessorPlugin
), name of the exposed method (e.g. detectFaces
) and language you want to use for plugin development (Java or Kotlin).
For reference see the CLI's docs.
- Register the package in MainApplication.java
@Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// ...
packages.add(new FaceDetectorFrameProcessorPluginPackage()); // <- add
return packages;
}
Manual setup​
- Java
- Kotlin
- Open your Project in Android Studio
- Create a Java source file, for the Face Detector Plugin this will be called
FaceDetectorFrameProcessorPlugin.java
. - Add the following code:
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import com.mrousavy.camera.frameprocessors.Frame;
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin;
import com.mrousavy.camera.frameprocessors.VisionCameraProxy;
public class FaceDetectorFrameProcessorPlugin extends FrameProcessorPlugin {
FaceDetectorFrameProcessorPlugin(@NonNull VisionCameraProxy proxy, @Nullable Map<String, Object> options) {}
@Nullable
@Override
public Object callback(@NonNull Frame frame, @Nullable Map<String, Object> arguments) throws Throwable {
// code goes here
return null;
}
}
- Implement your Frame Processing. See the Example Plugin (Java) for reference.
- Create a new Java file which registers the Frame Processor Plugin in a React Package, for the Face Detector plugin this file will be called
FaceDetectorFrameProcessorPluginPackage.java
:
import androidx.annotation.NonNull;
import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin;
import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry;
public class FaceDetectorFrameProcessorPluginPackage implements ReactPackage {
static {
FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectFaces", FaceDetectorFrameProcessorPlugin::new);
}
@NonNull
@Override
public List<NativeModule> createNativeModules(@NonNull ReactApplicationContext reactContext) {
return Collections.emptyList();
}
@NonNull
@Override
public List<ViewManager> createViewManagers(@NonNull ReactApplicationContext reactContext) {
return Collections.emptyList();
}
}
The Frame Processor Plugin can be initialized from JavaScript using initFrameProcessorPlugin("detectFaces")
.
- Register the package in MainApplication.java
@Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// ...
packages.add(new FaceDetectorFrameProcessorPluginPackage()); // <- add
return packages;
}
- Open your Project in Android Studio
- Create a Kotlin source file, for the Face Detector Plugin this will be called
FaceDetectorFrameProcessorPlugin.kt
. - Add the following code:
import com.mrousavy.camera.frameprocessors.Frame
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin
import com.mrousavy.camera.frameprocessors.VisionCameraProxy
class FaceDetectorFrameProcessorPlugin(proxy: VisionCameraProxy, options: Map<String, Any>?): FrameProcessorPlugin() {
override fun callback(frame: Frame, arguments: Map<String, Any>?): Any? {
// code goes here
return null
}
}
- Implement your Frame Processing. See the Example Plugin (Kotlin) for reference.
- Create a new Kotlin file which registers the Frame Processor Plugin in a React Package, for the Face Detector plugin this file will be called
FaceDetectorFrameProcessorPluginPackage.kt
:
import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager
import com.mrousavy.camera.frameprocessors.FrameProcessorPlugin
import com.mrousavy.camera.frameprocessors.FrameProcessorPluginRegistry
class FaceDetectorFrameProcessorPluginPackage : ReactPackage {
companion object {
init {
FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectFaces") { proxy, options ->
FaceDetectorFrameProcessorPlugin(proxy, options)
}
}
}
override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
return emptyList()
}
override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
return emptyList()
}
}
The Frame Processor Plugin can be initialized from JavaScript using initFrameProcessorPlugin("detectFaces")
.
- Register the package in MainApplication.java
@Override
protected List<ReactPackage> getPackages() {
@SuppressWarnings("UnnecessaryLocalVariable")
List<ReactPackage> packages = new PackageList(this).getPackages();
// ...
packages.add(new FaceDetectorFrameProcessorPluginPackage()); // <- add
return packages;
}