### Install react-native-executorch-webrtc Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch-webrtc/README.md Install the package using yarn. For iOS, navigate to the ios directory and run pod install. ```bash yarn add react-native-executorch-webrtc ``` ```bash cd ios && pod install ``` -------------------------------- ### Install Dependencies Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Navigate to the `docs` directory and install project dependencies using Yarn. ```bash cd docs yarn install ``` -------------------------------- ### Install React Native ExecuTorch and Dependencies Source: https://github.com/software-mansion/react-native-executorch/blob/main/README.md Installs the core package and necessary resource fetching libraries. Choose the platform-specific run command after installation. ```bash yarn add react-native-executorch yarn add react-native-executorch-expo-resource-fetcher yarn add expo-file-system expo-asset yarn ``` -------------------------------- ### Install Dependencies and Run Demo App Source: https://github.com/software-mansion/react-native-executorch/blob/main/README.md Navigate to a demo app's project directory, install dependencies, and run the app for iOS or Android. Replace with your target platform. ```bash yarn && yarn ``` -------------------------------- ### Build Example App for Native Libraries Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/README.md Build an example app to generate required native libraries like libfbjni.so and libc++_shared.so. These are automatically searched for by the test script. ```bash cd apps/computer-vision/android ./gradlew assembleDebug # or ./gradlew assembleRelease ``` -------------------------------- ### Install react-native-rag and Optional Persistence Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Install the core react-native-rag library. Optionally, install @react-native-rag/op-sqlite for SQLite-backed persistence. ```bash npm install react-native-rag # Optional SQLite-backed persistence (otherwise an in-memory store is used) npm install @react-native-rag/op-sqlite ``` -------------------------------- ### Start Development Server Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Start the Docusaurus development server. This command also generates the API reference from TypeScript source before starting the site. ```bash yarn start ``` -------------------------------- ### Install React Native ExecuTorch Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Install the core library and the appropriate resource fetcher for your project type. ```bash npm install react-native-executorch # Expo projects npm install react-native-executorch-expo-resource-fetcher # Bare React Native projects npm install react-native-executorch-bare-resource-resource-fetcher ``` -------------------------------- ### Image Classification - useClassification Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Demonstrates how to perform image classification using the `useClassification` hook. It shows model initialization with `models.classification.efficientnet_v2_s()` and how to get classification labels and probabilities from an image. ```APIDOC ## Image classification — `useClassification` ### Description Performs image classification using a specified model. Accepts various image input formats and returns a map of labels to their probabilities. ### Method Signature `useClassification({ model: ModelConfig })` ### Parameters #### Model Configuration - **model** (object) - Required - Configuration for the classification model. Use `models.classification.()`. ### Input Image Formats - Remote URL (`https://…`) - Local file URI (`file://…`) - Base64 string - Bundled assets (`require('../assets/img.jpg')`) ### Forward Pass `model.forward(imageInput)` ### Return Value - `labels`: `Record` - A map where keys are ImageNet1k labels and values are their corresponding probabilities. ### Example ```tsx import { useClassification, models } from 'react-react-native-executorch'; const model = useClassification({ model: models.classification.efficientnet_v2_s() }); const labels = await model.forward('https://example.com/puppy.png'); // labels: Record — ImageNet1k label → probability const topThree = Object.entries(labels) .sort(([, a], [, b]) => b - a) .slice(0, 3); ``` ``` -------------------------------- ### Initialize RAG with ExecuTorch Models Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Set up a MemoryVectorStore with ExecuTorch embeddings and an ExecuTorchLLM. This example uses specific models from the react-native-executorch registry. ```tsx import { useRAG, MemoryVectorStore, ExecuTorchEmbeddings, ExecuTorchLLM } from 'react-native-rag'; import { models } from 'react-native-executorch'; const vectorStore = new MemoryVectorStore({ embeddings: new ExecuTorchEmbeddings(models.text_embedding.all_minilm_l6_v2()), }); const llm = new ExecuTorchLLM(models.llm.lfm2_5_1_2b_instruct()); export default function App() { const rag = useRAG({ vectorStore, llm }); // rag.addDocument(text), rag.query(question) → rag.response (streamed) return {rag.response}; } ``` -------------------------------- ### Create New Version Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Snapshot the documentation for a new npm version. Run this command from the `docs/` folder, replacing `1.0.0` with the actual version number. ```bash yarn docs:version 1.0.0 ``` -------------------------------- ### Initialize Git Submodules Source: https://github.com/software-mansion/react-native-executorch/blob/main/README.md Initialize required git submodules for demo applications. Run this command from the repository root. ```bash git submodule update --init packages/react-native-executorch/third-party/common ``` -------------------------------- ### Deploy using SSH Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Deploy the documentation website using SSH. This command builds the static files and pushes them to the `gh-pages` branch. ```bash USE_SSH=true yarn deploy ``` -------------------------------- ### useTokenizer Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Provides a HuggingFace-compatible BPE/WordPiece tokenizer for counting tokens before sending text to embedding models or LLMs. It offers methods to encode text to token IDs, decode token IDs back to text, get vocabulary size, and map tokens to IDs and vice-versa. ```APIDOC ## useTokenizer ### Description HuggingFace-compatible BPE / WordPiece tokenizer. Mostly useful for counting tokens before sending text to embedding models or LLMs. ### Usage ```tsx import { useTokenizer, models } from 'react-native-executorch'; const tokenizer = useTokenizer({ tokenizer: models.text_embedding.all_minilm_l6_v2() }); // Encode text to token IDs const ids = await tokenizer.encode('Hello, world!'); // Decode token IDs to text const text = await tokenizer.decode(ids); // Get vocabulary size const vocab = await tokenizer.getVocabSize(); // Map token to ID const id = await tokenizer.tokenToId('hello'); // Map ID to token const token = await tokenizer.idToToken(id); ``` **Note:** You usually don't need this — `useLLM` and `useTextEmbeddings` tokenize internally. ``` -------------------------------- ### Clone Repository and Set Up Remote Source: https://github.com/software-mansion/react-native-executorch/blob/main/CONTRIBUTING.md Clone your forked repository and add the base repository as a remote. This is a standard Git workflow for contributing to open-source projects. ```bash git clone git@github.com:/react-native-executorch.git cd react-native-executorch git remote add upstream https://github.com/software-mansion/react-native-executorch.git ``` -------------------------------- ### Build Static Files Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Build the static files for a production deployment. The output will be generated in the `build/` directory. ```bash yarn build ``` -------------------------------- ### Basic CMake Configuration Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets the minimum CMake version, project name, and C++ standard for the build. ```cmake cmake_minimum_required(VERSION 3.13) project(RNExecutorchTests) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED TRUE) ``` -------------------------------- ### Configuring LLM for Tool/Function Calling Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/llm.md Set up LLM with tools for function calling. Define tools with names, descriptions, and parameter schemas. Implement a callback to execute these tools. ```tsx import { useLLM, models, DEFAULT_SYSTEM_PROMPT, LLMTool, ToolCall, } from 'react-native-executorch'; const TOOLS: LLMTool[] = [ { name: 'get_weather', description: 'Get current weather in a given location.', parameters: { type: 'dict', properties: { location: { type: 'string', description: 'Location to check weather for' }, }, required: ['location'], }, }, ]; const executeTool = async (call: ToolCall): Promise => { switch (call.toolName) { case 'get_weather': return 'It is sunny and 21°C.'; default: return null; } }; const llm = useLLM({ model: models.llm.hammer2_1_1_5b() }); useEffect(() => { llm.configure({ chatConfig: { systemPrompt: `${DEFAULT_SYSTEM_PROMPT} Current time: ${new Date().toString()}`, }, toolsConfig: { tools: TOOLS, executeToolCallback: executeTool, displayToolCalls: true, }, }); }, []); ``` -------------------------------- ### Add Sampler Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up the SamplerTests executable with specific source files. ```cmake add_rn_test(SamplerTests unit/SamplerTest.cpp SOURCES ${COMMON_DIR}/runner/sampler.cpp ${COMMON_DIR}/runner/arange_util.cpp LIBS ) ``` -------------------------------- ### LLM Accessor Options Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/llm.md Demonstrates how to select LLM models with different configurations like default quantization, disabling quantization, or specifying a backend. ```typescript models.llm.llama3_2_3b(); // Non-quantized variant. models.llm.llama3_2_3b({ quant: false }); // Explicit backend — only the backends the model actually ships are accepted. models.llm.qwen3_4b({ backend: 'xnnpack' }); ``` -------------------------------- ### Importing ExecuTorch Prebuilt Binaries Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Imports the prebuilt ExecuTorch shared library (`libexecutorch.so`) for the specified Android ABI. ```cmake # ExecuTorch Prebuilt binaries add_library(executorch_prebuilt SHARED IMPORTED) set_target_properties(executorch_prebuilt PROPERTIES IMPORTED_LOCATION "${ANDROID_THIRD_PARTY}/executorch/${ANDROID_ABI}/libexecutorch.so" ) ``` -------------------------------- ### Configure Tokenizer Dependencies Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up include directories for the tokenizers library. This is used when tokenizers are a dependency. ```cmake add_library(tokenizers_deps INTERFACE) target_include_directories(tokenizers_deps INTERFACE "${TOKENIZERS_DIR}") ``` -------------------------------- ### Initialize with Bare React Native Resource Fetcher Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Initialize the ExecuTorch library using the bare React Native resource fetcher adapter. This must be done before using any other ExecuTorch APIs. ```tsx // Bare React Native import { initExecutorch } from 'react-native-executorch'; import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher'; initExecutorch({ resourceFetcher: BareResourceFetcher }); ``` -------------------------------- ### useTextToSpeech Hook Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/speech.md Initializes the Text-to-Speech engine with a specified Kokoro model and provides a `forward` method to synthesize speech from text. The synthesized waveform can then be played using an audio context. ```APIDOC ## useTextToSpeech ### Description Initializes the Text-to-Speech engine with a specified Kokoro model. This hook provides a `forward` method to synthesize speech from text. ### Method Signature `useTextToSpeech(options: { model: Model }): { forward: (options: { text: string, speed: number }) => Promise }` ### Parameters #### `useTextToSpeech` Options - **model** (Model) - Required - A Kokoro preset that bundles the model, a voice, and the phonemizer for a specific language. #### `forward` Options - **text** (string) - Required - The text to be converted into speech. - **speed** (number) - Required - The speech speed, where 1.0 is normal speed. ### Returns - `forward`: A Promise that resolves to a `Float32Array` representing the audio waveform. ### Example ```tsx import { useTextToSpeech, models } from 'react-native-executorch'; import { AudioContext } from 'react-native-audio-api'; const tts = useTextToSpeech({ model: models.text_to_speech.kokoro.en_us.heart(), }); const audioContext = new AudioContext({ sampleRate: 24000 }); const speak = async (text: string) => { const waveform = await tts.forward({ text, speed: 1.0 }); // Float32Array @ 24 kHz const buffer = audioContext.createBuffer(1, waveform.length, 24000); buffer.getChannelData(0).set(waveform); const source = audioContext.createBufferSource(); source.buffer = buffer; source.connect(audioContext.destination); source.start(); }; ``` ``` -------------------------------- ### Initialize Executorch with Bare Resource Fetcher Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/bare-resource-fetcher/README.md Initialize the Executorch runtime with the BareResourceFetcher for native filesystem access in bare React Native projects. ```typescript import { initExecutorch } from 'react-native-executorch'; import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher'; initExecutorch({ resourceFetcher: BareResourceFetcher, }); ``` -------------------------------- ### Deploy without SSH Source: https://github.com/software-mansion/react-native-executorch/blob/main/docs/README.md Deploy the documentation website without using SSH. Replace `` with your actual GitHub username. This command builds the static files and pushes them to the `gh-pages` branch. ```bash GIT_USER= yarn deploy ``` -------------------------------- ### Object Detection - useObjectDetection Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Illustrates object detection using the `useObjectDetection` hook. It covers initializing models like `yolo26n`, specifying detection parameters such as `detectionThreshold` and `iouThreshold`, and processing the returned detections. ```APIDOC ## Object detection — `useObjectDetection` ### Description Performs object detection on an image, identifying bounding boxes, labels, and confidence scores for detected objects. Supports various YOLO and other object detection models. ### Method Signature `useObjectDetection({ model: ModelConfig })` ### Parameters #### Model Configuration - **model** (object) - Required - Configuration for the object detection model. Use `models.object_detection.()`. ### Input Image Formats - Remote URL (`https://…`) - Local file URI (`file://…`) - Base64 string - Bundled assets (`require('../assets/img.jpg')`) ### Forward Pass `model.forward(imageInput, options?)` ### Parameters for `forward` - **imageInput** (string | object) - Required - The image to process. Can be a URL, file URI, base64 string, or bundled asset. - **options** (object) - Optional - Configuration for detection. - **detectionThreshold** (number) - Optional - Minimum confidence score (0–1) for a detection to be considered valid. - **iouThreshold** (number) - Optional - Intersection over Union threshold (0–1) for Non-Maximum Suppression (NMS) aggressiveness. - **inputSize** (number) - Optional - Specifies the input size for multi-size YOLO models (e.g., 384, 512, 640). - **classesOfInterest** (string[]) - Optional - An array of class names to filter detections. ### Return Value - `detections`: `Detection[]` - An array of detected objects, where each object has: - **bbox**: `{ x1: number, y1: number, x2: number, y2: number }` - Bounding box coordinates in pixel space. - **label**: `string` - The detected object's class label. - **score**: `number` - The confidence score of the detection. ### Available Models - `models.object_detection.yolo26n` / `yolo26s` / `yolo26m` / `yolo26l` / `yolo26x` - `models.object_detection.rf_detr_nano` - `models.object_detection.ssdlite_320_mobilenet_v3_large` ### Example ```tsx import { useObjectDetection, models } from 'react-native-executorch'; const model = useObjectDetection({ model: models.object_detection.yolo26n() }); const detections = await model.forward('https://example.com/street.jpg', { detectionThreshold: 0.5, // minimum confidence (0–1) iouThreshold: 0.45, // NMS aggressiveness (0–1) inputSize: 640, // for multi-size YOLO models (384 / 512 / 640) classesOfInterest: ['PERSON', 'CAR'], // filter }); for (const d of detections) { console.log(d.bbox, d.label, d.score); } ``` **Note:** YOLO models support multiple input sizes. Call `model.getAvailableInputSizes()` to enumerate them. ``` -------------------------------- ### Add Image Processing Test Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up the ImageProcessingTest executable, including image utility sources and opencv_deps. ```cmake add_rn_test(ImageProcessingTest unit/ImageProcessingTest.cpp SOURCES ${IMAGE_UTILS_SOURCES} LIBS opencv_deps ) ``` -------------------------------- ### Add Frame Transform Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up FrameTransformTests, linking it with FrameTransform, image utilities, and opencv_deps. ```cmake add_rn_test(FrameTransformTests unit/FrameTransformTest.cpp SOURCES ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp LIBS opencv_deps ) ``` -------------------------------- ### Basic Text-to-Speech Usage Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/speech.md Initialize the text-to-speech engine with a specific model and use it to convert text to speech. Ensure the AudioContext is configured with the correct sample rate. ```tsx import { useTextToSpeech, models } from 'react-native-executorch'; import { AudioContext } from 'react-native-audio-api'; const tts = useTextToSpeech({ model: models.text_to_speech.kokoro.en_us.heart(), }); const audioContext = new AudioContext({ sampleRate: 24000 }); const speak = async (text: string) => { const waveform = await tts.forward({ text, speed: 1.0 }); // Float32Array @ 24 kHz const buffer = audioContext.createBuffer(1, waveform.length, 24000); buffer.getChannelData(0).set(waveform); const source = audioContext.createBufferSource(); source.buffer = buffer; source.connect(audioContext.destination); source.start(); }; ``` -------------------------------- ### Download Model with Progress Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Use ExpoResourceFetcher to download model files with a progress callback. It returns local file paths upon successful download or null if interrupted. ```tsx import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher'; // Download with progress const uris = await ExpoResourceFetcher.fetch( (p) => console.log(`${Math.round(p * 100)}%`), 'https://example.com/model.pte', 'https://example.com/tokenizer.bin' ); // uris: string[] of local file paths (no `file://` prefix), or null if interrupted ``` -------------------------------- ### Configure Metro for .pte Models Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Register the '.pte' and '.bin' extensions in your metro.config.js to allow bundling models as assets. ```javascript const { getDefaultConfig } = require('expo/metro-config'); const config = getDefaultConfig(__dirname); config.resolver.assetExts.push('pte'); config.resolver.assetExts.push('bin'); module.exports = config; ``` -------------------------------- ### Instance Segmentation with YOLOv26n Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Utilize `useInstanceSegmentation` to detect objects and generate per-instance masks. The output includes bounding box, label, score, and mask for each detected object. ```tsx import { useInstanceSegmentation, models } from 'react-native-executorch'; const model = useInstanceSegmentation({ model: models.instance_segmentation.yolo26n(), }); const instances = await model.forward('https://example.com/street.jpg'); // instances: { bbox, label, score, mask }[] ``` -------------------------------- ### Initialize ExecuTorch with Bare React Native Resource Fetcher Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Initialize the ExecuTorch library with the bare React Native resource fetcher. This must be called at the app's entry point before any other API calls. ```tsx import { initExecutorch } from 'react-native-executorch'; import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher'; initExecutorch({ resourceFetcher: BareResourceFetcher }); ``` -------------------------------- ### Initialize ExecuTorch and LLM Model Source: https://github.com/software-mansion/react-native-executorch/blob/main/README.md Sets up the ExecuTorch environment with a resource fetcher and initializes the Large Language Model (LLM) for use within a React Native component. ```tsx import { useLLM, models, Message, initExecutorch, } from 'react-native-executorch'; import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher'; initExecutorch({ resourceFetcher: ExpoResourceFetcher, }); function MyComponent() { // Initialize the model 🚀 const llm = useLLM({ model: models.llm.lfm2_5_1_2b_instruct() }); // ... rest of your component } ``` -------------------------------- ### Object Detection with useObjectDetection Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Employ the `useObjectDetection` hook for object detection. Configure the model with parameters like `detectionThreshold`, `iouThreshold`, `inputSize`, and `classesOfInterest`. The `forward` method returns an array of `Detection` objects, each containing bounding box coordinates, label, and confidence score. ```tsx import { useObjectDetection, models } from 'react-native-executorch'; const model = useObjectDetection({ model: models.object_detection.yolo26n() }); const detections = await model.forward('https://example.com/street.jpg', { detectionThreshold: 0.5, // minimum confidence (0–1) iouThreshold: 0.45, // NMS aggressiveness (0–1) inputSize: 640, // for multi-size YOLO models (384 / 512 / 640) classesOfInterest: ['PERSON', 'CAR'], // filter }); for (const d of detections) { console.log(d.bbox, d.label, d.score); } ``` -------------------------------- ### useTextToSpeech Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for text-to-speech synthesis using Kokoro TTS, supporting batch and streaming modes with phoneme input. ```APIDOC ## useTextToSpeech ### Description Synthesizes speech from text using Kokoro TTS. Supports batch and streaming modes, and accepts phoneme input. ### Reference [speech.md](./references/speech.md) ``` -------------------------------- ### Configure OpenCV Dependencies Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up OpenCV library paths and links necessary libraries for different Android ABIs. This is used when OpenCV is a dependency. ```cmake set(OPENCV_LIBS_DIR "${ANDROID_THIRD_PARTY}/opencv/${ANDROID_ABI}") set(OPENCV_THIRD_PARTY_DIR "${ANDROID_THIRD_PARTY}/opencv-third-party/${ANDROID_ABI}") if(ANDROID_ABI STREQUAL "arm64-v8a") set(OPENCV_THIRD_PARTY_LIBS "${OPENCV_THIRD_PARTY_DIR}/libkleidicv_hal.a" "${OPENCV_THIRD_PARTY_DIR}/libkleidicv_thread.a" "${OPENCV_THIRD_PARTY_DIR}/libkleidicv.a" ) elseif(ANDROID_ABI STREQUAL "x86_64") set(OPENCV_THIRD_PARTY_LIBS "") endif() add_library(opencv_deps INTERFACE) target_link_libraries(opencv_deps INTERFACE ${OPENCV_LIBS_DIR}/libopencv_core.a ${OPENCV_LIBS_DIR}/libopencv_features2d.a ${OPENCV_LIBS_DIR}/libopencv_highgui.a ${OPENCV_LIBS_DIR}/libopencv_imgproc.a ${OPENCV_LIBS_DIR}/libopencv_photo.a ${OPENCV_LIBS_DIR}/libopencv_video.a ${OPENCV_THIRD_PARTY_LIBS} ${EXECUTORCH_LIBS} z dl m log ) target_link_options(opencv_deps INTERFACE -fopenmp -static-openmp) ``` -------------------------------- ### usePoseEstimation Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for human pose estimation, detecting keypoints on a human body (COCO 17-keypoint format). ```APIDOC ## usePoseEstimation ### Description Estimates human pose by detecting keypoints, following the COCO 17-keypoint standard. ### Reference [vision.md](./references/vision.md) ``` -------------------------------- ### Initialize ExecuTorch with Expo Resource Fetcher Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Initialize the ExecuTorch library with the Expo-specific resource fetcher. This should be called at the app's entry point before any other API calls. ```tsx import { initExecutorch } from 'react-native-executorch'; import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher'; initExecutorch({ resourceFetcher: ExpoResourceFetcher }); ``` -------------------------------- ### Define Tokenizer Source Files Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Lists the source file for the Tokenizer module. This is used for text tokenization functionalities. ```cmake set(TOKENIZER_SOURCES ${RNEXECUTORCH_DIR}/TokenizerModule.cpp) ``` -------------------------------- ### Define Core Library Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Defines the static library 'rntests_core' with its associated source files and include directories. ```cmake add_library(rntests_core STATIC ${CORE_SOURCES}) target_include_directories(rntests_core PUBLIC ${RNEXECUTORCH_DIR}/data_processing ${TOKENIZERS_DIR} ${RNEXECUTORCH_DIR} ${COMMON_DIR} ${PACKAGE_ROOT}/third-party/include ${PACKAGE_ROOT}/third-party/include/cpuinfo ${PACKAGE_ROOT}/third-party/include/pthreadpool ${PHONEMIS_DIR}/src ${REACT_NATIVE_DIR}/ReactCommon ${REACT_NATIVE_DIR}/ReactCommon/jsi ${REACT_NATIVE_DIR}/ReactCommon/callinvoker ${COMMON_DIR}/ada ) target_link_libraries(rntests_core PUBLIC executorch_prebuilt gtest log ) ``` -------------------------------- ### Add FileUtils Test Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures the FileUtilsTest executable, linking it with the core library and gtest_main. ```cmake add_rn_test(FileUtilsTest unit/FileUtilsTest.cpp) ``` -------------------------------- ### useTextToImage Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for generating images from text descriptions using Stable Diffusion models. ```APIDOC ## useTextToImage ### Description Generates images based on textual prompts using Stable Diffusion models. ### Reference [vision.md](./references/vision.md) ``` -------------------------------- ### Configure Phonemis Subdirectory Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Adds the Phonemis library as a subdirectory and sets compile definitions and include directories. This is used for integrating the Phonemis library. ```cmake set(PHONEMIS_DIR "${PACKAGE_ROOT}/third-party/common/phonemis") add_subdirectory(${PHONEMIS_DIR} ${PROJECT_BINARY_DIR}/phonemis) target_compile_definitions(phonemis PRIVATE ET_ON) target_include_directories(phonemis PRIVATE "${PACKAGE_ROOT}/third-party/include") ``` -------------------------------- ### Generate Image with useTextToImage Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Use the `useTextToImage` hook to generate images from text prompts. Specify the model, prompt, image size, and number of steps. Image size must be a multiple of 32. Performance varies by device and parameters. ```tsx import { useTextToImage, models } from 'react-native-executorch'; const model = useTextToImage({ model: models.image_generation.bk_sdm_tiny_vpred_256() }); const image = await model.generate('a medieval castle by the sea', 256, 25); // image: base64 PNG. Render with ``` -------------------------------- ### Add Style Transfer Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up StyleTransferTests with StyleTransfer model, vision model, frame processing utilities, image utilities, and Android-specific libraries. ```cmake add_rn_test(StyleTransferTests integration/StyleTransferTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/style_transfer/StyleTransfer.cpp ${RNEXECUTORCH_DIR}/models/VisionModel.cpp ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### Add Classification Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up ClassificationTests with Classification model, vision model, frame processing utilities, image utilities, and Android-specific libraries. ```cmake add_rn_test(ClassificationTests integration/ClassificationTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/classification/Classification.cpp ${RNEXECUTORCH_DIR}/models/VisionModel.cpp ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### Define Image Utilities Source Files Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Lists the source files for image processing utilities. These are used for handling image data within the module. ```cmake set(IMAGE_UTILS_SOURCES ${RNEXECUTORCH_DIR}/data_processing/ImageProcessing.cpp ${RNEXECUTORCH_DIR}/data_processing/base64.cpp ${COMMON_DIR}/ada/ada.cpp ) ``` -------------------------------- ### Adding Google Test Subdirectory Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Includes the Google Test framework as a subdirectory for testing purposes. ```cmake # Add Gtest as a subdirectory add_subdirectory(${THIRD_PARTY_DIR}/googletest ${PROJECT_BINARY_DIR}/googletest) ``` -------------------------------- ### useLLM Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for text generation, chat, tool calling, and visual language modeling (VLM). ```APIDOC ## useLLM ### Description Provides functionality for text generation, chat interactions, tool calling, and visual language modeling. ### Reference [llm.md](./references/llm.md) ``` -------------------------------- ### Define DSP Source Files Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Lists the source file for the Digital Signal Processing (DSP) module. This is used for audio processing functionalities. ```cmake set(DSP_SOURCES ${RNEXECUTORCH_DIR}/data_processing/dsp.cpp) ``` -------------------------------- ### Define Core Source Files Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Lists the source files for the core module of React Native Executorch. These are essential for the base functionality. ```cmake set(CORE_SOURCES ${RNEXECUTORCH_DIR}/models/BaseModel.cpp ${RNEXECUTORCH_DIR}/data_processing/Numerical.cpp ${CMAKE_SOURCE_DIR}/integration/stubs/jsi_stubs.cpp ) ``` -------------------------------- ### Add Object Detection Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures ObjectDetectionTests with ObjectDetection model, vision model, frame processing utilities, image utilities, and Android-specific libraries. ```cmake add_rn_test(ObjectDetectionTests integration/ObjectDetectionTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/object_detection/ObjectDetection.cpp ${RNEXECUTORCH_DIR}/models/VisionModel.cpp ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp ${RNEXECUTORCH_DIR}/utils/computer_vision/Processing.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### Add Image Embeddings Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Sets up ImageEmbeddingsTests with ImageEmbeddings model, vision model, frame processing utilities, image utilities, and Android-specific libraries. ```cmake add_rn_test(ImageEmbeddingsTests integration/ImageEmbeddingsTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/embeddings/image/ImageEmbeddings.cpp ${RNEXECUTORCH_DIR}/models/VisionModel.cpp ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### useImageEmbeddings Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for generating image embeddings using CLIP models, useful for image similarity tasks. ```APIDOC ## useImageEmbeddings ### Description Generates image embeddings using CLIP models, suitable for tasks like image similarity search. ### Reference [vision.md](./references/vision.md) ``` -------------------------------- ### Add Text-to-Speech Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures the CMake build system to include tests for the Text-to-Speech functionality using the Kokoro model. This target depends on the phonemis library. ```cmake add_rn_test(TextToSpeechTests integration/TextToSpeechTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/text_to_speech/kokoro/Kokoro.cpp ${RNEXECUTORCH_DIR}/models/text_to_speech/kokoro/DurationPredictor.cpp ${RNEXECUTORCH_DIR}/models/text_to_speech/kokoro/Synthesizer.cpp ${RNEXECUTORCH_DIR}/models/text_to_speech/kokoro/Partitioner.cpp ${RNEXECUTORCH_DIR}/models/text_to_speech/kokoro/Utils.cpp LIBS phonemis ) ``` -------------------------------- ### Add Instance Segmentation Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures the CMake build system to include tests for Instance Segmentation models. This target includes source files for the base model, vision model utilities, frame processing, and computer vision processing, and depends on OpenCV and Android libraries. ```cmake add_rn_test(InstanceSegmentationTests integration/InstanceSegmentationTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/instance_segmentation/BaseInstanceSegmentation.cpp ${RNEXECUTORCH_DIR}/models/VisionModel.cpp ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${RNEXECUTORCH_DIR}/utils/FrameTransform.cpp ${RNEXECUTORCH_DIR}/utils/computer_vision/Processing.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### Add Numerical Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Adds the NumericalTests executable, linking it with the core library and gtest_main. ```cmake add_rn_test(NumericalTests unit/NumericalTest.cpp) ``` -------------------------------- ### useTextToImage Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/vision.md Generates images from text prompts using on-device Stable Diffusion models. Supports different model variants for various device capabilities. ```APIDOC ## Text-to-image — `useTextToImage` On-device Stable Diffusion (BK-SDM tiny). ```tsx import { useTextToImage, models } from 'react-native-executorch'; const model = useTextToImage({ model: models.image_generation.bk_sdm_tiny_vpred_256() }); const image = await model.generate('a medieval castle by the sea', 256, 25); // image: base64 PNG. Render with ``` ### Signature `generate(prompt, imageSize?, numSteps?)` ### Parameters - **prompt** (string) - Required - The text prompt to generate an image from. - **imageSize** (number) - Optional - The desired size of the generated image. Must be a multiple of 32. - **numSteps** (number) - Optional - The number of diffusion steps to perform. Affects generation time and quality. ### Notes - Image size must be a multiple of 32. - Expect 20–60 s per image depending on device, size, and step count. - Use `bk_sdm_tiny_vpred_256` on lower-end devices; `bk_sdm_tiny_vpred_512` on high-end devices. ``` -------------------------------- ### Add Log Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Adds the LogTests executable, which relies on the core library and gtest_main. ```cmake add_rn_test(LogTests unit/LogTest.cpp) ``` -------------------------------- ### Add Voice Activity Detection Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures VADTests with VoiceActivityDetection model, utility sources, and DSP sources. ```cmake add_rn_test(VADTests integration/VoiceActivityDetectionTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/voice_activity_detection/VoiceActivityDetection.cpp ${RNEXECUTORCH_DIR}/models/voice_activity_detection/Utils.cpp ${DSP_SOURCES} ) ``` -------------------------------- ### Add Runner Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures the RunnerTests executable, including additional source files and libraries. ```cmake add_rn_test(RunnerTests unit/RunnerTest.cpp SOURCES ${COMMON_DIR}/runner/base_llm_runner.cpp ${COMMON_DIR}/runner/sampler.cpp ${COMMON_DIR}/runner/arange_util.cpp integration/stubs/jsi_stubs.cpp LIBS tokenizers_deps ) ``` -------------------------------- ### Add Base Model Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Adds the BaseModelTests executable, linking it with the core library and gtest_main. ```cmake add_rn_test(BaseModelTests integration/BaseModelTest.cpp) ``` -------------------------------- ### Available TTS Presets Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/speech.md Lists the available Kokoro TTS presets, which are combinations of locale and voice. These presets are used to initialize the `useTextToSpeech` hook. ```APIDOC ## Available TTS Presets ### Description `models.text_to_speech.kokoro..` — locale + voice combinations available for Text-to-Speech. ### Presets Table | Locale | Voices | |---|---| | `en_us` | `heart`, `river`, `sarah`, `adam`, `michael`, `santa` | | `en_gb` | `emma`, `daniel` | | `fr` | `siwis` | | `es` | `dora`, `alex` | | `it` | `sara`, `nicola` | | `pt` | `dora`, `santa` | | `hi` | `alpha`, `omega`, `psi` | | `pl` | `mateusz` | | `de` | `anna` | ### Example Usage ```tsx import { models } from 'react-native-executorch'; // Using an English (US) preset with the 'heart' voice const model = models.text_to_speech.kokoro.en_us.heart(); // Using a French preset with the 'siwis' voice // const model = models.text_to_speech.kokoro.fr.siwis(); ``` ``` -------------------------------- ### Generate Structured Output with LLM Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/llm.md Use `getStructuredOutputPrompt` to create a system prompt for LLMs to return JSON based on a Zod schema. `fixAndValidateStructuredOutput` validates the LLM's response against the schema. ```tsx import * as z from 'zod/v4'; import { useLLM, models, getStructuredOutputPrompt, fixAndValidateStructuredOutput, } from 'react-native-executorch'; const schema = z.object({ username: z.string().meta({ description: 'User asking the question' }), bid: z.number().meta({ description: 'Offer in the user message' }), currency: z.optional(z.string()), }); const llm = useLLM({ model: models.llm.qwen3_4b() }); useEffect(() => { const instructions = getStructuredOutputPrompt(schema); llm.configure({ chatConfig: { systemPrompt: `Parse the user's message and return JSON. Don't reply to the user. ${instructions} /no_think`, }, }); }, []); useEffect(() => { const last = llm.messageHistory.at(-1); if (!llm.isGenerating && last?.role === 'assistant') { try { const parsed = fixAndValidateStructuredOutput(last.content, schema); console.log(parsed); // typed by Zod } catch (e) { console.warn('Output did not match schema', e); } } }, [llm.messageHistory, llm.isGenerating]); ``` -------------------------------- ### Generate Text with LFM2.5 Model Source: https://github.com/software-mansion/react-native-executorch/blob/main/README.md Demonstrates how to use the initialized LLM to generate a chat completion based on a provided conversation history. The response is logged to the console. ```typescript const handleGenerate = async () => { const chat: Message[] = [ { role: 'system', content: 'You are a helpful assistant' }, { role: 'user', content: 'What is the meaning of life?' } ]; // Chat completion await llm.generate(chat); console.log('LFM2.5 says:', llm.response); }; ``` -------------------------------- ### useInstanceSegmentation Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for instance segmentation, distinguishing individual object instances within an image. ```APIDOC ## useInstanceSegmentation ### Description Performs instance segmentation, identifying and segmenting individual instances of objects within an image. ### Reference [vision.md](./references/vision.md) ``` -------------------------------- ### Directory Path Definitions Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Defines various project directory paths relative to the current CMakeLists.txt file, essential for locating dependencies and source files. ```cmake # tests/ <- CMAKE_SOURCE_DIR (this file's location) # rnexecutorch/ <- RNEXECUTORCH_DIR (parent of tests) # common/ <- COMMON_DIR # react-native-executorch/ <- PACKAGE_ROOT # / <- MONOREPO_ROOT # /third-party/ <- THIRD_PARTY_DIR set(RNEXECUTORCH_DIR "${CMAKE_SOURCE_DIR}/..") set(COMMON_DIR "${RNEXECUTORCH_DIR}/..") set(PACKAGE_ROOT "${COMMON_DIR}/..") set(MONOREPO_ROOT "${PACKAGE_ROOT}/../..") set(THIRD_PARTY_DIR "${MONOREPO_ROOT}/third-party") set(REACT_NATIVE_DIR "${MONOREPO_ROOT}/node_modules/react-native") set(ANDROID_THIRD_PARTY "${PACKAGE_ROOT}/third-party/android/libs/") set(TOKENIZERS_DIR "${PACKAGE_ROOT}/third-party/include/executorch/extension/llm/tokenizers/include") ``` -------------------------------- ### useTokenizer Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/SKILL.md Hook for tokenization compatible with HuggingFace tokenizers. ```APIDOC ## useTokenizer ### Description Provides tokenization functionality compatible with HuggingFace tokenizers. ### Reference [setup.md](./references/setup.md) ``` -------------------------------- ### Add Text-to-Image Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures the CMake build system to include tests for the Text-to-Image model. This target includes source files for the TextToImage model components and text embeddings, and depends on tokenizer libraries. ```cmake add_rn_test(TextToImageTests integration/TextToImageTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/text_to_image/TextToImage.cpp ${RNEXECUTORCH_DIR}/models/text_to_image/Encoder.cpp ${RNEXECUTORCH_DIR}/models/text_to_image/UNet.cpp ${RNEXECUTORCH_DIR}/models/text_to_image/Decoder.cpp ${RNEXECUTORCH_DIR}/models/text_to_image/Scheduler.cpp ${RNEXECUTORCH_DIR}/models/embeddings/text/TextEmbeddings.cpp ${TOKENIZER_SOURCES} LIBS tokenizers_deps ) ``` -------------------------------- ### Running Custom Models with useExecutorchModule Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Use this hook for custom `.pte` models not covered by dedicated hooks. Ensure your input tensor shapes exactly match the exported model. Preprocessing and postprocessing are your responsibility. ```tsx import { useExecutorchModule, ScalarType } from 'react-native-executorch'; const m = useExecutorchModule({ modelSource: require('../assets/custom_model.pte'), }); const run = async () => { const input = { dataPtr: new Float32Array([1.0, 2.0, 3.0]), sizes: [1, 3], scalarType: ScalarType.FLOAT, }; const output = await m.forward([input]); // output: TensorPtr[] — output[0].dataPtr is an ArrayBuffer; interpret per scalarType }; ``` -------------------------------- ### Add Text Embeddings Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures TextEmbeddingsTests with TextEmbeddings model and tokenizer sources, linking with tokenizers_deps. ```cmake add_rn_test(TextEmbeddingsTests integration/TextEmbeddingsTest.cpp SOURCES ${RNEXECUTORCH_DIR}/models/embeddings/text/TextEmbeddings.cpp ${TOKENIZER_SOURCES} LIBS tokenizers_deps ) ``` -------------------------------- ### Add Frame Processor Tests Source: https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/tests/CMakeLists.txt Configures FrameProcessorTests with FrameProcessor, FrameExtractor, image utilities, and Android-specific libraries. ```cmake add_rn_test(FrameProcessorTests unit/FrameProcessorTest.cpp SOURCES ${RNEXECUTORCH_DIR}/utils/FrameProcessor.cpp ${RNEXECUTORCH_DIR}/utils/FrameExtractor.cpp ${IMAGE_UTILS_SOURCES} LIBS opencv_deps android ) ``` -------------------------------- ### Load Models using the Models Registry Source: https://github.com/software-mansion/react-native-executorch/blob/main/skills/react-native-executorch/references/setup.md Use typed accessors from the 'models' registry for recommended model loading. Supports platform defaults, quantized variants, and specific backends. ```tsx import { useLLM, useObjectDetection, useOCR, models } from 'react-native-executorch'; useLLM({ model: models.llm.llama3_2_3b() }); // platform default, quantized useLLM({ model: models.llm.llama3_2_3b({ quant: false }) }); // full precision useObjectDetection({ model: models.object_detection.rf_detr_nano({ backend: 'xnnpack' }) }); useOCR({ model: models.ocr.craft({ language: 'en' }) }); ```