OpenFaceFX

https://openfacefx.com/docs
Admin
OpenFaceFX is an open-source lip-sync library and tool that turns audio recordings and transcripts...
Tokens:182,809
Snippets:869
Trust Score:3.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Quickstart CLI usage

Source: https://openfacefx.com/docs

Clone the repository, install dependencies, and run the naive aligner to generate a track file from text and audio.

```bash
git clone https://github.com/OpenFaceFX/OpenFaceFX && cd OpenFaceFX
pip install -e .        # numpy is the only runtime dependency
python -m openfacefx naive --text "hello world" --wav examples/voice.wav -o track.json
```

--------------------------------

### CLI output example

Source: https://openfacefx.com/docs

The console output confirming the successful generation of the track file.

```text
wrote track.json: 7 channels, 93 keyframes, 1.60s
```

--------------------------------

### Example .dict file format

Source: https://openfacefx.com/docs/api/i18n

A sample dictionary file header declaring the locale and alphabet, followed by word-to-phoneme mappings.

```text
;;; locale = xx-Toy
;;; alphabet = ipa
ka  k ɑ
ki  k i
```

--------------------------------

### Install and build commands for FaceFXWrapper.exe

Source: https://openfacefx.com/docs/facefxwrapper

Build the Windows PE executable with PyInstaller. Must be run on Windows or cross-built for the Wine target. Output is dist/FaceFXWrapper.exe.

```bash
pip install -e .            # or: pip install openfacefx
pip install pyinstaller
pyinstaller --onefile --console --name FaceFXWrapper facefxwrapper_main.py
# -> dist/FaceFXWrapper.exe  (build on Windows, or cross-build for the Wine target)

```

--------------------------------

### StreamingGenerator usage example

Source: https://openfacefx.com/docs/api/streaming

Demonstrates how to initialize the StreamingGenerator, push phoneme chunks, and convert the resulting frames into a FaceTrack.

```python
from openfacefx import StreamingGenerator, frames_to_track

gen = StreamingGenerator(fps=60.0, look_ahead=0.5)   # look_ahead = latency dial
frames = []
for chunk in phoneme_segment_chunks:                 # e.g. from a live aligner
    frames += gen.push(chunk)
frames += gen.flush()
track = frames_to_track(frames, 60.0)                # -> a normal FaceTrack
```

--------------------------------

### Run diff command

Source: https://openfacefx.com/docs/api/inspect

Examples of using the diff command for exact matching, tolerance-based comparison, and JSON report generation.

```bash
python -m openfacefx diff golden.track.json candidate.track.json            # exact-match gate
python -m openfacefx diff golden.track.json candidate.track.json --tolerance 0.002
python -m openfacefx diff a.track.json b.track.json --json > drift.json
```

--------------------------------

### Sequence tracks via CLI

Source: https://openfacefx.com/docs/api/transforms

Examples of using the sequence command to stitch tracks together with optional gaps or crossfades.

```bash
python -m openfacefx sequence line1.json line2.json line3.json -o scene.json
python -m openfacefx sequence a.json b.json --gap 0.5 -o with_beat.json     # silence between
python -m openfacefx sequence a.json b.json --crossfade 0.15 -o blended.json  # soft seam
```

--------------------------------

### Generate and bake emotion via CLI

Source: https://openfacefx.com/docs/api/emotion

Example workflow showing how to generate a base speech track and then apply an emotion layer using the emotion command.

```bash
# generate a speech track, then bake emotion over it
python -m openfacefx naive --text "..." --duration 1.5 -o base.json
python -m openfacefx emotion base.json happy.emotion.json --intensity 1.0 -o baked.json
# --intensity 0 (or a neutral envelope) => baked.json is byte-identical to base.json
```

--------------------------------

### Run openfacefx from-timing for all vendor timing formats

Source: https://openfacefx.com/docs/timing

Invoke the from-timing adapter for each supported vendor format. Only start times are guaranteed; start-only sources (Azure, Polly) derive each event's end from the next event's start, and the final event is held for --final-duration seconds (default 0.08). The pho, piper, and cartesia formats auto-select a built-in IPA preset when no --mapping is given; an explicit --mapping still wins. The voicevox format also covers API-compatible forks COEIROINK, SHAREVOX, LMROID, and AivisSpeech.

```bash
openfacefx from-timing --file voice.pho    --format pho      -o track.json
openfacefx from-timing --file align.json   --format piper --sample-rate 22050 -o track.json
openfacefx from-timing --file marks.json   --format cartesia -o track.json
openfacefx from-timing --file visemes.json --format azure    -o track.json
openfacefx from-timing --file voice.marks  --format polly     -o track.json --retarget arkit
openfacefx from-timing --file query.json   --format voicevox  -o track.json   # JP TTS
```

--------------------------------

### Capture espeak-ng phoneme events with C API

Source: https://openfacefx.com/docs/timing

C API example that subscribes to espeakEVENT_PHONEME events to capture per-phoneme start times. The callback prints the start position in milliseconds and the phoneme name. These are start-only events, so they must be fed through the schema and let resolve_ends infer durations.

```c
/* link: espeak-ng.  cc capture.c -lespeak-ng */
#include <espeak-ng/speak_lib.h>
#include <stdio.h>
#include <string.h>
static int cb(short *wav, int n, espeak_EVENT *ev) {
    for (; ev->type != espeakEVENT_LIST_TERMINATED; ev++)
        if (ev->type == espeakEVENT_PHONEME)      /* start ms, phoneme name */
            printf("%d\t%s\n", ev->audio_position, ev->id.string);
    return 0;
}
int main(void) {
    espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, NULL, 0);
    espeak_SetSynthCallback(cb);
    const char *t = "hello world";
    espeak_Synth(t, strlen(t) + 1, 0, POS_CHARACTER, 0,
                 espeakCHARS_AUTO | espeakPHONEMES, NULL, NULL);
    return espeak_Synchronize();
}

```

--------------------------------

### Generate LOD variants via CLI

Source: https://openfacefx.com/docs/api/lod

Commands to generate tiered animation variants from a track file. The first example uses default settings, while subsequent examples demonstrate custom RDP tolerances and frame rates.

```bash
python -m openfacefx lod clip.track.json -o out/clip                    # default 3 tiers
python -m openfacefx lod clip.track.json --rdp 0.002,0.01,0.04 --fps 60,30,15 -o out/clip
python -m openfacefx lod clip.track.json --rdp 0.005,0.02 --fps 60,60 --format csv -o out/clip
```

--------------------------------

### word_timings function

Source: https://openfacefx.com/docs/api/captions

Generates per-word timing information (token, start, end) based on naive alignment. It preserves punctuation in tokens and handles hyphenated words by spanning from the first part's start to the last part's end.

```python
def word_timings(text: str, duration: float, g2p=None) -> List[WordTiming]:
    """Per-word ``(token, start, end)`` from the same naive alignment the lip
    curves use. Display tokens keep their punctuation (so cue grouping can see
    sentence ends); each is paired to its ``[A-Za-z']+`` word span(s) — a
    hyphenated token spans from its first part's start to its last part's end,
    and a pure-punctuation token (no word) contributes no timing."""
    from .g2p import G2P
    from .texttags import WORD_RE, naive_word_segments
    _, spans = naive_word_segments(text, duration, g2p or G2P())
    out: List[WordTiming] = []
    idx = 0
    for token in text.split():
        n = len(WORD_RE.findall(token))
        if n == 0:
            continue
        out.append((token, spans[idx][0], spans[idx + n - 1][1]))
        idx += n
    return out
```

--------------------------------

### Loading a dictionary and generating segments

Source: https://openfacefx.com/docs/api/i18n

Demonstrates initializing the G2P engine, loading a custom dictionary, and generating segments using the naive path.

```python
g = G2P()
g.load_dictionary("toy.dict")     # adds mapped entries; first spelling wins
naive_segments("ka ki", 2.0, g2p=g)   # a normal track, via the naive path
```

--------------------------------

### wav_duration

Source: https://openfacefx.com/docs/api/pipeline

Utility function to get the duration of a WAV file.

```APIDOC
## `wav_duration(path: str) -> float`

### Description
Returns the duration of a PCM WAV file in seconds, using only the standard library.

### Method
Function call

### Parameters
- **path** (str) - Required - Path to the WAV file.

### Returns
- **float** - Duration in seconds.

### Example
```python
from openfacefx.pipeline import wav_duration

duration = wav_duration("audio.wav")
```
```

--------------------------------

### main(argv=None) entry point

Source: https://openfacefx.com/docs/api/batch

The main function intercepts arguments for the FaceFXWrapper shim before proceeding to standard argparse initialization.

```python
def main(argv=None) -> int:
    # FaceFXWrapper.exe drop-in shim (issue #33): intercept BEFORE argparse so the
    # raw positional args pass through verbatim. A real consumer command carries
    # values argparse would choke on — a leading-dash token read as an option, or
    # a resampled-WAV/text path — so the whole tail goes straight to the shim,
    # dispatched on the literal 'facefxwrapper' token.
    raw = sys.argv[1:] if argv is None else list(argv)
    if raw and raw[0] == "facefxwrapper":
        return facefxwrapper.run(raw[1:])

    p = argparse.ArgumentParser(prog="openfacefx")
    sub = p.add_subparsers(dest="cmd", required=True)
```

--------------------------------

### Import mouth-cue files using from-cues

Source: https://openfacefx.com/docs/api/importers

Use the from-cues command to import various mouth-cue file formats. The format is auto-detected by extension and the first line of the file.

```bash
python -m openfacefx from-cues mouth.tsv  -o track.json      # Rhubarb TSV
python -m openfacefx from-cues mouth.xml  -o track.anim      # Rhubarb XML -> Unity
python -m openfacefx from-cues mouth.dat  --fps 24 -o track.json   # Moho / OpenToonz
python -m openfacefx from-cues mouth.pgo  --coarticulate -o track.json  # Papagayo
```

--------------------------------

### dominant_cues function

Source: https://openfacefx.com/docs/api/exporters

Flattens a FaceTrack into (start, end, shape) runs by selecting the dominant channel at each frame.

```python
def dominant_cues(track: FaceTrack, rest_name: str = "X",
                  silence_floor: float = _SILENCE_FLOOR) -> List[Cue]:
    """Flatten ``track`` to ``(start, end, shape)`` runs by dominant channel.

    At each frame (sampled at the track's own fps) the single highest-weight
    channel wins; a frame with nothing above ``silence_floor`` becomes
    ``rest_name``. Adjacent equal-shape frames merge into one run, and the runs
    tile ``[0, duration]`` with no gaps.
    """
    fps = track.fps or 1.0
    dur = track.duration
    samplers = [(c.name, _sampler(c)) for c in track.channels]
    n = int(round(dur * fps))
    labels: List[Tuple[float, str]] = []
    for i in range(n + 1):
        t = i / fps
        best_name, best_w = rest_name, silence_floor
        for name, sample in samplers:
            w = sample(t)
            if w > best_w:
                best_name, best_w = name, w
        labels.append((t, best_name))
    if not labels:
        return []
    cues: List[Cue] = []
    start, current = labels[0]
    for t, name in labels[1:]:
        if name != current:
            cues.append((start, min(t, dur), current))
            start, current = t, name
    if start < dur:  # a lone differing frame exactly at the end is negligible
        cues.append((start, dur, current))
    return cues
```

--------------------------------

### looks_like_ssml function

Source: https://openfacefx.com/docs/api/ssml

Checks if the provided text starts with a <speak> root element, enabling auto-detection of SSML input.

```python
def looks_like_ssml(text: str) -> bool:
    """True if ``text`` opens with a ``<speak>`` root -- the auto-detect the naive
    command uses to enable SSML parsing without an explicit ``--ssml`` flag."""
    return bool(text) and bool(_SPEAK_RE.match(text))
```

--------------------------------

### main(argv=None) command handling

Source: https://openfacefx.com/docs/api/batch

The main entry point handles various CLI commands including lod, export-layers, diff, and naive processing.

```python
    if args.cmd == "lod":
        import os
        from .io_export import read_json, write_csv, write_json
        from .lod import generate_lods, lod_metadata
        source_ranking = None
        try:
            track = read_json(args.infile)
            variants, levels = generate_lods(
                track, rdp=_parse_floats(args.rdp, "--rdp"),
                fps=_parse_floats(args.fps, "--fps"))
            budgets = _parse_ints(args.max_channels, "--max-channels")
            if budgets is not None:
                if len(budgets) != len(variants):
                    raise ValueError(
                        f"--max-channels must list one budget per tier "
                        f"({len(variants)}), got {len(budgets)}")
                # rank the SOURCE once so the kept channel sets nest across LODs
                # (pose channels pass through every tier; issue #37)
                from .budget import keep_top_weight, rank_channels
                source_ranking = rank_channels(track)
                variants = [keep_top_weight(v, source_ranking, n)
                            for v, n in zip(variants, budgets)]
        except (OSError, ValueError) as ex:
            raise SystemExit(f"lod: {ex}")
        prefix, ext = args.out, args.format
        outdir = os.path.dirname(prefix)
        if outdir:
            os.makedirs(outdir, exist_ok=True)
        writer = write_csv if ext == "csv" else write_json
        files = []
        for i, v in enumerate(variants):
            path = f"{prefix}_lod{i}.{ext}"
            writer(v, path)
            files.append(os.path.basename(path))
        meta = lod_metadata(track, levels, variants, files)
        if source_ranking is not None:                # issue #37: per-tier budget
            meta["ranking"] = source_ranking
            for i, entry in enumerate(meta["levels"]):
                entry["max_channels"] = budgets[i]
        meta_path = f"{prefix}_lod.json"
        with open(meta_path, "w", encoding="utf-8") as fh:
            json.dump(meta, fh, indent=2)
            fh.write("\n")
        _say(args, f"wrote {len(variants)} LOD variants + {meta_path}: " +
             ", ".join(f"lod{m['index']}={m['keyframes']}kf@{m['fps']}fps"
                       for m in meta["levels"]))
        return 0

    if args.cmd == "export-layers":
        from .io_export import read_json, to_dict
        from .layers import build_layers
        try:
            track = read_json(args.infile)
        except (OSError, ValueError) as ex:
            raise SystemExit(f"export-layers: {ex}")
        layers = build_layers(track)
        with open(args.out, "w", encoding="utf-8") as fh:
            json.dump(to_dict(track, layers=layers), fh, indent=2)
        _say(args, f"wrote {args.out}: flat track + {len(layers)} layer(s) "
             f"({', '.join(l.name for l in layers)})")
        return 0

    if args.cmd == "diff":
        from .io_export import read_json
        from .trackdiff import diff_tracks, render_diff
        try:
            a = read_json(args.a)
            b = read_json(args.b)
        except (OSError, ValueError) as ex:
            raise SystemExit(f"diff: {ex}")
        report = diff_tracks(a, b, tolerance=args.tolerance)
        print(json.dumps(report) if args.json else render_diff(report))
        return 0 if report["ok"] else 1

    mapping = Mapping.from_json(args.mapping) if args.mapping else None
    params = (_coart_params(args)
              if args.cmd in ("naive", "mfa", "from-timing") else None)

    if args.cmd == "naive":
        if bool(args.anchors) != bool(args.anchors_format):
            raise SystemExit(
                "--anchors and --anchors-format are only valid together")
        dur = (args.duration if args.duration
               else wav_duration(args.wav) if args.wav else None)
        if dur is None and args.anchors_format not in _PHONE_SEGMENT_FORMATS:
            raise SystemExit(
                "naive: one of --wav / --duration is required (only the phone-timing "
                f"formats {'/'.join(_PHONE_SEGMENT_FORMATS)} carry their own timeline)")
        subs = []
        if args.text and not args.no_normalize:
            args.text, subs = normalize_transcript(args.text)
        g2p = G2P()
        if args.cmudict:
            added = g2p.load_cmudict(args.cmudict)
            _say(args, f"loaded {added} CMUdict entries")
        from .texttags import has_tags
        from .ssml import looks_like_ssml, parse_ssml
        # SSML (issue #52) is a thin front-end over the #7 tag path: parse it to
        # the same (clean_text, tags) pair and route it through _naive_tagged.
        # Explicit --ssml or an auto-detected <speak> root; empty tags still land
        # on the byte-identical plain path.
        ssml_clean = ssml_tags = None
        if getattr(args, "ssml", False) or (bool(args.text)
                                            and looks_like_ssml(args.text)):
```

--------------------------------

### CLI usage for open-source aligner adapters

Source: https://openfacefx.com/docs/api/aligners

Shows how to use the CLI to generate timings from aligner output. For word timings, no --text is needed because the aligner supplies the words. For Gentle's phoneme timings, use --anchors-format gentle-phones to get the accurate path.

```bash
# word timings — the aligner supplies the words, so no --text needed
python -m openfacefx naive --anchors words.json --anchors-format whisper -o out.json
python -m openfacefx naive --anchors align.json --anchors-format whisperx --wav vo.wav -o out.json
# Gentle's free phoneme timings — the accurate path, straight to generate_from_alignment
python -m openfacefx naive --anchors gentle.json --anchors-format gentle-phones -o out.json
```

--------------------------------

### Run naive pipeline with SSML input

Source: https://openfacefx.com/docs/api/ssml

Enable SSML parsing with the `--ssml` flag, or pass a document with a `<speak>` root for auto-detection. The command runs the naive pipeline with a 3-second duration and writes to `out.track.json`.

```bash
python -m openfacefx naive --ssml --duration 3 -o out.track.json \
  --text '<speak>Say <emphasis level="strong">brave</emphasis> <break time="300ms"/> new world <mark name="hit"/></speak>'
```

--------------------------------

### looks_like_ssml(text: str) -> bool

Source: https://openfacefx.com/docs/api/ssml

Checks if a string starts with a `<speak>` root element, used for auto-detection of SSML input.

```APIDOC
## `looks_like_ssml(text: str) -> bool`

### Description
Determines whether a given string starts with a `<speak>` root element, enabling automatic SSML parsing without an explicit `--ssml` flag.

### Method
Function call

### Endpoint
`looks_like_ssml(text: str) -> bool`

### Parameters
#### Function Parameters
- **text** (str) - Required - The input string to check for SSML root.

### Returns
- **bool** - True if the text opens with a `<speak>` root, otherwise False.

### Example
```python
looks_like_ssml("<speak>hello</speak>")  # True
```
```

--------------------------------

### Example Mapping Structure

Source: https://openfacefx.com/docs/api/retarget

A mapping dictionary defines how each Oculus-15 viseme maps to one or more target shapes with specific weight scales.

```json
{"PP": [("mouthClose", 0.9), ("mouthPressLeft", 0.35)], ...}
```

--------------------------------

### parse_lip_header function

Source: https://openfacefx.com/docs/api/exporters

Parses the 12-byte header at the start of a modern .lip file. Raises a ValueError if the provided data is shorter than 12 bytes.

```python
def parse_lip_header(data: bytes) -> LipHeader:
    """Parse the 12-byte header at the start of a modern .lip file."""
    if len(data) < 12:
        raise ValueError(f"lip data too short for header: {len(data)} bytes")
    version, size, flags = struct.unpack_from("<iii", data)
    return LipHeader(version, size, flags)
```

--------------------------------

### Import blendshape-weight CSV files

Source: https://openfacefx.com/docs/api/importers

Use the from-csv command to import CSV files. The command supports different layouts, FPS settings, and timecode columns.

```bash
python -m openfacefx from-csv track.csv -o out.json                 # OpenFaceFX long
python -m openfacefx from-csv capture.csv --fps 60 -o out.anim       # wide, Live Link Face
python -m openfacefx from-csv capture.csv --timecode-col Timecode -o out.json
```

--------------------------------

### main(argv=None) -> int implementation

Source: https://openfacefx.com/docs/api/batch

The main entry point for the batch processing command-line interface, handling the 'energy' command and associated track generation logic.

```python
        _write(track, args.out, args)
        _emit_summary(args, track, segments=segs)
    elif args.cmd == "energy":
        from .energy import generate_from_energy
        track = generate_from_energy(args.wav, fps=args.fps,
                                     intensity=args.intensity, mapping=mapping,
                                     gestures=_gesture_params(args),
                                     smooth=_smooth_seconds(args),
                                     lag=_lag_seconds(args))
        track = _apply_edits_layer(track, args)
        et = ev = None
        if getattr(args, "events", False):
            from .energy import energy_envelope
            et, ev = energy_envelope(args.wav, fps=args.fps)
        _event_layer(track, args, env_times=et, env=ev)
        _write(track, args.out, args)
        _emit_summary(args, track)
    return 0
```

--------------------------------

### Transform track CLI commands

Source: https://openfacefx.com/docs/api/transforms

Examples of using the transform command to modify track files, including scaling duration, fitting to audio, mirroring, and trimming.

```bash
python -m openfacefx transform track.json --retime 1.5 -o slow.json         # 1.5x slower
python -m openfacefx transform track.json --duration 3.2 -o fit.json         # to 3.2 s
python -m openfacefx transform track.json --wav newvo.wav -o redub.json      # to a WAV length
python -m openfacefx transform track.json --mirror -o flipped.json           # L/R mirror
python -m openfacefx transform track.json --trim 0.5 2.0 -o slice.json        # keep [0.5, 2.0]
```

--------------------------------

### Import and use openfacefx library API

Source: https://openfacefx.com/docs/timing

Shows the core workflow: parse Azure viseme JSON, resolve event ends, convert to segments with warnings, and generate a track using the vendor mapping. The warnings are printed with a "QA:" prefix.

```python
from openfacefx import (parse_azure_visemes, resolve_ends,
                        viseme_events_to_segments, build_vendor_mapping,
                        AZURE_VISEME_TO_TARGET, generate_from_alignment)

events = resolve_ends(parse_azure_visemes(open("visemes.json").read()))
segs, warnings = viseme_events_to_segments(events, AZURE_VISEME_TO_TARGET)
for w in warnings:
    print("QA:", w)
track = generate_from_alignment(segs, mapping=build_vendor_mapping(AZURE_VISEME_TO_TARGET))

```

--------------------------------

### Co-generate track and captions from one run

Source: https://openfacefx.com/docs/api/captions

Runs naive mode to produce a track JSON and its captions in one command.

```bash
python -m openfacefx naive --text "Well met." --wav vo.wav -o vo.track.json --emit-captions vo.srt
```

--------------------------------

### from_whisperx(json_text: str) -> List[Anchor]

Source: https://openfacefx.com/docs/api/aligners

Converts WhisperX alignment output into word anchors. Words without a start timestamp are dropped, following the same deterministic rule as from_whisper_json.

```APIDOC
## from_whisperx(json_text: str) -> List[Anchor]

### Description
Converts WhisperX alignment output (segments[].words[] with word/start/end/score) into word anchors. Words without a start timestamp are dropped, following the same deterministic rule as from_whisper_json.

### Method
Function call

### Endpoint
`from_whisperx(json_text: str) -> List[Anchor]`

### Parameters
#### Path Parameters
- **json_text** (str) - Required - The JSON string containing WhisperX alignment output.

### Request Example
```json
{
  "segments": [
    {
      "words": [
        {"word": "hello", "start": 0.0, "end": 0.5, "score": 0.9}
      ]
    }
  ]
}
```

### Response
#### Success Response (200)
- **List[Anchor]** - A list of word anchors extracted from the input.

#### Response Example
```json
[
  {
    "word": "hello",
    "start": 0.0,
    "end": 0.5,
    "score": 0.9
  }
]
```
```
Tokens:
Raw