ffmpeg-framecraft – FFmpeg toolkit for 9:16 & short-form video

A Node.js-friendly FFmpeg toolkit for video processing: crop to 9:16, slice by time, add transitions between clips, subtitles, background music, and extract audio. Designed for vertical/short-form output (e.g. YouTube Shorts, TikTok, Reels) and general video pipelines.

Table of contents

• Prerequisites
• Installation
• Quick start
• Features
  • Crop to 9:16
  • Slice by timestamp
  • Multiple slices with transitions
  • Subtitles
  • Background music
  • Extract thumbnail
  • Extract audio only
• Pipelines (compose)
• Platform presets
• Transition presets
• Progress reporting
• API reference
• Advanced: filter builders
• Advanced AutoCrop-vertical integration
• License

Prerequisites

FFmpeg and ffprobe must be installed:

If the binaries are not in PATH, set:

• FFMPEG_PATH — path to ffmpeg
• FFPROBE_PATH — path to ffprobe

Installation

npm install ffmpeg-framecraft

Then:

const { FramecraftEngine } = require('ffmpeg-framecraft');

Quick start

const { FramecraftEngine } = require('ffmpeg-framecraft');

const engine = new FramecraftEngine();

// Crop a landscape video to vertical 9:16
await engine.cropTo916('input.mp4', 'shorts.mp4');

// Extract one segment
await engine.slice('input.mp4', 'clip.mp4', { start: 10, end: 30 });

// Combine several segments with a fade between them
await engine.slicesWithTransitions('input.mp4', 'highlight.mp4', {
  slices: [
    { start: 0, end: 10 },
    { start: 45, end: 55 },
    { start: 120, end: 130 },
  ],
  transition: 'fade',
});

Features

1. Crop to 9:16

Converts horizontal video to vertical (9:16) and scales to 720×1280 (baseline-friendly).

• Default (static): centered crop (fast, no extra dependencies).
• Smart (dynamic, Node): content-aware crop that tracks people, supports two-person shots, and biases framing toward the “speaking” person using a lightweight mouth-motion heuristic. No Python required, but does rely on optional JS AI deps.
• Python (AutoCrop-vertical, recommended for advanced): delegates to the original Autocrop-vertical Python script for scene-aware TRACK/LETTERBOX, VFR handling, per-frame processing, audio start-time compensation, and hardware encoding. Best quality and behavior for complex content (interviews, podcasts, multi-person scenes).

Example

await engine.cropTo916('landscape.mp4', 'vertical.mp4');

await engine.cropTo916('landscape.mp4', 'vertical.mp4', {
  onProgress: (p) => console.log(`${p.percent?.toFixed(1) ?? '-'}%`),
});

// Smart (dynamic) crop
await engine.cropTo916('landscape.mp4', 'vertical_smart.mp4', {
  smart: true,
  smartSampleEvery: 0.25,
});

// Smart crop tuned for interviews/podcasts (keep 2 people + speaker bias)
await engine.cropTo916('input.mp4', 'vertical_podcast.mp4', {
  smart: true,
  smartTwoShot: true,
  smartSpeakerBias: true,
  smartSampleEvery: 0.15,
});

// Python-backed AutoCrop-vertical (exact behavior, recommended for advanced use)
await engine.cropTo916AutoCropVertical('input.mp4', 'vertical_autocrop.mp4', {
  ratio: '9:16',       // same as --ratio
  quality: 'balanced', // 'fast' | 'balanced' | 'high'
  encoder: 'auto',     // 'auto' | 'hw' | specific encoder name
  // frameSkip: 0,
  // downscale: 0,
});

2. Slice by timestamp

Extracts a single segment from the video. Start/end can be seconds or time strings ("MM:SS", "HH:MM:SS").

Example

// Seconds
await engine.slice('input.mp4', 'clip.mp4', { start: 10, end: 30 });

// Time strings
await engine.slice('input.mp4', 'clip.mp4', { start: '0:10', end: '1:30' });
await engine.slice('input.mp4', 'clip.mp4', { start: '1:00:00', end: '1:05:00' });

3. Multiple slices with transitions

Trims several time ranges from one video and concatenates them with a transition (xfade + acrossfade). Progress is based on output duration.

Per-slice transition: Any slice (except the first) can override the transition:

{ start: 10, end: 20, transition: 'wipeleft' }  // only this boundary uses wipeleft

Example — same transition for all boundaries

await engine.slicesWithTransitions('input.mp4', 'highlight.mp4', {
  slices: [
    { start: 0, end: 8 },
    { start: 20, end: 28 },
    { start: 60, end: 68 },
  ],
  transition: 'fade',
});

Example — custom duration

await engine.slicesWithTransitions('input.mp4', 'out.mp4', {
  slices: [{ start: 0, end: 5 }, { start: 10, end: 15 }],
  transition: { type: 'dissolve', duration: 1.5 },
});

Example — platform preset (transition from preset)

await engine.slicesWithTransitions('input.mp4', 'out.mp4', {
  slices: [{ start: 0, end: 10 }, { start: 30, end: 40 }],
  preset: 'youtubeShort',  // uses preset's transition + duration
});

Example — per-slice transitions

await engine.slicesWithTransitions('input.mp4', 'out.mp4', {
  slices: [
    { start: 0, end: 9 },
    { start: 38, end: 62, transition: 'fadewhite', duration: 2 },
    { start: 172, end: 176.4, transition: 'dissolve', duration: 1 },
  ],
  transition: 'fade',  // default for boundaries that don't specify one
});

Slices are clamped to the file duration; invalid or zero-length ranges throw.

4. Subtitles

Burns an SRT file into the video. Optional ASS-style options for font/size/colour (useful for AI-generated captions).

Example

await engine.addSubtitles('input.mp4', 'with_subs.mp4', 'captions.srt');

await engine.addSubtitles('input.mp4', 'with_subs.mp4', 'captions.srt', {
  fontSize: 28,
  primaryColour: '&Hffffff',
  outline: 2,
  onProgress: (p) => console.log(p.percent?.toFixed(1) + '%'),
});

5. Background music

Mixes the video’s audio with a second audio file (e.g. music). If the video has no audio, only the music track is used.

Example

await engine.addBackgroundMusic('talk.mp4', 'with_music.mp4', 'background.mp3');

6. Extract thumbnail

Exports a single frame as an image. Format is taken from the file extension (.jpg, .png).

Example

await engine.extractThumbnail('input.mp4', 'frame.jpg', 5);
await engine.extractThumbnail('input.mp4', 'poster.png', '0:01:30');

7. Extract audio only

Re-encodes audio to AAC (.m4a) or MP3 (.mp3) based on the output extension. No video stream.

Example

await engine.extractAudioOnly('video.mp4', 'audio.m4a');
await engine.extractAudioOnly('video.mp4', 'audio.mp3');
await engine.extractAudioOnly('video.mp4', 'audio.mp3', { bitrate: 256 });

Pipelines (compose)

Run several operations in sequence with a single call. Intermediate files are written to the same directory as the final output and deleted after the pipeline finishes.

Preset: one named operation.

await engine.compose('input.mp4', 'output.mp4', 'shorts');
// Same as: cropTo916(input, output)

Pipeline: array of steps. Each step has an op and step-specific options.

Example — crop then slice

await engine.compose('input.mp4', 'output.mp4', [
  { op: 'crop916' },
  { op: 'slice', start: 0, end: 60 },
]);

Example — slice then extract audio only

await engine.compose('input.mp4', 'output.mp4', [
  { op: 'slice', start: 10, end: 20 },
  { op: 'audioOnly', outputPath: path.join(__dirname, 'data', 'clip_audio.mp3') },
], { onProgress: (p) => console.log(p.percent?.toFixed(1) + '%') });
// Only clip_audio.mp3 is kept; intermediate slice file is removed.

Example — full pipeline

await engine.compose('input.mp4', 'output.mp4', [
  { op: 'slicesWithTransitions', slices: [{ start: 0, end: 10 }, { start: 30, end: 40 }], preset: 'youtubeShort' },
  { op: 'subtitles', srtPath: 'subs.srt' },
  { op: 'music', musicPath: 'bgm.mp3' },
], { onProgress: (p) => console.log(p.percent?.toFixed(1) + '%') });

Platform presets

Presets define aspect ratio, resolution, transition, and codec hints. Use preset name (string) or the object from the package.

Example

const { youtubeShortPreset, getPreset } = require('ffmpeg-framecraft');

console.log(youtubeShortPreset.transition);  // 'fade'
console.log(getPreset('tiktok').aspectRatio); // '9:16'

await engine.slicesWithTransitions('input.mp4', 'out.mp4', {
  slices: [...],
  preset: 'youtubeShort',
});

Transition presets

Used by slicesWithTransitions. You can pass a preset name or { type, duration } (duration in seconds).

Available names:
fade, fadeLong, wipeleft, wiperight, wipeup, wipedown, slideleft, slideright, slideup, slidedown, circleopen, circleclose, rectcrop, distance, fadeblack, fadewhite, radial, dissolve, pixelize, zoomin, zoomout.

Example

const { TRANSITION_PRESETS, getTransition } = require('ffmpeg-framecraft');

// Preset name
transition: 'dissolve'

// Custom type + duration
transition: { type: 'fade', duration: 1 }

// Resolve preset to object
getTransition('fadeLong');  // { type: 'fade', duration: 1 }

Transition duration is automatically clamped so it does not exceed the length of either segment at that boundary.

Progress reporting

All operations that take opts.onProgress report:

• percent — 0–100, based on output duration when known (slice, slicesWithTransitions, cropTo916, addBackgroundMusic). Otherwise from FFmpeg input duration.
• timemark — current output time (e.g. "0:00:12.50").
• frames, currentFps, currentKbps when available.

On completion, the executor emits a final onProgress({ percent: 100 }) so the UI can show 100% before “Compose completed”.

API reference

Common opts: onProgress: (progress) => {}.

Advanced: filter builders

For tests or custom FFmpeg graphs you can use the low-level filter helpers.

const {
  cropTo916Filter,
  subtitleFilter,
  amixFilter,
  buildSlicesWithTransitionsFilter,
  SHORTS_WIDTH,
  SHORTS_HEIGHT,
} = require('ffmpeg-framecraft');

// Build -vf style filter strings (no execution)
cropTo916Filter(1920, 1080);
// => "crop=607:1080:656:0,scale=720:1280"

subtitleFilter('/path/to/subs.srt', { fontSize: 24 });
// => "subtitles='...':force_style='FontSize=24'"

amixFilter(true);
// => "[0:a][1:a]amix=inputs=2:duration=shortest[aout]"

// Slices + transitions: pass normalized slices and hasAudio
const slices = [
  { startSeconds: 0, endSeconds: 10, transition: null },
  { startSeconds: 20, endSeconds: 30, transition: { type: 'fade', duration: 0.5 } },
];
const { filterComplex, mapVideo, mapAudio } = buildSlicesWithTransitionsFilter(slices, true);
// Use filterComplex with -filter_complex and mapVideo/mapAudio with -map

Advanced AutoCrop-vertical integration

For the most advanced, scene-aware cropping (mirroring the behavior of Autocrop-vertical), ffmpeg-framecraft can delegate cropping to the Python tool.

• When to use this (recommended):

  • Multi-person scenes (interviews, podcasts, group shots).
  • You want the exact TRACK vs LETTERBOX behavior, scene detection, VFR normalization, and audio sync that AutoCrop-vertical provides.
  • You’re OK installing Python dependencies once on your machine or server.

• What it does:

  • Uses PySceneDetect to split the video into scenes.
  • For each scene, uses YOLOv8 + face detection to decide:
    • TRACK (tight crop on subject) vs
    • LETTERBOX (preserve full shot with black bars).
  • Processes frames with OpenCV and encodes via FFmpeg.
  • Normalizes VFR → CFR and compensates audio start-time offsets.

Recommended setup (one-time)

Run the built-in setup command from your project root:

npx ffmpeg-framecraft-setup-autocrop \
  --py-command python3.9 \
  --py-dir .

• Python requirement: The underlying Autocrop-vertical tool officially supports Python 3.8+. For best compatibility use a Python 3.8–3.11 interpreter when choosing --py-command (for example python3.9 or python3.10).
• --py-command: Python executable to use for the venv (e.g. python3, python3.9 or a full path).
• --py-dir: Base directory where autocrop-vertical/ should be cloned (default: current directory).

What this does:

• Clones Autocrop-vertical into <py-dir>/autocrop-vertical (if not already present).
• Creates a virtualenv in autocrop-vertical/.venv.
• Upgrades pip, pins numpy<2, and installs requirements.txt.
• Writes .autocrop-config.json in py-dir with:
  • pythonDir — path to the AutoCrop-vertical repo.
  • pythonCommand — path to the venv’s Python binary.

After this, you can call cropTo916AutoCropVertical without passing Python paths explicitly.

Using AutoCrop-vertical from Node

const { FramecraftEngine } = require('ffmpeg-framecraft');
const engine = new FramecraftEngine();

await engine.cropTo916AutoCropVertical('input.mp4', 'vertical_autocrop.mp4', {
  // All options are optional if you ran the setup script:
  // pythonDir: '/custom/path/to/autocrop-vertical',
  // pythonCommand: '/custom/path/to/python',
  ratio: '9:16',        // maps to --ratio
  quality: 'balanced',  // 'fast' | 'balanced' | 'high'
  encoder: 'auto',      // 'auto' | 'hw' | explicit encoder name
  // frameSkip: 0,
  // downscale: 0,
  // planOnly: true,
});

Resolution order for Python settings:

• opts.pythonDir / opts.pythonCommand (explicit in code)
• Environment:
  • AUTO_CROP_VERTICAL_DIR
  • AUTO_CROP_VERTICAL_PY
• .autocrop-config.json written by ffmpeg-framecraft-setup-autocrop

If pythonDir is still missing, the engine throws with a clear message suggesting running the setup command or setting environment variables.

License

ISC License

Copyright (c) 2025, the ffmpeg-framecraft contributors

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.