Creates a new directive that defines which Robots workflows should run against Mux assets, and under what resource conditions.
Human-readable directive name.
The entity type each run targets. V1 accepts "video.asset" only; "video.live_stream" is reserved and rejected at save time.
Resources the engine ensures on the asset before dependent workflows run. May be omitted for workflows-only directives.
Customer-chosen identifier that must be unique within the directive. Workflow inputs reference this identifier.
A caption or audio track on the asset.
The track flavor to match: "caption" for text tracks, "audio" for audio tracks.
BCP 47 language code the track must match (e.g. "en"). When omitted, any language of the declared kind matches.
How the resource is materialized on the asset. Defaults to { "via": "external" }.
The resource is produced out-of-band (e.g. a track uploaded directly to the asset). The engine waits for it to appear and become ready.
The resource must already be ready on the asset when the run starts. If it is absent, the dependency workflow fails immediately.
The resource is produced by another workflow binding in this directive. Dependent workflows wait for that binding to complete and the resource to become ready.
The reference_id of the workflow binding in this directive that produces the resource.
The engine creates the resource by calling the Mux Video API when the resource is absent at run start.
Generate subtitles from the asset's audio using Mux auto-generated captions.
The Mux Video API call the engine fires to create the resource.
Parameters for the generate_subtitles engine action.
BCP 47 language code for the generated subtitles (e.g. "en"). Defaults to "auto", which detects the language from the audio.
BCP 47 language code of the audio track to transcribe. When omitted, the first audio track on the asset is used.
Ingest a track file onto the asset from a URL. Text tracks are ingested as subtitles.
The Mux Video API call the engine fires to create the resource.
Parameters for the create_track engine action.
Publicly reachable URL of the track file to ingest.
BCP 47 language code of the track (e.g. "en").
The kind of track to create. "text" tracks are ingested as subtitles.
Request the asset's shot-detection output.
The Mux Video API call the engine fires to create the resource.
Parameters for the request_shots engine action.
How shot images are derived. Defaults to "create_storyboards".
Customer-chosen identifier, unique within the directive. Workflow inputs entries reference this handle.
The asset's shot-detection output.
How the resource is materialized on the asset. Defaults to { "via": "external" }.
The resource is produced out-of-band (e.g. a track uploaded directly to the asset). The engine waits for it to appear and become ready.
The resource must already be ready on the asset when the run starts. If it is absent, the dependency workflow fails immediately.
The resource is produced by another workflow binding in this directive. Dependent workflows wait for that binding to complete and the resource to become ready.
The reference_id of the workflow binding in this directive that produces the resource.
The engine creates the resource by calling the Mux Video API when the resource is absent at run start.
Generate subtitles from the asset's audio using Mux auto-generated captions.
The Mux Video API call the engine fires to create the resource.
Parameters for the generate_subtitles engine action.
BCP 47 language code for the generated subtitles (e.g. "en"). Defaults to "auto", which detects the language from the audio.
BCP 47 language code of the audio track to transcribe. When omitted, the first audio track on the asset is used.
Ingest a track file onto the asset from a URL. Text tracks are ingested as subtitles.
The Mux Video API call the engine fires to create the resource.
Parameters for the create_track engine action.
Publicly reachable URL of the track file to ingest.
BCP 47 language code of the track (e.g. "en").
The kind of track to create. "text" tracks are ingested as subtitles.
Request the asset's shot-detection output.
The Mux Video API call the engine fires to create the resource.
Parameters for the request_shots engine action.
How shot images are derived. Defaults to "create_storyboards".
The Robots workflows to dispatch on each run.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the summarize workflow — the same fields accepted by POST /jobs/summarize, minus engine-supplied fields such as asset_id.
Tone for the generated summary. "neutral" for straightforward analysis, "playful" for witty and conversational, "professional" for executive-level reporting.
Curated output_steering controls for execution scope, summary style, audience, brand terminology, and tag taxonomy. Scope is enforced; other controls guide model behavior but do not guarantee exact output.
Optional execution window in seconds on the original asset timeline. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. The summary and tags are generated only from media within this window.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Controlled vocabulary for tag generation. This steers tags and may be deterministically filtered after generation. Supports up to 50 values and 2000 serialized characters.
Optional customer-facing name for the taxonomy, up to 100 characters.
Controlled vocabulary values for generated tags. Supports 1-50 values.
Canonical tag value to prefer in generated tags, up to 100 characters.
Short explanation of when this tag applies, up to 300 characters.
Accepted alternate names that should normalize to the canonical label. Up to 10 aliases, each up to 100 characters.
When false, generated tags are filtered to taxonomy labels and aliases. When true, unmatched tags may remain.
Best-effort style guidance for the generated title and description.
Intended audience used as best-effort model guidance. Does not change the output schema.
Preferred brand or domain terms to use when supported by the source content.
Legacy/internal prompt-section overrides. Prefer output_steering for new integrations.
Override the core task instruction for summarization.
Override the title generation requirements.
Override the description generation requirements.
Override the keyword/tag extraction requirements.
Override the quality standards for analysis.
Maximum title length in words.
Maximum description length in words.
Maximum number of tags to include in the generated output. Defaults to 10.
BCP 47 language code of the caption track to analyze (e.g. "en", "fr"). When omitted, the SDK uses the default track.
BCP 47 language code for the generated summary output (e.g. "en", "fr", "ja"). Auto-detected from the transcript if omitted.
When true, the generated title is written to the Mux asset's metadata (asset.meta.title) once the summary completes. Best-effort: a metadata-write failure does not fail the summary.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the moderate workflow — the same fields accepted by POST /jobs/moderate, minus engine-supplied fields such as asset_id.
BCP 47 language code for transcript analysis. Used only for audio-only assets; ignored for video assets with visual content. If omitted for audio-only assets, the first ready text track is used. Defaults to "en".
Score thresholds that determine whether content is flagged. When combined with sampling_interval or max_samples, the exceeds_threshold flag reflects whether any category's highest observed score exceeds its configured threshold. Defaults to {sexual: 0.7, violence: 0.8}.
Score threshold for sexual content. Content scoring above this value triggers exceeds_threshold.
Score threshold for violent content. Content scoring above this value triggers exceeds_threshold.
Interval, in seconds, between sampled thumbnails. Minimum 5 seconds. When max_samples is also set, the actual sampling density is the more restrictive of the two constraints.
Maximum number of thumbnails to sample. Acts as a cap — if sampling_interval produces fewer samples than this limit, the interval is respected; otherwise samples are evenly distributed with first and last frames pinned.
Curated controls that optionally restrict moderation to an asset time range.
Optional execution window in seconds on the original asset timeline. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. Moderation is restricted to media within this window, while returned timestamps remain absolute asset timestamps.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Optional action taken automatically when exceeds_threshold is true. When omitted, the job only reports scores and takes no action. Note: in a directive run that targets the same asset with other workflows, deleting playback IDs can break sibling workflows that need a playback ID.
Action to take when exceeds_threshold is true. "delete_playback_ids" deletes every playback ID on the asset, making it unplayable while preserving the underlying asset so it can be re-published or reviewed.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the generate-chapters workflow — the same fields accepted by POST /jobs/generate-chapters, minus engine-supplied fields such as asset_id.
BCP 47 language code of the caption track to analyze (e.g. "en", "fr"). When omitted, the SDK prefers English if available.
BCP 47 language code for the output chapter titles. Auto-detected from the transcript if omitted.
Curated output_steering controls for chapter style, granularity, audience, and brand terminology. These controls guide model behavior but do not guarantee exact output.
Best-effort style guidance for generated chapter titles.
Best-effort guidance for how coarse or fine chapter boundaries should be.
Intended audience used as best-effort model guidance. Does not change the output schema.
Preferred brand or domain terms to use when supported by the source content.
Legacy/internal prompt-section overrides. Prefer output_steering for new integrations.
Override the core task instruction for chapter generation.
Override the JSON output format instructions.
Override the chapter density and timing constraints.
Override the chapter title style requirements.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the find-scenes workflow — the same fields accepted by POST /jobs/find-scenes, minus engine-supplied fields such as asset_id.
Preferred transcript language code to analyze when a matching transcript track is available. Defaults to the first ready transcript track on the asset.
Optional lower-bound hint for scene segmentation. When provided, the model should avoid collapsing clearly distinct beats below this count when the content supports more granular scene boundaries.
Preferred minimum scene duration in milliseconds for scaffolded transcript chunking. Defaults to 15000.
Curated output_steering controls for execution scope, segmentation strategy, title style, narration detail, audience, brand terms, and topic taxonomy. Scope is enforced; other controls guide model behavior but do not guarantee exact output.
Optional execution window in seconds on the original asset timeline. The range must contain a video frame at the asset frame rate. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. Returned scene timestamps remain absolute asset timestamps.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Best-effort guidance for the kinds of boundaries the scene finder should prefer.
Best-effort title style guidance for generated scene titles.
Best-effort guidance for how much detail scene narratives should include.
Intended audience used as best-effort model guidance. Does not change the output schema.
Preferred brand or domain terms to use when supported by source evidence.
Controlled vocabulary used to steer scene concepts without changing the response schema.
Optional customer-facing name for the taxonomy.
Controlled vocabulary values used as best-effort model guidance.
Canonical taxonomy value to prefer when supported by the source content.
Short explanation of when this value applies.
Accepted alternate names that should normalize to the canonical label.
When false, the model should prefer values from the taxonomy. When true, non-taxonomy values may be used when no taxonomy value applies.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the edit-captions workflow — the same fields accepted by POST /jobs/edit-captions, minus engine-supplied fields such as asset_id.
Optional LLM-driven profanity detection and censorship rules applied to the selected caption track.
How profanity is detected. Currently only llm is supported, which uses an LLM to identify profanity in cue text.
Replacement strategy for detected profanity: blank inserts bracketed underscores, remove drops the match, and mask replaces characters with question marks. Defaults to "blank".
Additional words or short phrases that should always be censored even if the model does not detect them.
Words or short phrases that should never be censored even if the model flags them.
Optional static word or phrase replacements applied directly to cue text.
Exact word or phrase to replace in cue text.
Replacement text to insert when a match is found.
When true, find is matched only with exact case. Defaults to false (case-insensitive matching), so "gonna" also matches "Gonna" and "GONNA".
Whether to upload the edited VTT back to the Mux asset as a new text track. Defaults to true.
Whether to delete the original source text track after the edited track upload succeeds. Has effect only when upload_to_mux is true. Defaults to true.
Optional suffix appended to the uploaded replacement track name. Defaults to "edited".
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the translate-captions workflow — the same fields accepted by POST /jobs/translate-captions, minus engine-supplied fields such as asset_id.
BCP 47 language code for the translated output (e.g. "es", "ja"). The asset must not already have a text track for this language.
Whether to upload the translated VTT and attach it as a text track on the Mux asset. Defaults to true.
Best-effort list of terms (brand names, proper nouns) to preserve verbatim in the translated captions. Does not guarantee exact output. Terms must not contain '<' or '>', invisible characters, or characters altered by Unicode normalization.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the translate-audio workflow — the same fields accepted by POST /jobs/translate-audio, minus engine-supplied fields such as asset_id.
BCP 47 language code for the target translation language (e.g. "es", "fr", "ja").
Whether to automatically upload the translated audio track to the Mux asset. Defaults to true.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the ask-questions workflow — the same fields accepted by POST /jobs/ask-questions, minus engine-supplied fields such as asset_id.
One or more questions to ask about the video. Each question can either select from answer_options (defaults to yes/no) or, by setting free_form_reply: true, receive a free-form prose answer.
The question to ask about the video content. Maximum 600 characters.
Allowed answer values for this question. Defaults to ["yes", "no"] when omitted and free_form_reply is not true. Mutually exclusive with free_form_reply. Each option is a short label of at most 150 characters.
Experimental. When true, the model replies with free-form prose instead of selecting from answer_options. Mutually exclusive with answer_options. Treat the answer as untrusted model output.
BCP 47 language code of the caption track to analyze (e.g. "en", "fr"). When omitted, the SDK uses the default track.
Experimental. Max character length for free-form answers, between 1 and 1000. Ignored unless at least one question sets free_form_reply: true.
Curated controls that optionally restrict question answering to an asset time range.
Optional execution window in seconds on the original asset timeline. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. Answers are generated only from media within this window, while returned timestamps remain absolute asset timestamps.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the find-key-moments workflow — the same fields accepted by POST /jobs/find-key-moments, minus engine-supplied fields such as asset_id.
Maximum number of key moments to extract. When omitted, defaults to 10 for assets up to one hour and 25 for longer assets.
Preferred highlight duration range in milliseconds. When provided, the model will aim to select moments within this range.
Preferred minimum highlight duration in milliseconds.
Preferred maximum highlight duration in milliseconds.
Controls whether video assets are analyzed using Mux shots. When true, shots are generated or reused and visual evidence drives selection. When false or omitted, selection uses transcript evidence and the asset must have a caption track. Not supported for audio-only assets.
Curated output_steering controls for execution scope, selection strategy, title style, audience, taxonomy, and rubric tie-breakers. Scope is enforced; other controls guide model behavior but do not guarantee exact output.
Optional execution window in seconds on the original asset timeline. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. Returned moment timestamps remain absolute asset timestamps.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Best-effort guidance for what qualifies as a strong standalone key moment.
Best-effort title style guidance for generated moment titles.
Intended audience used as best-effort model guidance. Does not change the output schema.
Preferred brand or domain terms to use when supported by the source content.
Controlled vocabulary used to steer notable audible concepts without changing the response schema.
Optional customer-facing name for the taxonomy.
Controlled vocabulary values used as best-effort model guidance.
Canonical taxonomy value to prefer when supported by the source content.
Short explanation of when this value applies.
Accepted alternate names that should normalize to the canonical label.
When false, the model should prefer values from the taxonomy. When true, non-taxonomy values may be used when no taxonomy value applies.
Rubric dimensions to use as tie-breakers after applying selection strategy. Best-effort guidance only.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the generate-engagement-insights workflow — the same fields accepted by POST /jobs/generate-engagement-insights, minus engine-supplied fields such as asset_id.
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the generate-premium-captions workflow — the same fields accepted by POST /jobs/generate-premium-captions, minus engine-supplied fields such as asset_id.
BCP 47 language code of the audio (e.g. "en", "es"). A best-effort hint that biases transcription toward this language — it is not verified against the audio and does not guarantee the output language. When supplied, language detection is skipped and the captions are labeled with this code. The language will be auto-detected when omitted.
When true, any existing text track with the same language code is deleted before uploading the new caption track. When false (default), the request is rejected if a matching track already exists.
Custom name for the uploaded Mux text track. Defaults to "{Language} (Generated)" using the resolved language code.
When true, speaker labels are identified and added to each caption cue. Useful for interviews, podcasts, and multi-speaker content.
When true, word-level timestamps are exported as a JSON file accessible via temporary_words_url in the job outputs. The URL expires 7 days after the job completes. Billed at a higher unit rate.
Whether to upload the generated VTT to the Mux asset as a new text track. Defaults to true. When false, no track is created and replace_existing must also be false; the generated SRT remains available via temporary_srt_url.
Best-effort list of words or short phrases (proper nouns, product names, jargon) likely to appear in the audio, used to bias recognition toward correct spellings. Does not guarantee exact output. Each phrase may contain at most 49 characters and 5 words, and must not contain the characters <, >, {, }, [, ], or .
Customer-chosen identifier, unique within the directive. Resource source.binding refs point at this handle.
The Robots workflow to dispatch — the same identifier as POST /jobs/{workflow}.
reference_ids of declared resources this workflow depends on. Each entry is a hard dependency: the workflow is not dispatched until the referenced resource is ready.
Parameters for the find-best-thumbnails workflow — the same fields accepted by POST /jobs/find-best-thumbnails, minus engine-supplied fields such as asset_id.
Maximum number of candidate thumbnails to return (1-5). Defaults to 1.
Curated output_steering controls for execution scope, thumbnail scoring strategy, explicit user intent, audience, campaign style, and scoring priorities. Scope is enforced; other controls guide model behavior but do not guarantee exact output.
Optional sampling window in seconds on the original asset timeline, normalized inward to whole-millisecond half-open bounds. The range must contain a video frame at the asset frame rate. Omit start_time to begin at the asset start and omit end_time to continue through the asset end. Returned thumbnail timestamps remain absolute asset timestamps.
Start of the execution window in seconds from the beginning of the media. When omitted, the window starts at the beginning.
End of the execution window in seconds from the beginning of the media. When omitted, the window extends to the end.
Best-effort guidance for what kind of thumbnail candidate should score highest.
Open-ended description of what the user is explicitly looking for in candidate thumbnails. Best-effort guidance only.
Intended audience used as best-effort scoring guidance. Does not change the output schema.
Short description of the campaign/channel thumbnail style to prefer when supported by the frame.
Rubric criteria to emphasize in close calls. Best-effort guidance only.
When true, the highest-scoring thumbnail's timestamp is written to the Mux asset's thumbnail_time once the job completes, making that frame the asset's default poster image. The new thumbnail will appear for some clients sooner than others, depending on local cache settings.
{
"name": "Generate captions, then translate",
"subject": {
"type": "video.asset"
},
"resources": [
{
"reference_id": "captions_en",
"type": "video.asset.track",
"kind": "caption",
"language": "en",
"source": {
"via": "workflow",
"binding": "premium_captions"
}
}
],
"workflows": [
{
"reference_id": "premium_captions",
"workflow": "generate-premium-captions",
"params": {
"language_code": "en"
}
},
{
"reference_id": "translate_es",
"workflow": "translate-captions",
"inputs": [
"captions_en"
],
"params": {
"to_language_code": "es"
}
}
]
}