Pipeline Architecture

This page explains the internals of the pipeline system: how definitions are discovered, how the spawner works, and how to extend it.

Pipeline Definition Format

Every pipeline is a Markdown file at knowledge/system/rules/pipelines/<name>.md. The filename (minus .md) becomes the trigger name.

Frontmatter

---
type: pipeline
domain: lead-generation        # lead-generation, client-delivery, operations
status: active                 # active, draft, deprecated
model: sonnet                  # Default model: haiku, sonnet, opus
passes:                        # Optional — omit for single-pass
  - name: research
    model: sonnet
  - name: synthesis
    model: opus
permissions: rw-knowledge      # read-only, rw-knowledge, full
review_gate: false             # If true, stops between passes for approval
summary: One-line description
---

Required Body Sections

• Preflight Check — validates inputs before execution
• Purpose — what the pipeline accomplishes
• Flow — step-by-step instructions for each pass
• Output Format — template for the output file
• Rules — constraints and guardrails
• Error Handling — what to do when tools are unavailable

Discovery Chain

1. pipeline-prompt.ts defines PIPELINES_DIR as $KNOWLEDGE_DIR/system/rules/pipelines/
2. When a trigger arrives, getSystemPrompt(pipelineName) reads the definition
3. getFrontmatterModel(pipelineName) extracts the model for routing
4. getPipelinePasses(pipelineName) parses multi-pass configuration
5. getPipelinePermissionsTier(pipelineName) enforces access levels

Spawner Behavior

The spawner in spawner.ts manages pipeline execution:

• Max turns — each pipeline has a configurable maximum number of tool-use turns (default: 25)
• Parallel limit — maximum 3 concurrent jobs (configurable via MAX_PARALLEL_PIPELINES)
• Shell safety — all commands use execFileSync with argument arrays (never string interpolation)
• Pipeline name validation — must match /^[a-z0-9_-]+$/

Permission Tiers

Adding a New Pipeline

1. Create the .md file in knowledge/system/rules/pipelines/
2. Add a max-turns entry in spawner.ts PIPELINE_MAX_TURNS map
3. Add the trigger to knowledge/system/CLAUDE.md Pipeline Triggers table
4. Restart the pipeline server
5. Trigger from Mission Control or via API

The Document Compiler

The compile feature is a document compiler with 7 types — it is architecturally different from the 17 pipelines. It sends a chat command rather than spawning a full Claude Code process. Use it to transform raw pipeline output into branded client-ready documents:

compile [doc-type] [client-name]

The compiler reads formatting rules from system/rules/compile-guidelines.md and applies brand tokens from the client's brand.md and voice.md files.