CLI Reference

This page provides a comprehensive reference for Prompt Spec’s command line interface.

Installation

The Prompt Spec CLI is installed automatically when you install the package:

npm install -g prompt-spec
# or
pnpm add -g prompt-spec

Command Overview

Prompt Spec provides the following main commands:

• test: Run benchmarks against an agent specification
• transpile: Convert a YAML specification to a test file
• optimize: Optimize an agent based on benchmark results
• init: Create a new agent specification template
• validate: Validate an agent specification

Common Options

These options are available across multiple commands:

• --help, -h: Show help information
• --version, -v: Show version information
• --config, -c: Specify a configuration file

Command: test

Run benchmarks against an agent specification.

prompt-spec test [options] <spec>

Options

• --output, -o: Output path for the generated test file
• --keep, -k: Keep the generated test file after running
• --watch, -w: Watch for changes and rerun tests
• --reporter, -r: Test reporter to use (default, json, verbose)
• --benchmark, -b: Run a specific benchmark by name
• --timeout, -t: Timeout for each benchmark in seconds (default: 30)

Examples

# Run all benchmarks in a specification
prompt-spec test agent-spec.yaml
 
# Run a specific benchmark
prompt-spec test agent-spec.yaml -b "Password Reset Test"
 
# Run tests with verbose output
prompt-spec test agent-spec.yaml -r verbose
 
# Run tests and save results to a file
prompt-spec test agent-spec.yaml -o results.json

Command: transpile

Convert a YAML specification to a test file.

prompt-spec transpile [options] <spec>

Options

• --output, -o: Output path for the generated test file
• --format, -f: Output format (ts, js) (default: ts)

Examples

# Transpile a specification to a test file
prompt-spec transpile agent-spec.yaml
 
# Specify output path
prompt-spec transpile agent-spec.yaml -o tests/agent-test.ts
 
# Transpile to JavaScript
prompt-spec transpile agent-spec.yaml -f js

Command: optimize

Optimize an agent based on benchmark results.

prompt-spec optimize [options] <spec>

Options

• --iterations, -i: Number of optimization iterations (default: 3)
• --output, -o: Output path for the optimized specification
• --model, -m: Model to use for optimization (default: gpt-4o)
• --strategy, -s: Optimization strategy (prompt, tools, full) (default: prompt)
• --verbose, -v: Show detailed optimization process

Examples

# Basic optimization
prompt-spec optimize agent-spec.yaml
 
# Optimization with 5 iterations
prompt-spec optimize agent-spec.yaml -i 5
 
# Full agent optimization
prompt-spec optimize agent-spec.yaml -s full

Command: init

Create a new agent specification template.

prompt-spec init [options] <name>

Options

• --type, -t: Template type (basic, customer-service, qa) (default: basic)
• --output, -o: Output directory (default: current directory)

Examples

# Create a basic agent template
prompt-spec init my-agent
 
# Create a customer service agent template
prompt-spec init customer-agent -t customer-service
 
# Specify output directory
prompt-spec init my-agent -o agents/

Command: validate

Validate an agent specification.

prompt-spec validate [options] <spec>

Options

• --strict, -s: Enable strict validation

Examples

# Validate a specification
prompt-spec validate agent-spec.yaml
 
# Validate with strict mode
prompt-spec validate agent-spec.yaml -s

Environment Variables

Prompt Spec CLI respects the following environment variables:

• AGENT_BENCH_CONFIG: Path to a configuration file
• AGENT_BENCH_API_KEY: API key for LLM providers
• AGENT_BENCH_MODEL: Default model to use
• AGENT_BENCH_TIMEOUT: Default timeout in seconds

Configuration File

You can create a configuration file (.prompt-spec.json or .prompt-spec.yaml) in your project root to set default options:

{
  "defaultModel": "gpt-4o",
  "timeout": 60,
  "reporters": ["default", "json"],
  "outputDir": "./results"
}

Next Steps

• Explore Agent Specifications to create your own agents
• Learn about Benchmarking to test your agents
• Check out the API Reference for programmatic usage