Optimization

This page explains how to use Prompt Spec’s optimization capabilities to automatically improve agent prompts.

What is Optimization?

Prompt Spec’s optimization feature uses AI to analyze benchmark results and generate improved system prompts for your agents. This process helps:

• Identify weaknesses in agent performance
• Generate targeted improvements to system prompts
• Validate that changes actually improve performance
• Iteratively refine agents over multiple optimization cycles

How Optimization Works

The optimization process follows these steps:

1. Analysis: The optimizer analyzes benchmark results to identify patterns of failure
2. Generation: Based on the analysis, it generates improved system prompts
3. Validation: The new prompts are tested against the same benchmarks
4. Selection: The best-performing prompt is selected for the next iteration

Using the Optimization CLI

To optimize an agent using the command line:

# Basic optimization with default settings
prompt-spec optimize path/to/spec.yaml
 
# Optimization with custom settings
prompt-spec optimize path/to/spec.yaml --iterations 5 --output optimized-agent.yaml

CLI Options

• --iterations, -i: Number of optimization iterations (default: 3)
• --output, -o: Output path for the optimized specification
• --model, -m: Model to use for optimization (default: gpt-4o)
• --verbose, -v: Show detailed optimization process

Using the Optimization API

To optimize an agent programmatically:

import { optimizeAgent } from "prompt-spec";
 
const optimizedSpec = await optimizeAgent("path/to/spec.yaml", {
  iterations: 3,
  outputPath: "./optimized-agent.yaml",
  model: "gpt-4o",
  verbose: true,
});
 
console.log(`Optimization complete. Final score: ${optimizedSpec.finalScore}`);

Optimization Strategies

Prompt Spec supports different optimization strategies:

Prompt Refinement

This strategy focuses on improving the system prompt while keeping the agent’s tools and configuration the same:

prompt-spec optimize path/to/spec.yaml --strategy prompt

Tool Selection

This strategy optimizes which tools the agent should use and when:

prompt-spec optimize path/to/spec.yaml --strategy tools

Full Agent Optimization

This strategy optimizes all aspects of the agent, including prompt, tools, and configuration:

prompt-spec optimize path/to/spec.yaml --strategy full

Example Optimization

Here’s an example of how optimization might improve a system prompt:

Original Prompt:

You are a customer service agent. Help users with their inquiries.

Optimized Prompt:

You are a customer service agent for an e-commerce platform. Your primary responsibilities are:

1. Help users track their orders using the checkOrderStatus tool
2. Provide shipping information using the getShippingOptions tool
3. Assist with returns using the initiateReturn tool

When users ask about order status, always collect their order ID first.
For shipping inquiries, ask for their region if not provided.
For returns, verify the order is eligible before proceeding.

Always be concise, professional, and solution-oriented.

Optimization Reports

After optimization, Prompt Spec generates a detailed report showing:

• Performance improvements across iterations
• Changes made to the system prompt
• Benchmark results before and after optimization
• Analysis of which changes had the most impact

Best Practices

For best results with optimization:

1. Start with diverse benchmarks that cover all expected agent capabilities
2. Run multiple optimization iterations (3-5 is usually sufficient)
3. Review optimized prompts manually to ensure they align with your goals
4. Test optimized agents on new benchmarks to verify generalization

Next Steps

• Check out the CLI Reference for all optimization commands
• Explore Examples of optimization in action
• Learn about Agent Specifications to customize optimization