Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems

Abstract

Large Language Model (LLM)-based automatic Multi-Agent Systems (MAS) generation has become a crucial frontier for tackling complex tasks. However, existing methods face a dilemma between model capability and experience retention. Inference-time MAS leverages frozen frontier LLMs but repeats identical searches without learning from past experience. Conversely, Training-time MAS internalizes experience via gradient updates but is constrained by the low capability ceiling of smaller models, and is hard to scale to large frontier LLMs.

To bridge this gap, we propose Skill-MAS, a novel third path that decouples experience retention from parametric updates by conceptualizing the high-level orchestration capability as an evolvable Meta-Skill. Skill-MAS refines this architectural knowledge through a closed optimization loop: (1) Multi-Trajectory Rollout samples a behavioral distribution for each task under the current Meta-Skill; and (2) Selective Reflection adaptively selects priority tasks and applies hierarchical contrastive analysis to distill systemic experience into generalizable, strategy-level principles. Extensive experiments across four complex benchmarks and four distinct LLMs demonstrate that Skill-MAS not only achieves remarkable performance gains but also maintains a favorable cost–performance trade-off. Further analysis reveals that the evolved Meta-Skills are highly robust and exhibit strong transferability across unseen tasks and different LLMs.

Key Contributions

A New Third Path

We pioneer a paradigm that conceptualizes high-level orchestration behavior as an evolvable Meta-Skill, enabling frozen frontier LLMs to refine architectural knowledge without costly parametric updates.

Multi-Trajectory Rollout

Multiple independent rollouts per task convert single-trial outcomes into distributional statistics — separating genuine capability from execution stochasticity via uncertainty and difficulty scores.

Selective Reflection

Priority-driven task selection with adaptive elbow truncation, plus hierarchical contrastive analysis (within-task then cross-task), distills failure modes into actionable skill patches.

Better Cost–Performance Trade-off

By generating MAS in one shot with an evolved Meta-Skill rather than re-optimizing per sample, Skill-MAS attains the best average performance at moderate cost — outperforming both inference-time and training-time baselines.

Transferable Meta-Skills

Evolved Meta-Skills encode generalizable orchestration strategies and transfer robustly across unseen tasks and different backbone LLMs — validating task-agnostic skill evolution.

The Capability–Experience Dilemma

Overview of MAS paradigms and the Skill-MAS third path — **Figure 1:** Overview of MAS paradigms. (a)-(b) Comparison of existing Inference-time and Training-time MAS, illustrating the dilemma between model capability and experience retention. (c) Evolvable Skill-MAS bridges this gap by iteratively evolving the Meta-Skill, successfully coupling high-capability LLMs with experience retentiveness.

Inference-time MAS

EvoAgent, AOrchestra, AFlow — strong reasoning from frontier LLMs, but the Meta-agent executes identical search routines every iteration with no cumulative memory across runs.

Training-time MAS

MAS², MAS-Orchestra — internalize orchestration via fine-tuning small LLMs, but constrained by low capability ceilings and prohibitively expensive to scale to >100B models.

Skill-MAS (Ours)

Conceptualizes orchestration as an evolvable Meta-Skill (SKILL.md). A frozen frontier Meta-agent progressively refines strategy-level principles through rollout and reflection — no parametric updates needed.

Skill-MAS Approach

Skill-MAS models the Meta-agent's orchestration behavior as a structured Meta-Skill S, organized into three modules spanning the full pipeline. Each evolution round r proceeds through Multi-Trajectory Rollout and Selective Reflection, producing an improved skill S^(r+1). After R rounds, the validation-optimal skill S* is selected for test evaluation.

Task Decomposition (the what) Agent Engineering (the who) Workflow Orchestration (the how)

Skill-MAS evolutionary loop — **Figure 2:** The evolutionary loop of Skill-MAS. The Meta-Skill S^(r) (task decomposition, agent engineering, and workflow topology) guides Multi-Trajectory Rollout to compute distributional statistics. These metrics feed into Selective Reflection to prioritize tasks and extract evidence E via hierarchical trajectory reflection. Skill Optimization then leverages E to refine the Meta-Skill into S^(r+1), driving iterative improvement in MAS generation.

Stage 1 · Multi-Trajectory Rollout

Sample K independent trajectories per validation task under the current Meta-Skill. Compute per-task uncertainty (score std) and difficulty (negated mean score) to characterize behavioral distributions.

Stage 2 · Priority Selection

Fuse uncertainty and difficulty into a priority score, sort tasks, and apply elbow detection to select the most informative subset for reflection — concentrating the optimization budget.

Stage 2 · Hierarchical Reflection

Within-task contrastive analysis compares high- vs. low-scoring trajectories; cross-task synthesis extracts recurring failure modes and ranks impactful patches into structured evidence.

Stage 2 · Skill Optimization

The optimizer rewrites the Meta-Skill while preserving the three-module scaffold — removing ineffective rules, adding generalizable orchestration principles, and passing a structural validity check.

Benchmarks

We evaluate on four challenging benchmarks spanning deep research, expert-level mathematics, multi-hop QA, and real-world interactive scenarios:

DeepResearchBench

Deep research tasks scored on comprehensiveness, insight, instruction-following, and readability

HLE-Math

Humanity's Last Exam — complex expert-level mathematical reasoning evaluated by accuracy

BrowseComp-Plus

Complex multi-hop dynamic question answering with retrieval, evaluated by accuracy

VitaBench

Real-world daily scenarios requiring multi-tool calling, scored by rubric-based evaluator

Baselines

Inference-time MAS: EvoAgent, AOrchestra, AFlow

Training-time MAS: MAS², MAS-Orchestra

Meta-agent LLMs: Gemini-3.1-Flash, GPT-5.4-Nano, Qwen3.5-Plus, DeepSeek-V4-Flash. All metrics are normalized to [0, 100%]. We report average performance and average inference cost (USD).

Main Results

Quantitative Comparison

Even without iterative evolution, Skill-MAS-init demonstrates competitive performance — e.g., with Gemini-3.1-Flash it slightly surpasses the best baseline AFlow (21.68 vs. 21.29 avg.). Skill-MAS-optimized outperforms all baselines by a large margin in nearly all settings, achieving the highest average performance across all four LLMs. With Qwen3.5-Plus, avg. performance rises from 32.61% (init) to 38.41% (optimized); with DeepSeek-V4-Flash, from 33.72% to 41.05%.

Superior Cost–Performance Trade-off

Training-time MAS achieves the lowest cost but worst performance; inference-time MAS improves performance but incurs the highest inference cost. Skill-MAS occupies the best position — highest performance at moderate cost by generating MAS in one shot with an evolved Meta-Skill:

Skill-MAS cost-performance trade-off — **Figure 1:** (d) Cost-performance trade-off analysis, highlighting Skill-MAS as a better “Third Path” for automatic-MAS.

Analysis & Insights

(a) Meta-Skill Transferability

We evaluate transfer across LLMs and tasks. No-transfer (diagonal) achieves maximum gains; cross-LLM transfer (same task) follows closely; cross-task transfer (same LLM) also delivers competitive performance, validating that evolved Meta-Skills learn task-agnostic strategies:

Table 2: Transfer performance across LLMs and tasks — **Table 2:** Transfer performance across LLMs and tasks. Skill Source denotes the (LLM, Task) where the Meta-skill is evolved, and Test Setting denotes the (LLM, Task) where that Meta-skill is evaluated. Δ is measured against Skill-MAS-init.

Transfer heatmap across LLMs and tasks — **Figure 3:** Left: Skill transferability heatmap across LLMs (DS: DeepSeek-V4-Flash, GPT: GPT-5.4-Nano) and tasks (BCP: BrowseComp-Plus, VITA: VitaBench).

Observation

Cross-LLM transfer (Panel A) works well because trajectory analysis yields similar Meta-Skills regardless of backbone. Cross-task transfer (Panel B) also succeeds when skills encode general orchestration patterns rather than domain-specific tricks.

Insight

Meta-Skills distill strategy-level principles — not task-specific fixes — enabling robust generalization across both unseen datasets and different LLMs.

(b) Scaling Multi-Trajectory Rollout

We ablate the rollout number K per task (default K=5). Performance consistently improves with more trajectories, but with diminishing returns from 5 to 7:

Performance scaling with rollout number K — **Figure 3:** Right: Performance scaling across increasing multitrajectory rollout numbers (K = 3, 5, 7).

Insight

Performance improves consistently as the rollout number increases, indicating a clear positive correlation between sampled trajectories and evolution quality. However, the gains exhibit diminishing returns: the improvement from K=3 to 5 is larger than from 5 to 7, while a higher rollout number inevitably incurs greater computational cost during evolution. In practice, this hyperparameter therefore requires a careful trade-off between maximizing performance and managing evolution overhead.

(c) Ablation on Selective Reflection

Table 3: Ablation on selective reflection and multi-task learning — **Table 3:** Results of Skill-MAS with different settings.

Without adaptive priority selection, both label-free variants (Full-Validation, Half-Validation) suffer a performance drop but still surpass most baselines — highlighting the importance of elbow-based task selection. Multi-task learning shows mixed results, with slight gains on VitaBench but drops on BrowseComp-Plus.

(d) Meta-Skill Evolution Trajectory

Meta-Skill evolution on BrowseComp-Plus — **Figure 4:** Meta-Skill Evolution on BrowseComp-Plus.

Skill evolution traces a coherent arc: Module 1 establishes epistemic scaffolding (constraint prioritization, fan-out retrieval); Module 2 introduces calibrated evaluation (weighted constraint satisfaction); Module 3 adds system-level resilience (cross-entity bridging, merge-node re-execution).

(e) What Evolved MAS Look Like

On BrowseComp-Plus, baselines produce superficial parallel searches or linear chains without constraint linking. Skill-MAS-optimized generates structured pipelines with clue-specific retrievers and link-verification before final decisions:

Case study on BrowseComp-Plus — **Table 7:** MAS structure comparison on BrowseComp-Plus (DeepSeek-V4-Flash).

Case study on VitaBench — **Table 8:** MAS structure comparison on VitaBench (DeepSeek-V4-Flash).

Representative Generated Workflows

A curated preview of Skill-MAS-generated MAS architectures from our Gallery (DeepSeek V4 Flash). Each case shows the Stage-3 workflow topology — serial chains, parallel fan-out/fan-in, and iterative feedback loops — laid out horizontally so you can read the full agent graph at a glance.

Gallery & Online Demo

Want more? Browse the full Gallery for every evolved Meta-Skill and rollout artifact, or jump into the Online Demo to build and run your own three-stage MAS pipeline interactively.

Open Gallery → Open Online Demo → Run Locally →

BibTeX

@article{lin2026skill,
  title   = {Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems},
  author  = {Lin, Hehai and Yang, Qi and Qin, Chengwei},
  journal = {arXiv preprint arXiv:2606.18837},
  year    = {2026}
}