APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News

Microsoft's Copilot SDK 2.0 Runs 95x Faster by Ditching Containers for Rust

Microsoft released Copilot SDK 2.0 at Ignite. It was formerly called the GitHub Copilot SDK. A rebuilt Rust core cut a warm session turn from 5.25s to 55.3ms, by Microsoft's own benchmark.

Microsoft's Copilot SDK 2.0 Runs 95x Faster by Ditching Containers for Rust
Source
Microsoft Developer
Published
Author
Microsoft Developer
Read
7 min read

Takeaways

  • Microsoft released Copilot SDK 2.0 at Ignite. It was formerly called the GitHub Copilot SDK.
  • A rebuilt Rust core cut a warm session turn from 5.25s to 55.3ms, by Microsoft's own benchmark.
  • One shared runtime now hosts many isolated sessions, so new sessions no longer need a container, process, or VM.
  • Each session runs as its own user, with a strict tool allow-list in empty mode.
  • Durable session journals let sessions resume across restarts and machines, but there is no built-in session locking.
  • Free SDK for six languages. Usage bills like Copilot CLI prompts, or through your own provider keys ( repo ).

Microsoft announced Microsoft Copilot SDK 2.0 at Ignite, renaming what developers previously knew as the GitHub Copilot SDK. Version 2.0 replaces the original container-per-session architecture with a shared Rust runtime that can serve many users concurrently. Session state can also survive process restarts and move between service instances.

The new architecture can reduce startup latency and container costs for products whose agents rarely need a shell. It also changes how developers handle user identity, persistence, authorization, and sandboxing. Microsoft organizes those choices into six deployment patterns.

Containers stop being the default

The SDK provides programmatic access to the Microsoft Copilot harness, the agent loop, tools, and runtime used across Microsoft and GitHub products. GitHub described the harness as a shared agent foundation in its earlier rename announcement .

Version 1 required a container for every session, including conversations that only called an API or answered a short question. That design created three recurring costs:

  • Slow first turns: infrastructure had to start before the agent could respond.
  • Linear infrastructure growth: each additional session required another container.
  • Process-level identity: sessions inherited the credentials of the process that launched them.

Containers remain appropriate when an agent compiles code, executes untrusted scripts, or depends on a custom image. Microsoft says most of its internal teams still place each code-executing session in a separate sandbox. The shared runtime separates session state; untrusted execution still requires an isolation boundary.

Rust removes the startup drag

Serving many users from one process requires a small runtime footprint, so Microsoft rewrote the core in Rust. Its warmed benchmark measures a runtime that has already loaded, then creates a client and session and completes one turn.

MetricTypeScript CLIRust coreReported change
One-turn latency5.25 seconds55.3 millisecondsAbout 95 times faster
Memory for 10 clients1,383 MB126 MBAbout 91% lower
Throughput7.55 sessions per second120 sessions per secondAbout 16 times higher

Those figures come from Microsoft, which provides no hardware details or fuller workload definition beyond “warmed.” Model inference will still dominate the latency of many agent turns. The Rust rewrite removes overhead around the model call rather than accelerating the model itself.

The default v2 path loads the native Rust runtime directly into an application through a C ABI, a stable C-style interface that many languages can call. The SDK exchanges framed JSON-RPC messages with that runtime over an in-memory connection. Version 1 instead launched the Copilot CLI in headless mode as a subprocess and communicated through a pipe, which also required a Node.js runtime. GitHub’s migration notes describe the port, which the team completed partly with Copilot.

Sessions gain identity and memory

Shared hosting depends on strong boundaries between users, so version 2.0 adds per-session identity, configurable tools, and external persistence:

  1. Many sessions can share one process. Each session keeps separate credentials, tools, files, history, and runtime state. Creating one does not require a new process, virtual machine, or container.
  2. Credentials belong to the session. Developers provide the user’s credential when creating a session, allowing model and tool calls to be attributed to that user. A gitHubTokenProvider callback can supply refreshed, short-lived tokens instead of storing long-lived credentials.
  3. Empty mode starts with no host access. Setting mode: "empty" removes the default shell, file-system access, and built-in tools. Applications can expose only approved tools that operate against their own data and services.
  4. Session state can live outside the process. The runtime records messages, plans, tool calls, and checkpoints in an event log called the session journal . A sessionFs provider stores that journal in object storage, a database, or a tenant-specific directory, allowing another service instance to resume the session by ID.

A restricted TypeScript session can combine empty mode, a per-user token, and an explicit tool allow-list:

Externalized state allows a browser to disconnect while a server continues a long-running task, or a webhook to resume the conversation later. Another client, including GitHub Copilot CLI, can attach to inspect or steer a live session.

Applications that need several clients connected to the same active session can place the SDK behind an Agent Host Protocol host. AHP is an open protocol for communication between agent clients and hosts, and VS Code’s agent host uses it.

Six paths from shared to isolated

Microsoft divides deployment design into two decisions: where the model loop runs and where shell or file operations execute. The resulting patterns range from a fully shared process to a dedicated sandbox for every session.

PatternRuntime layoutBest fit
IsolatedEach session runs entirely inside its own container.Agents that need a shell, files, or a custom image from the first turn.
SharedMany sessions use one process without a sandbox.API and Model Context Protocol workflows. Microsoft considers it of limited use for standalone coding agents.
PromotableA session starts on the shared runtime and moves into a sandbox when needed. The one-way move preserves its ID and conversation.Products where only some conversations eventually require code execution.
DirectThe model loop remains shared while bash, view, and edit run inside a sandbox.Code interpreters and agents that need isolated tools without moving the full session.
DelegateA coordinator launches worker agents, each with its own sandbox.Research swarms and software pipelines that divide a task among several workers.
MixedDirect sandbox tools and delegated workers operate together.Workloads requiring both execution models and accepting the highest operational complexity.

Sessions that always need shell and file access fit the Isolated pattern. API-first products can begin with Shared or Promotable, while Direct keeps tool execution isolated without relocating the model loop. Delegate and Mixed suit systems that coordinate several agents.

Shared runtimes shift the risk

Runtime-level separation cannot replace application authorization. GitHub’s multi-tenancy guide assigns session routing and access control to the host application. Every resume or delete request should verify that the caller owns the session, and tool allow-lists should avoid broad patterns such as builtin:* . Tool handlers must also scope every data request to the authenticated user.

Durable sessions introduce several operational constraints:

  • Concurrent access requires coordination. The SDK provides no built-in session locking, and simultaneous access to one session has undefined behavior. Applications need a lock or queue.
  • Tool memory is temporary. In-process tool state is absent after a restart. Tools should remain stateless or persist their own data.
  • BYOK credentials must be supplied again. The journal does not save a developer’s model API key, so the application must provide it when resuming a session.
  • Approval storage needs tenant boundaries. With sessionFs , remembered tool approvals are shared within a storage namespace rather than separated by session ID. Each tenant should receive a distinct namespace.

Microsoft currently expects developers to host these architectures on their own infrastructure. GitHub Copilot cloud sandboxes and Microsoft Foundry managed agents provide the two managed options named in the announcement. Microsoft says more hosting choices are planned but gives no dates.

Who pays for what

The SDK is free and open source, while model usage remains billable. Copilot-based use follows the Copilot CLI model, with prompts counting toward the user’s usage allowance. A Copilot subscription is required unless the application uses Bring Your Own Key, commonly shortened to BYOK.

BYOK lets developers supply credentials for providers such as OpenAI, Microsoft Foundry, or Anthropic. Those requests are billed through the selected provider and do not consume Copilot premium-request quotas.

Version 2.0 supports TypeScript, Python, Go, .NET, Java, and Rust without a separate Copilot CLI installation. Node.js and .NET packages include platform runtime artifacts, Python downloads them, and Rust bundles them through its default features. Installation instructions and examples are available in the source repository .

Where version 2.0 pays off

Products whose conversations mostly call APIs can now keep many sessions in one process, persist their journals externally, and resume them from any service instance. The Promotable pattern can reduce container usage and first-turn latency when only a minority of sessions eventually need a shell.

Agents that execute untrusted code from the first message should retain per-session sandboxes. Those systems can still use version 2.0 for its lower runtime overhead, per-session credentials, and resumable state.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Sources

  1. 01x.com
  2. 02commandline.microsoft.com
  3. 03github.blog
  4. 04github.blog
  5. 05github.com
  6. 06github.com