Why Do We Benchmark, Anyway?
Before we get into the “what,” it helps to talk about the “why.” Why do browser engineers care so much about benchmarks?
At its core, benchmarking serves as a critical safety net for catching performance regressions before they ever reach users. But beyond that, benchmarks act as a powerful motivation function—a sort of “gamification” for browser engineers. Having a clear target helps us prioritize our efforts and decide exactly which optimizations deserve our focus. It also drives healthy competitiveness between different browser engines, which ultimately lifts the entire web ecosystem.
Of course, the ultimate goal isn’t just to make a number on a chart go up; it’s to meaningfully improve user experience and real-world performance.
Driven by Open Governance
Just like Speedometer 3, JetStream 3 is the result of a massive collaborative effort across all major browser engines, including Apple, Mozilla, and Google.
We adopted a strict consensus model for this release. This means we only added new workloads when everyone agreed they were valuable and representative. This open governance model has led to an incredibly productive collaboration with buy-in from multiple parties, ensuring the benchmark serves the best interests of the overall Web ecosystem.
Ripe for an Update
The last major release, JetStream 2, came out in 2019. In the technology space—and especially on the Web—six years is an eternity.
There’s a well-known concept in economics called Goodhart’s Law, which states that when a measure becomes a target, it ceases to be a good measure. Over time, engines naturally optimize for the specific patterns of a benchmark, and the metrics slowly lose their correlation with real-world performance. Speedometer recently received a massive update to account for this, and it only makes sense that JetStream is next in line.
JetStream vs. Other Benchmarks
You might be wondering: with the recent release of Speedometer 3, why do we need another benchmark?
While Speedometer is fantastic for measuring UI rendering and DOM manipulation, JetStream has a different focus: the computationally intensive parts of Web applications. We’re talking about use cases like browser-based games, physics simulations, framework cores, cryptography, and complex algorithms.
There are also practical engineering considerations. JetStream is designed so that it can run in engine shells—like d8, the standalone shell for V8. For engine developers, this is a massive advantage. Building a shell is significantly quicker than compiling a full browser like Chrome, allowing engineers to iterate faster. Because d8 is single-process, it also produces far less background noise, leading to more stable testing. This shell-compatibility also makes JetStream highly valuable for hardware and device vendors running simulators. It is a trade-off—a shell is slightly further removed from a full, real-world browser environment—but the engineering velocity it unlocks is well worth it.
How We Select Workloads
Building a benchmark requires a delicate balance between microbenchmarks and real applications.
Microbenchmarks are great engineering tools; they have a high signal-to-noise ratio and make it easy to see the effects of one specific optimization. While they make sense for early improvements of new features, they also often encourage overfitting in the long run. Engines might optimize heavily for a tiny loop that looks great on the benchmark but does absolutely nothing to help real users.
Because of this, a primary criterion for inclusion in JetStream 3 is that a workload should represent a real, end-to-end use case (or at least a highly abstracted form of one).
We also heavily prioritized diversity. We don’t want workloads that all exercise the exact same hot loop. We want coverage across different frameworks, varied libraries, diverse source languages, and distinct toolchains.
