A useful quantum hardware benchmark compares identical circuits under controlled conditions, then separates device quality from queue time, shot noise, compilation choices and changing calibration data. This workflow gives developers a repeatable way to test backends and decide whether a different device or hardware upgrade improves the result.
Overview
Quantum backend comparison is more involved than checking qubit count or a provider’s headline specification. A backend may offer more qubits but produce less reliable results for the circuit you actually need to run. Connectivity, native gates, two-qubit error rates, calibration age, circuit depth, queue behaviour and measurement noise can all affect the outcome.
The central rule is simple: keep the experiment constant wherever possible. Use the same logical circuits, input states, shot count, result-processing rules and acceptance criteria across the backends being compared. Record the compiled circuit as well as the original circuit, because transpilation can change gate count, depth and qubit mapping.
A practical benchmark should produce more than a single score. At minimum, record:
- Logical circuit details: qubit count, gate types, two-qubit gate count and ideal circuit depth.
- Compiled circuit details: physical qubits, native operations, transpiled depth and added routing operations.
- Outcome quality: success probability, distribution distance or another clearly defined fidelity proxy.
- Device context: calibration timestamp, reported gate errors, readout errors, connectivity and availability.
- Operational context: queue time, execution time, shot count, retries and any cost assumptions.
For background on reducing depth and execution cost, see Qiskit circuit optimisation. If you need to estimate runtime before testing, use the quantum resource estimation guide.
Checklist by scenario
Scenario 1: Establish a baseline with ideal and noisy simulation
Before sending work to hardware, run the benchmark on an ideal simulator. This establishes the expected output distribution and catches errors in circuit construction, measurement interpretation and result aggregation. Then use a noise-aware simulator when the platform provides suitable device information or a configurable noise model.
- Define the expected result for each circuit, including acceptable alternative outcomes where the algorithm is probabilistic.
- Run enough simulated shots to understand sampling variation.
- Save the circuit representation and software environment details.
- Confirm that the hardware measurement format can be mapped to the same result format.
The simulator is not a substitute for hardware testing. It is a control experiment that helps distinguish a device problem from a circuit or analysis problem. For choosing between local and cloud simulation, see the quantum simulator comparison.
Scenario 2: Compare general-purpose backends
For a broad backend comparison, create a small suite rather than one large application circuit. Include a single-qubit preparation and measurement test, a repeated single-qubit gate test, a Bell-state circuit, a short entanglement chain and a depth-scaling circuit.
| Test | What it probes | Useful measures |
|---|---|---|
| Single-qubit basis preparation | State preparation and readout | Success probability, readout bias |
| Repeated single-qubit gates | Accumulated one-qubit error | Success decay against depth |
| Bell state | Entanglement and two-qubit operations | Expected bitstring probability, parity |
| Entanglement chain | Connectivity and routing overhead | Compiled depth, added swaps, success probability |
| Depth sweep | Noise sensitivity over time | Quality at several depths, variance across runs |
Use the same logical circuit on every backend, but do not hide the compilation differences. A device with direct connectivity may execute a circuit with fewer two-qubit operations than a device that must route through intermediate qubits. That difference is part of the backend’s practical performance.
Scenario 3: Test a production-shaped circuit
Once the small tests are complete, benchmark a reduced version of the real workload. Preserve the important features: qubit interaction pattern, measurement locations, parameter ranges and representative circuit depth. If the production workload is hybrid, include classical processing and data-transfer time in a separate end-to-end measurement.
Do not use a single successful run as evidence of an improvement. Repeat the circuit under the same shot budget and compare confidence intervals or run-to-run variation. A backend that occasionally produces a strong result may be less useful than one that gives consistent, predictable results.
Scenario 4: Investigate a suspected hardware upgrade
Run the old and new configurations as an A/B test. Keep the benchmark version, transpiler settings, shot count, result parser and acceptance threshold fixed. Record whether the change is a different backend, a different qubit selection, a new compiler version or a changed error-mitigation setting. Otherwise, an apparent hardware gain may actually come from software changes.
What to double-check
Calibration data: Capture the date and time associated with reported qubit, gate and readout metrics. Calibration values can change, so treat them as context for the run rather than permanent properties of the backend.
Success probability: Define success before running the experiment. For a known target bitstring, it may be the frequency of that bitstring. For a distribution, use a stated distance or overlap measure. Avoid changing the definition between backends.
Fidelity proxies: Hardware experiments rarely provide a complete process fidelity estimate for an application circuit. Practical proxies include target-state probability, parity, Hamming distance, distribution overlap and success-rate decay with circuit depth. Label these as proxies and use the same measure for every backend.
Two-qubit errors: Two-qubit operations often deserve particular attention because they can dominate the error budget in entangling circuits. Compare both the reported device metric and the number of two-qubit operations introduced by compilation.
Shot noise: A difference of a few counts may be sampling variation rather than a meaningful device difference. Use a consistent shot count, repeat important measurements and report uncertainty where practical.
Queue time versus execution time: Measure these separately. Queue time affects delivery schedules and developer productivity; execution time reflects the workload itself. A backend should not be declared faster simply because one job waited less.
Reproducibility: Save the circuit source, transpiled circuit, backend identifier, selected physical qubits, SDK and compiler versions, shot count, job identifiers, calibration snapshot and analysis code. Provider commands and object names change, so keep platform-specific submission code in a small adapter rather than spreading it throughout the benchmark.
When comparing providers, define the scope carefully. A platform may expose several hardware types, simulators and execution options. The cloud platform decision guide can help separate backend selection from broader platform considerations.
Common mistakes
- Ranking by qubit count alone: Capacity does not describe the quality of the qubits used by your circuit or the routing required.
- Comparing different compiled circuits: If one transpilation adds several two-qubit operations, the comparison needs to show that overhead.
- Ignoring physical-qubit selection: A device-wide average can conceal meaningful variation between qubit regions. Record the selected mapping when the platform allows it.
- Mixing mitigation settings: Error mitigation may improve a reported result while increasing sampling, runtime or cost. Compare both mitigated and unmitigated results when mitigation is relevant.
- Using one circuit or one run: A narrow test can favour a particular topology, and one run cannot reveal stability.
- Confusing provider metrics with application performance: Published calibration values are useful inputs, not a replacement for an application-shaped benchmark.
- Changing the analysis after seeing results: Predefine the success measure and acceptance threshold to reduce accidental cherry-picking.
If mitigation is part of your workflow, document it separately and compare its trade-offs. See quantum error mitigation techniques developers can test for a practical framework.
When to revisit
Run the benchmark again before seasonal planning cycles, after a significant SDK or compiler update, when a provider changes its backend configuration, or when a new hardware option becomes available. Revisit it when your circuit changes materially: additional qubits, greater depth, a different interaction graph or a new measurement strategy can alter the preferred backend.
Also repeat the test if queue behaviour, execution pricing or access conditions affect the business case. These inputs should be verified from current provider documentation rather than copied from an older report. Keep historical results, but mark each record with its software, backend and calibration context so that older and newer measurements are not treated as directly interchangeable.
For a repeatable operating routine, use this final checklist:
- Freeze the benchmark circuits and define success before submission.
- Run ideal and, where useful, noise-aware simulations.
- Compile with documented settings for each target backend.
- Record depth, gate counts, two-qubit operations and qubit mapping.
- Capture calibration, queue, execution, shot and retry information.
- Run repeated hardware trials and report variation, not just the best result.
- Compare quality, time and cost as separate dimensions.
- Store raw results and rerun the suite whenever the underlying inputs change.
This approach turns quantum hardware testing into an engineering process rather than a one-off leaderboard. It also makes the result easier to explain: the best backend is the one that performs reliably for the circuit, constraints and operating conditions that matter to your team.