Today must have been Quantum Benchmark Reveal Day! IonQ and Super.Tech have disclosed their own quantum benchmarking suites called Algorithmic Qubits and SuperMarQ, respectively. Both benchmark suites are based upon running common quantum algorithms rather than randomized circuits. And the Boston Consulting Group published a white paper that companies some of the existing benchmarks and discusses how benchmarks can be a key source of competitive advantage for investors and end users.

The IonQ Algorithmic Qubits are based with work performed by the QED-C Technical Advisory Committee (TAC) on Standards and Performance Benchmarks. A key concept is that the benchmarks should be based more closely on common algorithms that could be used in a quantum computer rather than a collection of randomized circuits. IonQ has made two changes to the QED-C definition. First, the IonQ measurement only tracks the number of two qubit gates while the QED-C measurement uses both one and two qubit gates. And second IonQ has defined a success criterion of 37% for the processor to show adequate fidelity in calculating its results. Once the various algorithms are run, the results are plotted on a graph below and the Algorithmic Qubit Measurement (#AQ) can be seen as the largest box of width N qubits and depth N2 two qubit gates where the results met the 37% success criteria. An example of this can be seen in the sample below as shown by the area inside the gray box:

Chart Demonstration Algorithmic Qubits Determination. Credit: IonQ

Associated with IonQ’s blog post on Algorithmic Qubits (#AQ), they also announced a new quantum processor, codenamed Aria, which is currently Ain private beta. They indicated that this new processor demonstrates an #AQ measure of 20, which they believe would be the highest of any currently existing quantum machine. However, this test has not yet been run on any other machine that we know of so we cannot verify this at the moment. It will be interesting to see if anyone else in the industry adopts this measure. You can read more about Algorithmic Qubits in a blog post located here and also see another announcement about their Aria processor and the results of its #AQ benchmarking here.

In another benchmarking announcement, Super.Tech a quantum software company spun out of research at the University of Chicago, has introduced its own benchmark suite based upon common quantum algorithms including GHZ, Mermin-Bell, Phase Code, Bit Code, ZZ-SWAP QAOA, Vanilla QAOA, VQE, and Hamiltonian Simulation. In a paper, they compare the results of these tests for each algorithm on a variety of IBM, AQT, and IonQ processors. Super.Tech believes it is important to match the device architecture to the use case rather than relying on just one single metric. Super.Tech has posted a press release announcing the SupermarQ suite here. Additional information on SupermarQ is available in a paper posted on arXiv here, a SupermarQ web page on the Super.Tech website here, and you can download the code which is available on GitHub from here and run the benchmarks yourself.

And to help better explain what benchmarks are and how they can be used, the Boston Consulting Group (BCG) has just published a white paper that helps to walk a reader through the Quantum Benchmarking Zoo. It explains the different types and how they provide value to investors and end users. The white paper has a table that shows the pros and cons of several other proposed benchmark metrics including IBM’s Quantum Value and CLOPS measures, Sandia’s Mirror Circuits, ATOS Q-Score and several others. The BCG paper can be found here.

The challenge for the quantum industry will be to get everyone to use the same benchmark. Although these are great proposals, if everyone is using a different benchmarking approach it will be difficult to make direct comparisons of one machine versus another. Hopefully, as the industry matures different companies will decide to do this and it will make it much easier for end users to gain an accurate picture of how the different solutions stack up.

February 23, 2022