IBM had introduced in 2017 a performance metric called Quantum Volume that would take into account both qubit count and qubit volume to help measure the power of a quantum computer. This metric had many advantages because it is a measure that is hard to game and helped emphasize the importance of qubit quality factors such as gate fidelities and coherence times that could have a great affect on how well a machine could execute a program. However, the measure had a deficiency because it did not provide any insight into the execution speed of a quantum computer. A program that might take 1 minutes to execute on one of IBM’s superconducting quantum computers could take 100 minutes or more on a machine built with a different technology even though the two machines had the same quantum volume. This became problematic for IBM because ion trap competitors Honeywell and IonQ started to mention quantum volume numbers that were better than IBM’s.

IBM also realized that the real time execution speed of a problem is a real concern of end users and has been working diligently in the past year to make improvements in this area. In order to provide more clarify to this issue, IBM has come up with a new performance metric called CLOPS (Circuit Layer Operations per Second) that can provide some visibility into speed performance. This metric is based upon the time it takes to run a quantum program that includes multiple shots, varying parameters, overheads associated with the control electronics and other factors. Areas that have been a focus at IBM to improve the CLOPS factor include faster gates, faster readouts, advanced control electronics with qubit reset, OpenQASM3 and the Qiskit runtime. They stated that in the past two years they have improved the CLOPS measure from approximately a level of 200 in 2019 to a current level of 1400 on their five qubit IBMQ_BOGOTA machine.

It is measured by a test method that IBM has outlines in an arXiv technical paper and calculated according to the following formula:
CLOPS = M × K × S × D / time taken
M = number of templates = 100
K = number of parameter updates = 10
S = number of shots = 100
D = number of QV layers = log2 QV

Although none of iBM’s competitors have adopted this measure yet as it is brand new, it is quite likely that IBM’s performance against this metric may currently be leading the industry. For one thing, the inherent gate delays in a superconducting gate are orders of magnitude faster than those reported in ion traps. Also, IBM’s recent work in optimizing the overall quantum program runtime stack appears to be more extensive than what we have seen from others. However, the relative leads in these types of measures can changes as vendors introduce new generations of machines so these advantages cannot be considered permanent. In addition, the solution accuracy provided by a quantum computer is still very important. So even if another machines may take longer to run a program than IBM’s, if it provides a more accurate answer for a certain application the other machine may still be a preferable choice for an end user.

Additional information about the CLOPS measure can be found in a blog artilcle posted on the IBM Research website here and an arXiv paper that describes it in much detail here.

November 2, 2021