There has been a lot of recent news regarding development of various benchmarks that provide ways to compare the performance of different quantum computers. However, the actual data that shows the results of the tests has been relatively sparse. In some cases, the reports that exist use different methods or algorithms to perform the benchmarks so it is a little like comparing apples versus oranges.
So we were very happy to see a report posted on arXiv by researchers at the Los Alamos National Laboratory (LANL) that run IBM’s Quantum Volume test on 24 different machines from five different manufacturers. Although one might argue that the Quantum Volume metric may have some room for improvement, the fact that we have the data on so many different quantum processors makes this report well worth reading.
Here are a few interesting takeaways that we found in the report:
- The team at LANL was generally not able to measure the same Quantum Volume level that was reported for the same machine by the vendor. This may be due to number 2 below.
- Optimized compiling can make a big difference. The LANL team reported that they did reasonable amount of optimization in their testing, but without white-box access to the device. Certainly, if the manufacturer who designed the machine is performing the test, they are able to understand all the idiosyncrasies and tweaks that might be needed to achieve the best performance. But this intense level of optimization may be too difficult or too time consuming for a typical user to do.
- The Honeywell Continuum processor did very well in these tests. The lower bound of the Quantum Volume measurement for this machine was 256 while the next best measured value of 32. And the team reported this level could be higher, but the team had usage limitations and was not able to measure even larger quantum volume circuits.
The report makes for interesting reading and highlights the observation that much more work needs to improve the overal fidelity of the qubit operations. In all but two cases, the Quantum Volume measure was limited by the depth of the circuit that was tested rather than the overall number of qubits available in the processor. You can find this report posted on arXiv here.
March 9, 2022