With the recent leaks about Google’s quantum supremacy experiment, it is worth asking the question of how can one measure how well a machine is doing.  There are multiple ways of doing this and we thought it would instructive to see how sports car manufacturers do it.

Sports car manufacturers use a few different techniques to get feedback on their automobiles and typically use a mixture of the techniques described below. Engineers working on quantum hardware also use multiple ways of getting feedback on their performance and at a high level there are similarities to auto manufacturers in the basic approaches they may take.

Parametric Approach
One approach is to develop specific component metrics, take measurements and work to improve them.  In the automotive world these would include dynamometer test to measure horsepower, wind tunnel tests to measure drag coefficients, and skip pad testing to test lateral acceleration.  Although getting good results on these tests may not directly reflect upon how satisfied a particular customer will be when buying the car, it is safe to say the better one does on these tests, the more disposed their customers will be to like the car. Read this blog article to see how Mercedes-AMG does it.

Comparable parameters in quantum computing might include the T1 and T2 coherence times, single qubit and two-qubit gate fidelities.  IBM has been promoting a parameter called Quantum Volume which takes into account parameters such as qubit count, gate fidelities, connectivity, and others.  But Quantum Volume does not actually test the machine at the application level.

Nürburgring Approach
In Germany, there is a famous race track called Nürburgring that almost all the world’s automakers use to test out their cars. It consists of about 13 miles of varied roadway including straightaways, turns, and hills and is rented out about 19 weeks per year by an Industry Pool of about 40 car manufacturers and their suppliers so they can test out their new cars and technology. A key component of the testing is to make sure the cars have the durability to hold up under harsh usage. Doing well on the Nürburgring circuit may not necessarily be called a “real world” example since the average customer will never have a chance to drive their cars there.  But if a car can do perform well on Nürburgring, it is highly likely that it will also have good performance on the challenging curves of roads like California’s Highway 1. This article from Autoweek provides more detail on how the automakers utilize Nürburgring for their development efforts.

So we might call the Nürburgring Approach a “non-real-world application level benchmark”.  We could also call Google’s Quantum Supremacy experiment the same thing. Although there may not be much commercial use for the Quantum Supremacy test which is based upon determining the output of a random quantum circuit, it is highly likely that applications such as quantum machine learning, computational chemistry, and optimizations will greatly benefit from the design improvements that were made in order to successfully complete the quantum supremacy demonstration.

Customer Feedback Approach
Of course, a manufacturer performing their own tests on an automobile or computer may not always be enough. Sometimes end customers have concerns or issues that a manufacturer had not thought of or even tested.  So for this reason, they hold customer feedback sessions.  Sometime it may just consist of online customer surveys, other times it may consist of specially organized focus group, and occasionally they may even bring potential customers out to a test track to drive a prototype car and get their responses.  This article from InterQ Research describes how Tesla used focus groups to design their Model X SUV.

One quantum company that has developed features based upon customer feedback is D-Wave.  Features that they introduced in the D-Wave 2000Q such as anneal offsets, reverse annealing, anneal quench and others resulted in discussions with users.  And some of the new features such as the improved qubit connectivity and lower noise qubits that will be available in their upcoming Advantage™ system were also encouraged by their interactions with customers.

So there is a lot that can be learned from the sports car industry (as well as others) on how they measure performance and get feedback to guide design improvements in their products.  Although we have given specific examples of IBM using the Parametric Approach, Google using the Nürburgring Approach, and D-Wave using the Customer Feedback Approach, all of these companies use a mixture of ways to get a reading on how well their machines are performing and we recommend that they continue to do so because no one single method can provide all the information needed to help one develop the best possible machine.

October 3, 2019