Testing and Demonstrating
Volvo has been working on an automated braking system. This YouTube video shows a test in which the system fails, and the car plows into the back of a truck. I’m told (by a thoroughly unreliable source - one of my colleagues) that the problem was with the car battery - it had been put on “quick charge” before the test, and for some reason this led the battery to fail. Volvo came in for ridicule over this test (and others - there are several videos to be found on YouTube of autonomous vehicles crashing). I doubt the videos are good for business, and the person who pointed them out to me says they are putting the future of driverless cars into doubt. People just aren’t confident about autonomous vehicles.
The thing is, in systems engineering terms, this test is “successful”. Technologies are tested in order to find faults, not show their absence. To paraphrase Glenford Myers Testing can never prove the absence of faults, but can be very helpful for showing their presence. In systems engineering then, a “successful” test is one that finds something wrong. … At least in theory. In the test in question, I am certain Volvo discovered something valuable - that there was a problem with the “quick charge” setting for the battery. No doubt they have fixed it now. The issue is that this test was done in front of an audience. In this situation testing becomes demonstrating. A successful test is not a successful demonstration. A successful demonstration is not a successful test.
On the 17th July 1984 an organisation called CEGB tested a flask designed for transporting nuclear materials by rail - by ramming a locomotive and three carriages into it at 100mph. The chairman of CEGB described it as “the most horrendous and pessimistic crash we could arrange”. The flask withstood the impact (you can see a big group of people standing around it near the rear train carriage in the image above). This test reassured a skeptical public that transporting nuclear materials was safe. However - as a test - this exercise was fairly worthless. Look at the picture above and ask yourself why the tracks stop where they do? Notice all the nice, soft mud. The train is a Type 46 locomotive, which has a relatively “soft nose”. The wheels had also been removed from the flask and the carriages weighted so as not to ride up over the engine. This was a demonstration designed to show the system working - not a test designed to make it fail.
On the 1st December 1984, NASA and the FAA tested AMK (anti-misting kerosene) - fuel modified in such a way that it was less likely to ignite and burn severely in the event of a crash. The test involved crashing a remote controlled 747 aircraft, wheels-up, just short of a runway. To make the situation more extreme, giant cutters were embedded into the runway (you can see them in the image above). The plane crashed, exploded and was destroyed by fire. To the observing media the test appeared to be a catastrophic failure. But the explosion was caused when one of the engines hit a cutter, stopping the turbine and causing a massive release of energy. The subsequent fire was much slower to spread and passengers would have had some time to disembark (although it is very unlikely anyone would have actually survived the impact). The test was designed to push the system to extremes, and the resulting data led to insights about how fire spreads in the event of an AMK fueled plane crash. It was a successful test. However, it effectively led to the cancellation of the project.
I’m not sure where this leaves Volvo. I think the video shows them demonstrating a system rather than testing one. But maybe we should be pleased that Volvo’s demos aren’t quite as carefully stage managed as CEGB’s. Maybe we should be more concerned when Volvo’s demonstrations start going smoothly. Certainly, when it comes to testing, we shouldn’t be quick to laugh when tests are successful. When it comes to testing, we mustn’t put pressure on testers to show that a system works. Its much easier to video vehicles being tested than many other forms of autonomous system, and we need to be careful that the increasing availability of such video does not lead to a reluctance to test them to extremes.
The train and plane crash tests are discussed in The Golem at Large by Collins and Pinch. The train crash image comes from here. The plane crash from here.
Volvo discuss their 2020 vision here.
