False positives in radiation detection

Bruce Shneier picks up on this article quoting Robert Bonner (Commisioner of Customs and Border Protection) on radiation detection equipment beginning to be installed in ports:

Robert Bonner, commissioner of U.S. Customs and Border Protection, told a Senate subcommittee on homeland security that since the first such devices were installed in May 2000, they had picked up over 10,000 radiation hits in vehicles or cargo shipments entering the country. All proved harmless.

As an example of how the system was working, Bonner said on Jan. 26, 2005, a machines got a hit from a South Korean vessel at the Los Angeles seaport. The radiation turned out to be emanating from the ship's fire extinguishing system and was no threat to safety.

I tracked down Bonner's testimony which has this to say:

Our investment in WMD Detection technology is paying off as demonstrated by the following recent event. On January 26, 2005, at the Los Angeles seaport a PRD activated in proximity to a vessel from Kwan Yang, South Korea. A search of the vessel revealed that the source of the radiation was located in the ship’s engine room. Subsequent screening with a Radiation Isotope Identifier and analysis by CBP Laboratory and Scientific Services Personnel stationed at the NTC revealed that the material was Cobalt 60, a material used in industrial and medical applications. Following coordination with the Science and Technology Directorate’s Secondary Reachback Program, scientists were dispatched from the Department of Energy Radiation Assistance Program and it was confirmed that the radiation levels posed no threat to safety and that it was emanating from a gauge in the ship’s fire extinguishing system. Although this alarm proved to be benign, the event demonstrates CBP’s improving ability to detect sources of radiation in conveyances arriving at our borders and quickly take appropriate action to resolve any potential threats. Indeed, since CBP installed the first RPMs in May 2002, we have resolved over 10,000 radiation hits of vehicles or cargo shipments crossing our borders.

Shneier is rightfully amazed that the large number of false positives generated by the system are actually cited as an example of how well the system is working. I remember coming across this article and registering a similar reaction: high numbers of false positives should not be used as evidence for a system that is functioning well.

False positives in radiation detection can occur due to a variety of causes including (1) fluctuations in the natural radioactive background, (2) presence of other radioactive isotopes whose radiation cannot be distinguished from that being detected, (3) equipment malfunction. A good system design should seek to minimize the frequency of false positives, since they impose a cost: each positive needs to be investigated and the total cost of dealing with false positives is the frequency of false positives times the average cost of conducting an inspection. There are ways to reduce the false positive rate including adequate link budget in the detector system design so that the radiation signature can be effectively discriminated from the background as well as use of detectors with sufficient resolution to be able to distinguish between, say, Cobalt-60 and highly-enriched uranium (HEU).

How well is the system actually working? We need to look at the number of false positives (10,000 in this case) as a fraction of the system throughput (number of containers inspected). We also need to look at the number of false negatives: cases where the detection system failed to recognize radioactive materials. Both false positives and false negatives should be minimized and either category of error represents a failing of the system.