Last Wednesday, Mustafa Anil Kocak from NYU’s Tandon School of Engineering came to CDS to explain how we can reduce algorithmic errors by instructing an algorithm not to make predictions under certain circumstances.
Machine learning algorithms are often used to make predictions in finance, medicine, or real estate, but it is still possible for them to miscalculate. This can have severe consequences. For example, if an algorithm misinterprets patient data, a doctor may inadvertently prescribe an ineffective treatment plan. The repercussions would then fall on both parties: the patient might be worse off, and the doctor could face a lawsuit.
Given the consequences of inaccurate predictions, Kocak is investigating how error rates can be reduced by not guessing. Using a new Conjugate Prediction approach, Kocak has crafted a meta-algorithm can be implemented on top of an algorithm to calculate the error rate of its predictions.
Imagine a lamp shade over a lit light bulb. The lit bulb is the algorithm producing predictions, while the lamp shade is the meta-algorithm that is placed over the bulb. The meta-algorithm analyzes the error rate of the algorithm’s predictions as they are generated and, if it determines that the error rate of a certain prediction is high, the meta-algorithm will force the algorithm not to make a prediction in that instance—forcing the light bulb to ‘turn off’ instead of continuing to emit light.
After testing his model on seven data sets, Kocak found that his meta-algorithm can reduce the error rates of algorithms by up to one quarter, at the cost of asking the algorithm to refuse guessing 11% to 58% of the time.
Whether Kocak’s model can be practically implemented in real-world scenarios still requires further investigation. But the meta-algorithm’s counterintuitive logic of not guessing is a fascinating intervention in a field that prizes predicting outcomes. Kocak’s model also gestures towards a possible paradox in the general predictive process: the most accurate way to guess may sometimes be not to guess at all…
by Cherrie Kwok