3. Putting robots in boxes
If you have not read anything by Sir Kazuo Ishiguro, I would highly recommend it. Ishiguro is a present day Japanese writer, known for novels like Never Let Me Go and The Remains of the Day. His newest work is called Klara and the Sun, and it is quite a relevant work for the topics that we are discussing here.
In this novel, “Artificial Friend” Klara stars as protagonist. She is a programmed entity, the goal of her life is to maximise happiness and comfort of her companion child. This is a noble goal, but there exists skepticism in the world of Klara and the Sun. Ishiguro writes:
“They’re afraid because they can’t follow what’s going on inside any more. They can see what you do. They accept that your decisions, your recommendations, are sound and dependable, almost always correct. But they don’t like not knowing how you arrive at them.”
We might not be nearing — say — Back to the Future’s hover boards yet, but science fiction like Kazuo Ishiguro’s work feels close to our reality. Even if not in the form of complete artificial entities like Klara, these described thoughts are a reality already. There exists a great fear of non-transparent machine learning, with the black box as its main target. I am not disagreeing with the proposed caution when it comes to opaque models, but why the black box?
An example of where the black box is singled out can be found in Europe’s plans to regulate the field of artificial intelligence. As can be read in Science Magazine, the 2020 version of this regulation proposal included a complete ban on “black box” AI models in certain areas of deployment. The current proposed regulations differ slightly from earlier versions, and there is still no more than a proposal at this moment rather than a full legal system. However, this fear of the unpredictable shines through: What are these black boxes we are so afraid of?
Let’s talk about that, in a few headlines:
1. There are no true black boxes.
This is an important point of clarity, there are no true black boxes. What I mean with this — very vague — description, is that in theory every model is understandable. The theoretical idea of a true black box, is that of a model which we give some input variables and from which we receive a certain output without any sort of idea what happened in between. However, in practice there is always some knowledge of the framework and the process that occurred to get the model to where it is.
In other words, if we were to have an infinite amount of time, we could follow the exact decision procedure of the model as it works its way through a single instance. We could simply work our way through all the learning that has occurred to get us to this point, this procedure is “computable”¹. This is due to our part in creating the models, they are not a completely alien-made object that we stumbled upon.²
2. We are the ones that decide whether a model is a black box.
As it would be written in a mathematical proof: This follows directly from point 1.
If there are no black box models that are theoretically uninterpretable, there is no inherent quality to any one model which makes it a black box model. All models are interpretable to a degree, even the ones that would take us millions of years to actually understand from the inside. Therefore, deciding what is interpretable is up to us. In the end, it is merely a useful distinction for us to make when speaking about models. In much the same way that we agree on a concept like it being “a nice day” out, we choose for ourselves.
It is valuable to realise that, as a result of this, there is no one set definition for a black box model. We might agree around the world that “2” is what we call the number succeeding “1”, but me and my neighbour might have very different ideas on what it actually means that it is “a nice day” out. So far, the definition of a black box model is not as agreed upon as the definition of natural numbers and this is good to keep in mind.
3. The definition of what we call a black box depends on who we are considering interacts with the model.
When we talk about black boxes, we talk about models that are treated as black boxes. This is necessary, as we encounter such intricate models that it is impossible to understand them practically speaking. Simply put, at some point we decide that it is easier to consider a model as one that we have no knowledge of rather than using the inner workings of the model itself.
One important aspect of this is that the way we treat a model depends on the one that is working with it. If I would explain to my grandmother what a decision tree model is, she would most likely still treat it as a black box model. There is an input and an output, that is about it. However, a computer scientist has quite a good idea of what a decision tree model looks like on the inside: For them, treating it as a black box model seems unnecessary.
As models get more adept at dealing with complicated problems — and therefore become more complicated in the process — they become less transparent. This could be seen as a continuous scale, as there is no inherent property that makes a model completely opaque. Therefore the humans that use words like “black box” or “interpretable” are the ones that decide where to draw the line between different terms.
I am still of the opinion that it is very valuable to distinguish between the interpretability of different models. However, proceeding to use terms like “black box” warrants that it remains very clear what we mean by this. Even though this is a more difficult and complex task, to speak in terms of opacity of a model might be more fit for some discussions than answering YES or NO to the question whether something is interpretable.
There might be room for a few grey-scales in-between the spectrum of merely black and white.
Footnotes:
As a footnote for the logically inclined reader, this does not mean that the properties of models are necessarily decidable. This article refers to a paper in which the undecidability of what is called learnability is proven through a connection with Gödel’s undecidability results.
Although we might learn a lot from considering them to be just that: completely alien. More on that later.