Image source: Spanish "El Pais" website
When it comes to black boxes, many people think of equipment used to record flight data on airplanes or small theaters full of nostalgia. However, in the field of artificial intelligence (AI), black box is also an important term that cannot be ignored.
Spain’s El Pais pointed out that when an AI neural network is running, even the most senior researchers know nothing about its internal operations. The discussion here is not about biology, but about AI algorithms, especially those based on deep learning that imitate the connections between neurons. These systems are like black boxes, and it is difficult for data scientists, top talents in academia, and Nobel Prize-winning engineers at OpenAI and Google to peer into their internal secrets.
Model and data are opaque
"Scientific American" magazine reported that an AI black box refers to an AI system whose inner workings are completely invisible to users. Users can input information into these systems and get output, but they cannot inspect their code or understand the logic that produces the output.
Machine learning, as the main branch of AI, is the cornerstone of generative AI systems such as ChatGPT. Machine learning consists of three core parts: algorithm, training data and model. An algorithm is a series of program instructions. In machine learning, the algorithm learns to recognize patterns in the data through large amounts of training data. When the machine learning algorithm completes training, the product is the machine learning model, which is also the part that users actually use.
Any of these three parts of a machine learning system may be hidden, that is, placed in a black box. Typically, algorithms are publicly available. But to protect intellectual property, AI software developers often put models or training data into black boxes.
The model architecture is so complex that it is difficult to explain
Although the mathematics behind many AI algorithms are well understood, the behavior produced by the networks formed by these algorithms is elusive.
ChatGPT, Gemini, Claude, Llama, and any image generator like DALL-E, as well as any system that relies on neural networks, including facial recognition applications and content recommendation engines, face this problem.
In contrast, other AI algorithms, such as decision trees or linear regression (commonly used in fields such as medicine and economics), are more interpretable. Their decision-making process is easy to understand and visualize. Engineers can follow the branches of a decision tree and clearly see how a specific outcome is arrived at.
This clarity is critical because it injects transparency into AI and provides safety and security to those who use the algorithms. It is worth noting that the EU Artificial Intelligence Act emphasizes the importance of having transparent and explainable systems. However, the architecture of neural networks themselves hinders this transparency. To understand the black box problem of these algorithms, one must imagine a network of interconnected neurons or nodes.
Juan Antonio, a professor at the AI Institute of the Spanish National Research Council, explained that when you feed data into the network, the values in the nodes trigger a series of calculations. Information is propagated from the first nodes in numerical form to subsequent nodes, each node calculates a number and sends it to all connections, taking into account the weight (i.e. numerical value) of each connection. The new node that receives this information will calculate another number.
It is worth noting that current deep learning models contain thousands to millions of parameters. These parameters represent the number of nodes and connections after training, which are large and varied, making it difficult to derive meaningful equations manually.
According to industry estimates, GPT-4 has nearly 1.8 trillion parameters. According to this analysis, each language model will use approximately 220 billion parameters. This means that every time a question is asked, there are 220 billion variables that could influence the algorithm's response.
Tech companies try to open black boxes
Systemic opacity makes it harder to correct biases and fuels distrust. Currently, major players in the AI field are aware of this limitation and are actively conducting research to better understand how their models work. For example, OpenAI uses a neural network to observe and analyze another neural network, and Anthropic studies node connections and information propagation circuits.
Decoding the black box is of great benefit to the language model, which can avoid erroneous reasoning and misleading information generated by AI, and solve the problem of inconsistent answers. However, without understanding the inner workings of the network, technology companies often put models through extensive training and then release products after passing tests. This approach can also have problems, such as Google Gemini generating the wrong images when it was first released.
The opposite concept to the black box is the glass box. The AI glass box means that its algorithms, training data and models can be seen by anyone. The ultimate goal of decoding black boxes is to maintain control of AI, especially when it is deployed in sensitive areas. Suppose a machine learning model has made a diagnosis of a human's health or financial situation, would one want the model to be a black box or a glass box? The answer is obvious. This is not only a strong focus on the inner workings of the algorithm, not only out of scientific curiosity, but also the protection of user privacy.