AI Black Box Problem
The cluster discusses the 'black box' nature of neural networks and deep learning models, debating their interpretability, explainability of decisions, and the challenges or progress in making AI understandable to humans.
Activity Over Time
Top Contributors
Keywords
Sample Comments
Aren't large neural network already black boxes we don't understand built by machines we understand?
In layman's terms, how would you explain that deep learning isn't a black box?
It feels like you’re blaming the AI engineers here, that they built it this way out of ignorance or something. Look into interpretability research. It is a hard problem!
Knowing almost nothing about neural nets, is it possible for a NN based AI to explain it's decisions in some manner that we would understand?
What is it with hacker news commenters and assuming other people don't know things?The comment I was responding to was talking about training sets, I riffed on that. Depending on your definition of "full explanatory power", the model itself might very well not be enough, especially in the case of neural networks. Could you take a set of weights in a 5-deep neural network, look at an input vector, and have any kind of intuition about the output? It could be that there&
There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example.Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable.[0] https://chr
Counterexample:Convolutional Neural Networks (CNNs) (or any ML model, really)We still don't fully understand how or why they work so well. If you have software that queries a CNN and the CNN returns some prediction, there's absolutely no way of knowing or understanding the reasoning behind the decision. It's a black box of magical layers and weights that have been "trained" to make good predictions.If you have fraud detection software using ML and it flags somet
Why on earth should we trust (for such things) a model that includes an opaque neural network?
Wouldn't the work on interpretability would solve these concerns?
If a neural network has let's say 50 million parameters, it doesn't matter if you can trace any of the calculations, you're never going to actually understand how the model operates.Sometimes it's possible to extract a sort of conceptual understanding in some cases, eg you might say that this layer performs edge detection or whatever. But that's not much of an explanation.They do it because it works. You trust your Uber driver to get you to your destination even th