Why does the chapter caution against referring to the output pi of a neural network trained with BCE loss as a probability?

The chapter cautions against calling pi a probability because a BCE-trained network minimizes the discrepancy between its output and the binary label, not the true probability distribution of that label. Treating pi as a true probability requires idealized assumptions such as infinite data and perfect calibration, which rarely hold in practice. Instead, pi is best described as a classification decision or confidence-like score, not an exact probability.

The chapter explains that while pi is often described as the predicted probability of the label yi, this description overstates what the model actually learns. The BCE loss function directly compares the network output to the binary label yi, which may represent just one instance of many possible outcomes, and does not account for the ground truth probability distribution for each label. In an idealized setting with infinite data and a perfectly calibrated network, pi might approximate a true probability, but real-world data is finite and noisy, so this equivalence breaks down. Moreover, the BCE loss is not designed to make pi as close as possible to the true probability; it minimizes classification error, which can drive outputs toward 0 or 1 even when that does not reflect the true underlying probability. The chapter therefore argues that such outputs should not be labeled as overconfident, since that implies a flaw in the model, whereas they are simply a byproduct of optimizing BCE. More accurate language would call the model highly decisive or strongly confident, and the output should be understood as a classification decision rather than an exact probability distribution.