Learning from examples
So far you wrote the knowledge yourself: the heuristic, the rules, the constraints. Machine learning gets the program to work it out from examples instead. A decision tree is the most readable kind of learned model — a flowchart of questions ending in answers — and you can see exactly why it decides what it does.
Russell and Norvig's example: will we wait for a table at a restaurant? Given past visits — how busy it was, whether we were hungry, the type of food — learn a tree that predicts the decision. The algorithm is Ross Quinlan's ID3 (1986), and its successors still power tools like random forests and gradient-boosted trees.
Entropy
A good first question splits the examples into groups that are as pure as possible — all Yes or all No. Entropy measures impurity in bits: H = −Σ p log₂ p over the label proportions. A group that all agrees has entropy 0; a 50/50 split has entropy 1, as uncertain as a fair coin; a 75/25 split is in between, about 0.81.
function entropy(labels) {
const counts = {};
for (const l of labels) counts[l] = (counts[l] ?? 0) + 1;
let h = 0;
for (const c of Object.values(counts)) {
const p = c / labels.length;
h -= p * Math.log2(p);
}
return h;
}