Understanding Bayesian Optimisation graph — Case study

taco_truck_tom

I came across the concept of Bayesian Occam Razor in the book Machine Learning: a Probabilistic Perspective . According to the book: Another way to understand the Bayesian Occam’s razor effect is to note that probabilities must sum to one. Hence $\sum_D' p(D' |m) = 1$ , where the sum is over all possible data sets. Complex models, which can predict many things, must spread their probability mass thinly, and hence will not obtain as large a...

Originally asked by user9947 on Stack Exchange (ai) (CC BY-SA 4.0).

Variant 957.

uphilltech

Great question — answering as the UphillTech team:

The original graph for the aforementioned Bayesian Optimisation is similar to the graph in these slides (slide 18) along with the calculations. So, according to the tutorial the graph shown should actually have the term $p(D|m)$ on the y-axis, thus making it a generative model.Now the graph starts to make sense, since a model with low complexity cannot produce very complex datasets and will be centred around 0, while very complex models can produce richer datasets which makes them assign probability thinly over all the datatsets (to keep $\sum_{D'}p(D'|m) = 1$ ).

Full answer by user9947: https://ai.stackexchange.com/a/18568 (CC BY-SA 4.0).

Want it for your business? UphillTech.