I just found the following interesting opinions of Zubin Ghahramani in the presentation he gave at the Machine Learning Summer School 2005:
- I have no idea why anyone would want to use non-subjective priors. Objective priors are fraught with inconsistencies and no modeling is truly objective anyway. If you want robustness make sure your prior captures a wide range of reasonable outcomes and use decision theory to capture your losses.
- Bayesian methods don’t over fit, because they don’t fit anything! Approximate Bayesian methods can have failure modes that look like overfitting.
- Anything you can do easily with an SVM you can do with a Gaussian Process better.
- Learning theory is useful to analyze bounds on the performance of algorithms but I’m not sure it should be used to design algorithms.
- Algorithms should be designed to be sensible given the problem at hand, ignoring prior knowledge seems very silly.
- Well designed MCMC methods can sometimes be much faster and perform better than optimization algorithms.
- MAP methods, i.e. using a log prior as a regularizer, are not Bayesian.