I say “hair-splitting,” but I think the philosophical difference between these fields is actually pretty profound.
- Attempts to answer questions about truth; epistemology. Rather idealistic.
- Proposes a set of possible models, and posits the existence of a “true” model within that set. This “true” model explains the observations.
- This description applies to frequentist and bayesian statistics.
- Both camps take their models seriously as descriptions of reality. They only differ in their description of epistemic state, and in their methods for generating that description.
- Attempts to quantify our epistemic state.
- Frequentists: confidence intervals, \(p\)-values
- Bayesians: posterior distributions; Bayes factors
- Machine Learning
- More pragmatic: what is useful?
- Seeks a model for accomplishing some task: e.g., prediction or decision making.
- Selects a model based on its performance at that task—fitness for purpose. (A splash of Darwinism.)
- Makes no pretense at determining truth—is fundamentally an engineering discipline.
- We may choose different models based on trade-offs between performance, available memory, available processing power, available data, or other details of the operating environment.
- Machine learning takes algorithmic concerns into account.
- Is there an efficient procedure for determining the best model?
- This is the result of its origins within computer science.
My personal inclinations can probably be guessed from these descriptions.