Ensemble Methods

Useful reading:

• Dietterich. Machine Learning Research: Four Current Directions. 1997.

  (Yes, it is ironic that I reference a paper from 1997 which has “Current Directions” in its title. It actually gives a really helpful overview of ensemble methods, though).

• Random Forest Wikipedia article

  (One important topic not covered in the “Current Directions” paper)

TL;DR

• Under some common circumstances, a group (“ensemble”) of fairly good models will outperform one smarter model.

• Sufficient conditions for this “group effect” to happen:

  • The members of the ensemble are sufficiently diverse—they make mistakes in an uncorrelated fashion.

  • Each member of the ensemble meets some minimal threshold of performance on its own.