Support Vector Machines

Useful reading:

• Ben-Hur and Weston. A User’s Guide to Support Vector Machines.

TL;DR

• Support Vector Machines (SVMs) are binary classifiers based on a simple idea: find a plane that divides the two classes. Then, to predict the class of a test point, see which side of the plane it falls on.

• This simple idea leads to some pretty interesting mathematics.

  • The dual optimization problem opens the door to kernel methods. This allows us to divide the classes with curved surfaces instead of flat planes.

  • The soft-margin SVM’s constrained optimization problem is actually equivalent to an unconstrained optimization problem with an interesting meaning.

• This simple idea does surprisingly well in practice.