Another Algorithm for Optimization : Newton's Method
- 1 min
What is the Classification Problem ?
In Machine Learning and statistics, classification is the problem of detecting / identifying to which of a set of categories (or sub-populations) a new observation belongs, on the basis of the training set of the data containing obseravations (or instances) whose category membership (or labels) are known. Examples of Classification Problem are assigning a given email to the “spam” or “non-spam” class,and assigning a diagnosis to a given patient based on observed characteristics of the patient (sex, blood pressure, presence or absence of certain symptoms, etc.). Classification is an example of pattern recognition.
In Machine Learning, classification is cosidered an instance of supervised learning, i.e., learning where a training set of correctly identified observations is available.
We could approach the classification problem ignoring the fact that y is discrete-valued, and use our old linear regression algorithm to try to predict y given x. However, it is easy to construct examples where this method performs very poorly. Intuitively, it also doesn’t make sense for h(x) to take values larger than 1 or smaller than 0 when we know that y ∈ {0, 1}.
To fix this, let’s change the form for our hypotheses h(x). We will choose
where
is called the logistic function or the sigmoid function.
The Newton’s Method can be used in place of Simple Logistic Regression Algorithm. It can be used to maximize the log likelihood function, l(θ).
References :
- http://mathworld.wolfram.com/NewtonsMethod.html
- https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization
- https://www.softcover.io/read/bf34ea25/math_for_finance/multivariable_methods
- https://en.wikipedia.org/wiki/Newton%27s_method