_About Naive Bayes

tags: #ML/supervised/classification/nb

Naive Bayes is a probabilistic classifier based on Bayes' Theorem under a simple assumption which is the attributes/features are conditionally independent given the class label, hence 'naive' (naturally, features are related but we assume naivety), such that the probability of an example/observation belonging to a class given the set of features:

P (C = i | X) = \frac{P (X | C = i) * P (C = i)}{P (X)}

Formula Breakdown

$P (C = i | X)$ - Posterior probability of class = i given the input features $X$
$P (X | C = i)$ - Likelihood of observing the features given class = i
$P (C = i)$ - Prior probability of class = i
$P (X)$ - Marginal probability of observing the features, $X$

The class with the highest probability is then assigned as the predicted class.

Example: Suppose we have a dataset of animals with their corresponding features such as number of legs, presence of a tail, and whether they have a tail. We want to use this data to train a Naive Bayes classifier to predict whether a new animal is a dog or not based on its features:

First, we calculate the probabilities of each class (dog or not dog) in the training data (e.g. $P (d o g) = 0.4$ ).

For each feature (number of legs, presence of a tail, and barking), we calculate the conditional probabilities of each feature given the class (dog or not dog).

When we want to classify a new animal based on its features (e.g., 4 legs and a tail), we calculate the probability of it belonging to each class (dog or not dog) using Bayes' theorem.

$P (dog|has 4 legs & tail) = \frac{P (4 legs, tail|dog) * P (dog)}{P (4 legs, tail)}$

What type of classification can NB be used on?

Naive Bayes can be used for both binary and multiclass classification problems. It works by estimating the probability distribution of each feature for each class, and then computing the probability of a new data point belonging to each class using Bayes' theorem.