GDA

Gaussian Discriminant Analysis

I am still deriving the parameters post applying MLE.

\begin{aligned} ℓ (ϕ, μ_{0}, μ_{1}, Σ) & = \log \prod_{i = 1}^{m} p (x^{(i)}, y^{(i)}; ϕ, μ_{0}, μ_{1}, Σ) \\ = \log \prod_{i = 1}^{m} p (x^{(i)} | y^{(i)}; μ_{0}, μ_{1}, Σ) p (y^{(i)}; ϕ) \\ = \sum_{i = 1}^{m} \log p (x^{(i)} | y^{(i)}; μ_{0}, μ_{1}, Σ) p (y^{(i)}; ϕ) \end{aligned}

abstracting it we get:

\begin{aligned} ℓ (ϕ, μ_{k}, Σ) & = \sum_{i = 1}^{m} \log p (x^{(i)} | y^{(i)}; μ_{k}, Σ) p (y^{(i)}; ϕ) \\ = \sum_{i = 1}^{m} [- \frac{n}{2} \log 2 π - \frac{1}{2} \log | Σ | - \frac{1}{2} (x^{i} - μ_{k})^{T} Σ^{- 1} (x^{i} - μ_{k}) + y^{i} \log ϕ + (1 - y^{i}) \log (1 - ϕ)] \end{aligned}

Taking Derivative on that equation we get the following values:

ϕ = \frac{1}{m} \sum_{i = 1}^{m} 1 {y^{(i)} = 1}

μ_{k} = \frac{\sum_{i = 1}^{m} 1 {y^{(i)} = k} x^{(i)}}{\sum_{i = 1}^{m} 1 {y^{(i)} = k}}

Σ = \frac{1}{m} \sum_{i = 1}^{m} (x^{(i)} - μ_{y^{(i)}}) (x^{(i)} - μ_{y^{(i)}})^{T}

Posterior Probability in Gaussian Discriminant Analysis

We have the posterior probability of class 1 given the input $x$ as:

p (y = 1 | x; ϕ) = \frac{p (y = 1 | x; ϕ) p (x | μ_{1}, Σ)}{p (y = 1 | x; ϕ) p (x | μ_{1}, Σ) + p (y = 0 | x; ϕ) p (x | μ_{0}, Σ)}

Given the class priors and the likelihoods, we can write this as:

p (y = 1 | x; ϕ) = \frac{ϕ N (x | μ_{1}, Σ)}{ϕ N (x | μ_{1}, Σ) + (1 - ϕ) N (x | μ_{0}, Σ)}

Rewriting the above equation, we have:

p (y = 1 | x; ϕ) = \frac{1}{1 + \frac{(1 - ϕ) N (x | μ_{0}, Σ)}{ϕ N (x | μ_{1}, Σ)}}

Since the Gaussian distribution is a member of the exponential family, we can eventually express the ratio in the denominator as $\exp (θ^{T} x)$ , where $θ$ is a function of $ϕ, μ_{0}, μ_{1},$ and $Σ$ .