logit, sigmoid function, and probability

step-by-step through how the sigmoid function turns a logit into a probability, with both math and plain English at each step.

---

✅ Step 1: What is a logit?

The logit of a probability $p$ is defined as:

$$\text{logit}(p) = \log\left(\frac{p}{1 - p}\right)$$

This is called the log-odds, because it’s the natural log of the odds.

---

✅ Step 2: What if we are given the logit and want to get the probability?

Let’s say we are given a number $x$, and we know it’s a logit.

That means:

$$x = \log\left(\frac{p}{1 - p}\right)$$

We want to solve for $p$, the probability.

---

✅ Step 3: Exponentiate both sides

To get rid of the logarithm, we exponentiate both sides:

$$e^x = \frac{p}{1 - p}$$

This tells us the odds in terms of $x$.

---

✅ Step 4: Solve for $p$

Now we solve this equation for $p$.

Start with:

$$e^x = \frac{p}{1 - p}$$

Multiply both sides by $1 - p$:

$$e^x (1 - p) = p$$

Distribute the left-hand side:

$$e^x - e^x p = p$$

Now get all the $p$'s on one side:

$$e^x = p + e^x p$$

Factor out $p$ on the right-hand side:

$$e^x = p(1 + e^x)$$

Now divide both sides by $1 + e^x$:

$$p = \frac{e^x}{1 + e^x}$$

---

✅ Step 5: Simplify into the sigmoid form

We now simplify that expression:

$$p = \frac{e^x}{1 + e^x} = \frac{1}{\frac{1 + e^x}{e^x}} = \frac{1}{1 + e^{-x}}$$

That’s the sigmoid function:

$$p = \sigma(x) = \frac{1}{1 + e^{-x}}$$

---

✅ Final Result

So if you’re given a logit value $x$, the sigmoid function:

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

turns it into a probability $p \in (0, 1)$.

---

🔁 Why this is useful

• In logistic regression, we compute:

  $$x = w^T x + b \quad \text{(a linear score)}$$

• Then use the sigmoid to convert that score (logit) into a probability of class 1:

  $$p = \frac{1}{1 + e^{-x}}$$

This way, the model’s output is interpretable as a probability — suitable for binary classification.

Let me know if you want a plot or visualization to make it even clearer!