In logistic regression, why is the binary cross-entropy loss function convex?

soup_sam

I am studying logistic regression for binary classification. The loss function used is cross-entropy . For a given input $x$ , if our model outputs $\hat{y}$ instead of $y$ , the loss is given by $$\text{L}_{\text{CE}}(y,\hat{y}) = -[y \log \hat{y} + (1 - y) (\log{1 - \hat{y}})]$$ Suppose there are $m$ such training examples, then the overall total loss function $\text{TL}_{\text{CE}}$ is given by $$\text{TL}_{\text{CE}} = \dfrac{1}{m}...

Originally asked by hanugm on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

The $L_{CE}$ that you provided is binary cross-entropy, the factor $y$ and $(1-y)$ is because $y$ is binary $({0,1})$ , careful with the name next time. The cross-entropy loss should have form: $$L_{CE}=-\displaystyle\sum_{i=1}^C y_i\log(\hat{y_i})$$ Where $C$ is the number of classes. Normally, the $y_i$ factor only is 1 when $i$ is the index of the correct class. Therefore, with each class, the function is just: $$f(x)=-log(x)$$ Now, to prove this one is convex, we have multiple ways, but my favorite one is computing the derivative and second derivative. $$\frac{\partial L}{\partial x}=-\frac{1}{x}\Rightarrow \frac{\partial^2 L}{\partial x^2}=\frac{1}{x^2}>0 \text{ for all }x\in(0,1]$$...

Full answer by CuCaRot: https://ai.stackexchange.com/a/33867 (CC BY-SA 4.0).

Need it done for your business? UphillTech.