1. Multi-class vs Multi-label

What is the difference between a multiclass problem and a multilabel problem?

(https://stats.stackexchange.com/questions/11859/what-is-the-difference-between-multiclass-and-multilabel-problem)

I suspect the difference is that in multi-class problems the classes are mutually exclusive, whereas for multi-label problems each label represents a different classification task, but the tasks are somehow related (so there is a benefit in tackling them together rather than separately).  For example, in the famous leptograspus crabs dataset there are examples of males and females of two colour forms of crab.  You could approach this as a multi-class problem with four classes (male-blue, female-blue, male-orange, female-orange) or as a multi-label problem, where one label would be male/female and the other blue/orange.  Essentially in multi-label problems a pattern can belong to more than one class.

• in the multilabel case, one sample might be assigned more than one class.

• in the multiclass case, there are more than 2 classes in total.

As a side note, nothing prevents you from having a multilabel-multiclass classification problem.

See also:

https://stats.stackexchange.com/questions/319874/multi-label-or-multi-class-or-both

 2. Binary_crossentropy vs Categorical_crossentropy

Should I use a categorical cross entropy or binary cross entropy loss for binary predictions?

Binary cross-entropy is used for binary or multi-class, multi-label classifications, with sigmoid activation (in the last layer); whereas categorical cross entropy is for multi-class classification (where each example belongs to a single class) with softmax activation (in the last layer).

See also:

https://stackoverflow.com/questions/42081257/keras-binary-crossentropy-vs-categorical-crossentropy-performance

https://stackoverflow.com/questions/47877083/keras-binary-crossentropy-categorical-crossentropy-confusion

3. How to choose cross-entropy loss ?

https://stackoverflow.com/questions/47034888/how-to-choose-cross-entropy-loss-in-tensorflow

In functional sense, the sigmoid is a partial case of the softmax function, when the number of classes equals 2. Both of them do the same operation: transform the logits to probabilities.

In simple binary classification, there's no big difference between the two, however in case of multinomial classification, sigmoid allows to deal with non-exclusive labels (a.k.a. multi-labels), while softmax deals with exclusive classes.

See also:

http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html