Think of all the False Positives

Recall that K-nn is a lazy learner. At training time the algorithm simply stores the training data - i.e. no calculations/training occurs. It is not until inference time when the algorithm checks the K nearest points to the unseen datapoint in question. All calculations occur at inference time, hence being called lazy rather than eager.

Underfitting is when the model lacks the capacity to fit the underlying pattern/trend of the data. A model that underfits a training set will perform no better on unseen data.

Underfitting is caused when the model lacks the capacity to fit the underlying trend/pattern of the data.

Confidence Intervals should be reported for the final model architecture. They are a prediction on how the final model will perform on unseen data. If this is calculated with a range of different models, this will clearly be an inaccurate prediction.

Think about which methods are deterministic.

Gradient descent - gradients are calculated and a step is taken based on the whole training step. This is a computationally heavy computation, as apposed to calculating the gradients and updating the parameters based on one sample. Stochastic gradient descent results in a very noising learning signal as the gradient is sensitive to the variability of each individual datapoint. The learning signal can be smoothed out by sampling groups of datapoints in mini-batch gradient descent. Here the gradient is averaged over all datapoints.

Think about local optima. Is there a mechanism in K-means to help it get out of local minima once the algorithm has converged?

The responsibilities, r_nK for a particular data point, n, represents the probability of that datapoint being generated by all the K mixture components. This is a probability distribution over all mixtures and therefore has to sum to 1. A mixture, k, however, can be responsible for generating the vast majority of data points, therefore, easily having a total responsibility over all datapoints over 1.

Check definitions