Sklearn Instructions: Lesson 2
In the realm of machine learning, visualizing concepts using tools such as Validation Curves and Learning Curves can be invaluable. These tools provide insights into a model's performance and help diagnose issues like overfitting and underfitting.
Scikit-learn, a popular Python library, offers the training score and test score as key indicators for this purpose. Overfitting is signalled when the training score is high, but the test score is significantly lower. This large gap indicates that the model has memorised the training data, including noise, and does not generalise well to new data. Conversely, underfitting occurs when both training and test scores are low, suggesting that the model is too simple or inadequately trained.
In a typical scikit-learn workflow, after splitting the dataset into training and test sets (commonly using ), the model is trained on the training set and then evaluated on both sets. Comparing these scores helps identify which issue is present and guides decisions such as increasing model complexity, applying regularization, or gathering more data.
| Scenario | Training Score | Test Score | Interpretation | |--------------------------------|----------------|------------|-------------------------| | High training, low test | High | Low | Overfitting | | Low training, low test | Low | Low | Underfitting | | High training, high test | High | High | Good fit, generalises well |
A Validation Curve plots the test score and train score as a function of model complexity. As the model's complexity increases, the coefficients and predictions can become significantly different, leading to high variance. In cases with a very high number of samples, the model may approach the Bayes error rate. However, with a fixed complexity of degree=2, the train score and test scores converge toward the same value for very high numbers of samples.
On the other hand, a Learning Curve plots the test score and train score as a function of input size. As the number of samples increases, the train error increases but the test error decreases. This reflects the trade-off between bias and variance, a key concept in machine learning. If the number of samples increases significantly, the train and test error will almost converge together. With few samples, both train and test errors are crucial.
Bias refers to the extent to which the fitted model deviates from the perfect model, and is relatively constant regardless of the input. Variance, on the other hand, refers to the variation a model's response changes with the train set. The Bayes error rate is the error of the best model trained on unlimited data, limited only by noise in the data.
By inspecting the performance of a model through examining how scores change with the number of sample data, we can make informed decisions about the model's complexity and the amount of data needed for a good fit. Both Validation Curves and Learning Curves can be generated easily with scikit-learn.
The goal is to find the right balance that minimises both bias and variance, leading to a model that generalises well to new, unseen data. For instance, a simple polynomial fit of a single variable can have either high bias or high variance, depending on the degree allowed for the model.
