Regression & Classification with MNIST. _ day 4

 A Comprehensive Guide to Machine Learning: Regression and Classification with the MNIST DatasetIntroduction to Supervised Learning: Regression and ClassificationIn the realm of machine learning, supervised learning involves training a model on a labeled dataset, which means the dataset includes both input data and the corresponding output labels. Supervised learning tasks can be broadly categorized into two types: regression and classification.  Regression tasks aim to predict continuous numerical values. For example, predicting house prices based on various features such as location, size, and number of bedrooms. The output is a continuous value that can range over an infinite set of possible values. Common regression algorithms include linear regression, decision trees, and support vector regression.  Classification, on the other hand, deals with predicting discrete categorical values. The goal is to assign input data to one of several predefined classes. For instance, classifying emails as either spam or not spam, or recognizing handwritten digits as one of the digits from 0 to 9. The output is a discrete value representing the class label. Popular classification algorithms include logistic regression, support vector machines, decision trees, and neural networks.The MNIST Dataset: A Benchmark for ClassificationThe MNIST dataset is a widely used benchmark in the field of machine learning. It contains 70,000 images of handwritten digits, with each image being a 28×28 pixel grid. This results in 784 features per image. The dataset is split into 60,000 training images and 10,000 test images, each labeled with the corresponding digit (0-9). The MNIST dataset is often used to evaluate and compare the performance of different classification algorithms.Loading and Exploring the MNIST DatasetTo begin working with the MNIST dataset, we first load the data and inspect its structure. This involves checking the dimensions of the data arrays to ensure they match the expected format (70,000 samples, each with 784 features for the pixel values). Visualizing the data is also crucial. By plotting some of the images, we can confirm that the data is correctly loaded and understand the visual patterns associated with different digits.Training a Binary ClassifierBinary classification involves distinguishing between two classes. To illustrate this, we can train a classifier to identify the digit ‘5’ from all other digits. The process involves the following steps:Preparing the Data: We create a binary target vector where the label is 1 if the digit is ‘5’ and 0 otherwise. This transforms our multiclass problem into a binary classification problem.Training the Classifier: We use a machine learning algorithm, such as stochastic gradient descent (SGD), to train the model. The algorithm iteratively adjusts the model parameters to minimize the difference between the predicted and actual labels. During training, the model learns patterns and features that are indicative of the digit ‘5’, such as the loop and vertical line structure.Making Predictions: Once trained, the model can predict whether a new image is a ‘5’ or not by applying the learned decision boundary. This boundary separates the feature space into two regions, one for each class.Evaluating the ClassifierTo ensure the classifier performs well, we need to evaluate its performance using various metrics and techniques.Cross-ValidationThis technique assesses the model’s ability to generalize to an independent dataset. The data is split into multiple folds, and the model is trained and evaluated on different combinations of these folds. This helps in obtaining a more reliable estimate of the model’s performance.Confusion MatrixA confusion matrix provides a detailed breakdown of the classifier’s performance by comparing actual and predicted labels. It includes counts of true positives, false positives, true negatives, and false negatives. This matrix helps us understand the types of errors the classifier is making and how often they occur.Precision and RecallPrecision measures the accuracy of positive predictions, i.e., the proportion of true positive predictions out of all positive predictions. Recall measures the ability of the classifier to identify all positive instances, i.e., the proportion of true positive predictions out of all actual positives. These metrics are particularly useful for imbalanced datasets where some classes are much more frequent than others.Precision/Recall Trade-offBy adjusting the decision threshold, we can find a balance between precision and recall. Lowering the threshold increases recall but decreases precision, while raising the threshold does the opposite. Precision-recall curves help visualize this trade-off and select an optimal threshold based on the specific requirements of the application.ROC Curve and AUC ScoreThe ROC (Receiver Operating Characteristic) curve plots the true positive rate (recall) against the false positive rate at various threshold settings. The area under the ROC curve (AUC) provides a single number to summarize the model’s performance. A higher AUC indicates better performance. The ROC curve and AUC score are particularly useful for comparing different classifiers.Multiclass ClassificationAfter mastering binary classification, we extend the model to distinguish between all ten digits (0-9). This is known as multiclass classification. Two common strategies for multiclass classification are:One-vs-All (OvA)This approach involves training a separate binary classifier for each digit. Each classifier distinguishes one digit from all other digits. During prediction, the classifier with the highest score determines the final class label.One-vs-One (OvO)This approach involves training a binary classifier for every pair of digits. For ten digits, this results in 45 classifiers. During prediction, a voting scheme determines the final class label based on the outputs of all classifiers.Error AnalysisAnalyzing errors is crucial for improving model performance. We use confusion matrices to understand the types of errors the classifier makes. For example, a confusion matrix might reveal that the classifier frequently confuses ‘3’ with ‘8’. By visualizing these misclassified images, we can identify patterns and features that lead to errors. This insight helps us refine the model and improve its accuracy.Multilabel and Multioutput ClassificationIn some tasks, each instance can belong to multiple classes simultaneously (multilabel classification) or have multiple outputs (multioutput classification).Multioutput ClassificationThis involves predicting multiple outputs for each instance. For example, predicting both the digit and the intensity level of the handwriting. The model outputs a vector of predictions, and the performance is evaluated using appropriate metrics for each output.ConclusionIn this comprehensive guide, we have explored the essential steps involved in building, evaluating, and improving a machine learning classifier using the MNIST dataset. We started with an introduction to regression and classification, moved on to exploring and visualizing the dataset, and then delved into training classifiers and evaluating their performance using various metrics. Finally, we discussed advanced topics like multiclass, multilabel, and multioutput classification, along with error analysis. By understanding these concepts and applying these techniques, you can develop robust machine learning models and tackle various classification problems effectively. Whether you are working with simple binary classification tasks or complex multiclass and multioutput problems, the methods and insights discussed here will serve as a solid foundation for your machine learning journey. You can understand what we discussed better by understand the code below and its results. Now it’s time to code to practice what we explained so far , do not forget to check the images which are the results at the end! # Part 1: Importing Libraries import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import fetch_openml from sklearn.linear_model import SGDClassifier from sklearn.model_selection import cross_val_score, cross_val_predict from sklearn.metrics import confusion_matrix, precision_score, recall_score, precision_recall_curve, roc_curve, roc_auc_score from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.multioutput import ClassifierChain from sklearn.svm import SVC # Importing SVC # Part 2: Loading and Exploring the MNIST Dataset # Load the MNIST dataset mnist = fetch_openml('mnist_784', as_frame=False) X, y = mnist.data, mnist.target # Check the shape of the data print(X.shape) # Output: (70000, 784) print(y.shape) # Output: (70000,) # Visualize one of the digits some_digit = X[0] some_digit_image = some_digit.reshape(28, 28) plt.imshow(some_digit_image, cmap='binary') plt.axis('off') plt.show() # Part 3: Training a Binary Classifier # Create binary target vector: 1 for digit '5', 0 for all other digits y = y.astype(np.int8) # Convert target to integer y_train_5 = (y == 5) # Split the data into training and test sets X_train, X_test = X[:60000], X[60000:] y_train, y_test = y[:60000], y[60000:] y_train_5, y_test_5 = y_train == 5, y_test == 5 # Train a SGDClassifier sgd_clf = SGDClassifier(random_state=42) sgd_clf.fit(X_train, y_train_5) # Part 4: Evaluating the Classifier # Perform cross-validation cross_val_scores = cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring='accuracy') print("Cross-validation accuracy:", cross_val_scores) # Make cross-validated predictions y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3) # Confusion Matrix conf_mx = confusion_matrix(y_train_5, y_train_pred) print("Confusion Matrix:n", conf_mx) # Precision and Recall precision = precision_score(y_train_5, y_train_pred) recall = recall_score(y_train_5, y_train_pred) print("Precision:", precision) print("Recall:", recall) # Part 5: Precision/Recall Trade-off # Get decision scores y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function") # Compute precision-recall curve precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores) # Plot precision-recall curve plt.plot(thresholds, precisions[:-1], "b–", label="Precision") plt.plot(thresholds, recalls[:-1], "g-", label="Recall") plt.xlabel("Threshold") plt.legend(loc="center right") plt.ylim([0, 1]) plt.show() # Part 6: ROC Curve and AUC Score # Compute ROC curve fpr, tpr, thresholds = roc_curve(y_train_5, y_scores) # Plot ROC curve plt.plot(fpr, tpr, linewidth=2) plt.plot([0, 1], [0, 1], 'k–') plt.axis([0, 1, 0, 1]) plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.show() # Compute ROC AUC score roc_auc = roc_auc_score(y_train_5, y_scores) print("ROC AUC Score:", roc_auc) # Part 7: Multiclass Classification # One-vs-All (OvA) Strategy ovr_clf = OneVsRestClassifier(SVC(random_state=42)) ovr_clf.fit(X_train, y_train) print("OvA Classifier predictions for first digit:", ovr_clf.predict([some_digit])) # One-vs-One (OvO) Strategy ovo_clf = OneVsOneClassifier(SVC(random_state=42)) ovo_clf.fit(X_train, y_train) print("OvO Classifier predictions for first digit:", ovo_clf.predict([some_digit])) # Part 8: Error Analysis # Confusion Matrix for multiclass classifier y_train_pred_full = cross_val_predict(ovr_clf, X_train, y_train, cv=3) conf_mx_full = confusion_matrix(y_train, y_train_pred_full) plt.matshow(conf_mx_full, cmap=plt.cm.gray) plt.show() # Part 9: Multilabel and Multioutput Classification # Multilabel classification: Creating multiple binary labels y_train_large = (y_train >= 7) y_train_odd = (y_train % 2 == 1) y_multilabel = np.c_[y_train_large, y_train_odd] # Training a KNeighborsClassifier for multilabel classification knn_clf = KNeighborsClassifier() knn_clf.fit(X_train, y_multilabel) # Multioutput classification: Predicting multiple outputs chain_clf = ClassifierChain(SVC(random_state=42)) chain_clf.fit(X_train, y_multilabel) print("Predictions for first digit (multilabel):", knn_clf.predict([some_digit])) print("Predictions for first digit (multioutput):", chain_clf.predict([some_digit])) Lets explain the code provided part by part therefore,  you can understand better why we used this code here step by step :Part 1: Code with Detailed Explanation (Steps 1 to 6)Importing Librariesimport numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import cross_val_score, cross_val_predict
from sklearn.metrics import confusion_matrix, precision_score, recall_score, precision_recall_curve, roc_curve, roc_auc_score
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multioutput import ClassifierChain
from sklearn.svm import SVC # Importing SVC
Explanation:In this section, we import essential libraries and modules for numerical operations, data visualization, and machine learning tasks:numpy: A library for numerical operations, providing support for arrays and matrices.matplotlib.pyplot: A plotting library for creating static, animated, and interactive visualizations.sklearn.datasets: Provides functions to fetch datasets like MNIST.sklearn.linear_model: Contains the SGDClassifier for training linear models using stochastic gradient descent.sklearn.model_selection: Offers tools like cross_val_score and cross_val_predict for model evaluation using cross-validation.sklearn.metrics: Provides functions for evaluating model performance, including confusion matrix, precision, recall, precision-recall curve, ROC curve, and AUC score.sklearn.multiclass: Contains strategies for multiclass classification, such as OneVsRestClassifier and OneVsOneClassifier.sklearn.neighbors: Includes the KNeighborsClassifier for k-nearest neighbors classification.sklearn.multioutput: Provides tools for multioutput classification, including ClassifierChain.sklearn.svm: Contains the SVC (Support Vector Classifier) for classification tasks.Loading and Exploring the MNIST Dataset# Load the MNIST dataset
mnist = fetch_openml('mnist_784', as_frame=False)
X, y = mnist.data, mnist.target

# Check the shape of the data
print(X.shape) # Output: (70000, 784)
print(y.shape) # Output: (70000,)

# Visualize one of the digits
some_digit = X[0]
some_digit_image = some_digit.reshape(28, 28)
plt.imshow(some_digit_image, cmap='binary')
plt.axis('off')
plt.show()
Explanation:We load the MNIST dataset using fetch_openml. The dataset consists of 70,000 images of handwritten digits, each represented as a 28×28 pixel grid (resulting in 784 features per image).We print the shape of the data to ensure it matches the expected dimensions.We visualize one of the digits to verify the data is loaded correctly and to understand the visual patterns associated with different digits.Training a Binary Classifier# Create binary target vector: 1 for digit '5', 0 for all other digits
y = y.astype(np.int8) # Convert target to integer
y_train_5 = (y == 5)

# Split the data into training and test sets
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]
y_train_5, y_test_5 = y_train == 5, y_test == 5

# Train a SGDClassifier
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_5)
Explanation:We create a binary target vector where the label is 1 if the digit is ‘5’ and 0 otherwise, transforming our multiclass problem into a binary classification problem.We split the dataset into training and test sets. The first 60,000 samples are used for training, and the remaining 10,000 are used for testing.We train an SGDClassifier (Stochastic Gradient Descent Classifier) on the training data. This classifier is well-suited for large datasets and performs online learning, adjusting model parameters iteratively.How SGDClassifier Works:Initialization: Initialize the model parameters (weights) to small random values.Training: For each training sample, the classifier calculates the prediction, computes the error (difference between predicted and actual label), and updates the weights to minimize this error. This process is repeated for each sample in the training set.Iterations: The training process involves multiple iterations (epochs) over the training data to refine the model parameters.Gradient Descent: The classifier uses gradient descent to find the optimal parameters by moving in the direction that reduces the error.Evaluating the Classifier# Perform cross-validation
cross_val_scores = cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring='accuracy')
print("Cross-validation accuracy:", cross_val_scores)

# Make cross-validated predictions
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)

# Confusion Matrix
conf_mx = confusion_matrix(y_train_5, y_train_pred)
print("Confusion Matrix:n", conf_mx)

# Precision and Recall
precision = precision_score(y_train_5, y_train_pred)
recall = recall_score(y_train_5, y_train_pred)
print("Precision:", precision)
print("Recall:", recall)
Explanation:Cross-Validation: We use cross_val_score to assess the model’s performance. Cross-validation splits the training data into multiple folds, trains the model on different combinations of these folds, and evaluates its accuracy.Cross-Validated Predictions: We use cross_val_predict to obtain predictions for each training instance. This function returns predictions that are generated by a model trained on different folds than the instance being predicted.Confusion Matrix: We compute the confusion matrix using confusion_matrix to understand the types of errors the classifier makes. The confusion matrix includes counts of true positives (correctly identified ‘5’s), false positives (incorrectly identified as ‘5’s), true negatives (correctly identified as not ‘5’s), and false negatives (incorrectly identified as not ‘5’s).Precision and Recall: We calculate precision and recall scores to evaluate the model’s ability to identify positive instances (digits ‘5’) and the accuracy of its positive predictions. Precision is the ratio of true positive predictions to the total positive predictions, while recall is the ratio of true positive predictions to the total actual positives.Precision/Recall Trade-off# Get decision scores
y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")

# Compute precision-recall curve
precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)

# Plot precision-recall curve
plt.plot(thresholds, precisions[:-1], "b–", label="Precision")
plt.plot(thresholds, recalls[:-1], "g-", label="Recall")
plt.xlabel("Threshold")
plt.legend(loc="center right")
plt.ylim([0, 1])
plt.show()
Explanation:Decision Scores: We obtain the decision scores for each instance using cross_val_predict…

Thank you for reading this post, don't forget to subscribe!