--- title: Stop Using Accuracy to Evaluate Your Classification Models description: How to Interpret a Confusion Matrix and When to use Different Evaluation Metrics image: https://dataknowsall.com/hubfs/modeleval_1.png --- [Skip to content](https://dataknowsall.com/blog/modeleval.html#main-content) [![Lambda with Text - Wide](https://dataknowsall.com/hs-fs/hubfs/Lambda%20with%20Text%20-%20Wide.png?width=3068&height=301&name=Lambda%20with%20Text%20-%20Wide.png)](https://dataknowsall.com) - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) Open main navigation Close main navigation - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) - [Contact us](https://dataknowsall.com/contact) [Contact us](https://dataknowsall.com/contact) [All posts](https://dataknowsall.com/blog/all) March 4, 2024 # Stop Using Accuracy to Evaluate Your Classification Models ![Picture of Brian Roepke](https://app.hubspot.com/settings/avatar/67adab425dfba5d6fa7e242d82fbe2c4) By  [Brian Roepke](https://dataknowsall.com/blog/author/brian-roepke)  ·  3 minute read ## Model Evaluation for Binary Classification Understanding how to evaluate models is a critical part of any Machine Learning (ML) project. Many times when we're starting with ML or when having conversations with people that aren't as well versed in ML, the term **Accuracy** is thrown out as a generalization of the model's performance. The problem with talking about the **Accuracy**of a model is that it specifically refers to a calculation that divides the number of observations it correctly classified vs. the total number in the test set. While **Accuracy** is acceptable for model performance when your dataset is perfectly balanced (equal positive and negative classes), the reality is that this is rarely the case. There are methods to deal with imbalanced data, which I'll talk about in a later post, but for now, let's understand how to interpret the results of a model's performance properly. **More:** For information on dealing with **imbalanced data**, check out my other post: [Working with Imbalanced Data](http://localhost:8000/imbalanced.html). ## Interpreting a Confusion Matrix After a model is trained and tested on data (Scikit-Learn has a great write-up of this process[1](http://localhost:8000/modeleval.html#fn:CROSS)), you can generate a **Confusion Matrix** to give a view of how observations the model misclassified. A Confusion Matrix is a 2x2 table (for binary classification) that plots the **actual** values vs. the **predicted** values on its axes. Once you understand how to interpret it, it is easy and intuitive to see your model's performance. ![modeleval_1](https://dataknowsall.com/hs-fs/hubfs/modeleval_1.png?width=800&height=731&name=modeleval_1.png) *Image by Author. * The matrix shows correctly classified observations in the top left and bottom right quadrants, while misclassified are in the other corners. Thinking about these as rows or columns independently helps you interpret each class. See the [Scikit-Learn](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) documentation for how to display a Confusion Matrix. ## Evaluation Metrics 1. **Accuracy:** This is the percentage of observations that the model correctly predicted across both classes. Be very careful with imbalanced data. You can have very high accuracy, but the performance of the minority class can be horrible. 2. **Precision:** Tells you how reliable the prediction is for the class. e.g., when the precision score is high, and the model predicts the class, you can trust it is correct. 3. **Recall:** Tells you how well the model can predict the class. A high recall with low Precision means that it is performing well in the class but may include other classes in the results. 4. **F1 Score:** Calculates the harmonic mean of Precision and Recall combining them into a single metric. The following image is the best way that I've seen to illustrate how each calculation is performed. It can also help you, once you understand what you're trying to optimize for, which type of metric you should use. ![modeleval_2](https://dataknowsall.com/hs-fs/hubfs/modeleval_2.png?width=800&height=615&name=modeleval_2.png) *Image by Author. * For example, if you're looking to minimize False Positives for Class 1, then **Class 1 Precision** is the right metric for you (*Dark Orange and Light Orange in the Image*). **Note:** *Precision* is the relationship of the *column* in the matrix, and *recall* is the *row* in the matrix. ## Optimize for the Outcome You Want A critical part of picking the right evaluation metric is to decide what is the most important outcome you want. I think of this broken down into three things: 1. **Minimize False Positives:** A use case for this might be a spam filter. Ideally, you never want to mark a real message as spam and have it filed into a spam folder. The potential for a user to miss an important message is high; therefore, you want to reduce false positives and choose a model with *high Precision*. 2. **Minimize False Negatives:** The best example here for me is medical screening. It would be incredibly bad if the algorithm had false negatives with the patient, but they were sick. It would be best to flag potential issues and have an expert follow up. In this case, you want to focus on a *high Recall*. 3. **Both:** If your use case doesn't have a bias to one or the other, then choose *F1* for a balanced metric for *both* Precision and Recall. ## Confustion Matrix in Practice Here is a recent Confusion Matrix from a Random Forest algorithm I trained. My objective was to predict when a user would convert from a trial account to a paid account for a SaaS offering. This example is created from **Scikit-Learn** and is also showing the various evaluation metrics above. In my case, the *positive* class (`1`) was those that converted to paid, and the *negative* class was for those that failed to convert. Take a look for yourself and see how you interpret the results. ![modeleval_3](https://dataknowsall.com/hs-fs/hubfs/modeleval_3.png?width=486&height=641&name=modeleval_3.png) ## Code To output the above confusion matrix, you can utilize the following code. ``` from sklearn.metrics import classification_reportfrom sklearn.metrics import ConfusionMatrixDisplay# print classification reportprint(classification_report(y_test, y_pred, digits=3))# show confusion matrixConfusionMatrixDisplay.from_predictions(y_test, y_pred, cmap=plt.cm.Blues)plt.show() ``` See the [Scikit-Learn documentation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) for more information on displaying a Confusion Matrix. ## Conclusion A confusion matrix is a powerful tool in your toolbox for evaluating both binary and multi-class classification models’ performance. The trick, however, is not just to understand how to interpret the matrix; it is to understand the correct evaluation metric to use truly. Remember not to default to accuracy; there is an appropriate metric for your desired outcome! Enjoy! Share: [facebook-f icon](http://www.facebook.com/share.php?u=https://dataknowsall.com/blog/modeleval.html) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://dataknowsall.com/blog/modeleval.html) [twitter icon](https://twitter.com/intent/tweet?url=https://dataknowsall.com/blog/modeleval.html) [pinterest-p icon](http://pinterest.com/pin/create/link/?url=https://dataknowsall.com/blog/modeleval.html) [envelope icon](mailto:?body=https://dataknowsall.com/blog/modeleval.html) ## Related posts [![Model Selection](https://dataknowsall.com/hubfs/modelselection_01.png)](https://dataknowsall.com/blog/modelselection.html) [Machine Learning](https://dataknowsall.com/blog/tag/machine-learning) ### [Demystify Machine Learning Model Selection, a Step by Step Guide](https://dataknowsall.com/blog/modelselection.html) [![Confusion Matrix](https://dataknowsall.com/hubfs/trialfire_cm.png)](https://dataknowsall.com/blog/trialfire.html) [Machine Learning](https://dataknowsall.com/blog/tag/machine-learning) ### [A Real-World Approach with XGBoost on Marketing Attribution Data](https://dataknowsall.com/blog/trialfire.html) [![classification](https://dataknowsall.com/hubfs/classification_1.png)](https://dataknowsall.com/blog/classification.html) [Machine Learning](https://dataknowsall.com/blog/tag/machine-learning) ### [Everything You Need to Know to Build an Amazing Binary Classifier](https://dataknowsall.com/blog/classification.html) [![Lambda Google Workspace](https://dataknowsall.com/hs-fs/hubfs/Lambda%20Google%20Workspace.png?width=320&height=132&name=Lambda%20Google%20Workspace.png "Lambda Google Workspace")](https://www.dataknowsall.com) Specialists in Data Engineering, Machine Learning, and Analytics Consultation [linkedin-in icon](https://www.linkedin.com/in/broepke/) [twitter icon](https://twitter.com/broepke) [Follow us on Facebook](https://github.com/broepke) **Pages** - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) ### Get Updates In Your Inbox Copyright © 2026, Data Knows All ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Brian Roepke", "url" : "https://dataknowsall.com/blog/author/brian-roepke" }, "dateModified" : "2024-03-04T18:06:53.149Z", "datePublished" : "2021-11-07T15:55:00.000Z", "headline" : "Stop Using Accuracy to Evaluate Your Classification Models", "image" : [ "https://dataknowsall.com/hubfs/modeleval_1.png" ], "mainEntityOfPage" : { "@id" : "https://dataknowsall.com/blog/modeleval.html", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://dataknowsall.com/hubfs/Lambda%20Google%20Workspace.png" }, "name" : "Data Knows All, LLC" } } ``` ```json { "@context" : "http://schema.org", "@type" : "Article", "author" : { "@type" : "Person", "name" : [ "Brian Roepke" ], "url" : "https://dataknowsall.com/blog/author/brian-roepke" }, "datePublished" : "2021-11-07T15:55:00+0000", "description" : "How to Interpret a Confusion Matrix and When to use Different Evaluation Metrics", "headline" : "Stop Using Accuracy to Evaluate Your Classification Models", "image" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/modeleval_1.png", "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/Lambda%20Google%20Workspace.png" }, "name" : "Data Knows All" }, "url" : "https://dataknowsall.com/blog/modeleval.html" } ```