--- title: Up Your Game in Social Media Sentiment Analysis description: Understand how your customers feel about your products and services image: https://dataknowsall.com/hubfs/sentiment_all.png --- [Skip to content](https://dataknowsall.com/blog/sentiment.html#main-content) [![Lambda with Text - Wide](https://dataknowsall.com/hs-fs/hubfs/Lambda%20with%20Text%20-%20Wide.png?width=3068&height=301&name=Lambda%20with%20Text%20-%20Wide.png)](https://dataknowsall.com) - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) Open main navigation Close main navigation - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) - [Contact us](https://dataknowsall.com/contact) [Contact us](https://dataknowsall.com/contact) [All posts](https://dataknowsall.com/blog/all) March 5, 2024 # Up Your Game in Social Media Sentiment Analysis ![Picture of Brian Roepke](https://app.hubspot.com/settings/avatar/67adab425dfba5d6fa7e242d82fbe2c4) By  [Brian Roepke](https://dataknowsall.com/blog/author/brian-roepke)  ·  4 minute read ## What is Sentiment Analysis? Sentiment analysis helps understand the tone of text data, positive, negative, or neutral. Capturing sentiment can help organizations better understand the **Voice of Customer (VOC)** and even direct product development to improve functionality. Frequently, sentiment analysis is lexicon and rule-based, meaning it performs a lookup on words tagged by humans as positive, negative, or neutral. There are more advanced models available such as BERT, which was open-sourced by Google in 2018 This post will cover the more traditional lexicon-based approach; we'll take a look at the deep learning models later. You need to be familiar with new terms to interpret the results. They are simple concepts but understanding them upfront will help you follow along better. - **Polarity**: A value expressing the degree of emotion of a sentence from negative to positive. A value within the range \[-1.0, 1.0\] (negative sentiment => -1.0, neutral => 0.0, positive sentiment => 1.0) - **Subjectivity**: A subjective sentence expresses personal feelings, views, or beliefs. The subjectivity is a value within the range \[0.0, 1.0\] where 0.0 is very objective, and 1.0 is subjective. ## Twitter Data For this example, we're going to use data extracted from Twitter. Twitter is an excellent source for **VOC** analysis. You can search for a product name, a hashtag, mentions, or a company name. For Python, we can use the [Tweepy](https://docs.tweepy.org/en/stable/#) wrapper of the [Twitter API](https://developer.twitter.com/en). In searching for great examples of data that would be **highly positive** or **highly negative**, I just happened to be writing this and downloading data the moment the [Ghislaine Maxwell](https://www.nytimes.com/2021/12/29/nyregion/ghislaine-maxwell-guilty-verdict.html) trial verdict was being read and figured that would be a great place to start for negative. Who would be the opposite, where nearly all tweets would be positive? I chose [Taylor Swift](https://www.youtube.com/watch?v=FuXNumBwDOM). I won't cover the actual code I used to extract the Tweets, but all of it is available here if you would like to try this on your own: [Tweet Extraction Notebook](https://github.com/broepke/SentimentAnalysis/blob/main/twitter.ipynb) As usual, let's import the libraries we'll be using and import our data to get started. ``` import pandas as pdfrom textblob import TextBlobfrom vaderSentiment.vaderSentiment import SentimentIntensityAnalyzerimport seaborn as snsimport matplotlib.pyplot as plt ``` ``` swift = pd.read_pickle('swift.pkl')maxwell = pd.read_pickle('maxwell.pkl') ``` ## Sentiment Analysis with TexBlob **TexBlob** is a package I like to use for quick NLP projects. It's a simple Python API that supports common tasks like sentiment analysis. It contains two different sentiment analyzers. The first is called [`PatternAnalyzer`](https://github.com/clips/pattern), and the other is NLTKs `NaiveBayesAnalyzer`, which is trained on a movie reviews corpus. Getting the **polarity** (or **subjectivity**) score from TextBlob is simple. We can use Panda's `apply` method to get the \`sentiment.polarity for each tweet. ``` # Apply the Polarity Scoring from TextBlobswift['blob'] = swift['text'].apply(lambda x: TextBlob(x).sentiment.polarity)swift.head() ``` ``` TWEET POLARITY0 RT @TSwiftFTC: 🥇 According to @HITSDD, Taylor ... 1.0000001 RT @taylorr_media: Taylor Swift - All Too Well... 0.0000002 Taylor Swift and Ed Sheeran music mainly. And ... 0.1666673 RT @soitgoes: Taylor Swift didn't write:... 0.2000004 Suporte list: Nial Horan, Demi Lovato, Taylor ... 0.000000 ``` As simple as that, we have our polarity scores. We can see in the above examples that we have some ranging from neutral to the max positive value of `1.0`. Quite often, your text will have a neutral sentiment. Since neutral tweets do not add any information for the analysis, we can remove them and leave only those positive and negative. ``` # Get rid of neutral sentimentfilter = swift['blob'] != 0swift = swift[filter] ``` And next plot our results. I'm using the `histplot` from [Seaborn](https://seaborn.pydata.org/generated/seaborn.histplot.html). Histograms are ideal for visualizing polarity scores since they can nicely be binned into the range of -1.0 to 1.0. ``` sns.histplot(swift, x='blob', color="#7C8C58", bins=10)plt.title("Taylor Swift Sentiment Distribution with TextBlob")plt.xlabel("Sentiment")plt.ylabel("") ``` Below, you can see how the Taylor Swift tweets are mostly positive, while the Maxwell tweets are mostly negative. ![sentiment_swift_blob](https://dataknowsall.com/hs-fs/hubfs/sentiment_swift_blob.png?width=2400&height=1500&name=sentiment_swift_blob.png) ![sentiment_maxwell_blob](https://dataknowsall.com/hs-fs/hubfs/sentiment_maxwell_blob.png?width=2400&height=1500&name=sentiment_maxwell_blob.png)   ## Social Media Sentiment Analysis with Vader TextBlob is not optimized for shorter text or social media posts. It's better suited for longer, clean strings written in common language patterns. However, tools tuned explicitly to social media have emerged that understand how to deal with the language patterns used in social media. One such is **VADER** (*Valence Aware Dictionary and sEntiment Reasoner*) library. Some of the things VADER handles specifically for social media are: - **Booster Words**: Words that increase the sentiment like **great** or **love** - **ALL CAPS**: Words that are all caps are often used to amplify emotion (**GREAT**) - **Punctuation**: Marks such as **!** or **!!!** will increase the sentiment - **Slang**: Words such as **SUX** or **BFF** are interpreted correctly - **Emoji**: **UTF-8** encoded emojis are handled correctly **Note:** *strings mustn't be [cleaned](https://dataknowsall.com/blog/textcleaning.html) like you normally might with other NLP methods. There is much context that VADER will take away from the unprocessed text.* Before we jump into our Twitter data, let's take a look at how a couple of strings containing only emoji are processed. ``` positive = "💕 😁 🎉"negative = "💔 😬 🙁" ``` ``` {'neg': 0.0, 'neu': 0.375, 'pos': 0.625, 'compound': 0.875}{'neg': 0.514, 'neu': 0.203, 'pos': 0.284, 'compound': -0.3415} ``` The two strings would have returned `0.0` with TextBlob, with **VADER** they show `compound` scores that appear to be in line with the emotion that the emojis represent. Let's do the same process as above but utilize the VADER library. We'll use the same `apply` method on the Data Frame. ``` # Add the polarity scoresswift['vader'] = swift['text'].apply(lambda x: sid.polarity_scores(x)) ``` ``` {'neg': 0.0, 'neu': 0.793, 'pos': 0.207, 'compound': 0.7184} ``` The resulting column is actually a **dictionary** of different **key-value pairs**. The key-value that represents the TextBlob `sentiment` is the one called `compound`. We can extract it with the following function applied to our dataframe: ``` # Extract only the compound score valueswift['compound'] = swift['vader'].apply(lambda score_dict: score_dict['compound']) ``` After plotting our results again, we can see that VADER better represents the sentiment. Scores for Swift are more positively biased, and Maxwell is more negatively biased. VADER captures more emotion than TextBlob in the Tweets, whether emojis or other common ways people emphasize their feelings. ![sentiment_swift_vader](https://dataknowsall.com/hs-fs/hubfs/sentiment_swift_vader.png?width=2400&height=1500&name=sentiment_swift_vader.png) ![sentiment_maxwell_vader](https://dataknowsall.com/hs-fs/hubfs/sentiment_maxwell_vader.png?width=2400&height=1500&name=sentiment_maxwell_vader.png) ## Conclusion There you have it! Voice of Customer analysis of sentiment on Social Media data. This post is just the start of what you can do and how you might leverage this information; however, it's a powerful tool in your toolbox to understand your customers and their emotions better. The full code for Twitter data extraction and Sentiment analysis is available on [GitHub](https://github.com/broepke/SentimentAnalysis) Share: [facebook-f icon](http://www.facebook.com/share.php?u=https://dataknowsall.com/blog/sentiment.html) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://dataknowsall.com/blog/sentiment.html) [twitter icon](https://twitter.com/intent/tweet?url=https://dataknowsall.com/blog/sentiment.html) [pinterest-p icon](http://pinterest.com/pin/create/link/?url=https://dataknowsall.com/blog/sentiment.html) [envelope icon](mailto:?body=https://dataknowsall.com/blog/sentiment.html) ## Related posts [![Mongo Free Tier](https://dataknowsall.com/hubfs/mongo_01.png)](https://dataknowsall.com/blog/mongo.html) [Data Engineering](https://dataknowsall.com/blog/tag/data-engineering) ### [A Quick Start for Taking MongoDB Collections into Pandas DataFrames](https://dataknowsall.com/blog/mongo.html) [![looker tips](https://dataknowsall.com/hubfs/lookertips_target_02.png)](https://dataknowsall.com/blog/lookertips.html) [Business Intelligence](https://dataknowsall.com/blog/tag/business-intelligence) ### [6 Looker Tips That Will Power Up Your Next Data Analysis Job](https://dataknowsall.com/blog/lookertips.html) [![Confusion Matrix](https://dataknowsall.com/hubfs/trialfire_cm.png)](https://dataknowsall.com/blog/trialfire.html) [Machine Learning](https://dataknowsall.com/blog/tag/machine-learning) ### [A Real-World Approach with XGBoost on Marketing Attribution Data](https://dataknowsall.com/blog/trialfire.html) [![Lambda Google Workspace](https://dataknowsall.com/hs-fs/hubfs/Lambda%20Google%20Workspace.png?width=320&height=132&name=Lambda%20Google%20Workspace.png "Lambda Google Workspace")](https://www.dataknowsall.com) Specialists in Data Engineering, Machine Learning, and Analytics Consultation [linkedin-in icon](https://www.linkedin.com/in/broepke/) [twitter icon](https://twitter.com/broepke) [Follow us on Facebook](https://github.com/broepke) **Pages** - [Home](https://dataknowsall.com) - [Blog](https://dataknowsall.com/blog/) - [Services](https://dataknowsall.com/services) - [Technologies](https://dataknowsall.com/technologies) ### Get Updates In Your Inbox Copyright © 2026, Data Knows All ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Brian Roepke", "url" : "https://dataknowsall.com/blog/author/brian-roepke" }, "dateModified" : "2024-03-05T12:55:14.242Z", "datePublished" : "2022-01-04T16:00:00.000Z", "headline" : "Up Your Game in Social Media Sentiment Analysis", "image" : [ "https://dataknowsall.com/hubfs/sentiment_all.png" ], "mainEntityOfPage" : { "@id" : "https://dataknowsall.com/blog/sentiment.html", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://dataknowsall.com/hubfs/Lambda%20Google%20Workspace.png" }, "name" : "Data Knows All, LLC" } } ``` ```json { "@context" : "http://schema.org", "@type" : "Article", "author" : { "@type" : "Person", "name" : [ "Brian Roepke" ], "url" : "https://dataknowsall.com/blog/author/brian-roepke" }, "datePublished" : "2022-01-04T16:00:00+0000", "description" : "Understand how your customers feel about your products and services", "headline" : "Up Your Game in Social Media Sentiment Analysis", "image" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/sentiment_all.png", "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/Lambda%20Google%20Workspace.png" }, "name" : "Data Knows All" }, "url" : "https://dataknowsall.com/blog/sentiment.html" } ```