--- title: A Quick Guide to Part of Speech Tagging description: Leverage this essential building block of NLP to perform a Voice of Customer analysis --- [Data Knows All](https://dataknowsall.com/blog) # [A Quick Guide to Part of Speech Tagging](https://dataknowsall.com/blog/pos.html) Written by [Brian Roepke](https://dataknowsall.com/blog/author/brian-roepke) | Dec 11, 2021 3:48:00 PM ## What is a Part of Speech? Part of Speech (POS) is a way to describe the grammatical function of a word. In Natural Language Processing (NLP), POS is an essential building block of language models and interpreting text. While POS tags are used in higher-level functions of NLP, it's important to understand them on their own, and it's possible to leverage them for useful purposes in your text analysis. > There is a hierarchy of tasks in NLP (see [Natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing#Major_tasks_in_NLP) for a list). At the bottom are sentence and word segmentation. POS tagging builds on top of that, and phrase chunking builds on top of POS tags. These tags, in turn, can be used as features for higher-level tasks such as building parse trees, which can, in turn, be used for Named Entity Resolution, Coreference Resolution, Sentiment Analysis, and Question Answering. There are **eight** (sometimes nine) different parts of speech in English that are commonly defined [3](http://localhost:8000/pos.html#fn:POS). 1. **Noun**: A noun is the name of a person, place, thing, or idea. 2. **Pronoun**: A pronoun is a word used in place of a noun. 3. **Verb**: A verb expresses action or being. 4. **Adjective**: An adjective modifies or describes a noun or pronoun. 5. **Adverb**: An adverb modifies or describes a verb, an adjective, or another adverb. 6. **Preposition**: A preposition is a word placed before a noun or pronoun to form a phrase modifying another word in the sentence. 7. **Conjunction**: A conjunction joins words, phrases, or clauses. 8. **Interjection**: An interjection is a word used to express emotion. 9. **Determiner or Article**: A grammatical marker of definiteness (the) or indefiniteness (a, an). These are not always considered POS but are often included in POS tagging libraries. ## The Basics of POS Tagging Let's start with some simple examples of POS tagging with three common Python libraries: NLTK, TextBlob, and Spacy. We'll do the absolute basics for each and compare the results. Start by importing all the needed libraries. ``` import nltkfrom textblob import TextBlobfrom textblob.taggers import PatternTaggerimport spacy ``` For our examples, we'll use two sentences with a common word, *book*, which can be a noun or a verb, to test how well the POS taggers work in context. - *Please book my flight to California* - *I read a very good book* ### NLTK Let's start with the most common library for NLP in Python; the **Natural Language Toolkit** or **NLTK**. ``` tokenized_sent = nltk.sent_tokenize("Please book my flight to California")[nltk.pos_tag(nltk.word_tokenize(word)) for word in tokenized_sent] ``` ``` [[('Please', 'NNP'), ('book', 'NN'), ('my', 'PRP$'), ('flight', 'NN'), ('to', 'TO'), ('California', 'NNP')]] ``` ``` tokenized_sent = nltk.sent_tokenize("I read a very good book")[nltk.pos_tag(nltk.word_tokenize(word)) for word in tokenized_sent] ``` ``` [[('I', 'PRP'), ('read', 'VBP'), ('a', 'DT'), ('very', 'RB'), ('good', 'JJ'), ('book', 'NN')]] ``` What we notice here is that for the most part, **NLTK** properly recognizes the word in context; however, a few errors such as *Please* is tagged as a *Proper Noun (NNP)* and *book*is tagged as a *Noun (NN)* in our first sentence when it should be a *Verb (VB)*. **Note:** For a list of what the tags mean, see [Penn Treebank Project](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). ### TextBlob Let's try another library called **TextBlob** which provides a simple API for diving into standard natural language processing (NLP) tasks [5](http://localhost:8000/pos.html#fn:BLOB). It's a very good Pythonic implementation of an NLP library and simplifies some of the common NLP tasks. Much of what **TextBlob** does is *wrap NLTK* and other popular NLP libraries to make them easier to use. ``` blob = TextBlob("Please book my flight to California", pos_tagger=PatternTagger())blob.tags ``` ``` [('Please', 'VB'), ('book', 'NN'), ('my', 'PRP$'), ('flight', 'NN'), ('to', 'TO'), ('California', 'NNP')] ``` ``` blob = TextBlob("I read a very good book", pos_tagger=PatternTagger())blob.tags ``` ``` [('I', 'PRP'), ('read', 'VB'), ('a', 'DT'), ('very', 'RB'), ('good', 'JJ'), ('book', 'NN')] ``` Notice the use of `PatternTagger` in the initialization of the Blob. The default is to use NLTK's tagger, yielding the same results as above. This allows us to try a different POS Tagger and check its performance. We can see that **TextBlob** correctly identifies *Please* as a *Verb* this time but still misses *Book* as a *Verb* in the first sentence. ### Spacy **Spacy** is the most modern and advanced of the three of these. It's incredibly robust for tons of NLP tasks and allows for customization if more power is needed. This is currently my favorite NLP library, and let's check it out with our sentences. ``` doc = nlp("Please book my flight to California")for token in doc: print(token.text, token.pos_) ``` ``` Please INTJbook VERBmy PRONflight NOUNto ADPCalifornia PROPN ``` ``` doc = nlp("I read a very good book")for token in doc: print(token.text, token.pos_) ``` ``` I PRONread VERBa DETvery ADVgood ADJbook NOUN ``` We see here that **Spacy** correctly tagged all of our words, and it identified *Please* like an *Interjection* as opposed to a *Verb*, which is more accurate and also identified *Book* as a *Verb* in the first sentence. Each of these libraries has its pros and cons. I believe you should start with **NLTK** to understand how it works, especially since it has so much robust support of different [corpora](https://www.nltk.org/book/ch02.html). **TextBlob** is great when you want simplicity across several NLP tasks, and **Spacy**when you want one of the most robust NLP libraries around. Check out this great Series [NLTK with Python for Natural Language](https://pythonprogramming.net/tokenizing-words-sentences-nltk-tutorial/) from *PythonProgramming.net*. ## Voice of Customer Analysis with Parts of Speech One of the most common tasks performed with NLP is analyzing customer feedback from various sources and determining what customers are talking about for your product or service. This type of analysis is called **Voice of Customer Analysis** or **VOC**. There are many ways to perform a VOC analysis. From *Sentiment Analysis* to *Topic Modeling*, one method you can use is Part of Speech tagging to narrow *what* customers are talking about and *how* they talk about your products and services. Text for your analysis can come from survey responses, support tickets, Facebook comments, Tweets, chat conversations, emails, call transcripts, and online reviews. Let's say you have a collection of customer reviews. One of the things you might want to identify is all the **products** that people are talking about. You may have a perfect categorization of your products in a database, but what if you don't at the granular level you need? For this example, we will use the dataset of [Women's E-Commerce Clothing Reviews](https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews) on Kaggle. After importing the dataset, we can create a new **DataFrame** of all the words and their POS tag. The function below takes each review and determines the POS tag for each word; an **important** distinction because we get the *context* of each word in the sentence, and as we saw above, this makes a big difference in which POS tag is associated. ``` def pos_tag(text): df = pd.DataFrame(columns = ['WORD', 'POS']) doc = nlp(text) for token in doc: df = df.append({'WORD': token.text, 'POS': token.pos_}, ignore_index=True) return df ``` Next, we can run the function on a subset of the reviews. Since we're using individual words, the quantity can be in the millions, and we most likely don't need the entire dataset. ``` # Take a random sample of reviewsdf2 = df.sample(10000, random_state=42).copy()# Create an empty dataframe to store the resultsdf_pos = pd.DataFrame(columns = ['WORD', 'POS'])# Iterate through the reviews and append each POS tag to the dataframedf_pos = pos_tag(df2['Review Text'].to_string())df_pos.shape ``` ``` (144498, 2) ``` Next, we can group and count each of the POS tags to see the most frequent ones. ``` df_top_pos = df_pos.groupby('POS')['POS'].count().\ reset_index(name='count').sort_values(['count'],ascending=False).head(15) ``` Great! We have lots of tags and words. However, these don't quite tell us much on their own, giving us a view of the distribution of different tags. However, now we can use our tags to pull out words that might represent **products** vs. those that represent other words in our reviews. For this, we can filter for just the **Nouns**. ``` df_nn = df_pos[df_pos['POS'] == 'NOUN'].copy()df_nn.groupby('WORD')['WORD'].count().reset_index(name='count').\ sort_values(['count'], ascending=False).head(10) ``` ``` WORD count667 dress 17792062 top 11761764 shirt 4631971 sweater 437453 color 3831807 size 312765 fabric 2871922 store 2741822 skirt 2561416 pants 246 ``` Look at that! We have the top 15 words used in this subset of reviews and *most* of them look like product categories. What if we now look at the top **Adjectives** for the same subset of reviews? ``` df_adj = df_pos[df_pos['POS'] == 'ADJ'].copy()df_adj.groupby('WORD')['WORD'].count().reset_index(name='count').\ sort_values(['count'], ascending=False).head(15) ``` ``` WORD count400 great 481144 beautiful 405248 cute 398784 soft 321218 comfortable 272632 perfect 243585 nice 196776 small 17641 Great 149666 pretty 146394 gorgeous 144511 little 142611 other 142523 lovely 125349 flattering 122 ``` Tada 🎉! the top-most used words that describes *how* our customers are talking about our products. We have lost some positive words but also some that might be worth looking into. Things such as *small* and *little* speak potentially to a sizing issue where some clothes are not true to fit. A product manager can take this information and dive deeper into the reviews that mention this word. As mentioned, there are other ways to analyze text that might be better, such as Sentiment Analysis and Topic Modeling. However, this is a fun way to apply POS tagging in a practical use case and even be combined with those other NLP tools to help you get the most out of your customer's feedback. All the code for this analysis is available on [GitHub](https://github.com/broepke/POS) [View full post](https://dataknowsall.com/blog/pos.html) ```json { "@context" : "http://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Brian Roepke" }, "dateModified" : "2024-03-04T17:58:01.213Z", "datePublished" : "2021-12-11T15:48:00Z", "headline" : "A Quick Guide to Part of Speech Tagging", "image" : { "@type" : "ImageObject", "height" : 1500, "url" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/pos_01.png", "width" : 3000 }, "mainEntityOfPage" : "https://dataknowsall.com/blog/pos.html", "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "height" : 58.86571, "url" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/Lambda%20with%20Text%20-%20Wide.png", "width" : 600.0 }, "name" : "Data Knows All" } } ```