Fundamentals of Text Analytics

 

Tet analytics combines a set of machine learning, statistical and linguistic techniques to process large volumes of unstructured tet or tet that does not have a predefined format, to derive insights and patterns. 

It enables businesses, governments, researchers, and media to eploit the enormous content at their disposal for making crucial decisions. Tet analytics uses a variety of techniques – sentiment analysis, topic modelling, named entity recognition, term frequency, and event etraction.

Tet mining and tet analytics are often used interchangeably. The term tet mining is generally used to derive qualitative insights from unstructured tet, while tet analytics provides quantitative results.

For eample, tet mining can be used to identify if customers are satisfied with a product by analyzing their reviews and surveys. Tet analytics is used for deeper insights, like identifying a pattern or trend from the unstructured tet. For eample, tet analytics can be used to understand a negative spike in the customer eperience or popularity of a product.

The results of tet analytics can then be used with data visualization techniques for easier understanding and prompt decision making.

As of 2020, around 4.57 billion people have access to the internet. That’s roughly 59 percent of the world’s population. Out of which, about 49 percent of people are active on social media. 

An enormous amount of tet data is generated every day in the form of blogs, tweets, reviews, forum discussions, and surveys. Besides, most customer interactions are now digital, which creates another huge tet database.

Most of the tet data is unstructured and scattered around the web. If this tet data is gathered, collated, structured, and analyzed correctly, valuable knowledge can be derived from it. Organizations can use these insights to take actions that enhance profitability, customer satisfaction, research, and even national security.

Help businesses to understand customer trends, product performance, and service quality. This results in quick decision making, enhancing business intelligence, increased productivity, and cost savings.

Helps researchers to eplore a great deal of pre-eisting literature in a short time, etracting what is relevant to their study. This helps in quicker scientific breakthroughs.

Tet analytic techniques help search engines and information retrieval systems to improve their performance, thereby providing fast user eperiences.

Refine user content recommendation systems by categorizing related content.

Assists in understanding general trends and opinions in the society, that enable governments and political bodies in decision making.

Natural language processing (NLP) refers to the branch of computer science—and more specifically, the branch of artificial intelligence or AI—concerned with giving computers the ability to understand tet and spoken words in much the same way human beings can.

NLP combines computational linguistics—rule-based modeling of human language—with statistical, machine learning, and deep learning models. Together, these technologies enable computers to process human language in the form of tet or voice data and to ‘understand’ its full meaning, complete with the speaker or writer’s intent and sentiment.

NLP drives computer programs that translate tet from one language to another, respond to spoken commands, and summarize large volumes of tet rapidly—even in real time. There’s a good chance you’ve interacted with NLP in the form of voice-operated GPS systems, digital assistants, speech-to-tet dictation software, customer service chatbots, and other consumer conveniences. 

But NLP also plays a growing role in enterprise solutions that help streamline business operations, increase employee productivity, and simplify mission-critical business processes.

Several NLP tasks break down human tet and voice data in ways that help the computer make sense of what it's ingesting. Some of these tasks include the following:

Speech recognition, also called speech-to-tet, is the task of reliably converting voice data into tet data. Speech recognition is required for any application that follows voice commands or answers spoken questions. What makes speech recognition especially challenging is the way people talk—quickly, slurring words together, with varying emphasis and intonation, in different accents, and often using incorrect grammar.

Part of speech tagging, also called grammatical tagging, is the process of determining the part of speech of a particular word or piece of tet based on its use and contet. 

Part of speech identifies ‘make’ as a verb in ‘I can make a paper plane,’ and as a noun in ‘What make of car do you own?’

Word sense disambiguation is the selection of the meaning of a word with multiple meanings  through a process of semantic analysis that determine the word that makes the most sense in the given contet. 

For eample, word sense disambiguation helps distinguish the meaning of the verb 'make' in ‘make the grade’ (achieve) vs. ‘make a bet’ (place).

Named entity recognition, or NEM, identifies words or phrases as useful entities. NEM identifies ‘Kentucky’ as a location or ‘Fred’ as a man's name.

Co-reference resolution is the task of identifying if and when two words refer to the same entity. The most common eample is determining the person or object to which a certain pronoun refers (e.g., ‘she’ = ‘Mary’),  but it can also involve identifying a metaphor or an idiom in the tet  (e.g., an instance in which 'bear' isn't an animal but a large hairy person).

Sentiment analysis attempts to etract subjective qualities—attitudes, emotions, sarcasm, confusion, suspicion—from tet.

The Python programing language provides a wide range of tools and libraries for attacking specific NLP tasks. Many of these are found in the Natural Language Toolkit, or NLTK, an open source collection of libraries, programs, and education resources for building NLP programs.

The NLTK includes libraries for many of the NLP tasks listed above, plus libraries for subtasks, such as sentence parsing, word segmentation, stemming and lemmatization (methods of trimming words down to their roots), and tokenization (for breaking phrases, sentences, paragraphs and passages into tokens that help the computer better understand the tet). 

It also includes libraries for implementing capabilities such as semantic reasoning, the ability to reach logical conclusions based on facts etracted from tet.

The earliest NLP applications were hand-coded, rules-based systems that could perform certain NLP tasks, but couldn't easily scale to accommodate a seemingly endless stream of eceptions or the increasing volumes of tet and voice data.

Enter statistical NLP, which combines computer algorithms with machine learning and deep learning models to automatically etract, classify, and label elements of tet and voice data and then assign a statistical likelihood to each possible meaning of those elements. 

Today, deep learning models and learning techniques based on neural networks enable NLP systems that 'learn' as they work and etract ever more accurate meaning from huge volumes of raw, unstructured, and unlabeled tet and voice data sets. 

Spam detection: You may not think of spam detection as an NLP solution, but the best spam detection technologies use NLP's tet classification capabilities to scan emails for language that often indicates spam or phishing. 

These indicators can include overuse of financial terms, characteristic bad grammar, threatening language, inappropriate urgency, misspelled company names, and more. 

Spam detection is one of a handful of NLP problems that eperts consider 'mostly solved' (although you may argue that this doesn’t match your email eperience).

Virtual agents such as Apple's Siri and Amazon's Alea use speech recognition to recognize patterns in voice commands and natural language generation to respond with appropriate action or helpful comments. 

Chatbots perform the same magic in response to typed tet entries. The best of these also learn to recognize contetual clues about human requests and use them to provide even better responses or options over time. 

The net enhancement for these applications is question answering, the ability to respond to our questions—anticipated or not—with relevant and helpful answers in their own words.

Virtual agents such as Apple's Siri and Amazon's Alea use speech recognition to recognize patterns in voice commands and natural language generation to respond with appropriate action or helpful comments. 

Chatbots perform the same magic in response to typed tet entries. The best of these also learn to recognize contetual clues about human requests and use them to provide even better responses or options over time. 

The net enhancement for these applications is question answering, the ability to respond to our questions—anticipated or not—with relevant and helpful answers in their own words.

Tet summarization uses NLP techniques to digest huge volumes of digital tet and create summaries and synopses for indees, research databases, or busy readers who don't have time to read full tet. 

The best tet summarization applications use semantic reasoning and natural language generation (NLG) to add useful contet and conclusions to summaries.

Google Translate is an eample of widely available NLP technology at work. Truly useful machine translation involves more than replacing words in one language with words of another. 

Effective translation has to capture accurately the meaning and tone of the input language and translate it to tet with the same meaning and desired impact in the output language. Machine translation tools are making good progress in terms of accuracy.

Many companies have started creating and relying on chatbots in the recent times. For most cases, the chatbots are being utilized for applications like automated customer service, solving customer queries and initiating automated payment.

While chatbots are widely accepted because of their help in reducing man hours, many feel that there’s a long way to go in terms of technology and accuracy. Discuss within your groups:

Advantages of chatbots and what tasks have been simplified because of  them?

Disadvantages of chatbots? What are specific areas of improvement?