KG, Author at RxNLP: Solutions for Text Mining and NLP

What is ROUGE and how it works for evaluation of summarization tasks?

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is essentially of a set of metrics for evaluating automatic summarization of texts as well as machine translation. It works by comparing an automatically produced summary or translation against a set

Why Use Stop Words for Text Mining?

When working with text mining applications, we often hear of the term “stop words” or “stop word list” or even “stop list”. Stop words are just a set of commonly used words in any language. Stop words are commonly eliminated

Obtaining Important Words or N-Grams in a Document using Probability Counts

The probability of an n-gram or word tells you how important a word is to a document. Unfortunately, since a document almost always has important words inter-mingled with conjunctions and determiners and other types of noise, using the raw probabilities may

What are N-Grams?

N-grams of texts are extensively used in text mining and natural language processing tasks. They are basically a set of co-occuring words within a given window and when computing the n-grams you typically move one word forward (although you can

Constructing a Domain Specific Stop Word List

While it is fairly easy to use a published set of stop words, in many cases, using such stop words is completely insufficient for certain applications. For example, in clinical texts, terms like “mcg” “dr.” and “patient” occur almost in