“How do you say good morning in Spanish?” – This is an early research example in the field of Natural Language Processing (NLP) dated back to the 1950s. Using statistical techniques to analyze the human language then became popular in the 1990s; fast forward to two decades later in the 2010s, research efforts have evolved to focus on using deep neural networks for NLP. We hear about more and more new models with millions and billions of parameters, e.g. BERT, XLNet, ALBERT, etc., but are they really entirely different from one another? How about methods like tf-idf, word2vec? Are they still relevant and useful? How do we keep up with the constant evolution and innovation in the NLP field? How do we start understanding the model details behind the scene, rather than just using these models? 

Start with the common and basic theoretical foundation.

Deep learning or not, probabilistic language modeling is a common thread across all NLP tasks.

Count-based Approaches

In the non-deep learning realm, probabilistic language models (LMs) leverage count-based approaches, which represent words in terms of their frequencies. As an example, let’s now try to set up n-grams and bag-of-words (BOW) in the framework of a Naive Bayes classification model.

Naive Bayes treats the entire input text as a bag of words and models the probability of text based on unigrams. 

The diagram above shows that Naive Bayes infers the probability distribution of unlabeled data based on the probability distribution of labeled data, all by counting!

On the other hand, positive pointwise mutual information (PPMI) frames the probabilistic LM problem by asking “do words w1 and w2 co-occur more than if they were independent?” While Term Frequency-Inverse Document Frequency (TF-IDF) calculates a term’s importance based on the term frequency relative to other terms in other documents, there have also been attempts to estimate the probability of any given document d containing a term t. 

Prediction-based Approaches 

Count-based methods result in high sparsity in word representation vectors and typically do not generalize well – what if there are new words not present in your training vocabulary? We encounter very different vocabulary when reading The Lord of the Rings vs. The New York Times vs. Twitter. The idea that we could use the text at hand as implicitly supervised training data was a paradigm shift in framing NLP tasks. NLP approaches started to pivot away from count-based to prediction-based. 

Word2Vec is a single-layered neural network that hinges upon training binary prediction tasks based on a window of nearby words; for instance, is the word “hobbit” likely to show up near “ring”? Then, we use the model weights as the word embeddings, which are vector representations of words. Word2Vec is quick to train but does not take global context into account. 

On the other hand, since words that occur together may encode some form of meaning, Global Vectors for Word Representation (GloVe) generates embeddings by aggregating a global matrix of word co-occurrence. For example, one would expect to see the words “ice” and “solid” more frequently together than “ice” and “gas”.

One fundamental limitation that both Word2Vec and GloVe share is that they do not know how to deal with out-of-vocabulary (OOV) words. This is where fastText comes in. fastText looks at substrings of words; therefore, each word is a sum of its N-grams and each N-gram ranges from 3 to 6 characters. 

Contextual Approaches (Deep Learning)

None of the methods discussed above is able to capture contextual information well.

Hence, we turn to neural network architecture to process long sequences. Here, probabilistic LMs manifest in that a neural network predicts the probability distribution of every word, given words that the model has seen so far.  First, let’s discuss recurrent neural networks (RNNs) briefly. RNNs are feedback loops that cycle over the sequences. The most popular RNN variant in NLP is Long Short-Term Memory (LSTM). While vanilla RNNs  have bottleneck difficulties accessing information from many steps back, LSTMs have a separate memory cell with built-in forget gates to forget unimportant information. 

A notable model architecture that uses LSTMs is Embeddings from Language Models (ELMo). ELMo is bidirectional and concatenates the input text from left to right, and right to left, mirroring a human reader’s experience. However, the concatenation means that the training process could not take advantage of both left and right contexts simultaneously nor resolve the bottleneck problem completely. This leads to our most popular architecture: Transformers. Transformers leverage attention mechanisms; as the name implies, attention attends to its inputs, which means it pays most attention to the most important words. Furthermore, transformers  allow simultaneous processing of text,, by additionally encoding the word positions. Therefore, transformers have a faster model training process and thanks to positional word encodings, we do not have to worry about word orders being jumbled up.

BERT (Bidirectional Encoder Representation from Transformers), published in 2019, is actually a variant of the original transformer architecture implemented in 2017. Unlike the original encoder-decoder transformer architecture meant for sequence generation tasks, i.e. machine translation, BERT does not have a decoder component. An architectural difference as such indicates  that BERT is not trained for sequence generation tasks. 

Today, we hear about a variety of other transformer architectures, namely RoBERTa, XLNet, ALBERT, and so on. These models are built upon BERT and share a lot of commonalities in the architecture, despite improving upon the training processes. In order to keep up with the evolution of the NLP landscape, it is necessary to understand only the foundation of attention and the BERT architecture.

Summary

In this article, you learned that probabilistic language modeling is at the core of NLP tasks. You also learned the motivation of moving from count-based to prediction-based to deep contextual models, and the high-level differences and similarities among these models. You now have just enough theoretical background to get started in NLP! 

To pay homage to my ecology background, it is only right that I begin discussing machine learning methods with random forest.

Random forest is one of my favorite algorithms - it’s effective and elegant. We know that a forest is made up of many trees in real life. Same goes to random forest - it is an ensemble of decision trees trained independently of each other to ultimately cast a majority vote. To answer the question that the title of this post begs, we must start by investigating:

What’s random about random forests?

There are two elements of randomness in a random forest:
1. random sampling of training data points when building trees
2. random subsets of features considered

During training, samples are drawn with replacement (bootstrapping). Some samples are used multiple times in a tree. Only approximately 2/3 of data instances are used during the training process. In other words, about 1/3 of instances in your training data are left out. Each tree learns from a random, bootstrapped sample of training data. Therefore, each tree learns different parts of training data. Random subsets of features are considered when splitting nodes. At test time, random forest makes predictions by taking average predictions of each tree (aggregating). This is why you often hear the term “bagging” — it is simply a combination of two processes: bootstrap and aggregating.

Why is randomness powerful then?

The random selection of features considered can minimize the correlation effect between inputs while maintaining the strength of the tree. According to Leo Breiman (2001), the author of random forest paper, random selection features help the resulting forest to be relatively robust to outliers and noise. Moreover, the randomness nature of feature selection allows this algorithm to be parallelized to reduce training time.

Random sampling of data points allows us to generate estimates of ongoing generalization errors, strength, and correlation (since every tree’s knowledge within the ensemble is different, we can truly test if the forest can generalize to unseen data).

It is perhaps obvious at this point, Breiman also pointed out that the accuracy of random forests depends on 2 things: the strength of individual trees and a measure of dependence between them.

It’s worth noting that while random forest can help reduce the problem of multicollinearity of features, the variable importance of the correlated features will be reduced. It is still important to use your domain knowledge, if possible, to inform you which feature to use/remove. If you lack domain expertise, it is worth running different random forest experiments to test different combination of features and see for yourself the differences in model performances and variable importance values.

Increasing number of trees (alone) does not result in overfitting

This is perhaps a shocking and confusing statement to you. Many of you know that as model complexity increases, the tendency of overfitting also increases. However, in the context of random forests, the number of trees built is not a problem because each tree is built independently of another, meaning each tree is always starting from scratch. If you need math to convince you, Leo Breiman (2001) proved this using the Strong Law of Large Numbers: error rates converge to a certain value, Simply put, while keeping all other hyperparameters constant, beyond a sufficiently large value of number of trees, error rates plateau out and do not increase anymore. Following up on Breiman’s paper, Probst and Boulesteix (2018) also empirically concluded that we should not tune the number of trees and more trees do not degrade performance. In other words, theoretically, you should pick the number of trees based on the consideration of computational cost.

However, Probst and Boulesteix (2018) also found that empirically 100 trees are sufficient to lead to the largest performance gain. Oftentimes, we see that the testing performance might decrease when we increase the number of trees, even if all else stays constant. When you see this, you should note this as a sign of the distributional differences between the training data and the testing data. If both training and testing data share the same distribution, perhaps your trees are too “fully grown”, which means your trees are too deep — then, it’s time to reduce the depth of trees. Generally, you need more trees when you have:

1. lower sample size for each tree while training

2. bigger constraints on tree depth

3. more variables (since they lead to less correlated trees)

to reach convergence of error rate.

Conclusion

The name of random forests gives away the reason why they are so powerful. It’s because of the embedded randomness within the algorithm. Don’t blame on the number of trees alone when your model overfits. Check for the tree depth, sample size for each tree, and the number of variables.

References

1. Breiman, Leo (2001), Random Forests, https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf

2. Probst, Phillip and Boulesteix, Anne-Laure (2018), To Tune or Not to Tune the

   Number of Trees in Random Forest, http://www.jmlr.org/papers/volume18/17-269/17-269.pdf

It’s hard to lead “normal” lives to confront the “new normal” as the COVID-19 pandemic sweeps across the nation and the rest of the world. Aside from social distancing, many of us do want to use our skills to help, as we spend more time at home.

This blog is meant to provide some starter or ideas around how to begin analyzing the COVID-19 Open Research Dataset (CORD-19) that compiles scholarly articles on coronaviruses. I first learned about this dataset because multiple tech giants and other organizations have partnered together (a rare effort) to create this dataset. They launched a Kaggle challenge which you can read more about here .

About the data

As mentioned, this data is text-based and is provided in json format. There are three data files altogether: peer-reviewed commercial use subset (9000 papers), peer-reviewed non-commercial use subset (1973 papers), and another non-peer-reviewed dataset (803 papers). In this post, I will be using the Databricks environment to analyze the dataset for commercial use, since it has more files than the other two.

comm_use_subset.printSchema()

As you can see, the file is very deeply nested. You can already tell by now, cleaning this data would be a significant undertaking. But I am going to take the easy way out here for now, and show you how we can very quickly make use of the data in its current state and generate some simple Natural Language Processing (NLP) analysis.

Many associate NLP with deep learning now and I won’t blame them, because deep learning is indeed a very powerful tool in analyzing text. However, I am going to show you simple NLP approaches with and without deep learning methods.

Non-deep learning

Say that I am interested in knowing, at a glance, which are the most common words that appear in all the titles of these peer-reviewed papers. I can generate a word cloud.

comm_use_subset.select("metadata.title").show(3)

First, I concatenated all the available titles of papers.

from pyspark.sql.functions import concat_ws, collect_list

all_title_df = comm_use_subset.agg(concat_ws(", ",     collect_list(comm_use_subset['metadata.title'])).alias('all_titles'))
display(all_title_df)

Then, I wrote a simple function to plot a word cloud leveraging existing Python library called wordcloud. I first tokenized the text, by splitting the sentence into individual words. Next, I utilized the default set of stopwords and removed any stopwords accordingly. The WordCloud().generate() function calculates the number of instances each word appears. This means that the size of words in the word cloud we see later is a direct reflection of the frequency of the word in the text.

def custom_wordcloud_draw(text, color = 'white'):
    """
    Plots wordcloud of string text after removing stopwords
    """
    cleaned_word = " ".join([word for word in text.split()])
    wordcloud = WordCloud(stopwords= STOPWORDS.update(['using',   'based', 'analysis', 'study', 'research', 'viruses']),
                  background_color=color,
                  width=1000,
                  height=1000
                 ).generate(cleaned_word)
    plt.figure(1,figsize=(8, 8))
    plt.imshow(wordcloud)
    plt.axis('off')
    display(plt.show())

wordcloud_draw(str(all_title_df.select('all_titles').collect()[0]))

As you can see, there is quite a number of non-meaningful words in the word cloud, e.g. “using”, “based”, “analysis”, etc. Let’s try to remove some of them by making a very minor change to our wordcloud_draw function: call the function UPDATE on STOPWORDS to add these custom stopwords.

def custom_wordcloud_draw(text, color = 'white'):
    """
    Plots wordcloud of string text after removing stopwords
    """
    cleaned_word = " ".join([word for word in text.split()])
    wordcloud = WordCloud(stopwords= STOPWORDS.update(['using', 'based', 'analysis', 'study', 'research', 'viruses']),
                      background_color=color,
                      width=1000,
                      height=1000
                     ).generate(cleaned_word)
    plt.figure(1,figsize=(8, 8))
    plt.imshow(wordcloud)
    plt.axis('off')
    display(plt.show())

custom_wordcloud_draw(str(all_title_df.select('all_titles').collect()[0]))

Yayy! We successfully removed the non-meaningful words we are not interested in.

Now, let’s move onto NLP analysis using deep learning methods.

Deep Learning

We are going to use a summarizer model trained on BERT and also K-means clustering that initially was used to summarize lectures! Refer to this paper to read more about it. To use this library, you need to pip install bert-extractive-summarizer.

As with all machine learning models, you should be well aware of your model’s limitations. For this summarizer model, the known limitation is that it does not do well at summarizing text that has over 100 sentences.

In this section, my goal is to summarize abstracts of papers, so that I need to read even fewer words to get the gist of the paper!

Let me start by reading the abstract of one of the papers.

from summarizer import Summarizer

abstract = str(comm_use_subset.select("abstract.text").take(2)[1])
abstract

Now, let me train a summarizer model and remove any sentences that has fewer than 20 characters using the min_length parameter.

model = Summarizer()
abstract_summary = model(str(abstract), min_length=20)

full_abstract = ''.join(abstract_summary)
print(full_abstract)

The End

By now, hopefully you feel more motivated to start your own analysis, without letting the messiness of this dataset deter you! You probably noticed that I didn’t clean my data at all prior to performing these two quick-and-dirty NLP methods (generating a word cloud visualization and an abstract summary). Yes, these two results don’t yield much actionable insight. But, hopefully this has helped you jumpstart some of your own analysis! You may try to cluster the categories of papers on coronaviruses published out there, so that other researchers can direct their effort to less well-studied areas!

If you are interested in non-NLP analysis on this particular dataset, check out the Youtube video that I shared. There are coding notebooks linked in the Youtube description too.

It’s hard to stay hopeful these days, but I hope the utmost hope that this disorienting and disheartening COVID-19 situation will soon come to pass! I hope you and your loved ones are able to draw near to each other and stay healthy!

Code Repository

1. Code presented in this post is available on my GitHub.

2. My other friends and I presented our analysis on coronavirus-related data at a webinar, feel free to checkout this Youtube page for the walkthroughs. They have also provided their notebook links in the webinar description as well.

https://www.databricks.com/blog/2020/11/16/fatal-force-exploring-police-shootings-with-sql-analytics.html

Subscribe now