word embeddings – Notes of a Neuropsychiatry Amateur

NLP – Word Embeddings – BERT

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained transformer-based neural network architecture for natural language processing tasks such as text classification, question answering, and language inference. One important feature of BERT is its use of word embeddings, which are mathematical representations of words in a continuous vector space.

In BERT, word embeddings are learned during the pre-training phase and are fine-tuned during the task-specific fine-tuning phase. These embeddings are learned by training the model on a large corpus of text, and they are able to capture semantic and syntactic properties of words.

The BERT model architecture is composed of multiple layers of transformer blocks, with the input being a sequence of tokens (e.g., words or subwords) and the output being a contextualized representation of each token in the sequence. The model also includes a pooled output which is used for many down stream task, which are generated by applying a pooling operation over the entire sequence representation.

How does BERT differ from Word2Vec or GloVe?

Training objective: The main difference between BERT and Word2Vec/GloVe is the training objective. BERT is trained to predict missing words in a sentence (masked language modeling) and predict the next sentence (next sentence prediction), this way the model learns to understand the context of the words. Word2Vec and GloVe, on the other hand, are trained to predict a word given its context or to predict the context given a word, this way the model learn the association between words.
Inputs: BERT takes a pair of sentences as input, and learns to understand the relationship between them, Word2Vec and GloVe only take a single sentence or a window of context words as input.
Directionality: BERT is a bidirectional model, meaning that it takes into account the context of a word before and after it in a sentence. This is achieved by training on both the left and the right context of the word. Word2Vec is unidirectional model which can be trained on either the left or the right context, and GloVe is also unidirectional but it is trained on the global corpus statistics.
Pre-training: BERT is a pre-trained model that can be fine-tuned on specific tasks, Word2Vec and GloVe are also pre-trained models, but the main difference is that their pre-training is unsupervised with no downstream task, this means that the fine-tuning of BERT can provide better performance on some tasks because it is pre-trained with the task objective in mind.

BERT Architecture

The core component of BERT is a stack of transformer encoder layers, which are based on the transformer architecture introduced by Vaswani et al. in 2017. Each transformer encoder layer in BERT consists of multiple self-attention heads. The self-attention mechanism allows the model to weigh the importance of different parts of the input sequence when generating the representation of each token. This allows the model to understand the relationships between words in a sentence and to capture the context in which a word is used.

The transformer architecture also includes a feed-forward neural network, which is applied to the output of each self-attention head to produce the final representation of each token.

Transformer Encoder Layer

In the transformer encoder layer, for every input token in a sequence, the self-attention mechanism computes key, value, and query vectors, which are used to create a weighted representation of the input. The key, value and query vectors are computed by applying different linear transformations (matrix multiplications) on the input embeddings, these linear transformations are learned during the training process.

In BERT, the input representations are computed by combining multiple embedding layers. The input is first tokenized into word pieces, which is a technique that allows the model to handle out-of-vocabulary words by breaking them down into subword units. The tokenized input is then passed through three embedding layers:

Token Embedding: Each word piece is represented as a token embedding
Position Embedding: Each word piece is also represented by a position embedding, which encodes information about the position of the word piece in the input sequence.
Segment Embedding: BERT also uses a segment embedding to represent the input segments, this embedding helps BERT to distinguish between the two sentences when the input is a pair of sentences.

These embeddings are concatenated or added together to obtain a fixed-length vector representation of each word piece. Special tokens [CLS] and [SEP] are used to indicate the beginning and the end of the input segments and the classification prediction respectively, [CLS] token is used as the representation of the entire input sequence, which is used in the classification tasks, while [SEP] token is used to separate the input segments, in the case of the input being a pair of sentences.

Masked Language Modeling

If you try to predict each word of the input sequence using the training data with cross-entropy loss, the learning task becomes trivial for the network. Since the network knows beforehand what it has to predict, it can easily learn weights to reach a 100% classification accuracy.

The masked language modeling (MLM) approach, also known as the “masked word prediction” task, addresses this problem by randomly masking a portion of the input words (e.g., 15%) during training and requiring the network to predict the original value of the masked words based on the context. By masking a portion of the input words, the network is forced to understand the context of the words and to learn meaningful representations of the input.

In the MLM approach, the network is only required to predict the value of the masked words, and the loss is calculated only over the masked words. This means that the model is not learning to predict words it has already seen, but instead, it is learning to predict words it hasn’t seen while seeing all the context around those words.

In addition to MLM, BERT also uses another objective during pre-training called “next sentence prediction” this objective is a binary classification task that is applied on the concatenation of two sentences, the model is trained to predict whether the second sentence is the real next sentence in the corpus or not. This objective helps BERT to understand the relationship between two sentences and how they are related.

NLP – Word Embeddings – ELMo

ELMo (Embeddings from Language Models) is a deep learning approach for representing words as vectors (also called word embeddings). It was developed by researchers at Allen Institute for Artificial Intelligence and introduced in a paper published in 2018.

ELMo represents words as contextualized embeddings, meaning that the embedding for a word can change based on the context in which it is used. For example, the word “bank” could have different embeddings depending on whether it is used to refer to a financial institution or the edge of a river.

ELMo has been shown to improve the performance of a variety of natural language processing tasks, including language translation, question answering, and text classification. It has become a popular approach for representing words in NLP models, and the trained ELMo embeddings are freely available for researchers to use.

How does ELMo differ from Word2Vec or GloVe?

ELMo (Embeddings from Language Models) is a deep learning approach for representing words as vectors (also called word embeddings). It differs from other word embedding approaches, such as Word2Vec and GloVe, in several key ways:

Contextualized embeddings: ELMo represents words as contextualized embeddings, meaning that the embedding for a word can change based on the context in which it is used. In contrast, Word2Vec and GloVe represent words as static embeddings, which do not take into account the context in which the word is used.
Deep learning approach: ELMo uses a deep learning model, specifically a bidirectional language model, to generate word embeddings. Word2Vec and GloVe, on the other hand, use more traditional machine learning approaches based on a neural network (Word2Vec) and matrix factorization (GloVe).

To generate context-dependent embeddings, ELMo uses a bi-directional Long Short-Term Memory (LSTM) network trained on a specific task (such as language modeling or machine translation). The LSTM processes the input sentence in both directions (left to right and right to left) and generates an embedding for each word based on its context in the sentence.

Overall, ELMo is a newer approach for representing words as vectors that has been shown to improve the performance of a variety of natural language processing tasks. It has become a popular choice for representing words in NLP models.

What is the model for training ELMo word embeddings?

The model used to train ELMo word embeddings is a bidirectional language model, which is a type of neural network that is trained to predict the next word in a sentence given the context of the words that come before and after it. To train the ELMo model, researchers at Allen Institute for Artificial Intelligence used a large dataset of text, such as news articles, books, and websites. The model was trained to predict the next word in a sentence given the context of the words that come before and after it. During training, the model learns to represent words as vectors (also called word embeddings) that capture the meaning of the word in the context of the sentence.

Explain in details the bidirectional language model

A bidirectional language model is a type of neural network that is trained to predict the next word in a sentence given the context of the words that come before and after it. It is called a “bidirectional” model because it takes into account the context of words on both sides of the word being predicted.

To understand how a bidirectional language model works, it is helpful to first understand how a unidirectional language model works. A unidirectional language model is a type of neural network that is trained to predict the next word in a sentence given the context of the words that come before it.

A unidirectional language model can be represented by the following equation:

P(w[t] | w[1], w[2], …, w[t-1]) = f(w[t-1], w[t-2], …, w[1])

This equation says that the probability of a word w[t] at time t (where time is the position of the word in the sentence) is determined by a function f of the words that come before it (w[t-1], w[t-2], …, w[1]). The function f is learned by the model during training.

A bidirectional language model extends this equation by also taking into account the context of the words that come after the word being predicted:

P(w[t] | w[1], w[2], …, w[t-1], w[t+1], w[t+2], …, w[n]) = f(w[t-1], w[t-2], …, w[1], w[t+1], w[t+2], …, w[n])

This equation says that the probability of a word w[t] at time t is determined by a function f of the words that come before it and the words that come after it. The function f is learned by the model during training.

In practice, a bidirectional language model is implemented as a neural network with two layers: a forward layer that processes the input words from left to right (w[1], w[2], …, w[t-1]), and a backward layer that processes the input words from right to left (w[n], w[n-1], …, w[t+1]). The output of these two layers is then combined and used to predict the next word in the sentence (w[t]). The forward and backward layers are typically implemented as recurrent neural networks (RNNs) or long short-term memory (LSTM) networks, which are neural networks that are designed to process sequences of data.

During training, the bidirectional language model is fed a sequence of words and is trained to predict the next word in the sequence. The model uses the output of the forward and backward layers to generate a prediction, and this prediction is compared to the actual next word in the sequence. The model’s weights are then updated to minimize the difference between the prediction and the actual word, and this process is repeated for each word in the training dataset. After training, the bidirectional language model can be used to generate word embeddings by extracting the output of the forward and backward layers for each word in the input sequence.

ELMo model training algorithm

Initialize the word vectors:

The word vectors are usually initialized randomly using a Gaussian distribution.
Alternatively, you can use pre-trained word vectors such as Word2Vec or GloVe.

Process the input sequence:

Input the sequence of words w[1], w[2], ..., w[t-1] into the forward layer and the backward layer.
The forward layer processes the words from left to right, and the backward layer processes the words from right to left.
Each layer has its own set of weights and biases, which are updated during training.

Compute the output:

The output of the forward layer and the backward layer are combined to form the final output o[t].
The final output is used to predict the next word w[t].

Compute the loss:

The loss is computed as the difference between the predicted word w[t] and the true word w[t].
The loss function is usually the cross-entropy loss, which measures the difference between the predicted probability distribution and the true probability distribution.

Update the weights and biases:

The weights and biases of the forward and backward layers are updated using gradient descent and backpropagation.

Repeat steps 2-5 for all words in the input sequence.

ELMo generates contextualized word embeddings by combining the hidden states of a bi-directional language model (BLM) in a specific way.

The BLM consists of two layers: a forward layer that processes the input words from left to right, and a backward layer that processes the input words from right to left. The hidden state of the BLM at each position t is a vector h[t] that represents the context of the word at that position.

To generate the contextualized embedding for a word, ELMo concatenates the hidden states from the forward and backward layers and applies a weighted summation. The hidden states are combined using a task-specific weighting of all biLM layers. The weighting is controlled by a set of learned weights γ_task and a bias term s_task. The ELMo embeddings for a word at position k are computed as a weighted sum of the hidden states from all L layers of the biLM:

ELMo_task_k = E(R_k; Θtask) = γ_task_L * h_LM_k,L + γ_task{L-1} * h_LM_k,{L-1} + … + γ_task_0 * h_LM_k,0 + s_task

Here, h_LM_k,j represents the hidden state at position k and layer j of the biLM, and γ_task_j and s_task are the task-specific weights and bias term, respectively. The task-specific weights and bias term are learned during training, and are used to combine the hidden states in a way that is optimal for the downstream task.

Using ELMo for NLP tasks

ELMo can be used to improve the performance of supervised NLP tasks by providing context-dependent word embeddings that capture not only the meaning of the individual words, but also their context in the sentence.

To use a pre-trained bi-directional language model (biLM) for a supervised NLP task, the first step is to run the biLM and record the layer representations for each word in the input sequence. These layer representations capture the context-dependent information about the words in the sentence, and can be used to augment the context-independent token representation of each word.

In most supervised NLP models, the lowest layers are shared across different tasks, and the task-specific information is encoded in the higher layers. This allows ELMo to be added to the model in a consistent and unified manner, by simply concatenating the ELMo embeddings with the context-independent token representation of each word.

The model then combines the ELMo embeddings with the context-independent token representation to form a context-sensitive representation h_k, typically using either bidirectional RNNs, CNNs, or feed-forward networks. The context-sensitive representation h_k is then used as input to the higher layers of the model, which are task-specific and encode the information needed to perform the target NLP task. It can be helpful to add a moderate amount of dropout to ELMo and to regularize the ELMo weights by adding a regularization term to the loss function. This can help to prevent overfitting and improve the generalization ability of the model.

NLP – Word Embeddings – FastText

What is the FastText method for word embeddings?

FastText is a library for efficient learning of word representations and sentence classification. It was developed by Facebook AI Research (FAIR).

FastText represents each word in a document as a bag of character n-grams. For example, the word “apple” would be represented as the following character n-grams: “a”, “ap”, “app”, “appl”, “apple”, “p”, “pp”, “ppl”, “pple”, “p”, “pl”, “ple”, “l”, “le”. This representation has two advantages:

It can handle spelling mistakes and out-of-vocabulary words. For example, the model would still be able to understand the word “apple” even if it was misspelled as “appel” or “aple”.
It can handle words in different languages with the same script (e.g., English and French) without the need for a separate model for each language.

FastText uses a shallow neural network to learn the word representations from this character n-gram representation. It is trained using the skip-gram model with negative sampling, similar to word2vec.

FastText can also be used for sentence classification by averaging the word vectors for the words in the sentence and training a linear classifier on top of the averaged vector. It is particularly useful for languages with a large number of rare words, or in cases where using a word’s subwords (also known as substrings or character n-grams) as features can be helpful.

How are word embeddings trained in FastText?

Word embeddings in FastText can be trained using either the skip-gram model or the continuous bag-of-words (CBOW) model.

In the skip-gram model, the goal is to predict the context words given a target word. For example, given the input sequence “I have a dog”, the goal would be to predict “have” and “a” given the target word “I”, and to predict “I” given the target word “have”. The skip-gram model learns to predict the context words by minimizing the negative log likelihood of the context words given the target word.

In the CBOW model, the goal is to predict the target word given the context words. For example, given the input sequence “I have a dog”, the goal would be to predict “I” given the context words “have” and “a”, and to predict “have” given the context words “I” and “a”. The CBOW model learns to predict the target word by minimizing the negative log likelihood of the target word given the context words.

Both the skip-gram and CBOW models are trained using stochastic gradient descent (SGD) and backpropagation to update the model’s parameters. The model is trained by minimizing the negative log likelihood of the words in the training data, given the model’s parameters.

Explain how FastText represents each word in a document as a bag of character n-grams

To represent a word as a bag of character n-grams, FastText breaks the word down into overlapping substrings (also known as character n-grams). For example, the word “apple” could be represented as the following character 3-grams (trigrams): [“app”, “ppl”, “ple”]. The number of characters in each substring is specified by the user and is typically set to between 3 and 6 characters.

For example, consider the following sentence:

“I have a dog”

If we set the number of characters in each substring to 3, FastText would represent each word in the sentence as follows:

“I”: [“I”] “have”: [“hav”, “ave”] “a”: [“a”] “dog”: [“dog”]

The use of character n-grams allows FastText to learn good vector representations for rare words, as it can use the vector representations of the character n-grams that make up the rare word to compute its own vector representation. This is particularly useful for handling out-of-vocabulary words that may not have a pre-trained vector representation available.

How are vector representations for each word computed from n-gram vectors?

In FastText, the vector representation for each word is computed as the sum of the vector representations of the character n-grams (subwords) that make up the word. For example, consider the following sentence:

“I have a dog”

If we set the number of characters in each substring to 3, FastText would represent each word in the sentence as a bag of character 3-grams (trigrams) as follows:

“I”: [“I”] “have”: [“hav”, “ave”] “a”: [“a”] “dog”: [“dog”]

FastText would then learn a vector representation for each character n-gram and use these vector representations to compute the vector representation for each word. For example, the vector representation for the word “have” would be computed as the sum of the vector representations for the character n-grams [“hav”, “ave”].

Since there can be huge number of unique n-grams, how does FastText deal with the memory requirement?

One of the ways that FastText deals with the large number of unique character n-grams is by using hashing to map the character n-grams to a fixed-size hash table rather than storing them in a dictionary. This allows FastText to store the character n-grams in a compact form, which can save memory.

What is hashing? How are character sequences hashed to integer values?

Hashing is the process of converting a given input (called the ‘key’) into a fixed-size integer value (called the ‘hash value’ or ‘hash code’). The key is typically some sort of string or sequence of characters, but it can also be a number or other data type.

There are many different ways to hash a character sequence, but most algorithms work by taking the input key, performing some mathematical operations on it, and then returning the hash value as an integer. The specific mathematical operations used will depend on the specific hashing algorithm being used.

One simple example of a hashing algorithm is the ‘modulo’ method, which works as follows:

Take the input key and convert it into a numerical value, for example by assigning each character in the key a numerical value based on its ASCII code.
Divide this numerical value by the size of the hash table (the data structure in which the hashed keys will be stored).
The remainder of this division is the hash value for the key.

This method is simple and fast, but it is not very robust and can lead to a high number of collisions (when two different keys produce the same hash value). More sophisticated algorithms are typically used in practice to improve the performance and reliability of hash tables.

How is the Skip-gram with negative sampling applied in FastText?

Skip-gram with negative sampling (SGNS) algorithm is used to learn high-quality word embeddings (i.e., dense, low-dimensional representations of words that capture the meaning and context of the words). The Skip-gram with negative sampling algorithm works by training a predictive model to predict the context words (i.e., the words that appear near a target word in a given text) given the target word. During training, the model is given a sequence of word pairs (a target word and a context word) and tries to predict the context words given the target words.

To train the model, the SGNS algorithm uses a technique called negative sampling, which involves sampling a small number of negative examples (random words that are not the true context words) and using them to train the model along with the positive examples (the true context words). This helps the model to learn the relationship between the target and context words more efficiently by focusing on the most informative examples.

The SGNS algorithm steps are as following:

The embedding for a target word (also called the ‘center word’) is calculated by taking the sum of the embeddings for the word itself and the character n-grams that make up the word.
The context words are represented by their word embeddings, without adding the character n-grams.
Negative samples are selected randomly from the vocabulary during training, with the probability of selecting a word being proportional to the square root of its unigram frequency (i.e., the number of times it appears in the text).
The dot product of the embedding for the center word and the embedding for the context word is calculated. We then need to normalize the similarity scores over all of the context words in the vocabulary, so that the probabilities sum to 1 and form a valid probability distribution.
Compute the cross-entropy loss between the predicted and true context words. Use an optimization algorithm such as stochastic gradient descent (SGD) to update the embedding vectors in order to minimize this loss. This involves bringing the actual context words closer to the center word (i.e., the target word) and increasing the distance between the center word and the negative samples.

The cross-entropy loss function can be expressed as:
L = – ∑i(y_i log(p(w_i|c)) + (1 – y_i)log(1 – p(w_i|c)))
where:
L is the cross-entropy loss.
y_i is a binary variable indicating whether context word i is a positive example (y_i = 1) or a negative example (y_i = 0).
p(w_i|c) is the probability of context word i given the target word c and its embedding.
∑i indicates that the sum is taken over all context words i in the vocabulary.

FastText and hierarchical softmax

FastText can use a technique called hierarchical softmax to reduce the computation time during training. Hierarchical softmax works by organizing the vocabulary into a binary tree, with the word at the root of the tree and its descendant words arranged in a hierarchy according to their probability of occurrence.

During training, the model uses the hierarchical structure of the tree to compute the loss and update the model weights more efficiently. This is done by traversing the tree from the root to the appropriate leaf node for each word, rather than computing the loss and updating the weights for every word in the vocabulary separately.

The standard softmax function has a computational complexity of O(Kd), where K is the number of classes (i.e., the size of the vocabulary) and d is the number of dimensions in the hidden layer of the model. This complexity arises from the need to normalize the probabilities over all potential classes in order to obtain a valid probability distribution. The hierarchical softmax reduces the computational complexity to O(d*log(K)). Huffman coding can be used to construct a binary tree structure for the softmax function, where the lowest frequency classes are placed deeper into the tree and the highest frequency classes are placed near the root of the tree.

In the hierarchical softmax function, a probability is calculated for each path through the Huffman coding tree, based on the product of the output vector v_n_i of each inner node n and the output value of the hidden layer of the model, h. The sigmoid function is then applied to this product to obtain a probability between 0 and 1.

The idea of this method is to represent the output classes (i.e., the words in the vocabulary) as the leaves on the tree and to use a random walk through the tree to assign probabilities to the classes based on the path taken from the root of the tree. The probability of a certain class is then calculated as the product of the probabilities along the path from the root to the leaf node corresponding to the class.

This allows the hierarchical softmax function to compute the probability of each class more efficiently, since it only needs to consider the path through the tree rather than the entire vocabulary. This can significantly reduce the computational complexity of the model, particularly for large vocabularies, making it practical to train word embeddings on very large datasets.

Hierarchical softmax and conditional probabilities

To compute the probability of each context word given the center word and its embedding using the hierarchical softmax function, we first organize the vocabulary into a binary tree, with the words at the nodes of the tree and their descendant words arranged in a hierarchy according to their probability of occurrence.

We then compute the probability of each context word by traversing the tree from the root to the appropriate leaf node for the word. For each inner node n in the tree, we compute the probability of traversing the left or right branch of the tree as follows:

p(left|n) = sigmoid(v_n_i · h) p(right|n) = 1 – p(left|n)

where:

v_n_i is the vector representation of inner node n
h is the output value of the hidden layer of the model

The probability of a context word w is then computed as the product of the probabilities of the branches along the path from the root to the leaf node corresponding to w.

NLP – Word Embeddings – GloVe

What are word embeddings?

Word embeddings are a type of representation for text data, which allows words with similar meaning to have a similar representation in a neural network model. Word embeddings are trained such that words that are used in similar contexts will have similar vectors in the embedding space. This is useful because it allows the model to generalize better and makes it easier to learn from smaller amounts of data. Word embeddings can be trained using a variety of techniques, such as word2vec and GloVe, and are commonly used as input to deep learning models for natural language processing tasks.

So are they represented as arrays of numbers?

Yes, word embeddings are typically represented as arrays of numbers. The length of the array will depend on the size of the embedding space, which is a parameter that is chosen when the word embeddings are created. For example, if the size of the embedding space is 50, each word will be represented as a vector of length 50, with each element of the vector representing a dimension in the embedding space.

In a neural network model, these word embedding vectors are typically fed into the input layer of the model, and the rest of the layers in the model are then trained to perform some task, such as language translation or sentiment analysis. The model learns to combine the various dimensions of the word embedding vectors in order to make predictions or decisions based on the input data.

How are word embeddings determined?

There are a few different techniques for determining word embeddings, but the most common method is to use a neural network to learn the embeddings from a large dataset of text. The basic idea is to train a neural network to predict a word given the words that come before and after it in a sentence, using the output of the network as the embedding for the input word. The network is trained on a large dataset of text, and the weights of the network are used to determine the embeddings for each word.

There are a few different variations on this basic approach, such as using a different objective function or incorporating additional information into the input to the network. The specific details of how word embeddings are determined will depend on the specific method being used.

What are the specific methods for generating word embeddings?

Word embeddings are a type of representation for natural language processing tasks in which words are represented as numerical vectors in a high-dimensional space. There are several algorithms for generating word embeddings, including:

Word2Vec: This algorithm uses a neural network to learn the vector representations of words. It can be trained using two different techniques: continuous bag-of-words (CBOW) and skip-gram.
GloVe (Global Vectors): This algorithm learns word embeddings by factorizing a matrix of word co-occurrence statistics.
FastText: This is an extension of Word2Vec that learns word embeddings for subwords (character n-grams) in addition to full words. This allows the model to better handle rare and out-of-vocabulary words.
ELMo (Embeddings from Language Models): This algorithm generates word embeddings by training a deep bi-directional language model on a large dataset. The word embeddings are then derived from the hidden state of the language model.
BERT (Bidirectional Encoder Representations from Transformers): This algorithm is a transformer-based language model that generates contextual word embeddings. It has achieved state-of-the-art results on a wide range of natural language processing tasks.

What is the word2vec CBOW model?

The continuous bag-of-words (CBOW) model is one of the two main techniques used to train the Word2Vec algorithm. It predicts a target word based on the context words, which are the words surrounding the target word in a text.

The CBOW model takes a window of context words as input and predicts the target word in the center of the window. The input to the model is a one-hot vector representation of the context words, and the output is a probability distribution over the words in the vocabulary. The model is trained to maximize the probability of predicting the correct target word given the context words.

During training, the model adjusts the weights of the input-to-output connections in order to minimize the prediction error. Once training is complete, the model can be used to generate word embeddings for the words in the vocabulary. These word embeddings capture the semantic relationships between words and can be used for various natural language processing tasks.

What is the word2vec skip-gram model?

The skip-gram model is the other main technique used to train the Word2Vec algorithm. It is the inverse of the continuous bag-of-words (CBOW) model, which predicts a target word based on the context words. In the skip-gram model, the target word is used to predict the context words.

Like the CBOW model, the skip-gram model takes a window of context words as input and predicts the target word in the center of the window. The input to the model is a one-hot vector representation of the target word, and the output is a probability distribution over the words in the vocabulary. The model is trained to maximize the probability of predicting the correct context words given the target word.

What are the steps for the GloVe algorithm?

GloVe learns word embeddings by factorizing a matrix of word co-occurrence statistics, which can be calculated from a large corpus of text.

The main steps of the GloVe algorithm are as follows:

Calculate the word co-occurrence matrix: Given a large corpus of text, the first step is to calculate the co-occurrence matrix, which is a symmetric matrix X where each element X_ij represents the number of times word i appears in the context of word j. The context of a word can be defined as a window of words around the word, or it can be the entire document.
Initialize the word vectors: The next step is to initialize the word vectors, which are the columns of the matrix W. The word vectors are initialized with random values.
Calculate the pointwise mutual information (PMI) matrix: The PMI matrix is calculated as follows:

PMI_ij = log(X_ij / (X_i * X_j))

where X_i is the sum of all the elements in the ith row of the co-occurrence matrix, and X_j is the sum of all the elements in the jth column of the co-occurrence matrix. The PMI matrix is a measure of the association between words and reflects the strength of the relationship between them.

Factorize the PMI matrix: The PMI matrix is then factorized using singular value decomposition (SVD) or another matrix factorization technique to obtain the word vectors. The word vectors are the columns of the matrix W.
Normalize the word vectors: Finally, the word vectors are normalized to have unit length.

Once the GloVe algorithm has been trained, the word vectors can be used to represent words in a high-dimensional space. The word vectors capture the semantic relationships between words and can be used for various natural language processing tasks.

How is the matrix factorization performed in GloVe? What is the goal?

The goal of matrix factorization in GloVe is to find two matrices, called the word matrix and the context matrix, such that the dot product of these matrices approximates the co-occurrence matrix. The word matrix contains the word vectors for each word in the vocabulary, and the context matrix contains the context vectors for each word in the vocabulary.

To find these matrices, GloVe minimizes the difference between the dot product of the word and context matrices and the co-occurrence matrix using a least-squares optimization method. This results in word vectors that capture the relationships between words in the corpus.

In GloVe, the objective function that is minimized during matrix factorization is the least-squares error between the dot product of the word and context matrices and the co-occurrence matrix. More specifically, the objective function is given by:

How is the objective function minimized?

In each iteration of SGD, a mini-batch of co-occurrence pairs (i, j) is selected from the co-occurrence matrix, and the gradients of the objective function with respect to the parameters are computed for each pair. The parameters are then updated using these gradients and a learning rate, which determines the step size of the updates.

This process is repeated until the objective function has converged to a minimum or a preset number of iterations has been reached. The process of selecting mini-batches and updating the parameters is often referred to as an epoch. SGD is an efficient method for minimizing the objective function in GloVe because it does not require computing the Hessian matrix, which is the matrix of second-order partial derivatives of the objective function.

When should GloVe be used instead of Word2Vec?

GloVe (Global Vectors) and Word2Vec are two widely used methods for learning word vectors from a large corpus of text. Both methods learn vector representations of words that capture the semantics of the words and the relationships between them, and they can be used in various natural language processing tasks, such as language modeling, information retrieval, and machine translation.

GloVe and Word2Vec differ in the way they learn word vectors. GloVe learns word vectors by factorizing a co-occurrence matrix, which is a matrix that contains information about how often words co-occur in a given corpus. Word2Vec, on the other hand, learns word vectors using a shallow neural network with a single hidden layer.

One advantage of GloVe is that it is computationally efficient, as it does not require training a neural network. This makes it well suited for use with large corpora. However, Word2Vec has been shown to perform better on some tasks, such as syntactic analogies and named entity recognition.

How is the co-occurrence matrix reduced to lower dimensions in GloVe?

In GloVe (Global Vectors), the co-occurrence matrix is not directly reduced to lower dimensions. Instead, the co-occurrence matrix is used to learn word vectors, which are then reduced to lower dimensions using dimensionality reduction techniques, such as principal component analysis (PCA) or t-distributed stochastic neighbor embedding (t-SNE).

To learn word vectors from the co-occurrence matrix in GloVe, the matrix is factorized into two matrices, called the word matrix and the context matrix, using a least-squares optimization method. The word matrix contains the word vectors for each word in the vocabulary, and the context matrix contains the context vectors for each word in the vocabulary.

After the word vectors have been learned, they can be reduced to lower dimensions using dimensionality reduction techniques. For example, PCA can be used to project the word vectors onto a lower-dimensional space, while t-SNE can be used to embed the word vectors in a two-dimensional space for visualization.

It is worth noting that reducing the dimensionality of the word vectors may result in some loss of information, as some of the relationships between words may be lost in the lower-dimensional space. Therefore, it is important to consider the trade-off between the dimensionality of the word vectors and their representational power.

Interpreting GloVe from the Ratio of Co-occurrence Probabilities

GloVe uses the ratio of co-occurrence probabilities to learn the word vectors and context vectors. Specifically, it minimizes the difference between the dot product of the word and context vectors and the log of the ratio of co-occurrence probabilities. This allows GloVe to learn word vectors that capture the meanings and relationships between words in the language.

Chatting with GPT – Natural Language Processing

Let’s talk about natural language processing. How does input text data get transformed to numerical vectors?

In natural language processing (NLP), input text data is often transformed into numerical vectors in order to make it more suitable for machine learning algorithms to work with. There are several methods that can be used to transform text data into numerical vectors, including:

One-hot encoding: This is a simple method that converts each word in the text into a unique integer index, and then represents each word as a binary vector, with a “1” in the position corresponding to the index of the word and “0”s in all other positions. One-hot encoding is a common method for converting text data into numerical vectors, but it can result in very large and sparse vectors, which may not be efficient to work with.
Word embeddings: Word embeddings are numerical vectors that represent the meaning of a word in a continuous, low-dimensional space. Word embeddings can be trained on large datasets of text data using techniques such as word2vec or GloVe, and they can capture the contextual relationships between words and the similarity between words. Word embeddings are a popular method for representing text data in NLP, as they can capture the meaning of words in a compact and dense form.
Bag of words: The bag of words model represents text data as a fixed-length vector, with each element in the vector representing the number of times a particular word appears in the text. The bag of words model is a simple and commonly used method for representing text data, but it does not capture the order or context of the words in the text.

There are many other methods for converting text data into numerical vectors, and the specific method that is used may depend on the specific NLP task and the goals of the model. It is important to choose an appropriate method for representing text data in order to effectively capture the meaning and context of the text.

What are word2vec word embeddings? How are they trained?

Word2vec is a method for training word embeddings, which are numerical vectors that represent the meaning of a word in a continuous, low-dimensional space. Word embeddings are a popular method for representing text data in natural language processing (NLP), as they can capture the meaning of words in a compact and dense form and can be used as input to machine learning algorithms.

Word2vec is a type of neural network model that is trained to predict the context of a word given a set of surrounding words, or to predict a target word given a set of context words. During training, the model is presented with a large dataset of text data and learns to predict the context of a word based on the words that appear around it. The learned model can then be used to generate word embeddings by taking the weights of the model’s hidden layer as the word vectors.

There are two main variants of the word2vec model: continuous bag of words (CBOW) and skip-gram. In the CBOW model, the goal is to predict the target word given a set of context words, while in the skip-gram model, the goal is to predict the context words given a target word.

Word2vec word embeddings can be trained on large datasets of text data using techniques such as stochastic gradient descent (SGD) or negative sampling.

Provide the exact steps for the Word2Vec algorithm, include equations.

Word2Vec is a technique for learning vector representations of words, also known as “word embeddings”, in a way that captures the context and meaning of the words. The basic idea behind Word2Vec is to use the context of words in a text corpus to predict a target word, based on a sliding window of surrounding words. The prediction task is accomplished using a shallow neural network with a single hidden layer.

Here are the steps for the Word2Vec algorithm:

Preprocess the text data: This involves cleaning and formatting the text data, such as lowercasing, stemming, and removing stop words.
Tokenize the text: This involves dividing the text into individual words or tokens.
Build a vocabulary: This involves creating a list of all the unique words in the text data and assigning each word a unique index.
Create word-context pairs: For each word in the text, create a word-context pair by considering a window of surrounding words. For example, if the window size is 2, the word-context pairs for the word “cat” in the sentence “The cat sat on the mat” would be (“cat”, “The”), (“cat”, “sat”), (“cat”, “on”), and (“cat”, “the”).
Train the model: Use the word-context pairs to train a shallow neural network with a single hidden layer. The input layer consists of the context words, and the output layer consists of the target word. The hidden layer contains a set of “neurons” that represent the word embeddings.

The training process involves minimizing the error between the predicted output (the target word) and the true output (the context words) using stochastic gradient descent. The error is measured using a loss function, such as the negative log likelihood loss.

Here is the equation for the negative log likelihood loss:

Loss = -log P(w | c)

where w is the target word and c is the context.

Extract the word embeddings: Once the model is trained, the word embeddings can be extracted from the hidden layer of the neural network. These word embeddings can then be used for various natural language processing tasks, such as text classification, machine translation, and information retrieval.

You mentioned that word2vec can be trained using negative sampling. How does that work?

Yes, that’s correct! In addition to the traditional Word2Vec algorithm that uses the negative log likelihood loss function, there is also a variant called “negative sampling” that can be used to train the model. Negative sampling is a simplified training method that can be used to learn word embeddings more efficiently, especially for large text corpora.

Here’s how negative sampling works:

Preprocess and tokenize the text data as described in the previous steps.
Build a vocabulary and create word-context pairs as described in the previous steps.
Train the model: For each word-context pair, randomly sample a number of negative examples (i.e., words that are not the target word). The objective is to maximize the probability of the target word given the context words, while minimizing the probability of the negative examples given the context words.

The training process involves minimizing the following loss function:

Loss = -log(sigmoid(u_w * v_c)) – ∑log(sigmoid(-u_n * v_c))

where u_w and v_c are the embeddings for the target word and the context words, respectively, and u_n is the embedding for a negative example. The sigmoid function is used to squash the output to a value between 0 and 1.

Extract the word embeddings: Once the model is trained, the word embeddings can be extracted from the hidden layer of the neural network as described in the previous steps.

So when word2vec is trained using negative sampling, that means that just the loss function changes, otherwise stochastic gradient descent is still used?

Yes, that’s correct. When training word2vec using negative sampling, the loss function changes, but the underlying training algorithm, stochastic gradient descent (SGD), remains the same.

In word2vec, the goal is to learn a set of word vectors that can predict the surrounding words in a text. Negative sampling is a variant of the word2vec model that uses a different loss function to achieve this goal. Specifically, it uses a simplified version of the traditional word2vec loss function that only considers a small, randomly-selected subset of the negative examples (words that are not the target word) at each training step, rather than considering all of the negative examples. This allows the model to be trained more efficiently, as it doesn’t need to compute the loss for all of the negative examples at each training step.

SGD is still used to optimize the model’s parameters (the word vectors) by updating them in the direction that reduces the loss. The key difference with negative sampling is that the loss function being optimized is different from the traditional word2vec loss function.

But basically word2vec is trained using a pretty simple neural network, with just one hidden layer

Yes, that’s correct. Word2vec is typically implemented as a shallow neural network with just one hidden layer. The input to the network is a one-hot encoded word, and the hidden layer is a dense (fully-connected) layer with a fixed number of units. The output of the hidden layer is passed through a softmax activation function to produce a probability distribution over the vocabulary, which represents the model’s prediction of the probability of each word in the vocabulary given the input word.

You mentioned that there are two variants of word2vec – CBOW and skip-gram, what is the difference between the two?

In the Continuous Bag-of-Words (CBOW) model, the goal is to predict the current word given the surrounding context. For example, given the context “the cat sat on,” the model would try to predict the word “mat.” CBOW uses a sliding window approach to build up the context from the surrounding words.

On the other hand, the Skip-Gram model does the opposite: it tries to predict the surrounding context given a current word. So in the same example, given the word “mat,” the model would try to predict the context “the cat sat on.” Skip-gram works well with small amounts of data and is faster to train than CBOW.

What is the algorithms for the CBOW model?

The CBOW model uses a neural network with a single hidden layer to predict the current word based on the surrounding context. The input to the model is a one-hot encoded vector representing the context words, and the output is a probability distribution over the vocabulary, with the goal of predicting the target word with the highest probability.

The model is trained using the negative sampling loss function, which tries to maximize the probability of the target word given the context and minimize the probability of randomly sampled negative examples (words that are not the target word).

Here’s a summary of the CBOW algorithm:

Initialize the model weights and biases.
Iterate through each training example, which consists of a target word and a context of surrounding words.
Convert the context words and target word to one-hot encoded vectors.
Calculate the hidden layer activations by taking the dot product of the input vector (one-hot encoded context words) and the input weight matrix, and adding the bias.
Apply the activation function (e.g. sigmoid or ReLU) to the hidden layer activations to get the hidden layer output.
Calculate the output layer activations by taking the dot product of the hidden layer output and the output weight matrix, and adding the bias.
Apply the softmax function to the output layer activations to get the predicted probability distribution over the vocabulary.
Calculate the loss using the negative sampling loss function, which compares the predicted probability of the target word to the actual label (1 for the target word, 0 for the negative samples).
Backpropagate the error and update the model weights and biases using gradient descent.
Repeat steps 2-9 until the model has converged (i.e. the loss is minimized).

The CBOW algorithm uses the negative sampling loss function to train the model. The negative sampling loss function is defined as:

Loss = -(log(sigmoid(u_o * v_c)) + ∑ log(sigmoid(-u_i * v_c)))
where:

u_o is the output weight vector for the target word.
v_c is the input weight vector for the context words.
u_i is the output weight vector for a negative sample (a word that is not the target word).

The first term in the loss function maximizes the probability of the target word given the context, while the second term minimizes the probability of the negative samples.

The sigmoid function is applied to the dot product of the weight vectors to squash the output to a value between 0 and 1, which can be interpreted as a probability. The log function is applied to the sigmoid output to make the loss function smoother and easier to optimize.

The negative sampling loss function is used because it is computationally more efficient than the full softmax loss function, which involves calculating the probabilities of all words in the vocabulary.

Why is the dot product used? What does it represent?

In the word2vec algorithm, the dot product of the input and output weight vectors is used to measure the similarity between the context words and the target word. The dot product is a measure of the angle between two vectors, and it is defined as the product of the magnitudes of the vectors and the cosine of the angle between them.

In the case of word2vec, the input and output weight vectors are learned embeddings for the context words and the target word, respectively. These embeddings are learned during training, and they represent the underlying relationships between the words in the vocabulary.

The dot product of the input and output weight vectors is then used to predict the probability of the target word given the context. If the dot product is high, it means that the angle between the vectors is small, indicating that the context and target words are similar. This results in a high probability for the target word. On the other hand, if the dot product is low, it means that the angle between the vectors is large, indicating that the context and target words are dissimilar. This results in a low probability for the target word.

Overall, the dot product is used in the word2vec algorithm as a measure of similarity between the context words and the target word, which is used to predict the probability of the target word given the context.

What is the algorithm for the skip-gram model?

The Skip-Gram model is a neural network that uses a single hidden layer to predict the surrounding context given a current word. The input to the model is a one-hot encoded vector representing the current word, and the output is a probability distribution over the vocabulary, with the goal of predicting the context words with the highest probability.

The Skip-Gram model is trained using the negative sampling loss function, which tries to maximize the probability of the context words given the current word and minimize the probability of randomly sampled negative examples (words that are not in the context).

Here’s a summary of the Skip-Gram algorithm:

Initialize the model weights and biases.
Iterate through each training example, which consists of a current word and a context of surrounding words.
Convert the current word and context words to one-hot encoded vectors.
Calculate the hidden layer activations by taking the dot product of the input vector (one-hot encoded current word) and the input weight matrix, and adding the bias.
Apply the activation function (e.g. sigmoid or ReLU) to the hidden layer activations to get the hidden layer output.
Calculate the output layer activations by taking the dot product of the hidden layer output and the output weight matrix, and adding the bias.
Apply the softmax function to the output layer activations to get the predicted probability distribution over the vocabulary.
Calculate the loss using the negative sampling loss function, which compares the predicted probabilities of the context words to the actual labels (1 for the context words, 0 for the negative samples).
Backpropagate the error and update the model weights and biases using gradient descent.
Repeat steps 2-9 until the model has converged (i.e. the loss is minimized).

The negative sampling loss function used in the Skip-Gram model is defined as:

Loss = -(∑ log(sigmoid(u_c * v_o)) + ∑ log(sigmoid(-u_i * v_o)))

where:

u_c is the output weight vector for a context word.
v_o is the input weight vector for the current word.
u_i is the output weight vector for a negative sample (a word that is not in the context).

The first term in the loss function maximizes the probability of the context words given the current word, while the second term minimizes the probability of the negative samples.

In which cases is it better to use the CBOW model vs. Skip-Gram for word2vec?

The Continuous Bag-of-Words (CBOW) model and the Skip-Gram model are two approaches for learning vector representations of words, also known as word embeddings, in natural language processing tasks. These models are trained to predict a target word given the context of surrounding words, using a large corpus of text as input.

The main difference between the CBOW and Skip-Gram models is the way in which they define the context of a word. The CBOW model predicts the target word based on the context provided by the surrounding words, while the Skip-Gram model predicts the surrounding context words based on the target word.

In general, the CBOW model is faster to train and performs better on smaller datasets, while the Skip-Gram model is slower to train but can provide better results on larger datasets. The CBOW model is also better at handling rare words, as it takes into account the context provided by multiple surrounding words, while the Skip-Gram model only considers the context provided by a single target word.

In practice, the choice between the CBOW and Skip-Gram models will depend on the specific application and the available resources. If you have a large dataset and the time and computational resources to train a model, the Skip-Gram model may provide better results. If you have a smaller dataset and need to train a model quickly, the CBOW model may be a better choice.