Glossary

Few-shot learning: The ability of a model to learn to perform a task with a small amount of task-specific training data. This is in contrast to traditional machine learning, where a model requires a large amount of labeled data to perform well on a task.

Language model: A type of AI model that is trained to generate natural language text. These models are typically trained on large amounts of text data and can be used for a wide range of natural language understanding tasks, such as language translation, question answering, and text classification.

GPT-3: Generative Pre-trained Transformer 3 is a state-of-the-art language model developed by OpenAI that has been trained on a massive amount of text data.

Natural language understanding: The ability of an AI model to understand and interpret human language. This encompasses a wide range of tasks, such as language translation, question answering, sentiment analysis, and text classification.

Fine-tuning: The process of adapting a pre-trained model to a specific task by training it on a small amount of task-specific data. Fine-tuning is often used to improve the performance of a pre-trained model on a specific task.

Transfer learning: The ability of a model to transfer knowledge learned from one task to another.

Idiomatic expressions: Phrases that cannot be understood based on the meanings of the individual words, but rather by the phrase as a whole.

Sarcasm recognition: The ability of a model to recognize when someone is being sarcastic in their language.

Internal representations: The representations of the model's knowledge, learned during the training process.

Labeled data: Data that has been labeled, or marked, with the correct output for a specific task.

RNN - Recurrent Neural Networks (RNNs) are a type of neural network that have the ability to process sequential data. They were widely used for NLP tasks such as text classification and machine translation. Introduced in the 1980s, the became popular in early 2010s as one of the models to understand textual data.

CNN - Convolutional Neural Networks (CNNs) are a type of neural network that are well suited for processing data with a grid-like structure, such as images. They were adapted for NLP tasks such as text classification and sentiment analysis. CNNs were first introduced in the late 1980s for image classification and became popular for NLP tasks in the mid-2010s.

LSTM - Long Short-Term Memory (LSTM) networks are a type of RNN that are designed to handle the vanishing gradient problem that can occur when training RNNs. They were widely used for NLP tasks such as language modeling and machine translation.

Parallelization - It refers to the process of dividing a computational task into smaller, independent subtasks that can be executed simultaneously on multiple processors. This allows for faster processing and can reduce the overall time required to complete the task. In deep learning models, parallelizing computations can be challenging because the computations often depend on each other.

Vanishing Gradient - The vanishing gradient problem is a challenge that can occur when training recurrent neural networks (RNNs). The problem arises because the gradients used to update the weights of the independent and dependent variables used in these models like RNNs and LSTMs during training can become very small, leading to slow or ineffective learning. Simply put, the weight of any variable being computed at any time step depends on the weight in the previous time step. If you keep adding these dependent weights over multiple steps, the importance and the value of the weight keep reducing over time. If these weights become very small as they are propagated through the network, this can lead to slow or ineffective learning. These weights of various variables are also called “Attention”

Attention – simplistically speaking, these are the weights of various variables (or tokens on words) in a sentence that are used to calculate the relative importance of one word against the other. Generally, in attention models for understanding text, a sentence is broken down into parts as per parts of a sentence in the English language, so you will have a subject, an object, a verb (or the action), etc. the Attention model already has, to put it simplistically, “preconceived notions” as to the weights or attention between types of words. Attentions are refined from here onwards to a final value iteratively

Self-Attention – in this model, there is no “preconceived” notion of what word is more important. Effectively, all words of a sentence, or token, start at the same weight, and then, the attention of each word is calculated iteratively