Scaling Challenges

GPT-4, the model underpinning ChatGPT, comes with a subscription fee of $20 per month, along with a strict limitation of 25 messages every three hours. This means that users are spending over $200 a year for a chatbot with restricted usage. Given OpenAI's market leadership, one might question why they impose such limitations on user experience.

The answer lies in the high operational costs associated with Transformers. The self-attention mechanism that enables ChatGPT's impressive capabilities incurs significant costs based on the input sequence's length. For context, Meta's LLaMa model, with 62 billion parameters, incurred training expenses of around $5 million in just 21 days. Imagine the training costs for GPT-4, which is likely ten times larger!

It's estimated that OpenAI spends approximately $700,000 daily to run ChatGPT, and these financial pressures have led AI labs to adopt measures to manage expenses, impacting their final designs.

The Challenge of Computation Costs

Faced with these costs, researchers had to make tough choices: either limit the number of inferences or drastically cut sequence length. While restricting inferences might be acceptable due to the availability of a free version like GPT-3.5, the latter poses a significant challenge.

Large Language Models (LLMs) have a constrained working memory. Although they have processed vast amounts of information, during interactions, they can only handle a limited amount of data. Exceeding the context window can lead to forgotten exchanges, a limitation imposed to keep operational costs manageable.

What if there were a way to optimize computation costs to allow LLMs to work with far larger context windows, perhaps stretching into the hundreds of thousands of words? This is precisely what Meta's MegaByte aims to achieve through a groundbreaking architectural approach.

The Innovative Solution

MegaByte was developed with a singular purpose: to unlock long-sequence modeling, which current models like ChatGPT cannot efficiently handle. So, how does MegaByte accomplish this?

Instead of predicting words, MegaByte predicts letters. This model comprises three distinct components, as illustrated below:

Source: Meta

1. Patch Embedder: Upon receiving user input, the model divides it into "patches"—fixed-size segments of the original input—transforming natural language into machine-readable numbers.
2. Global Model: All embedded patches are processed simultaneously within a model that performs self-attention, updating each patch's embedding with its position and relation to other text portions.
3. Local Model: For every patch, a smaller, local model decodes it back into text, aiming to replicate the original language accurately.

Why MegaByte Holds Great Promise

MegaByte introduces two significant shifts in text generation:

1. A per-byte approach to text generation
2. Simultaneous patch generation

This means that instead of generating text by predicting the next token (typically three to four characters), MegaByte generates text character by character. Consequently, the models responsible for decoding text from embeddings can be much smaller.

The rationale is straightforward: predicting the next character in a word is easier than predicting the subsequent word in a sentence. The range of possibilities increases significantly when forming entire sentences.

Additionally, MegaByte leverages greater parallelism. While traditional Transformers like ChatGPT generate text sequentially, MegaByte processes all patches at once. This approach optimizes GPU usage, enhancing speed and reducing operational costs.

Success of MegaByte

So, how effective has MegaByte proven to be? To put it mildly—remarkably successful. Currently, ChatGPT supports a maximum of approximately 6,000 words. In contrast, MegaByte claims to manage up to 1 million tokens. If this refers to byte-sized tokens, it translates to around 170,000 words, representing a substantial increase compared to ChatGPT—a 27-fold improvement.

However, if the term "token" refers to the standard definition, MegaByte could potentially handle 750,000 words, equating to a staggering 125 times more data than ChatGPT's best model!

The rapid pace of innovation is astounding. Just recently, Claude expanded its token window to 100,000 tokens (or 75,000 words), and now we are witnessing an architecture that promises at least 100,000 additional words.

The most thrilling aspect? The possibility of extending beyond language models, applying self-attention to other rich modalities like images or videos. Considering how effectively ChatGPT understands text, envision an AI model that learns from more descriptive inputs like visuals or motion. That would be a monumental leap forward!

What an exciting time to be involved in AI!

The first video, "ChatGPT Please Write Me A Piece Of Polymorphic Malware," discusses the implications of AI in cybersecurity, highlighting the balance between innovation and risk.

The second video, "ChatGPT and Large Language Model Bias | 60 Minutes," explores the biases present in AI models, shedding light on the ethical considerations in AI development.