2025-05-12 18:00:00
Methods
2.2 Indicators for detecting under-trained tokens and 2.3 Verification of candidate tokens
Results
B. A short primer on UTF-8 encoding
C. Outputs for API-based verification
Although many of our findings are dependent on model-specific details such as tokenizer training and configuration, model architecture, and training data, there are a number of commonalities that appear across many different model families.
\ 3.2.1 Single-byte tokens
\ Tokens representing a single byte are a common source of untrained tokens. The most common occurrence are the ‘fallback’ bytes 0xF5–0xFF which are not used in UTF-8 encoded text[2], and are a convenient source for quickly locating reference untrained tokens for indicators which require them. In addition, many tokenizers including from the Gemma, Llama2 and Mistral families include every byte as a token, but additionally assign a duplicate token to many characters in the normal ASCII range 0x00–0x7F. For example, A is both token 282 as an unused byte fallback token and as token 235280 a text-based ‘A’ in the Gemma models. These issues are not universal, and we also find models which include precisely the 243 bytes used in UTF-8
\
\
\ as tokens, including the models by EleutherAI [14]. Untrained single byte tokens are typically classified as ‘partial UTF-8 sequences’ or ‘unreachable’, and our indicators are effective in revealing which ones are never or rarely seen in training. We publish specific tables which shows the status of each single-byte token for each analyzed model in our repository.
\ 3.2.2 Fragments of merged tokens
\
\ 3.2.3 Special tokens
\ Many models include untrained special tokens, such as , , or . In the following discussion we generally omit mentioning them, unless their status as an (un)trained token is particularly surprising, as their inclusion in the tokenizer and training data is typically deliberate, for purposes such as the ability to fine-tune models without changing tokenizers. One common observation is that on many occasions tokens such as , which we expect to be completely untrained, nevertheless appear to have been seen in training. A likely source for this is code repositories or guides about language models using these tokens in normal text, along with tokenizers allowing such special control tokens in normal input text.
\
:::info Authors:
(1) Sander Land, Cohere s([email protected]);
(2) Max Bartolo, Cohere ([email protected]).
:::
:::info This paper is available on arxiv under CC BY-SA 4.0 DEED license.
:::
[2] See Appendix B for a primer on UTF-8 encoding.
\ [3] When mentioning fragments of more complete tokens, the tokens in parentheses were not detected or verified as under-trained, unless explicitly mentioned otherwise.
2025-05-12 17:56:45
Methods
2.2 Indicators for detecting under-trained tokens and 2.3 Verification of candidate tokens
Results
B. A short primer on UTF-8 encoding
C. Outputs for API-based verification
In this section, we present a summary of our key findings regarding under-trained token detection. Given the model-specific nature and the extensive volume of results, we discuss some common findings as well as showcase some representative examples for particular models. Detailed reports covering all tested models and token types are available in our repository.
Figure 1 shows that despite their relative simplicity, our indicators are highly predictive of the maximal probability of token prediction. To quantify the number of tokens detected in verification compared to our candidate selection, we applied the verification step to all tokens for the Zephyr-beta model [12]. This resulted in 137 out of 31,747 verified tokens compared to 76 of 637 when testing only the top 2% candidate tokens.
\ Secondly, although training data statistics are rarely available, we were able to verify that our under-trained token indicators are closely related to the frequency tokens appear in training data for the OLMo v1.7 model [13]. Figure 2 shows a strong correlation for all proposed indicators, not only predicting under-trained tokens, but extending to the entire range of token frequencies.
\ Finally, Figure 3 shows additional examples of indicator metrics, showing clear peaks in the histogram near zero, and high correlation between alternative indicators in this region.
\
\ There are certain cases where the indicators we use are more predictive of a token’s tendency to induce unwanted output compared to our prompting techniques. With respect to verification, there are certain cases where the indicators we use offer a more reliable indication of a token’s tendency to induce unwanted output in typical prompting compared to our verification prompting techniques. These cases include input/output asymmetry, where tokens are solely present as inputs (e.g., ), or situations where the model exhibits a strong bias towards English, consistently producing translated outputs Another common occurrence is output of the equivalent token without a leading space, although the variation in our verification prompts compensates for this. Additionally, there are false negatives where tokens are rejected by the verification process but can still induce incorrect behaviour, mainly due to our strict threshold and repetitive verification prompts, which are aimed at detecting the most reliable under-trained tokens. However, verification using prompting is highly effective in identifying a threshold below which candidate tokens induce unwanted behaviour, and selecting the most effective candidate tokens.
\ Table 1 presents verification statistics and example verified tokens for the models evaluated. The number of verified under-trained tokens varies significantly across different model families and tokenizer vocabulary size, as well as depending on the number of unused special tokens a model’s tokenizer allows as plain-text input. The percentage of verified tokens typically ranges between 5–50% of tested candidate tokens, corresponding to 0.1–1% of the total vocabulary.
\
:::info Authors:
(1) Sander Land, Cohere s([email protected]);
(2) Max Bartolo, Cohere ([email protected]).
:::
:::info This paper is available on arxiv under CC BY-SA 4.0 DEED license.
:::
\
2025-05-12 17:47:38
:::info Authors:
(1) Sander Land, Cohere s([email protected]);
(2) Max Bartolo, Cohere ([email protected]).
:::
Methods
2.2 Indicators for detecting under-trained tokens and 2.3 Verification of candidate tokens
Results
B. A short primer on UTF-8 encoding
C. Outputs for API-based verification
The disconnect between tokenizer creation and model training in language models has been known to allow for certain inputs, such as the infamous _SolidGoldMagikarp token, to induce unwanted behaviour. Although such ‘glitch tokens’ that are present in the tokenizer vocabulary, but are nearly or fully absent in training, have been observed across a variety of different models, a consistent way of identifying them has been missing. We present a comprehensive analysis of Large Language Model (LLM) tokenizers, specifically targeting this issue of detecting untrained and under-trained tokens. Through a combination of tokenizer analysis, model weight-based indicators, and prompting techniques, we develop effective methods for automatically detecting these problematic tokens. Our findings demonstrate the prevalence of such tokens across various models and provide insights into improving the efficiency and safety of language models. https://github.com/cohere-ai/magikarp
Large Language Models (LLMs) have undergone remarkable advancements, becoming increasingly capable of understanding and generating human-like text. While most components of these models are trained in an unsupervised fashion on vast amounts of data, the tokenizer typically remains a separately trained component based on custom algorithms and smaller datasets.
\ GPT-2 laid the foundation for much of current-day transformer-based language modelling [1], including a framework for tokenization building on previous work in byte-pair encoding (BPE) [2], that has since been widely adopted. Tokenization using BPE converts input text to a sequence of token ids by iteratively merging two neighbouring tokens using a fixed set of merge rules. These rules are learned using a greedy training algorithm on a smaller dataset. In addition to choosing this training dataset, which is ideally representative of the LLM’s training data, training a tokenizer involves optimizing various settings, such as vocabulary size [3], the addition of special tokens, and strategies for handling out-of-vocabulary tokens.
\ Recent work in this area has primarily focused on techniques to remove the need for tokenization altogether by moving to raw byte input [4]. This typically comes at a significant cost in inference speed, which can be compensated for by specialized architectures at the initial and final layers [5], or variable compute at intermediate layers [6]. However, the development of these techniques have not been widely adopted, and the vast majority of current models still rely on standard BPE tokenization.
\ Despite its widespread use, the tokenization step has generally been found to be unsatisfactory, being at the root of many unwanted behaviours and problems of LLMs [7]. In particular, the disconnect between tokenizer and model training creates the potential for some tokens to rarely or never be seen in training. Including such tokens in model inputs can lead to unexpected model behaviour including as hallucination or the generation of garbled outputs, leading to such tokens commonly being referred to as ‘glitch tokens’ [8]. We refer to these as ‘under-trained’ or ‘untrained’ tokens, reserving the latter term only for cases in which we have clear indication that the specific token had no model training data occurrences.
\ The presence of such under-trained tokens has several drawbacks. Firstly, they occupy capacity in a fixed-size tokenizer that could be better utilized for more common tokens, reducing input/output length and inference costs Secondly, their deliberate or accidental presence in input data has the potential to cause unwanted outputs and break downstream applications. Robustness to such unexpected or malicious input data is increasingly important with the proliferation of tool use and agents in LLMs that retrieve and process external data. Lastly, these tokens can potentially be exploited to more easily circumvent guardrails by pushing the model beyond its trained distribution [8]. Although some work has been done on identifying such tokens through model and tokenizer analysis [9, 10, 11], there is a lack of reliable and well-explained automated methods that are tested across a wide range of models. Reliable tools for detecting tokenizer problems provide not only a way to test and iteratively improve the development of tokenizers, but can also provide a way to protect deployed models from unwanted input via input sanitization.
\ In this work, we present effective and efficient techniques for identifying such problematic tokens based on the model (un)embedding weights and tokenizer configuration. We apply these methods to a range of popular and recent open-weight models, including the Cohere Command R, Google Gemma, Meta Llama2, Mistral, Alibaba Qwen and OpenAI GPT-2 models. Finally, we include a brief exploration of extensions of these techniques to closed-source models. We also publish a general analysis tool compatible with Hugging Face models, along with detailed results for each analyzed model.
\
Our method consists of three steps; i) first, we perform a tokenizer analysis by inspecting its vocabulary and observing its encoding/decoding behaviour, ii) second, we calculate a number of indicators that identify candidate tokens that have likely not been seen during model training, and iii) third, we verify whether identified candidate tokens are indeed out of distribution by prompting the target model.
We start by defining a number of useful categories for tokens:
\ • Partial UTF-8 sequences: The token contains a partial UTF-8 sequence and can not be converted to a string by itself. This is typical for ‘fallback byte’ tokens in the 0x80-0xFF range (also see Appendix B), but depending on tokenizer configuration, can also include a combination of full and partial characters.
\ • Unreachable: When no input string can result in the token id, we categorize it as ‘unreachable’. We test this by checking if decoding the token to a string, and re-encoding it again, results in the token. Such tokens are typically the result of tokenizer configuration errors or conflicts between trained and manually added vocabulary. As this test does not work when tokens can not be decoded to a string, we exclude partial UTF-8 sequences from this category.
\
• Special tokens: Manually defined tokens carrying specific meanings as control tokens, such as . We identify special tokens using the patterns and […] and list them separately from unreachable tokens, even if they might be considered as such due to input sanitization in tokenizer preprocessing.
\ • Tokens not in any of the other categories, which constitute the vast majority.
\ We detect and exclude partial UTF-8 sequences and unreachable tokens from our token detection pipeline, as they are not suitable for automatically building verification prompts. Our published model reports include tables with such tokens, and we briefly discuss some interesting model-specific results in section 3.3.
\
We propose and use model architecture-dependent indicators to identify potentially under-trained tokens. An key distinction is made based on whether a model uses the same matrix for its token embeddings E and the final model layer, consisting of the ‘unembedding’ matrix, U, which converts the final internal embeddings to a probability distribution over tokens.[1] Regardless of model architecture, all weights of the unembedding matrix influence the token predictions at every training step. Specifically, the training loss is minimized when the probability of unused tokens is predicted as zero, regardless of the input, making their logits converge towards −∞. The model can achieve such an input-independent prediction by a constant vector in the residual stream, and the negative of this vector in rows of the unembedding matrix, resulting in a constant negative contribution to the logit values of unused tokens. Using this intuition, we can find unused tokens from the unembedding weights as follows:
\
\
\
Our proposed indicators naturally provide a ranking of candidate under-trained tokens, but do not give a definitive threshold, and their relative simplicity is likely to result in a somewhat noisy relation between indicator and model behaviour. To confirm that candidate tokens indeed induce unwanted model outputs, we verify all tokens which rank among the most likely 2% according to the chosen indicator, excluding partial UTF-8 sequences and unreachable tokens. This verification process involves constructing specific repetitive prompts that induces a high output probability for normal tokens, and checking if a candidate token has a very low output probability (see Appendix A for details).
\ \
:::info Authors:
(1) Sander Land, Cohere s([email protected]);
(2) Max Bartolo, Cohere ([email protected]).
:::
:::info This paper is available on arxiv under CC BY-SA 4.0 DEED license.
:::
[1] We assume the conventional final layer structure, consisting solely of the unembedding matrix without a bias.
2025-05-12 16:35:45
On the night of May 5th to 6th, HyperSwap was hit by a massive DDoS attack that impacted both their website and application. As HyperSwap was already using some of our cloud solutions, we quickly stepped in to support them migrating their front-end and redirecting the attack traffic to mitigate its effects.
\ HyperSwap's infrastructure initially lacked a proxy and did not have sufficient protection against DDoS attacks. However, they had deployed a fallback version of their application via IPFS, pinned on our network, in case their main servers were compromised.
\ Unfortunately, on the night of the attack, HyperSwap’s main server was overwhelmed. While the API remained live, its performance degraded significantly. Their team redirected traffic to our network in an emergency move, which caused some turbulence before we could fully respond.
\ Meanwhile, our dev team activated our internal anti-DDoS system, offloading attack traffic to a black hole and stabilizing the situation.
\
The Aleph team quickly stepped in to support us by ensuring the IPFS-hosted version was accessible, allowing users to continue accessing the app while we worked on mitigation. At the same time, the Imperator team, with their experience in indexing and handling high-throughput environments, acted swiftly and effectively. They immediately understood the situation and deployed the necessary resources to counter the attack, setting up new proxies, implementing alert systems, and reinforcing our infrastructure. —
\ On our side, we helped the HyperSwap team migrate their website to our internal anti-DDoS platform and redirected the attack to a black hole. This incident marked the first successful deployment of Aleph Cloud’s experimental anti-DDoS solution, developed to protect decentralized applications from targeted disruptions.
\
\ \ What Is a Remotely-Triggered Black Hole (RTBH)?
\ RTBH filtering is a network security technique used to mitigate DDoS attacks by redirecting malicious traffic to a null route, effectively making it disappear. Think of RTBH as a trapdoor at your network's entrance: when a flood of harmful traffic is detected, RTBH diverts it into this black hole, preventing it from reaching or slowing down the network and its users.
HyperSwap plans to deploy their APIs on Aleph Cloud to create a fully decentralized version of their application, one that’s resilient to future attacks.
\ We will continue to develop and improve our anti-DDoS solution, which proved effective in this real-world test. This successful mitigation highlights the power of decentralized infrastructure to deliver security, resilience, and true censorship resistance.
2025-05-12 16:07:53
Hey everyone! It’s been a while since my last article, as life got pretty busy after wrapping up my Connect Four project. Anyway, I’ve been making slow but steady progress on the Chess Game, which is the final Ruby project from The Odin Project’s curriculum. Honestly, it’s been quite the journey: fun, massive, intense, and sometimes really frustrating. I figured I’d share my experience to help anyone else gearing up to tackle it, trust me, it’s not as scary as it seems!
Believe it or not, I started mentally preparing for Chess long before even beginning Connect Four. I’d caught snippets of conversations on Discord, and people kept saying how huge and complicated it was. What really caught my eye was how everyone seemed stuck around the 70% completion mark. Another thing is that many folks skipped Test Driven Development because of how huge this project was. That was enough info for me at this point, I didn’t want any spoilers, so I stopped reading!
\ When I finally got around to starting, my first step was drafting some pseudocode and getting the basic logic straight in my head. I had a few ideas, but I couldn’t shake the memory of the Knight Travails project, which I’d thought would help me solve the Chess challenge.
\
\
\ \
Initially, I focused on designing the chessboard and creating all the chess pieces to display them clearly in the console, no moves yet, just the basics. At first, being a good student, I wanted to follow TDD right from the start. But pretty quickly, I realized why everyone had warned it was tough.
\ I managed to write some basic tests for early methods, but soon enough, I was staring at my screen overwhelmed. Each problem I solved seemed to spawn two more, like some frustrating spiderweb. Eventually, I set TDD aside just to focus on making progress. I felt a bit guilty ditching tests because I genuinely liked them during Connect Four.
\ However, I promised myself I’d circle back once everything was up and running smoothly.
\
At this stage, I had a rough game class, a method to handle rounds, and my board displaying nicely but zero moves programmed. It was time to roll up my sleeves and start coding chess moves. I knew I’d need special pawn moves and king-rook castling eventually, but decided to start with basic movements first. Knight Travails was helpful, or so I thought.
\
\ \ It didn’t take long to hit another snag: chess isn’t just about movement, but also about dealing with other pieces. So I had to go back, refactor the code, and figure out how to handle situations like taking opponent pieces or preventing moves onto spaces already occupied by your own pieces.
\ Honestly, that refactoring part was a total pain, especially after finishing the knight’s moves only to realize the knight “jumps” rather than slides, forcing yet another rewrite.
\
\
\ \ Once basic moves were finally done, I tackled the special pawn moves like advancing two squares on their first move, which was tricky but manageable. Castling, though, was another beast altogether! I actually took a break to clear my head, switching back to refining the game class, adding clarity, extra variables, and cleaning things up.
\ By this point, things were coming together, and castling started to look less intimidating. But now I knew JSON serialization was looming, and frankly, that scared me even more than castling did!
\ After a few intense hours, I finally conquered castling, minus a couple of debugging sessions that nearly drove me nuts. Once sorted, I realized my prompts to the player were broken, so I fixed those too.
\
\
\ \
Right after this, I took a much-needed holiday in Poland. It was my first trip abroad in three years, delayed by waiting for settled status. I’d ambitiously promised myself I’d code every morning from 6:30 to 11, but you know how it goes…
\ Friends called, family invited me over for coffee and cake, and suddenly coding turned into just an hour per day. But honestly, the downtime helped me recharge, even though it completely wrecked my sleep schedule!
\
\ \ While in Poland, I tackled JSON serialization because I’d struggled with it before and needed to get comfortable with it. Honestly, it was incredibly frustrating. Everyone online seems to talk casually about JSON, but organizing everything into hashes, especially nested ones for chess pieces, was brutal. Maybe my design made things harder than necessary. Anyway, after three tough weeks, I finally nailed it!
\
\
\ \ Back in the UK, my final week was dedicated to tests (yes, I finally returned to TDD!) and polishing prompts with some colorful console messages to make gameplay clearer and more enjoyable. And guess what? It’s DONE!
\
\
\ \ I’m genuinely thrilled to be finished this project felt like it took forever! Now, I can finally move forward. No rush, obviously, but I’m excited for what’s next.
\
If you’re about to tackle this Chess Game, my main advice is patience. Take your time upfront to really map out your design. Grab a pen and paper, jot down your ideas, and plan thoroughly. Trust me, skipping proper planning landed me in a mess of redundant code and wasted effort. A couple more hours of preparation could’ve saved me heaps of trouble.
\ Good luck, and more importantly, have fun with it!
2025-05-12 16:05:36
In April 2025, Windsurf (formerly Codeium) rolled out significant pricing updates that made waves in the developer community. First, the company provided OpenAI’s o4-mini and GPT-4.1 (equipped with 1 million token context window) freely available to all users for a 2-week time period. Immediately after this free trial period, on April 29, Windsurf followed up with a major revamp of their pricing and offering.
\ Free plan users now receive 25 prompt credits per month (up from 5). That is equivalent to 100 prompts with GPT-4.1, o4-mini, and other premium models. Devs will be charged once per prompt, regardless of the number of actions Cascade performs in response. This simplification will make it easier for users to anticipate costs and manage their credit usage effectively.
\ Added to that package are unlimited Cascade, Fast Tab Completions, and Command. This is a significant upgrade designed to facilitate a more agentic experience. The free plan now also includes unlimited interactive app Preview and 1 App deploy per day (see Windsurf’s Netlify integration)). When assessing Copilot and Cursor’s free model, Windsurf’s proposal looks rather appealing.
\
The expanded free credits greatly encourages tinkering and exploration. With 100 GPT-4.1 prompts per month, devs have a decent starting point to build and test small scale projects. Simply put, developers have a cost-free sandbox enabling more experimentation and knowledge-sharing.
\ Price-sensitive developers, notably university students and junior devs, will also stand to benefit from this change. It’s worth noting that though many professional devs can get IDEs like Cursor reimbursed by their companies, many established traditional firms are slower with adopting these new tools. This is an opportunity for professional devs (either at work or in their own time) to dabble with it. Following this, we expect to see a wave of companies switch to the IDE and become part of Windsurf’s enterprise customer base.
The reaction across developer communities has mainly been energetic. Devs see it as a “breath of fresh air” in a market full of subscription-based tools. The inclusion of unlimited Cascade Base usage and Tab Completions also means the free version no longer feels “crippled” compared to Pro.
\ Students voiced excitement that they could use GPT-4.1 in a coding IDE; most enjoyed the pace and quality of the outputs. Many users who had not tried AI coding tools indicated they were now downloading Windsurf to give it a shot, since the usual cost barrier was gone. Developers also appreciated the pricing simplification and expressed relief that the confusing Flow Actions system was addressed.
\
\ People no longer feel they’re “burning credits” unknowingly in the middle of a coding session. By charging only for prompts (and not for each action made by the agent), Windsurf earned goodwill. On the whole, Windsurf’s strategy succeeded in getting the community’s attention, and if you haven’t yet tried Windsurf, there is merit in trying it out.
\ That said, watching how devs behave in the coming months (do they stick with Windsurf? Convert to paid? Revert to Copilot?) will be the true test to the viability of this pricing structure.
With the current subscription fatigue, Windsurf’s move reflects the entry towards a new free tier model, and could prompt competitors to adjust. We had already seen signs of this shift in the industry. For instance, GitHub Copilot, notorious for not providing a free tier for individuals, has introduced a limited free tier (2,000 completions/month, 50 chat requests, and more).
\ As a developer, you might wonder whether prompt-based metering is the right approach at all. It can indeed be a little stressful. Many feel it should be value-driven, and that a fixed-price model would be the game changer (also who is paying for this? The foundation model folks? Split?). It’s worth noting this may only be a temporary reshuffling, as others are likely to follow suit.
\ This also raises the bigger question: what would ideal pricing even look like? Free tiers aren’t sustainable forever. Will we see the shift toward premium features such as plugins, agents, team plans, or corporate offerings? For developers, this competition is fantastic news! It means more opportunities to try AI tools without committing upfront.
\ Perhaps the biggest wildcard in Windsurf’s future is the possibility of being acquired by a larger player. Notably, OpenAI is reportedly in talks to acquire Windsurf for ~$3 billion. We could see OpenAI investing in Windsurf’s development, providing it with the latest models (at a discounted price?) while keeping it as a product for developers. This would have similar flavours to Microsoft's acquisition of GitHub, and its decision to let it continue to operate the platform.
\ One sound piece of advice is that developers shouldn’t commit just yet. We are still in the beginning phases of the AI Native IDE war, and there is no clear winner. Windsurf’s free tier can easily be part of your toolbox, given its generous free offering. However, there’s always more to selecting an IDE product than pricing! Over time, and as the space evolves, you might consolidate if one tool leaps ahead, but for now, a hybrid approach hedges against any single platform’s cost and capability limits.
\ One thing’s for sure: the IDE is evolving and becoming increasingly AI Native. As developers, this is our time to have fun exploring these tools and raise the bar in our workflows.
\
