Qwen, Minus the E

Let’s take a look at the following exchange.

Prompt: “How much wood could a woodchuck chuck if a woodchuck could chuck wood?”

Response: “This is a classic riddiculous and humorous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous”

Ok, clearly there’s something wrong with the output. But do you notice anything else about this response? Right, not a single word has the character ‘e’.

A little while back, I saw an RL project on teaching a language model to draw flowers. Inspired, I also looked at tweaking a model for some “artistic” goal. I wanted to see if it could be trained to output lipograms, i.e., text that omits a specific character. Authors have famously written whole books (Gadsby, La Disparition) this way, and I wanted to see what an LLM could manage.

Turns out, this task is rather challenging, similar to why LLMs (used to) struggle with counting the number of ‘r’s in “strawberry”: they read text in tokens, and not words, so whether something contains an ‘e’ is something the model cannot directly see. Regardless, I decided to explore a bit.

I began with getting Qwen to output with no ‘e’s. I set up the forward pass, loaded in the weights, and began training some LoRA adapters. With some tweaking, I had RLed (see appendix) an ‘e’-free version within a couple of hours. I also realized I could just mask out the ‘e’ tokens from the logits, which gave the ‘e’-free requirement for “free”.

What do we do now?

Either way, the output is often incoherent (like the woodchuck answer at the top). Ideally, the answer should be understandable to the person reading it. So we need a more thorough goal.

We also use the mask in most experiments, since it handles the ‘e’ rule for free. To keep things clear, I’ll use two names throughout:

• Qwen: Qwen3-1.7B with no mask
• Baseline: Qwen3-1.7B with the mask

Defining the Goal

First, I had to decide what context I wanted to evaluate Qwen on. Should we evaluate its ability to write prose? To rewrite sentences? I decided a good outcome would be the ability for the model to answer prompts, since that’s a pretty general use-case for LLMs. I also kept the answers short (capped at 300 tokens, roughly 200 words).

For the prompts, I built a dataset from existing sources (Dolly and No Robots, see appendix) and split it into 18,000 training, 200 validation and 500 test prompts. Each experiment samples what it needs from these splits, with prompts that can’t be answered without an ‘e’ removed (like “Who painted the Mona Lisa?”).

Next, let’s define a clearer set of expectations for evaluations, beyond just being ‘e’-free. After all, the riddiculous response from earlier is technically ‘e’-free. I came up with the following guidelines:

• Contain no ‘e’ or ‘E’, including accented substitutes.
• Answer the actual prompt.
• Use normal English, with correct spelling and coherent ideas.

We can evaluate the first condition with a few lines of code. To evaluate the second and third requirements, I decided to use GPT-6 Luna to judge the responses. The judge scores each response from 1 to 5 on two things: coherence, and whether it responds to the prompt. A response is usable if it’s ‘e’-free and scores at least 4 on both. I report every result as the usable rate: the share of answers on the eval set (50 prompts sampled from the validation split, with 4 answers each, so 200 answers) that are usable. This fit nicely in our evaluation set-up, and eventually in our training set-up as well (the judge’s scores become the RL reward later on).

A couple of things to keep the judge fair:

• The judge isn’t told about the ‘e’ rule, so it can’t go easy on a weird answer just because the task is hard.
• Wrong facts don’t count against a response, as long as it attempts the prompt. A 1.7B model gets plenty of facts wrong anyway, and I only care about the writing here.

Before we continue, we want to make sure our eval is suitable; we ran Qwen and evaluated only the second and third requirement, and made sure it could pass the check reliably (scores of ≥ 4), aiming for >80% pass rate. We also tweaked the prompts to catch common issues (repeating lines, answers being cut off by token limits). Before any training, we had:

After we confirm the evaluation is suitable (see appendix for the judge code), the goal is then to improve that 26% as much as possible.

Experimenting with SFT

Although “coherence” and “responds to the question” are values that are more subjective than containing the letter ‘e’, I wanted to explore if it were possible to improve these two values through supervised training.

One idea I tried was to SFT on a large corpus of ‘e’-free text, with the hopes of the model learning more cohesive patterns of ‘e’-free tokens strung together. I did this by SFTing the model on the Gadsby novel, which contains 50,000 words or roughly 69,000 tokens. However, as the loss decreased on the Gadsby runs, the model’s responses were not becoming any better. On our eval set of prompts, it scored roughly the same as our baseline rate.

Some of the responses looked like this (the numbers in parentheses are the judge’s coherence and responds scores):

There were two doubts I had. These training runs were training the model for generation, but we were evaluating it based on response. In addition, 69,000 tokens might not have been enough to produce real differences. This led to our second experiment.

We want to SFT on response-type answers. There weren’t any readily available datasets online, so I ended up creating my own. From the original training dataset of prompts, I took 180 suitable prompts and used frontier models to generate ‘e’-less responses. As an observation, the frontier models aren’t able to always one-shot an ‘e’-free response, but can usually correct their mistakes quickly when told to replace certain words (this did not work with Qwen3-1.7B, though).

Then, I measured the usable rate on the eval set, with varying amounts of training examples used. For each group, we ran passes through the data until the validation loss grew (signs of overfitting).

As a side note, the 300-token cap only applies when the model generates answers (in the eval and during RL). SFT trained on the full answers, with chats up to 3,072 tokens, so the 180 examples come to about 117,000 tokens, compared to 69,000 for Gadsby. The 90-example run is the closest match in size, and it still did better than the Gadsby run (31.5%, against roughly the 26% baseline).

We observed the following:

• The best adapter (180 examples) is 9 points above the baseline. I’ll call it the SFT adapter from here on. This seems promising as another starting point to measure in our later experiments.
• The gains are leveling off. Doubling from 90 to 180 examples added only 3.5 points, so writing more examples of this kind probably won’t help much more.
• Small training sets hurt. With 15 examples, answers got longer and worse.
• Answers got cleaner but drifted off topic. When the key word needs an ‘e’, the adapter swaps in another word and answers about that one instead (“clik” for “cliché”).

At this point, I decided to begin our RL experiments, since the gains from SFT seemed limited.

RLing for Coherence

To apply RL on this problem, we can design a simple system:

pick some prompts → sample some responses for each prompt → score the responses based on our guidelines → use a reward function based on these scores to push up the better responses → let the model learn through mass amounts of prompts.

In more detail, each training step was:

1. Pick 4 prompts from a pool of 1,235 training prompts (see appendix for where these came from).
2. Generate 8 answers per prompt with the mask on. The mask blocks ‘e’, ‘E’ and non-ASCII characters. Sampling uses temperature 0.7 and up to 300 tokens, with thinking off. The no-‘e’ system message is the same one the eval uses.
3. Judge all 32 answers in parallel with GPT-6 Luna. Each answer gets a coherence score and a responds score, both from 1 to 5.
4. Turn the scores into a reward: reward = (coherence − 1 + responds − 1) / 8. That gives a number from 0 (both 1) to 1 (both 5).
5. We then compare each answer to its siblings (we used GRPO as our algorithm of choice). Within each prompt’s group of 8, subtract the group’s average reward and divide by its spread. An answer gets pushed up only if it beat the other answers to the same prompt.
6. Update the adapter once. Settings: learning rate 2e-5, only the rank-16 LoRA weights train, and a KL penalty (0.02) pulls the model back toward where it started.

For this training setup, I decided to train the baseline and the SFT adapter, to see if the SFT had any lasting benefits beyond the initial increase in score.

One important step before running the algorithm was to confirm the model had something to learn from. For our setup, we need both at least one usable answer and differing scores within a group. On a sample of 50 prompts with 16 tries per prompt, the baseline had at least one usable answer for 78% of prompts, and almost every prompt’s answers got different scores. We got similar results with the SFT adapter, so it seemed fine to continue.

I then ran 300 steps on each model. I decided on 300 as the cutoff since that would give us roughly one pass over that pool, but I made sure to log results every 25 steps as well.

Results

On our original eval set, we get the following results:

We see that both models improved after the run. We make the following observations:

• The SFT adapter is the better place to start RL from. It starts higher and gains more. So the SFT training data did have lasting benefits.
• Most of the improvement comes around step 125. After that, the gains were mostly flat for both models.

Here’s what that looks like on a few prompts. We can see real improvements, but also a pattern that we should investigate further.

Notice how short some answers got. “SNCF” and “football” are one word each. “Football” only half answers the question, and the cliché answer doesn’t answer it at all, but both score a 5 for coherence.

Answers got much shorter in both runs:

Is the improvement just from shorter answers then?

For the baseline, mostly yeah. All of the gain came from coherence, the score a shorter answer makes easiest to earn, and none from answering the question better. Clean answers that skipped part of the question went from 24 to 66 out of 200. The likely cause is the reward mechanism. It adds the two scores together, so a 5 for coherence and a 3 for responding earns the same reward as a 4 and a 4, even though only the 4 and 4 counts as usable. A short, clean half-answer was the safe bet.

For the SFT adapter, no. Both scores went up, and the usable rate rose at every answer length. So the gain there wasn’t just from being shorter.

In conclusion, the best result came from first running SFT on 180 ‘e’-free examples, then 300 steps of RL, taking the usable rate from 26% to 60.5%.

Lastly, let’s revisit the original question with the SFT adapter:

Prompt: “How much wood could a woodchuck chuck if a woodchuck could chuck wood?”

Response: “As many as it could, assuming it can and wanting to.”

Pretty cool.

Takeaways and Next Steps

Throughout this project, I gained a lot more insight into post-training methods for LLMs, implementation details of RL algorithms, and many other things (memory calculations, how to prepare datasets, how to run experiments, etc.).

Some general takeaways I had while working on this were:

• It’s important to have a clear goal defined. Before I defined the goal of the project, the project was just learning and exploring LLM fine-tuning, which felt aimless.
• For projects, it’s better to work in sprints. In the middle, I was balancing school and interview prep while working on the project, which made it really hard to think deeply about the project’s direction every day. Rather than chipping away at the project, it felt much better to define milestones and work until they were reached.
• Visibility (logging runs and saving responses) is really valuable in training runs, especially for RL. The loss is just a value we calculate off of some metrics, and it’s hard to cover all bases. Being able to check and validate that the model’s responses are actually heading in a desired direction is important.

Regarding the overall project, I definitely believe there are improvements to be made, especially in designing more rigorous experiments and finding training algorithms that lead to better results. However, I’m satisfied with what I was able to learn and the results I saw, and would like to spend time learning other things.

For what’s next, there are quite a few things that I want to explore:

• Learn more about the fundamentals of RL
  • i.e., learning the mathematical frameworks behind it, and implementing more common algorithms and understanding the ideas behind them
  • I definitely want to try applying RL to a video game at some point…
• Look at more modern post-training work with models/agents
  • for next projects on RLing an LLM, I think I’ll probably try using pre-existing packages/apis and paying for compute to run larger explorations.

References and Reading

• original blog on painting flowers
• paper on using LLMs to transform text into lipograms
• blog on masking out ‘e’s for Llama-8B

Appendix

Project Timeline

the process of the project was roughly (blog covers bold sections):

• scoping out the project, choosing model and doing preliminary reading
• writing out the forward pass, wiring in optimizations (see appendix for more on this)
• straightforward attempts at making the model generate a response without the character ‘e’ (RLing the base model to not use ‘e’ on a specific prompt, trying out the mask)
• defining the goal (what objective do we want to see after fine-tuning? what kind of data do we need?)
• fooling around with implementing “Trainer” classes from TRL on my own (see the “Engine” classes in the appendix)
• sft attempts
• rl for coherence
• cleanup + writeup

GitHub Repo

the code, data and results are all in kevinyvv/qwen-3-1.7b-finetune-proj. some parts of interest may be:

• forward pass
• simple_train.py (the first attempt)
• SFTEngine, RLEngine (REINFORCE), GRPOEngine
• the judge and the ‘e’-free answers used for SFT
• results and findings: run records and the full experiment notes

Model choice

I chose Qwen3-1.7B since I could fine-tune it locally with 16 GB of VRAM. Its architecture was also simpler than those of newer models, so it was easier to understand and implement the forward pass.

The only pre-planning I really did otherwise was learning how to choose a model that I could work on locally. The details are collapsed by default, but feel free to read further if you’re curious about some of the mem calculations.

Sourcing/Building the dataset

datasets used: No Robots, Databricks-Dolly

I removed code-writing requests, empty examples, duplicate prompts, prompts over 150 words, and examples whose human answers exceeded 300 words. I also dropped No Robots’ persona-based chat examples, since instructions like “answer in rhymes” would add another constraint.

After filtering and making my own splits, I had (please note that most prompts in the dataset ended up being unused):

One additional problem I discovered: some prompts just can’t be answered without an ‘e’. “What’s the capital of Belgium?” needs “Brussels”. “Who painted the Mona Lisa?” needs “Leonardo”. I filtered these out for all prompts used in training and testing.

Instead of using the splits whole, I sampled from them whenever an experiment needed data. The eval set is 50 prompts from the validation split. The RL pool started as 2,000 prompts from the training split, and subagents filtered it down to 1,235.

RLing for no ‘e’

The first step was making the model output a calculated response with no ‘e’ tokens (as opposed to by luck). After implementing the forward pass and adding LoRA, I froze the original weights and trained only the adapters. I used eight copies of the same prompt, with thinking disabled and a maximum of 64 generated tokens:

respond without using the letter 'e'

The idea was to overfit this one example before worrying about a dataset or generalization. Generate a response, score it, and use that score to update the model. How hard could avoiding one letter be?

I started by giving a response one point for every word without ‘e’. That still gave positive rewards to answers that broke the rule, as long as they contained some valid words. I tried centering rewards across the batch to favor better responses, but the samples were often very similar. I also wanted a more specific signal about which parts of a response were causing problems.

So I moved to a reward lookup table for individual tokens:

for tok, idx in tok_to_id.items():
    if 'e' in tok.lower() and tok != eos_token:
        e_table[idx] = -1.
    else:
        e_table[idx] = 0.5

Tokens aren’t necessarily whole words, though. With a split like gr + een, penalizing only the second token misses the earlier choice that helped create the word. I used discounted future rewards so a later penalty could also affect earlier choices. These are shortened excerpts from the return calculation and policy loss:

# Inside discount(), working backwards through the response:
for t in reversed(range(rewards.size(1))):
    running = rewards[:, t] + gamma * running
    returns[:, t] = running

# Back in the training loop:
rewards = get_feedback(gen_ids, e_table)
per_token_reward = discount(rewards)
advantage = (per_token_reward - per_token_reward[mask].mean()) * mask
loss = -(log_probs * advantage).sum()

Subtracting the mean gave each token an advantage relative to the valid token returns in that batch. A positive advantage pushed up the probability of the sampled token; a negative one pushed it down.

This sort of worked. Some responses settled on “i want to know what you want to know”, which follows the rule but isn’t much of an answer. Other samples found an easier way to collect rewards (excerpt):

i m k t r n l d g h j z p q w x y f v b c …

I was rewarding tokens without ‘e’, and a stream of letters could satisfy that objective without forming words. The reward values changed across retries, but none of these rules directly measured whether the output made sense.

My next idea was to penalize drift from the original model with a KL penalty. I ran the same generated sequence through the model with LoRA disabled, then compared the log probabilities assigned to the sampled tokens. We then added this adjustment to the reward:

kl_approx = (log_probs.detach() - ref_gen_log_probs) * mask
rewards = get_feedback(gen_ids, e_table) - kl_beta * kl_approx

That was supposed to discourage the model from abandoning its original language behavior, but the outputs still fell apart. I later found that the KL penalty was also too small relative to the other rewards. One saved sample was:

Tst, tkt, nkt, tkt, nkt, tkt.

I tried smaller bonuses, bonuses scaled by token length, stronger penalties for ‘e’, and changing the discount factor. Some runs produced nonsense; others got stuck repeating the same sentence. A stronger penalty from the start also encouraged single-letter and newline outputs.

From this initial testing, I realized looking at actual samples matters. These runs showed how easily rewards could end up being gamed, even with countermeasures.

1.7B vs 4B

At some point, I was hypothesizing that 1.7B was just too small of a model. I also tried comparing Qwen3-1.7B vs 4B to see if I should’ve just chosen a larger model to work on. On the eval set, I saw:

We see that neither model can follow the rule from the instruction alone. Not a single answer avoided ‘e’, so we need the mask. Furthermore, the 4B model is worse under the mask. It keeps writing the words it planned and just drops the ‘e’s, giving things like “th” for “the” and “amusrmt park.” The 1.7B model more often switches to a different word.

So we stayed on 1.7B.