Qwen, Minus the E

A little while back, I saw an RL project on teaching a language model to draw flowers. Inspired, I also looked at tweaking a model for some “artistic” goal. I wanted to see if it could be trained to output lipograms, i.e., text that omits a specific character. Authors have famously written whole books (Gadsby, La Disparition) this way, and I wanted to see what an LLM could manage.

Turns out, this task is rather challenging, similar to why LLMs (used to) struggle with counting the number of ‘r’s in “strawberry”: they read text in tokens, and not words, so whether something contains an ‘e’ is something the model cannot directly see.

Regardless, I decided to explore a bit. I ended up with some nice results, from fine-tuning Qwen3-1.7B on my own 16 GB GPU to answer everyday prompts without using the letter ‘e’.

To demonstrate, here’s a comparison of the before and after.

I'll walk through how I got there throughout this post: first removing the ‘e’, then deciding what counts as a good answer, and then the final SFT and RL experiments for coherence. For some of the technical implementations (forward pass, SFT, RL (GRPO/REINFORCE), etc.), see the appendix at the bottom.

Removing ‘e’ from the Output

My first attempt was to train the ‘e’ away with RL. Supervised training needs example answers for the model to copy, and it’s hard to say exactly what a good ‘e’-free answer should look like. RL only needs a way to score the model’s own answers, and checking for an ‘e’ takes a few lines of code.

I set up the forward pass, loaded in the weights, and began training some LoRA adapters. With some tweaking, I had RLed an ‘e’-free version within a couple of hours. The catch was that the model found plenty of ways to avoid ‘e’ without saying anything useful, like writing streams of random letters (the full story is in the appendix).

Then I realized I could get the same thing without any training. Before the model picks each token, I can set the score of every token containing an ‘e’ to negative infinity, so it can never be picked. (The mask also blocks ‘E’ and non-ASCII characters, so look-alikes such as ‘é’ can’t sneak in.) This mask gives the ‘e’-free requirement for “free”.

But there’s a problem. Here’s an answer with the mask on:

Ok, clearly there’s something wrong with the output. Not a single word has the character ‘e’, but the answer is nonsense. The mask only stops the model from writing certain tokens. It doesn’t teach the model to choose words that avoid ‘e’ in the first place, so it often ends up repeating itself or misspelling words.

So what separates a response like that from a response like this?

The answer is coherence. Ideally, the answer should be understandable to the person reading it, and actually answer the prompt. So we need to define a more thorough goal.

Since the mask handles the ‘e’ rule for free, we use it in most experiments. To keep things clear, I’ll use two names throughout:

• Qwen: Qwen3-1.7B with no mask
• Baseline: Qwen3-1.7B with the mask

Defining the Goal

I chose to test Qwen on answering prompts, since that’s a pretty general use for LLMs. Answers are kept short (capped at 300 tokens, about 200 words). The prompts come from two existing datasets, Dolly and No Robots (see appendix). I removed any prompt that can’t be answered without an ‘e’, like “Who painted the Mona Lisa?”

An answer counts as usable if it passes three checks:

• No ‘e’ or ‘E’, including accented versions. A few lines of code check this.
• It answers the prompt.
• It reads as normal English, with correct spelling and ideas that make sense.

For the last two, GPT-6 Luna acts as a judge and scores each answer from 1 to 5. An answer needs at least a 4 on both. (See the appendix for how I kept the judge fair.)

Every result in this post is a usable rate: the percentage of usable answers out of 200 (50 prompts from the validation split, 4 answers each). Later on, these judge scores also become the reward for RL training.

Before any training:

Without the mask, not one answer avoided ‘e’, even when told to. The goal for the rest of the post is to push the baseline’s 26% as high as possible.

Experimenting with SFT

Although “coherence” and “responds to the question” are values that are more subjective than containing the letter ‘e’, I wanted to explore if it were possible to improve these two values through supervised training.

One idea I tried was to SFT on a large corpus of ‘e’-free text, with the hopes of the model learning more cohesive patterns of ‘e’-free tokens strung together. I did this by SFTing the model on the Gadsby novel, which contains 50,000 words or roughly 69,000 tokens. However, as the loss decreased on the Gadsby runs, the model’s responses were not becoming any better. On our eval set of prompts, it scored roughly the same as our baseline rate.

See some of the responses below, with the judge’s coherence and responds scores under each answer:

There were two doubts I had. These training runs were training the model for generation, but we were evaluating it based on response. In addition, 69,000 tokens might not have been enough to produce real differences. This led to our second experiment.

We want to SFT on response-type answers. There weren’t any readily available datasets online, so I ended up creating my own. From the original training dataset of prompts, I took 180 suitable prompts and used frontier models to generate ‘e’-less responses. As an observation, the frontier models aren’t able to always one-shot an ‘e’-free response, but can usually correct their mistakes quickly when told to replace certain words (this did not work with Qwen3-1.7B, though).

Then, I measured the usable rate on the eval set, with varying amounts of training examples used. For each group, we ran passes through the data until the validation loss grew (signs of overfitting).

As a side note, the 300-token cap only applies when the model generates answers (in the eval and during RL). SFT trained on the full answers, with chats up to 3,072 tokens, so the 180 examples come to about 117,000 tokens, compared to 69,000 for Gadsby. The 90-example run is the closest match in size, and it still did better than the Gadsby run (31.5%, against roughly the 26% baseline).

We observed the following:

• The best adapter (180 examples) is 9 points above the baseline. I’ll call it the SFT adapter from here on. This seems promising as another starting point to measure in our later experiments.
• The gains are leveling off. Doubling from 90 to 180 examples added only 3.5 points, so writing more examples of this kind probably won’t help much more.
• Small training sets hurt. With 15 examples, answers got longer and worse.
• Answers got cleaner but drifted off topic. When the key word needs an ‘e’, the adapter swaps in another word and answers about that one instead (“clik” for “cliché”).

At this point, I decided to begin our RL experiments, since the gains from SFT seemed limited.

RLing for Coherence

To apply RL on this problem, we can design a simple system:

pick some prompts → sample some responses for each prompt → score the responses based on our guidelines → use a reward function based on these scores to push up the better responses → let the model learn through mass amounts of prompts.

In more detail, each training step was:

1. Pick 4 prompts from a pool of 1,235 training prompts (see appendix for where these came from).
2. Generate 8 answers per prompt with the mask on. The mask blocks ‘e’, ‘E’ and non-ASCII characters. Sampling uses temperature 0.7 and up to 300 tokens, with thinking off. The no-‘e’ system message is the same one the eval uses.
3. Judge all 32 answers in parallel with GPT-6 Luna. Each answer gets a coherence score and a responds score, both from 1 to 5.
4. Turn the scores into a reward: reward = (coherence − 1 + responds − 1) / 8. That gives a number from 0 (both 1) to 1 (both 5).
5. We then compare each answer to its siblings (we used GRPO as our algorithm of choice). Within each prompt’s group of 8, subtract the group’s average reward and divide by its spread. An answer gets pushed up only if it beat the other answers to the same prompt.
6. Update the adapter once. Settings: learning rate 2e-5, only the rank-16 LoRA weights train, and a KL penalty (0.02) pulls the model back toward where it started.

To show how GRPO works in this setup, here’s step 5 on an example group:

For this training setup, I decided to train the baseline and the SFT adapter, to see if the SFT had any lasting benefits beyond the initial increase in score.

One important step before running the algorithm was to confirm the model had something to learn from. For our setup, we need both at least one usable answer and differing scores within a group. On a sample of 50 prompts with 16 tries per prompt, the baseline had at least one usable answer for 78% of prompts, and almost every prompt’s answers got different scores. We got similar results with the SFT adapter, so it seemed fine to continue.

I then ran 300 steps on each model. I decided on 300 as the cutoff since that would give us roughly one pass over that pool, but I made sure to log results every 25 steps as well.

Results

On our original eval set, we get the following results:

We see that both models improved after the run. We make the following observations:

• The SFT adapter is the better place to start RL from. It starts higher and gains more. So the SFT training data did have lasting benefits.
• Most of the improvement comes around step 125. After that, the gains were mostly flat for both models.

Here’s what that looks like on a few prompts. We can see real improvements, but also a pattern that we should investigate further.

Notice how short some answers got (pick the station, baseball, and cliché prompts above to see them). “SNCF” and “football” are one word each. “Football” only half answers the question, and the cliché answer doesn’t answer it at all, but both score a 5 for coherence.

Answers got much shorter in both runs:

Is the improvement just from shorter answers then?

For the baseline, mostly yeah. All of the gain came from coherence, the score a shorter answer makes easiest to earn, and none from answering the question better. Clean answers that skipped part of the question went from 24 to 66 out of 200. The likely cause is the reward mechanism. It adds the two scores together, so a 5 for coherence and a 3 for responding earns the same reward as a 4 and a 4, even though only the 4 and 4 counts as usable. A short, clean half-answer was the safe bet.

For the SFT adapter, no. Both scores went up, and the usable rate rose at every answer length. So the gain there wasn’t just from being shorter.

So where does that leave us? At the start, about 1 in 4 of the baseline’s answers was usable. After SFT on 180 ‘e’-free examples and 300 steps of RL, it’s about 6 in 10. And for the SFT adapter, that gain held up when we checked whether it was just writing less. It still cuts corners sometimes (see “football”), but it’s a lot closer to the second woodchuck answer than the riddiculous one we started with.

Here’s every answer from the eval set at each stage. Hover over or tap a dot to read it:

Takeaways and Next Steps

Throughout this project, I gained a lot more insight into post-training methods for LLMs, implementation details of RL algorithms, and many other things (memory calculations, how to prepare datasets, how to run experiments, etc.).

Some general takeaways I had while working on this were:

• It’s important to have a clear goal defined. Before I defined the goal of the project, the project was just learning and exploring LLM fine-tuning, which felt aimless.
• For projects, it’s better to work in sprints. In the middle, I was balancing school and interview prep while working on the project, which made it really hard to think deeply about the project’s direction every day. Rather than chipping away at the project, it felt much better to define milestones and work until they were reached.
• Visibility (logging runs and saving responses) is really valuable in training runs, especially for RL. The loss is just a value we calculate off of some metrics, and it’s hard to cover all bases. Being able to check and validate that the model’s responses are actually heading in a desired direction is important.

Regarding the overall project, I definitely believe there are improvements to be made, especially in designing more rigorous experiments and finding training algorithms that lead to better results. However, I’m satisfied with what I was able to learn and the results I saw, and would like to spend time learning other things.

For what’s next, there are quite a few things that I want to explore:

• Learn more about the fundamentals of RL
  • i.e., learning the mathematical frameworks behind it, and implementing more common algorithms and understanding the ideas behind them
  • I definitely want to try applying RL to a video game at some point…
• Look at more modern post-training work with models/agents
  • for next projects on RLing an LLM, I think I’ll probably try using pre-existing packages/apis and paying for compute to run larger explorations.

References and Reading

• original blog on painting flowers
• paper on using LLMs to transform text into lipograms
• blog on masking out ‘e’s for Llama-8B

Appendix

Project Timeline

the process of the project was roughly (blog covers bold sections):

• scoping out the project, choosing model and doing preliminary reading
• writing out the forward pass, wiring in optimizations (see the memory section for information)
• straightforward attempts at making the model generate a response without the character ‘e’ (RLing the base model to not use ‘e’ on a specific prompt, trying out the mask)
• defining the goal (what objective do we want to see after fine-tuning? what kind of data do we need?)
• fooling around with implementing “Trainer” classes from TRL on my own (see the “Engine” classes in the github repo)
• sft attempts
• rl for coherence
• cleanup + writeup

GitHub Repo

the code, data and results are all in kevinyvv/qwen-3-1.7b-finetune-proj. some parts of interest may be:

• forward pass
• simple_train.py (the first attempt)
• SFTEngine, RLEngine (REINFORCE), GRPOEngine
• the judge and the ‘e’-free answers used for SFT
• results and findings: run records and the full experiment notes

Model Choice

I chose Qwen3-1.7B since I could fine-tune it locally with 16 GB of VRAM. Its architecture was also simpler than those of newer models, so it was easier to understand and implement the forward pass.

The only pre-planning I really did otherwise was learning how to choose a model that I could work on locally. The details are collapsed by default, but feel free to read further if you’re curious about some of the mem calculations.

Sourcing/Building the Dataset

datasets used: No Robots, Databricks-Dolly

I removed code-writing requests, empty examples, duplicate prompts, prompts over 150 words, and examples whose human answers exceeded 300 words. I also dropped No Robots’ persona-based chat examples, since instructions like “answer in rhymes” would add another constraint.

After filtering and making my own splits, I had (please note that most prompts in the dataset ended up being unused):

One additional problem I discovered: some prompts just can’t be answered without an ‘e’. “What’s the capital of Belgium?” needs “Brussels”. “Who painted the Mona Lisa?” needs “Leonardo”. I filtered these out for all prompts used in training and testing.

Instead of using the splits whole, I sampled from them whenever an experiment needed data. The eval set is 50 prompts from the validation split. The RL pool started as 2,000 prompts from the training split, and subagents filtered it down to 1,235.

Keeping the Judge Fair

A couple of rules keep the judge fair:

• The judge isn’t told about the ‘e’ rule, so it can’t go easy on a weird answer just because the task is hard.
• Wrong facts don’t count against a response, as long as it attempts the prompt. A 1.7B model gets plenty of facts wrong anyway, and I only care about the writing here.

I also checked that the judge wasn’t too strict. I ran Qwen on the eval set without the ‘e’ rule and judged only coherence and responding. Normal answers should pass most of the time, so I aimed for over 80%, and Qwen scored 80.5%. Along the way, I tweaked the judge’s prompt to catch common issues (repeating lines, answers being cut off by the token limit). The judge code is linked in the GitHub Repo section.

RLing for no ‘e’

The first step was making the model output a calculated response with no ‘e’ tokens (as opposed to by luck). After implementing the forward pass and adding LoRA, I froze the original weights and trained only the adapters. I used eight copies of the same prompt, with thinking disabled and a maximum of 64 generated tokens:

The idea was to overfit this one example before worrying about a dataset or generalization. Generate a response, score it, and use that score to update the model. How hard could avoiding one letter be?

I started by giving a response one point for every word without ‘e’. That still gave positive rewards to answers that broke the rule, as long as they contained some valid words. I tried centering rewards across the batch to favor better responses, but the samples were often very similar. I also wanted a more specific signal about which parts of a response were causing problems.

So I moved to a reward lookup table for individual tokens:

for tok, idx in tok_to_id.items():
    if 'e' in tok.lower() and tok != eos_token:
        e_table[idx] = -1.
    else:
        e_table[idx] = 0.5

Tokens aren’t necessarily whole words, though. With a split like gr + een, penalizing only the second token misses the earlier choice that helped create the word. I used discounted future rewards so a later penalty could also affect earlier choices. These are shortened excerpts from the return calculation and policy loss:

# Inside discount(), working backwards through the response:
for t in reversed(range(rewards.size(1))):
    running = rewards[:, t] + gamma * running
    returns[:, t] = running

# Back in the training loop:
rewards = get_feedback(gen_ids, e_table)
per_token_reward = discount(rewards)
advantage = (per_token_reward - per_token_reward[mask].mean()) * mask
loss = -(log_probs * advantage).sum()

Subtracting the mean gave each token an advantage relative to the valid token returns in that batch. A positive advantage pushed up the probability of the sampled token; a negative one pushed it down.

This sort of worked. Some responses settled on “i want to know what you want to know”, which follows the rule but isn’t much of an answer. Other samples found an easier way to collect rewards (excerpt):

I was rewarding tokens without ‘e’, and a stream of letters could satisfy that objective without forming words. The reward values changed across retries, but none of these rules directly measured whether the output made sense.

My next idea was to penalize drift from the original model with a KL penalty. I ran the same generated sequence through the model with LoRA disabled, then compared the log probabilities assigned to the sampled tokens. We then added this adjustment to the reward:

kl_approx = (log_probs.detach() - ref_gen_log_probs) * mask
rewards = get_feedback(gen_ids, e_table) - kl_beta * kl_approx

That was supposed to discourage the model from abandoning its original language behavior, but the outputs still fell apart. I later found that the KL penalty was also too small relative to the other rewards. One saved sample was:

I tried smaller bonuses, bonuses scaled by token length, stronger penalties for ‘e’, and changing the discount factor. Some runs produced nonsense; others got stuck repeating the same sentence. A stronger penalty from the start also encouraged single-letter and newline outputs.

From this initial testing, I realized looking at actual samples matters. These runs showed how easily rewards could end up being gamed, even with countermeasures.

1.7B vs 4B

At some point, I was hypothesizing that 1.7B was just too small of a model. I also tried comparing Qwen3-1.7B vs 4B to see if I should’ve just chosen a larger model to work on. On the eval set, I saw:

We see that neither model can follow the rule from the instruction alone. Not a single answer avoided ‘e’, so we need the mask. Furthermore, the 4B model is worse under the mask. It keeps writing the words it planned and just drops the ‘e’s, giving things like “th” for “the” and “amusrmt park.” The 1.7B model more often switches to a different word.

So we stayed on 1.7B for the experiment. This wasn't exactly a rigorous comparison, but 1.7B and 4B were the models I could reasonably run locally. Some other hypotheses to test would be: newer models, models trained with more English content, etc.