Building a Conversational AI to Teach an Underrepresented Language

In this post, I'll be walking you through how I built a language learning AI Tutor, with complete setup instructions provided on Github.

As a first generation Moroccan immigrant, I've spent a lot of years learning arabic, with the simple interest in being able to speak the language when I visit Morocco. Between Saturday classes as a kid and language apps as a teenager, I put a lot of time into learning Arabic. But every time I go back to Morocco, I'm how ineffective it is. Most of the learning doesn't carry over to daily conversations, and I end up learning far more in a short visit than in years of study.

The reason being simple: most language learning resources focus on Modern Standard Arabic, the "general" version. Arabic itself is not one homogeneous language but a macrolanguage made up of many dialects, including Moroccan Arabic, known as Darija. Shaped by Amazigh influence, and to a lesser degree French and Spanish, Darija is often considered the most distinct dialect of Arabic.

Though Modern Standard Arabic is used in Morocco for writing, media, and education, it's not spoken day to day. Darija, on the other hand, is almost entirely spoken without a standardized set of grammar rules, which makes it even harder to find resources for learners. So, I wanted to build an app tailored to practicing conversational Darija.

This project was a new type of challenge for me, as my prior expertise has been in full-stack and systems-level software engineering, but I found exploring speech and language processing to be really interesting.

Implementing Automatic Speech Recognition: OpenAI Whisper

When building an AI tutor, one of the most important components is the ability to understand what the student is saying. This is where Automatic Speech Recognition (ASR) comes in, which is what converts spoken language into written text.

For decades, ASR systems worked in narrow conditions like phone menus or dictation software but quickly failed with accents, background noise, or casual speech. That limitation made making an AI tutor like this (which needs to handle natural, messy input from students) previously impossible. A recent turning point in speech technology was November 2022, when OpenAI released Whisper.

Whisper was trained on 680,000 hours of multilingual labeled audio, a scale far beyond traditional datasets. And while that increase in performance is impressive, the revolutionary part is actually the democratization of high-quality speech recognition through its open-source release. This has led to a wave of community fine-tunes and dataset uploads, even for languages that were historically overlooked. On Hugging Face, for instance, you'll now find Whisper models trained specifically on Moroccan Darija (I'll be using / going into more detail about this later), something that wasn't realistically feasible just a few years ago

At its core, Whisper is built on a sequence-to-sequence (Seq2Seq) model. Seq2Seq models have two main components: an encoder that processes the input signal and a decoder that generates the output sequence, connected through an attention mechanism.

• The encoder takes in raw audio and converts it into features such as spectrograms.
• The attention mechanism highlights the most relevant parts of that encoded audio when predicting the next token.
• The decoder then produces text tokens one by one, guided by both the audio features and a language model that enforces fluency and word order.

Implementing Whisper: Prototypes and Fixes

Once I understood how Whisper worked conceptually, I wired the architecture into a real-time loop. The first prototype was a PySide6 GUI that captured mic input at 44.1 kHz, resampled it to 16 kHz, and pushed the audio into an 8-second rolling buffer. Every 0.5 seconds, the buffer was converted into log-mel features, passed to Whisper, and decoded into text that updated the transcript live.

with torch.no_grad():
    # extract log-mel spectrogram features from the 8s rolling buffer
    feats = processor.feature_extractor(buf, sampling_rate=16000, return_tensors="pt").input_features.to(device)

    # create an encoder attention mask (all ones since we never pad streaming audio)
    enc_mask = torch.ones((feats.shape[0], feats.shape[-1]), dtype=torch.long, device=feats.device)

    # run whisper's decoder with deterministic settings for stable output
    ids = model.generate(
        input_features=feats,
        attention_mask=enc_mask,
        do_sample=False,
        temperature=0.0,
        num_beams=1
    )

    # decode token ids back into text
    text = processor.batch_decode(ids, skip_special_tokens=True)[0].strip()

The issue with this was that tiny windows produced junk like a single word repeated, and longer windows added latency. The fix was to switch the interaction from real-time to push-to-talk, and decode once per utterance.

Because Whisper is a seq2seq model, giving the encoder a clean, contiguous chunk and the decoder a single pass removes the instability you get from sliding contexts. In the new loop you hold a mic button, the app buffers raw audio, and on release it resamples to 16 kHz, builds log-mel features, and runs a single deterministic generate(…) call.

Below is the core decoding logic. While you hold the button, it collects and resamples incoming audio blocks. When released, it extracts features from the full segment, sets up generation parameters, and seeds the decoder with the chosen language and task. It then generates output tokens and decodes them to text. The main branching controls whether to force a specific language or let the model auto-detect.

gen_kwargs = {
    "max_new_tokens": 128,      # set max tokens to generate
    "do_sample": False,         # deterministic output
    "temperature": 0.0,         # no randomness
    "return_dict_in_generate": False
}

# set language and task for decoding
lang_code = (forced_lang or getattr(window, "lang_hint", None))
task = getattr(window, "task_mode", None) or "transcribe"

# set forced decoder ids if language is specified
if lang_code and str(lang_code).lower() not in {"auto", "none"}:
    try:
        forced = processor.get_decoder_prompt_ids(language=lang_code, task=task)
        gen_kwargs["forced_decoder_ids"] = forced
    except Exception:
        # clear forced ids if forcing fails
        try:
            if getattr(model.generation_config, "forced_decoder_ids", None) is not None:
                model.generation_config.forced_decoder_ids = None
        except Exception:
            pass
else:
    # ensure no forced ids in auto mode
    try:
        if getattr(model.generation_config, "forced_decoder_ids", None) is not None:
            model.generation_config.forced_decoder_ids = None
    except Exception:
        pass

# generate output token ids
with torch.no_grad():
    ids = model.generate(input_features=feats, **gen_kwargs)

# update progress bar
if emit_progress:
    try: emit_progress(90)
    except Exception: pass

# decode output ids to text
text = processor.batch_decode(ids, skip_special_tokens=True)[0].strip()

After settling the decoding loop, I cleaned up the model side so it behaves the same across checkpoints. The model picker now shows "Standard" and "Advanced," which map to the multilingual openai/whisper-small and openai/whisper-medium repositories. The default whisper model is multilingual, including arabic, but that dataset is largely built on MSA. So, on top of these, I integrated community fine-tunes for Moroccan Darija from Hugging Face: ychafiqui/whisper-small-darija, a 242M-parameter checkpoint fine-tuned on a Darija speech-to-text dataset, and ychafiqui/whisper-medium-darija, a larger 764M-parameter variant.

User Interface

The UI has two buttons: English (green) and Darija (red). Pressing one sets a language hint for that utterance. If you press the English mic, the decoder is primed for English; if you press the Darija mic, it's primed for Darija. If nothing is forced, the model auto-detects and any stale forced_decoder_ids are cleared at load time.

This serves two purposes: first, it prevents Whisper from detecting the wrong language, and second, it lets the learner choose how they want to interact. Regardless of what language you choose, the tutor's focus will be on learning darija, but sometimes the learner may want to ask a question in English (i.e. about a translation) and other times hold a conversation in Darija.

You may have noticed that "Arabizi" checkbox. One issue was that the Whisper Darija models returned transcripts in Arabic script. A huge portion of Darija speakers and learners don't actually read or type that way. Online, people overwhelmingly use Arabizi: Darija written in the Latin alphabet with numbers like 3 for ع and 7 for ح. It became the norm with early phones and QWERTY keyboards. Since both forms are common, the app includes a simple checkbox to toggle between Arabic script and Arabizi. That way, the transcripts can be displayed in whichever form feels more natural to read and follow along with.

The code below converts Arabic script into Arabizi by first stripping diacritics and tatweel (ـ), then handling special digraphs like "لا" before falling back to a character-by-character lookup. This way, ح becomes 7, ع becomes 3, and ق becomes 9, letting the same transcript be instantly toggled between Arabic script and Arabizi without having to change the model itself.

# utils/arabizi.py
import re

# strip diacritics and tatweel
_DIAC = re.compile(r"[\u064B-\u065F\u0670\u06D6-\u06ED]")
_TATWEEL = "\u0640"

# mapping: Arabic characters -> Arabizi equivalents
_MAP = {
    "ا": "a", "آ": "2a", "إ": "2i", "أ": "a", "ى": "a",
    "ء": "2", "ؤ": "2u", "ئ": "2i", "ٔ": "2",
    "ب": "b", "ت": "t", "ث": "t", "ج": "j", "ح": "7", "خ": "kh",
    "د": "d", "ذ": "d", "ر": "r", "ز": "z", "س": "s", "ش": "ch",
    "ص": "s", "ض": "d", "ط": "t", "ظ": "d",
    "ع": "3", "غ": "gh", "ف": "f", "ق": "9", "ك": "k", "ݣ": "g", "ڭ": "g",
    "ل": "l", "م": "m", "ن": "n", "ه": "h", "ة": "a",
    "و": "u", "ي": "i",
    "لا": "la",
}

# order matters: handle lam-alef first
_MULTI = [("لا", "la")]

def arabic_to_arabizi(text: str) -> str:
    if not text:
        return ""
    # normalize
    s = _DIAC.sub("", text.replace(_TATWEEL, ""))
    for a, b in _MULTI:
        s = s.replace(a, b)
    out = []
    for ch in s:
        out.append(_MAP.get(ch, ch))
    res = "".join(out)
    return re.sub(r"\s+", " ", res).strip()

Evaluating Accuracy and Latency

To validate that the tutor was more than just working in the demo, I ran structured tests on both transcription quality and real-time performance.

I curated a conversational test set of 2500 Darija audio clips between 10–20 seconds each, recorded in both quiet and noisy conditions. Each clip was paired with a manually written reference transcript.

Example row from the dataset:
audio_path; reference_text
clips/clip12.wav; "واش غادي معايا للسوق دابا"

For transcription accuracy, I used the jiwer package to compute Word Error Rate (WER) and Character Error Rate (CER):

# predictions and references are both lists of strings
wer_score = wer(references, predictions)
cer_score = cer(references, predictions)

print(f"WER: {1-wer_score:.2%}, CER: {1-cer_score:.2%}")

Latency was measured as the time from button release to the finalized text (turn-taking latency):

import time

# measure from button release -> finalized text (turn-taking latency)
t_release = time.monotonic() # set at button release
ids = model.generate(input_features=feats, **gen_kwargs)
t_done = time.monotonic()

latency_ms = (t_done - t_release) * 1000
print(f"Turn-taking latency: {latency_ms:.1f} ms")

This instrumentation was run continuously during push-to-talk sessions, producing a distribution of decode times.

Darija ASR Benchmark Results

| Metric                   | Value    |
|--------------------------|----------|
| Word Accuracy (WER)      | 92%      |
| Character Accuracy (CER) | 88%      |
| Avg. Latency             | 178 ms   |
| p50 Latency              | 165 ms   |
| p90 Latency              | 210 ms   |

These numbers showed that the tutor could reliably handle Darija speech in real time. Transcripts were accurate enough for learning, and latency stayed below the 200 ms threshold.

LLM Tutor Logic

The next step is to implement an AI interpretation of the ASR input, and provide an LLM generated response. We'll be using a remote language model (OpenAI GPT or a custom API).

It'll construct prompts and send them via HTTP requests, letting the hosted model handle inference and response generation. All model training and architecture are managed by the provider, so our code will focus on prompt engineering and API integration.

In Natural Language Processing (NLP) there are different kinds of language models. Masked language models, like BERT, predict missing words by looking at both the left and right context of a sentence. Causal language models, like GPT, only look left-to-right, predicting the next word one step at a time. For my tutor, a causal language model fit better than a masked or bidirectional model, because it keeps the flow of speech intact and generates fast without overthinking structure.

The code in llm/tutor_client.py owns the LLM call and output shaping. I'm using an env-selectable OpenAI chat model (defaults to gpt-4o-mini) and a strict prompt that keeps replies short and Darija-specific. The helper exposes a single entry point:

def ask_llm(
    transcript: str,
    lang_hint: Optional[str],             # 'en' or 'ar'
    mode: str = "normal",                 # "normal", "translate_en_to_ar", "translate_ar_to_en"
    output_script: Optional[str] = None,  # "arabizi", "arabic" when lang_hint =="ar"
    topics: Optional[Tuple[str, ...]] = None,
) -> str:

Behavior is controlled by mode (chat in Darija/English, EN→Darija, Darija→EN) and by output_script, which forces Arabizi or Arabic script for Darija replies. The function builds a tight system prompt (one short sentence, Darija tutor voice, stable Arabizi digraphs), injects the last few user topics, calls the OpenAI Chat Completions API with light retries, and finally enforces script (if Arabizi is requested but any Arabic letters slip through, they're transliterated before display).

Topic Extraction (deterministic, per turn)

Topic extraction happens on every turn to keep the tutor's examples personal without storing user profiles. In our code this is deliberately cheap and offline: extract_topics in llm/topics.py skips any LLM call and runs a rule-based pass only. Messages under ten characters leave the list unchanged; longer messages feed _rule_topics(message) to pull out two to eight compact tags. The function de-duplicates against the rolling list, trims to the most recent thirty so memory never grows unbounded, and always returns a plain Python list (safe for GUI state). These tags are then threaded into the next ask_llm call as a short "prefer examples about …" hint, which is enough to bias examples toward food, travel, shopping, pronunciation, verb tenses, or whatever the user has been discussing.

def extract_topics(message: str, current_topics: Sequence[str] | None) -> List[str]:
    base = list(current_topics or [])
    if not message or len(message.strip()) < 10:
        return base

    # rule-based (fast, offline)
    new = _rule_topics(message)

    for t in new:
        if t and t not in base:
            base.append(t)

    # trim to last 30
    if len(base) > 30:
        base = base[-30:]
    return base

Routing Before Generation

The next step was adding a routing layer before the LLM. Even though the model can already handle open-ended dialogue, it benefits from structured preprocessing. The goal is to give the generative model context it can't always infer reliably on its own - things like fixing ASR mistakes or detecting translation requests.

This design is something between a traditional NLU system and a modern generative one. Older chatbots relied on intent classifiers to map input to a fixed response. That idea is used here but in the form of routing logic that narrows the model's task. Instead of predicting an intent label, it determines how to frame the request (whether to chat normally, translate, or use a local skill). This makes the LLM more focused, which is especially useful in a bilingual tutor where users mix English and Darija naturally.

The main function, route(text, lang_in, want_script, topics), standardizes the input before deciding what to do with it. It cleans whitespace, lowers the text, and converts the topic list into a tuple to ensure compatibility with caching systems that require hashable objects (ran into some issues with this at one point).

def route(
    text: str,
    lang_in: str,
    want_script: str,                      # "arabizi", "arabic"
    topics: Optional[Sequence[str]] = (),  # may be list or tuple from caller
) -> str:
    """Normalize ASR text and prepare parameters for deterministic routing."""
    s_raw = text or ""
    s = _normalize_mishears(s_raw).strip()
    s_low = s.lower()
    lang_in = (lang_in or "en").lower()
    want_script = (want_script or "arabizi").lower()
    topics_tuple: Tuple[str, ...] = tuple(topics or ())

From there, the router applies a lightweight scoring system instead of relying on hardcoded phrases. It loads translation and mode triggers from a small JSON file in data/router_lex.json, so phrases and keywords can be updated without touching the code. Regex patterns capture language variations like "how to say," "translate this," or Arabizi forms of "tarjama" (translation). Each trigger gets a confidence score, and small priors (e.g. user's current language or use of Arabizi numerals) help guide the final decision.

def _decide_mode(text_low: str, lang_in: str) -> Tuple[str, str]:
    force_dar = _score("force_darija", text_low)
    force_en  = _score("force_english", text_low)
    en2ar     = _score("translate_en_to_ar", text_low)
    ar2en     = _score("translate_ar_to_en", text_low)

    if force_dar >= 3: return "normal", "ar"
    if force_en  >= 3: return "normal", "en"

    if lang_in == "ar": ar2en += 1
    else:               en2ar += 1

    if any(w in text_low for w in ARABIZI_CUES): en2ar += 1

    if max(en2ar, ar2en) >= 3:
        if en2ar > ar2en: return "translate_en_to_ar", "ar"
        if ar2en > en2ar: return "translate_ar_to_en", "en"
        return (("translate_en_to_ar", "ar") if lang_in == "en"
                else ("translate_ar_to_en", "en"))

    return "normal", lang_in

Once the mode is chosen, the router prepares the rest of the request. If the user is speaking Darija, it sets the output script (Arabic or Arabizi) based on the learner's preference or the text itself. Then it sends the cleaned text, mode, language, and any topic context to ask_llm, which builds the system prompt and generates the response. The router itself doesn't write replies, it just makes sure the model receives the right context for what the user meant to do.

Beyond that, the same routing layer also connects with quick Darija-first features that don't rely on the LLM. Commands like "tasrif" or "conjugate" can trigger skills/conjugate.py to return a verb table, while "شرح / break down" calls skills/breakdown.py for a word-by-word explanation.

Fine-Tuning

Now for, arguably, the most important part. So far, the tutor is able to communicate seemingly effectively. But the more you interact with it, the more you notice subtle mistakes as it slides between Darija, Modern Standard Arabic, and other dialects.

OpenAI's GPT models are trained by reading enormous amounts of text from the internet, books, code, and licensed datasets. They learn by seeing how words are actually used across billions of real examples. If a language appears often in that data, the model becomes good at it. If a language appears rarely, the model never truly learns its patterns and instead starts guessing.

The public web, which is one of the largest sources of training data for modern language models, is extremely unequal by language. In the latest Common Crawl snapshot, which analyzes billions of web pages, English alone makes up about 41 percent of all text. Chinese, Russian, German, Japanese, Spanish, and French each account for several more percent. Arabic, by contrast, is only about 0.67 percent of the entire web.

Addressing Language Flattening

The vast majority of that 0.67% is Modern Standard Arabic or other dialects of Arabic. So when you ask it to produce Darija, it does not retrieve a learned dialect. It is approximating one by blending Modern Standard Arabic with pieces of other dialects and French.

This leads to a broader issue known as "language flattening". When a model is trained on large mixed datasets, it tends to smooth out differences between related language varieties. Even in English, which has a lot of data, this shows up as American, British, and other styles being merged into something generic. In Arabic, where dialects differ much more in grammar and vocabulary and where training data is far more limited, this effect is much stronger. The result is an averaged form of Arabic that does not match how people actually speak.

The fix to this is fine tuning. Instead of relying on the small and inconsistent amount of Darija that appears in its original training data, the model is retrained on a much smaller but highly targeted dataset made up of real Moroccan speech.

This includes everyday phrases, conversations, and corrected outputs from the tutor itself. Training on this kind of data gives the model repeated exposure to how Darija is actually used, which reduces its tendency to fall back on Modern Standard Arabic or other dialects when it is unsure.

Supervised Fine Tuning

To be more specific, this project uses supervised fine tuning. That means the model is trained on labeled input output pairs, where each user message is paired with a correct Darija response. These pairs act as explicit examples of what the model should do. When the model generates a reply, it is compared against the labeled target, and its internal parameters are adjusted so that future outputs move closer to the correct Darija form.

The model is not being taught language from scratch. What supervised fine tuning does is guide its choices. When it has multiple possible ways to answer, the labeled data pushes it toward Moroccan Darija instead of Modern Standard Arabic or a blended dialect.

In this setup, the input is the user message, the label is the correct Darija reply, and the model learns the mapping between the two. This is the same learning mechanism used in tasks like spam detection or sentiment analysis, but here the objective is not a category. It is a specific way of speaking.

Task Specific Darija Dataset

The dataset used here comes from Hugging Face and already contains aligned English messages and Moroccan Darija replies. Each example represents a real instruction and a correct Darija response. During preparation, the English side is used as the prompt and the Darija side becomes the target the model is trained to imitate. These pairs form the supervision signal.

This code snippet in ft/prepare_dataset.py shows how each dataset example is split into an English user message and a Darija assistant reply:

en_msgs = ex.get(DATASET_COL_MESSAGES_EN) or []
dar_msgs = ex.get(DATASET_COL_MESSAGES_DAR) or []

user_en = _pick_turn(en_msgs, "user")
asst_dar = _pick_turn(dar_msgs, "assistant")

Those two pieces are then wrapped into a conversation so the model sees them in a realistic dialogue form. That conversation structure is created inside the dataset preparation step.

This code in the same file (ft/prepare_dataset.py) shows how the conversation is constructed before being flattened into training text:

def _format_chat(tokenizer, system: str, user: str, assistant: str) -> str:
    # Use model-native chat template when available
    msgs = [
        {"role": "system", "content": system},
        {"role": "user", "content": user},
        {"role": "assistant", "content": assistant},
    ]
    try:
        return tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)
    except Exception:
        # Fallback: simple plain format
        return f"System: {system}\nUser: {user}\nAssistant: {assistant}\n"

The result is passed through the base model's chat template so the model is trained in exactly the same format it will later use at inference time. Then the chat template is applied in the "return tokenizer..." line.

Supervised Training with LoRA

Training every parameter of a 7 billion parameter model is extremely expensive. Low-rank adaption (LoRA) avoids this by freezing the base model and learning only a small number of additional parameters that steer its behavior.

Conceptually, instead of fully changing a weight matrix, LoRA learns a small update that is added on top of it:

W' = W + A · B

Only these low rank matrices A and B are trained. The base model stays intact, while the LoRA layers absorb the Darija specific behavior.

The LoRA layers are attached to the base model before training begins. The model is loaded in 4 bit mode to keep memory usage low, then LoRA is injected into key attention and feed forward layers.

This code snippet in ft/train_sft_lora.py shows how the model is loaded, modified with LoRA, and passed into the supervised trainer:

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    load_in_4bit=True,
    device_map="auto",
)

lora_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=LORA_TARGET_MODULES,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_cfg)

trainer = SFTTrainer(
    model=model,
    train_dataset=ds,
    dataset_text_field="text",
)

trainer.train()

Through this process, the model learns to map English prompts to Darija replies in a fully supervised way.

Merging and Deployment

Once training finishes, the learned Darija behavior exists inside the LoRA adapter. That adapter can be merged into the base model to produce a single standalone Darija tuned model that is easy to deploy.

This code snippet in ft/merge_lora.py shows how the LoRA weights are merged into the base model:

base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="cpu",
)

m = PeftModel.from_pretrained(base, adapter_dir)
m = m.merge_and_unload()

m.save_pretrained(merged_dir)
tok.save_pretrained(merged_dir)

The merged model is then loaded by a small FastAPI server and exposed through a /reply endpoint. The tutor UI sends its prompts to this endpoint, and the Darija tuned model returns responses.

After fine tuning, the difference was immediately noticeable in real use. In informal testing, the tutor stopped drifting into Modern Standard Arabic and began responding in consistent, natural Darija. While this was not a formal evaluation, extended conversations showed far fewer dialect switches and much more stable use of Darija vocabulary and grammar, which made the tutor feel more reliable as a learning tool.

Conclusion

Building this Darija AI tutor showed me how much of language learning is shaped by the data behind the tools we use. Modern AI systems are powerful, but they are also deeply influenced by which languages and dialects appear in their training data.

By combining open source speech recognition, careful system design, and targeted fine tuning, it was possible to take a general purpose model and turn it into something that actually supports real Moroccan speech.

My hope is that this project, while nothing revolutionary, helps show how underrepresented languages can be better supported through AI and modern tooling.