Adding Language Support to SqueakyCleanText¶

This guide covers how to add new language support to SqueakyCleanText, both as a user (via config) and as a maintainer (adding built-in support).

Overview¶

SqueakyCleanText has 4 language-dependent subsystems:

Subsystem	What it does	Default languages
Lingua detection	Auto-detects input language	All 75 Lingua languages (`detect_all_languages=True`; `False` = EN, NL, DE, ES + extras)
Stopwords	Removes common words for statistical model	EN, NL, DE, ES
Date patterns	Regex + fuzzy matching for month names	EN, NL, DE, ES, FR, PT, IT
NER models	Named entity recognition (ONNX models)	EN, NL, DE, ES + Multilingual

Each subsystem can be extended independently. A language doesn't need support in all 4 to be useful.

v0.6.0 Notes¶

PII mode (ner_mode='pii') applies to all supported languages — PII labels are language-agnostic
Replacement modes (replacement_mode='placeholder'|'synthetic'|'reversible') work for all languages
ModernBERT is available as an alternative English-only backend with 8192-token context (vs 512 for XLM-RoBERTa). Set via ner_models={**DEFAULT_NER_MODELS, 'ENGLISH': 'rhnfzl/modernbert-base-ner-conll03-english-onnx'} once exported (the dict must name ENGLISH and MULTILINGUAL)
GLiClass document classification is language-independent (zero-shot, works with any input language)

How to Specify Languages¶

All language parameters accept any of these formats (case-insensitive):

Format	Example	Resolves to
Lingua name	`'ENGLISH'`, `'english'`	`'ENGLISH'`
ISO 639-1	`'en'`, `'EN'`	`'ENGLISH'`
ISO 639-3	`'eng'`, `'ENG'`	`'ENGLISH'`

This applies to: language, extra_languages, custom_stopwords keys, and custom_month_names keys.

Single language vs. tuple of languages¶

from sct import TextCleanerConfig

# Pin to one language (skip auto-detection)
cfg = TextCleanerConfig(language='en')             # → detects nothing, always ENGLISH

# Restrict detection to listed languages (auto-detect among them)
cfg = TextCleanerConfig(language=('en', 'nl'))     # → detect per text, restricted to EN/NL

# Full auto-detection (default — all supported languages)
cfg = TextCleanerConfig(language=None)             # → detect among all 75 Lingua languages
                                                   #   (detect_all_languages=False: EN, NL, DE, ES + extras)

Option A: User-Level Extension (via TextCleanerConfig)¶

Users can add language support without modifying source code:

from sct.config import TextCleanerConfig
from sct.sct import TextCleaner

cfg = TextCleanerConfig(
    # 1. Extend Lingua detection to include Polish (ISO codes also work: 'pl')
    extra_languages=('POLISH',),

    # 2. Custom stopwords (optional — auto-detected from stop-words package if available)
    custom_stopwords={
        'POLISH': frozenset({'i', 'w', 'na', 'z', 'do', 'nie', 'to', 'jest'}),
    },

    # 3. Custom month names for date detection (optional)
    custom_month_names={
        'POLISH': (
            'styczen', 'luty', 'marzec', 'kwiecien', 'maj', 'czerwiec',
            'lipiec', 'sierpien', 'wrzesien', 'pazdziernik', 'listopad', 'grudzien',
        ),
    },

    # 4. Custom NER model (optional — falls back to MULTILINGUAL if not provided)
    ner_models={
        'ENGLISH': 'rhnfzl/xlm-roberta-large-conll03-english-onnx',
        'MULTILINGUAL': 'rhnfzl/wikineural-multilingual-ner-onnx',
        # 'POLISH': 'your-org/polish-ner-onnx',  # placeholder: your own ONNX NER repo; without it POLISH runs MULTILINGUAL
    },
)

cleaner = TextCleaner(cfg=cfg)
result = cleaner.process("Spotkanie z Janem Kowalskim 15 styczen 2025 w Warszawie")

What each field does¶

extra_languages: Tuple of language identifiers (e.g. 'POLISH', 'pl', 'pol'). Accepts Lingua names, ISO 639-1, or ISO 639-3 codes. They join supported_languages, which sets the stopword lists loaded up front and the languages legacy warmup() loads. Unified processing rejects warmup() because providers initialize per worker on first use. With the default detect_all_languages=True, Lingua already considers all 75 languages, so the detector does not change. Only with detect_all_languages=False does extra_languages add these languages to the detector.

custom_stopwords: Dict mapping language name to frozenset of stopwords. If provided for a language, replaces (not merges with) any auto-detected stopwords. If not provided, the stop-words package is queried using the language's ISO 639-1 code.

custom_month_names: Dict mapping language name to tuple of 12 month names (full form, in order Jan-Dec). These are compiled into the date regex at config time, enabling date detection for that language.

ner_models: Dict mapping language name to HuggingFace ONNX model repo ID. ENGLISH and MULTILINGUAL are always required. Languages without a dedicated NER model fall back to the MULTILINGUAL model.

Language discovery¶

The supported_languages field is automatically computed as the union of: - Default 4: {ENGLISH, DUTCH, GERMAN, SPANISH} - extra_languages keys - ner_models keys (minus MULTILINGUAL) - custom_stopwords keys - custom_month_names keys

You don't need to add a language to extra_languages if it's already declared in any other field.

Minimal examples¶

Just detection (no stopwords/dates/NER for the new language):

from sct import TextCleanerConfig

cfg = TextCleanerConfig(
    extra_languages=('FRENCH',),
    check_ner_process=False,
)

Detection + stopwords (auto-loaded from stop-words package):

from sct import TextCleanerConfig

cfg = TextCleanerConfig(
    extra_languages=('FRENCH',),  # stop-words has French built-in
    check_ner_process=False,
)

Pin language (skip auto-detection — ISO codes accepted):

from sct import TextCleanerConfig

cfg = TextCleanerConfig(
    language='fr',                # ISO 639-1 → resolves to 'FRENCH'; any Lingua language works
    check_ner_process=False,
)

Option B: Adding Built-In Language Support (Maintainer Guide)¶

There are two tiers of built-in support:

Tier	What it means	Steps required
Opt-in	Detected by default (all 75 Lingua languages); users add it to the `detect_all_languages=False` detector via `extra_languages=('FRENCH',)`	Steps 1, 3 (NER only), 7
Full default	In `DEFAULT_LANGUAGES`, so also in the `detect_all_languages=False` detector	Steps 1, 2, 3, 7

With the default detect_all_languages=True, every Lingua language is detected, so FR/PT/IT and Polish need no setting. The tiers only differ when a user sets detect_all_languages=False: then only supported_languages are detected, and an opt-in language needs extra_languages=('FRENCH',). build_detector() handles any Lingua language name.

To permanently add a new language (e.g. French) to the package defaults, follow these steps in dependency order.

Step 1: `sct/utils/constants.py` — Add month names¶

Add the language's month names to MONTH_NAMES_MULTILINGUAL:

MONTH_NAMES_MULTILINGUAL = {
    # ... existing entries ...
    # French (accented characters are supported — keys are lowercase canonical forms)
    "janvier": frozenset({"fr"}), "février": frozenset({"fr"}),
    "mars": frozenset({"fr"}),  # French March — NOT shared with English "march"
    "avril": frozenset({"fr"}), "juin": frozenset({"fr"}),
    "juillet": frozenset({"fr"}), "août": frozenset({"fr"}),
    "septembre": frozenset({"fr"}), "octobre": frozenset({"fr"}),
    "novembre": frozenset({"fr", "it"}),  # shared: FR novembre = IT novembre
    "décembre": frozenset({"fr"}),
    # Update existing entries when month names are shared:
    "mai": frozenset({"de", "fr"}),    # German Mai = French mai (both = May)
}

Shared month names — check these before adding new entries: - "agosto" (ES/PT/IT August): add new ISO codes to the existing frozenset - "marzo" (ES/IT March): already tagged {"es", "it"} - "abril" (ES/PT April): already tagged {"es", "pt"} - "mai" (DE/FR May): already tagged {"de", "fr"} - "novembre" (FR/IT November): already tagged {"fr", "it"}

Rules: - Keys must be lowercase (include accented characters as they appear in text) - Values are frozensets of ISO 639-1 codes - If a month name string is identical across languages, add all ISO codes to one entry - Abbreviations (≤ 4 chars) are caught by exact DATE_REGEX; include unambiguous ones only - Full names (> 4 chars) are used by fuzzy matching via FUZZY_MONTH_VOCABULARY_BY_LANG

Update _LANG_TO_VOCAB:

_LANG_TO_VOCAB = {
    "ENGLISH": "en",
    "DUTCH": "nl",
    "GERMAN": "de",
    "SPANISH": "es",
    "FRENCH": "fr",     # ADD
    "PORTUGUESE": "pt", # ADD
    "ITALIAN": "it",    # ADD
}

The module-level DATE_REGEX, FUZZY_MONTH_VOCABULARY, and FUZZY_MONTH_VOCABULARY_BY_LANG are automatically recomputed from MONTH_NAMES_MULTILINGUAL at import time. No further changes to datetime.py needed.

Step 2: `sct/utils/resources.py` — Update default languages (full-default tier only)¶

Skip this step for opt-in languages — build_detector() dynamically handles any language in supported_languages. Only needed to add the language to the detector used with detect_all_languages=False; the default detector already covers all 75 Lingua languages.

Add the language to DEFAULT_LANGUAGES:

DEFAULT_LANGUAGES = frozenset({'ENGLISH', 'DUTCH', 'FRENCH', 'GERMAN', 'SPANISH'})

Step 3: `sct/config.py`, update model routes¶

Add the language to the matching checkpoint record in _ONNX_CHECKPOINTS and _TORCH_CHECKPOINTS. Each record keeps its language keys, model ID, and pinned revision together. The exported DEFAULT_* maps are derived from those records.

A language without a dedicated model belongs to the multilingual checkpoint record:

(('FRENCH', 'PORTUGUESE', 'ITALIAN', 'MULTILINGUAL'),
 'rhnfzl/wikineural-multilingual-ner-onnx', '74191432b21c956b5b04b4778c0e92e569b1f80d'),

LANG_KEYS defines the default model routes. Add opt-in routes to the checkpoint records without changing that default set.

Step 3.5: `scripts/export_onnx_models.py` — Create dedicated ONNX model (optional)¶

FR/PT/IT (v0.5.0+) use wikineural-multilingual-ner-onnx as their NER model. This is the accepted design — per-language ONNX models for these languages are hard to source (few CoNLL/WikiNER-style XLM-RoBERTa models exist for FR/PT/IT on HuggingFace). The multilingual model provides adequate PER/LOC/ORG/MISC coverage across all three languages.

This step is only needed if you specifically want a dedicated ONNX model for a language and have identified a suitable source model. Skip this step entirely if the MULTILINGUAL fallback is acceptable (it is for FR/PT/IT).

1. Find a suitable source model on HuggingFace:

For languages with CoNLL-02/03 coverage (EN, NL, DE, ES), the canonical source models are the FacebookAI/xlm-roberta-large-finetuned-conll0{2,3}-* series. For other languages, search HuggingFace for XLM-RoBERTa-based NER models fine-tuned on a standard NER corpus (WikiNER, Evalita, HAREM, etc.):

# Search HuggingFace for French NER models
# Filter: task=token-classification, architecture=XLM-RoBERTa
# Prefer: high downloads, standard NER labels (PER/LOC/ORG/MISC), CoNLL-style evaluation

2. Add the source → target mapping to MODEL_MAP:

# scripts/export_onnx_models.py
MODEL_MAP = {
    # ... existing entries ...
    # Add when a suitable source model is identified:
    "your-org/source-french-ner-model": "rhnfzl/xlm-roberta-large-ner-french-onnx",
    "your-org/source-portuguese-ner-model": "rhnfzl/xlm-roberta-large-ner-portuguese-onnx",
    "your-org/source-italian-ner-model": "rhnfzl/xlm-roberta-large-ner-italian-onnx",
}

3. Export and upload the model:

# Install export dependencies (separate from runtime deps)
pip install torch transformers "optimum[onnxruntime]" huggingface_hub

# Authenticate to HuggingFace Hub
huggingface-cli login

# Export a single language model
python scripts/export_onnx_models.py --models your-org/source-french-ner-model

# Or export all models in MODEL_MAP (skips already-uploaded ones)
python scripts/export_onnx_models.py

# Preview without uploading
python scripts/export_onnx_models.py --dry-run

The script handles: ONNX export via Optimum CLI, verification of required files, repo creation on HuggingFace Hub, file upload, and cleanup.

4. Update config.py with the new ONNX repo ID:

After upload, move the language from the multilingual record to a dedicated record in _ONNX_CHECKPOINTS. Record the uploaded model ID and its resolved revision together.

Step 4: `sct/utils/stopwords.py` — No changes needed¶

The stop-words package supports 33 languages. If the new language is among them, stopwords are auto-loaded via language_to_iso(). Check coverage:

from stop_words import StopWordError, get_stop_words

print('le' in get_stop_words('fr'))
# -> True
try:
    get_stop_words('th')
except StopWordError:
    print('Thai: not supported, SCT uses no stopwords')
# -> Thai: not supported, SCT uses no stopwords

If the package doesn't support the language, users can provide custom_stopwords, or you can add a built-in set.

Step 5: `sct/utils/datetime.py` — No changes needed¶

Date regex is auto-built from MONTH_NAMES_MULTILINGUAL (Step 1). Fuzzy vocabulary is auto-built from the same data. No code changes needed.

Step 6: `sct/utils/ner.py` — No changes needed¶

NER models are loaded from the ner_models dict. If a default model was added in Step 3, it will be available automatically. Languages without dedicated models fall back to MULTILINGUAL.

Step 7: `tests/test_sct.py` — Add language-specific tests¶

Add date regex tests:

def test_date_regex_french(self):
    dt = datetime.ProcessDateTime()
    cases = [
        ("Reunion le 15 janvier 2024 a Paris", "15 janvier 2024"),
        ("Le 3 mars 2023 debut du projet", "3 mars 2023"),
    ]
    for input_text, date_str in cases:
        result = dt.replace_dates(input_text, "<DATE>")
        self.assertIn("<DATE>", result, f"Failed for French date: {date_str}")

Add NER test (if dedicated model added):

@requires_ner
def test_ner_process_french(self):
    processed = self.ner.ner_process(
        "Emmanuel Macron habite a Paris.",
        positional_tags=['PER', 'LOC'],
        ner_confidence_threshold=0.85,
        language='FRENCH',
    )
    self.assertNotIn("Emmanuel Macron", processed)
    self.assertNotIn("Paris", processed)

Step 8: Version bump¶

Bump version in sct/__init__.py (single source of truth for version).

Creating ONNX NER Models for New Languages¶

Use scripts/export_onnx_models.py to convert a PyTorch NER model from HuggingFace to ONNX format and upload it to the rhnfzl organization. This script is the standard tool for all ONNX model creation in this project.

Prerequisites¶

# Install export dependencies (not in the runtime requirements)
pip install torch transformers "optimum[onnxruntime]" huggingface_hub

# Log in to HuggingFace Hub (requires write access to rhnfzl org)
huggingface-cli login

Workflow¶

Step 1: Identify a suitable source model

For languages with CoNLL-02/03 data, prefer the FacebookAI/xlm-roberta-large-finetuned-* series (consistent architecture with existing EN/NL/DE/ES models). For other languages:

Search HuggingFace for XLM-RoBERTa models fine-tuned on a standard NER corpus
Prefer: high download counts, CoNLL-style labels (PER/LOC/ORG/MISC), public evaluation results
CoNLL-02 covers Spanish, Dutch; CoNLL-03 covers English, German; other languages typically use WikiNER, Evalita (Italian), HAREM (Portuguese), QUAERO (French), etc.

Step 2: Add the mapping to MODEL_MAP

Edit scripts/export_onnx_models.py:

MODEL_MAP = {
    # Existing entries (CoNLL-02/03):
    "FacebookAI/xlm-roberta-large-finetuned-conll03-english": "rhnfzl/xlm-roberta-large-conll03-english-onnx",
    # ... other existing entries ...

    # New language — add your chosen source model:
    "your-org/xlm-roberta-large-french-ner": "rhnfzl/xlm-roberta-large-ner-french-onnx",
}

Target repo IDs follow the pattern rhnfzl/xlm-roberta-large-ner-{language}-onnx.

Step 3: Export and upload

# Export a specific model only
python scripts/export_onnx_models.py --models your-org/xlm-roberta-large-french-ner

# Dry run first (exports but doesn't upload)
python scripts/export_onnx_models.py --models your-org/xlm-roberta-large-french-ner --dry-run

# Keep the export directory for inspection
python scripts/export_onnx_models.py --models your-org/xlm-roberta-large-french-ner --keep-exports

The script exports via optimum.exporters.onnx (token-classification task), verifies required files (model.onnx, config.json, tokenizer.json, tokenizer_config.json), creates the HuggingFace repo if it doesn't exist, uploads all files, and cleans up the local export directory.

Step 4: Update config.py

After upload, move the language from the multilingual checkpoint record to its own _ONNX_CHECKPOINTS record. Use the uploaded model ID and its pinned revision. The DEFAULT_NER_MODELS and DEFAULT_NER_REVISIONS maps then update together.

The ONNX pipeline (sct/utils/onnx_pipeline.py) handles download, caching, and inference automatically. INT8 dynamic quantization is also available at load time via the quantize=True parameter (install SqueakyCleanText[quantize] first), reducing model size ~4× with a small accuracy trade-off.

Language Support Matrix¶

Current built-in support (0.8.0). "Yes (auto)" means detected with the default detect_all_languages=True; the last column is only needed with detect_all_languages=False:

Language	Lingua (default)	Stopwords	Date Regex	Fuzzy Dates	NER Model	Opt-in via
English	Yes (auto)	Yes	Yes	Yes	`rhnfzl/xlm-roberta-large-conll03-english-onnx`	—
Dutch	Yes (auto)	Yes	Yes	Yes	`rhnfzl/xlm-roberta-large-conll02-dutch-onnx`	—
German	Yes (auto)	Yes	Yes	Yes	`rhnfzl/xlm-roberta-large-conll03-german-onnx`	—
Spanish	Yes (auto)	Yes	Yes	Yes	`rhnfzl/xlm-roberta-large-conll02-spanish-onnx`	—
French	Yes (auto)	Yes (stop-words)	Yes	Yes	`rhnfzl/wikineural-multilingual-ner-onnx` (multilingual)	`extra_languages=('FRENCH',)`
Portuguese	Yes (auto)	Yes (stop-words)	Yes	Yes	`rhnfzl/wikineural-multilingual-ner-onnx` (multilingual)	`extra_languages=('PORTUGUESE',)`
Italian	Yes (auto)	Yes (stop-words)	Yes	Yes	`rhnfzl/wikineural-multilingual-ner-onnx` (multilingual)	`extra_languages=('ITALIAN',)`

User-extensible via TextCleanerConfig (no source changes needed):

Feature	How to enable	Auto-detected?
Lingua detection	Default (all 75 languages); with `detect_all_languages=False`, `extra_languages=('POLISH',)`	Yes
Stopwords	Auto from `stop-words` pkg	Yes (33 languages)
Custom stopwords	`custom_stopwords={'POLISH': frozenset({...})}`	No
Date regex	`custom_month_names={'POLISH': (...)}`	No
NER model	`ner_models={**DEFAULT_NER_MODELS, 'POLISH': 'model-id'}`	Falls back to MULTILINGUAL

Languages supported by the `stop-words` package¶

Arabic, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Turkish, Ukrainian, Vietnamese (33, from the package's languages.json).

Checklist for Adding a New Built-In Language¶

Opt-in tier (auto-detected by default; `extra_languages` only for `detect_all_languages=False`)¶

[ ] Add month names to MONTH_NAMES_MULTILINGUAL in constants.py (check for shared names!)
[ ] Add ISO code to _LANG_TO_VOCAB in constants.py
[ ] Add a language key to _ONNX_CHECKPOINTS (dedicated model or multilingual fallback) in config.py
[ ] Add a language key to _TORCH_CHECKPOINTS (dedicated model or multilingual fallback) in config.py
[ ] Create dedicated ONNX model (optional — see Step 3.5; skip if MULTILINGUAL fallback is acceptable):
[ ] Find a suitable source model on HuggingFace (XLM-RoBERTa-based, CoNLL/WikiNER-style labels)
[ ] Add "source-model": "rhnfzl/xlm-roberta-large-ner-{lang}-onnx" to MODEL_MAP in scripts/export_onnx_models.py
[ ] Run python scripts/export_onnx_models.py --models <source-model> to export and upload to rhnfzl Hub
[ ] Move the language to its dedicated _ONNX_CHECKPOINTS record with the new ONNX repo ID and revision
[ ] Add date regex tests in test_sct.py
[ ] (Optional) Add NER tests if dedicated model added
[ ] Verify: ruff check sct/ tests/ clean
[ ] Verify: pytest tests/ -v all pass
[ ] Bump __version__ in sct/__init__.py (pyproject.toml reads it)

Full-default tier (also in the `detect_all_languages=False` detector)¶

All of the above, plus: - [ ] Add to DEFAULT_LANGUAGES in resources.py

Default routing¶

Keep LANG_KEYS unchanged when adding an opt-in route. Changing it expands the default model and language set.