Rubai PII Detectionv1.3

Why This Model Exists

Uzbekistan introduced a new Personal Data Law that requires organizations to protect citizens' personal information. However, there were no tools available to automatically detect PII in Uzbek text before this model.

Personal data leaks happen in log files, training datasets for AI/ML, customer service chat histories, document archives, API responses, and financial transaction records. This model provides automatic PII detection that can be integrated into data pipelines, document processing systems, compliance audit tools, and financial services.

Performance Metrics

Test Suite Results (96 tests)

Supported Entity Types

The model now detects 7 entity types including the new CARD_NUMBER type:

Live Examples

Quick Start

Installation

pip install transformers torch

Basic Usage

from transformers import pipeline

# Load the model
ner = pipeline(
    "ner",
    model="islomov/rubai-PII-detection-v1-latin",
    aggregation_strategy="simple"
)

# Detect PII (including card numbers!)
text = "Sardor Rustamov telefon 90 123 45 67, karta 8600 1234 5678 9012"
results = ner(text)

for entity in results:
    if entity['entity_group'] != 'TEXT':
        print(f"{entity['entity_group']}: {entity['word']}")

Check for Card Numbers

def has_card_number(text: str) -> bool:
    """Check if text contains any card numbers."""
    ner = pipeline(
        "ner",
        model="islomov/rubai-PII-detection-v1-latin",
        aggregation_strategy="simple"
    )
    results = ner(text)
    return any(r["entity_group"] == "CARD_NUMBER" for r in results)

# Examples
print(has_card_number("Karta: 8600 1234 5678 9012"))  # True
print(has_card_number("Telefon: 90 123 45 67"))       # False

Changelog

Model Architecture

Training Data

The model was trained on 475,185 samples - the largest Uzbek PII dataset to date, covering 100+ domains including banking, healthcare, education, e-commerce, government services, and financial transactions.

How to Train Such a Model

Training a PII detection model for a low-resource language like Uzbek requires careful attention to data preparation and augmentation. Here's a guide based on this model's development:

Step 1: Start with a Strong Base Model

Use a BERT model pre-trained on your target language. For Uzbek, tahrirchi/tahrirchi-bert-base provides strong language understanding. This dramatically reduces the amount of labeled data needed.

Step 2: Create a Diverse Dataset

Cover multiple domains (banking, healthcare, education, government, etc.) to ensure the model generalizes well. Include both formal and informal text styles.

Step 3: Data Augmentation

Augment your data with format variations:

• Phone numbers: With/without spaces, dashes, country codes
• Dates: Different formats (15-yanvar, 15.01.2025, etc.)
• Names: Different orderings (First Last, Last First)
• Card numbers: With/without spaces, different card types
• Denormalized text: Numbers written as words for ASR compatibility

Step 4: Training Configuration

# Recommended hyperparameters (v1.3)
{
    "num_epochs": 5,
    "batch_size": 32,
    "gradient_accumulation_steps": 2,  # effective batch 64
    "learning_rate": 2e-5,
    "weight_decay": 0.01,
    "warmup_ratio": 0.1,
    "max_seq_length": 256,
    "eval_split": 0.05,
    "fp16": True
}

Step 5: Label Alignment

When tokenizing, carefully align labels with subword tokens. Use -100 for special tokens and subword continuations to exclude them from loss calculation.

Use Cases

• Data Privacy Compliance - Detect PII before data sharing to comply with Uzbekistan's Personal Data Law
• Financial Services - Detect and mask bank card numbers in transaction logs
• Document Redaction - Automatically mask sensitive information in documents
• Customer Support - Flag conversations containing personal data for special handling
• ML Data Preparation - Clean training datasets by removing or masking PII
• Log Sanitization - Filter PII from application logs before storage
• Audit & Monitoring - Track PII exposure in communications and documents

Limitations

License: CC BY-NC 4.0

This model is released under Creative Commons Attribution-NonCommercial 4.0. This license was chosen deliberately to balance open access with sustainable development.

Why CC BY-NC 4.0?

Citation

@misc{rubai-pii-detection-2026,
  author = {Sardor Islomov},
  title = {Rubai PII Detection v1.3 - Uzbek Personal Information Detector},
  year = {2026},
  publisher = {Hugging Face},
  organization = {Rubai AI},
  url = {https://huggingface.co/islomov/rubai-PII-detection-v1-latin},
  note = {BERT-based NER model for detecting PII in Uzbek text, including bank card numbers}
}