Building an Enterprise Legal AI Platform with RAG

The Problem

A legal-tech company working with French-language case law needed to process and search 50,000+ legal documents intelligently. Traditional keyword search was failing — lawyers needed semantic understanding, not just keyword matches.

Architecture I Designed

The system needed three core capabilities: intelligent document ingestion, semantic search, and AI-powered answer generation with source citations.

Document Pipeline

Every legal document goes through a multi-stage pipeline. First, text extraction handles PDFs and Word documents. Then a chunking strategy splits documents into 512-token chunks with 50-token overlap to preserve context across boundaries. Finally, each chunk is embedded with OpenAI embedding models for semantic search, while Claude handles the reasoning-heavy work: anonymization, answer generation with citations, and drafting.

Triple-Index Elasticsearch

I built three separate indices for different search patterns. The main semantic index handles natural language queries. A metadata index enables filtering by court, date, and jurisdiction. An appeals index tracks decision chains so lawyers can traverse the full history of a case.

Claude-Powered PII Anonymization

French legal documents contain sensitive personal data. I built an anonymization module using Claude that achieves 98% PII detection accuracy — identifying names, addresses, ID numbers, and financial data before documents are indexed.

Key Technical Decisions

Why Pinecone over pure Elasticsearch for vectors? Pinecone handles billion-scale vector search with sub-second latency. For 50K documents with 512-token chunks, we generate roughly 200K vectors. Pinecone handles this effortlessly.

Why Claude Sonnet specifically? Legal French is complex. Claude Sonnet outperformed GPT-4 on French legal comprehension benchmarks in my testing, particularly for OHADA compliance language.

Hybrid search over pure semantic. Combining BM25 keyword search with vector similarity gave 15% better retrieval accuracy than either alone.

Results

• 50,000+ documents processed and indexed
• Sub 2-second response times end to end
• 95%+ confidence scores on retrieval
• 98% PII detection accuracy
• 80% reduction in document creation time via AI generation module

Lessons Learned

Chunk size matters more than model choice. I tested 256, 512, and 1024 token chunks — 512 with overlap was optimal for legal documents which have dense, structured paragraphs.

Always build an eval set before optimizing. I created 200 legal Q&A pairs with a lawyer and ran every architecture change against it. This prevented several regressions.

If you are planning a similar system, the full Data Greffe Justice case study covers the architecture in more detail, and my enterprise RAG development page explains how I approach new engagements.