Building a Zoho Mail RAG System with Python, FastAPI, and ChromaDB

By Khaled Md Tuhidul Hossain · 2026-01-30 · AI Engineering

Email search has always been frustrating. You remember getting an important email about a project deadline, but you cannot recall the exact subject line, the sender's name, or when it arrived. Traditional keyword search falls short because it demands you remember the right words. What if you could just ask your inbox a question in plain English — "What are my upcoming interviews this week?" or "Show me emails from the marketing team about the product launch" — and get a precise, contextual answer?

That is exactly what I set out to build. Zoho Mail RAG is a full-stack Retrieval-Augmented Generation system that connects to your Zoho Mail account, syncs your emails, indexes them with vector embeddings, and lets you query your entire mailbox using natural language. I will walk you through the architecture, the key engineering decisions, and the lessons learned.

Why I built this

The motivation behind this project was both practical and educational:

The Problem: I use Zoho Mail for my professional correspondence, and like many people, I accumulate thousands of emails across dozens of folders. Finding specific information buried in old threads — a meeting time, an attachment someone sent months ago, a conversation about a particular topic — was becoming a daily friction point. Zoho's built-in search is decent for exact keyword matches, but it struggles with semantic queries like "what did the client say about the budget revision?"

The Goal: Build an intelligent email assistant that understands the meaning behind your queries instead of matching keywords. I wanted a system that could:

Sync my entire mailbox efficiently, handling all folders in parallel
Understand natural language questions about my emails
Combine semantic and keyword search for the best possible results
Remember conversation context so I could ask follow-up questions
Run locally with Docker for privacy — my emails never leave my machine (except for the LLM API calls)

The Learning Opportunity: This project was also an exercise in building a production-grade RAG pipeline from scratch, one that handles tens of thousands of documents, needs real-time syncing, and must deliver answers in seconds.

Architecture overview

The system follows a three-phase pipeline: Sync, Index, and Query.

┌─────────────────┐     ┌──────────────┐     ┌─────────────┐
│  React Frontend │────▶│  FastAPI      │────▶│   Zoho      │
│  (Chat UI)      │◀────│  Backend      │◀────│   Mail      │
└─────────────────┘     └──────┬───────┘     │   (IMAP)    │
                               │              └─────────────┘
                  ┌────────────┼────────────┐
                  ▼            ▼            ▼
            ┌──────────┐ ┌──────────┐ ┌──────────┐
            │ChromaDB  │ │SQLite    │ │OpenRouter │
            │(Vectors) │ │(State)   │ │(LLM/Emb) │
            └──────────┘ └──────────┘ └──────────┘

Backend: FastAPI (Python) handles IMAP connectivity, email processing, vector storage, and LLM orchestration.

Frontend: React with Tailwind CSS provides a chat interface with real-time sync progress and markdown-rendered responses.

Storage: SQLite stores email metadata and sync state. ChromaDB stores vector embeddings for semantic search. SQLite FTS5 powers keyword search.

AI Services: OpenRouter API provides access to OpenAI's text-embedding-3-small for embeddings and GPT-4o-mini for response generation.

The tech stack

Layer	Technology	Why
Backend Framework	FastAPI	Async-first, fast, excellent for I/O-bound IMAP operations
Email Protocol	IMAP via imapclient	Direct mailbox access, folder-level syncing
Vector Database	ChromaDB	Lightweight, embeddable, cosine similarity out of the box
Keyword Search	SQLite FTS5	BM25 scoring, zero infrastructure overhead
Embeddings	text-embedding-3-small	Cost-effective, strong performance for email-length text
LLM	GPT-4o-mini	Fast, affordable, good at synthesis and summarization
Frontend	React + Tailwind CSS	Responsive chat UI with real-time progress updates
Deployment	Docker Compose	Single command to spin up the entire stack

Phase 1: Email syncing

The sync service connects to Zoho Mail via IMAP/SSL and fetches emails across all folders in parallel. This was one of the trickiest parts to get right.

class SyncService:
    def __init__(self, db_session, vector_store, embedding_service):
        self.db = db_session
        self.vector_store = vector_store
        self.embedding_service = embedding_service
        self.semaphore = asyncio.Semaphore(SYNC_PARALLEL_WORKERS)
        self._sync_state = {}

    async def start_sync(self):
        folders = await self._list_folders()
        tasks = [
            self._sync_folder(folder) for folder in folders
        ]
        await asyncio.gather(*tasks)

    async def _sync_folder(self, folder: str):
        async with self.semaphore:
            uids = await self._fetch_uids(folder)
            for batch in self._batched(uids, SYNC_BATCH_SIZE):
                emails = await self._fetch_batch(folder, batch)
                await self._store_emails(emails)
                await self._index_emails(emails)
                self._update_progress(folder, len(batch))

Key design decisions:

Parallel folder syncing with a configurable semaphore (default 3 workers) prevents overwhelming the IMAP server while maximizing throughput
Batch processing (50 emails per batch) keeps memory usage predictable
Incremental sync — the system tracks which emails have been fetched and only processes new ones on subsequent syncs
SQLite write locking with asyncio locks prevents concurrent write conflicts

The frontend polls the sync status endpoint and displays real-time progress with animated counters per folder — you can watch emails flow in as they are processed.

Phase 2: Indexing with hybrid search

This is where the RAG magic happens. Every email gets indexed in two parallel systems:

Vector index (semantic search)

Each email's content is converted to a 1536-dimensional vector using OpenAI's text-embedding-3-small model via OpenRouter. These embeddings capture the meaning of the email, so a query about "project deadlines" will match emails that talk about "due dates" or "delivery timelines" even without those exact words.

class VectorStore:
    def __init__(self, persist_dir: str):
        self.client = chromadb.PersistentClient(path=persist_dir)
        self.collection = self.client.get_or_create_collection(
            name="emails",
            metadata={"hnsw:space": "cosine"}
        )

    async def add_emails(self, emails: list, embeddings: list):
        self.collection.upsert(
            ids=[str(e.id) for e in emails],
            embeddings=embeddings,
            documents=[e.body_text for e in emails],
            metadatas=[{
                "subject": e.subject,
                "sender": e.sender,
                "date": e.date.isoformat(),
                "folder": e.folder
            } for e in emails]
        )

    async def search(self, query_embedding: list, n_results: int = 20):
        return self.collection.query(
            query_embeddings=[query_embedding],
            n_results=n_results
        )

Full-text index (keyword search)

SQLite FTS5 provides BM25-scored keyword search. This catches exact matches that semantic search might rank lower — like searching for a specific invoice number or a person's name.

class FTSService:
    async def search(self, query: str, limit: int = 20):
        results = await self.db.execute(text("""
            SELECT email_id, rank
            FROM emails_fts
            WHERE emails_fts MATCH :query
            ORDER BY rank
            LIMIT :limit
        """), {"query": query, "limit": limit})
        return results.fetchall()

Reciprocal rank fusion

The real power comes from combining both search systems using Reciprocal Rank Fusion (RRF). Instead of choosing between semantic and keyword results, RRF merges the two ranked lists into a single better ranking:

def reciprocal_rank_fusion(
    vector_results: list,
    keyword_results: list,
    vector_weight: float = 0.6,
    keyword_weight: float = 0.4,
    k: int = 60
) -> list:
    scores = {}

    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + (
            vector_weight / (k + rank + 1)
        )

    for rank, doc_id in enumerate(keyword_results):
        scores[doc_id] = scores.get(doc_id, 0) + (
            keyword_weight / (k + rank + 1)
        )

    return sorted(scores.keys(), key=lambda x: scores[x], reverse=True)

The default weighting is 60% vector / 40% keyword. This means semantic understanding drives most results, but exact keyword matches still surface strongly. In practice, this hybrid approach significantly outperforms either method alone — especially for email search where you might ask "emails from John about the budget" (semantic intent + specific keyword).

Phase 3: Query pipeline

When a user asks a question, the system orchestrates several steps:

async def query(self, question: str, conversation_id: str = None):
    # 1. Check cache — instant return if seen recently
    cached = self.cache.get(question)
    if cached:
        return cached

    # 2. Expand query with related terms
    expanded_terms = self._expand_query(question)

    # 3. Generate query embedding
    query_embedding = await self.embedding_service.embed(question)

    # 4. Hybrid search (vector + FTS5 + RRF)
    results = await self.hybrid_search.search(
        query_embedding=query_embedding,
        keyword_query=expanded_terms,
        n_results=15
    )

    # 5. Temporal filtering for date-aware queries
    if self._is_temporal_query(question):
        results = self._filter_by_date(results, question)

    # 6. Build context and generate LLM response
    context = self._build_context(results)
    history = self.memory.get_history(conversation_id)

    answer = await self._generate_response(
        question=question,
        context=context,
        history=history
    )

    # 7. Cache result and store in conversation memory
    self.cache.set(question, answer)
    self.memory.add_exchange(conversation_id, question, answer)

    return answer

Query expansion

The system automatically expands queries with related terms. If you search for "interview", it also searches for "meeting", "schedule", "zoom", "calendar". This simple dictionary-based expansion catches emails that are relevant but use different vocabulary.

Temporal intelligence

One of the most useful features is temporal query handling. When the system detects time-related language ("upcoming", "next week", "last month"), it applies date filters to the results. It uses regex patterns and the dateutil library to parse dates from email content, handling over 10 different date formats.

Conversation memory

The system maintains conversation context across multiple questions. Ask "What meetings do I have this week?" followed by "Who organized the one on Thursday?" and the system understands "the one" refers to the Thursday meeting from the previous answer. Memory is managed with LRU eviction (50 conversations max, 24-hour TTL).

Query caching

An LRU cache with 5-minute TTL stores recent query results. If you ask the same question within 5 minutes, the response is instant — no embedding generation, no vector search, no LLM call.

Advantages of this approach

Building a custom RAG system for email search has clear advantages over traditional search or off-the-shelf AI tools:

Privacy and control

Your emails are indexed locally. The raw email content stays on your machine in SQLite and ChromaDB. Only the query and retrieved context snippets are sent to the LLM API. You could go fully local by swapping GPT-4o-mini for a local model via Ollama.

Semantic understanding

Traditional email search is purely keyword-based. This system understands that "project deadline reminders" should match emails containing "due date approaching" or "please submit by Friday". The vector embeddings capture meaning rather than exact words.

Hybrid search advantages

Neither pure semantic search nor pure keyword search is optimal for emails. Semantic search excels at understanding intent but can miss specific identifiers (invoice numbers, names). Keyword search finds exact matches but misses semantic connections. The RRF hybrid approach gets you both.

Conversation continuity

Unlike traditional search where every query is independent, this system remembers context. You can drill down into results with follow-up questions, making it feel like talking to an assistant who has read all your emails.

Cost efficiency

Using text-embedding-3-small and GPT-4o-mini via OpenRouter keeps costs minimal. Embedding 10,000 emails costs roughly a few cents, and each query costs fractions of a cent. The caching layer further reduces API calls for repeated queries.

Speed

The combination of ChromaDB's HNSW index for approximate nearest neighbor search, SQLite FTS5's inverted index for keyword search, and the LRU cache means most queries return in under 2 seconds — even across thousands of emails.

The frontend experience

The React frontend provides a clean chat interface built with Tailwind CSS:

Real-time sync dashboard — Watch emails being synced with per-folder progress bars and animated counters
Chat interface — Ask questions in natural language and get markdown-formatted responses with source citations
Conversation management — Start new conversations or continue existing ones
Source references — Every answer includes links to the original emails that informed the response

The UI is intentionally simple. The power is in the backend — the frontend just needs to get out of your way and let you ask questions.

Docker deployment

The entire system runs with a single Docker Compose command:

services:
  backend:
    build: ./backend
    ports:
      - "8000:8000"
    volumes:
      - ./data:/app/data
    env_file:
      - .env
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]

  frontend:
    build: ./frontend
    ports:
      - "80:80"
    depends_on:
      backend:
        condition: service_healthy

The backend persists all data (SQLite database, ChromaDB vectors, attachments) in a mounted volume, so your index survives container restarts. The frontend builds to static files served by nginx.

Configuration

The system is highly configurable through environment variables:

# Zoho Mail IMAP
IMAP_SERVER=imappro.zoho.com
IMAP_PORT=993
[email protected]
IMAP_PASSWORD=your-app-password

# AI Services via OpenRouter
OPENROUTER_API_KEY=your-key
EMBEDDING_MODEL=openai/text-embedding-3-small
CHAT_MODEL=openai/gpt-4o-mini

# Performance Tuning
SYNC_BATCH_SIZE=50
SYNC_PARALLEL_WORKERS=3
SYNC_INTERVAL_SECONDS=300

Using OpenRouter as the API gateway gives flexibility to swap models without code changes. Want to try Claude instead of GPT-4o-mini? Just change the CHAT_MODEL variable.

Lessons learned

SQLite is underrated

For a single-user application like this, SQLite is phenomenal. Combined with FTS5 for full-text search, it eliminates the need for Elasticsearch or any external search infrastructure. The asyncio lock pattern handles concurrent writes cleanly.

Hybrid search is worth the complexity

I initially built the system with vector search only. It worked well for semantic queries but missed obvious keyword matches. Adding FTS5 and RRF was a modest amount of additional code that dramatically improved result quality.

Batch everything

IMAP operations, embedding API calls, and database inserts are all I/O-bound. Batching these operations (50 emails per IMAP fetch, batch embedding calls) reduced total sync time by roughly 4x compared to processing emails one at a time.

Cache aggressively

The LRU cache with TTL was a late addition that made a huge difference in perceived performance. Most users ask variations of the same questions, and returning cached results instantly makes the system feel responsive.

What is next

There are several directions I want to take this project:

Local LLM support — Integrate Ollama for fully offline operation
Attachment search — Index PDF, DOCX, and other attachment content (the infrastructure for parsing is already in place)
Multi-account support — Connect multiple email accounts into a unified search
Scheduled sync — Background sync on an interval to keep the index fresh without manual triggers
Advanced filtering — Date range, folder, sender, and attachment filters in the UI

Conclusion

Building Zoho Mail RAG was a rewarding exercise in applying RAG techniques to a real-world problem that I face daily. The combination of IMAP syncing, hybrid search with Reciprocal Rank Fusion, conversation memory, and intelligent caching creates a system that genuinely changes how I interact with my email.

The project shows that RAG works well beyond PDFs and documentation: it can turn any text-heavy data source into a queryable knowledge base. If you deal with large volumes of email, I encourage you to build something similar. The tooling (FastAPI, ChromaDB, FTS5) is mature enough that you can go from zero to a working prototype in a weekend.

The full source code is available on my GitHub. Feel free to fork it, adapt it for your email provider, or use it as a starting point for your own RAG projects.