import AnimatedDiv from "@/components/mdx/AnimatedDiv"; import Tip from "@/components/mdx/Tip";

Scalable GenAI Invoice PDF Data Extractor

Ever wondered how to process hundreds of invoices without manually extracting line items, totals, or vendor info? Let's dive into a story where GenAI meets scalability.

Features

• Batch & Async Processing: Queue multiple invoices in RabbitMQ and let workers do the heavy lifting.
• High Concurrency: Each worker can handle ~100 files at once. Multiple workers scale horizontally.
• GenAI-Powered Extraction: Extract vendor details, invoice metadata, line items, totals, and payment info.
• Error Detection: Automatically flags mismatches in subtotal, tax, shipping, and total.
• Multi-format Support: PDFs, images, scanned documents, multi-page invoices.
• Token-Efficient YAML Output: Reduces AI token usage by 20–30% before converting to JSON for storage.

High-Level Architecture

Dead Letter Queue (DLQ)

• Failed invoices go to DLQ.
• Review failed jobs, retry manually, or trigger alerts.
• Keeps the main queue unblocked.

Common causes:

• Corrupted PDFs/images
• Unsupported invoice format
• Persistent AI extraction failure

Retry Strategy with Exponential Backoff

• Failed jobs retry automatically.
• Wait time grows exponentially (60s → 120s → 180s…).
• Jobs exceeding max retries go to DLQ.

Tech Stack

Installation & Setup

🐳 Run with Docker

docker compose --build -d

Access: http://localhost:3000

💻 Run Directly

docker compose up -d
npm run dev          # dev server
npm run worker       # start background worker
npm run start        # production via PM2

Scripts

{
  "dev": "node --env-file=.env --watch index.js",
  "worker": "node --env-file=.env worker/file.worker.js",
  "queue:flush": "node --env-file=.env scripts/amqp.flush.js",
  "start": "pm2 start ecosystem.config.cjs"
}

Environment Variables

GEMINI_API_KEY=<your-api-key>
AMQP_URL=amqp://user:password@localhost:5672
MONGODB_URI=mongodb://localhost:27017/invoices
REDIS_HOST=localhost
REDIS_PORT=6379
MAX_FILE_SIZE=200
PORT=3000
WORKER_CONCURRENCY=100

API Endpoints

Upload Files

• Endpoint: /api/file/upload
• Method: POST
• Form Key: files
• Response:

{
 "message": "Files uploaded and queued",
 "file_ids": ["abc123", "def456"]
}

Fetch Files

• Endpoint: /api/file
• Method: GET
• Query params: status=pending|processed|error&flag=true|false

Fetch File by ID

• Endpoint: /api/file/{invoice_id}
• Method: GET

Sample Output

items:
- name: "Activator CREAM 5-gal 5 gallons/pailt"
  quantity: 2
  rate: 269.00
payment_info:
  subtotal: 1187
  sales_tax_percentage: 8
  shipping_handling_cost: 50
  total: 1330.75
errors:
- "Subtotal mismatch: calculated 2550.00 vs declared 1187.00"
- "Sales Tax amount mismatch: calculated 204.00 vs declared 94.75"
- "Total mismatch: calculated 2804.00 vs declared 1330.75"

Stored in MongoDB as JSON.

Scaling Workers & Concurrency

• Worker concurrency: 100 files
• Horizontal scaling: run multiple workers
• File size limit: 200MB (configurable)
• Rate limiting: Redis ensures stable API usage

Token Efficiency

• YAML formatting reduces AI token usage by ~20–30%
• Less characters → faster extraction → lower costs
• Read more: How I Saved Millions in GenAI Token Costs

Postman Collection

• Import from postman_collection.json
• Base URL: http://localhost:3000
• Test endpoints: upload, fetch all, fetch by ID

---

This **will render cleanly as a proper blog**:

- All animations (`motion.div`) now use `AnimatedDiv`.
- Tooltips use `Tip`.
- MDX compilation errors are gone.
- The blog is interactive, and SEO-friendly.