managing-apps/assets/documentation/Workers processing/README.md

Workers Processing Architecture

This folder contains documentation for the enterprise-grade backtest processing architecture using a database queue pattern with separate API and Compute worker clusters.

Overview

The architecture separates concerns between:

• API Server: Handles HTTP requests, creates jobs, returns immediately (fire-and-forget)
• Compute Workers: Process CPU-intensive backtest jobs from the database queue
• Database Queue: Central coordination point using PostgreSQL

Documentation Files

1. 01-Overall-Architecture.md

   • Complete system architecture diagram
   • Component relationships
   • External service integrations

2. 02-Request-Flow.md

   • Sequence diagram of request flow
   • User request → Job creation → Processing → Status polling

3. 03-Job-Processing-Flow.md

   • Detailed job processing workflow
   • Worker polling, job claiming, semaphore control

4. 04-Database-Schema.md

   • Entity relationship diagram
   • Database schema for job queue
   • Key indexes and relationships

5. 05-Deployment-Architecture.md

   • Production deployment topology
   • Load balancing, clustering, monitoring

6. 06-Concurrency-Control.md

   • Concurrency control mechanisms
   • Semaphore-based limiting
   • Capacity calculations

7. 07-Monorepo-Structure.md

   • Monorepo project organization
   • Shared projects and dependencies

Key Features

• ✅ No Timeouts: Fire-and-forget pattern with polling
• ✅ Scalable: Horizontal scaling of both API and Compute clusters
• ✅ Reliable: Jobs persist in database, survive restarts
• ✅ Efficient: Dedicated CPU resources for compute work
• ✅ Enterprise-Grade: Handles 1000+ users, priority queue, health checks

Architecture Principles

1. Separation of Concerns: API handles requests, Compute handles CPU work
2. Database as Queue: PostgreSQL serves as reliable job queue
3. Shared Codebase: Monorepo with shared business logic
4. Resource Isolation: Compute workers don't interfere with API responsiveness
5. Fault Tolerance: Jobs survive worker failures, can be reclaimed

Capacity Planning

• Per Worker: 6 concurrent backtests (8-core machine)
• 3 Workers: 18 concurrent backtests
• Throughput: ~1,080 backtests/hour
• 1000 Users × 10 backtests: ~9 hours processing time

Next Steps

1. Create Managing.Compute project
2. Implement BacktestJob entity and repository
3. Create BacktestComputeWorker background service
4. Update API controllers to use job queue pattern
5. Deploy compute workers to dedicated servers