# Workers Processing Architecture

This folder contains documentation for the enterprise-grade backtest processing architecture using a database queue pattern with separate API and Compute worker clusters.

## Overview

The architecture separates concerns between:
- **API Server**: Handles HTTP requests, creates jobs, returns immediately (fire-and-forget)
- **Compute Workers**: Process CPU-intensive backtest jobs from the database queue
- **Database Queue**: Central coordination point using PostgreSQL

## Documentation Files

1. **[01-Overall-Architecture.md](./01-Overall-Architecture.md)**
   - Complete system architecture diagram
   - Component relationships
   - External service integrations

2. **[02-Request-Flow.md](./02-Request-Flow.md)**
   - Sequence diagram of request flow
   - User request → Job creation → Processing → Status polling

3. **[03-Job-Processing-Flow.md](./03-Job-Processing-Flow.md)**
   - Detailed job processing workflow
   - Worker polling, job claiming, semaphore control

4. **[04-Database-Schema.md](./04-Database-Schema.md)**
   - Entity relationship diagram
   - Database schema for job queue
   - Key indexes and relationships

5. **[05-Deployment-Architecture.md](./05-Deployment-Architecture.md)**
   - Production deployment topology
   - Load balancing, clustering, monitoring

6. **[06-Concurrency-Control.md](./06-Concurrency-Control.md)**
   - Concurrency control mechanisms
   - Semaphore-based limiting
   - Capacity calculations

7. **[07-Monorepo-Structure.md](./07-Monorepo-Structure.md)**
   - Monorepo project organization
   - Shared projects and dependencies

## Key Features

- ✅ **No Timeouts**: Fire-and-forget pattern with polling
- ✅ **Scalable**: Horizontal scaling of both API and Compute clusters
- ✅ **Reliable**: Jobs persist in database, survive restarts
- ✅ **Efficient**: Dedicated CPU resources for compute work
- ✅ **Enterprise-Grade**: Handles 1000+ users, priority queue, health checks

## Architecture Principles

1. **Separation of Concerns**: API handles requests, Compute handles CPU work
2. **Database as Queue**: PostgreSQL serves as reliable job queue
3. **Shared Codebase**: Monorepo with shared business logic
4. **Resource Isolation**: Compute workers don't interfere with API responsiveness
5. **Fault Tolerance**: Jobs survive worker failures, can be reclaimed

## Capacity Planning

- **Per Worker**: 6 concurrent backtests (8-core machine)
- **3 Workers**: 18 concurrent backtests
- **Throughput**: ~1,080 backtests/hour
- **1000 Users × 10 backtests**: ~9 hours processing time

## Next Steps

1. Create `Managing.Compute` project
2. Implement `BacktestJob` entity and repository
3. Create `BacktestComputeWorker` background service
4. Update API controllers to use job queue pattern
5. Deploy compute workers to dedicated servers