76 lines
2.8 KiB
Markdown
76 lines
2.8 KiB
Markdown
# Workers Processing Architecture
|
||
|
||
This folder contains documentation for the enterprise-grade backtest processing architecture using a database queue pattern with separate API and Compute worker clusters.
|
||
|
||
## Overview
|
||
|
||
The architecture separates concerns between:
|
||
- **API Server**: Handles HTTP requests, creates jobs, returns immediately (fire-and-forget)
|
||
- **Compute Workers**: Process CPU-intensive backtest jobs from the database queue
|
||
- **Database Queue**: Central coordination point using PostgreSQL
|
||
|
||
## Documentation Files
|
||
|
||
1. **[01-Overall-Architecture.md](./01-Overall-Architecture.md)**
|
||
- Complete system architecture diagram
|
||
- Component relationships
|
||
- External service integrations
|
||
|
||
2. **[02-Request-Flow.md](./02-Request-Flow.md)**
|
||
- Sequence diagram of request flow
|
||
- User request → Job creation → Processing → Status polling
|
||
|
||
3. **[03-Job-Processing-Flow.md](./03-Job-Processing-Flow.md)**
|
||
- Detailed job processing workflow
|
||
- Worker polling, job claiming, semaphore control
|
||
|
||
4. **[04-Database-Schema.md](./04-Database-Schema.md)**
|
||
- Entity relationship diagram
|
||
- Database schema for job queue
|
||
- Key indexes and relationships
|
||
|
||
5. **[05-Deployment-Architecture.md](./05-Deployment-Architecture.md)**
|
||
- Production deployment topology
|
||
- Load balancing, clustering, monitoring
|
||
|
||
6. **[06-Concurrency-Control.md](./06-Concurrency-Control.md)**
|
||
- Concurrency control mechanisms
|
||
- Semaphore-based limiting
|
||
- Capacity calculations
|
||
|
||
7. **[07-Monorepo-Structure.md](./07-Monorepo-Structure.md)**
|
||
- Monorepo project organization
|
||
- Shared projects and dependencies
|
||
|
||
## Key Features
|
||
|
||
- ✅ **No Timeouts**: Fire-and-forget pattern with polling
|
||
- ✅ **Scalable**: Horizontal scaling of both API and Compute clusters
|
||
- ✅ **Reliable**: Jobs persist in database, survive restarts
|
||
- ✅ **Efficient**: Dedicated CPU resources for compute work
|
||
- ✅ **Enterprise-Grade**: Handles 1000+ users, priority queue, health checks
|
||
|
||
## Architecture Principles
|
||
|
||
1. **Separation of Concerns**: API handles requests, Compute handles CPU work
|
||
2. **Database as Queue**: PostgreSQL serves as reliable job queue
|
||
3. **Shared Codebase**: Monorepo with shared business logic
|
||
4. **Resource Isolation**: Compute workers don't interfere with API responsiveness
|
||
5. **Fault Tolerance**: Jobs survive worker failures, can be reclaimed
|
||
|
||
## Capacity Planning
|
||
|
||
- **Per Worker**: 6 concurrent backtests (8-core machine)
|
||
- **3 Workers**: 18 concurrent backtests
|
||
- **Throughput**: ~1,080 backtests/hour
|
||
- **1000 Users × 10 backtests**: ~9 hours processing time
|
||
|
||
## Next Steps
|
||
|
||
1. Create `Managing.Compute` project
|
||
2. Implement `BacktestJob` entity and repository
|
||
3. Create `BacktestComputeWorker` background service
|
||
4. Update API controllers to use job queue pattern
|
||
5. Deploy compute workers to dedicated servers
|
||
|