managing-apps/assets/documentation/Workers processing/05-Deployment-Architecture.md

Deployment Architecture

This diagram shows the production deployment architecture with load balancing, clustering, and monitoring.

graph TB
    subgraph "Load Balancer"
        LB[NGINX/Cloudflare]
    end
    
    subgraph "API Server Cluster"
        direction LR
        API1[API-1<br/>Orleans Silo<br/>Port: 11111]
        API2[API-2<br/>Orleans Silo<br/>Port: 11121]
        API3[API-3<br/>Orleans Silo<br/>Port: 11131]
    end
    
    subgraph "Compute Worker Cluster"
        direction LR
        W1[Worker-1<br/>8 CPU Cores<br/>6 Concurrent Jobs]
        W2[Worker-2<br/>8 CPU Cores<br/>6 Concurrent Jobs]
        W3[Worker-3<br/>8 CPU Cores<br/>6 Concurrent Jobs]
    end
    
    subgraph "Database Cluster"
        direction LR
        DB_MASTER[(PostgreSQL<br/>Master<br/>Job Queue)]
        DB_REPLICA[(PostgreSQL<br/>Replica<br/>Read Only)]
    end
    
    subgraph "Time Series DB"
        INFLUX[(InfluxDB<br/>Candles Data)]
    end
    
    subgraph "Monitoring"
        PROM[Prometheus]
        GRAF[Grafana]
    end
    
    LB --> API1
    LB --> API2
    LB --> API3
    
    API1 --> DB_MASTER
    API2 --> DB_MASTER
    API3 --> DB_MASTER
    
    W1 --> DB_MASTER
    W2 --> DB_MASTER
    W3 --> DB_MASTER
    
    W1 --> INFLUX
    W2 --> INFLUX
    W3 --> INFLUX
    
    W1 --> PROM
    W2 --> PROM
    W3 --> PROM
    API1 --> PROM
    API2 --> PROM
    API3 --> PROM
    
    PROM --> GRAF
    
    DB_MASTER --> DB_REPLICA
    
    style LB fill:#9B59B6
    style API1 fill:#4A90E2
    style API2 fill:#4A90E2
    style API3 fill:#4A90E2
    style W1 fill:#50C878
    style W2 fill:#50C878
    style W3 fill:#50C878
    style DB_MASTER fill:#FF6B6B
    style INFLUX fill:#FFD93D
    style PROM fill:#E67E22
    style GRAF fill:#E67E22

Deployment Components

Load Balancer

• NGINX/Cloudflare: Distributes incoming requests across API servers
• Health checks and failover support

API Server Cluster

• 3+ Instances: Horizontally scalable Orleans silos
• Each instance handles HTTP requests and Orleans grain operations
• Ports: 11111, 11121, 11131 (for clustering)

Compute Worker Cluster

• 3+ Instances: Dedicated CPU workers
• Each worker: 8 CPU cores, 6 concurrent backtests
• Total capacity: 18 concurrent backtests across cluster

Database Cluster

• Master: Handles all writes (job creation, updates)
• Replica: Read-only for status queries and reporting

Monitoring

• Prometheus: Metrics collection
• Grafana: Visualization and dashboards