Add doc for workers architecture

2025-11-07 15:34:13 +07:00
parent 2dc34f07d8
commit 5578d272fa
9 changed files with 672 additions and 0 deletions
--- a/processing/05-Deployment-Architecture.md
+++ b/processing/05-Deployment-Architecture.md
@@ -0,0 +1,103 @@
+# Deployment Architecture
+
+This diagram shows the production deployment architecture with load balancing, clustering, and monitoring.
+
+```mermaid
+graph TB
+    subgraph "Load Balancer"
+        LB[NGINX/Cloudflare]
+    end
+    
+    subgraph "API Server Cluster"
+        direction LR
+        API1[API-1<br/>Orleans Silo<br/>Port: 11111]
+        API2[API-2<br/>Orleans Silo<br/>Port: 11121]
+        API3[API-3<br/>Orleans Silo<br/>Port: 11131]
+    end
+    
+    subgraph "Compute Worker Cluster"
+        direction LR
+        W1[Worker-1<br/>8 CPU Cores<br/>6 Concurrent Jobs]
+        W2[Worker-2<br/>8 CPU Cores<br/>6 Concurrent Jobs]
+        W3[Worker-3<br/>8 CPU Cores<br/>6 Concurrent Jobs]
+    end
+    
+    subgraph "Database Cluster"
+        direction LR
+        DB_MASTER[(PostgreSQL<br/>Master<br/>Job Queue)]
+        DB_REPLICA[(PostgreSQL<br/>Replica<br/>Read Only)]
+    end
+    
+    subgraph "Time Series DB"
+        INFLUX[(InfluxDB<br/>Candles Data)]
+    end
+    
+    subgraph "Monitoring"
+        PROM[Prometheus]
+        GRAF[Grafana]
+    end
+    
+    LB --> API1
+    LB --> API2
+    LB --> API3
+    
+    API1 --> DB_MASTER
+    API2 --> DB_MASTER
+    API3 --> DB_MASTER
+    
+    W1 --> DB_MASTER
+    W2 --> DB_MASTER
+    W3 --> DB_MASTER
+    
+    W1 --> INFLUX
+    W2 --> INFLUX
+    W3 --> INFLUX
+    
+    W1 --> PROM
+    W2 --> PROM
+    W3 --> PROM
+    API1 --> PROM
+    API2 --> PROM
+    API3 --> PROM
+    
+    PROM --> GRAF
+    
+    DB_MASTER --> DB_REPLICA
+    
+    style LB fill:#9B59B6
+    style API1 fill:#4A90E2
+    style API2 fill:#4A90E2
+    style API3 fill:#4A90E2
+    style W1 fill:#50C878
+    style W2 fill:#50C878
+    style W3 fill:#50C878
+    style DB_MASTER fill:#FF6B6B
+    style INFLUX fill:#FFD93D
+    style PROM fill:#E67E22
+    style GRAF fill:#E67E22
+```
+
+## Deployment Components
+
+### Load Balancer
+- **NGINX/Cloudflare**: Distributes incoming requests across API servers
+- Health checks and failover support
+
+### API Server Cluster
+- **3+ Instances**: Horizontally scalable Orleans silos
+- Each instance handles HTTP requests and Orleans grain operations
+- Ports: 11111, 11121, 11131 (for clustering)
+
+### Compute Worker Cluster
+- **3+ Instances**: Dedicated CPU workers
+- Each worker: 8 CPU cores, 6 concurrent backtests
+- Total capacity: 18 concurrent backtests across cluster
+
+### Database Cluster
+- **Master**: Handles all writes (job creation, updates)
+- **Replica**: Read-only for status queries and reporting
+
+### Monitoring
+- **Prometheus**: Metrics collection
+- **Grafana**: Visualization and dashboards
+