# Job Processing Flow

This diagram shows the detailed flow of how compute workers process backtest jobs from the queue.

```mermaid
flowchart TD
    Start([User Creates<br/>BundleBacktestRequest]) --> CreateJobs[API: Generate<br/>BacktestJobs]
    CreateJobs --> InsertDB[(Insert Jobs<br/>Status: Pending)]
    
    InsertDB --> WorkerPoll{Worker Polls<br/>Database}
    
    WorkerPoll -->|Every 5s| CheckJobs{Jobs<br/>Available?}
    CheckJobs -->|No| Wait[Wait 5s]
    Wait --> WorkerPoll
    
    CheckJobs -->|Yes| ClaimJobs[Claim Jobs<br/>Advisory Lock]
    ClaimJobs --> UpdateStatus[Update Status:<br/>Running]
    
    UpdateStatus --> CheckSemaphore{Semaphore<br/>Available?}
    CheckSemaphore -->|No| WaitSemaphore[Wait for<br/>slot]
    WaitSemaphore --> CheckSemaphore
    
    CheckSemaphore -->|Yes| AcquireSemaphore[Acquire<br/>Semaphore]
    AcquireSemaphore --> LoadCandles[Load Candles<br/>from InfluxDB]
    
    LoadCandles --> ProcessBacktest[Process Backtest<br/>CPU-intensive]
    
    ProcessBacktest --> UpdateProgress{Every<br/>10%?}
    UpdateProgress -->|Yes| SaveProgress[Update Progress<br/>in DB]
    SaveProgress --> ProcessBacktest
    UpdateProgress -->|No| ProcessBacktest
    
    ProcessBacktest --> BacktestComplete{Backtest<br/>Complete?}
    BacktestComplete -->|No| ProcessBacktest
    BacktestComplete -->|Yes| SaveResult[Save Result<br/>Status: Completed]
    
    SaveResult --> UpdateBundle[Update Bundle<br/>Progress]
    UpdateBundle --> ReleaseSemaphore[Release<br/>Semaphore]
    ReleaseSemaphore --> WorkerPoll
    
    style Start fill:#4A90E2
    style ProcessBacktest fill:#50C878
    style SaveResult fill:#FF6B6B
    style WorkerPoll fill:#FFD93D
```

## Key Components

- **Worker Polling**: Workers continuously poll database for pending jobs
- **Advisory Locks**: PostgreSQL advisory locks prevent multiple workers from claiming the same job
- **Semaphore Control**: Limits concurrent backtests per worker (default: CPU cores - 2)
- **Progress Updates**: Progress is saved to database every 10% completion
- **Bundle Updates**: Individual job completion updates the parent bundle request