managing-apps/assets/documentation/Workers processing/IMPLEMENTATION-PLAN.md

Implementation Plan

Phase 1: Database & Domain Setup

• Create BacktestJob entity in Managing.Domain/Backtests/
• Create BacktestJobStatus enum (Pending, Running, Completed, Failed)
• Create database migration for BacktestJobs table
• Add indexes: idx_status_priority, idx_bundle_request, idx_assigned_worker
• Create IBacktestJobRepository interface
• Implement BacktestJobRepository with advisory lock support

Phase 2: Compute Worker Project

• Refactor Managing.Workers.Api project (or rename to Managing.Compute)
• Remove Orleans dependencies completely
• Add project references to shared projects (Application, Domain, Infrastructure)
• Configure DI container with all required services (NO Orleans)
• Create BacktestComputeWorker background service
• Implement job polling logic (every 5 seconds)
• Implement job claiming with PostgreSQL advisory locks
• Implement semaphore-based concurrency control
• Implement progress callback mechanism
• Implement heartbeat mechanism (every 30 seconds)
• Add configuration: MaxConcurrentBacktests, JobPollIntervalSeconds, WorkerId

Phase 3: API Server Updates

• Update BacktestController to create jobs instead of calling grains directly
• Implement CreateBundleBacktest endpoint (returns immediately)
• Implement GetJobStatus endpoint (polls database for single job)
• Implement GetBundleStatus endpoint (polls database, aggregates job statuses)
• Update Backtester.cs to generate BacktestJob entities from bundle variants
• Remove all Orleans grain calls for backtests (direct replacement, no feature flags)
• Remove IGrainFactory dependency from Backtester.cs

Phase 4: Shared Logic Extraction

• Create BacktestExecutor.cs service (new file)
• Extract backtest execution logic from BacktestTradingBotGrain to BacktestExecutor
• Make backtest logic Orleans-agnostic (no grain dependencies)
• Add progress callback support to execution method
• Ensure candle loading works in compute worker context
• Handle credit debiting/refunding in executor
• Handle user context resolution in executor

Phase 5: Monitoring & Health Checks

• Add health check endpoint to compute worker (/health or /healthz)
• Add metrics: pending jobs, running jobs, completed/failed counts
• Add stale job detection (reclaim jobs from dead workers, LastHeartbeat > 5 min)
• Add comprehensive logging for job lifecycle events
• Include structured logging: JobId, BundleRequestId, UserId, WorkerId, Duration

Phase 6: SignalR & Notifications

• Inject IHubContext<BacktestHub> into compute worker or executor
• Send SignalR progress updates during job execution
• Update BacktestJob.ProgressPercentage in database
• Update BundleBacktestRequest progress when jobs complete
• Send completion notifications via SignalR and Telegram

Phase 7: Deployment

• Create Dockerfile for Managing.Compute (or update existing)
• Update docker-compose.yml to add compute worker service
• Configure environment variables: MaxConcurrentBacktests, JobPollIntervalSeconds, WorkerId
• Set up health check configuration in Docker
• Configure auto-scaling rules for compute workers (min: 1, max: 10)

Phase 9: Testing & Validation

• Unit tests: BacktestJobRepository (advisory locks, job claiming, stale detection)
• Unit tests: BacktestExecutor (core logic, progress callbacks)
• Integration tests: Single backtest job processing
• Integration tests: Bundle backtest with multiple jobs
• Integration tests: Concurrent job processing (multiple workers)
• Integration tests: Job recovery after worker failure
• Integration tests: Priority queue ordering
• Load tests: 100+ concurrent users, 1000+ pending jobs, multiple workers

Phase 8: Cleanup & Removal

• Remove or deprecate BacktestTradingBotGrain.cs (no longer used)
• Remove or deprecate BundleBacktestGrain.cs (replaced by compute workers)
• Remove Orleans grain interfaces for backtests (if not used elsewhere)
• Update ApiBootstrap.cs to remove Orleans backtest grain registrations
• Remove Orleans dependencies from Backtester.cs (keep for other operations)
• Update documentation to reflect new architecture