Add todo for backtest performance

assets/BacktestPerformanceOptimizations.md

@@ -0,0 +1,169 @@
+# Backtest Performance Optimizations
+
+This document tracks identified performance optimization opportunities for `BacktestExecutor.cs` based on analysis of the foreach loop that processes thousands of candles.
+
+## Current Performance Baseline
+
+- **Processing Rate**: ~1,707 candles/sec
+- **Execution Time**: ~3.365 seconds for 5,760 candles
+- **Memory Peak**: ~36.29 MB
+
+## Optimization Opportunities
+
+### 🔴 Priority 1: Reuse HashSet Instead of Recreating (CRITICAL)
+
+**Location**: `BacktestExecutor.cs` line 267
+
+**Current Code**:
+```csharp
+var fixedCandles = new HashSet<Candle>(rollingWindowCandles);
+```
+
+**Problem**: Creates a new HashSet 5,760 times (once per candle iteration). This is extremely expensive in terms of:
+- Memory allocations
+- GC pressure
+- CPU cycles for hash calculations
+
+**Solution**: Reuse HashSet and update incrementally:
+```csharp
+// Initialize before loop
+var fixedCandles = new HashSet<Candle>(RollingWindowSize);
+
+// Inside loop (replace lines 255-267):
+if (rollingWindowCandles.Count >= RollingWindowSize)
+{
+    var removedCandle = rollingWindowCandles.Dequeue();
+    fixedCandles.Remove(removedCandle);
+}
+rollingWindowCandles.Enqueue(candle);
+fixedCandles.Add(candle);
+// fixedCandles is now up-to-date, no need to recreate
+```
+
+**Expected Impact**: 20-30% performance improvement
+
+---
+
+### 🟠 Priority 2: Optimize Wallet Balance Tracking
+
+**Location**: `BacktestExecutor.cs` line 283
+
+**Current Code**:
+```csharp
+lastWalletBalance = tradingBot.WalletBalances.Values.LastOrDefault();
+```
+
+**Problem**: `LastOrDefault()` on `Dictionary.Values` is O(n) operation, called every 10 candles.
+
+**Solution**: Track balance directly or use more efficient structure:
+```csharp
+// Option 1: Cache last balance when wallet updates
+// Option 2: Use SortedDictionary if order matters
+// Option 3: Maintain separate variable that updates when wallet changes
+```
+
+**Expected Impact**: 2-5% performance improvement
+
+---
+
+### 🟡 Priority 3: Optimize TradingBox.GetSignal Input
+
+**Location**: `TradingBox.cs` line 130
+
+**Current Code**:
+```csharp
+var limitedCandles = newCandles.ToList(); // Converts HashSet to List
+```
+
+**Problem**: Converts HashSet to List every time `GetSignal` is called.
+
+**Solution**: 
+- Modify `TradingBox.GetSignal` to accept `IEnumerable<Candle>` or `List<Candle>`
+- Pass List directly from rolling window instead of HashSet
+
+**Expected Impact**: 1-3% performance improvement
+
+---
+
+### 🟢 Priority 4: Cache Progress Percentage Calculation
+
+**Location**: `BacktestExecutor.cs` line 297
+
+**Current Code**:
+```csharp
+var currentPercentage = (currentCandle * 100) / totalCandles;
+```
+
+**Problem**: Integer division recalculated every iteration (minor but can be optimized).
+
+**Solution**:
+```csharp
+// Before loop
+const double percentageMultiplier = 100.0 / totalCandles;
+
+// Inside loop
+var currentPercentage = (int)(currentCandle * percentageMultiplier);
+```
+
+**Expected Impact**: <1% performance improvement (minor optimization)
+
+---
+
+### 🟢 Priority 5: Use Stopwatch for Time Checks
+
+**Location**: `BacktestExecutor.cs` line 298
+
+**Current Code**:
+```csharp
+var timeSinceLastUpdate = (DateTime.UtcNow - lastProgressUpdate).TotalMilliseconds;
+```
+
+**Problem**: `DateTime.UtcNow` is relatively expensive when called frequently.
+
+**Solution**: Use `Stopwatch` for timing:
+```csharp
+var progressStopwatch = Stopwatch.StartNew();
+// Then check: progressStopwatch.ElapsedMilliseconds >= progressUpdateIntervalMs
+```
+
+**Expected Impact**: <1% performance improvement (minor optimization)
+
+---
+
+## Future Considerations
+
+### Batching Candle Processing
+If business logic allows, process multiple candles before updating signals to reduce `UpdateSignals()` call frequency. Requires careful validation.
+
+### Object Pooling
+Reuse List/HashSet instances if possible to reduce GC pressure. May require careful state management.
+
+### Parallel Processing
+If signals are independent, consider parallel indicator calculations. Requires careful validation to ensure business logic integrity.
+
+## Implementation Checklist
+
+- [ ] Priority 1: Reuse HashSet instead of recreating
+- [ ] Priority 2: Optimize wallet balance tracking
+- [ ] Priority 3: Optimize TradingBox.GetSignal input
+- [ ] Priority 4: Cache progress percentage calculation
+- [ ] Priority 5: Use Stopwatch for time checks
+- [ ] Run benchmark-backtest-performance.sh to validate improvements
+- [ ] Ensure business logic validation passes (Final PnL matches baseline)
+
+## Expected Total Impact
+
+**Combined Expected Improvement**: 25-40% faster execution
+
+**Target Performance**:
+- Processing Rate: ~2,100-2,400 candles/sec (up from ~1,707)
+- Execution Time: ~2.0-2.5 seconds (down from ~3.365)
+- Memory: Similar or slightly reduced
+
+## Notes
+
+- Always validate business logic after optimizations
+- Run benchmarks multiple times to account for system variance
+- Monitor memory usage to ensure optimizations don't increase GC pressure
+- Priority 1 (HashSet reuse) should provide the largest performance gain
+

src/Managing.Application/Backtests/BacktestExecutor.cs

@@ -264,6 +264,8 @@ public class BacktestExecutor
             // Run with optimized backtest path (minimize async calls)
             var signalUpdateStart = Stopwatch.GetTimestamp();
             // Convert rolling window to HashSet for TradingBot.UpdateSignals compatibility
+            // NOTE: Recreating HashSet each iteration is necessary to maintain correct enumeration order
+            // Incremental updates break business logic (changes PnL results)
             var fixedCandles = new HashSet<Candle>(rollingWindowCandles);
             await tradingBot.UpdateSignals(fixedCandles, preCalculatedIndicatorValues);
             signalUpdateTotalTime += Stopwatch.GetElapsedTime(signalUpdateStart);

src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv

@@ -21,3 +21,4 @@ DateTime,TestName,CandlesCount,ExecutionTimeSeconds,ProcessingRateCandlesPerSec,
 2025-11-15T06:46:21Z,Telemetry_ETH_RSI_EMACROSS,5760,12.58,457.8,28.82,21.79,35.28,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,e814eb74,dev,development
 2025-11-15T06:50:04Z,Telemetry_ETH_RSI_EMACROSS,5760,4.84,1190.4,29.01,19.10,35.17,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,e814eb74,dev,development
 2025-11-15T07:11:55Z,Telemetry_ETH_RSI_EMACROSS,5760,5.44,1059.4,28.81,18.07,33.80,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,bed25e72,dev,development
+2025-11-15T07:22:05Z,Telemetry_ETH_RSI_EMACROSS,5760,10.71,537.9,28.81,18.06,33.84,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,49a693b4,dev,development

src/Managing.Workers.Tests/performance-benchmarks.csv

@@ -66,3 +66,4 @@ DateTime,TestName,CandlesCount,ExecutionTimeSeconds,ProcessingRateCandlesPerSec,
 2025-11-15T06:46:21Z,Telemetry_ETH_RSI,5760,4.83,1191.0,29.02,20.22,37.20,4105.51,0,0.0,499.39,0.00,0.09,-30689.97,24,-51.70,0.00,e814eb74,dev,development
 2025-11-15T06:50:04Z,Telemetry_ETH_RSI,5760,4.47,1286.2,28.81,20.58,34.89,3324.75,0,0.0,965.71,0.00,0.17,-30689.97,24,-51.70,0.00,e814eb74,dev,development
 2025-11-15T07:11:55Z,Telemetry_ETH_RSI,5760,3.365,1707.1,29.06,20.43,36.29,2872.29,0,0.0,371.33,0.00,0.06,-30689.97,24,-51.70,0.00,bed25e72,dev,development
+2025-11-15T07:22:05Z,Telemetry_ETH_RSI,5760,7.49,766.2,28.80,20.86,34.90,5992.19,0,0.0,916.71,0.00,0.16,-30689.97,24,-51.70,0.00,49a693b4,dev,development