From 428e36d744ba234a82f4ac29293274fcc04c62d6 Mon Sep 17 00:00:00 2001 From: cryptooda Date: Sat, 15 Nov 2025 20:53:08 +0700 Subject: [PATCH] Add todo for backtest performance --- assets/BacktestPerformanceOptimizations.md | 169 ++++++++++++++++++ .../Backtests/BacktestExecutor.cs | 2 + .../performance-benchmarks-two-scenarios.csv | 1 + .../performance-benchmarks.csv | 1 + 4 files changed, 173 insertions(+) create mode 100644 assets/BacktestPerformanceOptimizations.md diff --git a/assets/BacktestPerformanceOptimizations.md b/assets/BacktestPerformanceOptimizations.md new file mode 100644 index 00000000..50adc629 --- /dev/null +++ b/assets/BacktestPerformanceOptimizations.md @@ -0,0 +1,169 @@ +# Backtest Performance Optimizations + +This document tracks identified performance optimization opportunities for `BacktestExecutor.cs` based on analysis of the foreach loop that processes thousands of candles. + +## Current Performance Baseline + +- **Processing Rate**: ~1,707 candles/sec +- **Execution Time**: ~3.365 seconds for 5,760 candles +- **Memory Peak**: ~36.29 MB + +## Optimization Opportunities + +### 🔴 Priority 1: Reuse HashSet Instead of Recreating (CRITICAL) + +**Location**: `BacktestExecutor.cs` line 267 + +**Current Code**: +```csharp +var fixedCandles = new HashSet(rollingWindowCandles); +``` + +**Problem**: Creates a new HashSet 5,760 times (once per candle iteration). This is extremely expensive in terms of: +- Memory allocations +- GC pressure +- CPU cycles for hash calculations + +**Solution**: Reuse HashSet and update incrementally: +```csharp +// Initialize before loop +var fixedCandles = new HashSet(RollingWindowSize); + +// Inside loop (replace lines 255-267): +if (rollingWindowCandles.Count >= RollingWindowSize) +{ + var removedCandle = rollingWindowCandles.Dequeue(); + fixedCandles.Remove(removedCandle); +} +rollingWindowCandles.Enqueue(candle); +fixedCandles.Add(candle); +// fixedCandles is now up-to-date, no need to recreate +``` + +**Expected Impact**: 20-30% performance improvement + +--- + +### 🟠 Priority 2: Optimize Wallet Balance Tracking + +**Location**: `BacktestExecutor.cs` line 283 + +**Current Code**: +```csharp +lastWalletBalance = tradingBot.WalletBalances.Values.LastOrDefault(); +``` + +**Problem**: `LastOrDefault()` on `Dictionary.Values` is O(n) operation, called every 10 candles. + +**Solution**: Track balance directly or use more efficient structure: +```csharp +// Option 1: Cache last balance when wallet updates +// Option 2: Use SortedDictionary if order matters +// Option 3: Maintain separate variable that updates when wallet changes +``` + +**Expected Impact**: 2-5% performance improvement + +--- + +### 🟡 Priority 3: Optimize TradingBox.GetSignal Input + +**Location**: `TradingBox.cs` line 130 + +**Current Code**: +```csharp +var limitedCandles = newCandles.ToList(); // Converts HashSet to List +``` + +**Problem**: Converts HashSet to List every time `GetSignal` is called. + +**Solution**: +- Modify `TradingBox.GetSignal` to accept `IEnumerable` or `List` +- Pass List directly from rolling window instead of HashSet + +**Expected Impact**: 1-3% performance improvement + +--- + +### 🟢 Priority 4: Cache Progress Percentage Calculation + +**Location**: `BacktestExecutor.cs` line 297 + +**Current Code**: +```csharp +var currentPercentage = (currentCandle * 100) / totalCandles; +``` + +**Problem**: Integer division recalculated every iteration (minor but can be optimized). + +**Solution**: +```csharp +// Before loop +const double percentageMultiplier = 100.0 / totalCandles; + +// Inside loop +var currentPercentage = (int)(currentCandle * percentageMultiplier); +``` + +**Expected Impact**: <1% performance improvement (minor optimization) + +--- + +### 🟢 Priority 5: Use Stopwatch for Time Checks + +**Location**: `BacktestExecutor.cs` line 298 + +**Current Code**: +```csharp +var timeSinceLastUpdate = (DateTime.UtcNow - lastProgressUpdate).TotalMilliseconds; +``` + +**Problem**: `DateTime.UtcNow` is relatively expensive when called frequently. + +**Solution**: Use `Stopwatch` for timing: +```csharp +var progressStopwatch = Stopwatch.StartNew(); +// Then check: progressStopwatch.ElapsedMilliseconds >= progressUpdateIntervalMs +``` + +**Expected Impact**: <1% performance improvement (minor optimization) + +--- + +## Future Considerations + +### Batching Candle Processing +If business logic allows, process multiple candles before updating signals to reduce `UpdateSignals()` call frequency. Requires careful validation. + +### Object Pooling +Reuse List/HashSet instances if possible to reduce GC pressure. May require careful state management. + +### Parallel Processing +If signals are independent, consider parallel indicator calculations. Requires careful validation to ensure business logic integrity. + +## Implementation Checklist + +- [ ] Priority 1: Reuse HashSet instead of recreating +- [ ] Priority 2: Optimize wallet balance tracking +- [ ] Priority 3: Optimize TradingBox.GetSignal input +- [ ] Priority 4: Cache progress percentage calculation +- [ ] Priority 5: Use Stopwatch for time checks +- [ ] Run benchmark-backtest-performance.sh to validate improvements +- [ ] Ensure business logic validation passes (Final PnL matches baseline) + +## Expected Total Impact + +**Combined Expected Improvement**: 25-40% faster execution + +**Target Performance**: +- Processing Rate: ~2,100-2,400 candles/sec (up from ~1,707) +- Execution Time: ~2.0-2.5 seconds (down from ~3.365) +- Memory: Similar or slightly reduced + +## Notes + +- Always validate business logic after optimizations +- Run benchmarks multiple times to account for system variance +- Monitor memory usage to ensure optimizations don't increase GC pressure +- Priority 1 (HashSet reuse) should provide the largest performance gain + diff --git a/src/Managing.Application/Backtests/BacktestExecutor.cs b/src/Managing.Application/Backtests/BacktestExecutor.cs index 5219736f..31eaa020 100644 --- a/src/Managing.Application/Backtests/BacktestExecutor.cs +++ b/src/Managing.Application/Backtests/BacktestExecutor.cs @@ -264,6 +264,8 @@ public class BacktestExecutor // Run with optimized backtest path (minimize async calls) var signalUpdateStart = Stopwatch.GetTimestamp(); // Convert rolling window to HashSet for TradingBot.UpdateSignals compatibility + // NOTE: Recreating HashSet each iteration is necessary to maintain correct enumeration order + // Incremental updates break business logic (changes PnL results) var fixedCandles = new HashSet(rollingWindowCandles); await tradingBot.UpdateSignals(fixedCandles, preCalculatedIndicatorValues); signalUpdateTotalTime += Stopwatch.GetElapsedTime(signalUpdateStart); diff --git a/src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv b/src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv index 40c0b26d..38e89d5e 100644 --- a/src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv +++ b/src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv @@ -21,3 +21,4 @@ DateTime,TestName,CandlesCount,ExecutionTimeSeconds,ProcessingRateCandlesPerSec, 2025-11-15T06:46:21Z,Telemetry_ETH_RSI_EMACROSS,5760,12.58,457.8,28.82,21.79,35.28,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,e814eb74,dev,development 2025-11-15T06:50:04Z,Telemetry_ETH_RSI_EMACROSS,5760,4.84,1190.4,29.01,19.10,35.17,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,e814eb74,dev,development 2025-11-15T07:11:55Z,Telemetry_ETH_RSI_EMACROSS,5760,5.44,1059.4,28.81,18.07,33.80,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,bed25e72,dev,development +2025-11-15T07:22:05Z,Telemetry_ETH_RSI_EMACROSS,5760,10.71,537.9,28.81,18.06,33.84,0.0,0,0.0,0.0,0.0,0.0,-35450.45,20,-49.76,0.00,49a693b4,dev,development diff --git a/src/Managing.Workers.Tests/performance-benchmarks.csv b/src/Managing.Workers.Tests/performance-benchmarks.csv index e18d13ac..2b1bc897 100644 --- a/src/Managing.Workers.Tests/performance-benchmarks.csv +++ b/src/Managing.Workers.Tests/performance-benchmarks.csv @@ -66,3 +66,4 @@ DateTime,TestName,CandlesCount,ExecutionTimeSeconds,ProcessingRateCandlesPerSec, 2025-11-15T06:46:21Z,Telemetry_ETH_RSI,5760,4.83,1191.0,29.02,20.22,37.20,4105.51,0,0.0,499.39,0.00,0.09,-30689.97,24,-51.70,0.00,e814eb74,dev,development 2025-11-15T06:50:04Z,Telemetry_ETH_RSI,5760,4.47,1286.2,28.81,20.58,34.89,3324.75,0,0.0,965.71,0.00,0.17,-30689.97,24,-51.70,0.00,e814eb74,dev,development 2025-11-15T07:11:55Z,Telemetry_ETH_RSI,5760,3.365,1707.1,29.06,20.43,36.29,2872.29,0,0.0,371.33,0.00,0.06,-30689.97,24,-51.70,0.00,bed25e72,dev,development +2025-11-15T07:22:05Z,Telemetry_ETH_RSI,5760,7.49,766.2,28.80,20.86,34.90,5992.19,0,0.0,916.71,0.00,0.16,-30689.97,24,-51.70,0.00,49a693b4,dev,development