managing-apps/assets/BacktestPerformanceOptimizations.md

Backtest Performance Optimizations

This document tracks identified performance optimization opportunities for BacktestExecutor.cs based on analysis of the foreach loop that processes thousands of candles.

Current Performance Baseline

• Processing Rate: ~1,707 candles/sec
• Execution Time: ~3.365 seconds for 5,760 candles
• Memory Peak: ~36.29 MB

Optimization Opportunities

🔴 Priority 1: Reuse HashSet Instead of Recreating (CRITICAL)

Location: BacktestExecutor.cs line 267

Current Code:

var fixedCandles = new HashSet<Candle>(rollingWindowCandles);

Problem: Creates a new HashSet 5,760 times (once per candle iteration). This is extremely expensive in terms of:

• Memory allocations
• GC pressure
• CPU cycles for hash calculations

Solution: Reuse HashSet and update incrementally:

// Initialize before loop
var fixedCandles = new HashSet<Candle>(RollingWindowSize);

// Inside loop (replace lines 255-267):
if (rollingWindowCandles.Count >= RollingWindowSize)
{
    var removedCandle = rollingWindowCandles.Dequeue();
    fixedCandles.Remove(removedCandle);
}
rollingWindowCandles.Enqueue(candle);
fixedCandles.Add(candle);
// fixedCandles is now up-to-date, no need to recreate

Expected Impact: 20-30% performance improvement

---

🟠 Priority 2: Optimize Wallet Balance Tracking

Location: BacktestExecutor.cs line 283

Current Code:

lastWalletBalance = tradingBot.WalletBalances.Values.LastOrDefault();

Problem: LastOrDefault() on Dictionary.Values is O(n) operation, called every 10 candles.

Solution: Track balance directly or use more efficient structure:

// Option 1: Cache last balance when wallet updates
// Option 2: Use SortedDictionary if order matters
// Option 3: Maintain separate variable that updates when wallet changes

Expected Impact: 2-5% performance improvement

---

🟡 Priority 3: Optimize TradingBox.GetSignal Input

Location: TradingBox.cs line 130

Current Code:

var limitedCandles = newCandles.ToList(); // Converts HashSet to List

Problem: Converts HashSet to List every time GetSignal is called.

Solution:

• Modify TradingBox.GetSignal to accept IEnumerable<Candle> or List<Candle>
• Pass List directly from rolling window instead of HashSet

Expected Impact: 1-3% performance improvement

---

🟢 Priority 4: Cache Progress Percentage Calculation

Location: BacktestExecutor.cs line 297

Current Code:

var currentPercentage = (currentCandle * 100) / totalCandles;

Problem: Integer division recalculated every iteration (minor but can be optimized).

Solution:

// Before loop
const double percentageMultiplier = 100.0 / totalCandles;

// Inside loop
var currentPercentage = (int)(currentCandle * percentageMultiplier);

Expected Impact: <1% performance improvement (minor optimization)

---

🟢 Priority 5: Use Stopwatch for Time Checks

Location: BacktestExecutor.cs line 298

Current Code:

var timeSinceLastUpdate = (DateTime.UtcNow - lastProgressUpdate).TotalMilliseconds;

Problem: DateTime.UtcNow is relatively expensive when called frequently.

Solution: Use Stopwatch for timing:

var progressStopwatch = Stopwatch.StartNew();
// Then check: progressStopwatch.ElapsedMilliseconds >= progressUpdateIntervalMs

Expected Impact: <1% performance improvement (minor optimization)

---

Future Considerations

Batching Candle Processing

If business logic allows, process multiple candles before updating signals to reduce UpdateSignals() call frequency. Requires careful validation.

Object Pooling

Reuse List/HashSet instances if possible to reduce GC pressure. May require careful state management.

Parallel Processing

If signals are independent, consider parallel indicator calculations. Requires careful validation to ensure business logic integrity.

Implementation Checklist

• Priority 1: Reuse HashSet instead of recreating
• Priority 2: Optimize wallet balance tracking
• Priority 3: Optimize TradingBox.GetSignal input
• Priority 4: Cache progress percentage calculation
• Priority 5: Use Stopwatch for time checks
• Run benchmark-backtest-performance.sh to validate improvements
• Ensure business logic validation passes (Final PnL matches baseline)

Expected Total Impact

Combined Expected Improvement: 25-40% faster execution

Target Performance:

• Processing Rate: ~2,100-2,400 candles/sec (up from ~1,707)
• Execution Time: ~2.0-2.5 seconds (down from ~3.365)
• Memory: Similar or slightly reduced

Notes

• Always validate business logic after optimizations
• Run benchmarks multiple times to account for system variance
• Monitor memory usage to ensure optimizations don't increase GC pressure
• Priority 1 (HashSet reuse) should provide the largest performance gain