Add precalculated signals list + multi scenario test

This commit is contained in:
2025-11-11 14:05:09 +07:00
parent e810ab60ce
commit 903413692c
7 changed files with 359 additions and 17 deletions

View File

@@ -19,13 +19,15 @@ Or run the script directly:
## What it does
1. Runs the **main performance telemetry test** (`ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry`)
2. Runs **two business logic validation tests**:
2. Runs the **two-scenarios performance test** (`ExecuteBacktest_With_Two_Scenarios_Should_Show_Performance_Telemetry`) - tests pre-calculated signals with 2 indicators and validates business logic consistency
3. Runs **two business logic validation tests**:
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest`
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest`
3. **Validates Business Logic**: Compares Final PnL with the first run baseline to ensure optimizations don't break behavior
4. Extracts performance metrics from the test output
5. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv`
6. **Never commits changes automatically**
4. **Validates Business Logic**: Compares Final PnL with the first run baseline to ensure optimizations don't break behavior
5. Extracts performance metrics from the test output
6. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv` (main test)
7. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv` (two-scenarios test)
8. **Never commits changes automatically**
## CSV Format
@@ -90,6 +92,61 @@ The benchmark shows significant variance in execution times (e.g., 0.915s to 1.4
**Takeaway**: Always validate business logic after performance optimizations, even if they seem unrelated.
### ❌ **Pitfall: RSI Indicator Optimizations**
**What happened**: Attempting to optimize the RSI divergence indicator decreased performance by ~50%!
- Changed from **6446 candles/sec** back to **2797 candles/sec**
- **Complex LINQ optimizations** like `OrderByDescending().Take()` were slower than simple `TakeLast()`
- **Creating HashSet<Candle>** objects in signal generation added overhead
- **Caching calculations** added complexity without benefit
**Takeaway**: Not all code is worth optimizing. Some algorithms are already efficient enough, and micro-optimizations can hurt more than help. Always measure the impact before committing complex changes.
## Performance Bottleneck Analysis (Latest Findings)
Recent performance logging revealed the **true bottleneck** in backtest execution:
### 📊 **Backtest Timing Breakdown**
- **Total execution time**: ~1.4-1.6 seconds for 5760 candles
- **TradingBotBase.Run() calls**: 5,760 total (~87ms combined, 0.015ms average per call)
- **Unaccounted time**: ~1.3-1.5 seconds (94% of total execution time!)
### 🎯 **Identified Bottlenecks** (in order of impact)
1. **TradingBox.GetSignal()** - Indicator calculations (called ~1,932 times, ~0.99ms per call average)
2. **BacktestExecutor loop overhead** - HashSet operations, memory allocations
3. **Signal update frequency** - Even with 66.5% efficiency, remaining updates are expensive
4. **Memory management** - GC pressure from frequent allocations
### 🚀 **Next Optimization Targets**
1. **Optimize indicator calculations** - RSI divergence processing is the biggest bottleneck
2. **Reduce HashSet allocations** - Pre-allocate or reuse collections
3. **Optimize signal update logic** - Further reduce unnecessary updates
4. **Memory pooling** - Reuse objects to reduce GC pressure
## Major Optimization Success: Pre-Calculated Signals
### ✅ **Optimization: Pre-Calculated Signals**
**What was implemented**: Pre-calculated all signals once upfront instead of calling `TradingBox.GetSignal()` ~1,932 times during backtest execution.
**Technical Details**:
- Added `PreCalculateAllSignals()` method in `BacktestExecutor.cs`
- Pre-calculates signals for all candles using rolling window logic
- Modified `TradingBotBase.UpdateSignals()` to support pre-calculated signal lookup
- Updated backtest loop to use O(1) signal lookups instead of expensive calculations
**Performance Impact** (Average of 3 runs):
- **Processing Rate**: 2,800 → **~5,800 candles/sec** (2.1x improvement!)
- **Execution Time**: 1.4-1.6s → **~1.0s** (35-50% faster!)
- **Signal Update Time**: ~1,417ms → **Eliminated** (no more repeated calculations)
- **Consistent Results**: 5,217 - 6,871 candles/sec range (expected system variance)
**Business Logic Validation**:
- ✅ All validation tests passed
- ✅ Final PnL matches baseline (±0)
- ✅ Two-scenarios test includes baseline assertions for consistency over time (with proper win rate percentage handling)
- ✅ Live trading functionality preserved (no changes to live trading code)
**Takeaway**: The biggest performance gains come from eliminating redundant calculations. Pre-calculating expensive operations once upfront is far more effective than micro-optimizations.
## Safe Optimization Strategies
Based on lessons learned, safe optimizations include:
@@ -99,6 +156,7 @@ Based on lessons learned, safe optimizations include:
3. **Avoid state changes**: Don't modify the order or timing of business logic operations
4. **Skip intermediate calculations**: Reduce logging and telemetry overhead
5. **Always validate**: Run full benchmark suite after every change
6. **Profile before optimizing**: Use targeted logging to identify real bottlenecks
## Example Output
@@ -153,6 +211,7 @@ The benchmark includes **comprehensive business logic validation** on three leve
## Files Modified
- `src/Managing.Workers.Tests/performance-benchmarks.csv` - **Modified** (new benchmark row added)
- `src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv` - **Modified** (new two-scenarios benchmark row added)
**Note**: Changes are **not committed automatically**. Review the results and commit manually if satisfied.