Add precalculated signals list + multi scenario test
This commit is contained in:
@@ -19,13 +19,15 @@ Or run the script directly:
|
||||
## What it does
|
||||
|
||||
1. Runs the **main performance telemetry test** (`ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry`)
|
||||
2. Runs **two business logic validation tests**:
|
||||
2. Runs the **two-scenarios performance test** (`ExecuteBacktest_With_Two_Scenarios_Should_Show_Performance_Telemetry`) - tests pre-calculated signals with 2 indicators and validates business logic consistency
|
||||
3. Runs **two business logic validation tests**:
|
||||
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest`
|
||||
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest`
|
||||
3. **Validates Business Logic**: Compares Final PnL with the first run baseline to ensure optimizations don't break behavior
|
||||
4. Extracts performance metrics from the test output
|
||||
5. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv`
|
||||
6. **Never commits changes automatically**
|
||||
4. **Validates Business Logic**: Compares Final PnL with the first run baseline to ensure optimizations don't break behavior
|
||||
5. Extracts performance metrics from the test output
|
||||
6. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv` (main test)
|
||||
7. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv` (two-scenarios test)
|
||||
8. **Never commits changes automatically**
|
||||
|
||||
## CSV Format
|
||||
|
||||
@@ -90,6 +92,61 @@ The benchmark shows significant variance in execution times (e.g., 0.915s to 1.4
|
||||
|
||||
**Takeaway**: Always validate business logic after performance optimizations, even if they seem unrelated.
|
||||
|
||||
### ❌ **Pitfall: RSI Indicator Optimizations**
|
||||
**What happened**: Attempting to optimize the RSI divergence indicator decreased performance by ~50%!
|
||||
- Changed from **6446 candles/sec** back to **2797 candles/sec**
|
||||
- **Complex LINQ optimizations** like `OrderByDescending().Take()` were slower than simple `TakeLast()`
|
||||
- **Creating HashSet<Candle>** objects in signal generation added overhead
|
||||
- **Caching calculations** added complexity without benefit
|
||||
|
||||
**Takeaway**: Not all code is worth optimizing. Some algorithms are already efficient enough, and micro-optimizations can hurt more than help. Always measure the impact before committing complex changes.
|
||||
|
||||
## Performance Bottleneck Analysis (Latest Findings)
|
||||
|
||||
Recent performance logging revealed the **true bottleneck** in backtest execution:
|
||||
|
||||
### 📊 **Backtest Timing Breakdown**
|
||||
- **Total execution time**: ~1.4-1.6 seconds for 5760 candles
|
||||
- **TradingBotBase.Run() calls**: 5,760 total (~87ms combined, 0.015ms average per call)
|
||||
- **Unaccounted time**: ~1.3-1.5 seconds (94% of total execution time!)
|
||||
|
||||
### 🎯 **Identified Bottlenecks** (in order of impact)
|
||||
1. **TradingBox.GetSignal()** - Indicator calculations (called ~1,932 times, ~0.99ms per call average)
|
||||
2. **BacktestExecutor loop overhead** - HashSet operations, memory allocations
|
||||
3. **Signal update frequency** - Even with 66.5% efficiency, remaining updates are expensive
|
||||
4. **Memory management** - GC pressure from frequent allocations
|
||||
|
||||
### 🚀 **Next Optimization Targets**
|
||||
1. **Optimize indicator calculations** - RSI divergence processing is the biggest bottleneck
|
||||
2. **Reduce HashSet allocations** - Pre-allocate or reuse collections
|
||||
3. **Optimize signal update logic** - Further reduce unnecessary updates
|
||||
4. **Memory pooling** - Reuse objects to reduce GC pressure
|
||||
|
||||
## Major Optimization Success: Pre-Calculated Signals
|
||||
|
||||
### ✅ **Optimization: Pre-Calculated Signals**
|
||||
**What was implemented**: Pre-calculated all signals once upfront instead of calling `TradingBox.GetSignal()` ~1,932 times during backtest execution.
|
||||
|
||||
**Technical Details**:
|
||||
- Added `PreCalculateAllSignals()` method in `BacktestExecutor.cs`
|
||||
- Pre-calculates signals for all candles using rolling window logic
|
||||
- Modified `TradingBotBase.UpdateSignals()` to support pre-calculated signal lookup
|
||||
- Updated backtest loop to use O(1) signal lookups instead of expensive calculations
|
||||
|
||||
**Performance Impact** (Average of 3 runs):
|
||||
- **Processing Rate**: 2,800 → **~5,800 candles/sec** (2.1x improvement!)
|
||||
- **Execution Time**: 1.4-1.6s → **~1.0s** (35-50% faster!)
|
||||
- **Signal Update Time**: ~1,417ms → **Eliminated** (no more repeated calculations)
|
||||
- **Consistent Results**: 5,217 - 6,871 candles/sec range (expected system variance)
|
||||
|
||||
**Business Logic Validation**:
|
||||
- ✅ All validation tests passed
|
||||
- ✅ Final PnL matches baseline (±0)
|
||||
- ✅ Two-scenarios test includes baseline assertions for consistency over time (with proper win rate percentage handling)
|
||||
- ✅ Live trading functionality preserved (no changes to live trading code)
|
||||
|
||||
**Takeaway**: The biggest performance gains come from eliminating redundant calculations. Pre-calculating expensive operations once upfront is far more effective than micro-optimizations.
|
||||
|
||||
## Safe Optimization Strategies
|
||||
|
||||
Based on lessons learned, safe optimizations include:
|
||||
@@ -99,6 +156,7 @@ Based on lessons learned, safe optimizations include:
|
||||
3. **Avoid state changes**: Don't modify the order or timing of business logic operations
|
||||
4. **Skip intermediate calculations**: Reduce logging and telemetry overhead
|
||||
5. **Always validate**: Run full benchmark suite after every change
|
||||
6. **Profile before optimizing**: Use targeted logging to identify real bottlenecks
|
||||
|
||||
## Example Output
|
||||
|
||||
@@ -153,6 +211,7 @@ The benchmark includes **comprehensive business logic validation** on three leve
|
||||
## Files Modified
|
||||
|
||||
- `src/Managing.Workers.Tests/performance-benchmarks.csv` - **Modified** (new benchmark row added)
|
||||
- `src/Managing.Workers.Tests/performance-benchmarks-two-scenarios.csv` - **Modified** (new two-scenarios benchmark row added)
|
||||
|
||||
**Note**: Changes are **not committed automatically**. Review the results and commit manually if satisfied.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user