docs: enhance benchmark command with business logic validation tests

- Add 2 ETH-based validation tests to benchmark script
- Validates ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest
- Validates ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest
- Ensures performance optimizations don't break trading logic
- Update command documentation with comprehensive validation details
- All 3 validation levels must pass for benchmark success
This commit is contained in:
2025-11-11 12:32:56 +07:00
parent 578709d9b7
commit fc036bb7de
4 changed files with 65 additions and 33 deletions

View File

@@ -18,11 +18,14 @@ Or run the script directly:
## What it does
1. Runs the performance telemetry test (`ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry`)
2. **Validates Business Logic**: Compares Final PnL with the first run of the file to ensure optimizations don't break behavior
3. Extracts performance metrics from the test output
4. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv`
5. **Never commits changes automatically**
1. Runs the **main performance telemetry test** (`ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry`)
2. Runs **two business logic validation tests**:
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest`
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest`
3. **Validates Business Logic**: Compares Final PnL with the first run baseline to ensure optimizations don't break behavior
4. Extracts performance metrics from the test output
5. Appends a new row to `src/Managing.Workers.Tests/performance-benchmarks.csv`
6. **Never commits changes automatically**
## CSV Format
@@ -58,28 +61,49 @@ The command uses regex patterns to extract metrics from the test console output
```
🚀 Running backtest performance benchmark...
📊 Test Results:
• Processing Rate: 2686.2 candles/sec
• Execution Time: 2.115 seconds
• Memory Peak: 23.91 MB
• Signal Efficiency: 33.1%
📊 Running main performance test...
✅ Performance test passed!
📊 Running business logic validation tests...
✅ Business logic validation tests passed!
✅ Business Logic OK: Final PnL matches baseline (±0)
📊 Benchmark Results:
• Processing Rate: 5688.8 candles/sec
• Execution Time: 1.005 seconds
• Memory Peak: 24.66 MB
• Signal Efficiency: 33.2%
• Candles Processed: 5760
• Score: 0.00
• Score: 6015
✅ Business Logic OK: Final PnL consistent (±0.00)
✅ Benchmark data recorded in performance-benchmarks.csv
✅ Benchmark data recorded successfully!
```
### Business Logic Validation
The benchmark includes **business logic validation** to ensure performance optimizations don't break backtest behavior:
The benchmark includes **comprehensive business logic validation** on three levels:
- **✅ Consistent**: Final PnL matches previous run (±0.01 tolerance)
- **✅ Baseline OK**: First run validates against expected baseline (24560.79)
- **⚠️ Warning**: Large PnL differences may indicate broken business logic
- ** First Run**: Validates against established baseline
#### 1. **Dedicated ETH Backtest Tests** (2 tests)
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest`
- Tests backtest with ETH 15-minute data
- Validates specific trading scenarios and positions
- Ensures indicator calculations are correct
- `ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest`
- Tests with a different ETH dataset
- Validates consistency across different market data
- Confirms trading logic works reliably
**Expected Baseline**: Final PnL should be **24560.79** for correct business logic.
#### 2. **Large Dataset Telemetry Test** (1 test)
- `ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry`
- Validates performance metrics extraction
- Confirms signal updates and backtest steps
- Ensures telemetry data is accurate
#### 3. **PnL Baseline Comparison**
- **Consistent**: Final PnL matches first run (±0.01 tolerance)
- **Baseline OK**: Expected baseline is **24560.79**
- **⚠️ Warning**: Large differences indicate broken business logic
**All three validation levels must pass for the benchmark to succeed!**
**This prevents performance improvements from accidentally changing trading outcomes!**