Improve backtest run
This commit is contained in:
@@ -55,7 +55,50 @@ The CSV file contains clean numeric values for all telemetry metrics:
|
||||
|
||||
## Implementation Details
|
||||
|
||||
The command uses regex patterns to extract metrics from the test console output and formats them into CSV rows. It detects the current git branch and commit hash for tracking purposes but **never commits changes automatically**.
|
||||
The command uses regex patterns to extract metrics from the test console output and formats them into CSV rows. It detects the current git branch and commit hash for tracking purposes but **never commits and push changes automatically**.
|
||||
|
||||
## Performance Variance
|
||||
|
||||
The benchmark shows significant variance in execution times (e.g., 0.915s to 1.445s for the same code), which is expected:
|
||||
|
||||
- **System load affects results**: Background processes and system activity impact measurements
|
||||
- **GC pauses occur unpredictably**: Garbage collection can cause sudden performance drops
|
||||
- **Multiple runs recommended**: Run benchmarks 3-5 times and compare median values for reliable measurements
|
||||
- **Time of day matters**: System resources vary based on other running processes
|
||||
|
||||
**Best Practice**: When optimizing, compare the median of multiple runs before and after changes to account for variance.
|
||||
|
||||
## Lessons Learned from Optimization Attempts
|
||||
|
||||
### ❌ **Pitfall: Rolling Window Changes**
|
||||
**What happened**: Changing the order of HashSet operations in the rolling window broke business logic.
|
||||
- Changed PnL from `22032.78` to `24322.17`
|
||||
- The order of `Add()` and `Remove()` operations on the HashSet affected which candles were available during signal updates
|
||||
- **Takeaway**: Even "performance-only" changes can alter trading logic if they affect the state during calculations
|
||||
|
||||
### ❌ **Pitfall: LINQ Caching**
|
||||
**What happened**: Caching `candles.First()` and `candles.Last()` caused floating-point precision issues.
|
||||
- SharpeRatio changed from `-0.01779902594116203` to `-0.017920689062300373`
|
||||
- Using cached values vs. repeated LINQ calls introduced subtle precision differences
|
||||
- **Takeaway**: Financial calculations are sensitive to floating-point precision; avoid unnecessary intermediate variables
|
||||
|
||||
### ✅ **Success: Business Logic Validation**
|
||||
**What worked**: The benchmark's comprehensive validation caught breaking changes immediately:
|
||||
1. **PnL baseline comparison** detected the rolling window issue
|
||||
2. **Dedicated ETH tests** caught the SharpeRatio precision problem
|
||||
3. **Immediate feedback** prevented bad optimizations from being committed
|
||||
|
||||
**Takeaway**: Always validate business logic after performance optimizations, even if they seem unrelated.
|
||||
|
||||
## Safe Optimization Strategies
|
||||
|
||||
Based on lessons learned, safe optimizations include:
|
||||
|
||||
1. **Reduce system call frequency**: Cache `GC.GetTotalMemory()` checks (e.g., every 100 candles)
|
||||
2. **Fix bugs**: Remove duplicate counters and redundant operations
|
||||
3. **Avoid state changes**: Don't modify the order or timing of business logic operations
|
||||
4. **Skip intermediate calculations**: Reduce logging and telemetry overhead
|
||||
5. **Always validate**: Run full benchmark suite after every change
|
||||
|
||||
## Example Output
|
||||
|
||||
|
||||
Reference in New Issue
Block a user