#!/bin/bash # Benchmark Backtest Performance Script # This script runs backtest performance tests and records results in CSV set -e # Exit on any error echo "🚀 Running backtest performance benchmark..." # Colors for output RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' BLUE='\033[0;34m' NC='\033[0m' # No Color # Function to extract value from test output using regex extract_value() { local pattern="$1" local text="$2" echo "$text" | grep -o "$pattern" | head -1 | sed 's/.*: //' | sed 's/[^0-9.]*$//' | tr -d ',' } # Get current timestamp TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") # Get git information COMMIT_HASH=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") BRANCH_NAME=$(git branch --show-current 2>/dev/null || echo "unknown") ENVIRONMENT="development" # Run the main performance test and capture output echo "📊 Running main performance test..." TEST_OUTPUT=$(dotnet test src/Managing.Workers.Tests/Managing.Workers.Tests.csproj \ --filter "ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry" \ --verbosity minimal \ --logger "console;verbosity=detailed" 2>&1) # Check if test passed if echo "$TEST_OUTPUT" | grep -q "Passed.*1"; then echo -e "${GREEN}✅ Performance test passed!${NC}" else echo -e "${RED}❌ Performance test failed!${NC}" echo "$TEST_OUTPUT" exit 1 fi # Run business logic validation tests echo "📊 Running business logic validation tests..." VALIDATION_OUTPUT=$(dotnet test src/Managing.Workers.Tests/Managing.Workers.Tests.csproj \ --filter "ExecuteBacktest_With_ETH_FifteenMinutes_Data_Should_Return_LightBacktest|ExecuteBacktest_With_ETH_FifteenMinutes_Data_Second_File_Should_Return_LightBacktest" \ --verbosity minimal \ --logger "console;verbosity=detailed" 2>&1) # Check if validation tests passed if echo "$VALIDATION_OUTPUT" | grep -q "Passed.*2"; then echo -e "${GREEN}✅ Business logic validation tests passed!${NC}" else echo -e "${RED}❌ Business logic validation tests failed!${NC}" echo "$VALIDATION_OUTPUT" exit 1 fi # Extract performance metrics from the output - use more robust parsing CANDLES_COUNT=$(echo "$TEST_OUTPUT" | grep "📈 Total Candles Processed:" | sed 's/.*: //' | sed 's/[^0-9]//g' | xargs) EXECUTION_TIME=$(echo "$TEST_OUTPUT" | grep "âąī¸ Total Execution Time:" | sed 's/.*: //' | sed 's/s//' | sed 's/,/./g' | awk '{print $NF}' | xargs | awk -F' ' '{if (NF==2) print ($1+$2)/2; else print $1}') PROCESSING_RATE=$(echo "$TEST_OUTPUT" | grep "🚀 Processing Rate:" | sed 's/.*: //' | sed 's/ candles\/sec//' | sed 's/,/./g' | xargs) # Extract memory metrics MEMORY_LINE=$(echo "$TEST_OUTPUT" | grep "💾 Memory Usage:") MEMORY_START=$(echo "$MEMORY_LINE" | sed 's/.*Start=//' | sed 's/MB.*//' | xargs) MEMORY_END=$(echo "$MEMORY_LINE" | sed 's/.*End=//' | sed 's/MB.*//' | xargs) MEMORY_PEAK=$(echo "$MEMORY_LINE" | sed 's/.*Peak=//' | sed 's/MB.*//' | xargs) # Extract signal update metrics SIGNAL_LINE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Signal Updates:") SIGNAL_UPDATES=$(echo "$SIGNAL_LINE" | sed 's/.*Signal Updates: //' | sed 's/ms.*//' | sed 's/,/./g' | xargs) SIGNAL_SKIPPED=$(echo "$SIGNAL_LINE" | grep -o "[0-9,]* skipped" | sed 's/ skipped//' | tr -d ',' | xargs) SIGNAL_EFFICIENCY=$(echo "$SIGNAL_LINE" | grep -o "[0-9.]*% efficiency" | sed 's/% efficiency//' | xargs) # Extract backtest steps BACKTEST_LINE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Backtest Steps:") BACKTEST_STEPS=$(echo "$BACKTEST_LINE" | sed 's/.*Backtest Steps: //' | sed 's/ms.*//' | sed 's/,/./g' | xargs) # Extract timing metrics AVG_SIGNAL_UPDATE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Average Signal Update:" | sed 's/.*Average Signal Update: //' | sed 's/ms.*//' | sed 's/,/./g' | xargs) AVG_BACKTEST_STEP=$(echo "$TEST_OUTPUT" | grep "â€ĸ Average Backtest Step:" | sed 's/.*Average Backtest Step: //' | sed 's/ms.*//' | sed 's/,/./g' | xargs) # Extract trading results FINAL_PNL=$(echo "$TEST_OUTPUT" | grep "â€ĸ Final PnL:" | sed 's/.*Final PnL: //' | sed 's/,/./g' | xargs) WIN_RATE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Win Rate:" | sed 's/.*Win Rate: //' | sed 's/%//' | xargs) GROWTH_PERCENTAGE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Growth:" | sed 's/.*Growth: //' | sed 's/%//' | sed 's/,/./g' | xargs) SCORE=$(echo "$TEST_OUTPUT" | grep "â€ĸ Score:" | sed 's/.*Score: //' | sed 's/[^0-9.-]//g' | xargs) # Set defaults for missing or malformed values CANDLES_COUNT=${CANDLES_COUNT:-0} EXECUTION_TIME=${EXECUTION_TIME:-0.0} PROCESSING_RATE=${PROCESSING_RATE:-0.0} MEMORY_START=${MEMORY_START:-0.0} MEMORY_END=${MEMORY_END:-0.0} MEMORY_PEAK=${MEMORY_PEAK:-0.0} SIGNAL_UPDATES=${SIGNAL_UPDATES:-0.0} SIGNAL_SKIPPED=${SIGNAL_SKIPPED:-0} SIGNAL_EFFICIENCY=${SIGNAL_EFFICIENCY:-0.0} BACKTEST_STEPS=${BACKTEST_STEPS:-0.0} AVG_SIGNAL_UPDATE=${AVG_SIGNAL_UPDATE:-0.0} AVG_BACKTEST_STEP=${AVG_BACKTEST_STEP:-0.0} FINAL_PNL=${FINAL_PNL:-0.00} WIN_RATE=${WIN_RATE:-0} GROWTH_PERCENTAGE=${GROWTH_PERCENTAGE:-0.00} SCORE=${SCORE:-0.00} # Fix malformed values SCORE=$(echo "$SCORE" | sed 's/^0*$/0.00/' | xargs) # Business Logic Validation: Check Final PnL against first run baseline FIRST_RUN_FINAL_PNL=$(head -2 src/Managing.Workers.Tests/performance-benchmarks.csv 2>/dev/null | tail -1 | cut -d',' -f15 | xargs) if [ -n "$FIRST_RUN_FINAL_PNL" ] && [ "$FIRST_RUN_FINAL_PNL" != "FinalPnL" ]; then # Compare against the first run in the file (the baseline) DIFF=$(echo "scale=2; $FINAL_PNL - $FIRST_RUN_FINAL_PNL" | bc -l 2>/dev/null || echo "0") ABS_DIFF=$(echo "scale=2; if ($DIFF < 0) -$DIFF else $DIFF" | bc -l 2>/dev/null || echo "0") if (( $(echo "$ABS_DIFF > 0.01" | bc -l 2>/dev/null || echo "0") )); then echo -e "${RED}❌ BUSINESS LOGIC WARNING: Final PnL differs from baseline!${NC}" echo " Baseline (first run): $FIRST_RUN_FINAL_PNL" echo " Current: $FINAL_PNL" echo " Difference: $DIFF" echo -e "${YELLOW}âš ī¸ This may indicate that changes broke business logic!${NC}" echo -e "${YELLOW} Please verify that optimizations didn't change backtest behavior.${NC}" else echo -e "${GREEN}✅ Business Logic OK: Final PnL matches baseline (Âą$ABS_DIFF)${NC}" fi else # If no baseline exists, establish one echo -e "${BLUE}â„šī¸ Establishing new baseline - this is the first run${NC}" echo -e "${GREEN}✅ First run completed successfully${NC}" fi # Create CSV row CSV_ROW="$TIMESTAMP,ExecuteBacktest_With_Large_Dataset_Should_Show_Performance_Telemetry,$CANDLES_COUNT,$EXECUTION_TIME,$PROCESSING_RATE,$MEMORY_START,$MEMORY_END,$MEMORY_PEAK,$SIGNAL_UPDATES,$SIGNAL_SKIPPED,$SIGNAL_EFFICIENCY,$BACKTEST_STEPS,$AVG_SIGNAL_UPDATE,$AVG_BACKTEST_STEP,$FINAL_PNL,$WIN_RATE,$GROWTH_PERCENTAGE,$SCORE,$COMMIT_HASH,$BRANCH_NAME,$ENVIRONMENT" # Append to CSV file echo "$CSV_ROW" >> "src/Managing.Workers.Tests/performance-benchmarks.csv" # Display results echo -e "${BLUE}📊 Benchmark Results:${NC}" echo " â€ĸ Processing Rate: $PROCESSING_RATE candles/sec" echo " â€ĸ Execution Time: $EXECUTION_TIME seconds" echo " â€ĸ Memory Peak: $MEMORY_PEAK MB" echo " â€ĸ Signal Efficiency: $SIGNAL_EFFICIENCY%" echo " â€ĸ Candles Processed: $CANDLES_COUNT" echo " â€ĸ Score: $SCORE" echo -e "${GREEN}✅ Benchmark data recorded successfully!${NC}"