Oda/managing-apps
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add monitoring on queries with sentry alert + Fix check position list in db for backtest

Browse Source
This commit is contained in:
cryptooda
2025-10-10 00:15:02 +07:00
parent ffb98fe359
commit e4c2f8b7a5
24 changed files with 3340 additions and 179 deletions
Download Patch File Download Diff File Expand all files Collapse all files

336
SQL_MONITORING_README.md Normal file
View File

@@ -0,0 +1,336 @@
# SQL Query Monitoring and Loop Detection System
## Overview
This comprehensive SQL monitoring system has been implemented to identify and resolve the SQL script loop issue that was causing DDOS-like behavior on your server. The system provides detailed logging, performance monitoring, and automatic loop detection to help identify the root cause of problematic database operations.
## Features
### 🔍 **Comprehensive SQL Query Logging**
- **Detailed Query Tracking**: Every SQL query is logged with timing, parameters, and execution context
- **Performance Metrics**: Automatic tracking of query execution times, row counts, and resource usage
- **Connection State Monitoring**: Tracks database connection open/close operations with timing
- **Error Logging**: Comprehensive error logging with stack traces and context information
### 🚨 **Automatic Loop Detection**
- **Pattern Recognition**: Identifies repeated query patterns that may indicate infinite loops
- **Frequency Analysis**: Monitors query execution frequency and detects abnormally high rates
- **Performance Thresholds**: Automatically flags slow queries and high-frequency operations
- **Real-time Alerts**: Immediate notification when potential loops are detected
### 📊 **Performance Monitoring**
- **Query Execution Statistics**: Tracks execution counts, average times, and performance trends
- **Resource Usage Monitoring**: Monitors memory, CPU, and I/O usage during database operations
- **Connection Pool Monitoring**: Tracks database connection pool health and usage
- **Transaction Monitoring**: Monitors transaction duration and rollback rates
### 🎯 **Smart Alerting System**
- **Configurable Thresholds**: Customizable thresholds for slow queries, high frequency, and error rates
- **Multi-level Alerts**: Different alert levels (Info, Warning, Error, Critical) based on severity
- **Contextual Information**: Alerts include repository name, method name, and query patterns
- **Automatic Escalation**: Critical issues are automatically escalated with detailed diagnostics
## Components
### 1. SqlQueryLogger
**Location**: `src/Managing.Infrastructure.Database/PostgreSql/SqlQueryLogger.cs`
Provides comprehensive logging for individual database operations:
- Operation start/completion logging
- Query execution timing and parameters
- Connection state changes
- Error handling and exception logging
- Performance issue detection
### 2. SqlLoopDetectionService
**Location**: `src/Managing.Infrastructure.Database/PostgreSql/SqlLoopDetectionService.cs`
Advanced loop detection and performance monitoring:
- Real-time query pattern analysis
- Execution frequency tracking
- Performance threshold monitoring
- Automatic cleanup of old tracking data
- Configurable detection rules
### 3. BaseRepositoryWithLogging
**Location**: `src/Managing.Infrastructure.Database/PostgreSql/BaseRepositoryWithLogging.cs`
Base class for repositories with integrated monitoring:
- Automatic query execution tracking
- Performance monitoring for all database operations
- Error handling and logging
- Loop detection integration
### 4. Enhanced ManagingDbContext
**Location**: `src/Managing.Infrastructure.Database/PostgreSql/ManagingDbContext.cs`
Extended DbContext with monitoring capabilities:
- Query execution tracking
- Performance metrics collection
- Loop detection integration
- Statistics and health monitoring
### 5. SqlMonitoringController
**Location**: `src/Managing.Api/Controllers/SqlMonitoringController.cs`
REST API endpoints for monitoring and management:
- Real-time query statistics
- Alert management
- Performance metrics
- Health monitoring
- Configuration management
## API Endpoints
### Get Query Statistics
```http
GET /api/SqlMonitoring/statistics
```
Returns comprehensive query execution statistics including:
- Loop detection statistics
- Context execution counts
- Active query patterns
- Performance metrics
### Get Alerts
```http
GET /api/SqlMonitoring/alerts
```
Returns current alerts and potential issues:
- High frequency queries
- Slow query patterns
- Performance issues
- Loop detection alerts
### Clear Tracking Data
```http
POST /api/SqlMonitoring/clear-tracking
```
Clears all tracking data and resets monitoring counters.
### Get Query Details
```http
GET /api/SqlMonitoring/query-details/{repositoryName}/{methodName}
```
Returns detailed information about specific query patterns.
### Get Monitoring Health
```http
GET /api/SqlMonitoring/health
```
Returns overall monitoring system health status.
## Configuration
### SqlMonitoringSettings
**Location**: `src/Managing.Infrastructure.Database/PostgreSql/SqlMonitoringSettings.cs`
Comprehensive configuration options:
- **TrackingWindow**: Time window for query tracking (default: 5 minutes)
- **MaxExecutionsPerWindow**: Maximum executions per window (default: 10)
- **SlowQueryThresholdMs**: Slow query threshold (default: 1000ms)
- **HighFrequencyThreshold**: High frequency threshold (default: 20 executions/minute)
- **EnableDetailedLogging**: Enable detailed SQL logging (default: true)
- **EnableLoopDetection**: Enable loop detection (default: true)
- **EnablePerformanceMonitoring**: Enable performance monitoring (default: true)
## Usage Examples
### 1. Using Enhanced Repository
```csharp
public class MyRepository : BaseRepositoryWithLogging, IMyRepository
{
public MyRepository(ManagingDbContext context, ILogger<MyRepository> logger, SqlLoopDetectionService loopDetectionService)
: base(context, logger, loopDetectionService)
{
}
public async Task<User> GetUserAsync(string name)
{
return await ExecuteWithLoggingAsync(async () =>
{
// Your database operation here
return await _context.Users.FirstOrDefaultAsync(u => u.Name == name);
}, nameof(GetUserAsync), ("name", name));
}
}
```
### 2. Manual Query Tracking
```csharp
// Track a specific query execution
_context.TrackQueryExecution("GetUserByName", TimeSpan.FromMilliseconds(150), "UserRepository", "GetUserAsync");
```
### 3. Monitoring API Usage
```bash
# Get current statistics
curl -X GET "https://your-api/api/SqlMonitoring/statistics"
# Get alerts
curl -X GET "https://your-api/api/SqlMonitoring/alerts"
# Clear tracking data
curl -X POST "https://your-api/api/SqlMonitoring/clear-tracking"
```
## Logging Output Examples
### Query Execution Log
```
[SQL-OP-START] a1b2c3d4 | PostgreSqlUserRepository.GetUserByNameAsync | Started at 14:30:15.123
[SQL-CONNECTION] a1b2c3d4 | PostgreSqlUserRepository.GetUserByNameAsync | Connection OPENED (took 5ms)
[SQL-QUERY] a1b2c3d4 | PostgreSqlUserRepository.GetUserByNameAsync | Executed in 25ms | Rows: 1
[SQL-CONNECTION] a1b2c3d4 | PostgreSqlUserRepository.GetUserByNameAsync | Connection CLOSED (took 2ms)
[SQL-OP-COMPLETE] a1b2c3d4 | PostgreSqlUserRepository.GetUserByNameAsync | Completed in 32ms | Queries: 1 | Result: User
```
### Loop Detection Alert
```
[SQL-LOOP-DETECTED] e5f6g7h8 | PostgreSqlTradingRepository.GetPositionsAsync | Pattern 'GetPositionsAsync()' executed 15 times | Possible infinite loop!
[SQL-LOOP-ALERT] Potential infinite loop detected in PostgreSqlTradingRepository.GetPositionsAsync with pattern 'GetPositionsAsync()'
```
### Performance Warning
```
[SQL-PERFORMANCE] PostgreSqlTradingRepository | GetPositionsAsync took 2500ms (threshold: 1000ms)
[SQL-QUERY-DETAILS] i9j0k1l2 | Query: SELECT * FROM Positions WHERE Status = @status | Parameters: {"status":"Active"}
```
## Troubleshooting
### Common Issues and Solutions
#### 1. High Query Frequency
**Symptoms**: Multiple queries executing rapidly
**Detection**: `[SQL-LOOP-DETECTED]` logs with high execution counts
**Solution**:
- Check for recursive method calls
- Verify loop conditions in business logic
- Review async/await patterns
#### 2. Slow Query Performance
**Symptoms**: Queries taking longer than expected
**Detection**: `[SQL-PERFORMANCE]` warnings
**Solution**:
- Review query execution plans
- Check database indexes
- Optimize query parameters
#### 3. Connection Issues
**Symptoms**: Connection timeouts or pool exhaustion
**Detection**: `[SQL-CONNECTION]` error logs
**Solution**:
- Review connection management
- Check connection pool settings
- Verify proper connection disposal
#### 4. Memory Issues
**Symptoms**: High memory usage during database operations
**Detection**: Memory monitoring alerts
**Solution**:
- Review query result set sizes
- Implement pagination
- Check for memory leaks in entity tracking
## Integration Steps
### 1. Update Existing Repositories
Replace existing repository implementations with the enhanced base class:
```csharp
// Before
public class MyRepository : IMyRepository
{
private readonly ManagingDbContext _context;
// ...
}
// After
public class MyRepository : BaseRepositoryWithLogging, IMyRepository
{
public MyRepository(ManagingDbContext context, ILogger<MyRepository> logger, SqlLoopDetectionService loopDetectionService)
: base(context, logger, loopDetectionService)
{
}
// ...
}
```
### 2. Update Dependency Injection
The services are automatically registered in `Program.cs`:
- `SqlLoopDetectionService` as Singleton
- Enhanced `ManagingDbContext` with monitoring
- All repositories with logging capabilities
### 3. Configure Monitoring Settings
Add configuration to `appsettings.json`:
```json
{
"SqlMonitoring": {
"TrackingWindow": "00:05:00",
"MaxExecutionsPerWindow": 10,
"SlowQueryThresholdMs": 1000,
"HighFrequencyThreshold": 20,
"EnableDetailedLogging": true,
"EnableLoopDetection": true,
"EnablePerformanceMonitoring": true
}
}
```
## Monitoring Dashboard
### Key Metrics to Monitor
1. **Query Execution Count**: Track total queries per minute
2. **Average Execution Time**: Monitor query performance trends
3. **Error Rate**: Track database error frequency
4. **Connection Pool Usage**: Monitor connection health
5. **Loop Detection Alerts**: Immediate notification of potential issues
### Alert Thresholds
- **Critical**: >50 queries/minute, >5 second execution time
- **Warning**: >20 queries/minute, >1 second execution time
- **Info**: Normal operation metrics
## Best Practices
### 1. Repository Design
- Always inherit from `BaseRepositoryWithLogging`
- Use `ExecuteWithLoggingAsync` for all database operations
- Include meaningful parameter names in logging calls
- Handle exceptions properly with logging
### 2. Performance Optimization
- Monitor slow queries regularly
- Implement proper indexing strategies
- Use pagination for large result sets
- Avoid N+1 query problems
### 3. Error Handling
- Log all database errors with context
- Implement proper retry mechanisms
- Use circuit breaker patterns for external dependencies
- Monitor error rates and trends
### 4. Security Considerations
- Avoid logging sensitive data in query parameters
- Use parameterized queries to prevent SQL injection
- Implement proper access controls for monitoring endpoints
- Regular security audits of database operations
## Conclusion
This comprehensive SQL monitoring system provides the tools needed to identify and resolve the SQL script loop issue. The system offers:
- **Real-time monitoring** of all database operations
- **Automatic loop detection** with configurable thresholds
- **Performance tracking** with detailed metrics
- **Comprehensive logging** for debugging and analysis
- **REST API endpoints** for monitoring and management
- **Configurable settings** for different environments
The system is designed to be non-intrusive while providing maximum visibility into database operations, helping you quickly identify and resolve performance issues and potential infinite loops.