Refactor LlmController and GeminiProvider for improved message handling and redundant tool call detection
- Enhanced LlmController to detect and handle redundant tool calls, ensuring efficient processing and preventing unnecessary requests. - Updated message formatting in GeminiProvider to align with Gemini's expectations, improving the structure of requests sent to the API. - Improved logging in AiChat component to provide better insights into received responses and fallback mechanisms for empty content. - Adjusted handling of final responses in AiChat to ensure meaningful content is displayed, enhancing user experience during interactions.
This commit is contained in:
599
LLM_IMPROVEMENTS_TODO.md
Normal file
599
LLM_IMPROVEMENTS_TODO.md
Normal file
@@ -0,0 +1,599 @@
|
||||
# LLM Controller - Feature Improvements Roadmap
|
||||
|
||||
## 🎯 Quick Wins (1-2 days)
|
||||
|
||||
### ✅ Priority 1: Suggested Follow-up Questions
|
||||
**Status:** Not Started
|
||||
**Effort:** 4-6 hours
|
||||
**Impact:** High
|
||||
|
||||
**Description:**
|
||||
After each response, the LLM suggests 3-5 relevant follow-up questions to guide user exploration.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Update `BuildSystemMessage()` to include follow-up question instruction
|
||||
- [ ] Add `SuggestedQuestions` property to `LlmProgressUpdate` class
|
||||
- [ ] Create `ExtractFollowUpQuestions()` method to parse questions from response
|
||||
- [ ] Update `ChatStreamInternal()` to extract and send suggested questions
|
||||
- [ ] Update frontend to display suggested questions as clickable chips
|
||||
- [ ] Test with various query types (backtest, indicator, general finance)
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
- Frontend components (AiChat.tsx)
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 2: Feedback & Rating System
|
||||
**Status:** Not Started
|
||||
**Effort:** 6-8 hours
|
||||
**Impact:** High (Quality tracking)
|
||||
|
||||
**Description:**
|
||||
Users can rate LLM responses (👍👎) with optional comments to track quality and improve prompts.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `LlmFeedback` domain model (ResponseId, UserId, Rating, Comment, Timestamp)
|
||||
- [ ] Create `ILlmFeedbackRepository` interface
|
||||
- [ ] Implement `LlmFeedbackRepository` with MongoDB
|
||||
- [ ] Add `ResponseId` property to `LlmChatResponse`
|
||||
- [ ] Create new endpoint: `POST /Llm/Feedback`
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/Feedback`
|
||||
- [ ] Update frontend to show 👍👎 buttons after each response
|
||||
- [ ] Create analytics dashboard to view feedback trends
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Domain/Llm/LlmFeedback.cs`
|
||||
- `src/Managing.Application.Abstractions/Repositories/ILlmFeedbackRepository.cs`
|
||||
- `src/Managing.Infrastructure/Repositories/LlmFeedbackRepository.cs`
|
||||
- `src/Managing.Application/Services/LlmFeedbackService.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 3: Export Conversations
|
||||
**Status:** Not Started
|
||||
**Effort:** 4-6 hours
|
||||
**Impact:** Medium
|
||||
|
||||
**Description:**
|
||||
Export conversation to Markdown, JSON, or PDF for reporting and sharing.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `IConversationExportService` interface
|
||||
- [ ] Implement Markdown export (simple format with messages)
|
||||
- [ ] Implement JSON export (structured data)
|
||||
- [ ] Implement PDF export using QuestPDF or similar
|
||||
- [ ] Create new endpoint: `GET /Llm/Conversations/{id}/Export?format={md|json|pdf}`
|
||||
- [ ] Add "Export" button to conversation UI
|
||||
- [ ] Test with long conversations and special characters
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Application/Services/ConversationExportService.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/IConversationExportService.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 4: Query Categorization
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-4 hours
|
||||
**Impact:** Medium (Better analytics)
|
||||
|
||||
**Description:**
|
||||
Automatically categorize queries (BacktestAnalysis, GeneralFinance, etc.) for analytics.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `QueryCategory` enum (BacktestAnalysis, BundleAnalysis, IndicatorQuestion, GeneralFinance, HowTo, DataRetrieval, Comparison)
|
||||
- [ ] Add `QueryCategory` property to `LlmProgressUpdate`
|
||||
- [ ] Create `DetermineQueryCategory()` method using keyword matching
|
||||
- [ ] Update system prompt to include category in response
|
||||
- [ ] Send category in initial progress update
|
||||
- [ ] Track category distribution in analytics
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Medium Effort (3-5 days)
|
||||
|
||||
### ✅ Priority 5: Conversation Persistence
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Very High (Core feature)
|
||||
|
||||
**Description:**
|
||||
Save conversation history to database so users can resume previous conversations across sessions.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `ChatConversation` domain model (Id, UserId, Title, CreatedAt, UpdatedAt, LastMessageAt)
|
||||
- [ ] Create `ChatMessage` domain model (Id, ConversationId, Role, Content, Timestamp, TokenCount, ToolCalls)
|
||||
- [ ] Create `IChatConversationRepository` interface
|
||||
- [ ] Implement `ChatConversationRepository` with MongoDB
|
||||
- [ ] Create `IChatMessageRepository` interface
|
||||
- [ ] Implement `ChatMessageRepository` with MongoDB
|
||||
- [ ] Create new endpoint: `GET /Llm/Conversations` (list user's conversations)
|
||||
- [ ] Create new endpoint: `GET /Llm/Conversations/{id}` (get conversation with messages)
|
||||
- [ ] Create new endpoint: `POST /Llm/Conversations` (create new conversation)
|
||||
- [ ] Create new endpoint: `POST /Llm/Conversations/{id}/Messages` (add message to conversation)
|
||||
- [ ] Create new endpoint: `DELETE /Llm/Conversations/{id}` (delete conversation)
|
||||
- [ ] Update `ChatStream` to save messages automatically
|
||||
- [ ] Create conversation list UI component
|
||||
- [ ] Add "New Conversation" button
|
||||
- [ ] Add conversation sidebar with search/filter
|
||||
- [ ] Test with multiple concurrent conversations
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Domain/Llm/ChatConversation.cs`
|
||||
- `src/Managing.Domain/Llm/ChatMessage.cs`
|
||||
- `src/Managing.Application.Abstractions/Repositories/IChatConversationRepository.cs`
|
||||
- `src/Managing.Application.Abstractions/Repositories/IChatMessageRepository.cs`
|
||||
- `src/Managing.Infrastructure/Repositories/ChatConversationRepository.cs`
|
||||
- `src/Managing.Infrastructure/Repositories/ChatMessageRepository.cs`
|
||||
- `src/Managing.Application/Services/ChatConversationService.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 6: Response Streaming (Token-by-Token)
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** High (UX improvement)
|
||||
|
||||
**Description:**
|
||||
Stream LLM response as tokens arrive instead of waiting for complete response.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Update `ILlmService.ChatAsync()` to return `IAsyncEnumerable<LlmTokenChunk>`
|
||||
- [ ] Modify LLM provider implementations to support streaming
|
||||
- [ ] Update `ChatStreamInternal()` to stream tokens via SignalR
|
||||
- [ ] Add new progress update type: "token_stream"
|
||||
- [ ] Update frontend to display streaming tokens with typing animation
|
||||
- [ ] Handle tool calls during streaming (partial JSON parsing)
|
||||
- [ ] Add "Stop Generation" button in UI
|
||||
- [ ] Test with different LLM providers
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
- `src/Managing.Application/Services/LlmService.cs` (or provider-specific implementations)
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- Frontend components (AiChat.tsx)
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 7: Usage Analytics Dashboard
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** High (Cost monitoring)
|
||||
|
||||
**Description:**
|
||||
Track and visualize LLM usage metrics (tokens, cost, performance).
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `LlmUsageMetric` domain model (UserId, Timestamp, Provider, Model, PromptTokens, CompletionTokens, Cost, Duration, QueryCategory)
|
||||
- [ ] Create `ILlmUsageRepository` interface
|
||||
- [ ] Implement `LlmUsageRepository` with InfluxDB (time-series data)
|
||||
- [ ] Update `ChatStreamInternal()` to log usage metrics
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/Usage` (token usage over time)
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/PopularTools` (most called tools)
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/AverageIterations` (performance metrics)
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/ErrorRate` (quality metrics)
|
||||
- [ ] Create new endpoint: `GET /Llm/Analytics/CostEstimate` (current month cost)
|
||||
- [ ] Create analytics dashboard component with charts (Chart.js or Recharts)
|
||||
- [ ] Add filters: date range, category, provider
|
||||
- [ ] Display key metrics: total tokens, cost, avg response time
|
||||
- [ ] Test with large datasets
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Domain/Llm/LlmUsageMetric.cs`
|
||||
- `src/Managing.Application.Abstractions/Repositories/ILlmUsageRepository.cs`
|
||||
- `src/Managing.Infrastructure/Repositories/LlmUsageRepository.cs`
|
||||
- `src/Managing.Application/Services/LlmAnalyticsService.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 8: Quick Actions / Shortcuts
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Medium (Workflow improvement)
|
||||
|
||||
**Description:**
|
||||
Recognize patterns and offer action buttons (e.g., "Delete this backtest" after analysis).
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `QuickAction` model (Id, Label, Icon, Endpoint, Parameters)
|
||||
- [ ] Add `Actions` property to `LlmProgressUpdate`
|
||||
- [ ] Create `GenerateQuickActions()` method based on context
|
||||
- [ ] Update system prompt to suggest actions in structured format
|
||||
- [ ] Parse action suggestions from LLM response
|
||||
- [ ] Update frontend to display action buttons
|
||||
- [ ] Implement action handlers (call APIs)
|
||||
- [ ] Add confirmation dialogs for destructive actions
|
||||
- [ ] Test with various scenarios (backtest, bundle, indicator)
|
||||
|
||||
**Example Actions:**
|
||||
- After backtest analysis: "Delete this backtest", "Run similar backtest", "Export details"
|
||||
- After bundle analysis: "Delete bundle", "Run again with different params"
|
||||
- After list query: "Export to CSV", "Show details"
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
- Frontend components (AiChat.tsx)
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Long-term (1-2 weeks)
|
||||
|
||||
### ✅ Priority 9: Multi-Provider Fallback
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-5 days
|
||||
**Impact:** High (Reliability)
|
||||
|
||||
**Description:**
|
||||
Automatically fallback to alternative LLM provider on failure or rate limit.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `LlmProviderHealth` model to track provider status
|
||||
- [ ] Create `IProviderHealthMonitor` service
|
||||
- [ ] Implement health check mechanism (ping providers periodically)
|
||||
- [ ] Create provider priority list configuration
|
||||
- [ ] Update `LlmService.ChatAsync()` to implement fallback logic
|
||||
- [ ] Add retry logic with exponential backoff
|
||||
- [ ] Track provider failure rates
|
||||
- [ ] Send alert when provider is down
|
||||
- [ ] Update frontend to show current provider
|
||||
- [ ] Test failover scenarios
|
||||
|
||||
**Provider Priority Example:**
|
||||
1. Primary: OpenAI GPT-4
|
||||
2. Secondary: Anthropic Claude
|
||||
3. Tertiary: Google Gemini
|
||||
4. Fallback: Local model (if available)
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Application/Services/ProviderHealthMonitor.cs`
|
||||
- `src/Managing.Domain/Llm/LlmProviderHealth.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Application/Services/LlmService.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 10: Scheduled Queries / Alerts
|
||||
**Status:** Not Started
|
||||
**Effort:** 4-6 days
|
||||
**Impact:** High (Automation)
|
||||
|
||||
**Description:**
|
||||
Run queries on schedule and notify users of changes (e.g., "Alert when backtest scores > 80").
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `LlmAlert` domain model (Id, UserId, Query, Schedule, Condition, IsActive, LastRun, CreatedAt)
|
||||
- [ ] Create `ILlmAlertRepository` interface
|
||||
- [ ] Implement `LlmAlertRepository` with MongoDB
|
||||
- [ ] Create background service to process alerts (Hangfire or Quartz.NET)
|
||||
- [ ] Create new endpoint: `POST /Llm/Alerts` (create alert)
|
||||
- [ ] Create new endpoint: `GET /Llm/Alerts` (list user's alerts)
|
||||
- [ ] Create new endpoint: `PUT /Llm/Alerts/{id}` (update alert)
|
||||
- [ ] Create new endpoint: `DELETE /Llm/Alerts/{id}` (delete alert)
|
||||
- [ ] Implement notification system (SignalR, email, push)
|
||||
- [ ] Create alert management UI
|
||||
- [ ] Add schedule picker (cron expression builder)
|
||||
- [ ] Test with various schedules and conditions
|
||||
|
||||
**Example Alerts:**
|
||||
- "Notify me when a backtest scores > 80" (run every hour)
|
||||
- "Daily summary of new backtests" (run at 9am daily)
|
||||
- "Alert when bundle completes" (run every 5 minutes)
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Domain/Llm/LlmAlert.cs`
|
||||
- `src/Managing.Application.Abstractions/Repositories/ILlmAlertRepository.cs`
|
||||
- `src/Managing.Infrastructure/Repositories/LlmAlertRepository.cs`
|
||||
- `src/Managing.Application/Services/LlmAlertService.cs`
|
||||
- `src/Managing.Application/BackgroundServices/LlmAlertProcessor.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 11: Smart Context Window Management
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-5 days
|
||||
**Impact:** Medium (Better conversations)
|
||||
|
||||
**Description:**
|
||||
Intelligently compress conversation history instead of simple truncation.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Research and implement summarization approach (LLM-based or extractive)
|
||||
- [ ] Create `SummarizeConversation()` method
|
||||
- [ ] Update `TrimConversationContext()` to use summarization
|
||||
- [ ] Preserve key entities (IDs, numbers, dates)
|
||||
- [ ] Use embeddings to identify relevant context (optional, advanced)
|
||||
- [ ] Test with long conversations (50+ messages)
|
||||
- [ ] Measure token savings vs quality trade-off
|
||||
- [ ] Add configuration for compression strategy
|
||||
|
||||
**Approaches:**
|
||||
1. **Simple:** Summarize every N old messages into single message
|
||||
2. **Advanced:** Use embeddings to keep semantically relevant messages
|
||||
3. **Hybrid:** Keep recent messages + summarized older messages + key facts
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
|
||||
---
|
||||
|
||||
### ✅ Priority 12: Interactive Clarification Questions
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-4 days
|
||||
**Impact:** Medium (Reduce back-and-forth)
|
||||
|
||||
**Description:**
|
||||
When ambiguous, LLM asks structured multiple-choice questions instead of open-ended text.
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `ClarificationOption` model (Id, Label, Description)
|
||||
- [ ] Add `Options` property to `LlmProgressUpdate`
|
||||
- [ ] Update system prompt to output clarification questions in structured format
|
||||
- [ ] Create `ExtractClarificationOptions()` method
|
||||
- [ ] Update `ChatStreamInternal()` to handle clarification state
|
||||
- [ ] Update frontend to display radio buttons/chips for options
|
||||
- [ ] Handle user selection (send as next message automatically)
|
||||
- [ ] Test with ambiguous queries
|
||||
|
||||
**Example:**
|
||||
User: "Show me the backtest"
|
||||
LLM: "Which backtest would you like to see?"
|
||||
- 🔘 Best performing backtest
|
||||
- 🔘 Most recent backtest
|
||||
- 🔘 Specific backtest by name
|
||||
|
||||
**Files to Create:**
|
||||
- `src/Managing.Domain/Llm/ClarificationOption.cs`
|
||||
|
||||
**Files to Modify:**
|
||||
- `src/Managing.Api/Controllers/LlmController.cs`
|
||||
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
|
||||
- Frontend components (AiChat.tsx)
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Additional Features (Nice to Have)
|
||||
|
||||
### Voice Input Support
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Medium
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create new endpoint: `POST /Llm/VoiceChat` (accept audio file)
|
||||
- [ ] Integrate speech-to-text service (Azure Speech, OpenAI Whisper)
|
||||
- [ ] Process transcribed text as normal chat
|
||||
- [ ] Add microphone button in frontend
|
||||
- [ ] Handle audio recording in browser
|
||||
- [ ] Test with various audio formats and accents
|
||||
|
||||
---
|
||||
|
||||
### Smart Conversation Titling
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 hours
|
||||
**Impact:** Low (QoL)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] After first response, send summary request to LLM
|
||||
- [ ] Update conversation title in background
|
||||
- [ ] Don't block user while generating title
|
||||
- [ ] Test with various conversation types
|
||||
|
||||
---
|
||||
|
||||
### Tool Call Caching
|
||||
**Status:** Not Started
|
||||
**Effort:** 1-2 days
|
||||
**Impact:** Medium (Performance)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create cache key hash function (toolName + arguments)
|
||||
- [ ] Implement cache wrapper around `ExecuteToolAsync()`
|
||||
- [ ] Configure cache duration per tool type
|
||||
- [ ] Invalidate cache on data mutations
|
||||
- [ ] Test cache hit/miss rates
|
||||
|
||||
---
|
||||
|
||||
### Conversation Branching
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Low (Power user feature)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create new endpoint: `POST /Llm/Conversations/{id}/Branch?fromMessageId={id}`
|
||||
- [ ] Copy conversation history up to branch point
|
||||
- [ ] Create new conversation with copied history
|
||||
- [ ] Update UI to show branch option on messages
|
||||
- [ ] Test branching at various points
|
||||
|
||||
---
|
||||
|
||||
### LLM Model Selection
|
||||
**Status:** Not Started
|
||||
**Effort:** 1-2 days
|
||||
**Impact:** Medium (Cost control)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Add `PreferredModel` property to `LlmChatRequest`
|
||||
- [ ] Create model configuration (pricing, speed, quality scores)
|
||||
- [ ] Update frontend with model selector dropdown
|
||||
- [ ] Display model info (cost, speed, quality)
|
||||
- [ ] Test with different models
|
||||
|
||||
---
|
||||
|
||||
### Debug Mode
|
||||
**Status:** Not Started
|
||||
**Effort:** 4-6 hours
|
||||
**Impact:** Low (Developer tool)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Add `Debug` property to `LlmChatRequest`
|
||||
- [ ] Return full prompt, raw response, token breakdown when debug=true
|
||||
- [ ] Create debug panel in UI
|
||||
- [ ] Add toggle to enable/disable debug mode
|
||||
- [ ] Test with various queries
|
||||
|
||||
---
|
||||
|
||||
### PII Detection & Redaction
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Medium (Security)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Implement PII detection regex (email, phone, SSN, credit card)
|
||||
- [ ] Scan messages before sending to LLM
|
||||
- [ ] Warn user about detected PII
|
||||
- [ ] Option to redact or anonymize
|
||||
- [ ] Test with various PII patterns
|
||||
|
||||
---
|
||||
|
||||
### Rate Limiting Per User
|
||||
**Status:** Not Started
|
||||
**Effort:** 1-2 days
|
||||
**Impact:** Medium (Cost control)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create rate limit configuration (requests/hour, tokens/day)
|
||||
- [ ] Implement rate limit middleware
|
||||
- [ ] Track usage per user
|
||||
- [ ] Return 429 with quota info when exceeded
|
||||
- [ ] Display quota usage in UI
|
||||
|
||||
---
|
||||
|
||||
### Request Queueing
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Medium (Reliability)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Implement request queue with priority
|
||||
- [ ] Queue requests when rate limited
|
||||
- [ ] Send position-in-queue updates via SignalR
|
||||
- [ ] Process queue when rate limit resets
|
||||
- [ ] Test with high load
|
||||
|
||||
---
|
||||
|
||||
### Prompt Version Control
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Low (Optimization)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create `SystemPrompt` model (Version, Content, CreatedAt, IsActive, SuccessRate)
|
||||
- [ ] Store multiple prompt versions
|
||||
- [ ] A/B test prompts (rotate per conversation)
|
||||
- [ ] Track success metrics per prompt version
|
||||
- [ ] UI to manage prompt versions
|
||||
|
||||
---
|
||||
|
||||
### LLM Playground
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-4 days
|
||||
**Impact:** Low (Developer tool)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Create playground UI component
|
||||
- [ ] System prompt editor with syntax highlighting
|
||||
- [ ] Message history builder
|
||||
- [ ] Tool selector
|
||||
- [ ] Temperature/token controls
|
||||
- [ ] Side-by-side comparison
|
||||
- [ ] Test various configurations
|
||||
|
||||
---
|
||||
|
||||
### Collaborative Filtering
|
||||
**Status:** Not Started
|
||||
**Effort:** 3-5 days
|
||||
**Impact:** Low (Discovery)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Track query patterns per user
|
||||
- [ ] Implement collaborative filtering algorithm
|
||||
- [ ] Suggest related queries after response
|
||||
- [ ] Display "Users also asked" section
|
||||
- [ ] Test recommendation quality
|
||||
|
||||
---
|
||||
|
||||
### Conversation Encryption
|
||||
**Status:** Not Started
|
||||
**Effort:** 2-3 days
|
||||
**Impact:** Medium (Security)
|
||||
|
||||
**Implementation Tasks:**
|
||||
- [ ] Implement encryption/decryption service
|
||||
- [ ] Generate user-specific encryption keys
|
||||
- [ ] Encrypt messages before storing
|
||||
- [ ] Decrypt on retrieval
|
||||
- [ ] Test performance impact
|
||||
|
||||
---
|
||||
|
||||
## 📊 Progress Tracker
|
||||
|
||||
**Quick Wins:** 0/4 completed (0%)
|
||||
**Medium Effort:** 0/4 completed (0%)
|
||||
**Long-term:** 0/4 completed (0%)
|
||||
**Additional Features:** 0/15 completed (0%)
|
||||
|
||||
**Overall Progress:** 0/27 completed (0%)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Recommended Implementation Order
|
||||
|
||||
1. **Conversation Persistence** - Foundation for other features
|
||||
2. **Suggested Follow-up Questions** - Quick UX win
|
||||
3. **Feedback & Rating System** - Quality tracking
|
||||
4. **Usage Analytics Dashboard** - Monitor costs
|
||||
5. **Response Streaming** - Better UX
|
||||
6. **Export Conversations** - User requested feature
|
||||
7. **Quick Actions** - Workflow optimization
|
||||
8. **Multi-Provider Fallback** - Reliability
|
||||
9. **Query Categorization** - Better analytics
|
||||
10. **Smart Context Management** - Better conversations
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- All features should follow the Controller → Application → Repository pattern
|
||||
- Regenerate `ManagingApi.ts` after adding new endpoints: `cd src/Managing.Nswag && dotnet build`
|
||||
- Use MongoDB for document storage, InfluxDB for time-series metrics
|
||||
- Test all features with real user scenarios
|
||||
- Consider token costs when implementing LLM-heavy features (summarization, titling)
|
||||
- Ensure all features respect user privacy and data security
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-07
|
||||
**Maintained By:** Development Team
|
||||
File diff suppressed because it is too large
Load Diff
@@ -166,22 +166,13 @@ public class GeminiProvider : ILlmProvider
|
||||
|
||||
private object ConvertToGeminiRequest(LlmChatRequest request)
|
||||
{
|
||||
var contents = request.Messages
|
||||
.Where(m => m.Role != "system") // Gemini doesn't support system messages in the same way
|
||||
.Select(m => new
|
||||
{
|
||||
role = m.Role == "assistant" ? "model" : "user",
|
||||
parts = new[]
|
||||
{
|
||||
new { text = m.Content }
|
||||
}
|
||||
}).ToList();
|
||||
|
||||
// Add system message as first user message if present
|
||||
var contents = new List<object>();
|
||||
|
||||
// Add system message as first user message if present (Gemini only uses first system message)
|
||||
var systemMessage = request.Messages.FirstOrDefault(m => m.Role == "system");
|
||||
if (systemMessage != null && !string.IsNullOrWhiteSpace(systemMessage.Content))
|
||||
{
|
||||
contents.Insert(0, new
|
||||
contents.Add(new
|
||||
{
|
||||
role = "user",
|
||||
parts = new[]
|
||||
@@ -191,6 +182,70 @@ public class GeminiProvider : ILlmProvider
|
||||
});
|
||||
}
|
||||
|
||||
// Process non-system messages in order
|
||||
// Gemini expects: user -> model (assistant) -> user (tool results) -> model -> ...
|
||||
foreach (var message in request.Messages.Where(m => m.Role != "system"))
|
||||
{
|
||||
if (message.Role == "assistant")
|
||||
{
|
||||
// Assistant message - check if it has tool calls
|
||||
if (message.ToolCalls != null && message.ToolCalls.Any())
|
||||
{
|
||||
// This is a function call request - Gemini handles this automatically
|
||||
// We still need to add the text content if any
|
||||
if (!string.IsNullOrWhiteSpace(message.Content))
|
||||
{
|
||||
contents.Add(new
|
||||
{
|
||||
role = "model",
|
||||
parts = new[]
|
||||
{
|
||||
new { text = message.Content }
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
// Regular assistant response
|
||||
contents.Add(new
|
||||
{
|
||||
role = "model",
|
||||
parts = new[]
|
||||
{
|
||||
new { text = message.Content ?? "" }
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
else if (message.Role == "tool")
|
||||
{
|
||||
// Tool results - Gemini expects these as functionResponse parts in the model's response
|
||||
// But since we're sending them as separate messages, we'll format them as user messages
|
||||
// with clear indication they're tool results
|
||||
contents.Add(new
|
||||
{
|
||||
role = "user",
|
||||
parts = new[]
|
||||
{
|
||||
new { text = $"Tool result (call_id: {message.ToolCallId}): {message.Content}" }
|
||||
}
|
||||
});
|
||||
}
|
||||
else
|
||||
{
|
||||
// User message
|
||||
contents.Add(new
|
||||
{
|
||||
role = "user",
|
||||
parts = new[]
|
||||
{
|
||||
new { text = message.Content ?? "" }
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
var geminiRequest = new
|
||||
{
|
||||
contents,
|
||||
|
||||
@@ -159,22 +159,46 @@ function AiChat({ onClose }: AiChatProps): JSX.Element {
|
||||
|
||||
if (update.type === 'final_response' && update.response) {
|
||||
finalResponse = update.response
|
||||
console.log('Received final response from LLM:', {
|
||||
hasContent: !!finalResponse.content,
|
||||
contentLength: finalResponse.content?.length || 0,
|
||||
contentPreview: finalResponse.content?.substring(0, 100) || '(empty)',
|
||||
fullContent: finalResponse.content,
|
||||
requiresToolExecution: finalResponse.requiresToolExecution
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Add final response if we got one
|
||||
if (finalResponse) {
|
||||
// Backend should always send meaningful content, but handle edge cases
|
||||
const rawContent = finalResponse.content?.trim() || ''
|
||||
const isContentValid = rawContent.length > 0
|
||||
|
||||
const assistantMessage: Message = {
|
||||
role: 'assistant',
|
||||
content: finalResponse.content || 'No response from AI',
|
||||
content: isContentValid
|
||||
? rawContent
|
||||
: 'I received your request but the response was empty. Please try rephrasing your question or ask for specific details.',
|
||||
timestamp: new Date()
|
||||
}
|
||||
console.log('Adding final assistant message to chat:', {
|
||||
rawContentLength: finalResponse.content?.length || 0,
|
||||
trimmedContentLength: rawContent.length,
|
||||
isContentValid,
|
||||
contentPreview: assistantMessage.content.substring(0, 100),
|
||||
fullContent: assistantMessage.content
|
||||
})
|
||||
setMessages(prev => [...prev, assistantMessage])
|
||||
} else if (lastUpdate && lastUpdate.type === 'final_response' && lastUpdate.response) {
|
||||
// Fallback: check lastUpdate in case finalResponse wasn't set
|
||||
console.log('Using fallback: final response from lastUpdate')
|
||||
const rawContent = lastUpdate.response.content?.trim() || ''
|
||||
const assistantMessage: Message = {
|
||||
role: 'assistant',
|
||||
content: lastUpdate.response.content || 'No response from AI',
|
||||
content: rawContent.length > 0
|
||||
? rawContent
|
||||
: 'I received your request but the response was empty. Please try rephrasing your question or ask for specific details.',
|
||||
timestamp: new Date()
|
||||
}
|
||||
setMessages(prev => [...prev, assistantMessage])
|
||||
@@ -188,9 +212,14 @@ function AiChat({ onClose }: AiChatProps): JSX.Element {
|
||||
setMessages(prev => [...prev, errorMessage])
|
||||
} else {
|
||||
// If we didn't get a final response, show the last progress message
|
||||
console.warn('No final response received. Last update:', {
|
||||
type: lastUpdate?.type,
|
||||
message: lastUpdate?.message,
|
||||
hasResponse: !!lastUpdate?.response
|
||||
})
|
||||
const assistantMessage: Message = {
|
||||
role: 'assistant',
|
||||
content: lastUpdate?.message || 'Response incomplete',
|
||||
content: lastUpdate?.message || 'Response incomplete. Please try again.',
|
||||
timestamp: new Date()
|
||||
}
|
||||
setMessages(prev => [...prev, assistantMessage])
|
||||
|
||||
Reference in New Issue
Block a user