managing-apps/LLM_IMPROVEMENTS_TODO.md

# LLM Controller - Feature Improvements Roadmap

## 🎯 Quick Wins (1-2 days)

### ✅ Priority 1: Suggested Follow-up Questions
**Status:** Not Started
**Effort:** 4-6 hours
**Impact:** High

**Description:**
After each response, the LLM suggests 3-5 relevant follow-up questions to guide user exploration.

**Implementation Tasks:**
- [ ] Update `BuildSystemMessage()` to include follow-up question instruction
- [ ] Add `SuggestedQuestions` property to `LlmProgressUpdate` class
- [ ] Create `ExtractFollowUpQuestions()` method to parse questions from response
- [ ] Update `ChatStreamInternal()` to extract and send suggested questions
- [ ] Update frontend to display suggested questions as clickable chips
- [ ] Test with various query types (backtest, indicator, general finance)

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
- Frontend components (AiChat.tsx)

---

### ✅ Priority 2: Feedback & Rating System
**Status:** Not Started
**Effort:** 6-8 hours
**Impact:** High (Quality tracking)

**Description:**
Users can rate LLM responses (👍👎) with optional comments to track quality and improve prompts.

**Implementation Tasks:**
- [ ] Create `LlmFeedback` domain model (ResponseId, UserId, Rating, Comment, Timestamp)
- [ ] Create `ILlmFeedbackRepository` interface
- [ ] Implement `LlmFeedbackRepository` with MongoDB
- [ ] Add `ResponseId` property to `LlmChatResponse`
- [ ] Create new endpoint: `POST /Llm/Feedback`
- [ ] Create new endpoint: `GET /Llm/Analytics/Feedback`
- [ ] Update frontend to show 👍👎 buttons after each response
- [ ] Create analytics dashboard to view feedback trends

**Files to Create:**
- `src/Managing.Domain/Llm/LlmFeedback.cs`
- `src/Managing.Application.Abstractions/Repositories/ILlmFeedbackRepository.cs`
- `src/Managing.Infrastructure/Repositories/LlmFeedbackRepository.cs`
- `src/Managing.Application/Services/LlmFeedbackService.cs`

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`

---

### ✅ Priority 3: Export Conversations
**Status:** Not Started
**Effort:** 4-6 hours
**Impact:** Medium

**Description:**
Export conversation to Markdown, JSON, or PDF for reporting and sharing.

**Implementation Tasks:**
- [ ] Create `IConversationExportService` interface
- [ ] Implement Markdown export (simple format with messages)
- [ ] Implement JSON export (structured data)
- [ ] Implement PDF export using QuestPDF or similar
- [ ] Create new endpoint: `GET /Llm/Conversations/{id}/Export?format={md|json|pdf}`
- [ ] Add "Export" button to conversation UI
- [ ] Test with long conversations and special characters

**Files to Create:**
- `src/Managing.Application/Services/ConversationExportService.cs`
- `src/Managing.Application.Abstractions/Services/IConversationExportService.cs`

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`

---

### ✅ Priority 4: Query Categorization
**Status:** Not Started
**Effort:** 3-4 hours
**Impact:** Medium (Better analytics)

**Description:**
Automatically categorize queries (BacktestAnalysis, GeneralFinance, etc.) for analytics.

**Implementation Tasks:**
- [ ] Create `QueryCategory` enum (BacktestAnalysis, BundleAnalysis, IndicatorQuestion, GeneralFinance, HowTo, DataRetrieval, Comparison)
- [ ] Add `QueryCategory` property to `LlmProgressUpdate`
- [ ] Create `DetermineQueryCategory()` method using keyword matching
- [ ] Update system prompt to include category in response
- [ ] Send category in initial progress update
- [ ] Track category distribution in analytics

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`

---

## 🚀 Medium Effort (3-5 days)

### ✅ Priority 5: Conversation Persistence
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Very High (Core feature)

**Description:**
Save conversation history to database so users can resume previous conversations across sessions.

**Implementation Tasks:**
- [ ] Create `ChatConversation` domain model (Id, UserId, Title, CreatedAt, UpdatedAt, LastMessageAt)
- [ ] Create `ChatMessage` domain model (Id, ConversationId, Role, Content, Timestamp, TokenCount, ToolCalls)
- [ ] Create `IChatConversationRepository` interface
- [ ] Implement `ChatConversationRepository` with MongoDB
- [ ] Create `IChatMessageRepository` interface
- [ ] Implement `ChatMessageRepository` with MongoDB
- [ ] Create new endpoint: `GET /Llm/Conversations` (list user's conversations)
- [ ] Create new endpoint: `GET /Llm/Conversations/{id}` (get conversation with messages)
- [ ] Create new endpoint: `POST /Llm/Conversations` (create new conversation)
- [ ] Create new endpoint: `POST /Llm/Conversations/{id}/Messages` (add message to conversation)
- [ ] Create new endpoint: `DELETE /Llm/Conversations/{id}` (delete conversation)
- [ ] Update `ChatStream` to save messages automatically
- [ ] Create conversation list UI component
- [ ] Add "New Conversation" button
- [ ] Add conversation sidebar with search/filter
- [ ] Test with multiple concurrent conversations

**Files to Create:**
- `src/Managing.Domain/Llm/ChatConversation.cs`
- `src/Managing.Domain/Llm/ChatMessage.cs`
- `src/Managing.Application.Abstractions/Repositories/IChatConversationRepository.cs`
- `src/Managing.Application.Abstractions/Repositories/IChatMessageRepository.cs`
- `src/Managing.Infrastructure/Repositories/ChatConversationRepository.cs`
- `src/Managing.Infrastructure/Repositories/ChatMessageRepository.cs`
- `src/Managing.Application/Services/ChatConversationService.cs`

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`

---

### ✅ Priority 6: Response Streaming (Token-by-Token)
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** High (UX improvement)

**Description:**
Stream LLM response as tokens arrive instead of waiting for complete response.

**Implementation Tasks:**
- [ ] Update `ILlmService.ChatAsync()` to return `IAsyncEnumerable<LlmTokenChunk>`
- [ ] Modify LLM provider implementations to support streaming
- [ ] Update `ChatStreamInternal()` to stream tokens via SignalR
- [ ] Add new progress update type: "token_stream"
- [ ] Update frontend to display streaming tokens with typing animation
- [ ] Handle tool calls during streaming (partial JSON parsing)
- [ ] Add "Stop Generation" button in UI
- [ ] Test with different LLM providers

**Files to Modify:**
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
- `src/Managing.Application/Services/LlmService.cs` (or provider-specific implementations)
- `src/Managing.Api/Controllers/LlmController.cs`
- Frontend components (AiChat.tsx)

---

### ✅ Priority 7: Usage Analytics Dashboard
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** High (Cost monitoring)

**Description:**
Track and visualize LLM usage metrics (tokens, cost, performance).

**Implementation Tasks:**
- [ ] Create `LlmUsageMetric` domain model (UserId, Timestamp, Provider, Model, PromptTokens, CompletionTokens, Cost, Duration, QueryCategory)
- [ ] Create `ILlmUsageRepository` interface
- [ ] Implement `LlmUsageRepository` with InfluxDB (time-series data)
- [ ] Update `ChatStreamInternal()` to log usage metrics
- [ ] Create new endpoint: `GET /Llm/Analytics/Usage` (token usage over time)
- [ ] Create new endpoint: `GET /Llm/Analytics/PopularTools` (most called tools)
- [ ] Create new endpoint: `GET /Llm/Analytics/AverageIterations` (performance metrics)
- [ ] Create new endpoint: `GET /Llm/Analytics/ErrorRate` (quality metrics)
- [ ] Create new endpoint: `GET /Llm/Analytics/CostEstimate` (current month cost)
- [ ] Create analytics dashboard component with charts (Chart.js or Recharts)
- [ ] Add filters: date range, category, provider
- [ ] Display key metrics: total tokens, cost, avg response time
- [ ] Test with large datasets

**Files to Create:**
- `src/Managing.Domain/Llm/LlmUsageMetric.cs`
- `src/Managing.Application.Abstractions/Repositories/ILlmUsageRepository.cs`
- `src/Managing.Infrastructure/Repositories/LlmUsageRepository.cs`
- `src/Managing.Application/Services/LlmAnalyticsService.cs`

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`

---

### ✅ Priority 8: Quick Actions / Shortcuts
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Medium (Workflow improvement)

**Description:**
Recognize patterns and offer action buttons (e.g., "Delete this backtest" after analysis).

**Implementation Tasks:**
- [ ] Create `QuickAction` model (Id, Label, Icon, Endpoint, Parameters)
- [ ] Add `Actions` property to `LlmProgressUpdate`
- [ ] Create `GenerateQuickActions()` method based on context
- [ ] Update system prompt to suggest actions in structured format
- [ ] Parse action suggestions from LLM response
- [ ] Update frontend to display action buttons
- [ ] Implement action handlers (call APIs)
- [ ] Add confirmation dialogs for destructive actions
- [ ] Test with various scenarios (backtest, bundle, indicator)

**Example Actions:**
- After backtest analysis: "Delete this backtest", "Run similar backtest", "Export details"
- After bundle analysis: "Delete bundle", "Run again with different params"
- After list query: "Export to CSV", "Show details"

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
- Frontend components (AiChat.tsx)

---

## 🎨 Long-term (1-2 weeks)

### ✅ Priority 9: Multi-Provider Fallback
**Status:** Not Started
**Effort:** 3-5 days
**Impact:** High (Reliability)

**Description:**
Automatically fallback to alternative LLM provider on failure or rate limit.

**Implementation Tasks:**
- [ ] Create `LlmProviderHealth` model to track provider status
- [ ] Create `IProviderHealthMonitor` service
- [ ] Implement health check mechanism (ping providers periodically)
- [ ] Create provider priority list configuration
- [ ] Update `LlmService.ChatAsync()` to implement fallback logic
- [ ] Add retry logic with exponential backoff
- [ ] Track provider failure rates
- [ ] Send alert when provider is down
- [ ] Update frontend to show current provider
- [ ] Test failover scenarios

**Provider Priority Example:**
1. Primary: OpenAI GPT-4
2. Secondary: Anthropic Claude
3. Tertiary: Google Gemini
4. Fallback: Local model (if available)

**Files to Create:**
- `src/Managing.Application/Services/ProviderHealthMonitor.cs`
- `src/Managing.Domain/Llm/LlmProviderHealth.cs`

**Files to Modify:**
- `src/Managing.Application/Services/LlmService.cs`

---

### ✅ Priority 10: Scheduled Queries / Alerts
**Status:** Not Started
**Effort:** 4-6 days
**Impact:** High (Automation)

**Description:**
Run queries on schedule and notify users of changes (e.g., "Alert when backtest scores > 80").

**Implementation Tasks:**
- [ ] Create `LlmAlert` domain model (Id, UserId, Query, Schedule, Condition, IsActive, LastRun, CreatedAt)
- [ ] Create `ILlmAlertRepository` interface
- [ ] Implement `LlmAlertRepository` with MongoDB
- [ ] Create background service to process alerts (Hangfire or Quartz.NET)
- [ ] Create new endpoint: `POST /Llm/Alerts` (create alert)
- [ ] Create new endpoint: `GET /Llm/Alerts` (list user's alerts)
- [ ] Create new endpoint: `PUT /Llm/Alerts/{id}` (update alert)
- [ ] Create new endpoint: `DELETE /Llm/Alerts/{id}` (delete alert)
- [ ] Implement notification system (SignalR, email, push)
- [ ] Create alert management UI
- [ ] Add schedule picker (cron expression builder)
- [ ] Test with various schedules and conditions

**Example Alerts:**
- "Notify me when a backtest scores > 80" (run every hour)
- "Daily summary of new backtests" (run at 9am daily)
- "Alert when bundle completes" (run every 5 minutes)

**Files to Create:**
- `src/Managing.Domain/Llm/LlmAlert.cs`
- `src/Managing.Application.Abstractions/Repositories/ILlmAlertRepository.cs`
- `src/Managing.Infrastructure/Repositories/LlmAlertRepository.cs`
- `src/Managing.Application/Services/LlmAlertService.cs`
- `src/Managing.Application/BackgroundServices/LlmAlertProcessor.cs`

---

### ✅ Priority 11: Smart Context Window Management
**Status:** Not Started
**Effort:** 3-5 days
**Impact:** Medium (Better conversations)

**Description:**
Intelligently compress conversation history instead of simple truncation.

**Implementation Tasks:**
- [ ] Research and implement summarization approach (LLM-based or extractive)
- [ ] Create `SummarizeConversation()` method
- [ ] Update `TrimConversationContext()` to use summarization
- [ ] Preserve key entities (IDs, numbers, dates)
- [ ] Use embeddings to identify relevant context (optional, advanced)
- [ ] Test with long conversations (50+ messages)
- [ ] Measure token savings vs quality trade-off
- [ ] Add configuration for compression strategy

**Approaches:**
1. **Simple:** Summarize every N old messages into single message
2. **Advanced:** Use embeddings to keep semantically relevant messages
3. **Hybrid:** Keep recent messages + summarized older messages + key facts

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`

---

### ✅ Priority 12: Interactive Clarification Questions
**Status:** Not Started
**Effort:** 3-4 days
**Impact:** Medium (Reduce back-and-forth)

**Description:**
When ambiguous, LLM asks structured multiple-choice questions instead of open-ended text.

**Implementation Tasks:**
- [ ] Create `ClarificationOption` model (Id, Label, Description)
- [ ] Add `Options` property to `LlmProgressUpdate`
- [ ] Update system prompt to output clarification questions in structured format
- [ ] Create `ExtractClarificationOptions()` method
- [ ] Update `ChatStreamInternal()` to handle clarification state
- [ ] Update frontend to display radio buttons/chips for options
- [ ] Handle user selection (send as next message automatically)
- [ ] Test with ambiguous queries

**Example:**
User: "Show me the backtest"
LLM: "Which backtest would you like to see?"
- 🔘 Best performing backtest
- 🔘 Most recent backtest
- 🔘 Specific backtest by name

**Files to Create:**
- `src/Managing.Domain/Llm/ClarificationOption.cs`

**Files to Modify:**
- `src/Managing.Api/Controllers/LlmController.cs`
- `src/Managing.Application.Abstractions/Services/ILlmService.cs`
- Frontend components (AiChat.tsx)

---

## 🔧 Additional Features (Nice to Have)

### Voice Input Support
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Medium

**Implementation Tasks:**
- [ ] Create new endpoint: `POST /Llm/VoiceChat` (accept audio file)
- [ ] Integrate speech-to-text service (Azure Speech, OpenAI Whisper)
- [ ] Process transcribed text as normal chat
- [ ] Add microphone button in frontend
- [ ] Handle audio recording in browser
- [ ] Test with various audio formats and accents

---

### Smart Conversation Titling
**Status:** Not Started
**Effort:** 2-3 hours
**Impact:** Low (QoL)

**Implementation Tasks:**
- [ ] After first response, send summary request to LLM
- [ ] Update conversation title in background
- [ ] Don't block user while generating title
- [ ] Test with various conversation types

---

### Tool Call Caching
**Status:** Not Started
**Effort:** 1-2 days
**Impact:** Medium (Performance)

**Implementation Tasks:**
- [ ] Create cache key hash function (toolName + arguments)
- [ ] Implement cache wrapper around `ExecuteToolAsync()`
- [ ] Configure cache duration per tool type
- [ ] Invalidate cache on data mutations
- [ ] Test cache hit/miss rates

---

### Conversation Branching
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Low (Power user feature)

**Implementation Tasks:**
- [ ] Create new endpoint: `POST /Llm/Conversations/{id}/Branch?fromMessageId={id}`
- [ ] Copy conversation history up to branch point
- [ ] Create new conversation with copied history
- [ ] Update UI to show branch option on messages
- [ ] Test branching at various points

---

### LLM Model Selection
**Status:** Not Started
**Effort:** 1-2 days
**Impact:** Medium (Cost control)

**Implementation Tasks:**
- [ ] Add `PreferredModel` property to `LlmChatRequest`
- [ ] Create model configuration (pricing, speed, quality scores)
- [ ] Update frontend with model selector dropdown
- [ ] Display model info (cost, speed, quality)
- [ ] Test with different models

---

### Debug Mode
**Status:** Not Started
**Effort:** 4-6 hours
**Impact:** Low (Developer tool)

**Implementation Tasks:**
- [ ] Add `Debug` property to `LlmChatRequest`
- [ ] Return full prompt, raw response, token breakdown when debug=true
- [ ] Create debug panel in UI
- [ ] Add toggle to enable/disable debug mode
- [ ] Test with various queries

---

### PII Detection & Redaction
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Medium (Security)

**Implementation Tasks:**
- [ ] Implement PII detection regex (email, phone, SSN, credit card)
- [ ] Scan messages before sending to LLM
- [ ] Warn user about detected PII
- [ ] Option to redact or anonymize
- [ ] Test with various PII patterns

---

### Rate Limiting Per User
**Status:** Not Started
**Effort:** 1-2 days
**Impact:** Medium (Cost control)

**Implementation Tasks:**
- [ ] Create rate limit configuration (requests/hour, tokens/day)
- [ ] Implement rate limit middleware
- [ ] Track usage per user
- [ ] Return 429 with quota info when exceeded
- [ ] Display quota usage in UI

---

### Request Queueing
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Medium (Reliability)

**Implementation Tasks:**
- [ ] Implement request queue with priority
- [ ] Queue requests when rate limited
- [ ] Send position-in-queue updates via SignalR
- [ ] Process queue when rate limit resets
- [ ] Test with high load

---

### Prompt Version Control
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Low (Optimization)

**Implementation Tasks:**
- [ ] Create `SystemPrompt` model (Version, Content, CreatedAt, IsActive, SuccessRate)
- [ ] Store multiple prompt versions
- [ ] A/B test prompts (rotate per conversation)
- [ ] Track success metrics per prompt version
- [ ] UI to manage prompt versions

---

### LLM Playground
**Status:** Not Started
**Effort:** 3-4 days
**Impact:** Low (Developer tool)

**Implementation Tasks:**
- [ ] Create playground UI component
- [ ] System prompt editor with syntax highlighting
- [ ] Message history builder
- [ ] Tool selector
- [ ] Temperature/token controls
- [ ] Side-by-side comparison
- [ ] Test various configurations

---

### Collaborative Filtering
**Status:** Not Started
**Effort:** 3-5 days
**Impact:** Low (Discovery)

**Implementation Tasks:**
- [ ] Track query patterns per user
- [ ] Implement collaborative filtering algorithm
- [ ] Suggest related queries after response
- [ ] Display "Users also asked" section
- [ ] Test recommendation quality

---

### Conversation Encryption
**Status:** Not Started
**Effort:** 2-3 days
**Impact:** Medium (Security)

**Implementation Tasks:**
- [ ] Implement encryption/decryption service
- [ ] Generate user-specific encryption keys
- [ ] Encrypt messages before storing
- [ ] Decrypt on retrieval
- [ ] Test performance impact

---

## 📊 Progress Tracker

**Quick Wins:** 0/4 completed (0%)
**Medium Effort:** 0/4 completed (0%)
**Long-term:** 0/4 completed (0%)
**Additional Features:** 0/15 completed (0%)

**Overall Progress:** 0/27 completed (0%)

---

## 🎯 Recommended Implementation Order

1. **Conversation Persistence** - Foundation for other features
2. **Suggested Follow-up Questions** - Quick UX win
3. **Feedback & Rating System** - Quality tracking
4. **Usage Analytics Dashboard** - Monitor costs
5. **Response Streaming** - Better UX
6. **Export Conversations** - User requested feature
7. **Quick Actions** - Workflow optimization
8. **Multi-Provider Fallback** - Reliability
9. **Query Categorization** - Better analytics
10. **Smart Context Management** - Better conversations

---

## 📝 Notes

- All features should follow the Controller → Application → Repository pattern
- Regenerate `ManagingApi.ts` after adding new endpoints: `cd src/Managing.Nswag && dotnet build`
- Use MongoDB for document storage, InfluxDB for time-series metrics
- Test all features with real user scenarios
- Consider token costs when implementing LLM-heavy features (summarization, titling)
- Ensure all features respect user privacy and data security

---

**Last Updated:** 2026-01-07
**Maintained By:** Development Team