managing-apps/LLM_IMPROVEMENTS_TODO.md

LLM Controller - Feature Improvements Roadmap

🎯 Quick Wins (1-2 days)

✅ Priority 1: Suggested Follow-up Questions

Status: Not Started Effort: 4-6 hours Impact: High

Description: After each response, the LLM suggests 3-5 relevant follow-up questions to guide user exploration.

Implementation Tasks:

• Update BuildSystemMessage() to include follow-up question instruction
• Add SuggestedQuestions property to LlmProgressUpdate class
• Create ExtractFollowUpQuestions() method to parse questions from response
• Update ChatStreamInternal() to extract and send suggested questions
• Update frontend to display suggested questions as clickable chips
• Test with various query types (backtest, indicator, general finance)

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs
• src/Managing.Application.Abstractions/Services/ILlmService.cs
• Frontend components (AiChat.tsx)

---

✅ Priority 2: Feedback & Rating System

Status: Not Started Effort: 6-8 hours Impact: High (Quality tracking)

Description: Users can rate LLM responses (👍👎) with optional comments to track quality and improve prompts.

Implementation Tasks:

• Create LlmFeedback domain model (ResponseId, UserId, Rating, Comment, Timestamp)
• Create ILlmFeedbackRepository interface
• Implement LlmFeedbackRepository with MongoDB
• Add ResponseId property to LlmChatResponse
• Create new endpoint: POST /Llm/Feedback
• Create new endpoint: GET /Llm/Analytics/Feedback
• Update frontend to show 👍👎 buttons after each response
• Create analytics dashboard to view feedback trends

Files to Create:

• src/Managing.Domain/Llm/LlmFeedback.cs
• src/Managing.Application.Abstractions/Repositories/ILlmFeedbackRepository.cs
• src/Managing.Infrastructure/Repositories/LlmFeedbackRepository.cs
• src/Managing.Application/Services/LlmFeedbackService.cs

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs
• src/Managing.Application.Abstractions/Services/ILlmService.cs

---

✅ Priority 3: Export Conversations

Status: Not Started Effort: 4-6 hours Impact: Medium

Description: Export conversation to Markdown, JSON, or PDF for reporting and sharing.

Implementation Tasks:

• Create IConversationExportService interface
• Implement Markdown export (simple format with messages)
• Implement JSON export (structured data)
• Implement PDF export using QuestPDF or similar
• Create new endpoint: GET /Llm/Conversations/{id}/Export?format={md|json|pdf}
• Add "Export" button to conversation UI
• Test with long conversations and special characters

Files to Create:

• src/Managing.Application/Services/ConversationExportService.cs
• src/Managing.Application.Abstractions/Services/IConversationExportService.cs

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs

---

✅ Priority 4: Query Categorization

Status: Not Started Effort: 3-4 hours Impact: Medium (Better analytics)

Description: Automatically categorize queries (BacktestAnalysis, GeneralFinance, etc.) for analytics.

Implementation Tasks:

• Create QueryCategory enum (BacktestAnalysis, BundleAnalysis, IndicatorQuestion, GeneralFinance, HowTo, DataRetrieval, Comparison)
• Add QueryCategory property to LlmProgressUpdate
• Create DetermineQueryCategory() method using keyword matching
• Update system prompt to include category in response
• Send category in initial progress update
• Track category distribution in analytics

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs
• src/Managing.Application.Abstractions/Services/ILlmService.cs

---

🚀 Medium Effort (3-5 days)

✅ Priority 5: Conversation Persistence

Status: Not Started Effort: 2-3 days Impact: Very High (Core feature)

Description: Save conversation history to database so users can resume previous conversations across sessions.

Implementation Tasks:

• Create ChatConversation domain model (Id, UserId, Title, CreatedAt, UpdatedAt, LastMessageAt)
• Create ChatMessage domain model (Id, ConversationId, Role, Content, Timestamp, TokenCount, ToolCalls)
• Create IChatConversationRepository interface
• Implement ChatConversationRepository with MongoDB
• Create IChatMessageRepository interface
• Implement ChatMessageRepository with MongoDB
• Create new endpoint: GET /Llm/Conversations (list user's conversations)
• Create new endpoint: GET /Llm/Conversations/{id} (get conversation with messages)
• Create new endpoint: POST /Llm/Conversations (create new conversation)
• Create new endpoint: POST /Llm/Conversations/{id}/Messages (add message to conversation)
• Create new endpoint: DELETE /Llm/Conversations/{id} (delete conversation)
• Update ChatStream to save messages automatically
• Create conversation list UI component
• Add "New Conversation" button
• Add conversation sidebar with search/filter
• Test with multiple concurrent conversations

Files to Create:

• src/Managing.Domain/Llm/ChatConversation.cs
• src/Managing.Domain/Llm/ChatMessage.cs
• src/Managing.Application.Abstractions/Repositories/IChatConversationRepository.cs
• src/Managing.Application.Abstractions/Repositories/IChatMessageRepository.cs
• src/Managing.Infrastructure/Repositories/ChatConversationRepository.cs
• src/Managing.Infrastructure/Repositories/ChatMessageRepository.cs
• src/Managing.Application/Services/ChatConversationService.cs

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs

---

✅ Priority 6: Response Streaming (Token-by-Token)

Status: Not Started Effort: 2-3 days Impact: High (UX improvement)

Description: Stream LLM response as tokens arrive instead of waiting for complete response.

Implementation Tasks:

• Update ILlmService.ChatAsync() to return IAsyncEnumerable<LlmTokenChunk>
• Modify LLM provider implementations to support streaming
• Update ChatStreamInternal() to stream tokens via SignalR
• Add new progress update type: "token_stream"
• Update frontend to display streaming tokens with typing animation
• Handle tool calls during streaming (partial JSON parsing)
• Add "Stop Generation" button in UI
• Test with different LLM providers

Files to Modify:

• src/Managing.Application.Abstractions/Services/ILlmService.cs
• src/Managing.Application/Services/LlmService.cs (or provider-specific implementations)
• src/Managing.Api/Controllers/LlmController.cs
• Frontend components (AiChat.tsx)

---

✅ Priority 7: Usage Analytics Dashboard

Status: Not Started Effort: 2-3 days Impact: High (Cost monitoring)

Description: Track and visualize LLM usage metrics (tokens, cost, performance).

Implementation Tasks:

• Create LlmUsageMetric domain model (UserId, Timestamp, Provider, Model, PromptTokens, CompletionTokens, Cost, Duration, QueryCategory)
• Create ILlmUsageRepository interface
• Implement LlmUsageRepository with InfluxDB (time-series data)
• Update ChatStreamInternal() to log usage metrics
• Create new endpoint: GET /Llm/Analytics/Usage (token usage over time)
• Create new endpoint: GET /Llm/Analytics/PopularTools (most called tools)
• Create new endpoint: GET /Llm/Analytics/AverageIterations (performance metrics)
• Create new endpoint: GET /Llm/Analytics/ErrorRate (quality metrics)
• Create new endpoint: GET /Llm/Analytics/CostEstimate (current month cost)
• Create analytics dashboard component with charts (Chart.js or Recharts)
• Add filters: date range, category, provider
• Display key metrics: total tokens, cost, avg response time
• Test with large datasets

Files to Create:

• src/Managing.Domain/Llm/LlmUsageMetric.cs
• src/Managing.Application.Abstractions/Repositories/ILlmUsageRepository.cs
• src/Managing.Infrastructure/Repositories/LlmUsageRepository.cs
• src/Managing.Application/Services/LlmAnalyticsService.cs

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs

---

✅ Priority 8: Quick Actions / Shortcuts

Status: Not Started Effort: 2-3 days Impact: Medium (Workflow improvement)

Description: Recognize patterns and offer action buttons (e.g., "Delete this backtest" after analysis).

Implementation Tasks:

• Create QuickAction model (Id, Label, Icon, Endpoint, Parameters)
• Add Actions property to LlmProgressUpdate
• Create GenerateQuickActions() method based on context
• Update system prompt to suggest actions in structured format
• Parse action suggestions from LLM response
• Update frontend to display action buttons
• Implement action handlers (call APIs)
• Add confirmation dialogs for destructive actions
• Test with various scenarios (backtest, bundle, indicator)

Example Actions:

• After backtest analysis: "Delete this backtest", "Run similar backtest", "Export details"
• After bundle analysis: "Delete bundle", "Run again with different params"
• After list query: "Export to CSV", "Show details"

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs
• src/Managing.Application.Abstractions/Services/ILlmService.cs
• Frontend components (AiChat.tsx)

---

🎨 Long-term (1-2 weeks)

✅ Priority 9: Multi-Provider Fallback

Status: Not Started Effort: 3-5 days Impact: High (Reliability)

Description: Automatically fallback to alternative LLM provider on failure or rate limit.

Implementation Tasks:

• Create LlmProviderHealth model to track provider status
• Create IProviderHealthMonitor service
• Implement health check mechanism (ping providers periodically)
• Create provider priority list configuration
• Update LlmService.ChatAsync() to implement fallback logic
• Add retry logic with exponential backoff
• Track provider failure rates
• Send alert when provider is down
• Update frontend to show current provider
• Test failover scenarios

Provider Priority Example:

1. Primary: OpenAI GPT-4
2. Secondary: Anthropic Claude
3. Tertiary: Google Gemini
4. Fallback: Local model (if available)

Files to Create:

• src/Managing.Application/Services/ProviderHealthMonitor.cs
• src/Managing.Domain/Llm/LlmProviderHealth.cs

Files to Modify:

• src/Managing.Application/Services/LlmService.cs

---

✅ Priority 10: Scheduled Queries / Alerts

Status: Not Started Effort: 4-6 days Impact: High (Automation)

Description: Run queries on schedule and notify users of changes (e.g., "Alert when backtest scores > 80").

Implementation Tasks:

• Create LlmAlert domain model (Id, UserId, Query, Schedule, Condition, IsActive, LastRun, CreatedAt)
• Create ILlmAlertRepository interface
• Implement LlmAlertRepository with MongoDB
• Create background service to process alerts (Hangfire or Quartz.NET)
• Create new endpoint: POST /Llm/Alerts (create alert)
• Create new endpoint: GET /Llm/Alerts (list user's alerts)
• Create new endpoint: PUT /Llm/Alerts/{id} (update alert)
• Create new endpoint: DELETE /Llm/Alerts/{id} (delete alert)
• Implement notification system (SignalR, email, push)
• Create alert management UI
• Add schedule picker (cron expression builder)
• Test with various schedules and conditions

Example Alerts:

• "Notify me when a backtest scores > 80" (run every hour)
• "Daily summary of new backtests" (run at 9am daily)
• "Alert when bundle completes" (run every 5 minutes)

Files to Create:

• src/Managing.Domain/Llm/LlmAlert.cs
• src/Managing.Application.Abstractions/Repositories/ILlmAlertRepository.cs
• src/Managing.Infrastructure/Repositories/LlmAlertRepository.cs
• src/Managing.Application/Services/LlmAlertService.cs
• src/Managing.Application/BackgroundServices/LlmAlertProcessor.cs

---

✅ Priority 11: Smart Context Window Management

Status: Not Started Effort: 3-5 days Impact: Medium (Better conversations)

Description: Intelligently compress conversation history instead of simple truncation.

Implementation Tasks:

• Research and implement summarization approach (LLM-based or extractive)
• Create SummarizeConversation() method
• Update TrimConversationContext() to use summarization
• Preserve key entities (IDs, numbers, dates)
• Use embeddings to identify relevant context (optional, advanced)
• Test with long conversations (50+ messages)
• Measure token savings vs quality trade-off
• Add configuration for compression strategy

Approaches:

1. Simple: Summarize every N old messages into single message
2. Advanced: Use embeddings to keep semantically relevant messages
3. Hybrid: Keep recent messages + summarized older messages + key facts

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs

---

✅ Priority 12: Interactive Clarification Questions

Status: Not Started Effort: 3-4 days Impact: Medium (Reduce back-and-forth)

Description: When ambiguous, LLM asks structured multiple-choice questions instead of open-ended text.

Implementation Tasks:

• Create ClarificationOption model (Id, Label, Description)
• Add Options property to LlmProgressUpdate
• Update system prompt to output clarification questions in structured format
• Create ExtractClarificationOptions() method
• Update ChatStreamInternal() to handle clarification state
• Update frontend to display radio buttons/chips for options
• Handle user selection (send as next message automatically)
• Test with ambiguous queries

Example: User: "Show me the backtest" LLM: "Which backtest would you like to see?"

• 🔘 Best performing backtest
• 🔘 Most recent backtest
• 🔘 Specific backtest by name

Files to Create:

• src/Managing.Domain/Llm/ClarificationOption.cs

Files to Modify:

• src/Managing.Api/Controllers/LlmController.cs
• src/Managing.Application.Abstractions/Services/ILlmService.cs
• Frontend components (AiChat.tsx)

---

🔧 Additional Features (Nice to Have)

Voice Input Support

Status: Not Started Effort: 2-3 days Impact: Medium

Implementation Tasks:

• Create new endpoint: POST /Llm/VoiceChat (accept audio file)
• Integrate speech-to-text service (Azure Speech, OpenAI Whisper)
• Process transcribed text as normal chat
• Add microphone button in frontend
• Handle audio recording in browser
• Test with various audio formats and accents

---

Smart Conversation Titling

Status: Not Started Effort: 2-3 hours Impact: Low (QoL)

Implementation Tasks:

• After first response, send summary request to LLM
• Update conversation title in background
• Don't block user while generating title
• Test with various conversation types

---

Tool Call Caching

Status: Not Started Effort: 1-2 days Impact: Medium (Performance)

Implementation Tasks:

• Create cache key hash function (toolName + arguments)
• Implement cache wrapper around ExecuteToolAsync()
• Configure cache duration per tool type
• Invalidate cache on data mutations
• Test cache hit/miss rates

---

Conversation Branching

Status: Not Started Effort: 2-3 days Impact: Low (Power user feature)

Implementation Tasks:

• Create new endpoint: POST /Llm/Conversations/{id}/Branch?fromMessageId={id}
• Copy conversation history up to branch point
• Create new conversation with copied history
• Update UI to show branch option on messages
• Test branching at various points

---

LLM Model Selection

Status: Not Started Effort: 1-2 days Impact: Medium (Cost control)

Implementation Tasks:

• Add PreferredModel property to LlmChatRequest
• Create model configuration (pricing, speed, quality scores)
• Update frontend with model selector dropdown
• Display model info (cost, speed, quality)
• Test with different models

---

Debug Mode

Status: Not Started Effort: 4-6 hours Impact: Low (Developer tool)

Implementation Tasks:

• Add Debug property to LlmChatRequest
• Return full prompt, raw response, token breakdown when debug=true
• Create debug panel in UI
• Add toggle to enable/disable debug mode
• Test with various queries

---

PII Detection & Redaction

Status: Not Started Effort: 2-3 days Impact: Medium (Security)

Implementation Tasks:

• Implement PII detection regex (email, phone, SSN, credit card)
• Scan messages before sending to LLM
• Warn user about detected PII
• Option to redact or anonymize
• Test with various PII patterns

---

Rate Limiting Per User

Status: Not Started Effort: 1-2 days Impact: Medium (Cost control)

Implementation Tasks:

• Create rate limit configuration (requests/hour, tokens/day)
• Implement rate limit middleware
• Track usage per user
• Return 429 with quota info when exceeded
• Display quota usage in UI

---

Request Queueing

Status: Not Started Effort: 2-3 days Impact: Medium (Reliability)

Implementation Tasks:

• Implement request queue with priority
• Queue requests when rate limited
• Send position-in-queue updates via SignalR
• Process queue when rate limit resets
• Test with high load

---

Prompt Version Control

Status: Not Started Effort: 2-3 days Impact: Low (Optimization)

Implementation Tasks:

• Create SystemPrompt model (Version, Content, CreatedAt, IsActive, SuccessRate)
• Store multiple prompt versions
• A/B test prompts (rotate per conversation)
• Track success metrics per prompt version
• UI to manage prompt versions

---

LLM Playground

Status: Not Started Effort: 3-4 days Impact: Low (Developer tool)

Implementation Tasks:

• Create playground UI component
• System prompt editor with syntax highlighting
• Message history builder
• Tool selector
• Temperature/token controls
• Side-by-side comparison
• Test various configurations

---

Collaborative Filtering

Status: Not Started Effort: 3-5 days Impact: Low (Discovery)

Implementation Tasks:

• Track query patterns per user
• Implement collaborative filtering algorithm
• Suggest related queries after response
• Display "Users also asked" section
• Test recommendation quality

---

Conversation Encryption

Status: Not Started Effort: 2-3 days Impact: Medium (Security)

Implementation Tasks:

• Implement encryption/decryption service
• Generate user-specific encryption keys
• Encrypt messages before storing
• Decrypt on retrieval
• Test performance impact

---

📊 Progress Tracker

Quick Wins: 0/4 completed (0%) Medium Effort: 0/4 completed (0%) Long-term: 0/4 completed (0%) Additional Features: 0/15 completed (0%)

Overall Progress: 0/27 completed (0%)

---

🎯 Recommended Implementation Order

1. Conversation Persistence - Foundation for other features
2. Suggested Follow-up Questions - Quick UX win
3. Feedback & Rating System - Quality tracking
4. Usage Analytics Dashboard - Monitor costs
5. Response Streaming - Better UX
6. Export Conversations - User requested feature
7. Quick Actions - Workflow optimization
8. Multi-Provider Fallback - Reliability
9. Query Categorization - Better analytics
10. Smart Context Management - Better conversations

---

📝 Notes

• All features should follow the Controller → Application → Repository pattern
• Regenerate ManagingApi.ts after adding new endpoints: cd src/Managing.Nswag && dotnet build
• Use MongoDB for document storage, InfluxDB for time-series metrics
• Test all features with real user scenarios
• Consider token costs when implementing LLM-heavy features (summarization, titling)
• Ensure all features respect user privacy and data security

---

Last Updated: 2026-01-07 Maintained By: Development Team