Add influxdb export/import + add new influxdb instance for prod

2025-10-28 12:56:42 +07:00
parent ffe1bed051
commit 5cef270d64
5 changed files with 993 additions and 2 deletions
--- a/scripts/influxdb/README.md
+++ b/scripts/influxdb/README.md
@@ -0,0 +1,345 @@
+# InfluxDB Export and Import Scripts
+
+This directory contains scripts for exporting and importing InfluxDB data for the Managing Apps project using query-based methods that work with standard read/write tokens.
+
+## Prerequisites
+
+1. **InfluxDB CLI** - Required for export/import operations
+   ```bash
+   brew install influxdb-cli
+   ```
+
+2. **jq** - JSON parser for reading configuration files
+   ```bash
+   brew install jq
+   ```
+
+## Available Scripts
+
+### 1. `export-prices-bucket.sh`
+Exports OHLCV candle/price data from the `prices-bucket`.
+
+**What it exports:**
+- All candle data (open, high, low, close, volume)
+- Multiple exchanges, tickers, and timeframes
+- Configurable time ranges
+
+**Usage:**
+```bash
+./export-prices-bucket.sh
+```
+
+**Interactive Prompts:**
+- Select environment (SandboxLocal or ProductionLocal)
+- Select time range (7 days, 30 days, 90 days, 1 year, all data, or custom)
+
+**Output:**
+- CSV export: `./exports/<ENVIRONMENT>/<TIMESTAMP>/prices-bucket_data.csv`
+- Metadata file with export details
+
+**Advantages:**
+- ✅ Works with regular read tokens (no admin required)
+- ✅ Flexible time range selection
+- ✅ Exports in standard CSV format
+- ✅ Can be imported to any InfluxDB instance
+
+---
+
+### 2. `import-csv-data.sh`
+Imports prices-bucket CSV export data into any InfluxDB environment.
+
+**What it imports:**
+- Prices-bucket data only
+- Supports large files (1.6M+ data points)
+- Automatically creates bucket if needed
+
+**Usage:**
+```bash
+./import-csv-data.sh
+```
+
+**Interactive Prompts:**
+1. Select source environment (which export to import from)
+2. Select export timestamp
+3. Select target environment (where to import to)
+4. Confirm the import operation
+
+**Features:**
+- ✅ Imports CSV exports to any environment
+- ✅ Works with regular read/write tokens
+- ✅ Batch processing for large files (5000 points per batch)
+- ✅ Automatic bucket creation if needed
+- ✅ Progress tracking for large imports
+
+**⚠️ Note:** Import adds data to the bucket. Existing data with the same timestamps will be overwritten.
+
+---
+
+## Configuration
+
+The scripts automatically read InfluxDB connection settings from:
+- `src/Managing.Api/appsettings.SandboxLocal.json`
+- `src/Managing.Api/appsettings.ProductionLocal.json`
+
+**Required settings in appsettings files:**
+```json
+{
+  "InfluxDb": {
+    "Url": "https://influx-db.apps.managing.live",
+    "Organization": "managing-org",
+    "Token": "your-token-here"
+  }
+}
+```
+
+## Export/Import Structure
+
+```
+exports/
+├── SandboxLocal/
+│   └── 20241028_143022/
+│       ├── prices-bucket_data.csv
+│       └── export-metadata.txt
+└── ProductionLocal/
+    └── 20241028_160000/
+        ├── prices-bucket_data.csv
+        └── export-metadata.txt
+```
+
+## Data Structure
+
+### prices-bucket (Managed by these scripts)
+- **Measurement**: `price`
+- **Contains**: OHLCV candle data
+- **Tags**: 
+  - `exchange` (e.g., Evm, Binance)
+  - `ticker` (e.g., BTC, ETH, AAVE)
+  - `timeframe` (e.g., FifteenMinutes, OneHour, OneDay)
+- **Fields**:
+  - `open`, `high`, `low`, `close` (price values)
+  - `baseVolume`, `quoteVolume` (volume data)
+  - `TradeCount` (number of trades)
+  - `takerBuyBaseVolume`, `takerBuyQuoteVolume` (taker buy volumes)
+
+### agent-balances-bucket (Not included in export/import scripts)
+- **Measurement**: `agent_balance`
+- **Contains**: User balance history over time
+- **Note**: This bucket is not managed by these scripts. Balance data is derived from operational data and should be regenerated rather than migrated.
+
+## Common Workflows
+
+### Quick Export
+```bash
+cd scripts/influxdb
+./export-prices-bucket.sh
+# Select: 1 (SandboxLocal)
+# Select: 5 (All data)
+```
+
+### Export Specific Time Range
+```bash
+cd scripts/influxdb
+./export-prices-bucket.sh
+# Select: 1 (SandboxLocal)
+# Select: 3 (Last 90 days)
+```
+
+### Migrate Sandbox to Production
+```bash
+cd scripts/influxdb
+
+# Step 1: Export from sandbox
+./export-prices-bucket.sh
+# Select: 1 (SandboxLocal)
+# Select: 5 (All data)
+
+# Step 2: Import to production
+./import-csv-data.sh
+# Select source: 1 (SandboxLocal)
+# Select: Latest export timestamp
+# Select target: 2 (ProductionLocal)
+# Confirm: yes
+```
+
+### Backup Before Major Changes
+```bash
+cd scripts/influxdb
+
+# Export current production data
+./export-prices-bucket.sh
+# Select: 2 (ProductionLocal)
+# Select: 5 (All data)
+
+# If something goes wrong, restore it:
+./import-csv-data.sh
+# Select the backup you just created
+```
+
+### Clone Environment
+```bash
+# Export from source
+./export-prices-bucket.sh
+# Select source environment
+
+# Import to target
+./import-csv-data.sh
+# Select target environment
+```
+
+## Token Permissions
+
+### Read Token (Export)
+Required for `export-prices-bucket.sh`:
+- ✅ Read access to buckets
+- ✅ This is what you typically have in production
+
+### Write Token (Import)
+Required for `import-csv-data.sh`:
+- ✅ Read/Write access to target bucket
+- ✅ Ability to create buckets (optional, for auto-creation)
+
+### How to Check Your Token Permissions
+```bash
+influx auth list --host <URL> --token <TOKEN>
+```
+
+## Data Retention
+
+- Exports are stored indefinitely by default
+- Manual cleanup recommended:
+  ```bash
+  # Remove exports older than 90 days
+  find ./exports -type d -mtime +90 -exec rm -rf {} +
+  ```
+
+## Troubleshooting
+
+### "influx command not found"
+Install InfluxDB CLI:
+```bash
+brew install influxdb-cli
+```
+
+### "jq command not found"
+Install jq:
+```bash
+brew install jq
+```
+
+### "Failed to parse configuration"
+Ensure the appsettings JSON file exists and is valid JSON.
+
+### "Connection refused"
+- Check that InfluxDB URL is accessible
+- Verify network connectivity to the server
+- Check firewall rules
+
+### "401 Unauthorized"
+- Verify the InfluxDB token in appsettings is correct
+- For exports: ensure token has read permissions for the bucket
+- For imports: ensure token has write permissions for the bucket
+
+### "Bucket not found"
+The import script will automatically create the bucket if you have permissions.
+
+Or create it manually:
+```bash
+influx bucket create \
+  --name prices-bucket \
+  --org managing-org \
+  --retention 0 \
+  --host https://influx-db.apps.managing.live \
+  --token YOUR_TOKEN
+```
+
+### Import is slow
+- This is normal for large files (240MB+ with 1.6M+ data points)
+- Expected time: 5-15 minutes depending on network speed
+- The script processes data in batches of 5000 points
+- Progress is shown during import
+
+### Duplicate data after import
+- Imports overwrite data with the same timestamp
+- To avoid duplicates, don't import the same data twice
+- To replace all data: delete the bucket first, then import
+
+## Performance Tips
+
+### For Large Exports
+- Export specific time ranges instead of all data when possible
+- Exports are faster than full database dumps
+- CSV files compress well (use `gzip` for storage)
+
+### For Large Imports
+- Import during low-traffic periods
+- Monitor InfluxDB memory usage during import
+- Consider splitting very large imports into time ranges
+
+## Verify Data After Import
+
+```bash
+# Check recent data
+influx query 'from(bucket:"prices-bucket") |> range(start:-7d) |> limit(n:10)' \
+  --host https://influx-db.apps.managing.live \
+  --org managing-org \
+  --token YOUR_TOKEN
+
+# Count total records
+influx query 'from(bucket:"prices-bucket") |> range(start:2020-01-01T00:00:00Z) |> count()' \
+  --host https://influx-db.apps.managing.live \
+  --org managing-org \
+  --token YOUR_TOKEN
+
+# Check specific ticker
+influx query 'from(bucket:"prices-bucket") |> range(start:-30d) |> filter(fn: (r) => r.ticker == "BTC")' \
+  --host https://influx-db.apps.managing.live \
+  --org managing-org \
+  --token YOUR_TOKEN
+```
+
+## Manual Export/Import Commands
+
+If you need to run commands manually:
+
+**Export:**
+```bash
+influx query 'from(bucket: "prices-bucket") |> range(start: -30d)' \
+  --host https://influx-db.apps.managing.live \
+  --org managing-org \
+  --token YOUR_TOKEN \
+  --raw > export.csv
+```
+
+**Import:**
+```bash
+influx write \
+  --host https://influx-db.apps.managing.live \
+  --org managing-org \
+  --token YOUR_TOKEN \
+  --bucket prices-bucket \
+  --format csv \
+  --file export.csv
+```
+
+## Best Practices
+
+1. **Regular Exports**: Schedule regular exports of production data
+2. **Test Imports**: Test imports on sandbox before production
+3. **Verify After Import**: Always verify data integrity after import
+4. **Document Changes**: Keep notes of what data was imported when
+5. **Backup Before Major Changes**: Export before major data operations
+
+## Support
+
+For issues or questions, refer to:
+- [InfluxDB Query Documentation](https://docs.influxdata.com/influxdb/v2.0/query-data/)
+- [InfluxDB Write Documentation](https://docs.influxdata.com/influxdb/v2.0/write-data/)
+- Project documentation in `/docs`
+
+## Script Locations
+
+All scripts are located in: `/Users/oda/Desktop/Projects/managing-apps/scripts/influxdb/`
+
+Configuration files:
+- Sandbox: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.SandboxLocal.json`
+- Production: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.ProductionLocal.json`