346 lines
8.9 KiB
Markdown
346 lines
8.9 KiB
Markdown
# InfluxDB Export and Import Scripts
|
|
|
|
This directory contains scripts for exporting and importing InfluxDB data for the Managing Apps project using query-based methods that work with standard read/write tokens.
|
|
|
|
## Prerequisites
|
|
|
|
1. **InfluxDB CLI** - Required for export/import operations
|
|
```bash
|
|
brew install influxdb-cli
|
|
```
|
|
|
|
2. **jq** - JSON parser for reading configuration files
|
|
```bash
|
|
brew install jq
|
|
```
|
|
|
|
## Available Scripts
|
|
|
|
### 1. `export-prices-bucket.sh`
|
|
Exports OHLCV candle/price data from the `prices-bucket`.
|
|
|
|
**What it exports:**
|
|
- All candle data (open, high, low, close, volume)
|
|
- Multiple exchanges, tickers, and timeframes
|
|
- Configurable time ranges
|
|
|
|
**Usage:**
|
|
```bash
|
|
./export-prices-bucket.sh
|
|
```
|
|
|
|
**Interactive Prompts:**
|
|
- Select environment (SandboxLocal or ProductionLocal)
|
|
- Select time range (7 days, 30 days, 90 days, 1 year, all data, or custom)
|
|
|
|
**Output:**
|
|
- CSV export: `./exports/<ENVIRONMENT>/<TIMESTAMP>/prices-bucket_data.csv`
|
|
- Metadata file with export details
|
|
|
|
**Advantages:**
|
|
- ✅ Works with regular read tokens (no admin required)
|
|
- ✅ Flexible time range selection
|
|
- ✅ Exports in standard CSV format
|
|
- ✅ Can be imported to any InfluxDB instance
|
|
|
|
---
|
|
|
|
### 2. `import-csv-data.sh`
|
|
Imports prices-bucket CSV export data into any InfluxDB environment.
|
|
|
|
**What it imports:**
|
|
- Prices-bucket data only
|
|
- Supports large files (1.6M+ data points)
|
|
- Automatically creates bucket if needed
|
|
|
|
**Usage:**
|
|
```bash
|
|
./import-csv-data.sh
|
|
```
|
|
|
|
**Interactive Prompts:**
|
|
1. Select source environment (which export to import from)
|
|
2. Select export timestamp
|
|
3. Select target environment (where to import to)
|
|
4. Confirm the import operation
|
|
|
|
**Features:**
|
|
- ✅ Imports CSV exports to any environment
|
|
- ✅ Works with regular read/write tokens
|
|
- ✅ Batch processing for large files (5000 points per batch)
|
|
- ✅ Automatic bucket creation if needed
|
|
- ✅ Progress tracking for large imports
|
|
|
|
**⚠️ Note:** Import adds data to the bucket. Existing data with the same timestamps will be overwritten.
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
The scripts automatically read InfluxDB connection settings from:
|
|
- `src/Managing.Api/appsettings.SandboxLocal.json`
|
|
- `src/Managing.Api/appsettings.ProductionLocal.json`
|
|
|
|
**Required settings in appsettings files:**
|
|
```json
|
|
{
|
|
"InfluxDb": {
|
|
"Url": "https://influx-db.apps.managing.live",
|
|
"Organization": "managing-org",
|
|
"Token": "your-token-here"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Export/Import Structure
|
|
|
|
```
|
|
exports/
|
|
├── SandboxLocal/
|
|
│ └── 20241028_143022/
|
|
│ ├── prices-bucket_data.csv
|
|
│ └── export-metadata.txt
|
|
└── ProductionLocal/
|
|
└── 20241028_160000/
|
|
├── prices-bucket_data.csv
|
|
└── export-metadata.txt
|
|
```
|
|
|
|
## Data Structure
|
|
|
|
### prices-bucket (Managed by these scripts)
|
|
- **Measurement**: `price`
|
|
- **Contains**: OHLCV candle data
|
|
- **Tags**:
|
|
- `exchange` (e.g., Evm, Binance)
|
|
- `ticker` (e.g., BTC, ETH, AAVE)
|
|
- `timeframe` (e.g., FifteenMinutes, OneHour, OneDay)
|
|
- **Fields**:
|
|
- `open`, `high`, `low`, `close` (price values)
|
|
- `baseVolume`, `quoteVolume` (volume data)
|
|
- `TradeCount` (number of trades)
|
|
- `takerBuyBaseVolume`, `takerBuyQuoteVolume` (taker buy volumes)
|
|
|
|
### agent-balances-bucket (Not included in export/import scripts)
|
|
- **Measurement**: `agent_balance`
|
|
- **Contains**: User balance history over time
|
|
- **Note**: This bucket is not managed by these scripts. Balance data is derived from operational data and should be regenerated rather than migrated.
|
|
|
|
## Common Workflows
|
|
|
|
### Quick Export
|
|
```bash
|
|
cd scripts/influxdb
|
|
./export-prices-bucket.sh
|
|
# Select: 1 (SandboxLocal)
|
|
# Select: 5 (All data)
|
|
```
|
|
|
|
### Export Specific Time Range
|
|
```bash
|
|
cd scripts/influxdb
|
|
./export-prices-bucket.sh
|
|
# Select: 1 (SandboxLocal)
|
|
# Select: 3 (Last 90 days)
|
|
```
|
|
|
|
### Migrate Sandbox to Production
|
|
```bash
|
|
cd scripts/influxdb
|
|
|
|
# Step 1: Export from sandbox
|
|
./export-prices-bucket.sh
|
|
# Select: 1 (SandboxLocal)
|
|
# Select: 5 (All data)
|
|
|
|
# Step 2: Import to production
|
|
./import-csv-data.sh
|
|
# Select source: 1 (SandboxLocal)
|
|
# Select: Latest export timestamp
|
|
# Select target: 2 (ProductionLocal)
|
|
# Confirm: yes
|
|
```
|
|
|
|
### Backup Before Major Changes
|
|
```bash
|
|
cd scripts/influxdb
|
|
|
|
# Export current production data
|
|
./export-prices-bucket.sh
|
|
# Select: 2 (ProductionLocal)
|
|
# Select: 5 (All data)
|
|
|
|
# If something goes wrong, restore it:
|
|
./import-csv-data.sh
|
|
# Select the backup you just created
|
|
```
|
|
|
|
### Clone Environment
|
|
```bash
|
|
# Export from source
|
|
./export-prices-bucket.sh
|
|
# Select source environment
|
|
|
|
# Import to target
|
|
./import-csv-data.sh
|
|
# Select target environment
|
|
```
|
|
|
|
## Token Permissions
|
|
|
|
### Read Token (Export)
|
|
Required for `export-prices-bucket.sh`:
|
|
- ✅ Read access to buckets
|
|
- ✅ This is what you typically have in production
|
|
|
|
### Write Token (Import)
|
|
Required for `import-csv-data.sh`:
|
|
- ✅ Read/Write access to target bucket
|
|
- ✅ Ability to create buckets (optional, for auto-creation)
|
|
|
|
### How to Check Your Token Permissions
|
|
```bash
|
|
influx auth list --host <URL> --token <TOKEN>
|
|
```
|
|
|
|
## Data Retention
|
|
|
|
- Exports are stored indefinitely by default
|
|
- Manual cleanup recommended:
|
|
```bash
|
|
# Remove exports older than 90 days
|
|
find ./exports -type d -mtime +90 -exec rm -rf {} +
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### "influx command not found"
|
|
Install InfluxDB CLI:
|
|
```bash
|
|
brew install influxdb-cli
|
|
```
|
|
|
|
### "jq command not found"
|
|
Install jq:
|
|
```bash
|
|
brew install jq
|
|
```
|
|
|
|
### "Failed to parse configuration"
|
|
Ensure the appsettings JSON file exists and is valid JSON.
|
|
|
|
### "Connection refused"
|
|
- Check that InfluxDB URL is accessible
|
|
- Verify network connectivity to the server
|
|
- Check firewall rules
|
|
|
|
### "401 Unauthorized"
|
|
- Verify the InfluxDB token in appsettings is correct
|
|
- For exports: ensure token has read permissions for the bucket
|
|
- For imports: ensure token has write permissions for the bucket
|
|
|
|
### "Bucket not found"
|
|
The import script will automatically create the bucket if you have permissions.
|
|
|
|
Or create it manually:
|
|
```bash
|
|
influx bucket create \
|
|
--name prices-bucket \
|
|
--org managing-org \
|
|
--retention 0 \
|
|
--host https://influx-db.apps.managing.live \
|
|
--token YOUR_TOKEN
|
|
```
|
|
|
|
### Import is slow
|
|
- This is normal for large files (240MB+ with 1.6M+ data points)
|
|
- Expected time: 5-15 minutes depending on network speed
|
|
- The script processes data in batches of 5000 points
|
|
- Progress is shown during import
|
|
|
|
### Duplicate data after import
|
|
- Imports overwrite data with the same timestamp
|
|
- To avoid duplicates, don't import the same data twice
|
|
- To replace all data: delete the bucket first, then import
|
|
|
|
## Performance Tips
|
|
|
|
### For Large Exports
|
|
- Export specific time ranges instead of all data when possible
|
|
- Exports are faster than full database dumps
|
|
- CSV files compress well (use `gzip` for storage)
|
|
|
|
### For Large Imports
|
|
- Import during low-traffic periods
|
|
- Monitor InfluxDB memory usage during import
|
|
- Consider splitting very large imports into time ranges
|
|
|
|
## Verify Data After Import
|
|
|
|
```bash
|
|
# Check recent data
|
|
influx query 'from(bucket:"prices-bucket") |> range(start:-7d) |> limit(n:10)' \
|
|
--host https://influx-db.apps.managing.live \
|
|
--org managing-org \
|
|
--token YOUR_TOKEN
|
|
|
|
# Count total records
|
|
influx query 'from(bucket:"prices-bucket") |> range(start:2020-01-01T00:00:00Z) |> count()' \
|
|
--host https://influx-db.apps.managing.live \
|
|
--org managing-org \
|
|
--token YOUR_TOKEN
|
|
|
|
# Check specific ticker
|
|
influx query 'from(bucket:"prices-bucket") |> range(start:-30d) |> filter(fn: (r) => r.ticker == "BTC")' \
|
|
--host https://influx-db.apps.managing.live \
|
|
--org managing-org \
|
|
--token YOUR_TOKEN
|
|
```
|
|
|
|
## Manual Export/Import Commands
|
|
|
|
If you need to run commands manually:
|
|
|
|
**Export:**
|
|
```bash
|
|
influx query 'from(bucket: "prices-bucket") |> range(start: -30d)' \
|
|
--host https://influx-db.apps.managing.live \
|
|
--org managing-org \
|
|
--token YOUR_TOKEN \
|
|
--raw > export.csv
|
|
```
|
|
|
|
**Import:**
|
|
```bash
|
|
influx write \
|
|
--host https://influx-db.apps.managing.live \
|
|
--org managing-org \
|
|
--token YOUR_TOKEN \
|
|
--bucket prices-bucket \
|
|
--format csv \
|
|
--file export.csv
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Regular Exports**: Schedule regular exports of production data
|
|
2. **Test Imports**: Test imports on sandbox before production
|
|
3. **Verify After Import**: Always verify data integrity after import
|
|
4. **Document Changes**: Keep notes of what data was imported when
|
|
5. **Backup Before Major Changes**: Export before major data operations
|
|
|
|
## Support
|
|
|
|
For issues or questions, refer to:
|
|
- [InfluxDB Query Documentation](https://docs.influxdata.com/influxdb/v2.0/query-data/)
|
|
- [InfluxDB Write Documentation](https://docs.influxdata.com/influxdb/v2.0/write-data/)
|
|
- Project documentation in `/docs`
|
|
|
|
## Script Locations
|
|
|
|
All scripts are located in: `/Users/oda/Desktop/Projects/managing-apps/scripts/influxdb/`
|
|
|
|
Configuration files:
|
|
- Sandbox: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.SandboxLocal.json`
|
|
- Production: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.ProductionLocal.json`
|