# InfluxDB Export and Import Scripts

This directory contains scripts for exporting and importing InfluxDB data for the Managing Apps project using query-based methods that work with standard read/write tokens.

## Prerequisites

1. **InfluxDB CLI** - Required for export/import operations
   ```bash
   brew install influxdb-cli
   ```

2. **jq** - JSON parser for reading configuration files
   ```bash
   brew install jq
   ```

## Available Scripts

### 1. `export-prices-bucket.sh`
Exports OHLCV candle/price data from the `prices-bucket`.

**What it exports:**
- All candle data (open, high, low, close, volume)
- Multiple exchanges, tickers, and timeframes
- Configurable time ranges

**Usage:**
```bash
./export-prices-bucket.sh
```

**Interactive Prompts:**
- Select environment (SandboxLocal or ProductionLocal)
- Select time range (7 days, 30 days, 90 days, 1 year, all data, or custom)

**Output:**
- CSV export: `./exports/<ENVIRONMENT>/<TIMESTAMP>/prices-bucket_data.csv`
- Metadata file with export details

**Advantages:**
- ✅ Works with regular read tokens (no admin required)
- ✅ Flexible time range selection
- ✅ Exports in standard CSV format
- ✅ Can be imported to any InfluxDB instance

---

### 2. `import-csv-data.sh`
Imports prices-bucket CSV export data into any InfluxDB environment.

**What it imports:**
- Prices-bucket data only
- Supports large files (1.6M+ data points)
- Automatically creates bucket if needed

**Usage:**
```bash
./import-csv-data.sh
```

**Interactive Prompts:**
1. Select source environment (which export to import from)
2. Select export timestamp
3. Select target environment (where to import to)
4. Confirm the import operation

**Features:**
- ✅ Imports CSV exports to any environment
- ✅ Works with regular read/write tokens
- ✅ Batch processing for large files (5000 points per batch)
- ✅ Automatic bucket creation if needed
- ✅ Progress tracking for large imports

**⚠️ Note:** Import adds data to the bucket. Existing data with the same timestamps will be overwritten.

---

## Configuration

The scripts automatically read InfluxDB connection settings from:
- `src/Managing.Api/appsettings.SandboxLocal.json`
- `src/Managing.Api/appsettings.ProductionLocal.json`

**Required settings in appsettings files:**
```json
{
  "InfluxDb": {
    "Url": "https://influx-db.apps.managing.live",
    "Organization": "managing-org",
    "Token": "your-token-here"
  }
}
```

## Export/Import Structure

```
exports/
├── SandboxLocal/
│   └── 20241028_143022/
│       ├── prices-bucket_data.csv
│       └── export-metadata.txt
└── ProductionLocal/
    └── 20241028_160000/
        ├── prices-bucket_data.csv
        └── export-metadata.txt
```

## Data Structure

### prices-bucket (Managed by these scripts)
- **Measurement**: `price`
- **Contains**: OHLCV candle data
- **Tags**: 
  - `exchange` (e.g., Evm, Binance)
  - `ticker` (e.g., BTC, ETH, AAVE)
  - `timeframe` (e.g., FifteenMinutes, OneHour, OneDay)
- **Fields**:
  - `open`, `high`, `low`, `close` (price values)
  - `baseVolume`, `quoteVolume` (volume data)
  - `TradeCount` (number of trades)
  - `takerBuyBaseVolume`, `takerBuyQuoteVolume` (taker buy volumes)

### agent-balances-bucket (Not included in export/import scripts)
- **Measurement**: `agent_balance`
- **Contains**: User balance history over time
- **Note**: This bucket is not managed by these scripts. Balance data is derived from operational data and should be regenerated rather than migrated.

## Common Workflows

### Quick Export
```bash
cd scripts/influxdb
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 5 (All data)
```

### Export Specific Time Range
```bash
cd scripts/influxdb
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 3 (Last 90 days)
```

### Migrate Sandbox to Production
```bash
cd scripts/influxdb

# Step 1: Export from sandbox
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 5 (All data)

# Step 2: Import to production
./import-csv-data.sh
# Select source: 1 (SandboxLocal)
# Select: Latest export timestamp
# Select target: 2 (ProductionLocal)
# Confirm: yes
```

### Backup Before Major Changes
```bash
cd scripts/influxdb

# Export current production data
./export-prices-bucket.sh
# Select: 2 (ProductionLocal)
# Select: 5 (All data)

# If something goes wrong, restore it:
./import-csv-data.sh
# Select the backup you just created
```

### Clone Environment
```bash
# Export from source
./export-prices-bucket.sh
# Select source environment

# Import to target
./import-csv-data.sh
# Select target environment
```

## Token Permissions

### Read Token (Export)
Required for `export-prices-bucket.sh`:
- ✅ Read access to buckets
- ✅ This is what you typically have in production

### Write Token (Import)
Required for `import-csv-data.sh`:
- ✅ Read/Write access to target bucket
- ✅ Ability to create buckets (optional, for auto-creation)

### How to Check Your Token Permissions
```bash
influx auth list --host <URL> --token <TOKEN>
```

## Data Retention

- Exports are stored indefinitely by default
- Manual cleanup recommended:
  ```bash
  # Remove exports older than 90 days
  find ./exports -type d -mtime +90 -exec rm -rf {} +
  ```

## Troubleshooting

### "influx command not found"
Install InfluxDB CLI:
```bash
brew install influxdb-cli
```

### "jq command not found"
Install jq:
```bash
brew install jq
```

### "Failed to parse configuration"
Ensure the appsettings JSON file exists and is valid JSON.

### "Connection refused"
- Check that InfluxDB URL is accessible
- Verify network connectivity to the server
- Check firewall rules

### "401 Unauthorized"
- Verify the InfluxDB token in appsettings is correct
- For exports: ensure token has read permissions for the bucket
- For imports: ensure token has write permissions for the bucket

### "Bucket not found"
The import script will automatically create the bucket if you have permissions.

Or create it manually:
```bash
influx bucket create \
  --name prices-bucket \
  --org managing-org \
  --retention 0 \
  --host https://influx-db.apps.managing.live \
  --token YOUR_TOKEN
```

### Import is slow
- This is normal for large files (240MB+ with 1.6M+ data points)
- Expected time: 5-15 minutes depending on network speed
- The script processes data in batches of 5000 points
- Progress is shown during import

### Duplicate data after import
- Imports overwrite data with the same timestamp
- To avoid duplicates, don't import the same data twice
- To replace all data: delete the bucket first, then import

## Performance Tips

### For Large Exports
- Export specific time ranges instead of all data when possible
- Exports are faster than full database dumps
- CSV files compress well (use `gzip` for storage)

### For Large Imports
- Import during low-traffic periods
- Monitor InfluxDB memory usage during import
- Consider splitting very large imports into time ranges

## Verify Data After Import

```bash
# Check recent data
influx query 'from(bucket:"prices-bucket") |> range(start:-7d) |> limit(n:10)' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN

# Count total records
influx query 'from(bucket:"prices-bucket") |> range(start:2020-01-01T00:00:00Z) |> count()' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN

# Check specific ticker
influx query 'from(bucket:"prices-bucket") |> range(start:-30d) |> filter(fn: (r) => r.ticker == "BTC")' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN
```

## Manual Export/Import Commands

If you need to run commands manually:

**Export:**
```bash
influx query 'from(bucket: "prices-bucket") |> range(start: -30d)' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN \
  --raw > export.csv
```

**Import:**
```bash
influx write \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN \
  --bucket prices-bucket \
  --format csv \
  --file export.csv
```

## Best Practices

1. **Regular Exports**: Schedule regular exports of production data
2. **Test Imports**: Test imports on sandbox before production
3. **Verify After Import**: Always verify data integrity after import
4. **Document Changes**: Keep notes of what data was imported when
5. **Backup Before Major Changes**: Export before major data operations

## Support

For issues or questions, refer to:
- [InfluxDB Query Documentation](https://docs.influxdata.com/influxdb/v2.0/query-data/)
- [InfluxDB Write Documentation](https://docs.influxdata.com/influxdb/v2.0/write-data/)
- Project documentation in `/docs`

## Script Locations

All scripts are located in: `/Users/oda/Desktop/Projects/managing-apps/scripts/influxdb/`

Configuration files:
- Sandbox: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.SandboxLocal.json`
- Production: `/Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.ProductionLocal.json`