Files

cryptooda 5cef270d64 Add influxdb export/import + add new influxdb instance for prod

2025-10-28 12:56:42 +07:00

8.9 KiB

Raw Blame History

InfluxDB Export and Import Scripts

This directory contains scripts for exporting and importing InfluxDB data for the Managing Apps project using query-based methods that work with standard read/write tokens.

Prerequisites

InfluxDB CLI - Required for export/import operations
```
brew install influxdb-cli
```
jq - JSON parser for reading configuration files
```
brew install jq
```

Available Scripts

1. `export-prices-bucket.sh`

Exports OHLCV candle/price data from the prices-bucket.

What it exports:

All candle data (open, high, low, close, volume)
Multiple exchanges, tickers, and timeframes
Configurable time ranges

Usage:

./export-prices-bucket.sh

Interactive Prompts:

Select environment (SandboxLocal or ProductionLocal)
Select time range (7 days, 30 days, 90 days, 1 year, all data, or custom)

Output:

CSV export: ./exports/<ENVIRONMENT>/<TIMESTAMP>/prices-bucket_data.csv
Metadata file with export details

Advantages:

✅ Works with regular read tokens (no admin required)
✅ Flexible time range selection
✅ Exports in standard CSV format
✅ Can be imported to any InfluxDB instance

2. `import-csv-data.sh`

Imports prices-bucket CSV export data into any InfluxDB environment.

What it imports:

Prices-bucket data only
Supports large files (1.6M+ data points)
Automatically creates bucket if needed

Usage:

./import-csv-data.sh

Interactive Prompts:

Select source environment (which export to import from)
Select export timestamp
Select target environment (where to import to)
Confirm the import operation

Features:

✅ Imports CSV exports to any environment
✅ Works with regular read/write tokens
✅ Batch processing for large files (5000 points per batch)
✅ Automatic bucket creation if needed
✅ Progress tracking for large imports

⚠️ Note: Import adds data to the bucket. Existing data with the same timestamps will be overwritten.

Configuration

The scripts automatically read InfluxDB connection settings from:

src/Managing.Api/appsettings.SandboxLocal.json
src/Managing.Api/appsettings.ProductionLocal.json

Required settings in appsettings files:

{
  "InfluxDb": {
    "Url": "https://influx-db.apps.managing.live",
    "Organization": "managing-org",
    "Token": "your-token-here"
  }
}

Export/Import Structure

exports/
├── SandboxLocal/
│   └── 20241028_143022/
│       ├── prices-bucket_data.csv
│       └── export-metadata.txt
└── ProductionLocal/
    └── 20241028_160000/
        ├── prices-bucket_data.csv
        └── export-metadata.txt

Data Structure

prices-bucket (Managed by these scripts)

Measurement: price
Contains: OHLCV candle data
Tags:
- exchange (e.g., Evm, Binance)
- ticker (e.g., BTC, ETH, AAVE)
- timeframe (e.g., FifteenMinutes, OneHour, OneDay)
Fields:
- open, high, low, close (price values)
- baseVolume, quoteVolume (volume data)
- TradeCount (number of trades)
- takerBuyBaseVolume, takerBuyQuoteVolume (taker buy volumes)

agent-balances-bucket (Not included in export/import scripts)

Measurement: agent_balance
Contains: User balance history over time
Note: This bucket is not managed by these scripts. Balance data is derived from operational data and should be regenerated rather than migrated.

Common Workflows

Quick Export

cd scripts/influxdb
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 5 (All data)

Export Specific Time Range

cd scripts/influxdb
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 3 (Last 90 days)

Migrate Sandbox to Production

cd scripts/influxdb

# Step 1: Export from sandbox
./export-prices-bucket.sh
# Select: 1 (SandboxLocal)
# Select: 5 (All data)

# Step 2: Import to production
./import-csv-data.sh
# Select source: 1 (SandboxLocal)
# Select: Latest export timestamp
# Select target: 2 (ProductionLocal)
# Confirm: yes

Backup Before Major Changes

cd scripts/influxdb

# Export current production data
./export-prices-bucket.sh
# Select: 2 (ProductionLocal)
# Select: 5 (All data)

# If something goes wrong, restore it:
./import-csv-data.sh
# Select the backup you just created

Clone Environment

# Export from source
./export-prices-bucket.sh
# Select source environment

# Import to target
./import-csv-data.sh
# Select target environment

Token Permissions

Read Token (Export)

Required for export-prices-bucket.sh:

✅ Read access to buckets
✅ This is what you typically have in production

Write Token (Import)

Required for import-csv-data.sh:

✅ Read/Write access to target bucket
✅ Ability to create buckets (optional, for auto-creation)

How to Check Your Token Permissions

influx auth list --host <URL> --token <TOKEN>

Data Retention

Exports are stored indefinitely by default

Manual cleanup recommended:

# Remove exports older than 90 days
find ./exports -type d -mtime +90 -exec rm -rf {} +

Troubleshooting

"influx command not found"

Install InfluxDB CLI:

brew install influxdb-cli

"jq command not found"

Install jq:

brew install jq

"Failed to parse configuration"

Ensure the appsettings JSON file exists and is valid JSON.

"Connection refused"

Check that InfluxDB URL is accessible
Verify network connectivity to the server
Check firewall rules

"401 Unauthorized"

Verify the InfluxDB token in appsettings is correct
For exports: ensure token has read permissions for the bucket
For imports: ensure token has write permissions for the bucket

"Bucket not found"

The import script will automatically create the bucket if you have permissions.

Or create it manually:

influx bucket create \
  --name prices-bucket \
  --org managing-org \
  --retention 0 \
  --host https://influx-db.apps.managing.live \
  --token YOUR_TOKEN

Import is slow

This is normal for large files (240MB+ with 1.6M+ data points)
Expected time: 5-15 minutes depending on network speed
The script processes data in batches of 5000 points
Progress is shown during import

Duplicate data after import

Imports overwrite data with the same timestamp
To avoid duplicates, don't import the same data twice
To replace all data: delete the bucket first, then import

Performance Tips

For Large Exports

Export specific time ranges instead of all data when possible
Exports are faster than full database dumps
CSV files compress well (use gzip for storage)

For Large Imports

Import during low-traffic periods
Monitor InfluxDB memory usage during import
Consider splitting very large imports into time ranges

Verify Data After Import

# Check recent data
influx query 'from(bucket:"prices-bucket") |> range(start:-7d) |> limit(n:10)' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN

# Count total records
influx query 'from(bucket:"prices-bucket") |> range(start:2020-01-01T00:00:00Z) |> count()' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN

# Check specific ticker
influx query 'from(bucket:"prices-bucket") |> range(start:-30d) |> filter(fn: (r) => r.ticker == "BTC")' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN

Manual Export/Import Commands

If you need to run commands manually:

Export:

influx query 'from(bucket: "prices-bucket") |> range(start: -30d)' \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN \
  --raw > export.csv

Import:

influx write \
  --host https://influx-db.apps.managing.live \
  --org managing-org \
  --token YOUR_TOKEN \
  --bucket prices-bucket \
  --format csv \
  --file export.csv

Best Practices

Regular Exports: Schedule regular exports of production data
Test Imports: Test imports on sandbox before production
Verify After Import: Always verify data integrity after import
Document Changes: Keep notes of what data was imported when
Backup Before Major Changes: Export before major data operations

Support

For issues or questions, refer to:

Script Locations

All scripts are located in: /Users/oda/Desktop/Projects/managing-apps/scripts/influxdb/

Configuration files:

Sandbox: /Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.SandboxLocal.json
Production: /Users/oda/Desktop/Projects/managing-apps/src/Managing.Api/appsettings.ProductionLocal.json

8.9 KiB Raw Blame History