{"id":37674412,"url":"https://github.com/hypertrial/stacking_sats_pipeline","last_synced_at":"2026-01-16T12:10:08.464Z","repository":{"id":295120799,"uuid":"989178569","full_name":"hypertrial/stacking_sats_pipeline","owner":"hypertrial","description":"Stacking Sats - A Hypertrial Open-Source Initiative","archived":false,"fork":false,"pushed_at":"2025-07-15T02:50:49.000Z","size":6376,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T17:24:07.518Z","etag":null,"topics":["bitcoin","data-engineering","data-science","python","stacking-sats"],"latest_commit_sha":null,"homepage":"https://hypertrial.github.io/stacking_sats_product/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hypertrial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-23T17:23:27.000Z","updated_at":"2025-07-15T02:50:53.000Z","dependencies_parsed_at":"2025-05-23T18:45:00.177Z","dependency_job_id":"5296c916-56ed-43a4-beab-2ce82ba7dc1f","html_url":"https://github.com/hypertrial/stacking_sats_pipeline","commit_stats":null,"previous_names":["hypertrial/stacking_sats_data","hypertrial/stacking_sats_pipeline"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/hypertrial/stacking_sats_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypertrial%2Fstacking_sats_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypertrial%2Fstacking_sats_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypertrial%2Fstacking_sats_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypertrial%2Fstacking_sats_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hypertrial","download_url":"https://codeload.github.com/hypertrial/stacking_sats_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypertrial%2Fstacking_sats_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478479,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","data-engineering","data-science","python","stacking-sats"],"created_at":"2026-01-16T12:10:07.634Z","updated_at":"2026-01-16T12:10:08.438Z","avatar_url":"https://github.com/hypertrial.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stacking Sats Pipeline\n\nA data engineering pipeline for extracting, loading, and merging cryptocurrency and financial data from multiple sources.\n\n## Requirements\n\n- Python 3.11 or 3.12\n- pip\n\n## Installation\n\n```bash\npip install stacking-sats-pipeline\n```\n\n## Quick Start\n\n### Data Extraction\n\nExtract all data sources to local files for offline analysis:\n\n#### CLI Usage\n\n```bash\n# Extract all data to CSV format\nstacking-sats --extract-data csv\n\n# Extract all data to Parquet format (smaller files, better compression)\nstacking-sats --extract-data parquet\n\n# Extract to specific directory\nstacking-sats --extract-data csv --output-dir data/\nstacking-sats --extract-data parquet -o exports/\n```\n\n#### Python API\n\n```python\nfrom stacking_sats_pipeline import extract_all_data\n\n# Extract all data to CSV in current directory\nextract_all_data(\"csv\")\n\n# Extract all data to Parquet in specific directory\nextract_all_data(\"parquet\", \"data/exports/\")\n```\n\n### Data Loading\n\n```python\nfrom stacking_sats_pipeline import load_data\n\n# Load Bitcoin price data\ndf = load_data()\n\n# Load specific data source\nfrom stacking_sats_pipeline.data import CoinMetricsLoader\nloader = CoinMetricsLoader()\nbtc_data = loader.load_from_web()\n```\n\n**What gets extracted:**\n\n- 📈 **Bitcoin Price Data** (CoinMetrics) → `btc_coinmetrics.csv/parquet`\n- 😨 **Fear \u0026 Greed Index** (Alternative.me) → `fear_greed.csv/parquet`\n- 💵 **U.S. Dollar Index** (FRED) → `dxy_fred.csv/parquet`\\*\n\n_\\*Requires `FRED_API_KEY` environment variable. Get a free key at [FRED API](https://fred.stlouisfed.org/docs/api/api_key.html)_\n\n**File Format Benefits:**\n\n- **CSV**: Human-readable, universally compatible\n- **Parquet**: ~50% smaller files, faster loading, preserves data types\n\n### Multi-Source Data Loading\n\n```python\nfrom stacking_sats_pipeline.data import MultiSourceDataLoader\n\n# Load and merge data from all available sources\nloader = MultiSourceDataLoader()\navailable_sources = loader.get_available_sources()\nmerged_df = loader.load_and_merge(available_sources)\n\n# Available sources: coinmetrics, feargreed, fred (if API key available)\nprint(f\"Available data sources: {available_sources}\")\nprint(f\"Merged data shape: {merged_df.shape}\")\n```\n\n## Data Sources\n\n### CoinMetrics (Bitcoin Price Data)\n\n```python\nfrom stacking_sats_pipeline.data import CoinMetricsLoader\n\nloader = CoinMetricsLoader(data_dir=\"data/\")\ndf = loader.load_from_web()  # Fetch latest data\ndf = loader.load_from_file()  # Load cached data (fetches if missing)\n\n# Extract to files\ncsv_path = loader.extract_to_csv()\nparquet_path = loader.extract_to_parquet()\n```\n\n### Fear \u0026 Greed Index\n\n```python\nfrom stacking_sats_pipeline.data import FearGreedLoader\n\nloader = FearGreedLoader(data_dir=\"data/\")\ndf = loader.load_from_web()\n```\n\n### FRED (Federal Reserve Economic Data)\n\n```python\nimport os\nos.environ['FRED_API_KEY'] = 'your_api_key_here'\n\nfrom stacking_sats_pipeline.data import FREDLoader\n\nloader = FREDLoader(data_dir=\"data/\")\ndf = loader.load_from_web()  # DXY (Dollar Index) data\n```\n\n## Development\n\nFor development and testing:\n\n**Requirements**: Python 3.11 or 3.12\n\n```bash\n# Clone the repository\ngit clone https://github.com/hypertrial/stacking_sats_pipeline.git\ncd stacking_sats_pipeline\n\n# Set up development environment (installs dependencies + pre-commit hooks)\nmake setup-dev\n\n# OR manually:\npip install -e \".[dev]\"\npre-commit install\n\n# Run tests\nmake test\n# OR: pytest\n\n# Code quality (MANDATORY - CI will fail if not clean)\nmake lint          # Fix linting issues\nmake format        # Format code\nmake check         # Check without fixing (CI-style)\n\n# Run specific test categories\npytest -m \"not integration\"  # Skip integration tests\npytest -m integration        # Run only integration tests\n```\n\n### Code Quality Standards\n\n**⚠️ MANDATORY**: All code must pass ruff linting and formatting checks.\n\n- **Linting/Formatting**: We use [ruff](https://docs.astral.sh/ruff/) for both linting and code formatting\n- **Pre-commit hooks**: Automatically run on every commit to catch issues early\n- **CI enforcement**: Pull requests will fail if code doesn't meet standards\n\n**Quick commands:**\n\n```bash\nmake help          # Show all available commands\nmake lint          # Fix ALL issues (autopep8 + ruff + format)\nmake autopep8      # Fix line length issues specifically\nmake format        # Format code with ruff only\nmake format-all    # Comprehensive formatting (autopep8 + ruff)\nmake check         # Check code quality (what CI runs)\n```\n\nFor detailed testing documentation, see [TESTS.md](tests/TESTS.md).\n\n### Contributing Data Sources\n\nThe data loading system is designed to be modular and extensible. To add new data sources (exchanges, APIs, etc.), see the [Data Loader Contribution Guide](stacking_sats_pipeline/data/CONTRIBUTE.md) which provides step-by-step instructions for implementing new data loaders.\n\n## Command Line Options\n\n```bash\n# Extract data\nstacking-sats --extract-data csv --output-dir data/\nstacking-sats --extract-data parquet -o exports/\n\n# Show help\nstacking-sats --help\n```\n\n## Project Structure\n\n```\n├── stacking_sats_pipeline/\n│   ├── main.py                    # Pipeline orchestrator and CLI\n│   ├── config.py                  # Configuration constants\n│   ├── data/                      # Modular data loading system\n│   │   ├── coinmetrics_loader.py  # CoinMetrics data source\n│   │   ├── fear_greed_loader.py   # Fear \u0026 Greed Index data source\n│   │   ├── fred_loader.py         # FRED economic data source\n│   │   ├── data_loader.py         # Multi-source data loader\n│   │   └── CONTRIBUTE.md          # Guide for adding data sources\n│   └── __init__.py                # Package exports\n├── tutorials/examples.py          # Interactive examples\n└── tests/                         # Comprehensive test suite\n```\n\n## API Reference\n\n### Core Functions\n\n```python\nfrom stacking_sats_pipeline import (\n    extract_all_data,           # Extract all data sources to files\n    load_data,                  # Load Bitcoin price data\n    validate_price_data,        # Validate price data quality\n    extract_btc_data_to_csv,    # Extract Bitcoin data to CSV\n    extract_btc_data_to_parquet # Extract Bitcoin data to Parquet\n)\n```\n\n### Configuration Constants\n\n```python\nfrom stacking_sats_pipeline import (\n    BACKTEST_START,    # Default start date for data range\n    BACKTEST_END,      # Default end date for data range\n    CYCLE_YEARS,       # Default cycle period\n    MIN_WEIGHT,        # Minimum weight threshold\n    PURCHASE_FREQ      # Default purchase frequency\n)\n```\n\n## Data Validation\n\nAll data sources include built-in validation:\n\n```python\nfrom stacking_sats_pipeline import validate_price_data\n\n# Validate Bitcoin price data\ndf = load_data()\nis_valid = validate_price_data(df)\n\n# Custom validation with specific requirements\nrequirements = {\n    'required_columns': ['PriceUSD', 'Volume'],\n    'min_price': 100,\n    'max_price': 1000000\n}\nis_valid = validate_price_data(df, **requirements)\n```\n\n## File Format Support\n\nThe pipeline supports both CSV and Parquet formats:\n\n- **CSV**: Universal compatibility, human-readable\n- **Parquet**: Better compression (~50% smaller), faster loading, preserves data types\n\n```python\n# CSV format\nextract_all_data(\"csv\", \"output_dir/\")\n\n# Parquet format\nextract_all_data(\"parquet\", \"output_dir/\")\n```\n\n## Timestamp Handling\n\nAll data sources normalize timestamps to midnight UTC for consistent merging:\n\n```python\nloader = MultiSourceDataLoader()\nmerged_df = loader.load_and_merge(['coinmetrics', 'fred'])\n\n# All timestamps are normalized to 00:00:00 UTC\nprint(merged_df.index.tz)  # UTC\nprint(merged_df.index.time[0])  # 00:00:00\n```\n\n## Error Handling\n\nThe pipeline includes comprehensive error handling:\n\n```python\ntry:\n    df = extract_all_data(\"csv\")\nexcept Exception as e:\n    print(f\"Data extraction failed: {e}\")\n    # Partial extraction may have succeeded\n```\n\nIndividual data sources fail gracefully - if one source is unavailable, others will still be extracted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypertrial%2Fstacking_sats_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhypertrial%2Fstacking_sats_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypertrial%2Fstacking_sats_pipeline/lists"}