{"id":26682712,"url":"https://github.com/jpcodes44/jpquant","last_synced_at":"2026-02-02T14:36:03.911Z","repository":{"id":284259236,"uuid":"942085713","full_name":"JPCodes44/JPQuant","owner":"JPCodes44","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-14T07:15:12.000Z","size":107775,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-22T05:37:52.027Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JPCodes44.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"SupportResistanceBreakout.html","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-03T14:51:18.000Z","updated_at":"2025-05-05T00:08:14.000Z","dependencies_parsed_at":"2025-05-14T07:39:18.708Z","dependency_job_id":null,"html_url":"https://github.com/JPCodes44/JPQuant","commit_stats":null,"previous_names":["jpcodes44/jpquant"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JPCodes44/JPQuant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPCodes44%2FJPQuant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPCodes44%2FJPQuant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPCodes44%2FJPQuant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPCodes44%2FJPQuant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JPCodes44","download_url":"https://codeload.github.com/JPCodes44/JPQuant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JPCodes44%2FJPQuant/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29013091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-02T12:48:30.580Z","status":"ssl_error","status_checked_at":"2026-02-02T12:46:38.384Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-26T08:27:59.079Z","updated_at":"2026-02-02T14:36:03.903Z","avatar_url":"https://github.com/JPCodes44.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📘 Full Guide to Using `backtesting.py` like a Pro\n\nThis guide covers how to use `backtesting.py` effectively and responsibly for developing trading strategies. Whether you're backtesting a simple moving average or designing adaptive regression channels, **this guide keeps you from overfitting and helps you code like you're trading live**.\n\n---\n\n## 🧠 Core Concepts of `backtesting.py`\n\n### What is `backtesting.py`?\nA lightweight Python framework for simulating trading strategies on historical data with built-in support for:\n- Candlestick charts\n- Portfolio management\n- Strategy definition\n- Custom indicators\n\n### Key Components\n- `Strategy`: Your custom logic\n- `self.data`: OHLCV historical data\n- `self.I(...)`: For defining indicators\n- `self.buy()`, `self.sell()`, `self.position.close()`: Trading commands\n- `next(self)`: Called every new candle\n\n---\n\n## 🔀 Data Handling in Real-Time Context\n\nWhen building indicators in `init`, **you must not cheat** by using future data (aka overfitting). You should:\n\n### ✅ Use a manual `for` loop inside `self.I()`\n```python\nself.I(your_indicator_function, self.data.Close, ...)\n```\nAnd inside your indicator:\n```python\ndef your_indicator(close):\n    result = np.full_like(close, np.nan)\n    for i in range(window, len(close)):\n        result[i] = some_logic_using(close[i-window:i])\n    return result\n```\nThis ensures **only past data** is used.\n\n### ❌ Never do this:\n```python\nnp.mean(close[-20:])  # uses future data if done in init()\n```\n\n---\n\n## ⚙️ The `init()` Function\n\nThis is where you calculate indicators using `self.I()` and store them as instance variables.\n\n```python\ndef init(self):\n    self.sma = self.I(sma_function, self.data.Close, period=20)\n```\n\nIf you're making a custom indicator that reacts to trends, spikes, or volatility — simulate live conditions using a loop with index `i`, just like a trading system would.\n\n---\n\n## 🔄 The `next()` Function\n\nThis runs at every candle and should contain your trading logic:\n\n```python\ndef next(self):\n    if not self.position and self.data.Close[-1] \u003e self.sma[-1]:\n        self.buy()\n    elif self.position and self.data.Close[-1] \u003c self.sma[-1]:\n        self.position.close()\n```\n\nUse `-1` to reference the most recent value of your indicator or data.\n\n---\n\n## 🧪 Real-Time Simulation Indicators\n\nHere are some key techniques we've used:\n\n### ✅ Proper Real-Time Calculation (regression channels)\n```python\ndef indicator(close):\n    result = np.full_like(close, np.nan)\n    for i in range(lookback, len(close)):\n        model = LinearRegression()\n        X = np.arange(i - lookback, i).reshape(-1, 1)\n        y = close[i - lookback:i]\n        model.fit(X, y)\n        result[i] = model.predict([[i]])[0]\n    return result\n```\n\n### 💨 Spike Detection (volatility filter)\n```python\nfrom scipy.signal import find_peaks\n\ndef detect_spike(close, i, window, prominence):\n    segment = np.asarray(close[i-window:i], dtype=np.float64)\n    if len(segment) \u003c 3: return False\n    scaled = max(prominence * abs(np.mean(segment)), 1e-6)\n    peaks, _ = find_peaks(segment, prominence=scaled)\n    troughs, _ = find_peaks(-segment, prominence=scaled)\n    return (len(peaks) \u003e 0 and peaks[-1] \u003e window - 5) or \\\n           (len(troughs) \u003e 0 and troughs[-1] \u003e window - 5)\n```\nUsed this to **skip drawing bands during pump/dump candles**.\n\n---\n\n## 🎨 Plotting with `self.I()`\n\nYou can use `self.I()` to show:\n- Regression lines\n- Support/resistance\n- Volatility filters\n- Zones (returning arrays of 1/0 or `np.nan`)\n\nUse `self.I()` only to call indicator logic that **returns the same shape as data**.\n\n```python\nself.upper_band = self.I(get_upper_band, self.data.Close, ...)\n```\n\n---\n\n## 💸 Tips to Avoid Overfitting\n\n- ✅ Use only historical data for indicator computation\n- ✅ Reset logic when large spikes break channel structure\n- ✅ Simulate real-time decisions\n- ✅ Validate with multiple lookbacks\n- ✅ Add time-based constraints (cooldowns)\n\n---\n\n## 📈 Advanced Use Cases\n\n- Segmented regression channels\n- Volatility-based band resetting\n- Spike filtering using `find_peaks`\n- Drawing different bands for intraday vs trend-timeframe\n- Adaptive lookbacks based on channel width or price breakout\n\n---\n\n## 🧰 Coinbase Fetcher \u0026 Data Preparation\n\nBefore running any backtest, you need historical data.\nUse the script you wrote to fetch clean 1m/1h/1d windows from Coinbase:\n\n```bash\npython coinbase_fetcher.py  # Your file with csvs_of_random_windows()\n```\n\nThis:\n- Randomly selects windows\n- Pulls candles in compliant chunks\n- Handles authentication\n- Avoids duplicates\n- Saves directly to folders like `1m_data` / `1h_data`\n\nBelow is the **full script** used for fetching Coinbase data:\n\n```python\n\"\"\"\nSTEPS TO USE\n1. create a .env file that looks like this\n    COINBASE_API_KEY=\"organizations/{org_id}/apiKeys/{key_id}\"\n    COINBASE_API_SECRET=\"-----BEGIN EC PRIVATE KEY-----\\nYOUR PRIVATE KEY\\n-----END EC PRIVATE KEY-----\\n\"\n2. select the symbol you want to fetch data for\n3. select the timeframe you want to fetch data for\n4. select the number of weeks of data to fetch\n5. run the script\n\"\"\"\n\n# ====== Imports ======\nimport pandas as pd\nimport numpy as np\nimport datetime\nimport os\nfrom dotenv import load_dotenv\nfrom pathlib import Path\nimport requests\nimport time\n\n# ====== Moon Dev's Configuration 🌙 ======\nSYMBOL_LIST = [\n    \"BTC-USD\",\n    \"ETH-USD\",\n    \"SOL-USD\",\n    \"AVAX-USD\",\n    \"ADA-USD\",\n    \"DOGE-USD\",\n    # \"MATIC-USD\",\n    \"LTC-USD\",\n    \"LINK-USD\",\n    \"BCH-USD\",\n    # \"ATOM-USD\",\n    # \"XLM-USD\",\n    \"ARB-USD\",\n    # \"APT-USD\",\n    # \"OP-USD\",\n    \"SUI-USD\",\n    # \"RNDR-USD\",\n    \"ICP-USD\",\n    # \"AAVE-USD\",\n    # \"UNI-USD\",\n]\nTIMEFRAME = \"1m\"  # Timeframe (e.g., '1m', '5m', '1h', '6h', '1d')\n\nif \"m\" in TIMEFRAME:\n    SAVE_DIR = \"/Users/jpmak/JPQuant/data/1m_data\"  # Directory to save the data files\nelif \"h\" in TIMEFRAME:\n    SAVE_DIR = \"/Users/jpmak/JPQuant/data/1h_data\"  # Directory to save the data files\nelif \"d\" in TIMEFRAME:\n    SAVE_DIR = \"/Users/jpmak/JPQuant/data/1d_data\"  # Directory to save the data files\n\n# DATE_RANGE = pd.date_range(start=\"2019-03-06\", end=\"2025-03-07\") 1d timeframe\n\n\nNUM_CSV = 20\n\n# Create save directory if it doesn't exist\nos.makedirs(SAVE_DIR, exist_ok=True)\nprint(f\"📂 Save directory ready: {SAVE_DIR}\")\n\n# Get the project root directory (2 levels up from this file)\nproject_root = Path(__file__).parent.parent.parent\nenv_path = project_root / \"coinbase.env\"\n\nprint(f\"🔍 Looking for .env file in: {project_root}\")\nprint(f\"📁 .env file exists: {'✅' if env_path.exists() else '❌'}\")\n\n# Load environment variables from the specific path\nload_dotenv(env_path)\n\n# Debug prints for API credentials (without revealing them)\napi_key = os.getenv(\"COINBASE_API_KEY\")\napi_secret = os.getenv(\"COINBASE_API_SECRET\")\nprint(\"🔑 API Key loaded:\", \"✅\" if api_key else \"❌\")\nprint(\"🔒 API Secret loaded:\", \"✅\" if api_secret else \"❌\")\n\nif not api_key or not api_secret:\n    print(\"❌ Error: API credentials not found in .env file\")\n    print(\"💡 Make sure your .env file exists and contains:\")\n    print(\"   COINBASE_API_KEY=organizations/{org_id}/apiKeys/{key_id}\")\n    print(\"   COINBASE_API_SECRET=your-secret-key\")\n    raise ValueError(\"Missing API credentials\")\n\nprint(\"🌙 Moon Dev's Coinbase Data Fetcher Initialized! 🚀\")\n\n\ndef sign_request(method, path, body=\"\", timestamp=None):\n    \"\"\"Sign a request using the API secret\"\"\"\n    timestamp = timestamp or str(int(time.time()))\n\n    # Remove the '/api/v3/brokerage' prefix from path for signing\n    if path.startswith(\"/api/v3/brokerage\"):\n        path = path[len(\"/api/v3/brokerage\") :]\n\n    # Create the message to sign\n    message = f\"{timestamp}{method}{path}{body}\"\n\n    try:\n        # Print debug info without exposing secrets\n        print(f\"🔏 Signing request for: {method} {path}\")\n\n        # Create JWT token\n        headers = {\n            \"CB-ACCESS-KEY\": api_key,\n            \"CB-ACCESS-TIMESTAMP\": timestamp,\n            \"accept\": \"application/json\",\n            \"content-type\": \"application/json\",\n        }\n\n        return headers\n\n    except Exception as e:\n        print(f\"❌ Error generating signature: {str(e)}\")\n        raise\n\n\ndef timeframe_to_granularity(timeframe):\n    \"\"\"Convert timeframe to granularity in seconds\"\"\"\n    if \"m\" in timeframe:\n        return int(\"\".join([char for char in timeframe if char.isnumeric()])) * 60\n    elif \"h\" in timeframe:\n        return int(\"\".join([char for char in timeframe if char.isnumeric()])) * 60 * 60\n    elif \"d\" in timeframe:\n        return (\n            int(\"\".join([char for char in timeframe if char.isnumeric()]))\n            * 24\n            * 60\n            * 60\n        )\n\n\ndef get_historical_data(symbol, timeframe, dates):\n    print(f\"🔍 Moon Dev is fetching {timeframe} data for {symbol}\")\n\n    try:\n        # Test connection with a simple request\n        print(\"🌎 Testing API connection...\")\n        base_url = \"https://api.exchange.coinbase.com\"  # Changed to exchange URL\n\n        # Get product details first\n        path = \"/products/\" + symbol\n        headers = sign_request(\"GET\", path)\n        print(\"🔐 Generated authentication headers\")\n        print(\"🔄 Making API request...\")\n\n        response = requests.get(f\"{base_url}{path}\", headers=headers)\n\n        if response.status_code != 200:\n            print(f\"❌ Response Headers: {response.headers}\")\n            print(f\"❌ Response Body: {response.text}\")\n            raise Exception(f\"API Error: {response.status_code} - {response.text}\")\n\n        print(\"✨ Connection test successful!\")\n\n        # Calculate time ranges\n        end_time = dates[-1]\n        start_time = dates[0]\n        granularity = timeframe_to_granularity(timeframe)\n\n        # Calculate appropriate chunk size based on granularity\n        # Coinbase limit is 300 candles per request\n        max_candles = 300\n        chunk_hours = max(\n            1, int((max_candles * granularity) / 3600)\n        )  # Convert to hours, minimum 1 hour\n        print(f\"📊 Using {chunk_hours} hour chunks for {timeframe} timeframe\")\n\n        # Fetch candles in chunks to avoid rate limits\n        all_candles = []\n        current_start = start_time\n\n        dataframe = pd.DataFrame()\n\n        while current_start \u003c end_time:\n            current_end = min(\n                current_start + datetime.timedelta(hours=chunk_hours), end_time\n            )\n\n            print(\n                f\"📊 Fetching data from {current_start.strftime('%Y-%m-%d %H:%M')} to {current_end.strftime('%Y-%m-%d %H:%M')}\"\n            )\n\n            params = {\n                \"start\": current_start.isoformat(),\n                \"end\": current_end.isoformat(),\n                \"granularity\": str(granularity),\n            }\n\n            path = f\"/products/{symbol}/candles\"\n            headers = sign_request(\"GET\", path)\n\n            response = requests.get(f\"{base_url}{path}\", params=params, headers=headers)\n\n            if response.status_code != 200:\n                print(f\"❌ Response Headers: {response.headers}\")\n                print(f\"❌ Response Body: {response.text}\")\n                raise Exception(f\"API Error: {response.status_code} - {response.text}\")\n\n            candles = response.json()\n            all_candles.extend(candles)  # Changed to handle direct response\n            current_start = current_end\n            time.sleep(0.5)  # Rate limit compliance\n\n        print(f\"✨ Successfully fetched {len(all_candles)} candles!\")\n\n        # Convert to DataFrame\n        df = pd.DataFrame(all_candles)\n        df.columns = [\"datetime\", \"open\", \"high\", \"low\", \"close\", \"volume\"]\n        df[\"datetime\"] = pd.to_datetime(df[\"datetime\"], unit=\"s\")\n        dataframe = pd.concat([df, dataframe])\n        dataframe = dataframe.set_index(\"datetime\")\n        dataframe = dataframe[[\"open\", \"high\", \"low\", \"close\", \"volume\"]]\n\n        return dataframe\n\n    except Exception as e:\n        print(f\"❌ Error: {str(e)}\")\n        print(\"💡 Tips:\")\n        print(\"   1. Make sure you're using a Coinbase Exchange API key\")\n        print(\"   2. Check if your API key has the required permissions\")\n        print(\"   3. Verify your API key is active\")\n        raise\n\n\ndef generate_random_date_range(timeframe: str):\n    import random\n\n    # Set bounds for how big/small windows can be\n    if \"m\" in timeframe:\n        earliest = pd.to_datetime(\"2020-01-01 09:10:00\")\n        latest = pd.to_datetime(\"2025-03-23 09:10:00\")\n        abs_min = datetime.timedelta(minutes=5)\n        abs_max = datetime.timedelta(days=5)\n        freq = \"min\"\n    elif \"h\" in timeframe:\n        earliest = pd.to_datetime(\"2020-01-01 09:00:00\")\n        latest = pd.to_datetime(\"2025-03-23 09:00:00\")\n        abs_min = datetime.timedelta(hours=24)\n        abs_max = datetime.timedelta(days=240)\n        freq = \"h\"\n    elif \"d\" in timeframe:\n        earliest = pd.to_datetime(\"2020-01-01\")\n        latest = pd.to_datetime(\"2025-03-23\")\n        abs_min = datetime.timedelta(days=7)\n        abs_max = datetime.timedelta(days=680)\n        freq = \"D\"\n    else:\n        raise ValueError(\"Invalid timeframe\")\n\n    # Randomly scale the window size within bounds (e.g. 0.5x to 1x of max)\n    scale = random.uniform(0.5, 1.0)\n    window = abs_min + (abs_max - abs_min) * scale\n\n    # Ensure we don’t overshoot the latest date\n    max_start = latest - window\n    start = earliest + (max_start - earliest) * random.random()\n    end = start + window\n\n    # Round down start and end to clean timestamps based on frequency\n    start = start.floor(freq)\n    end = end.ceil(freq)\n\n    print(\n        f\"🌀 Using randomized window:\\n   Start: {start}\\n   End:   {end}\\n   Window: {window} (Timeframe: {timeframe})\"\n    )\n    return start, end\n\n\ndef csvs_of_random_windows(timeframe, num_csv):\n    for i in range(num_csv):\n        # Select a random symbol from the symbol list\n        symbol = SYMBOL_LIST[np.random.randint(0, len(SYMBOL_LIST))]\n\n        start_date, end_date = generate_random_date_range(TIMEFRAME)\n\n        DATE_RANGE = pd.date_range(\n            start=start_date,\n            end=end_date,\n            freq=\"min\" if \"m\" in timeframe else \"h\" if \"h\" in timeframe else \"d\",\n        )\n\n        DATES = pd.to_datetime(DATE_RANGE).sort_values()  # Ensure datetime \u0026 sort\n\n        # Fetch the historical data for the selected symbol and timeframe\n        try:\n            dataframe = get_historical_data(symbol, timeframe, DATES)\n        except Exception as e:\n            print(f\"There was an error cz of {e}\")\n\n        if dataframe.empty:\n            print(f\"⚠️ No data returned for {symbol}, skipping.\")\n            continue\n\n        # Create a filename for the CSV file\n        csv_filename = f\"{symbol[0:3]}-{timeframe}-{start_date}-{end_date}_data.csv\"\n\n        # Construct the full path to the CSV file\n        csv_path = os.path.join(SAVE_DIR, csv_filename)\n\n        # Check if the CSV file already exists\n        if os.path.exists(csv_path):\n            print(\"file alr exists\")\n            continue\n\n        # Print a message indicating the creation of a new CSV file\n        print(f\"🎨✨ Creating sheet #{i + 1} from {start_date} to {end_date}\")\n\n        # Check if the DataFrame contains enough data\n        try:\n            sliced = dataframe.loc[start_date:end_date]\n        except KeyError:\n            print(f\"Skipping csv: {start_date} to {end_date} not in index\")\n            continue\n\n        if sliced.shape[0] \u003c= 1:\n            print(\n                \"Skipping csv because not enough data is fetched or Start/end times are too early/late.\"\n            )\n\n        # Save the DataFrame to a CSV file\n        sliced.to_csv(csv_path)\n\n    # Print a message indicating the completion of the process\n    print(\"Done boiiiiiiiii\")\n\n\ncsvs_of_random_windows(TIMEFRAME, NUM_CSV)\n\n```\n\nEach file is perfect for single-trade backtest sessions using `run_backtest(...)`\n\n---\n\n## 🔧 Testing \u0026 Running\n\nUse your runner script like:\n```python\nrun_backtest(SegmentedRegressionWithFinalFitBands, DATA_FOLDER)\n```\nMake sure `DATA_FOLDER` matches the resolution of your candles (`1m`, `1h`, `1d`, etc).\n\n---\n\n## 🧼 Debug Tips\n- Print shape/type of any array inside indicator\n- Use `np.asarray(..., dtype=np.float64)` to convert weird backtesting.py arrays\n- Plot spike zones for sanity\n- Use `cooldown_counter` to delay logic after extreme events\n\n---\n\n## 🏑 Final Words\nBacktesting isn’t just about coding strategies — it’s about thinking like you’re trading **live**.\n\nBuild tools that adapt.\nCode like the future isn’t visible.\nTest like your money depends on it — because one day, it might.\n\nYou got this, professor trader. 🚫🚀\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpcodes44%2Fjpquant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpcodes44%2Fjpquant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpcodes44%2Fjpquant/lists"}