{"id":18388756,"url":"https://github.com/imperialcollegelondon/recode-decodingmarketsignals","last_synced_at":"2025-04-12T04:27:13.290Z","repository":{"id":235780659,"uuid":"727007564","full_name":"ImperialCollegeLondon/ReCoDE-DecodingMarketSignals","owner":"ImperialCollegeLondon","description":"In this Exemplar we investigate whether we can use signals form technical and quantitative analysis to predict future stock returns","archived":false,"fork":false,"pushed_at":"2024-05-28T09:28:07.000Z","size":10276,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-16T00:45:26.647Z","etag":null,"topics":["recode"],"latest_commit_sha":null,"homepage":"https://imperialcollegelondon.github.io/ReCoDE-DecodingMarketSignals/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImperialCollegeLondon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-04T01:39:15.000Z","updated_at":"2024-05-28T09:27:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"de5832bd-ae30-42e3-90ed-9bdf176053b0","html_url":"https://github.com/ImperialCollegeLondon/ReCoDE-DecodingMarketSignals","commit_stats":null,"previous_names":["imperialcollegelondon/recode-decodingmarketsignals"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImperialCollegeLondon%2FReCoDE-DecodingMarketSignals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImperialCollegeLondon%2FReCoDE-DecodingMarketSignals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImperialCollegeLondon%2FReCoDE-DecodingMarketSignals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImperialCollegeLondon%2FReCoDE-DecodingMarketSignals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImperialCollegeLondon","download_url":"https://codeload.github.com/ImperialCollegeLondon/ReCoDE-DecodingMarketSignals/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248515659,"owners_count":21117216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["recode"],"created_at":"2024-11-06T01:37:14.129Z","updated_at":"2025-04-12T04:27:13.261Z","avatar_url":"https://github.com/ImperialCollegeLondon.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- Your Project title, make it sound catchy! --\u003e\n\n# Decoding Market Signals\n\n## Leveraging candlestick patterns, machine learning and alpha signals for enhanced trading strategy analysis\n\n\u003c!-- Provide a short description to your project --\u003e\n\n### Description\n\nThis project aims to rigorously back-test a trading strategy, focusing on evaluating the informational value of \ncandlestick patterns. Utilising `Python`, a pipeline of functions will systematically scan and evaluate \nthe components of the S\u0026P 500 stock market index for patterns upon which one can base a trading decision.\n\nAdvanced functionalities of the `Pandas` library will be employed to load, manipulate and store detailed statistical \ndata, particularly `method chains`, `multi-indexed data frames` and `user-defined functions` acting on rows and columns of data frames.\n\nThe project's core involves assessing the predictive capabilities of these trading signals using nuanced binary classification performance \nmetrics, thereby determining their practical applicability. Additionally, a logistic regression model will be deployed \nto explore the intersection of finance and machine learning. \nThis phase aims to ascertain whether machine learning algorithms can outperform traditional methods in predicting market \nmovements based on identified signals.\n\nThis multifaceted project integrates financial analysis, data science, and machine learning, \npromising insights with both academic and practical implications. \nIts methodologically sound approach, coupled with detailed documentation and learning annotations, \nis designed to make it an exemplary contribution to the ReCoDE initiative, \nshowcasing the transformative potential of research computing and data science in interdisciplinary research.\n\n\u003c!-- What should the students going through your exemplar learn --\u003e\n\n### Learning Outcomes\n\n- Setting up a custom computational environment for financial data science.\n- Making a custom technical analysis library written in C++ work with recent Python. \n- Obtaining and pre-processing high-quality financial data.\n- Using Pandas' best-practices like method-chaining, and multi-index data frames for data manipulation  \n- Independently testing and analysing trading actions proposed on a hypothesis.\n\n\u003c!-- How long should they spend reading and practising using your Code.\nProvide your best estimate --\u003e\n\n| Task       | Time    |\n| ---------- | ------- |\n| Reading    | 4 hours |\n| Practising | 6 hours |\n\n### Requirements\n\n\u003c!--\nIf your exemplar requires students to have a background knowledge of something\nespecially this is the place to mention that.\n\nList any resources you would recommend to get the students started.\n\nIf there is an existing exemplar in the ReCoDE repositories link to that.\n--\u003e\n\n- Foundational knowledge of Python and Pandas.\n- An interest in stock markets and trading signals.\n- An interest in statistical analysis and hypothesis testing.\n- Resilience in troubleshooting and adapting older libraries to work with recent Python versions.\n  - Particularly, we will make use of a library called `ta-lib` that contains a pattern-recognition library detecting candlestick patterns in Open-High-Low-Close `(OHCL)` data.\n- Familiarity with Jupyter notebooks, type annotations, and automation.\n\n#### Academic\n\n\u003c!-- List the system requirements and how to obtain them, that can be as simple\nas adding a hyperlink to as detailed as writing step-by-step instructions.\nHow detailed the instructions should be will vary on a case-by-case basis.\n\nHere are some examples:\n\n- 50 GB of disk space to hold Dataset X\n- Anaconda\n- Python 3.11 or newer\n- Access to the HPC\n- PETSc v3.16\n- gfortran compiler\n- Paraview\n--\u003e\n\nThe repository is self-contained. Additional references are provided in the Jupyter notebooks.\nWhen we move form a single-stock analysis to the whole investment-universe, the resulting data frames \nbecome too large to work with on a standard machine leading to Kernel crashes. They are themselves not dangerous\nto the hardware of the computer at all, and one can mitigate this by selecting a subset of the data. If you wish to run the \ncode on all the data, you need a potent machine, or alternatively execute the code on the HPC facilities. \nThat does not hinder you from getting started, though.\n\n\n#### System\n\n\u003c!-- Instructions on how the student should start going through the exemplar.\n\nStructure this section as you see fit but try to be clear, concise and accurate\nwhen writing your instructions.\n\nFor example:\nStart by watching the introduction video,\nthen study Jupyter notebooks 1-3 in the `intro` folder\nand attempt to complete exercise 1a and 1b.\n\nOnce done, start going through the PDF in the `main` folder.\nBy the end of it you should be able to solve exercises 2 to 4.\n\nA final exercise can be found in the `final` folder.\n\nSolutions to the above can be found in `solutions`.\n--\u003e\n\nA recent mid-class laptop is sufficient to follow along the code. The more data you wish to analyse, the more RAM\nit should have. The code was developed on a Linux machine. \n\nIn this code exemplary, we make use of `Python 3.11`. \nIdentifying the candlestick patterns in financial markets data is obtained by using a library that is called `ta-lib`.\nIt works well for our task, but its Python wrapper is no longer maintained. If you are comfortable with an older version of Python, precisely \n`Python 3.8` or `Python 3.9`, or just want to get started, it is straightforward to install an older version using `pip` or `conda`. \n\n`ta-lib` was tested to be installable from `pypi` on `Python 3.8` and `3.9`.\nIf you just want to get started, use `Python 3.8`. `ta-lib` can then be installed using `pip install TA-Lib`. \nAlternatively, if you want to make use of the `conda` package manager, use `conda install ta-lib`\nFor `Python 3.9`, the author observed on a Linux operating system, that `conda install ta-lib` worked straightforward, \nwhereas `pip install ta-lib` did not.\n\nIf you want to make use of later versions of Python such as the environment this project was developed on, precisely \n`Python v. 3.11`, the process is more involved and requires compiling `ta-lib's C++` files from source.\n\nWe now encountered two common problems, we frequently face as computer and data scientists:\ni) Making `legacy` code run on modern systems,\nii) Facing multiple choices of which package manager to use.\n**If you are just interested on getting started, use `Python 3.8` and skip the following section.**\n\nFor the interested reader, is follows some background information on i) and ii).\nIssue i) is commonly encountered in practice, especially in larger corporations operating with custom software, whose author's stopped \nmaintaining their code. There is no silver bullet on working with legacy code and custom problems often need tailor-made solutions.\nFor this project, the author wrote\na `shell` script that is custom-made to set up `ta-lib` for `Python 3.11`. Whenever you are not sure whether a solution works and you\nhave reasons to believe your attempt is error-prone, might have side effects, or spoil the operating system, it is advisable \nto work from within a virtual environment, for example using `Docker`, before employing a working solution on the user's machine. \nDiscussing `Docker` is beyond the scope of this documentation, however.\n\nThe `shell script` can also be directly applied to work on macOS as the latter uses the `Z shell` by default. \nThe `Z shell`, is also known as `zsh` and is a `Unix shell` that is built on top of `bash`. Hence, compatibility \nis likely and the script should run without reservation. \nFor Windows users the `shell script` can also be modified to work on the `Windows shell` or `PowerShell`.\nThe equivalent of a Linux `shell script` on Windows is a `batch script` and the commands expressed have to be translated \nto make them compatible on Windows.\n\nLet us now quickly address issue ii):\nIf your Python environment is set up using miniconda (recommended), see also `https://docs.conda.io/projects/miniconda/en/latest/`,\nboth, `conda` and `pip` are installed by default, and you can make use of both them.   \nIf your Python environment is set up using the source files from `https://www.python.org/` you might have to install `pip` separately\nand cannot use the benefits of conda.\n\nWhat then is the difference between `pip` and `conda`?\n`Pip` is a package manager specifically designed for Python packages.\nIt primarily focuses on installing and managing Python libraries and packages from the `Python Package Index (PyPI)`.\nPip is used for managing Python dependencies within a Python environment.\nOn the other hand, `Conda` is a more comprehensive package manager and environment manager.\nWhile it can manage Python packages, it is not limited to Python and can handle packages and libraries from various programming languages.\nConda is often used to create isolated environments that can include different versions of Python and non-Python dependencies.\nIt can manage both Python packages and system-level packages and is capable of handling complex dependency resolution.\n\nManaging and detecting version conflicts of a large Python setup is again a topic on its own. Granted `conda` is diligent, but slow, the \nreader is encouraged to look into promising alternatives like `mamba`, which is a package manager written in `C++` and hence more performant \nthan `conda`, although less tested.\n\nA final note regarding code-formatting. To comply with the PEP-8 style guide for Python code,  \n`https://peps.python.org/pep-0008/`, we make use of a code-formatter, that automatically spots issues concerning spacing\nand style. It is applied on code that runs error-free and ensures style consistency. There are several open-source code-formatters\nout there and arguably the most popular are `black`, see `https://github.com/psf/black`, and `Ruff`, see `https://docs.astral.sh/ruff/formatter/`.\nThe former is well-tested, however the latter is more performant and has recently gained increasing attention. Hence, we make use of Ruff.\n\n\n### Getting Started\n\n\n#### Windows/MacOS\n\nFor Windows and MacOS based machines, you will have to install the `ta-lib` library separately. Instructions\nto do so can be [found here](https://github.com/ta-lib/ta-lib-python?tab=readme-ov-file#dependencies). Once installed, you can create a virtual environment as suggested below and use `pip install ta-lib` to install the required Python wrapper, and `pip install -r requirements.txt` to install the other dependencies. \n\n#### Linux\n\n\nThe following code was tested on `Ubuntu 22.04.4 LTS` using `Python 3.11`. For other Linux distributions you need to modify the commands \nthat install software on your system. For example, on `Fedora`, the default package manager is `dnf` rather than Ubuntu's `apt`. Also,\nyou might modify `python3.11 -m venv MarketSignals` to whatever python version you have. Typically, one can also use the default system Python. The first \nline in the below sequence then reads\n`python -m venv MarketSignals` rather than `python3.11 -m venv MarketSignals`. However, on my setup, I opted to install a more recent version of Python\n, here 3.11. I then have to explicitly call this version to create a virtual environment. Besides, I opted to leave Ubuntu's default system Python \nalone. The reason is somewhat more intricate. In short, in contrast to Windows, Ubuntu's bootloader `grub` critically depends on this system\nPython. If, for any reason you spoil this system Python, you cannot just delete and re-install it. Should you try to do, your system will not \nboot anymore. Hence, I leave the system Python untouched and instead opted for a separate Python version explicitly set for development work. \nThis `dev work Python`, I can modify, fine-tune, delete, and re-install without ever even touching the system Python, making it safer to experiment. \n\nExecute the following sequences in your Linux terminal to set up your Python environment needed to run the `DecodingMarketSignals` repository.\n```\npython3.11 -m venv MarketSignals  # create a virtual environment\nsource MarketSignals/bin/activate  # activate the virtual environment\nsudo apt-get install python3.11-dev  # installs the Python 3.11 development files on your Ubuntu or Debian-based Linux system.\npip install -r requirements.txt  # install dependencies via pip \nchmod +x install-talib.sh  # set execution rights for the shell script to install TA-lib from source\n./install-talib.sh  # run the script installing TA-lib  \njupyter-notebook  # optional: launch an instance of Jupyter notebook and run the examples (assumes you downloaded the CRSP data already).\n```\n\n\n\u003c!-- An overview of the files and folder in the exemplar.\nNot all files and directories need to be listed, just the important\nsections of your project, like the learning material, the code, the tests, etc.\n\nA good starting point is using the command `tree` in a terminal(Unix),\ncopying its output and then removing the unimportant parts.\n\nYou can use ellipsis (...) to suggest that there are more files or folders\nin a tree node.\n--\u003e\n\nStart by opening and reading through the Jupyter notebook. All essential steps are separated in respective sub-sections.\nOnce you have an understanding of the overall goal, you can start setting up your Python environment along `ta-lib and \neither replicate the results or apply the techniques demonstrated to your own data. The reader is encouraged to apply the methods outlined \non their own data from different markets, for instance, the futures and forex markets. \n\n\u003c!--\nThe below is a TODO\n--\u003e\n\n### Project Structure\n\n\n```log\n\n.\n\n├── data\n\n│   ├── SP500_daily_data_1980_to_2023.csv.gz (to be downloaded by the user via WRDS)\n\n│   └── SP500_tickers_one_per_line.txt\n\n├── figures\n\n|   ├── candlestick_anatomy.png\n\n|   ├── WRDS_overview.png\n\n|   ├── CRSP1_select_quarterly.png\n\n|   ├── ...\n\n│   └── WRDS_overview.png\n\n├── notebooks\n\n|   ├── BSquant.py\n\n|   ├── 1_obtaining_financial_data.ipynb\n\n|   ├── 2_single_stock_case.ipynb\n\n|   ├── 3_SP500_case.ipynb\n\n├── install-talib.sh\n\n├── requirements.txt\n\n├── mkdocs.yml\n\n├── LICENSE.md\n\n└── README.md\n\n```\n\n\u003c!-- Change this to your License. Make sure you have added the file on GitHub --\u003e\n\n### License\n\nThis project is licensed under the [BSD-3-Clause license](LICENSE.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimperialcollegelondon%2Frecode-decodingmarketsignals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimperialcollegelondon%2Frecode-decodingmarketsignals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimperialcollegelondon%2Frecode-decodingmarketsignals/lists"}