{"id":25117925,"url":"https://github.com/brody-lab/bl_pipeline_python","last_synced_at":"2026-01-29T23:02:46.851Z","repository":{"id":53379102,"uuid":"306686478","full_name":"Brody-Lab/bl_pipeline_python","owner":"Brody-Lab","description":"Python Data pipeline for BrodyLab ","archived":false,"fork":false,"pushed_at":"2024-07-30T18:12:48.000Z","size":1915,"stargazers_count":0,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-09-09T20:12:10.043Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Brody-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-10-23T16:13:31.000Z","updated_at":"2024-07-30T18:12:51.000Z","dependencies_parsed_at":"2024-06-08T18:46:39.145Z","dependency_job_id":"ecf73b8a-0f7d-4a71-b349-91e747e989a7","html_url":"https://github.com/Brody-Lab/bl_pipeline_python","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Brody-Lab/bl_pipeline_python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Brody-Lab%2Fbl_pipeline_python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Brody-Lab%2Fbl_pipeline_python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Brody-Lab%2Fbl_pipeline_python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Brody-Lab%2Fbl_pipeline_python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Brody-Lab","download_url":"https://codeload.github.com/Brody-Lab/bl_pipeline_python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Brody-Lab%2Fbl_pipeline_python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28889863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T21:06:44.224Z","status":"ssl_error","status_checked_at":"2026-01-29T21:06:42.160Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-08T03:36:46.524Z","updated_at":"2026-01-29T23:02:46.837Z","avatar_url":"https://github.com/Brody-Lab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Brody Lab Python-based Data Pipeline\n\nThe python data pipeline defined for BrodyLab.\n\n# Installation\n\n## Prerequisites\n\n1. Request access to DataJoint: https://frevvo-prod.princeton.edu/frevvo/web/tn/pu.nplc/u/84fd5e8d-587a-4f6a-a802-0c3d2819e8fe/app/_sO14QHzSEemyQZ_M7RLPOg/formtype/_-XYdEEK2Eeqtf7JjRFmYDQ/popupform\n2. Install conda on your system:  https://conda.io/projects/conda/en/latest/user-guide/install/index.html\n3. If running in Windows get [git](https://gitforwindows.org/)\n4. (Optional for ERDs) [Install graphviz](https://graphviz.org/download/)\n\n## Installation with conda\n\n1. Open a new terminal \n2. Clone this repository: either with https (`https://github.com/Brody-Lab/bl_pipeline_python.git`) or ssh (`git@github.com:Brody-Lab/bl_pipeline_python.git`)\n    - If you cannot clone repositories with ssh, [set keys overview](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent), [set keys windows](https://github.com/Brody-Lab/jbreda_labnotebook/blob/master/helpful_code.md#ssh)\n3. Create a conda environment: `conda create -n bl_pipeline_python_env python==3.10`\n    - the name can be changed, just keep it consistent for your kernal below\n    - this repository has been updated to use python 3.10 (3.7 was previously in use) \n4. Activate environment: `conda activate bl_pipeline_python_env`.   **(Activate environment each time you use the project)**\n5. Change directory to this repository `cd bl_pipeline_python`.\n6. Install the primary required libraries `pip install -e .`\n    - this will take a few minutes\n7. Install jupyter and ipykernal libraries in series\n    ```\n    conda install -c conda-forge jupyterlab\n    conda install -c anaconda ipykernel\n    python -m ipykernel install --user --name=bl_pipeline_python_env # allows you to run notebooks on environment kernal\n    ```\n8. Run the Configuration notebook under `notebooks/00-datajoint-configuration.ipynb`\n    - make sure you select the correct kernal from the top right menu\n9. If you have install issues, especially on a windows machine, see [here](https://github.com/Brody-Lab/bl_pipeline_python/blob/main/install_debug.md)\n10. Additional libraries you may want to install:\n    ```\n    pip install seaborn pyarrow openpyxl\n    ```\n\n## Tutorials\n\nWe have created some tutorial notebooks to help you start working with datajoint\n\n1. Querying data (**Strongly recommended**) \n - `jupyter notebook notebooks/tutorials/Explore_Sessions_Data.ipynb`\n - `jupyter notebook notebooks/tutorials/1-Explore U19 data pipeline with DataJoint.ipynb`\n\n\n## Array Electrophysiology with Neuropixels Probes\n\n+ This portion of the workflow usings DataJoint's standard python package \n[element-array-ephys](https://github.com/datajoint/element-array-ephys).\n\n+ This workflow uses the `ephys_chronic` module from `element-array-ephys`.\n\n## Developer Notes\n\n### Database transfer\n+ `bl_pipeline_python/datajoint01_pipeline/process/process.py`\n+ Copy data from source tables (to shadow tables) to new tables\n+ Shadow table allows for renaming of primary key\n+ Shadow table has same definition of the new table, except that the primary key\n is the same as the source table\n+ For each shadow table set the keys as a secondary field when not used as \nprimary key\n\n### Data integrity issues\n+ `bl_pipeline_python/notebooks/debugging_data_integrity.ipynb`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrody-lab%2Fbl_pipeline_python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrody-lab%2Fbl_pipeline_python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrody-lab%2Fbl_pipeline_python/lists"}