{"id":23421295,"url":"https://github.com/nhsdigital/sde_example_analysis","last_synced_at":"2025-10-25T21:15:19.909Z","repository":{"id":189960655,"uuid":"679639616","full_name":"NHSDigital/sde_example_analysis","owner":"NHSDigital","description":"Example of what you can do in Databricks in the Secure Data Environment (SDE) using Python, SQL, and R.","archived":false,"fork":false,"pushed_at":"2024-01-23T15:20:22.000Z","size":85,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-15T03:43:34.605Z","etag":null,"topics":["data-analysis","data-science","databricks-notebooks","machine-learning","mlflow"],"latest_commit_sha":null,"homepage":"https://digital.nhs.uk/services/secure-data-environment-service/secure-data-environment/user-guides","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NHSDigital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-17T09:36:16.000Z","updated_at":"2025-02-14T09:44:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"4a5a75ed-2505-4802-a602-eae97201abd5","html_url":"https://github.com/NHSDigital/sde_example_analysis","commit_stats":null,"previous_names":["nhsdigital/sde_example_analysis"],"tags_count":0,"template":false,"template_full_name":"NHSDigital/rap-package-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fsde_example_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fsde_example_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fsde_example_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NHSDigital%2Fsde_example_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NHSDigital","download_url":"https://codeload.github.com/NHSDigital/sde_example_analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248016630,"owners_count":21034015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-science","databricks-notebooks","machine-learning","mlflow"],"created_at":"2024-12-23T02:14:42.485Z","updated_at":"2025-10-25T21:15:14.867Z","avatar_url":"https://github.com/NHSDigital.png","language":"Python","readme":"# Secure Data Environment (SDE) example analysis\n\nReproducible and flexible example notebooks that may be used by researchers using Databricks within the [NHS England SDE](https://digital.nhs.uk/services/secure-data-environment-service/secure-data-environment).\n\n## Contact\n**This repository is maintained by [NHS England Data Wranglers team](mailto:england.sdeservice@nhs.net?subject=GitHub%20sde_example_notebooks)**.\n\u003e _To contact us raise an issue on Github or via email._\n\u003e \n\u003e See our (and our colleagues') other work here:\n\u003e- [SDE Data Wranglers](https://github.com/orgs/NHSDigital/teams/sde_wranglers/repositories)\n\u003e- [NHS England](https://github.com/orgs/NHSDigital/repositories)\n\n## Description\n\nThe NHS England SDE is a data and research analysis platform, which is built to uphold the highest standards of privacy and security of NHS health and social care data when used for research and analysis.\n\nWithin the SDE, users have access to [Databricks](https://digital.nhs.uk/services/secure-data-environment-service/secure-data-environment/user-guides/using-databricks-in-sde). Some users may be new to Databricks, so we provide example notebooks and useful functions for what they can do with Databricks inside the SDE. In addition, the analysis explores machine learning (ML) capabilities in the SDE.\n\n### Available analysis\n\n#### :snake: python\n\npyspark_code_example: PySpark is the Python API for Aparche Spark - it can handle Big Data better than pandas.\n\n##### :sparkles: machine_learning_big_data\n\nUsually, if a dataframe has more than one million rows, it is big data. However, a dataset may have less rows than this and still be classed as \"big\" due to complexity.\n\nThis folder contains notebooks exploring the machine learning capabilities of PySpark in the SDE. Examples include:\n- decision_tree_regressor\n- random_forest_classifier\n\n##### :panda_face: machine_learning_small_data\n\nThis folder contains analysis for smaller data sets, using libraries like pandas and sklearn. Examples include:\n- regression_multivariable\n- regression_simple\n\n#### :pirate_flag:\tr\n\nSparkR is an R package that provides a light-weight frontend to use Apache Spark from R. The example notebook demonstrates what you can do with SparkR.\n\n#### :mechanic: wrangler_utilities\n\nUseful functions and code, including code to create and save toy data sets.\n\n## Licence\n\nThe codebase is released under the [MIT License](LICENCE). This covers both the codebase and any sample code in the documentation.\n\nAny HTML or Markdown documentation is [© Crown copyright](https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/) and available under the terms of the [Open Government 3.0 licence](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).\n\n## Acknowledgements\n\n[Data Wranglers](https://github.com/orgs/NHSDigital/teams/sde_wranglers), Data \u0026 Analytics, NHS England\n\nFor individuals involved see [CONTRIBUTORS](CONTRIBUTORS.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnhsdigital%2Fsde_example_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnhsdigital%2Fsde_example_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnhsdigital%2Fsde_example_analysis/lists"}