nhsdigital/sde_example_analysis

Example of what you can do in Databricks in the Secure Data Environment (SDE) using Python, SQL, and R.

Secure Data Environment (SDE) example analysis

Reproducible and flexible example notebooks that may be used by researchers using Databricks within the NHS England SDE.

Contact

This repository is maintained by NHS England Data Wranglers team.

To contact us raise an issue on Github or via email.

See our (and our colleagues') other work here:

• SDE Data Wranglers
• NHS England

Description

The NHS England SDE is a data and research analysis platform, which is built to uphold the highest standards of privacy and security of NHS health and social care data when used for research and analysis.

Within the SDE, users have access to Databricks. Some users may be new to Databricks, so we provide example notebooks and useful functions for what they can do with Databricks inside the SDE. In addition, the analysis explores machine learning (ML) capabilities in the SDE.

Available analysis

🐍 python

pyspark_code_example: PySpark is the Python API for Aparche Spark - it can handle Big Data better than pandas.

✨ machine_learning_big_data

Usually, if a dataframe has more than one million rows, it is big data. However, a dataset may have less rows than this and still be classed as "big" due to complexity.

This folder contains notebooks exploring the machine learning capabilities of PySpark in the SDE. Examples include:

• decision_tree_regressor
• random_forest_classifier

🐼 machine_learning_small_data

This folder contains analysis for smaller data sets, using libraries like pandas and sklearn. Examples include:

• regression_multivariable
• regression_simple

🏴‍☠️ r

SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. The example notebook demonstrates what you can do with SparkR.

🧑‍🔧 wrangler_utilities

Useful functions and code, including code to create and save toy data sets.

Licence

The codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

Any HTML or Markdown documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.

Acknowledgements

Data Wranglers, Data & Analytics, NHS England

For individuals involved see CONTRIBUTORS.

sde_example_analysis's Issues

Github topics

hey guys - do you mind using topics in keeping with this framework we've been working on? It's no published yet, but It's be cool to see how you find it:

nhsengland/open-health-statistics#35