{"id":21554396,"url":"https://github.com/millengustavo/demo-datasus-streamlit","last_synced_at":"2025-04-10T09:26:12.775Z","repository":{"id":112833708,"uuid":"228036315","full_name":"millengustavo/demo-datasus-streamlit","owner":"millengustavo","description":"Demo Application with DataSUS death records and Streamlit","archived":false,"fork":false,"pushed_at":"2019-12-14T14:36:22.000Z","size":7043,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T08:12:54.274Z","etag":null,"topics":["data-science","datasus","health","healthcare","streamlit"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/millengustavo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-14T14:27:37.000Z","updated_at":"2024-08-07T19:34:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"617d172d-bc95-428d-aadd-4f8b4ad1d9d8","html_url":"https://github.com/millengustavo/demo-datasus-streamlit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/millengustavo%2Fdemo-datasus-streamlit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/millengustavo%2Fdemo-datasus-streamlit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/millengustavo%2Fdemo-datasus-streamlit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/millengustavo%2Fdemo-datasus-streamlit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/millengustavo","download_url":"https://codeload.github.com/millengustavo/demo-datasus-streamlit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248190750,"owners_count":21062334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","datasus","health","healthcare","streamlit"],"created_at":"2024-11-24T07:14:38.877Z","updated_at":"2025-04-10T09:26:12.764Z","avatar_url":"https://github.com/millengustavo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Demo Application with DataSUS death records and Streamlit\n\n![datasus_streamlit](assets/datasus_streamlit.gif)\n\nIn Brazil, [more than 70% of the population depends only on the medical assistance provided by the government](http://bvsms.saude.gov.br/bvs/pacsaude/diretrizes.php). The Brazilian public healthcare system is called SUS (Sistema Único de Saúde). \n\nThere is a public SUS data repository available online (DataSUS). Although the data is not always clean and complete, we can derive many insights from DataSUS. \n\nIn this post we are going to build and deploy a Streamlit application inspired on the [Uber pickups example](https://github.com/streamlit/demo-uber-nyc-pickups), but using DataSUS death records (2006-2017) and geographic coordinates from health facilities.\n\n## Downloading the data from DataSUS\n\n### SIM\n\nFrom the [DataSUS website](http://www2.datasus.gov.br/DATASUS/index.php?area=060701) we have the definition of SIM:\n\u003e The Mortality Information System (SIM) was created by DATASUS for the regular collection of mortality data in the country. From the creation of the SIM it was possible to comprehensively capture mortality data to subsidize the various management spheres in public health. Based on this information it is possible to perform situation analysis, planning and evaluation of actions and programs in the area.\n\nLet's download the SIM data for the São Paulo state. The prefixes of the files are \"DOSP\". \n\n#### Downloading the data from the ftp\n```python\nfrom ftplib import FTP\n\nftp = FTP(\"ftp.datasus.gov.br\")\nftp.login()\nftp.cwd(\"dissemin/publicos/SIM/CID10/DORES/\")\nall_files = ftp.nlst(\".\")\nstate_prefix = \"DOSP\"\n# We sort the list and keep only the last 12 records\n# This is because they share the same layout of the current data (2006-2017)\nfiles = sorted([file for file in all_files if state_prefix in file])[-12:]\n\nfor file in files:\n    print(\"Downloading {}...\".format(file))\n    with open(file, \"wb\") as fp:\n        ftp.retrbinary(\"RETR {}\".format(file), fp.write)\n```\n\n#### Renaming the files which the extension is capitalized\n\n```python\nimport os\n\nfiles = [file for file in os.listdir() if \"DOSP\" in file and \".DBC\" in file]\n\nfor file in files:\n    os.rename(file, file[:-4] + \".dbc\")\n```\n\n#### Converting from .dbc to .csv\nAs you may have noticed, the files are in a `.dbc` format. This is a proprietary format of the SUS Department of Informatics (DATASUS).\n\nA kind developer provided a [tool](https://github.com/greatjapa/dbc2csv) to convert files from `.dbc` to `.csv`. To use this tool we will need to have `git` and `docker` installed.\n\n##### Build the docker image\n```bash\ngit clone https://github.com/greatjapa/dbc2csv.git\ncd dbc2csv\ndocker build -t dbc2csv .\n```\n\n##### Convert the files\n1. Navigate to the folder where you download the `.dbc` files. Copy the full path to the directory, you can get this path by running:\n```bash\npwd\n```\n2. Run:\n```bash\ndocker run -it -v \u003cfull_path_to_the_directory\u003e:/usr/src/app/data dbc2csv make\n```\n3. A `/csv` folder will be populated with the converted files.\n\n![dbc2csv](assets/dbc2csv.png)\n\n### CNES\nFrom their [website](http://cnes.datasus.gov.br/):\n\u003e The National Register of Health Facilities (CNES) is a public document and official information system for registering information about all health facilities in the country, regardless of their legal nature or integration with the Unified Health System (SUS).\n\nThe process to download the data is simpler this time, they are already in a `.zip` file you can download from this link:\n\nftp://ftp.datasus.gov.br/cnes/BASE_DE_DADOS_CNES_201910.ZIP\n\nWe are going to use only one `.csv` file from this data: `tbEstabelecimento201910.csv`\n\n## Processing the data\n\n### Reading the facilities table with pandas\n\nTo be efficient, we will pass only the columns that matter to our application.\n\n```python\nimport pandas as pd\n\ncnes = pd.read_csv(\n    \"tbEstabelecimento201910.csv\",\n    sep=\";\",\n    usecols=[\n        \"CO_CNES\",\n        \"CO_CEP\",\n        \"NO_FANTASIA\",\n        \"NO_LOGRADOURO\",\n        \"CO_ESTADO_GESTOR\",\n        \"NU_LATITUDE\",\n        \"NU_LONGITUDE\",\n    ],\n)\n```\n\n### Filtering the data for the São Paulo state. \nFrom the dictionary available on the DataSUS website we know that '35' is the code for São Paulo. For this application we are only going to keep this data\n\n```python\ncnes = cnes[cnes[\"CO_ESTADO_GESTOR\"]==35]\n```\n\n### Merging with the death records\nMy converted `.csv` SIM files are in the path `../data/SIM/csv/`, make sure you modify the path accordingly\n\n```python\nfiles = sorted(os.listdir(\"../data/SIM/csv/\"))\n\ndfs = [\n    pd.read_csv(\n        \"../data/SIM/csv/\" + file,\n        usecols=[\"NUMERODO\", \"DTOBITO\", \"HORAOBITO\", \"CODESTAB\"],\n    )\n    for file in files\n]\ndf = pd.concat(dfs)\n\n# We will drop the null CODESTABs (data without CNES code)\ndf = df.dropna()\n```\n\nBefore proceeding to fill the missing coordinates, join the CODESTAB with the CO_CNES, so we have fewer facilities to fill.\n\n```python\ncnes = cnes.rename(columns={\"CO_CNES\": \"CODESTAB\"})\nmerged = df.merge(cnes, on=\"CODESTAB\")\n\n# Since we merged with the death records file, we have many duplicates, \n# let's drop it to see which facilities have coordinates missing\nunique_merged = merged[\n    [\"CODESTAB\", \"CO_CEP\", \"NU_LATITUDE\", \"NU_LONGITUDE\"]\n].drop_duplicates()\n# Filtering the data for only the records where the coordinates are missing\nmissing_coords = unique_merged[unique_merged[\"NU_LATITUDE\"].isnull()]\n# The CEP was automatically converted to int and we have lost the first zero digit.\n# This line converts to string and pad with zero so we have a valid CEP\nmissing_coords[\"CO_CEP\"] = (\n    missing_coords[\"CO_CEP\"].astype(str).apply(lambda x: x.zfill(8))\n```\n\nWe have 697 CEPs without coordinates, let's try to fill them up.\n\n## Enriching the data from DataSUS with latitude and longitude (cep_to_coords)\n\nThe data we downloaded from DataSUS is not complete. Geographic coordinates of various health facilities are missing. While latitude and longitude are not present in all cases, we do have the Brazilian zip code (CEP) for some. \n\nA quick search on Google for converting from CEP to latitude and longitude has shown that we had some scripts that mixed R and Python to achieve this task. \n\nInvestigating the scripts further, it became clear that it was simple and valuable to implement this in Python. So, I removed the dependency of R to achieve the same result with just Python (https://github.com/millengustavo/cep_to_coords). \n\n### Install geocode from source\n```bash\ngit clone https://github.com/millengustavo/cep_to_coords.git\ncd cep_to_coords\ngit checkout master\npip install -e .\n```\n\nThe package usage is simple. Call the `cep_to_coords` function with a valid CEP string, it will search the correios API for an address, concatenate it with the city and country and hit an [API](http://photon.komoot.de/) to get the coordinates.\n\nIf you find it useful, please leave a star on [Github](https://github.com/millengustavo/cep_to_coords). The project is still in its infancy, so it is a great opportunity to [contribute to your first open source project](https://medium.com/@austintackaberry/why-you-should-contribute-to-open-source-software-right-now-bec8bd83cfc0) adding features or refactoring the code!\n\n### Fill the coordinates\n```python\nfrom cep_to_coords.geocode import cep_to_coords\n\ncep_column = \"CO_CEP\"\n\nunique_ceps = missing_coords[cep_column].unique()\n# cep_to_coords returns a [lat, lon] list if it finds the coordinates\n# else it returns [NaN, NaN]\nmissing_coords[\"lat\"] = float(\"nan\")\nmissing_coords[\"lon\"] = float(\"nan\")\nfor ind, elem in enumerate(unique_ceps):\n    try:\n        coords = cep_to_coords(elem)\n        missing_coords.loc[ind, \"lat\"] = coords[0]\n        missing_coords.loc[ind, \"lon\"] = coords[1]\n    except Exception as e:\n        print(elem, coords, e)\n    print(\"{}%...\".format(ind * 100 / len(unique_ceps)))\n```\n\n\u003e Using the cep_to_coords function we were able to fill **78%** of the missing coordinates!\n\n### Compiling the final CEP table\n\nTo complete the data preparation, we need to take our filled coordinates and replace the NaNs on the death records table.\n\n```python\nunique_merged[\"CO_CEP\"] = (\n    unique_merged[\"CO_CEP\"].astype(str).apply(lambda x: x.zfill(8))\n)\n# unfortunately we didn't fill all coordinates, let's drop them\nmissing_coords = missing_coords.drop(columns=[\"NU_LATITUDE\", \"NU_LONGITUDE\"]).dropna()\n# joining the datasets\nfull_table = unique_merged.merge(missing_coords, on=\"CO_CEP\", how=\"left\")\n# filling the missing data\nfull_table[\"lat\"] = full_table.apply(\n    lambda x: x[\"lat\"] if pd.isnull(x[\"NU_LATITUDE\"]) else x[\"NU_LATITUDE\"], axis=1\n)\nfull_table[\"lon\"] = full_table.apply(\n    lambda x: x[\"lon\"] if pd.isnull(x[\"NU_LONGITUDE\"]) else x[\"NU_LONGITUDE\"], axis=1\n)\n# compiling the CEP final table\nfull_table = (\n    full_table.drop(columns=[\"NU_LATITUDE\", \"NU_LONGITUDE\", \"CODESTAB_y\"])\n    .dropna()\n    .rename(columns={\"CODESTAB_x\": \"CODESTAB\"})\n    .reset_index(drop=True)\n)\n```\n\n### Merging the facilities back to the death records dataframe and cleaning the data\n\n```python\ndf_enriched = df.merge(full_table, on=\"CODESTAB\")\ndf_enriched[\"HORAOBITO\"] = pd.to_numeric(\n    df_enriched[\"HORAOBITO\"], downcast=\"integer\", errors=\"coerce\"\n)\ndf_enriched = df_enriched.dropna()\ndf_enriched[\"DTOBITO\"] = df_enriched[\"DTOBITO\"].astype(str).apply(lambda x: x.zfill(8))\ndf_enriched[\"HORAOBITO\"] = (\n    df_enriched[\"HORAOBITO\"].astype(int).astype(str).apply(lambda x: x.zfill(4))\n)\n# Creating a timestamp column with both date and hour of death\ndf_enriched[\"DATA\"] = df_enriched[\"DTOBITO\"] + \" \" + df_enriched[\"HORAOBITO\"]\ndf_enriched[\"DATA\"] = pd.to_datetime(\n    df_enriched[\"DATA\"], format=\"%d%m%Y %H%M\", errors=\"coerce\"\n)\ndf_enriched = df_enriched.dropna()\n\ndf_enriched[\"NUMERODO\"] = df_enriched[\"NUMERODO\"].astype(str)\n\ndf_enriched[\"lat\"] = (\n    df_enriched[\"lat\"].astype(str).str.replace(\",\", \".\", regex=False).astype(float)\n)\n\ndf_enriched[\"lon\"] = (\n    df_enriched[\"lon\"].astype(str).str.replace(\",\", \".\", regex=False).astype(float)\n)\n```\n\n### Saving to a .parquet file\n\n```python\ndf_enriched.to_parquet(\"../data/clean/dataset.parquet.gzip\", compression=\"gzip\", index=False)\n```\n\n## Creating the app using the Uber pickups example\n\nStreamlit according to the website is \n\u003e “The fastest way to build custom ML tools”. \n\nIt is indeed a bold statement, but what sold me on it was the sentence on the subtitle:\n\u003e “So you can stop spending time on frontend development and get back to what you do best.”.\n\nFor this experiment, we are going to spend even less time on frontend development by using an example gently posted by the Streamlit team (https://github.com/streamlit/demo-uber-nyc-pickups). The demo presents the Uber pickups on New York City by hour. Our goal here is to replace pickups with deaths registered on SIM and New York City with the state of São Paulo. \n\nThere are only a few things we need to change in the code to adapt the application to our use. \n\n### Clone the repository\n```bash\ngit clone https://github.com/streamlit/demo-uber-nyc-pickups.git\ncd demo-uber-nyc-pickups\n```\n\n### Open app.py on your favorite text editor and change the following lines (commented here)\n\n```python\n# OLD -\u003e DATE_TIME = \"date/time\"\n# NEW -\u003e DATE_TIME = \"data\"\n\n# OLD -\u003e data = pd.read_csv(DATA_URL, nrows=nrows)\n# NEW -\u003e data = pd.read_parquet(\"../data/clean/dataset.parquet.gzip\")\n```\n\nFor cosmetic purposes you may also change the title and other specific references\n\n### Install streamlit\n```bash\npip install streamlit\n```\n\n### Run the app\n```bash\nstreamlit run app.py\n```\n\n![terminal_streamlit](assets/terminal_streamlit.png)\n\nVoilà! This command will automatically open the app on your browser (port 8051 by default).\n\n# Conclusion\nThis is a very simple project that shows some amazing libraries that are being developed lately for Machine Learning applications. Although we didn't used any complex techniques here, we covered an interesting part of what a data scientist do. Data ingestion, cleaning, enriching and finally visualization.\n\nI hope you learned something from this and I encourage you to play around with Streamlit, it's definitely amazing!\n\n![datasus_app](assets/datasus_app_pt.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmillengustavo%2Fdemo-datasus-streamlit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmillengustavo%2Fdemo-datasus-streamlit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmillengustavo%2Fdemo-datasus-streamlit/lists"}