{"id":26956604,"url":"https://github.com/uomresearchit/mine-the-gaps","last_synced_at":"2025-04-03T03:32:57.652Z","repository":{"id":37676109,"uuid":"216416665","full_name":"UoMResearchIT/mine-the-gaps","owner":"UoMResearchIT","description":"App for showing sensor data on a map and comparing with estimates.","archived":false,"fork":false,"pushed_at":"2022-07-02T15:12:28.000Z","size":33674,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-03-17T16:33:52.602Z","etag":null,"topics":["django","geopandas","javascript","jquery","leafletjs","postgis","postgresql","python3"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UoMResearchIT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-20T19:36:12.000Z","updated_at":"2024-03-17T16:33:52.603Z","dependencies_parsed_at":"2022-09-19T03:51:08.463Z","dependency_job_id":null,"html_url":"https://github.com/UoMResearchIT/mine-the-gaps","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UoMResearchIT%2Fmine-the-gaps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UoMResearchIT%2Fmine-the-gaps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UoMResearchIT%2Fmine-the-gaps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UoMResearchIT%2Fmine-the-gaps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UoMResearchIT","download_url":"https://codeload.github.com/UoMResearchIT/mine-the-gaps/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246933235,"owners_count":20857046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","geopandas","javascript","jquery","leafletjs","postgis","postgresql","python3"],"created_at":"2025-04-03T03:30:41.359Z","updated_at":"2025-04-03T03:32:57.561Z","avatar_url":"https://github.com/UoMResearchIT.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Django CI](https://github.com/UoMResearchIT/mine-the-gaps/actions/workflows/django.yml/badge.svg)](https://github.com/UoMResearchIT/mine-the-gaps/actions/workflows/django.yml)\n\nThe sections below are:\n- [About](#about)\n- [Example screenshots](#example-screenshots)\n- [User instructions](#user-instructions)\n- [Install and test locally](#install-and-test-locally)\n  - [Clone this repository](#clone-this-repository)\n  - [QUICK START Instructions](#quick-start-instructions)  \n  - [Instructions for full database and Django set-up](#instructions-for-full-database-and-django-set-up-not-using-makefile--docker)\n- [Deployment](#deployment)  \n\n- [Research](#research)  \n- [Copyright and Licensing](#copyright--licensing)\n\n\u003c!-- toc --\u003e\n\n# About\nA Django web application to display sensor data on a map and allow comparison with regional estimates.\nAll data is loaded into the web app's database by the (admin) user at run-time, from 4 CSV files \nwhich comprise 2 data files and 2 metadata files. \nExample data files are provided in the `/examples` folder.  \n\nRegional estimates can also be calculated at run-time. The estimation methods \nare those available in the python library: https://pypi.org/project/region-estimators \n('concentric-regions' and 'distance-simple' at time of writing)\n\nUsers can also upload their own data locally (within a browser session), \nagain via a CSV file upload, allowing their own data to be compared alongside the \nactuals/estimates data pre-loaded by admin.\n\n\n\n# Example screenshots\n\n## Example with daily UK AQ data\n(currently running on our web app deployment: see http://minethegaps.manchester.ac.uk/)\n![mine-the-gaps](images/mine-the-gaps_general_screenshot.png)\n\n## Example with yearly Arizona Chloride PM2.5 LC data\n![Arizona_Chloride](images/mine-the-gaps_screenshot_US_Arizona_Chloride.png)\n\n# User instructions\n\nSee our [user instructions readme](README_instructions.md) for general and admin user instructions on\nhow to use this web application\n(and try this out on our example deployment: [mine-the-gaps](http://minethegaps.manchester.ac.uk/))\n\n\n# Install and test locally\nThe following instructions are to get this Django web app running on any machine, using a browser\nand localhost.\n\n## Clone this repository \nMake a copy of this code on your computer by:\n\n`cd ~/Code/` [cd to a folder on your local machine where you would like to install this code]\n\nand then either:\n\n`git clone  https://github.com/UoMResearchIT/mine-the-gaps.git`\n\nor:\n\n click on 'Code' on this github repository homepage, then 'Download ZIP' and save to your chosen folder.\n Then extract this zip file.\n\n## QUICK START (Makefile) Instructions \nThese instructions are for Linux, Mac OS and Windows Sub-system Linux users only. We have created a Makefile which \nyou can easily call from the shell. It runs a docker container which runs the Mine-the-Gaps web app, accessible from \na browser, at localhost.\n\n### Open a shell/command prompt and check you are in the mine-the-gaps project directory:\n`cd [folder into which you cloned the mine-the-gaps code]/mine-the-gaps`\nYou should be in the same folder as the `Makefile` file.\n\n### Call the Makefile to start the app\n`make docker-serve`\\\nThis installs and runs the app on localhost.\n\n### Check the app is running\nOpen a browser and navigate to `http://127.0.0.1:8000/` . You should see the mine-the-gaps web app running, but with no\ndata. (You may need to wait a few seconds and/or refresh the browser.)\n\n### Set up super-users (admin users) on the web app (after Makefile/docker quick installation)\nTo load data into the mine-the-gaps app, an admin user is required. To set up an \nadmin user (aka superuser), we need to run the Django management tool (which runs within the docker container). \nRun:\\\n`docker-compose run web python manage.py createsuperuser`\\\nThis will ask several questions via the command line. Once the superuser is set up,\nthat user can log in to the web application.\n\n### Log in as superuser and upload data\nReturn to the web application on your browser (`localhost:8000`) and click on `admin login` in the \ntop right-hand side of the main page.  Use the new admin log-in credentials to log in.\n\nSee our [instructions for admin users](README_instructions.md#admin-users) for admin user instructions \non how to use this web application, including how to upload data.\n\n### Further details when using the Quick Start set-up (using Makefile and docker)\n\nThe docker containers will now run in the background (`-d` specifies run as detached process) \nuntil they are stopped.\n\nTo see a list of all running containers, run either:\\\n`docker ps`\\\nor\\\n`docker container ls`\n\nIf both containers are running as expected, the output should show the two containers running: \none for the `mine-the-gaps_webapp` and another for a `postgis` database image.\nThe output should look something like:\n```\nCONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS                                       NAMES\n88a4aeebe9a8   mine-the-gaps_web   \"/entrypoint /start\"     56 minutes ago   Up 56 minutes   0.0.0.0:8000-\u003e8000/tcp, :::8000-\u003e8000/tcp   mine-the-gaps_web_1\n31f5a454ff2b   postgis/postgis      \"docker-entrypoint.s…\"   56 minutes ago   Up 56 minutes   5432/tcp                                    mine-the-gaps_db_1\n```\nIf the output does not look like above, then try running:\\\n`docker-compose logs -f`\\\nwhich gives you a detailed log file, showing all logs, including errors, for both containers.\n\nTo stop the docker container run:\\\n`docker-compose  down`\\\nor \\\n`make docker-stop`\\\n*Note that in the current set-up, once the container is stopped, any admin users and loaded data \n(added using instructions below) will be lost.*\n\nTo clean out the docker settings files run:\\\n`make clean`\n\nTo clean out the docker settings files and the Secret Key, run:\\\n`make clean-all`\n\nTo create a new settings file, run:\\\n`make settings`\\\nAlternatively, you can open the `/geo_sensor_gaps/settings/local.py` file to set the `SECRET_KEY` and the \n`MAX_NUM_PROCESSORS` settings.\n\nTo create a new secret key for docker (stored in text file), run:\\\n`make keys`\\\nAlternatively: \n1) you can open the `/geo_sensor_gaps/settings/local.py` file to set the `SECRET_KEY` setting.\n2) you can update the secret key in the file `keystring.txt`.\n\nTo set the maximum number of processors to be used:\\\nOpen the `/geo_sensor_gaps/settings/local.py` file to set the `MAX_NUM_PROCESSORS` setting.\n\nTo access the web app's source code, run:\\\n`docker exec -it mine-the-gaps_web_1 /bin/bash`\\\n*Note: `mine-the-gaps_web_1` should match the `NAME` of the container, listed when you get the list\nof all running containers (see above).*\n\n\n## Instructions for full database and Django set-up (NOT using Makefile / Docker)\n  \n  These instructions are based on the Ubuntu OS. They will need to be adapted to run on other Linux distributions,\n  Windows and other OSs.\n\n  This web app is tested on python versions 3.7, 3.8 and 3.9\n\n        sudo apt-get update\n        sudo apt-get install -y libproj-dev libgeos-dev gdal-bin libgdal-dev libsqlite3-mod-spatialite\n        python -m pip install --upgrade pip\n\n### Virtual Environment\n\nSee https://docs.python.org/3/tutorial/venv.html for instruction on how to set up a virtual environment \n(recommended). This virtual environment will hold the required version of Python, Django and other \nnecessary modules.\n\nTo activate your new virtual environment run:\n\n`source venv/bin/activate` [Replace 'venv' with the path to your new virtual environment]\n      \n### Install Django and additional modules. \n\nEnsure that your newly created virtual environment is activated (see above), and then run:\n\n    cd [code-directory] [Replace [code-directory] with the path of the project folder that contains requirements.txt]\n    pip install -r requirements.txt\n\n### Install the PostGIS (Spatial PostreSQL database)\nAs this web app requires geographical functionality, we can't rely only on Django's default\ndatabase set-up. We require a database that can hold and process geo-spatial data and \nPostGIS is used for this purpose.\nFor general documentation on PostGIS see https://postgis.net/docs/\n\nWithin the above instructions, there are installation instructions found at: \nhttps://postgis.net/docs/postgis_installation.html#install_short_version\n\nFor instructions on using PostGIS with Django:\nhttps://docs.djangoproject.com/en/3.2/ref/contrib/gis/install/postgis/\n\n### Add a .envs file\n\nUsing the `mine-the-gaps/settings/.env.template`, copy this file to `mine-the-gaps/settings/.env`\nthen fill in the `SECRET_KEY` value with a newly generated key (string) \n(e.g try this online key generator https://djecrety.ir/)\n\nFill in the other database log-in values with your own: Replace the NAME, USER and PASSWORD values with \nthose set up when you created your  PostGIS database in previous step.\n\n```text\nGEO_SENSOR_GAPS_SECRET_KEY=[INSERT_YOUR_SECRET_KEY]\nGEO_SENSOR_GAPS_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]\nGEO_SENSOR_GAPS_SQL_DATABASE=geo_sensor_gaps [OR REPLACE WITH YOUR DB NAME]\nGEO_SENSOR_GAPS_SQL_USER=geo_sensor_gaps_user [OR REPLACE WITH YOUR DB USER NAME]\nGEO_SENSOR_GAPS_SQL_PASSWORD=[INSERT_YOUR_DB_PASSWORD]\nGEO_SENSOR_GAPS_SQL_HOST=localhost\n```\n\n### Check Django installation\nEnsure your new virtual environment is activated, and then you can check Django is installed using the following\nonline set-up instructions.\n\n*Note that Django has already been installed in the previous step*\nhttps://docs.djangoproject.com/en/3.2/intro/install/\n\n\n### Run the development server\nEnsure that the new virtual environment is activated and run:\n\n `cd [code-directory]` [Replace [code-directory] with the path of the project folder that contains requirements.txt]\n\noptional test:\\\n`python manage.py test`\\\nand run:\\\n`python manage.py runserver`\n\nThe output to the last command should be similar to:\n\n```shell\n$ python manage.py runserver\n\nWatching for file changes with StatReloader\nPerforming system checks...\n\nSystem check identified no issues (0 silenced).\nJuly 22, 2021 - 16:15:44\nDjango version 3.2.5, using settings 'mine-the-gaps.settings.dev'\nStarting development server at http://127.0.0.1:8000/\nQuit the server with CONTROL-C.\n```\nUsing your browser, navigate to http://127.0.0.1:8000/ (or the link shown in your output) and this should \nopen up the web-application. *Note that no data has been loaded yet, so the map will be empty.*\n\n### Set up super-users (admin users) on the web app (when not using Makefile / docker install)\nTo load data into the mine-the-gaps app, an admin user is required. To set up an \nadmin user (aka superuser), we need to run the Django management tool (which runs within the docker container). \nRun:\\\n`python manage.py createsuperuser`\\\nThis will ask several questions via the command line. Once the superuser is set up,\nthat user can log in to the web application.\n\n### Log in as superuser and upload data\nReturn to the web application on your browser (`localhost:8000`) and click on `admin login` in the \ntop right-hand side of the main page.  Use the new admin log-in credentials to log in.\n\nSee our [instructions for admin users](README_instructions.md#admin-users) for admin user instructions \non how to use this web application, including how to upload data.\n\nSee the Django documents for more details on setting up user authentication:\nhttps://docs.djangoproject.com/en/3.2/topics/auth/default/\n\n# Deployment\nSee the Django documentation for deploying Django applications: \nhttps://docs.djangoproject.com/en/3.2/howto/deployment/\n\n# Research\n\n## Acknowledgement\nThis web application is part of the project \"Understanding the relationship between human health \nand the environment' funded by the Alan Turing Institute\n\n## Authors\nAnn Gledson, Douglas Lowe, Manuele Reani, David Topping and Caroline Jay\n\n\n# Copyright \u0026 Licensing\n\nThis software has been developed by Ann Gledson from the \n[Research IT](https://research-it.manchester.ac.uk/) \ngroup at the [University of Manchester](https://www.manchester.ac.uk/).\n\n(c) 2020-2021 University of Manchester.\nLicensed under the MIT license, see the file LICENSE for details.\n\n\n\n  \n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuomresearchit%2Fmine-the-gaps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuomresearchit%2Fmine-the-gaps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuomresearchit%2Fmine-the-gaps/lists"}