{"id":26691080,"url":"https://github.com/olahsymbo/data-extraction-service","last_synced_at":"2025-04-12T22:12:03.019Z","repository":{"id":41947449,"uuid":"269582975","full_name":"olahsymbo/data-extraction-service","owner":"olahsymbo","description":"upload large dataset from csv, xlsx into Postgres DB, and render the data on a large table on HTML page","archived":false,"fork":false,"pushed_at":"2023-07-31T20:05:29.000Z","size":45437,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"dev","last_synced_at":"2025-03-26T16:17:16.614Z","etag":null,"topics":["django","html","pipenv","postgres-db","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olahsymbo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-05T09:12:09.000Z","updated_at":"2024-10-05T01:48:19.000Z","dependencies_parsed_at":"2023-01-18T11:17:21.250Z","dependency_job_id":null,"html_url":"https://github.com/olahsymbo/data-extraction-service","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olahsymbo%2Fdata-extraction-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olahsymbo%2Fdata-extraction-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olahsymbo%2Fdata-extraction-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olahsymbo%2Fdata-extraction-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olahsymbo","download_url":"https://codeload.github.com/olahsymbo/data-extraction-service/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248637771,"owners_count":21137538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","html","pipenv","postgres-db","postgresql","python"],"created_at":"2025-03-26T16:17:20.245Z","updated_at":"2025-04-12T22:12:02.993Z","avatar_url":"https://github.com/olahsymbo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![data-extraction-service](https://github.com/olahsymbo/DataTask/actions/workflows/datatask-ci.yml/badge.svg)](https://github.com/olahsymbo/DataTask/actions/workflows/datatask-ci.yml)\n\n## Data Extraction Service\n\nThis django web application is developed for uploading data in \".csv\" format to a postgresql database and serving (rendering) the data from the DB in an html.\n\n### Getting Started\n\nTo enable this app function properly, install the dependencies in the `pipenv`. Simply run:\n\n```\npipenv shell\npipenv install\n```\n\n\nOnce all the dependencies have been installed, proceed as follows:\n\n- Create a postgres DB\n- Run upload_data app\n- Launch DataTask server\n\nThe details of the steps are provided as follows:\n\n### Create a Postgres DB\n\n-  First install postgresql 12.2 (incase it's not installed already).\n\n    One way of doing this is to download the postgresql app from https://postgresapp.com/downloads.html\n    or use homebrew by running:\n\n    `brew install postgresql`\n\n    After installing postgresql, start all postgresql services from terminal by either running (postgres must be running at all times):\n\n    `pg_ctl -D /usr/local/var/postgres start`\n    or\n    `brew services start postgresql`\n\n- Second, create a new postgres database named `task` using:\n\n    `CREATEDB task;`\n\n    Afterward, launch the `task` db shell by running:\n\n    `psql task`\n\n    create username and password for the database using:\n\n    `CREATE USER admin1 with encrypted password 'admin1';`\n\n    grant all privileges of the db to USER `admin1` using:\n\n    `GRANT ALL PRIVILEGES ON DATABASE task TO admin1;`\n\n    grant the USER the role to create new db using:\n\n    `ALTER USER admin1 CREATEDB;`\n\n\n### Run upload_data app\n\nBy completing the first step, we have created a db `task`. The next step is uploading csv data to postgres db. The app for achieving this is `upload_data`.\n\nIn order to upload the data, we need to create a db table which will be accessed by our django project model, we need to:\n\n- Create db table \n\n    create db table `Taskdata` in the db `task` by running the following:\n\n    `python upload_data/create_db_table.py`\n\n- Upload csv data to db table\n\n    start the web server using:\n\n    `python manage.py runserver`\n\n    open another terminal and goto project directory\n\n    `cd data-extraction-service`\n\n    to upload the `task.csv` data into table `Taskdata`, run:\n\n    `python upload_data/stack_data.py`\n\n    we can now turn off the web application server (Ctrl + C)\n\n### Launch data-extraction-service server\n\nThe main app in `data-extraction-service` is `showdata`. It contains the data model, views, and url routes\n\n- Again launch the server using:\n\n    `python manage.py runserver`\n\n- to view the uploaded data in html, go to:\n\n    `http://127.0.0.1:8000/index.html/`\n\n- to access the admin page, go to:\n\n    `http://127.0.0.1:8000/admin/`\n  \nThe admin page login details is:\n\n```\nusername: admin\npassword: user1234\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folahsymbo%2Fdata-extraction-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folahsymbo%2Fdata-extraction-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folahsymbo%2Fdata-extraction-service/lists"}