{"id":16419714,"url":"https://github.com/salaah01/visualcsv","last_synced_at":"2026-01-21T16:37:24.284Z","repository":{"id":44248971,"uuid":"253027281","full_name":"Salaah01/visualCSV","owner":"Salaah01","description":"VisualCSV is a web application built using Django, React and Redux for state management which allows users to upload CSVs and build graphs based on the data in the CSVs.","archived":false,"fork":false,"pushed_at":"2023-07-20T10:59:20.000Z","size":22658,"stargazers_count":1,"open_issues_count":16,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-14T03:48:21.371Z","etag":null,"topics":["database","django","docker","postgresql","python","react","redux"],"latest_commit_sha":null,"homepage":"https://www.iamsalaah.com/projects/visual-csv","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Salaah01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-04T15:07:33.000Z","updated_at":"2022-07-24T12:21:10.000Z","dependencies_parsed_at":"2024-10-28T15:28:35.708Z","dependency_job_id":"b507bfd2-f77f-4277-a3c0-ee047e4e9d3e","html_url":"https://github.com/Salaah01/visualCSV","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2FvisualCSV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2FvisualCSV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2FvisualCSV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Salaah01%2FvisualCSV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Salaah01","download_url":"https://codeload.github.com/Salaah01/visualCSV/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247785920,"owners_count":20995642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","django","docker","postgresql","python","react","redux"],"created_at":"2024-10-11T07:25:42.023Z","updated_at":"2026-01-21T16:37:24.252Z","avatar_url":"https://github.com/Salaah01.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VisualCSV\n\nVisualCSV is a web application built using [Django](https://www.djangoproject.com/), [React](https://reactjs.org/) and [Redux](https://redux.js.org/) for state management which allows users to upload CSVs and build graphs based on the data in the CSVs.\n\n## How it Works\n\nOnce authenticated, users are able to upload CSVs. With the uploaded CSVs users are asked to set any primary and foreign keys. Once the users are happy with their inputs they are able to send their CSVs to the backend by pressing an upload button. This converts a user's choices to JSON and along with the CSV data which is also converted to JSON and send this off to the backend.\n\nThe backend then creates database tables based on the POST data and updates the `user_auth` table which contains information on which table belongs to which user.\n\nWith this completed, the user is then able to access the graph_builder where they are able to use the UI to build graphs.\n\nPresently, the only way to save the graph is to physically save the a copy of the graph as an image. However, in the future, this will be updated so that this can be saved directly onto the database.\n\n## Getting Started\n\nThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.\n\n## Installation\n\nThe guide will cover installation using [Docker](https://docs.docker.com/) and installing manually without Docker.\nIt is recommended to use Docker to install and deploy the application as it is a lot faster and easier.\n\n### Installation Using Docker\n\n**Clone the repository**\n\nClone (or fork) the repository to your working directory `git clone https://github.com/Salaah01/visualCSV.git`.\n\n**Setting up the environment variables**\n\nThe docker version of the installation will use 3 containers:\n\n- Main web application\n- PostgreSQL database containing the main tables for the Django application.\n- PostgreSQL database for containing the user tables.\n- PostgreSQL test database for tests relating to the user database.\n\nDocker will expect files from which it will retrieve values to set as environment variables.\nThe get started, you can run `bash quick_start_env_files.sh`.\n**This is for testing that the docker commands do work, in production you must replace these files with your own environment variables and once deployed, remove the files from your file system.**\n\nI recommend run the bash script and then running `find | grep \".*\\.env$\"`.\nThis will return a lit the files where the environment variables are stored.\n\n**Build the services**\n\n```bash\ndocker-compose build\ndocker-compose up -d\n```\n\nThis can take a little while the first time round as it does install the node packages.\nThe script will bind the visualCSV directory in the host machine to the same directory in the container. Therefore, updating files in the host will update the files in the container.\nOnce docker-compose has finished it's magic, you are ready to access the site via port 8000 on localhost and start developing!\n\n## Installation Manually\n\n### Prerequisites\n\nYou will need to ensure that the following are installed:\n* [Python 3](https://www.python.org/downloads/)\n* [Node.js](https://nodejs.org/en/download/)\n* [PostgreSQL](https://www.postgresql.org/) (The application has not yet been tested on other database systems.)\n* [Git](https://git-scm.com/downloads) (Recommended)\n\n### Installing\n\nThe system uses two databases. One which is handled by Django's ORM and another that is handled via pure SQL. The Django ORM controlled database we will call visualCSV and the self managed database we will call visualCSVClient.\n\n#### Database setup\n\nTwo separate databases are used to separate concerns. The visualCSV database will be used to handle authentication and other tables set up by Django. Whereas visualCSVClient will be used to store the end user's data. As each CSV is assigned its own table, we cannot control how fast the database grows. By designating the end-user tables to its own database, we can better control what is in the visualCSV (the main database).\n\nI recommend using [pgAdmin](https://www.pgadmin.org/) to set up the database as it provides a great GUI for interacting with postgreSQL.\nYou will need to create three databases, visualCSV, visualCSVClient and visualCSVTest.\n\nIf you are using pgAdmin, access the query tool in visualCSVClient and run the following sql:\n\n```sql\nCREATE TABLE user_auth (id SERIAL PRIMARY KEY, user_id INTEGER NOT NULL, table_name VARCHAR(100) NOT NULL, table_alias VARCHAR(100) NOT NULL);\n```\n\nRepeat the query in the visualCSVClient.\n\nIf you choose not to use pgAdmin, you can achieve the same using the command line:\n\n```bash\npsql -U postgres\nCREATE DATABASE \"visualCSV\";\nCREATE DATABASE \"visualCSVClient\";\n\\c visualCSVClient\nCREATE TABLE user_auth (id SERIAL PRIMARY KEY, user_id INTEGER NOT NULL, table_name VARCHAR(100) NOT NULL, table_alias VARCHAR(100) NOT NULL);\n```\n\n#### Clone the repository\n\nClone (or fork) the repository to your working directory `git clone https://github.com/Salaah01/visualCSV.git`. If you do not have `git` installed, you can [download the files directly](https://github.com/Salaah01/visualCSV/archive/master.zip).\n\n### Create and activate a virtual environment\n\nTo create a virtual environment run `python -m venv venv` . Note: If you are using Linux you may need to run `sudo apt-get install python3-venv`.\nActivate the virtual environment.\nFor Windows users: `venv\\scripts\\activate`.\nFor Linux/Mac users: `venv/bin/activate`.\n\n### Install Python packages\n\nRun `pip install -r requirements.txt`\n\n### Install Node packages\n\nRun `npm install`.\nYou may receive a warning about vulnerabilities in the installed node packages. By running `npm audit fix` you may be able to fix these vulnerabilities (only vulnerabilities which will not break the code).\n\n### Run Node build scripts\n\nRun `npm run build`.\nThis will compile the React, TypeScript and SASS modules to JavaScript and CSS.\n\n### Django migrations\n\nRun `python visualCSV\\manage.py migrate`. This will create the Django tables onto the visualCSV database.\n\n### Start development\n\nRun `python visualCSV\\manage.py runserver` to start the development start and run `npm start` to start watching for changes which would require node to recompile certain files.\n\nA script exists which will start the JavaScript unit test on watch mode. This can be run using: `npm run test:watch`.\n\n## Production\n\nThe following instructions will enable you to deploy the application using [Docker Stack](https://docs.docker.com/engine/reference/commandline/stack/) to deploy a stack to a swarm.\nThe documentation will describe the steps needed to install the application on an Ubuntu 18.04.3 x64 machine.\n\n### Install Docker\n\nEnsure that whichever cloud provider you are using does have docker installed. To quickly check if your machine has docker installed, run `docker --version`.\nIf you need to install docker, you can do so by running the following:\n\nbash\n```\ncurl -fsSL https://get.docker.com -o get-docker.sh\nsudo sh get-docker.sh\n```\n\nThe application runs 5 services with 3 replicas of the web service. You may wish to edit this to include other replicas and settings.\nHaving 5 services does not mean that you must have 5 machines, all of the services can be run from a single machine.\n\n### Initialise Swarm\n\nOn one of your machines, run `docker swarm init  --advertise-addr \u003cip address\u003e`.\nThis will initialise the swarm and provide two commands using which you can either add a worker to the swarm or add a manager.\nWe will add 4 other managers, to do so, run `docker swarm join-token manager` on each of the other machines.\n\n### Update Environment Variables\n\n```bash\nbash quick_start_env_files.sh\n```\nThis will very quickly create .env files which docker will expect when building the stack.\nWe will test that the stack can be deployed, and once we are happy, we will stop the stack, update the .env files and redeploy the stack.\n\n```bash\ndocker stack deploy -c output.yml visualCSV\n```\n\nRun `docker stack services visualCSV` to check the progress of the swarm. It is ready when all the replicas show the same number on the left and right. e.g: 0/3 means not ready, 3/3 means ready.\n\nThis may take a while as the web service needs to run through a number of installations and so you may find that web service and the nginx service (it depends on the web service) take longer than others to initialise.\n\nOnce all the services are running, access the public IP address to test that the application is running.\n\nNext stop the stack and update the env file:\n\n```bash\ndocker stack rm visualCSV\nfind | grep \".*\\.env$\"\n```\n\nThis returns a list of env files that need to be updated. Update the value in each file and re-run `docker stack deploy -c output.yml visualCSV`.\nOnce everything is running again run `find | grep \".*\\.env$\" | while read in; do rm \"$in\"; done` to remove all the .env files.\n\n## Updating Webiste\nThe image is held in Docker Hub and automation is setup were commits to the master branch will be picked up by Docker Hub resulting in a new image to be created. Once complete, run ``docker service update {{service name}}``.\n\n## Testing\n\nThe packages comes with Python and JavaScript unit tests.\n\nTo run the Python unit tests run `python visualCSV\\manage.py test`.\n\nTo test JavaScript unit tests run `npm run test`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Fvisualcsv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalaah01%2Fvisualcsv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalaah01%2Fvisualcsv/lists"}