{"id":18720848,"url":"https://github.com/msichterman/basic-data-analysis-system","last_synced_at":"2026-04-06T01:33:23.499Z","repository":{"id":105343573,"uuid":"256076979","full_name":"msichterman/basic-data-analysis-system","owner":"msichterman","description":"Web-based, interactive, end-to-end data analysis system using careful and intentional relational database modeling techniques.","archived":false,"fork":false,"pushed_at":"2022-12-12T12:15:13.000Z","size":45279,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-03T14:21:50.081Z","etag":null,"topics":["c","css","database","expressjs","html","javascript","nodejs","postgresql"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msichterman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-16T01:21:11.000Z","updated_at":"2023-03-09T02:29:04.000Z","dependencies_parsed_at":"2023-05-18T07:45:48.934Z","dependency_job_id":null,"html_url":"https://github.com/msichterman/basic-data-analysis-system","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/msichterman/basic-data-analysis-system","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msichterman%2Fbasic-data-analysis-system","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msichterman%2Fbasic-data-analysis-system/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msichterman%2Fbasic-data-analysis-system/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msichterman%2Fbasic-data-analysis-system/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msichterman","download_url":"https://codeload.github.com/msichterman/basic-data-analysis-system/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msichterman%2Fbasic-data-analysis-system/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31456659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"ssl_error","status_checked_at":"2026-04-05T21:22:51.943Z","response_time":75,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","css","database","expressjs","html","javascript","nodejs","postgresql"],"created_at":"2024-11-07T13:32:48.359Z","updated_at":"2026-04-06T01:33:23.482Z","avatar_url":"https://github.com/msichterman.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![MSICH - Personal Site](https://img.shields.io/badge/MSICH-Personal%20Site-0892d0)](https://msich.dev/)\u0026emsp;\n[![Twitter Follow](https://img.shields.io/twitter/follow/mattsichterman?style=social)](https://twitter.com/mattsichterman)\u0026emsp;\n[![Follow on GitHub](https://img.shields.io/github/followers/msichterman?label=Follow%20on%20Github\u0026style=social)](https://github.com/msichterman)\u0026emsp;\n# Build An End-to-End E-R Data Analysis Workflow\n***Disclaimer: This repository is public to showcase skills developed from this assignment. I am in no way reponsible for any academic integrity issues should any code be re-used or copied from any part of this assignment.**\n\n## Motivation\nThis site was created to satisfy the requirements of an assignment in my `CSCE 411: Data Modeling for Systems Development` class at the University of Nebraska-Lincoln. The assignment was intended to help build the following skills:\n\n* Build a web-based interactive data analysis system\n* Understand the three types of data modeling techniques for database design\n\nIn real-world settings, a dataset may not be directly used with a relational database management system (RDBMS). Different datasets can be provided with different formats(e.g., text, binary, images, and so on). In order to use these datasets in RDBMS, data processing and modeling techniques are typically required. In this assignment, we were provided with a dataset in a zip archive file, and after decompression there were about 2,000 data files, named from record_000000.datto to record_001999.dat. Each data file corresponds to a synthesized record containing User ID, User Name, User Location, Number of Messages, and Message information (Send Time and Send Text) in a binary format. After processing these datasets, we were instructed to conceptually, logically, and physically model the data. From there, we added the data to a PostgreSQL database and created a web interface that displays the results of four required database queries.\n\n![](End-to-EndAppDemo.gif)\n\n## Technologies\n### Data Preprocessing\n* C\n### Back-End\n* Node.js\n    * [Express.js](https://expressjs.com/)\n* PostgreSQL\n    * [node-postgres](https://node-postgres.com/)\n    * [pgAdmin 4](https://www.pgadmin.org/download/)\n### Front-End\n* JavaScript\n* HTML\n* CSS\n\n## Getting Started\nIn order to use the application locally, you'll need to do a few things first.\n\n1. Clone this repository and `cd` into the repository's directory\n```\ngit clone https://github.com/msichterman/CSCE411-Assignment4-DataAnalysisWorkflow.git \u0026\u0026 cd ./CSCE411-Assignment4-DataAnalysisWorkflow\n```\n\n2. Install all of the `node_modules` required for the project. Depending on your computer's configuration, you may need to prefix this command with a `sudo`.\n```\nnpm install\n```\nor\n```\nsudo npm install\n```\n\n3. Lastly, run the following command to get the project off the ground. This command will not only build your JS files, but it will also auto-compile your files on every file save. This allows for hot reloading on-save when the backend and/or frontend code is updated.\n\n```\nnpm run start\n```\n\n4. Head over to [http://localhost:3000](http://localhost:3000) to see the application live!\n\n## Data Preprocessing\nThe data for this assignment was provided in a zip archive file (Assignment4_Dataset_Code.zip). After decompression, there are about 2,000 data files, named from record_000000.dat to record_001999.dat. Each data file corresponds to a synthesized record containing User ID, User Name, User Location, Number of Messages, and Message information (Send Time and Send Text) in a binary format. In order to import this data, we had to preprocess it into .csv files, where each file represents a table in our database, namely ```user``` and ```message```. We were given a few C language files as the code stubs for the assignment and the files were able to read in the data from the binary files and print the data from each file. We extended this functionality by creating the .csv files and adding the necessary information to each. These files could then be imported into our PostgreSQL database. These are the steps to run the preprocessing code:\n1. ```cd``` into the ```Assignment4_Dataset_Code``` folder.\n* ```cd Assignment4_Dataset_Code```\n\n2. A makefile was generated in order to simplify the build process:\n* ```make read_record``` - builds the ```read_record``` executable which prints the contents of a specified record.\n   * ```./read_record \u003crecord_number\u003e``` - For example, running ```./read_record 999``` will print the record of User ID: 999. The program also shows the processing time at the end of its output.\n* ```make process_record``` - builds the ```process_record``` executable which we modified to generate the .csv files for each of our tables given a range of records.\n   * ```./process_record \u003cstart_record_number\u003e \u003cend_record_number\u003e``` - For example, running ```./process_record 14 223``` simply goes through all records from 14 to 223 and generates the .csv files.\n* ```make distclean``` - removes the exectutables from the directory\n\n3. Process all records from 0 to 1999 to generate the .csv files for our user and message tables.\n* ```make process_record \u0026\u0026 ./process_record 0 1999```\n\n4. The .csv files are generated and ready to be imported to PostgresSQL\n\n## Importing Assignment4CompleteDump.sql to Create the Database\n### Installation\nIf you’re using Windows, download a [Windows installer](https://www.postgresql.org/download/windows/) of PostgreSQL.\n\nIf you’re using a Mac, this tutorial assumes you have [Homebrew](https://brew.sh/) installed on your computer as a package manager for installing new programs. If you don’t, simply click on the link and follow the instructions to install Homebrew.\n\nOpen up the Terminal and install ```postgresql``` with ```brew```.\n```\nbrew install postgresql\n```\nAfter the installation is complete, we’ll want to get the ```postgresql``` up and running, which we can do with ```services start```.\n```\nbrew services start postgresql\n==\u003e Successfully started `postgresql` (label: homebrew.mxcl.postgresql)\n```\nIf at any point you want to stop the ```postgresql``` service, you can run ```brew services stop postgresql```.\n\nPostgreSQL is installed now, so the next step is to connect to the postgres command line, where we can run SQL commands to import the database.\n\n**Note:** This installation guide was found [here](https://blog.logrocket.com/setting-up-a-restful-api-with-node-js-and-postgresql-d96d6fc892d8/), much thanks to the [author](http://tania.dev).\n\n### Importing the Database with the PostgreSQL Command Prompt\n1. Start PostgreSQL using the command ```psql```\n2. Create a new database called ```assignment4``` using the command ```CREATE DATABASE assignment4;```\n3. Connect to the created database using the command ```\\c assignment4```\n4. Insert the contents of the ```Assignment4CompleteDump.sql``` file into the new database using the command:\n```\n\\i /\u003cPATH TO LOCAL REPO\u003e/CSCE411-Assignment4-DataAnalysisWorkflow/Assignment4_Dataset_Code/Assignment4CompleteDump.sql\n```\n   * **Note:** This may take a few minutes since there are over 98,000 message records\n5. Now the database should be populated and running, head over to the web application to see the queries in action\n\n## Get In Touch\n* Follow me on [Twitter](https://twitter.com/mattsichterman)\n* Connect with me on [LinkedIn](https://www.linkedin.com/in/msichterman/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsichterman%2Fbasic-data-analysis-system","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsichterman%2Fbasic-data-analysis-system","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsichterman%2Fbasic-data-analysis-system/lists"}