{"id":27693180,"url":"https://github.com/jesperkuipers/dataproccesing","last_synced_at":"2026-05-14T21:36:39.184Z","repository":{"id":121617241,"uuid":"339476553","full_name":"JesperKuipers/DataProccesing","owner":"JesperKuipers","description":"📊 School assignment to build a REST API and an application that uses the API","archived":false,"fork":false,"pushed_at":"2021-04-12T11:24:14.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-07T13:38:16.403Z","etag":null,"topics":["json","school","xml"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JesperKuipers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-02-16T17:28:57.000Z","updated_at":"2021-04-12T11:24:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"29f471a3-cb6d-44c1-afee-71c9b89087ea","html_url":"https://github.com/JesperKuipers/DataProccesing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JesperKuipers/DataProccesing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JesperKuipers%2FDataProccesing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JesperKuipers%2FDataProccesing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JesperKuipers%2FDataProccesing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JesperKuipers%2FDataProccesing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JesperKuipers","download_url":"https://codeload.github.com/JesperKuipers/DataProccesing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JesperKuipers%2FDataProccesing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33044337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","school","xml"],"created_at":"2025-04-25T12:55:21.818Z","updated_at":"2026-05-14T21:36:39.178Z","avatar_url":"https://github.com/JesperKuipers.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/44586411/110249754-f6009080-7f77-11eb-9148-ff56c1f26c50.gif\" width=\"80%\"\u003e\n\u003c/p\u003e\n\n# DataProccesing\n\u003cp\u003eWelcome to my school project for the class dataproccesing.\u003c/p\u003e\n\n## Introduction\n\u003cp\u003eThis project is made for a school assignment where I needed to process data from a database to a consumer application.\u003cbr\u003e\nThe goal is to make a REST API that uses XML and JSON.\nWe need to make an application that uses the REST API, the apps need to switch between XML and JSON.\u003c/p\u003e\n\n## Table of Contents\n* [Introduction](#introduction)\n* [Table of Contents](#table-of-contents)\n* [Tools used](#tools-used)\n* [Database Import](#database-import)\n* [REST API](#rest-api)\n* [Consumer application](#consumer-application)\n* [How to install the database](#how-to-install-the-database)\n\n\n## Tools used\n| Tool            |  Used for                         | Website                                                                                                   |\n| :-------------: | :-------------:                   | :-----------------------:                                                                                 |\n| XAMPP           |  Hosting local MySQL DB           | \u003ca href=\"https://www.apachefriends.org/index.html\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e  |\n| HeidiSQL        |  Importing SQL dump               | \u003ca href=\"https://www.heidisql.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e                 |\n| EmEditor        |  Editing and splitting .sql file  | \u003ca href=\"https://www.emeditor.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e                 |\n| SSMA            |  Migration tool from MySQL to MSSQL | \u003ca href=\"https://www.microsoft.com/en-us/download/details.aspx?id=54257\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e                               |\n| SSMS            |  Management Studio for MSSQL      | \u003ca href=\"https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver15\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e |\n| Swagger         |  REST API Documentation           | \u003ca href=\"https://swagger.io/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eWebsite\u003c/a\u003e                       |\n\n\n## Database Import\n\u003cp\u003eI downloaded the Dataset here: https://steam.internet.byu.edu/ \u003c/p\u003e\n\u003cp\u003eAfter downloading the dataset I found out that importing a 160 GB dataset is a challenge in itself, not even talking about making a MySQL Dump into a working MS SQL Database.\u003c/p\u003e\n\nI used the following command to import it into PhpMyAdmin\n```\nmysql --init-command=\"SET SESSION FOREIGN_KEY_CHECKS=0;SET UNIQUE_CHECKS=0;\" -u root -p DataBaseNamePlaceholder \u003c C:\\NameOfTheSqlFile\n```\n\u003cbr\u003e\n\u003cp\u003eBut quickly switched to \u003ca href=\"https://www.heidisql.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eHeidiSQL\u003c/a\u003e mainly to check the progress. HeidiSQL also splits the file into multiple queries, see the GIF for an example.\u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://user-images.githubusercontent.com/44586411/110246660-c302d080-7f68-11eb-9403-fcaa29c3c854.gif\"\u003e\u003c/img\u003e\u003c/p\u003e\n\n\u003cp\u003eThe SQL file (160 GB) is too big, even tho HeidiSQL splits the file into multiple queries.\u003cbr\u003e\nI solved this issue by using \u003ca href=\"https://www.emeditor.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eEmEditor\u003c/a\u003e.\u003cbr\u003e \nEmEditor is made to load big files and also works great to edit big files.\u003cbr\u003e \nBut the feature I used the most is the splitting tool, with this I was able to split the SQL file down into 8 files.\u003cbr\u003e \nEach file consists out of 1 table, except for some small tables.\u003c/p\u003e\n\n\u003cp\u003eWith the smaller SQL files I was able to import the tables I need into MySQL with the help of HeidiSQL.\u003c/p\u003e\n\n\u003cp\u003eThe SQL file is imported in MySQL, but the end goal is to have it in MSSQL.\u003cbr\u003e\n  I used \u003ca href=\"https://www.microsoft.com/en-us/download/details.aspx?id=54257\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eSSMA (SQL Server Migration Assistant)\u003c/a\u003e to migrate the MySQL datbase into MSSQL\n\n## REST API\n\u003cp\u003eI used ASP.NET Core Web API for the API. It was the first time using Core instead of Net and the first time I made a REST API.\u003cbr\u003e\n  I used \u003ca href=\"https://swagger.io/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eSwagger\u003c/a\u003e to create documentation for my REST API which saved me a lot of time which is really great.\u003cbr\u003e\n    - First step was making models so I could scaffold them to create controllers.\u003cbr\u003e\n    - Second step was to add pagination so the API wouldn't crash with the huge amount of rows in the database.\u003cbr\u003e\n    - As of last I created one GET, POST, PUT and DELETE for each database table as are the requirements from school.\u003cbr\u003e\u003c/p\u003e\n\n## Consumer application\nI didn't have enough time to create the application. Lucky enough I got a 7.5/10 so I won't be making the application.\n\n## How to install the database\n\u003cimg align=\"right\" src=\"https://user-images.githubusercontent.com/44586411/113137878-62904780-9225-11eb-91ca-7d19ca978e76.gif\" alt=\"Gif that shows the process to import a SSMS back-up\"\u003e\u003c/img\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjesperkuipers%2Fdataproccesing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjesperkuipers%2Fdataproccesing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjesperkuipers%2Fdataproccesing/lists"}