{"id":18870716,"url":"https://github.com/mortonprod/glabs","last_synced_at":"2026-04-16T05:33:42.237Z","repository":{"id":144402085,"uuid":"108979671","full_name":"mortonprod/glabs","owner":"mortonprod","description":"This is a very basic utility page to show financial transactions.","archived":false,"fork":false,"pushed_at":"2017-10-31T15:19:20.000Z","size":41,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-30T18:13:57.635Z","etag":null,"topics":["jade","mongodb","nodejs","xml2js"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mortonprod.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-31T10:23:37.000Z","updated_at":"2017-10-31T12:08:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"7bb595b3-5ab6-417c-aaa6-15d88ca90043","html_url":"https://github.com/mortonprod/glabs","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mortonprod/glabs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortonprod%2Fglabs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortonprod%2Fglabs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortonprod%2Fglabs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortonprod%2Fglabs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mortonprod","download_url":"https://codeload.github.com/mortonprod/glabs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortonprod%2Fglabs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31872653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"online","status_checked_at":"2026-04-16T02:00:06.042Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jade","mongodb","nodejs","xml2js"],"created_at":"2024-11-08T05:22:26.131Z","updated_at":"2026-04-16T05:33:42.215Z","avatar_url":"https://github.com/mortonprod.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Glab\n\nThe app will convert XML files to JSON and store this to a mongodb. \nThe app uses xml2js to convert the XML. This is relatively agnostic about the format of the xml.\nTherefore any information stored as xml can be transformed.\n\n\nThe app allows the output json to be formatted before storing which allows you to easily extend the app without having to rewrite code again and again.\n\n\nThe XML files can be from an s3 bucket or saved locally on the server.\n\n\n*At the moment it only automatically uses a local store but access to s3 is provided.*\n\n\nThe app manages the xml files to ensure we have a complete list of all files from all sources. \nIt also keeps a record of the files processed so we don't lose track of what files need attention.\n\nThe conversion is viewed through a UI rendered by jade(pug). \n\n\n*A submit form has not been added but the backend API route has been added.*\n\n## Building\n\nYou will need the repository:\n\n```\ngit clone https://github.com/mortonprod/glabs .\n```\n\nThen you will need to install the npm packages:\n\n\n```\nnpm install\n```\n\nThe app is built with typescript so to transpile you will need to run \n\n```\nnpm run build\n```\n\nwhich will create the dist folder. \n\n## Documentation\n\nTo build the documentation and server it to your browser run:\n\n```\nnpm run docs\n```\n\nThen go to: [http://localhost:3001/](http://localhost:3001/)\n\nServerDocs.js is the node server which will serve the static documentation assets. \n\n\n## Deployment with docker\n\nThe app had been deployed through a digital ocean droplet with the database on a docker volume.\n\nIf you want to run the docker droplet build: \n\n```\nnpm run docker:build\n```\n\nthen run in detach mode with:\n\n\n```\nnpm run docker:start\n```\n\nYou can then see this running at [http://localhost:3001/](http://localhost:3001/).\nMake sure to kill this afterwards using:\n\n```\ndocker stop \u003cYour process ID\u003e\n```\n\n\nIf you would like to see this running from digital ocean go to [https://glabs.services](https://glabs.services)\n\n## Deployment without docker\n\nIf you want to run this on your local machine you will need to have mongodb installed and then run\n\n```\nnpm run start:db\n```\n\nto start the database. \n\n\nYou can then start your app from another terminal after building it with:\n\n```\nnpm run start\n```\n\n\n## Test\n\n**Note when you run the tests the databases will reset by removing all the old collections**\n\n**The keys provided are for access to s3 only**\n\nNeed to start your local version of mongodb. When this is done run: \n\n```\nnpm run start:db\n```\n\n```\nnpm run test\n```\n\nThe tests only cover the converter and xmlManager since they are the most complicated parts of the app.\n\n## Components\n\n### Converter \n\nThis class exposes the functions needed to convert the xml files to json. \nIt also deals with storing the output to mongodb. \nFurthermore, it will also search the database for projection to store in a separate database.\nThis is done using mongo's aggregate function. \nIt simply takes a command and the collection name it should give the output.\n\n### XML Manager\n\nSince the app should work with remote and local files it needs to keep a store of\n\n* What files we have\n* Where they are\n* Have they been processed\n\nIt also needs to update on certain events\n\n* Had new files been uploaded\n* Have more files been added to s3.\n\nThe app is designed to expect a signal from AWS when the s3 bucket has been changed. \nThis signal will be sent to a particular route and xmlManager will get a list of all the files. \n\n\n*The signal from s3 is not added yet but the backend node code is designed to quickly implement this*\n\n### Server\n\nThe server is the entry point of the app and provides API points to:\n\n* Upload Files\n* Specify s3 directory has been modified\n* Serve UI.\n    * Will provide jade with the formated ReturnedDebitItem information from the mongodb\n\nIt will also initialise everything needed to run the app.\n\n**At the moment all collections in the database are removed on startup to make debugging easier.**\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmortonprod%2Fglabs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmortonprod%2Fglabs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmortonprod%2Fglabs/lists"}