Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mortonprod/glabs
This is a very basic utility page to show financial transactions.
https://github.com/mortonprod/glabs
jade mongodb nodejs xml2js
Last synced: 1 day ago
JSON representation
This is a very basic utility page to show financial transactions.
- Host: GitHub
- URL: https://github.com/mortonprod/glabs
- Owner: mortonprod
- Created: 2017-10-31T10:23:37.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2017-10-31T15:19:20.000Z (about 7 years ago)
- Last Synced: 2024-11-08T05:26:25.452Z (about 2 months ago)
- Topics: jade, mongodb, nodejs, xml2js
- Language: TypeScript
- Size: 40 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Glab
The app will convert XML files to JSON and store this to a mongodb.
The app uses xml2js to convert the XML. This is relatively agnostic about the format of the xml.
Therefore any information stored as xml can be transformed.
The app allows the output json to be formatted before storing which allows you to easily extend the app without having to rewrite code again and again.
The XML files can be from an s3 bucket or saved locally on the server.
*At the moment it only automatically uses a local store but access to s3 is provided.*
The app manages the xml files to ensure we have a complete list of all files from all sources.
It also keeps a record of the files processed so we don't lose track of what files need attention.
The conversion is viewed through a UI rendered by jade(pug).
*A submit form has not been added but the backend API route has been added.*
## Building
You will need the repository:
```
git clone https://github.com/mortonprod/glabs .
```
Then you will need to install the npm packages:
```
npm install
```
The app is built with typescript so to transpile you will need to run
```
npm run build
```
which will create the dist folder.
## Documentation
To build the documentation and server it to your browser run:
```
npm run docs
```
Then go to: [http://localhost:3001/](http://localhost:3001/)
ServerDocs.js is the node server which will serve the static documentation assets.
## Deployment with docker
The app had been deployed through a digital ocean droplet with the database on a docker volume.
If you want to run the docker droplet build:
```
npm run docker:build
```
then run in detach mode with:
```
npm run docker:start
```
You can then see this running at [http://localhost:3001/](http://localhost:3001/).
Make sure to kill this afterwards using:
```
docker stop
```
If you would like to see this running from digital ocean go to [https://glabs.services](https://glabs.services)
## Deployment without docker
If you want to run this on your local machine you will need to have mongodb installed and then run
```
npm run start:db
```
to start the database.
You can then start your app from another terminal after building it with:
```
npm run start
```
## Test
**Note when you run the tests the databases will reset by removing all the old collections**
**The keys provided are for access to s3 only**
Need to start your local version of mongodb. When this is done run:
```
npm run start:db
```
```
npm run test
```
The tests only cover the converter and xmlManager since they are the most complicated parts of the app.
## Components
### Converter
This class exposes the functions needed to convert the xml files to json.
It also deals with storing the output to mongodb.
Furthermore, it will also search the database for projection to store in a separate database.
This is done using mongo's aggregate function.
It simply takes a command and the collection name it should give the output.
### XML Manager
Since the app should work with remote and local files it needs to keep a store of
* What files we have
* Where they are
* Have they been processed
It also needs to update on certain events
* Had new files been uploaded
* Have more files been added to s3.
The app is designed to expect a signal from AWS when the s3 bucket has been changed.
This signal will be sent to a particular route and xmlManager will get a list of all the files.
*The signal from s3 is not added yet but the backend node code is designed to quickly implement this*
### Server
The server is the entry point of the app and provides API points to:
* Upload Files
* Specify s3 directory has been modified
* Serve UI.
* Will provide jade with the formated ReturnedDebitItem information from the mongodb
It will also initialise everything needed to run the app.
**At the moment all collections in the database are removed on startup to make debugging easier.**