https://github.com/ecopque/ozmap-sync
OZmap Sync is a service built with Node.js + TypeScript for data synchronization between an ISP system (mocked using json-server) and an external management system, simulating OZmap.
https://github.com/ecopque/ozmap-sync
node nodejs typescript
Last synced: 2 months ago
JSON representation
OZmap Sync is a service built with Node.js + TypeScript for data synchronization between an ISP system (mocked using json-server) and an external management system, simulating OZmap.
- Host: GitHub
- URL: https://github.com/ecopque/ozmap-sync
- Owner: ecopque
- Created: 2025-09-08T15:46:17.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-09-09T00:29:58.000Z (10 months ago)
- Last Synced: 2025-09-09T02:46:27.493Z (10 months ago)
- Topics: node, nodejs, typescript
- Language: TypeScript
- Homepage: https://linktr.ee/edsoncopque
- Size: 5.46 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# OZmap Sync
## Overview
**OZmap Sync** is a service built with **Node.js + TypeScript** for **data synchronization** between an ISP system (mocked using `json-server`) and an external management system, simulating **OZmap**.
The system:
- Periodically fetches data from the ISP system.
- Transforms the data into the required format.
- Sends processed data to the target system.
- Manages failures and respects request rate limits.
- Logs all operations and events.
- Stores synchronized records in **MongoDB**.
---
## Technologies Used
- **Node.js** + **TypeScript** → Core of the application
- **json-server** → Mock ISP system
- **Mongoose** → MongoDB integration
- **Axios** → HTTP requests
- **Winston** → Advanced logging
- **Node-cron** → Scheduling synchronization jobs
- **dotenv** → Environment configuration
---
## Project Structure
```bash
ozmap-sync/
├── src/
│ ├── config/
│ │ ├── database.ts # MongoDB connection configuration
│ │ └── env.ts # Environment variables loader (dotenv)
│ │
│ ├── database/
│ │ ├── models/
│ │ │ └── syncRecord.ts # Mongoose schema for sync records
│ │ └── index.ts # Database connection initializer
│ │
│ ├── integrations/
│ │ ├── ispClient.ts # Fetch data from the ISP (json-server)
│ │ └── ozmapClient.ts # Simulated OZmap SDK client
│ │
│ ├── jobs/
│ │ └── scheduler.ts # Cron job to trigger synchronization
│ │
│ ├── services/
│ │ ├── syncService.ts # Main sync process orchestration
│ │ └── transformService.ts # Transforms ISP data to OZmap format
│ │
│ ├── utils/
│ │ ├── logger.ts # Centralized logging (Winston)
│ │ └── rateLimiter.ts # Handles request throttling per minute
│ │
│ ├── app.ts # Initializes app, scheduler, and configs
│ └── index.ts # Main entry point
│
├── logs/
│ ├── combined.log # All application logs
│ ├── error.log # Only error logs
│ ├── exceptions.log # Exceptions captured automatically
│ └── rejections.log # Promise rejections logs
│
├── docs/
│ └── architecture-diagram.png # Architecture diagram for README
│
├── db.json # Mock ISP data (json-server)
├── .env.example # Example environment variables
├── .gitignore # Git ignored files and folders
├── package.json
├── tsconfig.json # TypeScript configuration
└── README.md
```
---
## Project Setup
### **1. Clone the repository**
```bash
$ git clone https://github.com/your-username/ozmap-sync.git
$ cd ozmap-sync
```
### **2. Install dependencies**
```bash
$ npm install
```
### **3. Configure environment variables**
```bash
Create a .env file in the project root based on .env.example:
ISP_BASE_URL=http://localhost:4000
RATE_LIMIT_PER_MIN=50
SYNC_CRON=*/1 * * * *
MONGODB_URI=mongodb://localhost:27017/ozmap_sync
LOG_LEVEL=debug
```
### **4. ISP Mock Server**
```bash
This project uses json-server to simulate the ISP data source.
$ npx json-server --watch db.json --port 4000
```
### **5. Running the Application**
```bash
Development mode
$ npm run dev
. The app will:
. Connect to MongoDB.
. Fetch data from json-server.
. Transform and send data to the target system (mocked OZmap).
. Store processed records in MongoDB.
. Generate detailed logs in the console and the logs/ folder.
```
### **6. Logs & Data Storage**
```bash
Logs are stored in:
logs/
├── combined.log
├── error.log
├── exceptions.log
└── rejections.log
```
### **7. To check synchronized records in MongoDB**
```bash
$ mongosh
$ use ozmap_sync
$ db.syncrecords.find().pretty()
```
### **8. Architecture**
```bash
The architecture is designed to be modular and scalable, with clear separation of concerns:
+--------------------+
| json-server (ISP) |
+--------------------+
↓
fetchISPData()
↓
+--------------------+
| Transform Service |
+--------------------+
↓
+--------------------+
| OZmap Client (SDK) |
+--------------------+
↓
+--------------------+
| MongoDB Storage |
+--------------------+
```
---
## About the Author
. Developer: Edson Copque
. Website: https://linktr.ee/edsoncopque
. GitHub: https://github.com/ecopque
. Signal Messenger: ecop.01