{"id":23036740,"url":"https://github.com/miguelamello/python-fleet-logbook","last_synced_at":"2026-04-12T10:48:09.581Z","repository":{"id":171801294,"uuid":"648428476","full_name":"miguelamello/python-fleet-logbook","owner":"miguelamello","description":"This is an implementation of a Fleet Logbook Service responsible for receiving, collecting, processing and presenting data from \"VDR - Vessel Data Recorders\" and making it available for further analysis or storage.","archived":false,"fork":false,"pushed_at":"2023-06-11T20:12:04.000Z","size":8406,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-08T13:24:47.951Z","etag":null,"topics":["api-gateway","dashboard","flask","graphql","microservices","mongodb","python","socket-io","software-engineering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miguelamello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-02T00:33:46.000Z","updated_at":"2023-06-10T17:51:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"ebf5472b-d12c-467d-b0f8-42e0dcdeef21","html_url":"https://github.com/miguelamello/python-fleet-logbook","commit_stats":null,"previous_names":["miguelamello/python-stream-ingestor","miguelamello/python-ship-logbook","miguelamello/python-fleet-logbook"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelamello%2Fpython-fleet-logbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelamello%2Fpython-fleet-logbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelamello%2Fpython-fleet-logbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelamello%2Fpython-fleet-logbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miguelamello","download_url":"https://codeload.github.com/miguelamello/python-fleet-logbook/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246905871,"owners_count":20852819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-gateway","dashboard","flask","graphql","microservices","mongodb","python","socket-io","software-engineering"],"created_at":"2024-12-15T17:27:19.624Z","updated_at":"2025-12-30T23:10:33.725Z","avatar_url":"https://github.com/miguelamello.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python Fleet Logbook\n\nThis is an implementation of a Fleet Logbook Service responsible for receiving, collecting, processing and presenting data from \"VDR - Vessel Data Recorders\" and making it available for further analysis, storage and searchability. The primary goal of a Fleet Logbook Service is maintaining a precise log of essential vessel instruments readings such as vessel position, speed, heading, engine parameters, rudder movements, communications, alarms and many others. The secondary goal is to provide a way to access this data in a convenient way. For such a task the service provides a \"Ingestor Service\" to receive data from the VRD and a \"Dashboard Service\" to provide access to the data though a dashboard. The services also provides a \"Disgestor Service\" to process the data and make it available for further analysis,  storage or disposal through a API so other services can access it.\n\nNote: In this implementation it's been target a fleet of ships, but the service can be used for any kind of vehicle, as long as it has a VDR - Vessel Data Recorder or similar device.\n\n## 1) Architecture\n\nThe Fleet Logbook Service is composed of three main components:\n\n- Ingestor Service\n- Dashboard Service\n- Disgestor Service\n\n**1.1) Ingestor Service**\n\nThe Ingestor Service is responsible for receiving data from the VDRs and storing it in a database. The Ingestor connects to the VDRs through a TCP connection and receives data in the form of NMEA sentences. The Ingestor parses the NMEA sentences and stores the data in a database. As the ship may have issues regarding connectivity, the Ingestor is responsible for keeping trying to connect to the VDRs and each time it connects it will pull all the data that was not received yet. Readings should be saved in a cronomological order, so the Ingestor should be able to detect if readings are out of order, it should be able to detect duplicate entries. Readings should be saved to database in a way that it is easy to query them by time range, by type of reading, by ship, by ship and time range, by ship and type of reading, by ship, type of reading and time range. The Ingestor should be able to detect if the VDR is sending data that is not in the NMEA format and alert developers. \n\n**1.2) Dashboard Service**\n\nThe Dashboard Service is responsible for providing a user graphic interface to access the data. The dashboard should be able to show the data in a way that is easy to understand and navigate. The dashboard should be compatible with desktop and mobile browsers, it also should have a responsive design. Graphic interface should be easy to understand and should be as clear as possible. The dashboard should permit to choose from all available ships, may they be active or inactive. Active meaning when the VDR - Vessel Data Recorder is turned on, and inactive meaning when the VDR is turned off. The dashboard should inform if any expected reading from VDR is not present and alert developers. \n\n**1.3) Disgestor Service**\n\nThe Disgestor Service is responsible to post-processing VDR data and make this data searchable through a GraphQL API. Disgestor should allow direct result data from database, but also asking for data in a asyncronous way. For example, the client may query for a specific data and the Disgestor should return a \"pending\" status and a \"request id\". The client may then query for the same data using the \"request id\" and the Disgestor should return the data if it is ready.\n\n## 2) VDR - Vessel Data Recorder\n\nShips often have dedicated devices known as VDR, which stands for \"Voyage Data Recorders\" or \"Vessel Data Recorders\", that collect and store various vessel data readings. These recorders are specifically designed to capture and record critical data related to the vessel's navigation, operation, and safety.\n\nVDRs are similar to \"black boxes\" used in aircraft and are mandated by international maritime regulations, such as the International Maritime Organization's (IMO) Safety of Life at Sea (SOLAS) convention. They are typically installed on larger ships, including commercial vessels, passenger ships, and certain types of offshore vessels.\n\nVDRs are equipped with sensors and interfaces to collect data from various onboard systems, including navigation instruments, radar systems, gyrocompasses, GPS receivers, engine and machinery sensors, bridge audio communications, and other relevant equipment. The collected data can include information such as vessel position, speed, heading, engine parameters, rudder movements, communications, and alarms.\n\nIn addition to data collection, VDRs often have the capability to record audio and capture images from bridge cameras or other surveillance systems. This comprehensive data recording helps in investigating accidents, understanding vessel operations, analyzing incidents, and improving safety and operational practices.\n\nThe recorded data from VDRs is typically stored in a secure and tamper-proof manner, and in the event of an incident, it can be retrieved for analysis and investigation purposes by authorities or ship operators.\nHowever, VDRs can also streamming data in real time to a remote server or allow remote connections to download the data. This is the case of the VDRs used in this project.\n\n**2.1) NMEA - National Marine Electronics Association**\n\nThe NMEA 0183 standard defines an electrical interface and data protocol for communications between marine electronic devices such as echo sounder, sonars, anemometer (wind speed and direction), gyrocompass, autopilot, GPS receivers and many other types of instruments. It has been defined by, and is controlled by, the US-based National Marine Electronics Association.\n\nNMEA sentences are standardized data formats used in marine navigation and communication systems. These sentences are ASCII-based and provide a structured way to transmit various types of data between marine electronic devices. NMEA sentences typically start with a dollar sign ($) followed by a two-letter talker ID that identifies the type of device or system providing the data.\n\nHere are a few examples of commonly used NMEA sentences:\n\n**Global Positioning System (GPS) Data:**\n\n**$GPGGA: GPS Fix Data**\n\n\u003ccode\u003e$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47\u003c/code\u003e\n\nThis sentence provides GPS fix information, including time, latitude, longitude, number of satellites in use, altitude, and more.\n\n**$GPRMC: Recommended Minimum Navigation Information**\n\n\u003ccode\u003e$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A\u003c/code\u003e\n\nThis sentence provides essential GPS data such as time, date, latitude, longitude, speed, and course.\n\n**Depth Data:**\n\n**$SDDBT: Depth below transducer**\n\n\u003ccode\u003e$SDDBT,12.3,f,3.7,M,2.0,F*0C\u003c/code\u003e\n\nThis sentence provides depth information measured by the transducer in both feet and meters.\n\n**Wind Data:**\n\n**$WIMWV: Wind Speed and Angle**\n\n\u003ccode\u003e$WIMWV,84.3,R,12.4,N,A*0B\u003c/code\u003e\n\nThis sentence provides wind speed and angle information, including relative (R) or true (T) wind direction.\n\n**AIS (Automatic Identification System) Data:**\n\n**$AIVDM: AIS VHF Data Link Message**\n\n\u003ccode\u003e$AIVDM,1,1,,A,13aH52P0tP00l4CNOvNk9An00SGt,0*7E\u003c/code\u003e\n\nThis sentence contains encoded AIS information transmitted by nearby vessels.\n\n**Compass Data:**\n\n**$HCHDG: Heading, Deviation, and Variation**\n\n\u003ccode\u003e$HCHDG,179.8,,,,*14\u003c/code\u003e\n\nThis sentence provides heading information from a compass, including deviation and variation.\n\n\nThese examples represent just a few of the many NMEA sentence formats available. Different marine devices and systems support various NMEA sentences to facilitate data exchange and interoperability between equipment on board a vessel.\n\n## 3) Ingestor Service - Implementation\n\nThe Ingestor Service is implemented in Python 3.11 using the following libraries: \n\n- signal \n- socket \n- sys, \n- time\n- threading\n- logging\n- dotenv\n- mongo_client\n- nmea\n\nThe service is composed of the following main tasks taht should be implemented:\n\n**3.1) VDR - Vessel Data Recorder remote access**\n\nIn a real production ready service, we would have to implement a way to access the VDRs remotely. This could be done in many ways, but the most common way is to use a TCP connection. The VDRs would be configured to open a TCP port and listen for connections. The Ingestor Service would then connect to the VDRs and start receiving data. In this project we will simulate this by using a NMEA datalog file filled with NMEA sentences. The Ingestor Service will read the file and parse the sentences, and deliver each sequence in a 10 seconds interval.\n\nHere is an example of a NMEA datalog file:\n\n```\n  $GPGLL,4916.45,N,12311.12,W,2023-06-06T22:54:44,A,*1D\n  $GPGLL,4917.60,N,12310.25,W,2023-06-06T22:54:45,A,*45\n  $GPGLL,4918.75,N,12309.38,W,2023-06-06T22:54:46,A,*5A\n  $GPGLL,4919.90,N,12308.51,W,2023-06-06T22:54:47,A,*6F\n  $GPGLL,4921.05,N,12307.64,W,2023-06-06T22:54:48,A,*78\n  $GPGLL,4922.20,N,12306.77,W,2023-06-06T22:54:49,A,*89\n  $GPGLL,4923.35,N,12305.90,W,2023-06-06T22:54:50,A,*9A\n  $GPGLL,4924.50,N,12305.03,W,2023-06-06T22:54:51,A,*AB\n  $GPGLL,4925.65,N,12304.16,W,2023-06-06T22:54:52,A,*BC\n  $GPGLL,4926.80,N,12303.29,W,2023-06-06T22:54:53,A,*CD\n  $GPGLL,4927.95,N,12302.42,W,2023-06-06T22:54:54,A,*DE\n  $GPGLL,4929.10,N,12301.55,W,2023-06-06T22:54:55,A,*EF\n  $GPGLL,4930.25,N,12300.68,W,2023-06-06T22:54:56,A,*F0\n  ...\n```\n\nFor this demonstration project we will use only `$GPGLL: Geographic Latitude and Longitude` sentences. However, in a real production ready service we would have to implement a way to parse all the NMEA sentences that the VDRs are sending.\n\n**3.2) NMEA sentence parsing**\n\nThe Ingestor Service should be able to parse the NMEA sentences in a better fitted format for storage in a database and for further processing. This can be achieved by using the standard Python Type called `dict` that can hold data in many formats. In our specific usecase we will use the `dict` type to hold the data in a key-value format. For example, the `$GPGLL` sentence will be parsed as follows:\n\n```\n  {\n    'source': '$GPGLL', \n    'latitude': '5019.70N', \n    'longitude': '12223.27W', \n    'utctime': '2023-06-06T22:54:56' \n  }\n```\n\nWith this format we can easily store the data in a database and also process it in a easy way.\n\nNote: By now we are collecting NMEA sentences from a unique device only. In a real production ready service we would have to implement a way to collect data from multiple devices, and so we would have to add a `device_id` field to the `dict` type. This field would be used to identify the device that sent the data. Maybe wwe could use the serial number of the device as the `device_id`.\n\n**3.3) Database storage of sentences**\n\nThe Ingestor Service should be able to store the parsed sentences in a database. For this demonstration project we will use MongoDB as the database. MongoDB is a document-oriented database that stores data in flexible, JSON-like documents, meaning fields can vary from document to document and data structure can be changed over time. The document model maps to the objects in your application code, making data easy to work with. Ad hoc queries, indexing, and real time aggregation provide powerful ways to access and analyze your data. MongoDB is a distributed database at its core, so high availability, horizontal scaling, and geographic distribution are built in and easy to use.\n\nFor performance reasons we setted up a `time-series` colletion, and so before saving, some transformation should be made, as follows:\n\n```\n{\n  'timestamp': '2023-06-06T22:58:03',\n  'metadata': {\n    'source': '$GPGLL', \n    'latitude': '5019.70N', \n    'longitude': '12223.27W', \n    'utctime': '2023-06-06T22:54:56' \n  }\n}\n```\nNow we can use all the power of MongoDB TimeSeries to query the data in a easy way. The beauty of NoSQL databases like MongoDB is that we can store data in a flexible way, without the need to define a schema in advance. This is very useful when we are dealing with data that is not well defined, as is the case of NMEA sentences. Another advantage on MongoDB is that we can store data in a JSON format, and JSON is a very popular format that is easy to understand and manipulate.\n\n## 4) Disgestor Service - Implementation\n\nThe Disgestor Service will is implemented as a GraphQL API using the following libraries: \n\n  - **Flask**, which is a lightweight web application framework, designed to make getting \n  started quick and easy, with the ability to scale up to complex applications.\n\n  - **Ariadne**, which is a Python library for implementing GraphQL servers \n  using schema-first approach.\n\nWe choosed to make the Disgestor Service a GraphQL API because GraphQL is a query language for APIs and a runtime for fulfilling those queries with existing data. GraphQL provides a complete and understandable description of the data as an API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.\n\nThe service is composed of the following main tasks taht should be implemented:\n\n**4.1) GraphQL Schema**\n\nThe GraphQL schema is the core of any GraphQL server implementation. It describes the functionality available to the client applications that connect to it. The schema defines the available API operations and their parameters and return types. It also defines the relationships between the different data types exposed by the API. For this project we will use the following schema:\n\n```\n  # This is a GraphQL schema file\n\n  # This type defines the readings sent by the VDRs\n  type Reading {\n    _id: ID!\n    timestamp: String!\n    metadata: Device!\n  }\n\n  # This type defines the devices that sent the readings\n  type Device {\n    source: String!\n    latitude: String!\n    longitude: String!\n    utctime: String!\n  }\n\n  # This type defines the vessels that the devices are installed\n  type Vessel {\n    _id: ID!\n    name: String!\n    type: VesselType!\n    capacity: Int\n    owner: String\n    registrationNumber: String\n    manufacturingDate: String\n    length: Float\n    width: Float\n    height: Float\n    weight: Float\n    status: VesselStatus!\n    createdAt: String!\n    updatedAt: String!\n  }\n\n  # This type defines the vessel types\n  enum VesselType {\n    CARGO\n    PASSENGER\n    FISHING\n    CRUISE\n    TANKER\n    TUGBOAT\n    DRILLER \n    ORTHER\n  }\n\n  # This type defines the vessel status\n  enum VesselStatus {\n    STANDBY\n    MAINTENANCE\n    DECOMMISSIONED\n    DOCKED\n    OPERATING\n  }\n\n  # This type defines the queries that can be made to the API\n  type Query {\n    getReadings: [Reading]\n    getReadingsByTimeRange(start: timespamp, end: timestamp): [Reading]\n    getReadingsById(id: ID!): [Reading]\n    getReadingsByDevice(serial: String!): [Reading]\n    getReadingsByDeviceAndTimeRange(serial: String!, start: timespamp, end: timestamp): [Reading]\n    getVessels: [Vessel]\n    getVesselsByType(type: VesselType!): [Vessel]\n    getVesselsByStatus(status: VesselStatus!): [Vessel]\n    getVesselById(id: ID!): Vessel\n    getVesselByName(name: String!): Vessel\n  }\n```\n\n**4.2) Queries**\n\nThe Disgestor Service should be able to answer the following queries:\n\n - getReadings\n - getReadingsByTimeRange\n - getReadingsById\n - getReadingsByDevice\n - getReadingsByDeviceAndTimeRange\n - getVessels\n - getVesselsByType\n - getVesselsByStatus\n - getVesselById\n - getVesselByName\n\n **4.2.1) getReadings**\u003cbr\u003e\nReturn all readings from all devices limiting in 1000 newest records.\n\nQuery example:\n\n```\n  query {\n    getReadings {\n      _id\n      timestamp\n      metadata {\n        source\n        latitude\n        longitude\n        utctime\n      }\n    }\n  }\n```\n\nResponse example:\n\n```\n  {\n    \"data\": {\n      \"getReadings\": [\n        {\n          \"_id\": \"6480d862b189fec83374c22f\",\n          \"timestamp\": \"2023-06-07 19:20:02.256000\", \n          \"metadata\": {\n            \"latitude\": \"5008.20N\",\n            \"longitude\": \"12231.97W\",\n            \"source\": \"$GPGLL\",\n            \"utctime\": \"2023-06-06T22:55:29\", \n            \"serialNumber\": \"9876543210\"\n          }\n        },\n        {\n          \"_id\": \"6480d86cb189fec83374c230\",\n          \"timestamp\": \"2023-06-07 19:20:12.259000\", \n          \"metadata\": {\n            \"latitude\": \"4950.95N\",\n            \"longitude\": \"12245.02W\",\n            \"source\": \"$GPGLL\",\n            \"utctime\": \"2023-06-06T22:55:14\",\n            \"serialNumber\": \"4321098765\"\n          }\n        },\n        ...\n      ]\n    }\n  }\n```\n\n **4.2.2) getReadingsByTimeRange**\u003cbr\u003e\nReturn all readings from all devices within the time range, limiting in 1000 newest records.\n\nQuery example:\n\n```\n  query {\n    getReadingsByTimeRange(\n      start: \"2023-06-07 19:20:02.256000\"\n      end: \"2023-06-07 19:23:45.489000\"\n    ) {\n      _id\n      timestamp\n      metadata {\n        source\n        longitude\n        latitude\n        utctime\n      }\n    }\n  }\n```\n\nResponse example:\n\n```\n  {\n    \"data\": {\n      \"getReadingsByTimeRange\": [\n        {\n          \"_id\": \"6480d862b189fec83374c22f\",\n          \"metadata\": {\n            \"latitude\": \"5008.20N\",\n            \"longitude\": \"12231.97W\",\n            \"source\": \"$GPGLL\",\n            \"utctime\": \"2023-06-06T22:55:29\"\n          },\n          \"timestamp\": \"2023-06-07 19:20:02.256000\"\n        },\n        {\n          \"_id\": \"6480d86cb189fec83374c230\",\n          \"metadata\": {\n            \"latitude\": \"4950.95N\",\n            \"longitude\": \"12245.02W\",\n            \"source\": \"$GPGLL\",\n            \"utctime\": \"2023-06-06T22:55:14\"\n          },\n          \"timestamp\": \"2023-06-07 19:20:12.259000\"\n        },\n        {\n          \"_id\": \"6480d941f028896ff7c6d17c\",\n          \"metadata\": {\n            \"latitude\": \"5025.45N\",\n            \"longitude\": \"12218.92W\",\n            \"source\": \"$GPGLL\",\n            \"utctime\": \"2023-06-06T22:55:44\"\n          },\n          \"timestamp\": \"2023-06-07 19:23:45.489000\"\n        }\n      ]\n    }\n  }\n```\n\n## 5) Dashboard Service - Implementation\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiguelamello%2Fpython-fleet-logbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiguelamello%2Fpython-fleet-logbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiguelamello%2Fpython-fleet-logbook/lists"}