{"id":14956920,"url":"https://github.com/raymondcm/topic_store","last_synced_at":"2026-04-24T03:01:39.527Z","repository":{"id":41902003,"uuid":"257617815","full_name":"RaymondCM/topic_store","owner":"RaymondCM","description":"ROS package used for serialising common ROS messages to a database or filesystem","archived":false,"fork":false,"pushed_at":"2023-03-22T13:49:23.000Z","size":420,"stargazers_count":6,"open_issues_count":7,"forks_count":12,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-20T18:21:52.065Z","etag":null,"topics":["mongodb","ros","storage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RaymondCM.png","metadata":{"files":{"readme":"docs/README.md","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-21T14:13:47.000Z","updated_at":"2025-02-13T22:54:52.000Z","dependencies_parsed_at":"2025-04-12T06:01:38.835Z","dependency_job_id":null,"html_url":"https://github.com/RaymondCM/topic_store","commit_stats":null,"previous_names":["raymondkirk/topic_store"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/RaymondCM/topic_store","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaymondCM%2Ftopic_store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaymondCM%2Ftopic_store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaymondCM%2Ftopic_store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaymondCM%2Ftopic_store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RaymondCM","download_url":"https://codeload.github.com/RaymondCM/topic_store/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaymondCM%2Ftopic_store/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32207191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T01:12:49.758Z","status":"online","status_checked_at":"2026-04-24T02:00:07.115Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mongodb","ros","storage"],"created_at":"2024-09-24T13:13:44.404Z","updated_at":"2026-04-24T03:01:39.522Z","avatar_url":"https://github.com/RaymondCM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Topic Store\n\n![CI](https://github.com/RaymondKirk/topic_store/workflows/Topic%20Store/badge.svg?branch=master)\n[![PyPi](http://badge.fury.io/py/topic-store.svg)](https://pypi.org/project/topic-store/)\n\nTopic Store is a ROS package for storing ROS messages to a database or filesystem.\n\nUnlike ROS bags Topic Store adds flexibility by serialising all messages into a data hierarchy that's easily \nsearchable with database queries and allows for remote storage. \n\nYou can also use Topic Store as a standalone python package to read and write data without a ROS installation!\n\n# Installation\n### ROS \n\n```bash\n# From source\ncd catkin_ws/src\ngit clone https://github.com/RaymondKirk/topic_store \ncatkin build topic_store\n\n# From apt\nsudo apt install ros-melodic-topic-store  # you need to add the L-CAS ros source\n```\n### Installing without ROS\n\nYou can install topic_store as an independent python2/3 package without a working ROS installation. \n\n```bash\n# From source \ngit clone https://github.com/RaymondKirk/topic_store\ncd topic_store/src\npip install --extra-index-url https://rospypi.github.io/simple/ -e .\n\n# From PyPi\npip install --extra-index-url https://rospypi.github.io/simple/ topic-store\n```\n\nTo install other dependencies i.e. `ros_numpy` you can run the following\n\n```bash\npip install --extra-index-url https://rospypi.github.io/simple/ ros_numpy\n```\n\n# Usage\n\n[![asciicast](https://asciinema.org/a/Cq9i3a41fzuULw52tRLkHvBQS.svg)](https://asciinema.org/a/Cq9i3a41fzuULw52tRLkHvBQS)\n\nCreate a scenario file by following the documentation in [scenarios](https://github.com/RaymondKirk/topic_store/wiki/Scenarios) or by following the example file\n[default_config.yaml](../scenarios/default_config.yaml). The example below will save a history of all messages sent to the ros log topic.\n\n```yaml\n# Save to /path/to/your/scenario/file.yaml. This collection behaviour will save your log history.\ncontext: \"save_ros_logs\"\n\nstorage: \n  \"method\": \"filesystem\"\n  \"location\": \"default\" \n\ndata: { \n  \"ros_msg\": \"/rosout\", \n}\n\ncollection: \n  method: \"event\" \n  watch_topic: \"/rosout\"\n```\n\nIf ```storage.method``` is database ensure that your database is accessible at the host/port in the config file found at\n ```storage.config```. To launch the default database ```roslaunch topic_store start_database.launch```, the default \ndatabase creates/uses a Mongo 4.2 server instance in the default ```${HOME}/.ros/topic_store/database``` \nfolder exposed on ```localhost:65530``` (defined in  ```storage.config``` of the scenario file). \nDocker is a requirement to use a database backend to avoid conflicts with system mongo. A utility script is provided in \n`scripts/install_docker.sh` if you do not have it installed.\n\nLaunch your data collection scenario! \n\n```\nroslaunch topic_store run_scenario.launch scenario_file:=\"/path/to/your/scenario/file.yaml\"\n```\n\n# Examples\n\n## Database data\n\nThe below example shows how to load and use files stored in a database. \n\n```python\nimport topic_store as ts\n\n# Read data\nstorage = ts.load(\"/path/to/scenario/file/containing/db/connection/info.yaml\")\nfor item in storage:\n    print(\"As Python Dict\", item.dict)  # or item[\"key\"]\n    print(\"As ROS Msgs\", item.msgs)  # or item(\"key\")\n\n# Write data\nstorage.insert_one({\"important_data\": \"topic store is great!\"})\n```\n\n## Filesystem data\n\nThe below example shows how to load and use `.topic_store` files, saved from when scenarios are ran with the \n`storage_method=\"filesystem\"` option.\n\n```python\nimport topic_store as ts\n\n# Read data\nstorage = ts.load(\"/path/to/file.topic_store\")\nfor item in storage:\n    print(\"As Python Dict\", item.dict)  # or item[\"key\"]\n    print(\"As ROS Msgs\", item.msgs)  # or item(\"key\")\n\n# Write data\nstorage.insert_one({\"important_data\": \"topic store is great!\"})\n```\n\n## Launch a database\n\nWhen launching a data collection scenario where ```storage.method==database``` you must also launch the database or \nensure it's already running at the uri defined in the host/port parameters of the config file located at ```storage.config```.\n\nTo launch a database prior to running a data collection scenario.\n\n```bash\nroslaunch topic_store start_database.launch scenario_file:=\"/path/to/your/scenario/file.yaml\"\n```\n \n## Convert to ROS bags\n\nFilesystem `.topic_store` files and database collections can be converted to ROS bags.\n\n```bash\n# Convert filesystem files to rosbags\nrosrun topic_store convert.py -i input.topic_store -o output.bag\n\n# Convert database collections to rosbags \n# Pass scenario file containing connection and collection info as the input\nrosrun topic_store convert.py -i scenario_config.yaml -o output.bag\n```\n\n## Convert between Filesystem and Database\n\nFilesystem `.topic_store` files can be migrated to a MongoDB database. Pass the scenario file that contains the database \nconnection information as the output file.\n\n```bash\nrosrun topic_store convert.py -i input.topic_store -o scenario_config.yaml\n```\n\nDatabase collections can be converted to a filesystem `.topic_store` file. Pass the scenario file that contains the database \nconnection information as the input file.\n\n```bash\nrosrun topic_store convert.py -i scenario_config.yaml -o output.topic_store\n```\n\n## Export from Mongodb query into rosbag/filesystem\n\nExample call:\n  \n* With typical mongodb URI for SSL and authentication\n* On database `ff_rasberry` and collection `2020_riseholme_framos_cameras`\n* Specify database connection URI (`-i`) with db name `authSource=database`\n* Including a query (`-q`) for a specific document\n* Add projection to return only sub-documents (`-p`)\n\n```bash\nconvert.py -i \"mongodb://USER:PASS@HOST:PORT/?authSource=ff_rasberry\u0026tls=true\u0026tlsAllowInvalidCertificates=true\" -c 2020_riseholme_framos_cameras -q '{\"_id\":\"ObjectId(5f115ee6af915351df739757)\"}' -p '{\"cameras.top.color\":1, \"robot\": 1}' -o out.bag\n```\n\n## Storing high bandwidth image data\n\nTo store a lot of image data you will be limited to the IO of your machine. \n[Topic Compression](https://github.com/RaymondKirk/topic_compression) offers a compression solution to increase capture performance.\n\n```bash\n# Install the ROS package\ncd catkin_ws/src\ngit clone https://github.com/RaymondKirk/topic_compression\ncatkin build topic_compression\n\n# Compression/Decompression is chosen automatically so just pass the input topic name and optionally the out topic name\nrosrun topic_compression run in:=/camera/colour/image_raw  # out=/camera/colour/image_raw/compressed \nrosrun topic_compression run in:=/camera/depth/image_raw  # out=/camera/depth/image_raw/compressed \n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraymondcm%2Ftopic_store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraymondcm%2Ftopic_store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraymondcm%2Ftopic_store/lists"}