{"id":19900073,"url":"https://github.com/m-lab/jostler","last_synced_at":"2025-09-13T00:36:44.688Z","repository":{"id":62207711,"uuid":"545650316","full_name":"m-lab/jostler","owner":"m-lab","description":"An agent on M-Lab nodes to upload JSON data to GCS in JSONL format for autoloading to BigQuery.","archived":false,"fork":false,"pushed_at":"2024-10-07T13:42:03.000Z","size":301,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-28T06:44:54.560Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-04T18:43:06.000Z","updated_at":"2024-10-07T13:31:12.000Z","dependencies_parsed_at":"2023-11-15T22:33:23.024Z","dependency_job_id":"6be085f0-fa7c-410b-95b2-96626ec51f45","html_url":"https://github.com/m-lab/jostler","commit_stats":{"total_commits":48,"total_committers":5,"mean_commits":9.6,"dds":"0.35416666666666663","last_synced_commit":"8f93f2954f02d43a3ea14d4229fa82629d79c369"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/m-lab/jostler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fjostler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fjostler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fjostler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fjostler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/jostler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fjostler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274902264,"owners_count":25371074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T20:11:04.586Z","updated_at":"2025-09-13T00:36:44.670Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Coverage Status](https://coveralls.io/repos/github/m-lab/jostler/badge.svg?branch=main)](https://coveralls.io/github/m-lab/jostler?branch=main)\n\n# Jostler Design\n\n## 1. Background\n\n`jostler` is the uploader agent of the `Autoloading` project.\nPlease see the design doc\n[Autoloading Measurement Results into BigQuery](https://docs.google.com/document/d/1kJ2oy5MAwYBBCq2mVOoJBjq4zU2xiEy1gRAYNoim920/edit)\nfor details.\n\n## 2.`jostler` operation\n\n`jostler` will support two modes of operation:\n\n1. A **short-lived interactive local mode** to create a BigQuery\ntable schema file for a given datatype.  The\ntable schema file is built using the standard columns and the datatype\nschema. This mode is mainly meant for an operator to examine and verify\nthe table schema before creating a table.\n2. A **long-lived non-interactive daemon mode** to serve as an\nupload agent on M-Lab's nodes.\n\nIn the long-lived non-interactive mode, `jostler` will run as a sidecar\ncontainer of the new measurement container.  The new measurements\nwill write new format files in JSON format to a predefined location\non the local filesystem as described earlier and `jostler` will use\n\u003ccode\u003e[inotify](https://man7.org/linux/man-pages/man7/inotify.7.html)\u003c/code\u003e\nto monitor filesystem events in the predefined location.\n\u003ccode\u003ejostler\u003c/code\u003e will read individual new format files, bundle them\ntogether, compress the bundle, and upload the bundle to GCS.  After a\nsuccessful upload, \u003ccode\u003ejostler\u003c/code\u003e will delete from the local\nfilesystem new format and other files that the new measurement generated.\n\nAny files that were not included in a bundle due to\nany error such as wrong extension (not `.json`), read error, or invalid\nJSON will also be deleted from the local filesystem in order to avoid\nfilling up the node's disk space.\n\nThere are two configurable parameters that control triggering of an\nupload operation:\n\n1. Bundle size (e.g., 30 megabytes)\n2. Bundle age (e.g., 3 hours)\n\nOnce a bundle reaches its maximum allowable size or age, it will be\nuploaded to GCS.\n\n### 2.1. Bundle names\n\nThe location of new format files is predefined in the new measurement\ncontainer as follows:\n\n```\n    /var/spool/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e/\u003cnew-format-data\u003e\n```\n\nThe reason new format pathnames must follow the above convention is\nthat upload agents,  [`pusher`](https://github.com/m-lab/pusher) and\n`jostler`, use the same string of the pathname after `/var/spool`\nas a prefix for GCS object names.  For details, see [Uniform\nNames: Experiments by Any Other Name [Would Not Be As\nSweet]](https://docs.google.com/document/d/1BPt11RK5x6FZTXvWbZaC_GhSlH9vHzhrRP6ca0TBosI/edit#).\n\nFor example, `pusher` creates object names prefixed by\n`ndt/scamper1/2022/09/12`\n\nin the `pusher-mlab-oti` bucket for traceroute data (`scamper1` datatype)\ngenerated as a sidecar service of NDT measurements on 2022/09/12 as we\ncan see below:\n\n```\n    $ gsutil ls gs://pusher-mlab-oti/ndt/scamper1/2022/09/12\n    gs://pusher-mlab-oti/ndt/scamper1/2022/09/12/20220912T143138.409697Z-scamper1-mlab2-gru01-ndt.tgz\n    gs://pusher-mlab-oti/ndt/scamper1/2022/09/12/20220912T143139.800575Z-scamper1-mlab3-gig03-ndt.tgz\n    ...\n```\n\n`jostler` will upload JSONL bundles to a GCS bucket specified by\na flag which can be the same as the current `pusher`'s buckets\n`pusher-mlab-{sandbox,staging,oti}`.  And because `jostler`'s\nGCS object names have the `autoload/\u003cversion\u003e` prefix before\n`\u003cexperiment\u003e/\u003cdatatype\u003e/...` they will be easily distinguished from\n`pusher`'s objects:\n\n```\n    autoload/\u003cversion\u003e/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e\n```\n\nThe purpose of `autoload/\u003cversion\u003e` in the prefix of the object name\nis to support breaking changes to autoloading implementation.\n\nEach data bundle will have the following naming convention:\n\n```\n    prefix=autoload/\u003cversion\u003e/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e\n    \u003cprefix\u003e/\u003ctimestamp\u003e-\u003cdatatype\u003e-\u003cnode\u003e-\u003cexperiment\u003e-data.jsonl.gz\n```\n\n### 2.2. Bundle contents\n\n#### 2.2.1. Standard columns\n\nEach bundle will consist of individual JSON objects (new format files),\none per line, and each line will include a subset of standard columns\nin the first version (v1) of autoloading.  With respect to the standard\ncolumns, it's important to highlight the following:\n\n\n* Since the main objective of autoloading is to avoid parsing, there\nwill be no `parser` record.  Instead, there will be an `archiver` record\nthat `jostler` will add by wrapping raw JSON from new format files\nwithin an outer record.  In this way `jostler` would make it easier\nfor the new measurement to satisfy the standard columns requirement.\nBut third-parties that don't use `jostler` would still be better if they\nincluded fields like `date` (and others in time when we specify more).\n* It will be helpful to have an `id` field.  In fact, this will be a\nrequirement in the future if we ever want to join autoloaded data with,\nsay, the annotation data. However, since this requires more semantic\nawareness of the raw JSON and some way of specifying the format of\nthe `id`, it is not a requirement for autoloading v1.  Aside from\nautoloaded data, we should keep this in mind with the possible future\ngoal of migrating existing JSON `parser` datatypes to be autoloaded.\nThe `id` field could be the filename minus any filename extension to\nencourage services to name files with the UUID or similarly meaningful\nunique identifier. This would preserve semantic opaqueness of the raw\ndata while providing a convention to populate the id field.\n\nVersion 1 of a JSONL bundle will look like the following, pretty printed,\nabbreviated, and showing standard column names in boldface:\n\n```\n    {\n      \"**date**\": \"2022/09/29\",\n      \"**archiver**\": {\n        \"Version\": \"jostler@0.1.7\",\n        \"GitCommit\": \"3ac4528\",\n        \"ArchiveURL\": \"gs://\u003cbucket\u003e/\u003cprefix\u003e/\u003cbundlename\u003e.jsonl.gz\",\n        \"Filename\": \"\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e/\u003cfilename1\u003e.json\"\n      },\n      \"**raw**\": {\n        \"UUID\": \"1234\",\n        \"MeasurementVersion\": \"0.1.2\",\n        \"Field1\": 42\n      }\n    }\n    {\n      \"**date**\": \"2022/09/29\",\n      \"**archiver**\": {\n        \"Version\": \"jostler@0.1.7\",\n        \"GitCommit\": \"3ac4528\",\n        \"ArchiveURL\": \"gs://\u003cbucket\u003e/\u003cprefix\u003e/\u003cbundlename\u003e.jsonl.gz\",\n        \"Filename\": \"\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e/\u003cfilename2\u003e.json\"\n      },\n      \"**raw**\": {\n        \"UUID\": \"1234\",\n        \"MeasurementVersion\": \"0.1.2\",\n        \"Field1\": 420\n        \"Field2\": 31.41\n      }\n    }\n    ...\n```\n\n* \u003cstrong\u003e\u003ccode\u003edate\u003c/code\u003e\u003c/strong\u003e is the date component of the directory\npathname where new format files were discovered.  For example,\nthe \u003ccode\u003edate\u003c/code\u003e field of the bundle that contains new format\nfiles in \u003ccode\u003e/var/spool/ndt/foo1/2022/09/29\u003c/code\u003e will be\n\u003ccode\u003e2022/09/29\u003c/code\u003e.\n\n* \u003cstrong\u003e\u003ccode\u003earchiver\u003c/code\u003e\u003c/strong\u003e defines the details of the\nrunning instance of \u003ccode\u003ejostler.\u003c/code\u003e\n\n* \u003cstrong\u003e\u003ccode\u003eraw\u003c/code\u003e\u003c/strong\u003e contains individual new format\ncontents in JSON format without any modification.  The fields\n\u003ccode\u003eUUID\u003c/code\u003e, \u003ccode\u003eMeasurementVersion\u003c/code\u003e, \u003ccode\u003eField1\u003c/code\u003e,\nand \u003ccode\u003eField2\u003c/code\u003e are simply examples.  The new measurement provider\nwill decide what fields will be included in their new format.\n\nNotice that not all data fields are necessarily included in each\n\u003ccode\u003eraw\u003c/code\u003e JSON object (new format files).  The above example\nshows that \u003ccode\u003eField2\u003c/code\u003e and \u003ccode\u003eField1\u003c/code\u003e are missing from\nthe first and the second new format files respectively.\n\n### 2.3. Datatype schema\n\nNew measurements should provide the schema of their measurement data as\na file in JSON format.\n\nWhen `jostler` starts, it looks for datatype schema files of each\nspecified datatype, generates the corresponding BigQuery table schema\n(which includes M-Lab's standard columns), and uploads the table schema\nfiles to GCS.  The location of a datatype schema file can be specified via\na command line flag (`-datatype-schema-file`) but its default location is:\n\n```\n    /var/spool/datatypes/\u003cdatatype\u003e.json\n```\n\nIn the interactive mode, the operator can use the `-schema` flag to create\nthe schema and examine it.  For example, below is the command to create\nBigQuery table schemas for tables `foo1` and `bar1`.  In this example,\n`jostler` is told to look for `foo1`'s measurement data schema in the\ndefault location and for `bar1`'s in `/path/to/bar1.json`.\n\n```\n    $ ./jostler -schema -datatype foo1 -datatype bar1 \\\n        -datatype-file bar1:/path/to/bar1.json\n```\n\n`jostler` uploads table schema files to GCS as the following objects:\n\n```\n    autoload/v1/tables/\u003cexperiment\u003e/foo1-table.json\n    autoload/v1/tables/\u003cexperiment\u003e/bar1-table.json\n```\n\nThe purpose of version `v1` is to support breaking changes to autoloading\nimplementation (i.e., conventions agreed on between `jostler` and the\nloader agent in the pipeline).\n\n### 2.4. Index bundles\n\nFor every JSONL bundle that `jostler` uploads to GCS, it will also upload\nan index file also in JSONL format that contains the list of filenames\ncontained in the bundle in the same order that new format data appears\nin the `raw` fields of the bundle.\n\n`jostler` creates index files as a special datatype of `index1` so the\nautoload agent in the pipeline does not have to distinguish between\nmeasurement data files and index files.  In other words, as far as the\npipeline is concerned, `index1` is just another datatype.\n\nIndex bundles will have the same name as the bundle they describe.\n\n### 2.5. Default paths and object names\n\nIn summary, by default:\n\n1. Measurement data files will be read from the local filesystem at:\n    ```\n    /var/spool/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e\n    ```\n2. Datatype schema files will be read from the local filesystem at:\n    ```\n    /var/spool/datatypes/\u003cdatatype\u003e.json\n    ```\n3. Table schema files will be uploaded to GCS as:\n    ```\n    autoload/v1/tables/\u003cexperiment\u003e/\u003cdatatype\u003e.table.json\n    ```\n4. JSONL data bundles will be uploaded to GCS as:\n    ```\n    autoload/v1/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e/\u003ctimestamp\u003e-\u003cdatatype\u003e-\u003cnode\u003e-\u003cexperiment\u003e-data.jsonl.gz\n    ```\n5. JSONL index bundles will be uploaded to GCS as:\n    ```\n    autoload/v1/\u003cexperiment\u003e/\u003cdatatype\u003e/\u003cyyyy\u003e/\u003cmm\u003e/\u003cdd\u003e/\u003ctimestamp\u003e-\u003cdatatype\u003e-\u003cnode\u003e-\u003cexperiment\u003e-index1.jsonl.gz\n    ```\n\n### 2.6. `jostler` configuration\n\n**GCS configuration**\n\n* bucket name: for example` pusher-mlab-{sandbox,staging,oti}`\n* home folder: object name starts with this string (e.g., `autoload/v1)`\n* M-Lab node name:` `parsed and used in object names (examples in\n\n**Bundle configuration**\n* maximum size: maximum size before it is uploaded\n* maximum age: maximum duration since a bundle was created in memory until it is uploaded\n\n**Filesystem configuration**\n* home directory: directory under which measurement data is created (e.g., `/var/spool`)\n* extensions: filename extensions of interest (default `.json`); other files will be ignored\n* experiment: name of the measurement service (e.g., `ndt`)\n* datatypes: name(s) of the datatype(s) the experiment generates (e.g., `scamper1`)\n* minimum file age: minimum duration since a file's last modification time before it is considered a missed data file\n* scan interval: the interval for scanning filesystem for missed files\n\n**Execution**\n* flush timeout: maximum duration for flushing active bundles to GCS before exiting\n* schema: run in the interactive mode and create schema files\n* verbose: enable verbose mode for more logging\n\n### 2.7. `jostler` architecture\n\n`jostler` architecture consists of a public `api` package that defines\nstandard columns and `index1` datatype, and the following internal packages:\n\n* `internal/gcs`: handles downloading and uploading files to Google Cloud Storage (GCS).\n* `internal/jsonlbundle`:  implements logic to process a single JSONL bundle.\n* `internal/schema implements logic to handle datatype and table schemas.\n* `internal/testhelper`: implements logic to help in unit and integration (e2e) testing.\n* `internal/uploadbundle`: implements logic to bundle multiple local JSON files into JSONL bundles and upload to Google Cloud Storage (GCS)\n* `internal/watchdir`: watches a directory and sends notifications to its client when it notices a new file.\n\nFiles that do not have a .json suffix or are not in proper JSON format\nwill be ignored.  As mentioned earlier, jostler is different from pusher\nby not indiscriminately including all files in the bundle regardless\nof their content.  This behavior of `jostler` will provide [better\nsecurity](https://github.com/m-lab/pusher/blob/main/DESIGN.md#7-security-considerations).\n\nIt is highly desirable that `jostler` guarantees it will not upload the\nsame new format file more than once.  With this guarantee there will be\nno need to deduplicate data.  Due to asynchronous pod reboots and GCS\nfailures, the feasibility of this guarantee is currently unclear but\nevery effort will be made to obviate the need for data deduplication.\n\n### 2.8. Concurrency and shared data\n\nTo be written.\n\n### 2.9. Missed data files\n\nFor all _planned_ reboots, upload agents on M-Lab nodes will have a\nduration to flush out their active data and wrap up gracefully so that\nno files are missed.  For `pusher`, the duration is specified with the\n`-sigtermWait` flag and for `jostler` it will be specified with the\n`-flushTimeout` flag.\n\nHowever, because pods can have _unplanned_ restarts at any\ntime, it is possible for `jostler` (or any other agent) to\nmiss the Writable file was closed (`IN_CLOSE_WRITE`) or\nFile was moved to (`IN_MOVED_FROM`) `inotify` events.\nAlso if too many events occur at once, the `inotify` event\nqueue can overflow and lose some events (see [Limitations and\ncaveats](https://man7.org/linux/man-pages/man7/inotify.7.html)).\nAdditionally, if upload to GCS fails, the individual new format files\nthat were in the bundle will not be deleted.\n\nWhen a file's last modification time is more than a configurable\nduration (e.g., 2 hours), `jostler` assumes it either missed the file's\n`IN_CLOSE_WRITE` or `IN_MOVED_FROM` event or uploading to GCS wasn't\nsuccessful.  In cases like this, `jostler` considers the file eligible\nfor upload.  This also means that files that are open but are not modified\nfor more than the configurable duration will be uploaded _prematurely_.\nThis is why it is required that new measurements should not keep a file\nopen without writing to it for more than a few minutes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fjostler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fjostler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fjostler/lists"}