{"id":18414914,"url":"https://github.com/xerial/silk-format","last_synced_at":"2025-04-13T00:58:54.353Z","repository":{"id":141493998,"uuid":"14546869","full_name":"xerial/silk-format","owner":"xerial","description":"A compact text format for structured data","archived":false,"fork":false,"pushed_at":"2013-12-16T14:53:42.000Z","size":280,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-04-13T00:58:49.575Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xerial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-20T04:38:40.000Z","updated_at":"2013-12-16T14:53:43.000Z","dependencies_parsed_at":"2023-03-12T05:30:18.436Z","dependency_job_id":null,"html_url":"https://github.com/xerial/silk-format","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xerial%2Fsilk-format","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xerial%2Fsilk-format/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xerial%2Fsilk-format/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xerial%2Fsilk-format/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xerial","download_url":"https://codeload.github.com/xerial/silk-format/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650435,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T03:52:40.629Z","updated_at":"2025-04-13T00:58:54.323Z","avatar_url":"https://github.com/xerial.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Silk Text Format\n====\n\nSilk text format is a compact data format for describing structured data.\n\n## Specification  (draft)\n\n### Comment-line\n\nStarts with `#`\n\n```\n# This is a comment line\n```\n\n### Preambles\nPreamble line starts with `%`:\n\n#### Header \nSpecifies a silk format version.\n```\n%silk - version:\"1.0\"\n```\n#### Record schema definition\n\n```\n# Detailed definition\n%record person(id:int, name:string)\n\n# Simplified record syntax. You can use `-` instead of wrapping parameters with parentheses. \n# If the type name is ommitted, the default is string type\n%record person - id:int, name\n```\n\n#### Primitive data types\n\nIf no type is specified in a record definition, the default data type becomes `string`.\n\n* `string` (UTF8 encoding)\n* `int`\n* `float`\n* `double`\n* `boolean`\n* Array type `(type name)*`\n\n```\n%record person - id:int, name, phone:string*\n# string type can be omitted\n%record person - id:int, name, phone*\n```\n\n* Map type `map[K,V]`\n\n```\n%record symbolTable:map[string, int]\n-symbolTable\nA 0\nB 1\nC 2\n```\n\n \n* `json` \n  * Silk uses an extended JSON format that can use QName (alphabet and number characters with spaces) token instead of double-quoted `\"String\"` \n\n* `option[A]` or `A?`\n   * Optional appearances of a parameter value, allowing a column to be empty.\n    \n### Line format\n\nSilk represents a list of records using tab-separated format. The line starting with `-` represents a data type name to be described. The following lines\ndescribe records line by line in a tab-separated format.\n\n```\n%record person(id:int, name) \n\n-person\n1\tleo\n2\tyui\n```\n\n```\n%record fruit_table:map[string, string]\n\n-fruit_table\nA\tapple\nB\tbanana\nC\tcoconut\n```\n\n\n#### Embedding (extended) JSON in a column\n\n```\n%record log - date, level, param:json\n\n-log\n2013-11-20\tinfo\t{message:\"hello silk\"}\n2013-11-20\tdebug\t{result:\"success\", elapsed time:12.3}\n```\n\nThis is equivalent to write as follows:\n```\n-log\n -date:2013-11-20\n -level:info\n -param\n  -message:hello silk\n-log\n -date:2013-11-20\n -level:debug\n -param\n  -result:success\n  -elapsed time:12.3\n```\n\n#### Embedding an array into a column\n```\n%record check_sheet - pid:int, answer:boolean*\n-check_sheet\n1\t[true, true, false, false, true]\n2\t[true, false, false, true, true]\n```\n\n#### In-line representation of nested records\n\n```\n%record person - id:int, name, address:address\n%record address - address, phone, country\n\n-person\n1\tleo\t[\"ABC Street\", \"XXX-XXXX\", \"Japan\"]\n2\tyui\t[\"YYY Town\", \"ZZZ-ZZZZ\", \"US\"]\n```\n\n#### Embedding json data in a column\n\n```\n%record read - qname, flag:int, chr:alnum, start:int, score:int, cigar, mname, mstart:int, isize:int, qseq, qv, tag:json\n-read\nread_28833_29006_6945        99        chr20        28833        20\t10M1D25M        =        28993        195\tAGCTTAGCTAGCTACCTATATCTTGGTCTTGGCCG        \u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c:\u003c9/,\u0026,22;;\u003c\u003c\u003c {MF:130, Nm:1, H0:0, H1:0, RG:L1}\nread_28701_28881_323b        147        chr20        28834        30\t35M        =        28701        -168\tACCTATATCTTGGCCTTGGCCGATGCGGCCTTGCA        \u003c\u003c\u003c\u003c\u003c;\u003c\u003c\u003c\u003c7;:\u003c\u003c\u003c6;\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c7\u003c\u003c\u003c\u003c\t{MF:18, Nm:0, H0:1, H1:0, RG:L2}\n```\n\n#### Polymorphic types\n\n```\n%record log - date, message\n# Embedding log record parameters into info/error records\n%record info: _:log\n%record error - _:log, cause\n\n-log\n@info  2013-11-20\tsystem started\n@error 2013-11-20\terror occurred\tNullPointerException\n```\n\nWhen the line starts with `@`, the first column is a type description. The remaining part is tab-separated values of the corresponding type.\n\n### Indentation format\n\nA more human readable data description, suited to configuration files, data\ndescriptions, etc. that need to be edited by hand.\n\n```\n-person\n -id: leo\n -name: yui\n```\n\n\n### Import statment\n\nImporting another silk data via `import` statement.\n\n`record.silk`\n```\n%silk version:\"1.0\"\n%record person - id:int, name\n```\n\n```\n%import \"record.silk\"\n-person\n1\tleo\n2\tyui\n```\n\n\n### Context line\n\nContext line starting with `\u003e` is a meta data for annotating or grouping records.\n\n```\n-log\n\u003eserver - name:sv01, type:server log\n@info 2013-11-20\tsystem started\n@info 2013-11-20\treceived a task request\n@error 2013-11-20\terror occurred\tNullPointerException\n\u003eclient - name:sv100, type:client log\n@info 2013-11-20\tclient has started\n@info 2013-11-20\tclient sending a task to server\n```\n\n### Schema-less data\n\nIf no record name or schema is specified, silk parses each data line as a tab-separated record of json. \n```\nhost:127.0.0.1  ident:- user:frank  time:\"[10/Oct/2000:13:55:36 -0700]\" req:\"GET /apache_pb.gif HTTP/1.0\" status:\"200\"  size:2326 referer:\"http://www.example.com/start.html\" ua:\"Mozilla/4.08 [en] (Win98; I ;Nav)\"\n```\n\nIf your schema of the data becomes stable, you should define a record schema to create compact records:\n\n```\n%record weblog - host, ident, user, time, req, status, size:int, referer, ua\n-weblog\n127.0.0.1  -  frank  [10/Oct/2000:13:55:36 -0700] GET /apache_pb.gif HTTP/1.0 200 2326  http://www.example.com/start.html Mozilla/4.08 [en] (Win98; I ;Nav)\n```\n\nYou can split the above weblog into schema and log record files:\n\nWith context information, you can enhance the data description. \n\n`weblog.silk`\n```\n%record weblog - host, ident, user, time, req, status, size:int, referer, ua\n\u003enote - description:\"Imported from Apache Logs\", server:sx03\n-weblog\n%import \"weblog.tsv\"\n```\n\n`weblog.tsv`\n```\n127.0.0.1  -  frank  [10/Oct/2000:13:55:36 -0700] GET /apache_pb.gif HTTP/1.0 200 2326  http://www.example.com/start.html Mozilla/4.08 [en] (Win98; I ;Nav)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxerial%2Fsilk-format","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxerial%2Fsilk-format","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxerial%2Fsilk-format/lists"}