{"id":18248824,"url":"https://github.com/igorski/binarcular","last_synced_at":"2025-04-08T19:55:51.147Z","repository":{"id":57203591,"uuid":"300007138","full_name":"igorski/binarcular","owner":"igorski","description":"Read/write binary data into/from JSON structures, with all data types converted to JS friendly values. Data can be searched, sliced into separate meaningful structures and files can be generated and downloaded, all in the web browser.","archived":false,"fork":false,"pushed_at":"2020-10-06T17:06:56.000Z","size":472,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-14T16:50:06.245Z","etag":null,"topics":["binary-file-search","client-side","es6-module","file","fileparser","filereader","filesearch","filestream","javascript","struct","typed-struct"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/igorski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-30T17:57:46.000Z","updated_at":"2024-01-30T22:45:24.000Z","dependencies_parsed_at":"2022-09-17T15:00:37.022Z","dependency_job_id":null,"html_url":"https://github.com/igorski/binarcular","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igorski%2Fbinarcular","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igorski%2Fbinarcular/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igorski%2Fbinarcular/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igorski%2Fbinarcular/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/igorski","download_url":"https://codeload.github.com/igorski/binarcular/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247918927,"owners_count":21018044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-file-search","client-side","es6-module","file","fileparser","filereader","filesearch","filestream","javascript","struct","typed-struct"],"created_at":"2024-11-05T09:38:25.277Z","updated_at":"2025-04-08T19:55:51.128Z","avatar_url":"https://github.com/igorski.png","language":"JavaScript","readme":"# Binarcular\n\nA library that allows you to read/write the contents of a binary file into/from a JSON Object,\ntaking care of all data type conversion using JavaScript-friendly values. You can search for data\nby value, slice blocks into separate, meaningful structures or generate a binary downloadable\nfile, all inside your browser.\n\nPractical use cases are:\n\n* Validating whether a file header contains the appropriate description, by comparing\n its parsed data to the respective data types\n* Scanning a file for specific metadata to determine the location of other meaningful data\n* Creating a binary file in your browser, without having to repeatedly write meaningless byte\nvalues in sequence\n\nSee API and Example below.\n\n## Compatibility\n\n_binarcular_ should work fine on Internet Explorer 10 and up. You can verify\nsupport at runtime by querying the result of the _isSupported()_-method:\n\n```\nimport { isSupported } from 'binarcular';\nif ( isSupported( optRequire64bitConversion = false ) ) {\n    ...do stuff!\n} else {\n    ...do other, less cool stuff! Actually, not cool at all!! \u003e=/\n}\n```\n\nNOTE: if you require support for 64-bit types there are [additional requirements](https://caniuse.com/?search=bigint). Pass boolean _true_ for optional argument _optRequire64bitConversion_ to determine whether the environment supports 64-bit conversion.\n\n## Installation\n\nYou can get it via NPM:\n\n```\nnpm install binarcular\n```\n\n### Project integration\n\nThe library is compatible with CommonJS / ES6 modules or can be included in a document\nusing AMD/RequireJS. See the contents of the _/dist/_ folder and include as your project sees fit.\n\n## API\n\nThe module exports the following:\n\n```\nimport {\n\n    isSupported:     fn( optRequire64bitConversion = false ),\n    types:           Object\u003cString\u003e,\n    parse:           async fn( dataSource, structureDefinition, optReadOffset = 0 ),\n    seek:            async fn( uint8Array, searchStringOrByteArray, optReadOffset = 0 ),\n    write:           async fn( uint8Array, structureDefinition, dataToWrite, optWriteOffset = 0 )\n    fileToByteArray: async fn( file, optSliceOffset = 0, optSliceSize = file.size )\n    byteArrayToFile: fn( uint8Array, filename, optMimeType = 'application/octet-stream' )\n\n} from 'binarcular';\n```\n\nWe'll look into each of these below:\n\n### Reading a chunk of data into an Object of a specific structure type\n\nHandled by the _parse_ method:\n\n```\nasync parse( dataSource, structureDefinition, optReadOffset = 0 )\n```\n\nWhere:\n\n* _dataSource_ is the file to parse (can be either _File_, _Blob_, _Uint8Array_ or (base64 encoded) _String_)\n* _structureDefinition_ is an Object defining a data structure ([as described here](#define-a-structure))\n* _optReadOffset_ is a numerical index describing where in the file's ByteArray reading should start\n  this defaults to 0 to start at the beginning of the file.\n\nWhen the Promise resolves, the result is the following structure:\n\n```\n{\n    data: Object,\n    end: Number,\n    error: Boolean,\n    byteArray: Uint8Array\n}\n```\n\nIf all has been read successfully, _data_ is an Object that follows\nthe structure of _wavHeader_ and is populated with the actual file data.\n\n_end_ describes at what offset in given file the structure's definition has ended.\nThis can be used for subsequent read operations where different data types are\nextracted from the binary data.\n\nIf _error_ is true, this indicates that something went wrong during parsing. The _data_\nObject will be populated with all data that could've been harvested up until the error occurred.\nThis allow you to harvest what you can from corrupted files.\n\n#### A note on using Uint8Array as dataSource\n\nYou can see that another property is defined in the result, namely _byteArray_.\nIf the _dataSource_ provided to the parse method was a _Uint8Array_ which you intend to\nreuse inside your project, be sure to reassign your byteArray reference to the\nreturned instance.\n\nThe rationale here is that for minimal overhead, the ownership of the ByteArray's binary\ncontent is transferred during the read operations. _You will not be able to perform\nany actions on your ByteArray without updating the reference_.\n\n### Looking for a specific entry in a file\n\nIf you are working with a file where the content of interest is preceded by some\nmetadata at an arbitrary point, it makes sense to first look for this metadata\ndeclaration so you know from where you can retrieve the actual data of interest.\n\nFor this purpose you can use _seek_:\n\n```\nasync seek( uint8Array, searchStringOrByteArray, optReadOffset = 0 )\n```\n\nwhere:\n\n* _uint8Array_ is the ByteArray containing the binary data.\n* _searchStringOrByteArray_ can be either a String (in case the meta data is a\ncharacter sequence) or a Uint8Array holding a byte sequence that functions as the \"search query\".\n* _optReadOffset_ determines the offset within the data from where to start searching, this\ndefaults to 0 to read from the start.\n\nThe method returns a numerical index at which the data was found or _Infinity_\nif no match were found.\n\n### Writing JSON as binary content\n\nIf you have a JSON structure that you wish to write into a binary file, you can do\nso using _write_:\n\n```\nasync write( uint8Array, structureDefinition, dataToWrite, optWriteOffset = 0 )\n```\n\nwhere:\n\n* _uint8Array_ is the ByteArray containing the binary data.\n* _structureDefinition_ is an Object defining a data structure ([as described here](#define-a-structure))\n* _dataToWrite_ is an Object following the data structure, except the value here is the\n  data you wish to write in the binary file.\n* _optWriteOffset_ is the index at which data will be written. This defaults to _0_ to\n start writing at the beginning of the file. Data will be written for the length of\n given _structureDefinition_, all existing data beyond this point will remain unchanged.\n\nThe result of this operation is the following:\n\n```\n{\n    data: Object,\n    end: Number,\n    error: Boolean,\n    byteArray: Uint8Array\n}\n```\n\nWhere _byteArray_ should replace the reference of the _byteArray_ you passed into\nthe method. This ByteArray contains the original data except that the data block starting\nat requested _optWriteOffset_ has been replaced with the binary equivalent of the _dataToWrite_-Object.\nAll data beyond the size of the written data block remains unchanged.\n\n### Converting a File reference to a ByteArray\n\n```\nasync fileToByteArray( fileReference, optSliceOffset = 0, optSliceSize = fileReference.size )\n```\n\nwhere:\n\n* _fileReference_ is the File (or Blob) of which the contents should be read into a _Uint8Array_.\n* _optSliceOffset_ is the optional offset from where to read the data, defaults to 0 to\nstart from the beginning.\n* _optSliceSize_ is the optional size of the resulting ByteArray. This defaults to the\nsize of the file to read the file in its entirety. When using a custom _optSliceOffset_\noverflow checking is performed to prevent reading out of the file boundaries.\n\n### Converting a ByteArray to a File\n\n```\nbyteArrayToFile( byteArray, filename, optMimeType = 'application/octet-stream' )\n```\n\nThis will generate a download of given _filename_, containing the data of\n_byteArray_ as its content using given _optMimeType_.\n\nTo prevent blocking the download, this should be called directly from a click handler.\n\n## Example\n\nLet's say we want to read the binary data of a well known proprietary format.\nFirst up we will get to...\n\n### Define a structure\n\nDefining a structure is nothing more than declaring an Object where the keys\ndefine names meaningful to your purpose and the values consist of Strings describing:\n\n* one of the available type enumerations (the names of the imported types are equal to their value).\n* optional Array declaration where by adding a numerical value between brackets _[n]_, will\n  make the value an Array of given length _n_.\n* optional modifier defining the endianness of the file's byte order, separated by a pipeline\n  (either _|BE_ for Big Endian or _|LE_ for Little Endian). When unspecified, the\n  endianness of the clients system is used (assuming the file has been encoded on/by a similar\n  system, which usually means Little Endian these days).\n\nAn example structure that defines the [header of a .WAV file](http://soundfile.sapp.org/doc/WaveFormat)\nwould look like:\n\n```\nconst wavHeader = {\n    type:           'CHAR[4]',\n    size:           'INT32|LE',\n    format:         'CHAR[4]',\n    formatName:     'CHAR[4]',\n    formatLength:   'INT32|LE',\n    audioFormat:    'INT16|LE',\n    channelAmount:  'INT16|LE',\n    sampleRate:     'INT32|LE',\n    bytesPerSecond: 'INT32|LE',\n    blockAlign:     'INT16|LE',\n    bitsPerSample:  'INT16|LE',\n    dataChunkId:    'CHAR[4]',\n    dataChunkSize:  'INT32|LE'\n};\n```\n\nNote that the order of the keys (and more importantly: their type definition) should match\nthe order of the values as described the particular file's type!\n\n#### A teeny tiny note on Endianness\n\nNote that specifying endianness can be omitted if you're certain that the file's\nencoding is equal to that of the platform you will be parsing the file on (most likely\nonly Big Endianness will require an explicit definition). _And I hope you will never\nbe in the unfortunate situation where you work with a file that uses different\nendianness for different blocks!_\n\n#### Back to talking types\n\nAll available data types are listed in the _{ types }_ export. Note that definitions\nfor _CHAR_ will return as a String. If you want an 8-bit integer/byte value, use\n_BYTE_ or _INT8_ instead.\n\nWe can now proceed to read the file:\n\n```\nimport { parse } from 'binarcular';\n\nasync function readWaveHeader( fileReference ) {\n    const { data, end, error, byteArray } = await parse( fileReference, wavHeader, 0 );\n\n    console.log( data );  // will contain the properties of a WAV file header\n    console.log( end );   // will describe the end offset of the header\n    console.log( error ); // when true, a file reading error occurred\n}\n```\n\nYou can also view the [demo](https://htmlpreview.github.io/?https://github.com/igorski/binarcular/blob/master/dist/index.html) provided in this repository's _example.html_ file, which\nparses .WAV files and provides advanced examples using seeking, slicing and error\ncorrection before finally providing you with the instruction on how to extract the\nmeaningful data from the file.\n\n## Performance\n\nDepending on the size of the files you're working with, memory allocation can become a problem.\n\nThe parser will only read the block that is requested (e.g. starting from the\nrequested offset and only for the size of the requested _structureDefinition_) and\nshould thus be light on resources. Additionally, all read operations happen in a\ndedicated Web Worker which keeps your main application responsive (you can safely\nparse several hundred megabytes of data without blocking your UI).\n\nDepending on your use case, it helps to take the following guidelines into consideration:\n\n* Use base64 _only when you have no choice_ as a base64 String describes the\n  file _in its entirety_. Also, the way JavaScript handles Strings is by\n  allocating the entire value (and not by reference!) whenever you assign\n  it to a new variable.\n* If you intend to do multiple reads on the same file (for instance: first reading\n  its header to determine where in the file the meaningful content begins) it\n  is recommended to use the _fileToByteArray()_-method to create a single\n  reusable _Uint8Array_. This also makes sense if you need to read the file in its entirety.\n\n## Build instructions\n\nIn case you want to aid in development of the library:\n\nThe project dependencies are maintained by NPM, you can resolve them using:\n\n```\nnpm install\n```\n\nYou can develop (and test against the example app by navigating to _http://localhost:8080_) by running:\n\n```\nnpm run dev\n```\n\nTo create a production build:\n\n```\nnpm run build\n```\n\nAfter which a folder _dist/_ is created which contains the prebuilt AMD/RequireJS\nand CommonJS/ES module libraries (as well as the example application).\n\nThe source code is transpiled to ES5 for maximum browser compatibility.\n\n## Unit testing\n\nUnit tests are run via jest, you can run the tests by running:\n\n```\nnpm run test\n```\n\nUnit tests go in the _./test_-folder. The file name for a unit test must be equal to the file it is testing, but contain the suffix \".spec\", e.g. _functions.js_ should have a test file _functions.spec.js_.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figorski%2Fbinarcular","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Figorski%2Fbinarcular","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figorski%2Fbinarcular/lists"}