https://github.com/klirichek/sphtool

Tool for converting index headers used by manticore search
https://github.com/klirichek/sphtool

json manticore sph

Last synced: 24 days ago
JSON representation

Tool for converting index headers used by manticore search

• Host: GitHub
• URL: https://github.com/klirichek/sphtool
• Owner: klirichek
• License: mit
• Created: 2020-10-28T03:41:41.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2021-06-29T06:58:00.000Z (almost 5 years ago)
• Last Synced: 2025-09-08T14:47:14.720Z (9 months ago)
• Topics: json, manticore, sph
• Language: Go
• Homepage:
• Size: 12.7 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# SPHtool - render/parse sph/meta files

Manticore search indexes consists from huge datafiles (as dictionaries, attributes, etc) and small headers containing

basic meta info as index settings, paths, etc. That is binary internal format and usually not capable to be edited from

outside (so consider it as 'black box').

This tool renders whole header to json (which is obviously much more human-friendly).

Also it performs opposite operation - parses json back to header. 

 

    ./sphtool -i index.sph

    ./sphtool -i index.meta

    

parses `index.sph` and produce `index.sph.json` as output. The same way works with 'meta' headers (both from realtime

and percolate indexes). File extension (`.sph` or `.meta`) right now used to determine file format and is mandatory

(I know, file fingerprint is enouth go determine format, however right now extension is used). By default target file is

original one plus `.json` extension.

    ./sphtool -i index.sph.json

    

renders `index.sph.json` and produce `index.sph` as output. File extension `.json` is mandatory. Internal flavour (sph,

meta from rt or saved percolate queries) is determined from magic field in json. By default target file is original one

without `.json` extension.

Optionally target file may be provided with `-o` option, as:

    ./sphtool -i pqindex.json -o pqindex.meta

    ./sphtool -i plainindex.json -o plainindex.sph

Supported versions of headers are visible in banner:

    SPHtool v $Format:%h$ - sph and meta files converter

    Copyright (c) 2020 Alexey N. Vinogradov (a.n.vinogradov@gmail.com)

    Supports sph v. 60, meta v. 17, pq meta v. 8

    Usage of ./sphtool:

      -i string

        	path to sph, meta, sph.json or meta.json file

      -o string

        	path to target file (optional)

        	

Tool may convert to json and back following headers:

 * `.sph` - which is 'sphinx header' - main file containing plain index params as schema, tokenenizer settings, etc.

 That is both part of standalone plain  indexes and rt disk chunks. In json you may change, say, path to files 

 (wordforms, stopwords, whatever) if they changed.  Or fixup timestamps of the files, as they used to check files 

 consistency.

 * `.meta` of rt - is header of realtime index. It has similar sense as sph file of plain index, but in aggregate

 meaning over all index chunks.

 * `.meta` of pq - is actually whole pq index (it has nothing else apart this meta). Apart general settings (like schema)

 and embedded files/hashes it includes stored queries.

 

 ## Building

 

 Use

 

    go get github.com/klirichek/sphtool

    

or (if you have clone sources):

    cd /src/of/sphtool

    go build -o sphtool . 

## Caveats

It might look useful to format json if you want to edit it.

Many sources advice to use 'jq' tool for it, like:

    cat index.meta.json | jq . > index_formatted.meta.json

    

However that `jq` corrupts some numbers when reformat, so that final file became unusable. Look at this live

example (one and same file first printed, then passed to jq). Note the difference! BEWARE! 

    $ cat ex.json

    {

      "numbers": [

        623237308590442816,

        2696721519739033356,

        15902905282948881040

      ]

    }

    

    $ cat ex.json | jq .

    {

      "numbers": [

        623237308590442800,

        2696721519739033600,

        15902905282948880000

      ]

    }