{"id":13409291,"url":"https://github.com/AydinSakar/lucene-layer","last_synced_at":"2025-03-14T14:31:09.023Z","repository":{"id":14557698,"uuid":"17273356","full_name":"AydinSakar/lucene-layer","owner":"AydinSakar","description":"Lucene on FoundationDB","archived":false,"fork":false,"pushed_at":"2013-09-04T22:53:37.000Z","size":484,"stargazers_count":3,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-07-31T20:35:57.329Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AydinSakar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-02-28T02:57:29.000Z","updated_at":"2022-03-12T20:45:38.000Z","dependencies_parsed_at":"2022-08-26T21:52:46.723Z","dependency_job_id":null,"html_url":"https://github.com/AydinSakar/lucene-layer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AydinSakar%2Flucene-layer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AydinSakar%2Flucene-layer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AydinSakar%2Flucene-layer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AydinSakar%2Flucene-layer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AydinSakar","download_url":"https://codeload.github.com/AydinSakar/lucene-layer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243593336,"owners_count":20316170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:00:59.637Z","updated_at":"2025-03-14T14:31:08.393Z","avatar_url":"https://github.com/AydinSakar.png","language":"Java","funding_links":[],"categories":["Layers"],"sub_categories":[],"readme":"# FoundationDB Lucene Layer\n\nThis layer provides two integration points with Lucene, `FDBDirectory` and \n`FDBCodec`. These are full implementations of the [Directory](https://lucene.apache.org/core/4_4_0/core/org/apache/lucene/store/Directory.html)\nand [Codec](https://lucene.apache.org/core/4_4_0/core/org/apache/lucene/codecs/Codec.html)\ninterfaces which are backed entirely by [FoundationDB](https://foundationdb.com/).\n\n`FDBDirectory` can be used on its own with the default `Codec` doing the\ninteresting work. Files generated by Lucene are stored as blobs in the\ndatabase instead of the file system.\n\n`FDBCodec`, which must be used in conjunction with `FDBDirectory`, implements \nnew serialization and data models for Lucene. This results in explicit keys and\nvalues in the database instead of file-like blobs.\n\n\n## _Warning: Alpha Stage_\n\nThis layer is at an early alpha stage (note the 0.0.1 version number). While\nmost of the stock Lucene tests pass when using `FDBDirectory`, many currently\nfail when running with `FDBCodec`. There are no known _correctness_ issues at\nthis time but slowness and timeout issues could easily be hiding such problems.\n\nPlease try it out and let us know how it works (e.g. on our\n[community site](http://community.foundationdb.com/)), but production usage\nis *not* recommended.\n\n\n## FDBCodec Data Model\n\nThe [Subspace](https://foundationdb.com/documentation/data-modeling.html#subspaces-of-keys)\nconcept is used extensively to provide a simple, logical mapping and easy\nstorage and retrieval. Each directory, segment and format are identified by a\nunique string. These identifier strings are then concatenated together to yield\nkey ranges associated with each logical format being stored.\n\nFor example, assume we have a `FDBDirectory` created with the path\n`(\"lucene\")` and a segment named `\"_0\"`. That would result in the following\n[Tuples](https://foundationdb.com/documentation/data-modeling.html#tuples):\n\n- `(\"lucene\", \"_0\", \"dat\")` for DocValues\n- `(\"lucene\", \"_0\", \"inf\")` for FieldInfos\n- `(\"lucene\", \"_0\", \"liv\")` for LiveDocs\n- _etc_\n\nAdditional keys and values exist under each of those subspaces for storing the \ninformation associated with each format. In the documentation below, the full\nsubspace is the concatenation of the directory, segment and format subspaces.\n\n\n### DocValuesFormat\n\nEncodes/decodes strongly typed, per document values. See\n[DocValuesFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/DocValuesFormat.html)\nand\n[FieldInfo.DocValuesType](https://lucene.apache.org/core/4_4_0/core/org/apache/lucene/index/FieldInfo.DocValuesType.html).\n\nThe `long_BINARY`, `long_NUMERIC`, `long_SORTED` and `long_SORTED_SET` key parts\nbelow refer to the `DocValuesType` enum `ordinal()` values.\n\nSubspace: `(\"dat\")`\n\n    (str_fieldName, long_BINARY, long_doc0) =\u003e (bytes_value)\n    (str_fieldName, long_BINARY, long_doc1) =\u003e (bytes_value)\n    ...\n    (str_fieldName, long_NUMERIC, long_doc0) =\u003e (long_value)\n    (str_fieldName, long_NUMERIC, long_doc1) =\u003e (long_value)\n    ...\n    (str_fieldName, long_SORTED, \"bytes\", long_ordinal0) =\u003e (bytes_value)\n    (str_fieldName, long_SORTED, \"bytes\", long_ordinal1) =\u003e (bytes_value)\n    ...\n    (str_fieldName, long_SORTED_SET, \"ord\", long_doc0) =\u003e (long_ordinal)\n    (str_fieldName, long_SORTED_SET, \"ord\", long_doc1) =\u003e (long_ordinal)\n    ...\n    (str_fieldName, long_SORTED_SET, \"bytes\", long_ordinal0) =\u003e (bytes_value)\n    (str_fieldName, long_SORTED_SET, \"bytes\", long_ordinal1) =\u003e (bytes_value)\n    ...\n    (str_fieldName, long_SORTED_SET, \"doc_ord\", long_doc0, long_ordinal0) =\u003e ()\n    (str_fieldName, long_SORTED_SET, \"doc_ord\", long_doc0, long_ordinal1) =\u003e ()\n    (str_fieldName, long_SORTED_SET, \"doc_ord\", long_doc1, long_ordinal0) =\u003e ()\n    ...\n\n\n### FieldInfosFormat\n\nEncodes/decodes filed metadata. See\n[FieldInfosFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/FieldInfosFormat.html)\nand\n[FieldInfos](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/FieldInfos.html):\n\nSubspace: `(\"inf\")`\n\n    (long_field0, \"name\") =\u003e (string_fieldName)\n    (long_field0, \"has_index\") =\u003e (boolean_value)\n    (long_field0, \"has_payloads\") =\u003e (boolean_value)\n    (long_field0, \"has_norms\") =\u003e (boolean_value)\n    (long_field0, \"has_vectors\") =\u003e (boolean_value)\n    (long_field0, \"doc_values_type\") =\u003e (string_docValuesType)\n    (long_field0, \"norms_type\") =\u003e (string_normsType)\n    (long_field0, \"index_options\") =\u003e (string_indexOptions)\n    (long_field0, \"attr\", string_attr0) =\u003e (string_value)\n    (long_field0, \"attr\", string_attr1) =\u003e (string_value)\n    ...\n    (long_field1, \"name\") =\u003e (string_fieldName)\n    ...\n\n\n### LiveDocsFormat\n\nEncodes/decodes live-ness of documents. See\n[LiveDocsFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/LiveDocsFormat.html).\n\nSubspace: `(\"liv\")`\n\n    (long_liveGen0) =\u003e (long_totalSize)\n    (long_liveGen0, long_setBitIndex0) =\u003e ()\n    (long_liveGen0, long_setBitIndex1) =\u003e ()\n    (long_liveGen1) =\u003e (long_totalSize)\n    ...\n\n\n### NormsFormat\n\nEncodes/decodes per-document score normalization values. See\n[NormsFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/NormsFormat.html).\n\nSubspace: `(\"len\")`\n\n_Uses `DocValuesFormat` with a different subspace extension._\n\n\n### PostingsFormat\n\nEncodes/decodes terms, postings, and proximity data. See\n[PostingsFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/PostingsFormat.html).\n\nSubspace: `(\"pst\")`\n\n    (long_field0, bytes_term0, \"numDocs\") =\u003e (littleEndianLong_value)\n    (long_field0, bytes_term0, long_doc0) =\u003e (long_termDocFreq)\n    (long_field0, bytes_term0, long_doc0, long_pos0) =\u003e (long_startOffset, long_endOffset, bytes_payload)\n    ...\n    (long_field1, bytes_term1, \"numDocs\") =\u003e (littleEndianLong_value)\n    ...\n\n\n### SegmentInfoFormat\n\nEncodes/decodes segment metadata. See\n[SegmentInfoFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/SegmentInfoFormat.html).\n\nSubspace: `(\"si\")`\n\n    (\"doc_count\")=\u003e (long_docCount)\n    (\"is_compound_file\") =\u003e (boolean_value)\n    (\"version\") =\u003e (long_version)\n    (\"attr\", string_attr0) =\u003e (string_value)\n    (\"attr\", string_attr1) =\u003e (string_value)\n    ...\n    (\"diag\", string_diag0) =\u003e (string_value)\n    (\"diag\", string_diag1) =\u003e (string_value)\n    ...\n    (\"file\", string_file0) =\u003e ()\n    (\"file\", string_file1) =\u003e ()\n    ...\n\n\n### StoredFieldsFormat\n\nEncodes/decodes per-document fields. See\n[StoredFieldsFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/SegmentInfoFormat.html).\n\nThe key parts `long_TYPE` and `long_DATA` below refer to constants values,\ncurrently `0` and `1`.\n\nSubspace: `(\"fld\")`\n\n    (long_doc0, long_TYPE, long_field0) =\u003e (string_typeName, long_dataIndex)\n    (long_doc0, long_TYPE, long_field1) =\u003e (string_typeName, long_dataIndex)\n    ...\n    (long_doc0, long_DATA, long_field0, long_dataIndex, long_offset0) =\u003e (bytes_value)\n    (long_doc0, long_DATA, long_field0, long_dataIndex, long_offset1) =\u003e (bytes_value)\n    (long_doc0, long_DATA, long_field1, long_dataIndex, long_offset0) =\u003e (bytes_value)\n    ...\n    (long_doc1, long_TYPE, long_field0) =\u003e (string_typeName, long_dataIndex)\n    ...\n\n\n### TermVectorsFormat\n\nEncodes/decodes per-document term vectors. See\n[TermVectorsFormat](https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/codecs/TermVectorsFormat.html).\n\nSubspace: `(\"vec\")`\n\n    (long_doc0, \"field\", string_field0) =\u003e (long_fieldNum, long_numTerms, boolean_hasPositions, boolean_hasOffsets, boolean_hasPayloads)\n    (long_doc0, \"field\", string_field1) =\u003e (long_fieldNum, long_numTerms, boolean_hasPositions, boolean_hasOffsets, boolean_hasPayloads)\n    ...\n    (long_doc0, \"term\", string_field0, bytes_term0) =\u003e (long_freq)\n    (long_doc0, \"term\", string_field0, bytes_term0, long_pos0) =\u003e (long_startOffset, long_endOffset, bytes_payload)\n    (long_doc0, \"term\", string_field0, bytes_term0, long_pos1) =\u003e (long_startOffset, long_endOffset, bytes_payload)\n    (long_doc0, \"term\", string_field0, bytes_term1) =\u003e (long_freq)\n    ...\n    (long_doc1, \"field\", string_fieldName0) =\u003e (long_fieldNum, long_numTerms, boolean_hasPositions, boolean_hasOffsets, boolean_hasPayloads)\n    ...\n\n\n## Running Built-In Tests\n\n[Maven](http://maven.apache.org/) is used for building, packaging and running\ntests.\n\n    $ mvn test\n\n\n## Running Lucene and Solr Tests\n\n1. Package fdb-lucene-layer\n\n        $ mvn package\n\n2. Download the Solr source\n\n        $ curl -O http://mirror.nexcess.net/apache/lucene/solr/4.4.0/solr-4.4.0-src.tgz\n        $ tar xzf solr-4.4.0-src.tgz\n        $ cd solr-4.4.0/\n\n3. Run the full test suite\n\n        $ ant test -Dtests.codec=FDBCodec \\\n                   -Dtests.directory=com.foundationdb.lucene.FDBTestDirectory \\\n                   -lib  ../target/fdb-lucene-layer-0.0.1-SNAPSHOT.jar \\\n                   -lib ../target/dependency/fdb-java-1.0.0.jar\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAydinSakar%2Flucene-layer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAydinSakar%2Flucene-layer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAydinSakar%2Flucene-layer/lists"}