https://github.com/bcdh/exist-algolia-index
Uses eXist-db's internal mechanisms to upload and sync indexes with Algolia's cloud services
https://github.com/bcdh/exist-algolia-index
Last synced: about 1 month ago
JSON representation
Uses eXist-db's internal mechanisms to upload and sync indexes with Algolia's cloud services
- Host: GitHub
- URL: https://github.com/bcdh/exist-algolia-index
- Owner: BCDH
- License: gpl-3.0
- Created: 2016-11-12T15:07:25.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2023-12-21T21:30:58.000Z (over 2 years ago)
- Last Synced: 2024-04-24T12:26:46.719Z (about 2 years ago)
- Language: Scala
- Homepage:
- Size: 230 KB
- Stars: 4
- Watchers: 5
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# eXist-db Indexer for Algolia
   [](https://www.gnu.org/licenses/gpl-3.0.html)
eXist Indexer for Algolia is a configurable index plug-in for the [eXist-db](https://github.com/eXist-db/exist) native XML database. It uses eXist's own indexing mechanisms to create, upload and incrementally sync local indexes with [Algolia's](http://www.algolia.com) cloud services.
Example deployment: autocomplete search on http://raskovnik.org
## Installation
This README covers the build, manual installation, and general configuration of the plugin.
### Build
Requirements: Java 17, `sbt`.
```bash
sbt assembly
```
The assembly is written to:
```bash
target/scala-2.13/exist-algolia-index-assembly-.jar
```
### Manual install
The plugin JAR must be built and then installed into eXist manually.
1. Build the assembly:
```bash
sbt assembly
```
2. Copy the resulting JAR into eXist's plugin/library directory.
3. Add the Algolia module to `conf.xml` inside `indexer/modules`:
```xml
```
4. Add the dependency entry to `startup.xml`:
```xml
org.humanistika.exist.index.algolia
exist-algolia-index
VERSION_FROM_VERSION_SBT
exist-algolia-index-assembly-VERSION_FROM_VERSION_SBT.jar
```
5. Restart eXist.
6. Reindex the configured collections so already-present data is pushed into Algolia.
The correct reindex target depends on your own collection structure. Reindex the collection or subcollection whose `collection.xconf` contains the Algolia index configuration.
## Configuration
For a single collection in eXist, you can put data into one or more indexes in Algolia, just create an "index" element inside the "algolia" element for each index and give it the name of the Algolia index, if the index doesn't exist in Algolia it will be automatically created for you.
For incremental indexing to work, you need to have two sets of unique ids, one for each document in the collection (documentId) and one for each rootObject (nodeId).
Algolia writes are sent in batches. The global `batch-size` module attribute defaults to `1000` operations per request. A collection-level `` can override it with `batchSize` if a specific Algolia index needs smaller or larger chunks.
```xml
xml
http://www.w3.org/XML/1998/namespace
```
An optional `visibleBy` attribute can be used to restrict data access when searching the Algolia index.
A `rootObject` is equivalent to an object inside an Algolia Index. We create one "rootObject" either for each document, or document fragment (if you specify a path attribute on the rootObject).
An `attribute` (represents a JSON object attribute, not to be confused with an XML attribute) is a simple key/value pair that is extracted from the XML and placed into the Algolia object ("rootObject" as we call it). All of the text nodes or attribute values indicated by the "path" on the "attribute" element will be serialized to a string (and then converted if you set an explicit "type" attribute).
The path for an "attribute" may point to either an XML element or XML attribute node. Paths must be simple, you can use namespace prefixes in the path, but you must also set the namespaceMappings element in the `collection.xconf`.
The XML Schema file [exist-algolia-index-config.xsd](https://github.com/BCDH/exist-algolia-index/blob/master/src/main/resources/xsd/exist-algolia-index-config.xsd) defines and documents the index configuration.
An `object` represents a JSON object, and this is where things become fun, we basically serialize the XML node pointed to by the "path" attribute on the "object" element to a JSON equivalent. This allows you to create highly complex and structured objects in the Algolia index from your XML.
The `name` attribute that is available on the "attribute" and "object" elements allows you to set the name of the field in the JSON object of the Algolia index, this means that name names of your data fields can be different in Algolia to eXist if you wish.
### Reindexing Existing Data
Installing or updating the plugin does not by itself upload already-present XML documents to Algolia. After installation, reindex each configured collection in eXist so the configured `rootObject`s are serialized and pushed to Algolia.
In general:
- reindex the full configured collection for a first-time backfill
- reindex a narrower subcollection if your deployment replaced only part of the XML corpus and that subcollection has the relevant Algolia collection config
- avoid reindexing broad parent collections unless they are the intended scope of the Algolia configuration
### Indexing Status
The plugin writes deployment-readable indexing status to `algolia-index/status.json` under eXist's configured data directory. The status records are keyed by Algolia index and collection path where a collection is known.
Status states:
- `current`: the latest tracked operation for that index or collection completed successfully
- `degraded`: Algolia rejected or failed a terminal operation such as a batch write, document delete, collection delete, or index drop
- `stale_local_store`: the plugin could not derive collection-delete object IDs from the local Algolia store, usually because the collection was removed before a successful backfill created local state
The local and staging helper scripts fail verification when `status.json` contains `degraded` or `stale_local_store` records. Resolve those states before treating a deployment as successful. In practice, check the failure message in `status.json` and the Algolia/eXist logs, then retry the targeted reindex or run a wider backfill if the local store is missing the needed collection state.
### Limiting Object Access
You can limit data access by setting the `visibleBy` attribute in `collection.xconf` and mapping it to the corresponding path in your XML data, preferably in the document header.
See the test fixture examples:
- XML: [VSK.TEST.xml](https://github.com/BCDH/exist-algolia-index/tree/master/src/test/resources/integration/user-specified-visibleBy/VSK.TEST.xml)
- Configuration: [collection.xconf](https://github.com/BCDH/exist-algolia-index/tree/master/src/test/resources/integration/user-specified-visibleBy/collection.xconf)
## Enable logging in eXist (optional)
You can see what we are sending to Algolia by adding the following to your `$EXIST_HOME/log4j2.xml` file:
Add this as a child of the `` element:
```xml
```
And add this as a child of the `` element:
```xml
```
The log output will then appear in eXist's configured log directory, usually `logs/algolia-index.log` under the active eXist home or container layout, the next time eXist is started.
## Current limitations
When you back up eXist, you should also back up the `algolia-index` directory inside eXist's configured data directory, because it holds the local representation of what is stored on the remote Algolia server. Support for integrating that local store into a native backup/restore workflow may be added later.
## Acknowledgements
Hats off to [Adam Retter](https://github.com/adamretter) for sharing his superb programming skills with us in this project.
This tool was developed in the context of ongoing work at [BCDH](http://www.humanistika.org), including Raskovnik, a Serbian dictionary platform built together with the Institute of Serbian Language.