{"id":39497271,"url":"https://github.com/clarin-eric/fcs-sru-aggregator","last_synced_at":"2026-01-18T05:43:53.106Z","repository":{"id":36088925,"uuid":"154054840","full_name":"clarin-eric/fcs-sru-aggregator","owner":"clarin-eric","description":"CLARIN Federated Content Search v3 Aggregator – Augmenting your Search Engine","archived":false,"fork":false,"pushed_at":"2025-11-27T16:55:37.000Z","size":15883,"stargazers_count":7,"open_issues_count":16,"forks_count":2,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-11-28T05:44:40.268Z","etag":null,"topics":["clarin","fcs","java","reactjs","search","search-engine"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clarin-eric.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-10-21T21:42:02.000Z","updated_at":"2025-11-27T16:55:41.000Z","dependencies_parsed_at":"2025-03-12T16:33:00.964Z","dependency_job_id":"62b2b1e8-4b8f-41c5-884f-eacbcc0082d5","html_url":"https://github.com/clarin-eric/fcs-sru-aggregator","commit_stats":null,"previous_names":[],"tags_count":61,"template":false,"template_full_name":null,"purl":"pkg:github/clarin-eric/fcs-sru-aggregator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clarin-eric%2Ffcs-sru-aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clarin-eric%2Ffcs-sru-aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clarin-eric%2Ffcs-sru-aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clarin-eric%2Ffcs-sru-aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clarin-eric","download_url":"https://codeload.github.com/clarin-eric/fcs-sru-aggregator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clarin-eric%2Ffcs-sru-aggregator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28531368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clarin","fcs","java","reactjs","search","search-engine"],"created_at":"2026-01-18T05:43:28.323Z","updated_at":"2026-01-18T05:43:53.100Z","avatar_url":"https://github.com/clarin-eric.png","language":"Java","funding_links":[],"categories":["Reference Implementations for SRU / FCS"],"sub_categories":[],"readme":"# FCS SRU Aggregator\n\nCLARIN Federated Content Search Aggregator – Augmenting your Search Engine\n\n## Introduction\n\nThe [CLARIN Federated Content Search (CLARIN-FCS)](https://www.clarin.eu/content/content-search) introduces an _interface specification_ that decouples the _search engine_ functionality from its _exploitation_, i.e. user-interfaces, third-party applications, to allow services to access heterogeneous search engines in a uniform way.\n\n## How does it work?\n\n1. a _Client_ submits a query to an _Endpoint_\n2. The _Endpoint_ translates the query from CQL or FCS-QL to the query dialect used by the _Search Engine_ and submits the translated query to the _Search Engine_\n3. The _Search Engine_ processes the query and generates a result set, i.e. it compiles a set of hits that match the search criteria.\n4. The _Endpoint_ then translates the results from the Search Engine-specific result set format to the CLARIN-FCS result format and sends them to the _Client_.\n\n## The Aggregator\n\nThe FCS Aggregator is a search engine for language resources hosted at a [variety of institutions](https://centres.clarin.eu/fcs). The user interface allows search queries to access heterogeneous data sources in a uniform way. It is called _Aggregator_ because it sends out search queries to remote search engines and then collects the search results.\n\nThe [FCS Aggregator](https://contentsearch.clarin.eu/) is running at [CLARIN](https://www.clarin.eu/).\n\n## The Specification\n\nThe Specification for **Federated Content Search Core 2.0** can be found as a\n[PDF document](https://office.clarin.eu/v/CE-2017-1046-FCS-Specification-v20230426.pdf). Sources for the various specification documents can be found at the [FCS Misc Webpage](https://clarin-eric.github.io/fcs-misc/).\n\nFor more details visit at the [CLARIN FCS – Technical Details](https://www.clarin.eu/content/federated-content-search-clarin-fcs-technical-details) page.\n\n## Changelog\n\nFor a detailed list of changes, please take a look at [`CHANGELOG.md`](CHANGELOG.md).\n\n## Building\n\n### Requirements\n\nThe backend Java web server requires **Java 11** or newer, with Maven version 3.6.3.\n\nThe frontend React web application requires **Node 22** (with NPM 10). It is only required for re-compiling the frontend. You can also use the bundled, pre-compiled files if no further customization is required.\n\nDocker might be required if you want to run everything containerized. Then both the Java and Node requirements are also not necessary as the multi-stage build process handles everything.\n\n### Source Code\n\n```bash\n# fetch the source code\ngit clone https://github.com/clarin-eric/fcs-sru-aggregator.git \u0026\u0026 cd fcs-sru-aggregator/\n\n# initialize the git submodules\ngit submodule update --init --recursive aggregator-webui/\n```\n\n### Dropwizard Server Application (Backend)\n\nYou can build the Java [dropwizard](https://www.dropwizard.io/en/stable/) server application by running the [`scripts/build.sh`](scripts/build.sh) script. This will generate a [`aggregator-app-*.jar`](aggregator-app/target/) binary that bundles backend, frontend and all relevant dependencies.\n\n```bash\n./scripts/build.sh\n```\n\n### React Web Frontend\n\nThe React web frontend is already included as assets in [`aggregator-app/src/main/resources/assets/webapp`](aggregator-app/src/main/resources/assets/webapp) with the [index page `aggregator-app/src/main/resources/eu/clarin/sru/fcs/aggregator/app/index.mustache`](aggregator-app/src/main/resources/eu/clarin/sru/fcs/aggregator/app/index.mustache).\n\nIf you want to modify or update the web application, you can use the [`scripts/update-webui.sh`](scripts/update-webui.sh) script. This will use the [`aggregator-webui/`](aggregator-webui/) git submodule containing the sources and rebuild everything and then place the assets back into the Java [asset folder](aggregator-app/src/main/resources/assets/webapp).\n\n```bash\n# optionally update the git submodule code\n#git submodule update --remote aggregator-webui/\n\n# rebuild and update web UI files\n./scripts/update-webui.sh\n```\n\nThe React web frontend can also be run as a standalone web application. However, it requires the FCS Aggregator REST API to get data and process search requests. You can find the source code at https://github.com/clarin-eric/fcs-sru-aggregator-ui/.\n\n### Using Docker\n\nTo create the docker image, simply build the [`Dockerfile`](Dockerfile). This also allows to configure the frontend a bit by modifying the `webui.env` ([template](webui.env.template.textplus)) file. The aggregator also uses the default configurations in [`aggregator-app/aggregator.yml`](aggregator-app/aggregator.yml) which can be overriden by mounting another file over it.\n\n```bash\ndocker build --tag=fcs-sru-aggregator .\n```\n\nIf you do not want to change anything, you can also use the [`Dockerfile.simple`](Dockerfile.simple) which uses the prebuild web frontend assets and only recompiles the Java sources. The [`aggregator-app/aggregator.yml`](aggregator-app/aggregator.yml) configurations will also be used.\n\n## Running\n\n### Configuration\n\nSee [`DEPLOYMENT.md`](aggregator-app/DEPLOYMENT.md) for example deployment configurations and descriptions about settings.\n\n### Natively\n\nYou can run the application in development and production mode. Please look at the [`scripts/run-(devel|prod).sh`](scripts/) files for more details.\n\nIn development the [`aggregator_devel.yml`](aggregator-app/aggregator_devel.yml) and for production the [`aggregator.yml`](aggregator-app/aggregator.yml) configuration files are used.\n\n```bash\n# Run in development mode\n# Also allows to attach a debugger\n./scripts/run-devel.sh\n\n# Run in production mode (uses the build aggregator-app-*.jar file)\n./scripts/run-prod.sh\n```\n\nYou then can access the locally running Aggregator at [http://localhost:4019/](http://localhost:4019/).\n\n### Using Docker\n\nUse docker mounts to allow access to generated data and logs, and to override default configuration files.\n\n```bash\n# copy of resources for faster startups\ntouch fcsAggregatorResources.json fcsAggregatorResources.backup.json\n# statistics output directory\nmkdir stats\n\n# run in background (detached), and restart (unless stopped by operator)\n#   -d --restart unless-stopped\n# configure public port (internal 4019, external 4020)\n#   -p 4020:4019\n# aggregator configuration\n#   -v $(pwd)/aggregator-app/aggregator.yml:/work/aggregator.yml:ro\n# (cached) aggregator resources\n#   -v $(pwd)/fcsAggregatorResources.json:/var/lib/aggregator/fcsAggregatorResources.json\n#   -v $(pwd)/fcsAggregatorResources.backup.json:/var/lib/aggregator/fcsAggregatorResources.backup.json\n# if statistics logging (fcsstats logger) is enabled, mount the output directory\n#   -v $(pwd)/stats:/var/log/aggregator/stats\n# provide a container name with `--name`\n# and specify the docker image (see build step above)\ndocker run \\\n    -d \\\n    --restart unless-stopped \\\n    -p 4020:4019 \\\n    -v $(pwd)/aggregator-app/aggregator.yml:/work/aggregator.yml:ro \\\n    -v $(pwd)/fcsAggregatorResources.json:/var/lib/aggregator/fcsAggregatorResources.json \\\n    -v $(pwd)/fcsAggregatorResources.backup.json:/var/lib/aggregator/fcsAggregatorResources.backup.json \\\n    -v $(pwd)/stats:/var/log/aggregator/stats \\\n    --name fcs-sru-aggregator \\\n    fcs-sru-aggregator\n\n# if you did not specify the `--rm` option and used the `--restart` parameter,\n# you have to manually stop and remove the container to quit the server and allow for future starts\ndocker stop fcs-sru-aggregator\ndocker rm fcs-sru-aggregator\n```\n\nYou then can access the locally running Aggregator at [http://localhost:4020/](http://localhost:4020/).\n\n## FCS Endpoints\n\n### Endpoint Reference Implementations\n\nIf you have any kind of RESTful API to your Search Engine using the [Korp Endpoint Reference Implementation](https://github.com/clarin-eric/fcs-korp-endpoint/) as a starting point should be the way to go. If you more specifically are using Korp it should only be a simple adaptation to corpora and tagsets needed. In any case do not forget to look at the tests.\n\nFor lexical resources, an [FCS Endpoint with Solr search engine](https://github.com/saw-leipzig/lexfcs-endpoint) might be a good starting point. It allows index XML structured data in the Solr search engine and handles search query conversion. It can also be used as a template if you want to connect other search engines by modifying the query translation and API handling.\n\nAn older [CQI Bridge endpoint](https://github.com/clarin-eric/fcs-sru-cqi-bridge) also exists but only support simple full-texts search.\n\nFor more endpoint implementations, please take a look at the [**Awesome FCS** list](https://github.com/clarin-eric/awesome-fcs).\n\n### Endpoint Validation\n\nTo test your Endpoint you can point the [FCS Endpoint Validator](https://www.clarin.eu/fcsvalidator/) ([code](https://github.com/saw-leipzig/fcs-endpoint-validator)) to your Endpoint.\n\n\u003c!--\n\n### What is new in FCS Aggregator v3.0?\n\n- A new graphical query builder (GQB) to support the new Query Language FCS-QL\n- Support for the AdvancedDataView, ADV, with layer capabilities\n- Backwards compatibility to earlier versions of Endpoints and protocols, _legacy_ and _version1_\n\nThe backwards compatibility gives you as a Centre search engine maintainer a smooth transtion\nto the new features and capabilities at your own convenience.\n\n### What is new in FCS v2.0?\n\n- A new Query Language\n- A matching display of query results, the AdvancedDataView, ADV, with layer capabilities\n- Backwards compatibility to earlier versions\n\nThese new additions to the CLARIN-FCS will not only enhance the power user experience and\npossibilities when performing queries from repositories, but also that less experienced\nusers will find it easier to explore different corpora.\n\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclarin-eric%2Ffcs-sru-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclarin-eric%2Ffcs-sru-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclarin-eric%2Ffcs-sru-aggregator/lists"}