{"id":38540704,"url":"https://github.com/b3j4y/unidisk","last_synced_at":"2026-01-17T07:09:34.810Z","repository":{"id":37686616,"uuid":"83029792","full_name":"B3J4y/UniDisk","owner":"B3J4y","description":"A Crawler to search for keywords and compare the score","archived":false,"fork":false,"pushed_at":"2022-11-25T12:03:25.000Z","size":89515,"stargazers_count":0,"open_issues_count":31,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-05T02:45:17.786Z","etag":null,"topics":["comparison","crawler","nlp","solr-client"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/B3J4y.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-24T10:22:36.000Z","updated_at":"2021-10-04T11:29:55.000Z","dependencies_parsed_at":"2025-04-13T10:00:26.263Z","dependency_job_id":"522e2f09-f6d4-462f-8b34-5d20c56ff4bb","html_url":"https://github.com/B3J4y/UniDisk","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/B3J4y/UniDisk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/B3J4y%2FUniDisk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/B3J4y%2FUniDisk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/B3J4y%2FUniDisk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/B3J4y%2FUniDisk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/B3J4y","download_url":"https://codeload.github.com/B3J4y/UniDisk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/B3J4y%2FUniDisk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28503090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["comparison","crawler","nlp","solr-client"],"created_at":"2026-01-17T07:09:34.697Z","updated_at":"2026-01-17T07:09:34.767Z","avatar_url":"https://github.com/B3J4y.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# UniDisk\n\nA Crawler to search for keywords and compare the score\n\n[![Build Status](https://travis-ci.org/B3J4y/UniDisk.svg?branch=master)](https://travis-ci.org/B3J4y/UniDisk)\n\n## Installation\n\n### Prerequisites\n\n- Java 8\n- Maven\n- Tomcat 8.X\n- IntelliJ Ultimate (preferred)\n\n### Installation steps for primefaces\n\n- build an artifact\n- mvn install\n- mvn test\n- run with tomcat and artifact war\n\n### Docker\n\nWe use Docker to simplify development.\nRun `docker compose up -d` to create a development environment with a MySQL, Solr and Tomcat Server.\nThe API is then available under _localhost:8081/unidisk/rest_.\nIn order for code changes to take effect, the services need to be rebuilt (`docker-compose up -d --build`). This takes some time therefore\nit makes sense to instead kill the web service (`docker rm -fv unidisk_api_1`) and start the server via IntelliJ (or any other way).\nIf the server is started via IntelliJ the API is accessible from _localhost:8080/unidisk_war/rest_.\n\n#### Admin Dashboards\n\nGo to _localhost:8082_ to open Adminer (MySQL Dashboard) or _http://localhost:8983_ for the Solr dashboard.\nEnter the following crendetials for Adminer:\nServer: _db_\nBenutzer: _user_\nPasswort: _secret_\nDatenbank: _unidisk_\n\n### Authentication\n\nWe use Firebase for authentication purposes. Accounts must be created via the Firebase console or CLI. Self sign up is currently not possible.\n\n#### Authentication Method\n\nThe authentication method can be changed by modifying the `authentication` property in _unidisk.properties_. If the specified value is not `firebase` all incoming requests are authenticated if any bearer token is set in the request header.\n\n### Database\n\nThe project contains configuration files for an in memory as well as a MySQL database.\nThe configuration can be changed by replacing the content of the config variable in the\nHibernateUtil class.\n\nAdditional configurations can be added by placing a configuration file into the resource directory. Afterwards\nreference that file in the HibernateUtil class by changing the config variable to\n\n```\npublic class HibernateUtil {\n    private static final String someOtherConfig = \"hibernate.foo.bar.cfg.xml\";\n    private static String config = someOtherConfig;\n    ...\n}\n```\n\n#### Mock setup\n\nYou can populate the database by changing the content of the TestSetupBean class. The init\nfunction stores predefined data from the ApplicationState in the currently configured database.\nThis happens every time the server starts. It's therefore recommended to remove the content\nof the init function after the first start of the app (unless you use an in memory database).\n\n##### In Memory Problems\n\nIf you are on pages that require a specific entity id (e.g. project page) and redeploy the server, the\nformer id might not exist anymore and the page is unable to load.\n\n#### MySQL Development Problems\n\n##### org.hibernate.id.IdentifierGenerationException: could not read a hi value - you need to populate the table: hibernate_sequence\n\nThis error occurs if you truncate the hibernate_sequence table. Delete the database instance and restart the server to fix this problem.\n\n### Solr\n\nFollow the installation guide from https://lucene.apache.org/solr/guide/7_0/index.html.\n\n#### Setup\n\nIf solr isn't running execute `solr start -p 8983`. The further setup will use port 8983 as default.\nIf you chose another one make sure to adapt the urls and commands.\n\n##### Firebase\n\nAsk one of the maintainers for the Firebase service account file and place it into _crawler/src/main/resources_ as _firebase-sa.json_. This is only necessary if you want to use Firebase authentication.\n\n##### Create Core\n\nRun `solr create -c unidisc`. After the command finished you should see the message\n_Created new core 'unidisc'_ in the console/terminal. You should now see the\nunidisc core section at http://localhost:8983/solr/#/unidisc/core-overview.\n\n##### Verify Setup\n\nThe test case `testFieldInputAndQuery` in _SimpleCrawlTest_ should now run successfully.\nYou can also run `shootTheMoon` which crawls websites and posts the result to solr. The test doesn't\nterminate but the number of documents in the _unidisc core_ should increase.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fb3j4y%2Funidisk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fb3j4y%2Funidisk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fb3j4y%2Funidisk/lists"}