{"id":24998163,"url":"https://github.com/namoshek/laravel-scout-database","last_synced_at":"2025-04-13T04:14:02.805Z","repository":{"id":42205517,"uuid":"255608136","full_name":"Namoshek/laravel-scout-database","owner":"Namoshek","description":"A generic Laravel Scout driver which performs full-text search on indexed model data using an SQL database as storage backend. Indexed data is stored in normalized form, allowing efficient search.","archived":false,"fork":false,"pushed_at":"2025-03-28T13:48:39.000Z","size":162,"stargazers_count":16,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-13T04:13:51.108Z","etag":null,"topics":["laravel","laravel-scout","laravel-scout-driver","search-engine","sql-database"],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Namoshek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-14T12:50:48.000Z","updated_at":"2025-04-09T10:15:47.000Z","dependencies_parsed_at":"2023-01-31T19:15:37.165Z","dependency_job_id":"7fab2bf8-48bb-4595-838c-d03e40e8e085","html_url":"https://github.com/Namoshek/laravel-scout-database","commit_stats":{"total_commits":49,"total_committers":4,"mean_commits":12.25,"dds":"0.40816326530612246","last_synced_commit":"637430cce297194fb982d10fe81f49db2a7d9051"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Namoshek%2Flaravel-scout-database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Namoshek%2Flaravel-scout-database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Namoshek%2Flaravel-scout-database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Namoshek%2Flaravel-scout-database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Namoshek","download_url":"https://codeload.github.com/Namoshek/laravel-scout-database/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248661719,"owners_count":21141451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["laravel","laravel-scout","laravel-scout-driver","search-engine","sql-database"],"created_at":"2025-02-04T17:39:34.178Z","updated_at":"2025-04-13T04:14:02.778Z","avatar_url":"https://github.com/Namoshek.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Laravel Scout driver for SQL Database based Search Indexing\n\n[![Latest Version on Packagist](https://img.shields.io/packagist/v/namoshek/laravel-scout-database.svg?style=flat-square)](https://packagist.org/packages/namoshek/laravel-scout-database)\n[![Total Downloads](https://img.shields.io/packagist/dt/namoshek/laravel-scout-database.svg?style=flat-square)](https://packagist.org/packages/namoshek/laravel-scout-database)\n[![Tests](https://github.com/Namoshek/laravel-scout-database/workflows/Tests/badge.svg)](https://github.com/Namoshek/laravel-scout-database/actions?query=workflow%3ATests)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=namoshek_laravel-scout-database\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=namoshek_laravel-scout-database)\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=namoshek_laravel-scout-database\u0026metric=sqale_rating)](https://sonarcloud.io/dashboard?id=namoshek_laravel-scout-database)\n[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=namoshek_laravel-scout-database\u0026metric=reliability_rating)](https://sonarcloud.io/dashboard?id=namoshek_laravel-scout-database)\n[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=namoshek_laravel-scout-database\u0026metric=security_rating)](https://sonarcloud.io/dashboard?id=namoshek_laravel-scout-database)\n[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=namoshek_laravel-scout-database\u0026metric=vulnerabilities)](https://sonarcloud.io/dashboard?id=namoshek_laravel-scout-database)\n[![License](https://poser.pugx.org/namoshek/laravel-scout-database/license)](https://packagist.org/packages/namoshek/laravel-scout-database)\n\nThis package provides a generic Laravel Scout driver which performs full-text search on indexed model data using an SQL database as storage backend.\nIndexed data is stored in normalized form, allowing efficient and fuzzy search which does not require a full and/or exact match.\n\nThis driver is an alternative to [`teamtnt/laravel-scout-tntsearch-driver`](https://github.com/teamtnt/laravel-scout-tntsearch-driver).\nThe primary difference is that this driver provides fewer features (like geo search).\nInstead, it works with all database systems supported by Laravel itself (which are basically all PDO drivers).\nAlso, the search algorithm is slightly different.\n\nAll tests are run through GitHub Actions for PHP 8.0, 8.1 and 8.2 on the following database systems:\n- SQLite 3\n- MySQL 8.0\n- PostgreSQL 13.1\n- SQL Server 2017\n\nActual limitations regarding supported database systems are mostly related to the use of _Common Table Expression_ using [staudenmeir/laravel-cte](https://github.com/staudenmeir/laravel-cte).\nPlease make sure your database system is supported before using the package, or you might run into database errors.\n\n## Installation\n\nYou can install the package via composer:\n\n```bash\ncomposer require namoshek/laravel-scout-database\n```\n\nAfter installing the package, the configuration file as well as the migrations need to be published:\n\n```bash\nphp artisan vendor:publish --provider=\"Namoshek\\Scout\\Database\\ScoutDatabaseServiceProvider\" --tag=\"config\"\nphp artisan vendor:publish --provider=\"Namoshek\\Scout\\Database\\ScoutDatabaseServiceProvider\" --tag=\"migrations\"\n```\n\nIf you would like to use a different table prefix than `scout_` for the tables created by this package,\nyou should change the configuration as well as the copied migrations accordingly.\nWhen you have done so, you can then apply the database migrations:\n\n```bash\nphp artisan migrate\n```\n\n### Upgrading from `v0.x` to `v1.x`\n\n#### Migrations\n\nWith the new version, the database schema has changed and new migrations need to be published using:\n\n```bash\nphp artisan vendor:publish --provider=\"Namoshek\\Scout\\Database\\ScoutDatabaseServiceProvider\" --tag=\"migrations\"\n```\n\nThe same hint as mentioned above in the _Installation_ section applies to the newly published migrations as well.\n\n#### Config\n\nThe configuration has been reduced slightly and you might want to compare the new configuration file with the old one to remove obsolete settings.\nSkipping this part has no negative impact on the performance of the Scout driver, though.\n\n#### Commands\n\nThe `\\Namoshek\\Scout\\Database\\Commands\\CleanWordsTable::class` command has been removed and you should un-schedule it, if you added it previously.\n\n#### Noteworthy\n\nMost occurrences of `protected` fields and methods have been changed to `private` to simplify development in regard to backwards-compatibility breaking changes in the future.\nIf you have not been actively overriding parts of the implementation, this does not affect you at all.\n\n## Configuration\n\nIn order to instruct Scout to use the driver provided by this package, you need to change the `driver` option in `config/scout.php`\nto `database`. If you did not change the Scout configuration file, you can also set the `SCOUT_DRIVER` environment variable to `database` instead.\n\nAll available configuration options of the package itself can be found in `config/scout-database.php`.\nThe options are described thoroughly in the file itself. By default, the package uses the [`UnicodeTokenizer`](src/Tokenizer/UnicodeTokenizer.php)\nand the [`PorterStemmer`](src/Stemmer/PorterStemmer.php) which is suitable for the English language. The search adds a trailing wildcard to the\nlast token and not all search terms need to be found in order for a document to show up in the results (there must be at least one match though).\n\nYou may also add a wildcard to each search token by enabling `wildcard_all_tokens` in the config file altough this is not recommended for performance reasons.\n\n_A basic installation most likely does not require you to change any of these settings. Just to make sure, you should have a look at the\n`connection` option though. If you want to change this, do so before running the migrations or the tables will be created using the wrong\ndatabase connection._\n\n### Supported Tokenizers\n\nCurrently, only a [`UnicodeTokenizer`](src/Tokenizer/UnicodeTokenizer.php) is available. It will split strings at any character which is neither\na letter, nor a number according to the `\\p{L}` and `\\p{N}` regex patterns. This means that dots, colons, dashes, whitespace, etc. are split criteria.\n\nIf you have different requirements for a tokenizer, you can provide your own implementation via the configuration. Just make sure it implements the\n[`Tokenizer`](src/Contracts/Tokenizer.php) interface.\n\n### Supported Stemmers\n\nCurrently, all stemmers implemented by the [`wamania/php-stemmer`](https://github.com/wamania/php-stemmer) package are available. A wrapper class\nhas been added for each of them:\n\n- [`DanishStemmer`](src/Stemmer/DanishStemmer.php)\n- [`DutchStemmer`](src/Stemmer/DutchStemmer.php)\n- [`EnglishStemmer`](src/Stemmer/EnglishStemmer.php)\n- [`FrenchStemmer`](src/Stemmer/FrenchStemmer.php)\n- [`GermanStemmer`](src/Stemmer/GermanStemmer.php)\n- [`ItalianStemmer`](src/Stemmer/ItalianStemmer.php)\n- [`NorwegianStemmer`](src/Stemmer/NorwegianStemmer.php)\n- [`NullStemmer`](src/Stemmer/NullStemmer.php) _(can be used to disable stemming)_\n- [`PorterStemmer`](src/Stemmer/PorterStemmer.php) _(default, same as [`EnglishStemmer`](src/Stemmer/EnglishStemmer.php))_\n- [`PortugueseStemmer`](src/Stemmer/PortugueseStemmer.php)\n- [`RomanianStemmer`](src/Stemmer/RomanianStemmer.php)\n- [`RussianStemmer`](src/Stemmer/RussianStemmer.php)\n- [`SpanishStemmer`](src/Stemmer/SpanishStemmer.php)\n- [`SwedishStemmer`](src/Stemmer/SwedishStemmer.php)\n\nIf you have different requirements for a stemmer, you can provide your own implementation via the configuration. Just make sure it implements the\n[`Stemmer`](src/Contracts/Stemmer.php) interface.\n\n## Usage\n\nThe package follows the available use cases described in the [official Scout documentation](https://laravel.com/docs/9.x/scout).\nPlease be aware of the listed [limitations](#limitations) though.\n\n### How does it work?\n\n#### The Indexing\n\nThe search driver internally uses a single table, which contains terms and the association to documents. When indexing documents (i.e. adding\nor updating models in the search index) the engine will use the configured tokenizer to split the input of each column into tokens.\nThe tokenizer configured by default simply splits inputs into words consisting of any unicode letter or number, which means any other character\nlike `,`, `.`, `-`, `_`, `!`, `?`, `/`, whitespace and all other special characters are considered separators for the tokens and will be removed\nby the tokenizer. This way such characters will never end up in the search index itself.\n\nAfter the inputs have been tokenized, each token (and at this point we actually expect our tokens to be words) is run through the configured\nstemmer to retrieve the stem (i.e. _root word_). Performing this action allows us to search for similar words later.\nThe [`PorterStemmer`](src/Stemmer/PorterStemmer.php) for example will produce `intellig` as output for both `intelligent` as well as \n`intelligence` as input. How this helps when searching will be clear in a moment.\n\nFinally, the results of this process are stored in the database. The _index_ table is filled with the results of the stemming process\nand the associations to the indexed models (model type and identifier).\nOn top of that, for each row in the index, the database also contains the number of occurences in a document.\nWe use this information for scoring within the search part of our engine.\n\n#### The Search\n\nWhen executing a search query, the same tokenizing and stemming process as used for indexing is applied to the search query string. The result of\nthis process is a list of stems (or _root words_) which are then used to perform the actual search. Depending on the configuration of the package,\nthe search will return documents which contain at least one or all of the stems. This is done by calculating a score for each match in the index\nbased on the inverse document frequency (i.e. the ratio between indexed documents and documents containing one of the searched words),\nthe term frequency (i.e. the number of occurrences of a search term within a document) and the term deviation (which is only relevant for the\nwildcard search). Returned are documents ordered by their score in descending order, until the desired limit is reached.\n\n### Extending the Search Index\n\nIt is possible to extend the search index table (`scout_index`) with custom columns.\nDuring indexing, these columns may be filled with custom content and during searching the searches can be scoped to these columns (exact match).\nThis feature is particularly useful when working with a multi-tenancy application where the search index is used by multiple tenants.\n\n#### Example Migration\n\nIn our example, we add a mandatory `tenant_id` column to the search index.\n\n```php\nreturn new class extends Migration {\n    public function up(): void\n    {\n        Schema::table('scout_index', function (Blueprint $table) {\n            $table-\u003euuid('tenant_id');\n        });\n    }\n\n    public function down(): void\n    {\n        Schema::table('scout_index', function (Blueprint $table) {\n            $table-\u003edropColumn(['tenant_id']);\n        });\n    }\n};\n```\n\n#### Indexing Example\n\nThe `tenant_id` is added during indexing for each model:\n\n```php\nclass User extends Model\n{\n    public function toSearchableArray(): array\n    {\n        return [\n            'id' =\u003e $this-\u003eid,\n            'name' =\u003e $this-\u003ename,\n            'tenant_id' =\u003e new StandaloneField($this-\u003etenant_id),\n        ];\n    }\n}\n```\n\n#### Search Example\n\nThe `tenant_id` is filtered during search based on the `$tenantId`, which may for example be taken from the HTTP request:\n\n```php\nUser::search('Max Mustermann')\n    -\u003ewhere('tenant_id', $tenantId)\n    -\u003eget();\n```\n\n## Limitations\n\nObviously, this package does not provide a search engine which (even remotely) brings the performance and quality a professional search engine\nlike Elasticsearch offers. This solution is meant for smaller to medium-sized projects which are in need of a rather simple-to-setup solution.\n\nAlso worth noting, the following Scout features are currently not implemented:\n- Soft Deletes\n- Search custom index using `User::search('Mustermann')-\u003ewithin('users_without_admins')`\n- Search with custom order using `User::search('Musterfrau')-\u003eorderBy('age', 'desc')`\n  - Implementing this feature would be difficult in combination with the scoring algorithm. Only the result of the database query could be ordered, while this could then lead to issues with pagination.\n\n### Known Issues\n\nOne issue with this search engine is that it can lead to issues if multiple queue workers work on the indexing of a single document concurrently\n(database will deadlock).\nTo circumvent this issue, a the number of attempts used for transactions is configurable. By default, each transaction is tried a maximum of three\ntimes if a deadlock (or any other error) occurs.\n\n## License\n\nThe MIT License (MIT). Please see [License File](LICENSE.md) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnamoshek%2Flaravel-scout-database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnamoshek%2Flaravel-scout-database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnamoshek%2Flaravel-scout-database/lists"}