https://github.com/wellcomecollection/search-logger
https://github.com/wellcomecollection/search-logger
aws digital-platform kinesis terraform
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/wellcomecollection/search-logger
- Owner: wellcomecollection
- Created: 2019-02-25T16:01:57.000Z (over 7 years ago)
- Default Branch: main
- Last Pushed: 2025-01-15T16:37:03.000Z (over 1 year ago)
- Last Synced: 2025-01-15T18:52:31.072Z (over 1 year ago)
- Topics: aws, digital-platform, kinesis, terraform
- Language: HCL
- Homepage:
- Size: 971 KB
- Stars: 0
- Watchers: 9
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
# search-logger
This is a tool which logs user searches to our "reporting" Elasticsearch cluster.
These logs include:
* what search term somebody typed in
* how many results we returned
* which results (if any) that they clicked on.
We use this to analyse search behaviour and improve our queries.
For example, we can look at searches that return 0 results and discuss whether there really are no results, or whether we should change the results we return.
## How search events get logged
```mermaid
flowchart LR
W[website] --> S[Segment]
S[Segment] --> K[Kinesis stream]
K --> L[search logger Lambda]
L --> R[(reporting
cluster)]
classDef externalNode fill:#e8e8e8,stroke:#8f8f8f
class S,R,W externalNode
classDef repoNode fill:#c8ecee,stroke:#298187,stroke-width:2px
class K,L repoNode
```
Our website [sends tracking events][track.ts] to [Segment].
Those segments are forwarded to a [Kinesis data stream][kinesis], which triggers a Lambda function.
That Lambda function writes the search logs into the [reporting cluster].
This repo contains:
* the source code for the Lambda function
* the Terraform definitions for the Kinesis stream and the Lambda function
[track.ts]: https://github.com/wellcomecollection/wellcomecollection.org/blob/9115873707b411a1ecfe2a93f5ebf7f240861c8f/common/services/conversion/track.ts#L6
[Segment]: https://segment.com/
[kinesis]: https://segment.com/docs/destinations/amazon-kinesis/
[reporting cluster]: https://reporting.wellcomecollection.org
## Developer info
* To deploy a new version the function:
```console
$ cd lambda
$ yarn deploy
$ cd ../terraform
$ ./run_terraform.sh apply
```
* You can see the Lambda logs in the logging cluster.