https://github.com/julien040/hn-recommendation-api

A recommendation system for Hacker News. Get the most similar posts for a given URL
https://github.com/julien040/hn-recommendation-api

embedding embeddings faiss hacker-news hnsw nextjs openai recommendation

Last synced: about 1 month ago
JSON representation

A recommendation system for Hacker News. Get the most similar posts for a given URL

• Host: GitHub
• URL: https://github.com/julien040/hn-recommendation-api
• Owner: julien040
• Created: 2023-06-13T15:59:18.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-06-23T16:11:26.000Z (about 1 year ago)
• Last Synced: 2024-06-30T15:48:21.944Z (3 months ago)
• Topics: embedding, embeddings, faiss, hacker-news, hnsw, nextjs, openai, recommendation
• Language: TypeScript
• Homepage: https://hn-recommend.julienc.me
• Size: 144 KB
• Stars: 80
• Watchers: 1
• Forks: 7
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# hn-recommendation-api

This project provides a recommendation engine for Hacker News posts based on their content. It utilizes the concept of embeddings to compute the similarity between posts. Specifically, the text of each post is extracted using the Diffbot API, and embeddings are computed using the OpenAI API with the "text-embedding-ada-002" model. An HNSW index is built using Faiss to quickly find the nearest items.

## Components

The project consists of three parts:

-   `root`: The API using Python and FastAPI.

-   `next`: The website using Next.JS and Typescript.

-   `createIndex.piynb`: The Jupyter notebook to convert the dataset to a Faiss HNSW index.

## Dataset

The dataset used to train the embeddings is available on Kaggle: https://www.kaggle.com/datasets/julien040/hacker-news-openai-embeddings

## Blog Post

A blog post about extracting the embeddings of Hacker News posts to get a recommendation engine is available at https://julienc.me/articles/Extract_embeddings_Hacker_News_article

## Contact

For any questions or API access requests, please contact me at contact[at]julienc.me.

## Disclaimer

This project is not affiliated with Y Combinator or Hacker News.

## License

This project is licensed under the MIT License.