https://github.com/pranav-kural/products-vectro
Shopify app enabling merchants to load products data into a vector database.
https://github.com/pranav-kural/products-vectro
ai generative-ai shopify shopify-app vector-database vectordb
Last synced: 7 months ago
JSON representation
Shopify app enabling merchants to load products data into a vector database.
- Host: GitHub
- URL: https://github.com/pranav-kural/products-vectro
- Owner: pranav-kural
- Created: 2024-08-22T00:17:21.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-08-22T23:07:54.000Z (about 1 year ago)
- Last Synced: 2025-01-23T03:29:12.522Z (9 months ago)
- Topics: ai, generative-ai, shopify, shopify-app, vector-database, vectordb
- Language: TypeScript
- Homepage:
- Size: 369 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# Products Vectro
An open-source project providing template of a Shopify app that enables merchants to load products data into a vector database.
## Features
- **Full Data Ingestion Cycle:** Handles full data ingestion pipeline from collecting data from GraphQL Admin API, pre-processing data, and generating embeddings to storing embedding vectors in vector database.
- **Vector Store Providers:** Supports multiple vector store providers like Pinecone, Datastax (AstraDB), and Elastic (Elasticsearch) out of the box, giving you the flexibility to choose the best provider for your use case.
- **Embedding Model Providers:** Supports multiple state-of-the-art embedding model providers like OpenAI, Google Generative AI, and Hugging Face out of the box, giving you the flexibility to choose the best provider for your use case.
### Supported Vector Store Providers
### Supported Embedding Model Providers
### Screenshot of Interface
## How it works?
You can simply clone this project and create your own Shopify app to load products data into a vector database. The app will provide the required user interface in the Shopify Admin UI to configure the data ingestion pipeline, and will handle the data ingestion process in the background.
Basic data ingestion workflow is as follows:
1. **Vector store configuration**: Merchant configures the vector store provider and provides the required credentials.
2. **Embedding model configuration**: Merchant configures the embedding model provider and provides the required credentials.
3. **Data Configuration**: Merchant can specify custom configurations for loading products data, for example, the number of products to load, the fields to include, products to exclude, etc.
4. **Data Ingestion**: The app will start the data ingestion process in the background. It will collect products data from the Shopify Admin API, pre-process the data, generate embeddings using the specified model, and store the embeddings in the vector store.
## Quick start
Begin by cloning the repository and installing the dependencies.
```bash
git clone
```
### Prerequisites
Before you begin, you'll need the following:
1. **Node.js**: [Download and install](https://nodejs.org/en/download/) it if you haven't already.
2. **Shopify Partner Account**: [Create an account](https://partners.shopify.com/signup) if you don't have one.
3. **Test Store**: Set up either a [development store](https://help.shopify.com/en/partners/dashboard/development-stores#create-a-development-store) or a [Shopify Plus sandbox store](https://help.shopify.com/en/partners/dashboard/managing-stores/plus-sandbox-store) for testing your app.
### Setup
Use below commands to install the dependencies.
Using yarn:
```shell
yarn install
```
Using npm:
```shell
npm install
```
Using pnpm:
```shell
pnpm install
```
### Local Development
Using yarn:
```shell
yarn dev
```
Using npm:
```shell
npm run dev
```
Using pnpm:
```shell
pnpm run dev
```
Press P to open the URL to your app. Once you click install, you can start development.
Local development is powered by [the Shopify CLI](https://shopify.dev/docs/apps/tools/cli). It logs into your partners account, connects to an app, provides environment variables, updates remote config, creates a tunnel and provides commands to generate extensions.
Please read the [documentation for @shopify/shopify-app-remix](https://www.npmjs.com/package/@shopify/shopify-app-remix#authenticating-admin-requests) to understand what other API's are available.
## Deployment
### Application Storage
This project uses [Prisma](https://www.prisma.io/) to store session data, by default using an [SQLite](https://www.sqlite.org/index.html) database.
The database is defined as a Prisma schema in `prisma/schema.prisma`.
This use of SQLite works in production if your app runs as a single instance.
The database that works best for you depends on the data your app needs and how it is queried.
You can run your database of choice on a server yourself or host it with a SaaS company.
Here’s a short list of databases providers that provide a free tier to get started:
| Database | Type | Hosters |
| ---------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MySQL | SQL | [Digital Ocean](https://www.digitalocean.com/try/managed-databases-mysql), [Planet Scale](https://planetscale.com/), [Amazon Aurora](https://aws.amazon.com/rds/aurora/), [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql) |
| PostgreSQL | SQL | [Digital Ocean](https://www.digitalocean.com/try/managed-databases-postgresql), [Amazon Aurora](https://aws.amazon.com/rds/aurora/), [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres) |
| Redis | Key-value | [Digital Ocean](https://www.digitalocean.com/try/managed-databases-redis), [Amazon MemoryDB](https://aws.amazon.com/memorydb/) |
| MongoDB | NoSQL / Document | [Digital Ocean](https://www.digitalocean.com/try/managed-databases-mongodb), [MongoDB Atlas](https://www.mongodb.com/atlas/database) |
To use one of these, you can use a different [datasource provider](https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#datasource) in your `schema.prisma` file, or a different [SessionStorage adapter package](https://github.com/Shopify/shopify-api-js/blob/main/packages/shopify-api/docs/guides/session-storage.md).
### Build
Remix handles building the app for you, by running the command below with the package manager of your choice:
Using yarn:
```shell
yarn build
```
Using npm:
```shell
npm run build
```
Using pnpm:
```shell
pnpm run build
```
## Hosting
When you're ready to set up the app in production, you can follow [our deployment documentation](https://shopify.dev/docs/apps/deployment/web) to host the app on a cloud provider like [Heroku](https://www.heroku.com/) or [Fly.io](https://fly.io/).
When you reach the step for [setting up environment variables](https://shopify.dev/docs/apps/deployment/web#set-env-vars), you also need to set the variable `NODE_ENV=production`.
### Hosting on Vercel
Using the Vercel Preset is recommended when hosting your Shopify Remix app on Vercel. You'll also want to ensure imports that would normally come from `@remix-run/node` are imported from `@vercel/remix` instead. Learn more about hosting Remix apps on Vercel [here](https://vercel.com/docs/frameworks/remix).
```diff
// vite.config.ts
import { vitePlugin as remix } from "@remix-run/dev";
import { defineConfig, type UserConfig } from "vite";
import tsconfigPaths from "vite-tsconfig-paths";
+ import { vercelPreset } from '@vercel/remix/vite';
installGlobals();
export default defineConfig({
plugins: [
remix({
ignoredRouteFiles: ["**/.*"],
+ presets: [vercelPreset()],
}),
tsconfigPaths(),
],
});
```
## Gotchas / Troubleshooting
### Database tables don't exist
If you get this error:
```
The table `main.Session` does not exist in the current database.
```
You need to create the database for Prisma. Run the `setup` script in `package.json` using your preferred package manager.
### Navigating/redirecting breaks an embedded app
Embedded Shopify apps must maintain the user session, which can be tricky inside an iFrame. To avoid issues:
1. Use `Link` from `@remix-run/react` or `@shopify/polaris`. Do not use ``.
2. Use the `redirect` helper returned from `authenticate.admin`. Do not use `redirect` from `@remix-run/node`
3. Use `useSubmit` or `` from `@remix-run/react`. Do not use a lowercase ``.
This only applies if you app is embedded, which it will be by default.
### Non Embedded
Shopify apps are best when they are embedded into the Shopify Admin. This template is configured that way. If you have a reason to not embed your please make 2 changes:
1. Change the `isEmbeddedApp` prop to false for the `AppProvider` in `/app/routes/app.jsx`
2. Remove any use of App Bridge APIs (`window.shopify`) from your code
3. Update the config for shopifyApp in `app/shopify.server.js`. Pass `isEmbeddedApp: false`
### OAuth goes into a loop when I change my app's scopes
If you change your app's scopes and authentication goes into a loop and fails with a message from Shopify that it tried too many times, you might have forgotten to update your scopes with Shopify.
To do that, you can run the `deploy` CLI command.
Using yarn:
```shell
yarn deploy
```
Using npm:
```shell
npm run deploy
```
Using pnpm:
```shell
pnpm run deploy
```
### My webhook subscriptions aren't being updated
This template registers webhooks after OAuth completes, using the `afterAuth` hook when calling `shopifyApp`.
The package calls that hook in 2 scenarios:
- After installing the app
- When an access token expires
During normal development, the app won't need to re-authenticate most of the time, so the subscriptions aren't updated.
To force your app to update the subscriptions, you can uninstall and reinstall it in your development store.
That will force the OAuth process and call the `afterAuth` hook.
### Admin created webhook failing HMAC validation
Webhooks subscriptions created in the [Shopify admin](https://help.shopify.com/en/manual/orders/notifications/webhooks) will fail HMAC validation. This is because the webhook payload is not signed with your app's secret key.
Create [webhook subscriptions](https://shopify.dev/docs/api/shopify-app-remix/v1/guide-webhooks) using the `shopifyApp` object instead.
Test your webhooks with the [Shopify CLI](https://shopify.dev/docs/apps/tools/cli/commands#webhook-trigger) or by triggering events manually in the Shopify admin(e.g. Updating the product title to trigger a `PRODUCTS_UPDATE`).
### Incorrect GraphQL Hints
By default the [graphql.vscode-graphql](https://marketplace.visualstudio.com/items?itemName=GraphQL.vscode-graphql) extension for VS Code will assume that GraphQL queries or mutations are for the [Shopify Admin API](https://shopify.dev/docs/api/admin). This is a sensible default, but it may not be true if:
1. You use another Shopify API such as the storefront API.
2. You use a third party GraphQL API.
in this situation, please update the [.graphqlrc.ts](https://github.com/Shopify/shopify-app-template-remix/blob/main/.graphqlrc.ts) config.
### First parameter has member 'readable' that is not a ReadableStream.
See [hosting on Vercel](#hosting-on-vercel).
### Admin object undefined on webhook events triggered by the CLI
When you trigger a webhook event using the Shopify CLI, the `admin` object will be `undefined`. This is because the CLI triggers an event with a valid, but non-existent, shop. The `admin` object is only available when the webhook is triggered by a shop that has installed the app.
Webhooks triggered by the CLI are intended for initial experimentation testing of your webhook configuration. For more information on how to test your webhooks, see the [Shopify CLI documentation](https://shopify.dev/docs/apps/tools/cli/commands#webhook-trigger).
### Using Defer & await for streaming responses
To test [streaming using defer/await](https://remix.run/docs/en/main/guides/streaming) during local development you'll need to use the Shopify CLI slightly differently:
1. First setup ngrok: https://ngrok.com/product/secure-tunnels
2. Create an ngrok tunnel on port 8080: `ngrok http 8080`.
3. Copy the forwarding address. This should be something like: `https://f355-2607-fea8-bb5c-8700-7972-d2b5-3f2b-94ab.ngrok-free.app`
4. In a separate terminal run `yarn shopify app dev --tunnel-url=TUNNEL_URL:8080` replacing `TUNNEL_URL` for the address you copied in step 3.
By default the CLI uses a cloudflare tunnel. Unfortunately it cloudflare tunnels wait for the Response stream to finish, then sends one chunk.
This will not affect production, since tunnels are only for local development.
### Using MongoDB and Prisma
By default this template uses SQLlite as the database. It is recommended to move to a persisted database for production. If you choose to use MongoDB, you will need to make some modifications to the schema and prisma configuration. For more information please see the [Prisma MongoDB documentation](https://www.prisma.io/docs/orm/overview/databases/mongodb).
Alternatively you can use a MongDB database directly with the [MongoDB session storage adapter](https://github.com/Shopify/shopify-app-js/tree/main/packages/apps/session-storage/shopify-app-session-storage-mongodb).
#### Mapping the id field
In MongoDB, an ID must be a single field that defines an @id attribute and a @map("\_id") attribute.
The prisma adapter expects the ID field to be the ID of the session, and not the \_id field of the document.
To make this work you can add a new field to the schema that maps the \_id field to the id field. For more information see the [Prisma documentation](https://www.prisma.io/docs/orm/prisma-schema/data-model/models#defining-an-id-field)
```prisma
model Session {
session_id String @id @default(auto()) @map("_id") @db.ObjectId
id String @unique
...
}
```
#### Error: The "mongodb" provider is not supported with this command
MongoDB does not support the [prisma migrate](https://www.prisma.io/docs/orm/prisma-migrate/understanding-prisma-migrate/overview) command. Instead, you can use the [prisma db push](https://www.prisma.io/docs/orm/reference/prisma-cli-reference#db-push) command and update the `shopify.web.toml` file with the following commands. If you are using MongoDB please see the [Prisma documentation](https://www.prisma.io/docs/orm/overview/databases/mongodb) for more information.
```toml
[commands]
predev = "npx prisma generate && npx prisma db push"
dev = "npm exec remix vite:dev"
```
#### Prisma needs to perform transactions, which requires your mongodb server to be run as a replica set
See the [Prisma documentation](https://www.prisma.io/docs/getting-started/setup-prisma/start-from-scratch/mongodb/connect-your-database-node-mongodb) for connecting to a MongoDB database.
### I want to use Polaris v13.0.0 or higher
Currently, this project is set up to work on node v18.20 or higher. However, `@shopify/polaris` is limited to v12 because v13 can only run on node v20+.
You don't have to make any changes to the code in order to be able to upgrade Polaris to v13, but you'll need to do the following:
- Upgrade your node version to v20.10 or higher.
- Update your `Dockerfile` to pull `FROM node:20-alpine` instead of `node:18-alpine`
## Shopify App Template - Remix
This project was scaffolded using the [Shopify App Template - Remix](https://github.com/Shopify/shopify-app-template-remix). The template is a starting point for building Shopify apps using the [Remix](https://remix.run) framework.
This template comes with the following out-of-the-box functionality:
- [OAuth](https://github.com/Shopify/shopify-app-js/tree/main/packages/shopify-app-remix#authenticating-admin-requests): Installing the app and granting permissions
- [GraphQL Admin API](https://github.com/Shopify/shopify-app-js/tree/main/packages/shopify-app-remix#using-the-shopify-admin-graphql-api): Querying or mutating Shopify admin data
- [REST Admin API](https://github.com/Shopify/shopify-app-js/tree/main/packages/shopify-app-remix#using-the-shopify-admin-rest-api): Resource classes to interact with the API
- [Webhooks](https://github.com/Shopify/shopify-app-js/tree/main/packages/shopify-app-remix#authenticating-webhook-requests): Callbacks sent by Shopify when certain events occur
- [AppBridge](https://shopify.dev/docs/api/app-bridge): This template uses the next generation of the Shopify App Bridge library which works in unison with previous versions.
- [Polaris](https://polaris.shopify.com/): Design system that enables apps to create Shopify-like experiences
## Tech Stack
This project uses [Remix](https://remix.run). The following Shopify tools are also included to ease app development:
- [Shopify App Remix](https://shopify.dev/docs/api/shopify-app-remix) provides authentication and methods for interacting with Shopify APIs.
- [Shopify App Bridge](https://shopify.dev/docs/apps/tools/app-bridge) allows your app to seamlessly integrate your app within Shopify's Admin.
- [Polaris React](https://polaris.shopify.com/) is a powerful design system and component library that helps developers build high quality, consistent experiences for Shopify merchants.
- [Webhooks](https://github.com/Shopify/shopify-app-js/tree/main/packages/shopify-app-remix#authenticating-webhook-requests): Callbacks sent by Shopify when certain events occur
- [Polaris](https://polaris.shopify.com/): Design system that enables apps to create Shopify-like experiences
## Resources
- [Remix Docs](https://remix.run/docs/en/v1)
- [Shopify App Remix](https://shopify.dev/docs/api/shopify-app-remix)
- [Introduction to Shopify apps](https://shopify.dev/docs/apps/getting-started)
- [App authentication](https://shopify.dev/docs/apps/auth)
- [Shopify CLI](https://shopify.dev/docs/apps/tools/cli)
- [App extensions](https://shopify.dev/docs/apps/app-extensions/list)
- [Shopify Functions](https://shopify.dev/docs/api/functions)
- [Getting started with internationalizing your app](https://shopify.dev/docs/apps/best-practices/internationalization/getting-started)
## Trademark Disclaimer
All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this project are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.