{"id":24617462,"url":"https://github.com/snaacky/theindex","last_synced_at":"2025-04-05T10:09:49.857Z","repository":{"id":38400229,"uuid":"306121706","full_name":"Snaacky/theindex","owner":"Snaacky","description":"The frontend, editor panel, and API of TheIndex.moe","archived":false,"fork":false,"pushed_at":"2025-03-28T22:16:31.000Z","size":27511,"stargazers_count":134,"open_issues_count":4,"forks_count":27,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-29T09:09:53.853Z","etag":null,"topics":["index","tables","website"],"latest_commit_sha":null,"homepage":"https://theindex.moe","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Snaacky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-21T19:04:21.000Z","updated_at":"2025-03-20T15:41:46.000Z","dependencies_parsed_at":"2023-02-08T04:46:30.221Z","dependency_job_id":"1afc9de5-5490-442c-a278-c130292624ba","html_url":"https://github.com/Snaacky/theindex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snaacky%2Ftheindex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snaacky%2Ftheindex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snaacky%2Ftheindex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snaacky%2Ftheindex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Snaacky","download_url":"https://codeload.github.com/Snaacky/theindex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318745,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["index","tables","website"],"created_at":"2025-01-24T23:39:19.078Z","updated_at":"2025-04-05T10:09:49.840Z","avatar_url":"https://github.com/Snaacky.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Website](https://img.shields.io/website?down_message=offline\u0026label=theindex.moe\u0026up_message=online\u0026url=https%3A%2F%2Ftheindex.moe)](https://theindex.moe)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/Snaacky/theindex/publish-docker-image.yml?logo=github)](https://github.com/Snaacky/theindex)\n[![CodeFactor](https://www.codefactor.io/repository/github/snaacky/theindex/badge)](https://www.codefactor.io/repository/github/snaacky/theindex)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/a91c8d3ec420474cbac1a524073c83ee)](https://www.codacy.com/gh/Snaacky/theindex/dashboard)\n[![Discord](https://img.shields.io/discord/974468300304171038?label=Discord\u0026logo=discord)](https://discord.gg/snackbox)\n\n# TheIndex\n\nThe frontend, editor panel, and API of the TheIndex.\n\n## Getting started\n\nThe easiest way is to get started is by using Docker Compose. You need to copy\nthe [`docker-compose.yml`](docker-compose.yml) and [`example.env`](example.env) file from GitHub. Rename `example.env`\nto `.env` and adjust the environment variables as you need to. with the following command:\n\n```shell\ndocker-compose up -d\n```\n\nYou'll need to change `\u003chost-port\u003e` to your port of choice. The web-server is not secured via SSL/TLS, it is in your\nresponsibility to put a reverse proxy in front of this container. When you run the image for the first time, don't\nforget to set your own [discord-id](https://discord.com/developers/docs/resources/user) in the\nenv `SETUP_WHITELIST_DISCORD_ID` to be able to login and edit. Once your container has set up itself once, you can\nremove the env variable from your setup.\n\n## Database\n\nWe use [mongodb](https://github.com/mongodb/mongo) as our database server. You can deploy your own mongo setup as HA\nservice or just a simple single docker container via e.g.:\n\n\u003e Note: The database will start empty, you have to fill the data yourself.\n\n```shell\ndocker run -d \\\n    --name mongo \\\n    -v ./db:/data/db \\\n    mongo\n```\n\nFor development or testing purposes it is highly recommended to\nuse [mongo-express](https://github.com/mongo-express/mongo-express) for accessing, viewing and editing the current state\nof the database. If you make it publicly accessible, **don't forget** to secure it with _login credentials_.\n\nTo simply spin up a mongo-express docker container, run:\n\n```shell\ndocker run -d \\\n    --name mongo-express \\\n    -p 8081:8081 \\\n    mongo-express\n```\n\nYou can also take a look at our provided [`docker-compose`](docker-compose.yml) file on how to set it up.\n\n## Cache-DB\n\nTo increase performance we use [redis](https://redis.io/) to cache results from the mongoDB. The cache is being\nautopopulated on cache-misses.\n\nYou can create a new instance with docker by running:\n\n```shell\ndocker run -d \\\n    --name redis \\\n    redis\n```\n\nThe redis db is already included in the example [`docker-compose`](docker-compose.yml) file\n\n## Updating container image\n\nWarning: be aware, that we do not offer any kind of official support and every update may be with breaking changes. Be\nsure to make backups before you update\n\nTo get the newest version of the container image, you will need to run:\n\n```shell\ndocker pull ghcr.io/snaacky/theindex:latest\n```\n\nAfterwards, you will need to stop and remove your current running instance and start it again.\n\n## Parameters\n\nHere is a collection of the possible environment variables with their default values you should set in your `.env` file:\n\n| Parameter                    | Function                                                                                             | Default                              |\n| ---------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------ |\n| `NEXT_PUBLIC_SITE_NAME`      | The name of your site                                                                                | `\"The Index\"`                        |\n| `NEXT_PUBLIC_DOMAIN`         | Your domain or IP, remove trailing slash                                                             | `\"https://theindex.moe\"`             |\n| `DATABASE_URL`               | Take a look at [mongodb docs](https://docs.mongodb.com/manual/reference/connection-string/)          | `\"mongodb://mongo:27017/index\"`      |\n| `CACHE_URL`                  | Connection string for the redis cache database                                                       | `\"redis://redis:6379\"`               |\n| `CHROME_URL`                 | WebSocket URL to a running chrome instance                                                           | `\"ws://chrome:3300\"`                 |\n| `AUDIT_WEBHOOK`              | WebHook-URL for audit-log, leave empty to disable support                                            | `\"\"`                                 |\n| `AUTH_DISCORD_ID`            | Discord OAuth2 client ID                                                                             | `\"your_discord_oauth_client_id\"`     |\n| `AUTH_DISCORD_SECRET`        | Discord OAuth2 client secret                                                                         | `\"your_discord_oauth_client_secret\"` |\n| `DISCORD_BOT_TOKEN`          | Required to access BOT resources                                                                     | `\"your_discord_bot_token\"`           |\n| `SETUP_WHITELIST_DISCORD_ID` | If you need help getting your id, check out this [guide](https://wiki.discord.id/obtain-ids/desktop) | `\"your_discord_id\"`                  |\n\nAnd the following env variables are only needed when you are in dev mode and debugging the db\n\n| Parameter                      | Function                                                                 | Default        |\n| ------------------------------ | ------------------------------------------------------------------------ | -------------- |\n| `ME_CONFIG_BASICAUTH_USERNAME` | [mongo-express](https://github.com/mongo-express/mongo-express) username | \"admin\"        |\n| `ME_CONFIG_BASICAUTH_PASSWORD` | [mongo-express](https://github.com/mongo-express/mongo-express) password | \"SUPER_SECRET\" |\n\nIf you want to verify how the docker-compose file fills the envs in, use `docker-compose config`\n\n## Getting started to code\n\n### Setup around the web app\n\n1. Getting started isn't that straight forward. You will need to have installed the latest version\n   of [docker with docker-compose](https://docs.docker.com/get-docker/) on your machine.\n\n2. Start by cloning the repo via a graphical git client (highly recommended) or use the cli via\n\n```shell\ngit clone https://github.com/snaacky/theindex\n```\n\n3. Copy the [`example.env`](example.env) file to `.env`.\n\n### How to fill the .env\n\n- Replace `NEXT_PUBLIC_DOMAIN` and `NEXTAUTH_URL` with `http://localhost:3000`\n- Generate a random strong string of at least 32 characters and use it for `NEXTAUTH_SECRET`. You can use generators\n  e.g. [1Password](https://1password.com/password-generator/) or create it yourself.\n- Change `DATABASE_URL`, `CACHE_URL` and `CHROME_URL` to use localhost instead of `mongo`, `redis` and `chrome` for\n  example: `mongodb://mongo:27017` -\u003e becomes `mongodb://localhost:27017`\n- Go to `https://discord.com/developers` -\u003e Create a new Application -\u003e Go into `Auth2` inside your Application panel -\u003e\n  Copy the `CLIENT ID` and `CLIENT SECRET` into the `.env` file.\n- In `Redirects` in `Auth2` copy and paste the following URL needed to verify your Discord\n  login `http://localhost:3000/api/auth/callback/discord`.\n- Fill `SETUP_WHITELIST_DISCORD_ID` with your Discord ID to have an admin account when you login.\n- Generate again another random strong string of at least 32 characters and use it for `MEILI_MASTER_KEY`. You can use\n  generators e.g. [1Password](https://1password.com/password-generator/) or create it yourself.\n- Lastly if you want to change the password for mongo-express, you can do so with the last two variables, usually though\n  none of us do while developing locally.\n\n4. Add the following ports to the images in the [`docker-compose`](docker-compose.yml) file:\n\n| service  | port-mapping  |\n| :------- | ------------- |\n| `mongo`  | `27017:27017` |\n| `redis`  | `6379:6379`   |\n| `meili`  | `7700:7700`   |\n| `chrome` | `3300:3000`   |\n\nAs an example, the setup for `mongo` should look similar to this:\n\n```yml\nmongo:\n  image: mongo\n  container_name: index-db\n  restart: unless-stopped\n  ports:\n    - '27017:27017'\n  volumes:\n    - ./db:/data/db\n```\n\n5. Now run the command to start all the needed backend processes\n\n```shell\ndocker-compose up -d mongo redis meili chrome mongo-express\n```\n\nAlternatively you can also just comment or remove the index service and run the command\n\n```shell\ndocker-compose up -d\n```\n\n### Web service\n\nTo start coding on the frontend, you will need to make sure, you have the latest version of\n[node.js](https://nodejs.org/) correctly installed. To install all the required dependencies run once:\n\n```shell\nbun install\n```\n\n\u003e Note: We decided to stick with npm instead of yarn to manage dependencies.\n\nYou should now have a folder called `node_modules`, which contains all the dependencies we need. We use\n[Next.js](https://nextjs.org) as framework for our [React](https://reactjs.org) web service. To test the web service you\nwill have to run a db server in the background and start the frontend via:\n\n```shell\nbun run dev\n```\n\nAfter compiling you can open [http://localhost:3000](http://localhost:3000) in your browser of choice and see the\nrunning web application.\n\nAs we use [Next.js](https://nextjs.org), the frontend supports hot reloading, so you can just leave the page open, while\nyou modify the code and see the changes on the fly in your browser.\n\n### Docker image\n\nTo create a ready-made docker image, just run:\n\n```shell\ndocker build . -t index\n```\n\nYou now have a local image with tag `index` that contains a distributable version of the code which can now be run.\n\n### Auto-formatter\n\nWe use [prettier](https://prettier.io) to ensure a consistent code style across all participants. You can simply\nauto-format everything with e.g. running the command\n\n```shell\nbunx prettier --write .\n```\n\n## Design of the web app\n\n### ISR/SSR\n\nWhere possible we use [ISR](https://vercel.com/docs/next.js/incremental-static-regeneration) to pre generate all\npublicly accessible pages for caching by CDNs or proxies while validating and fetching new data\nwith [SWR](https://web.dev/stale-while-revalidate/) requesting our own api.\n\n### API\n\nWe serve every api request over the endpoint `/api`, the corresponding code can be viewed at [pages/api](pages/api).\n\n- `/api/auth` is reserved for [NextAuth.js](https://next-auth.js.org/).\n- `/api/edit/...` requires a logged-in user and usually (at least) editor rights and is for modifying or creating new\n  content. The `_id` keyword `_new` is reserved for creating new content.\n- `/api/delete/...` requires a logged-in user and usually (at least) editor rights and is for deleting content.\n- `/api/[content]s` are public endpoints for fetching a list of all items of a certain content.\n- `/api/[content]/...` are public endpoints for fetching information about a specific content.\n\n### Page restrictions\n\nEvery page request first has to go through [\\_app.ts](pages/_app.ts), where a basic layout is being applied and if a\npage has an `auth` property, it also validates whether the user can access the given page. Valid auth attributes are:\n\n- `auth` itself is `null` or `typeof auth === \"undefined\"`, no restrictions. This seems to be a public page.\n- `requireLogin`, not really needed, but set it for logic reasons. User must be logged-in.\n- `requireAdmin`, only a user with admin rights can access this page.\n- `requireEditor`, only editors can view this page.\n\n## Contribution\n\nPull-requests are always welcome, but may not be always merged as it has to be in align with our idea of the index. If\nyou want a certain feature or have an idea, you can always open a feature request\nin [Issues](https://github.com/snaacky/theindex/issues/new?assignees=\u0026labels=enhancement\u0026template=feature_request.md\u0026title=%5BFEAT%5D)\nor report it on our [Discord](https://discord.gg/snackbox) in `#index` to be discussed. If it is not bad, in align with\nour ideas, and we find some time, we will certainly implement your requested feature (sometime...).\n\n### Things we use\n\n- [next.js](https://nextjs.org/) as Webserver\n- [React](https://reactjs.org/) as JS framework\n- Styles from [bootstrap](https://getbootstrap.com/)\n- Icons from [Font Awesome](https://fontawesome.com)\n- We bake everything into a [docker image](https://www.docker.com/)\n- Running on [node js](https://nodejs.org/)\n- User-authentication via [NextAuth.js](https://next-auth.js.org/)\n- [mongodb](https://www.mongodb.com) as database backend\n- [redis](https://redis.io) as cache\n- [puppeteer](https://github.com/puppeteer/puppeteer) to create screenshots of websites\n- [browserless/chrome](https://github.com/browserless/chrome) to run our chrome instance\n- [react-in-viewport](https://github.com/roderickhsiao/react-in-viewport) for checking if components are visible or not\n- [react-draggable](https://github.com/react-grid-layout/react-draggable) for dragging components by the end user\n- [react-toastify](https://github.com/fkhadra/react-toastify) for handling notifications\n- [iso-639-3](https://www.npmjs.com/package/iso-639-3) for the languages\n\nAnd most importantly:\n\n\u003e The help of our awesome community :3\n\n### Technical debts\n\n- We use JS instead of TS (currently in migration)\n- Unify `Editor`-views\n- Unify db insert and updates to the format of `func(_id, dataObject)` and update only as needed, GraphQL would be\n  nice...\n- Move to `add`, `remove` api instead of having to send whole arrays to update lists -\u003e GraphQL?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnaacky%2Ftheindex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnaacky%2Ftheindex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnaacky%2Ftheindex/lists"}