{"id":15792858,"url":"https://github.com/atlasacademy/fgo-game-data-api","last_synced_at":"2025-04-10T00:19:01.163Z","repository":{"id":37612566,"uuid":"264085065","full_name":"atlasacademy/fgo-game-data-api","owner":"atlasacademy","description":"HTTP API for FGO game data. Transform the raw game data into something a bit more manageable.","archived":false,"fork":false,"pushed_at":"2024-04-14T02:14:43.000Z","size":16015,"stargazers_count":73,"open_issues_count":11,"forks_count":8,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-04-14T04:11:05.516Z","etag":null,"topics":["fastapi","fate-grand-order","fgo","hacktoberfest","python"],"latest_commit_sha":null,"homepage":"https://api.atlasacademy.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atlasacademy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-15T03:19:43.000Z","updated_at":"2024-04-15T06:04:06.496Z","dependencies_parsed_at":"2024-01-20T04:19:56.301Z","dependency_job_id":"56a7909a-ac65-476b-8981-b776109f1521","html_url":"https://github.com/atlasacademy/fgo-game-data-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlasacademy%2Ffgo-game-data-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlasacademy%2Ffgo-game-data-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlasacademy%2Ffgo-game-data-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlasacademy%2Ffgo-game-data-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atlasacademy","download_url":"https://codeload.github.com/atlasacademy/fgo-game-data-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131403,"owners_count":21052835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","fate-grand-order","fgo","hacktoberfest","python"],"created_at":"2024-10-04T23:05:56.747Z","updated_at":"2025-04-10T00:19:01.134Z","avatar_url":"https://github.com/atlasacademy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FGO game data API \u003c!-- omit in toc --\u003e\n\nHTTP API for FGO game data. Transform the raw game data into something a bit more manageable.\n\nView the API documentation here: https://api.atlasacademy.io.\n\nIf you are looking for only the type definitions and enums. You can download the [`fgo-api-types` package](https://pypi.org/project/fgo-api-types/).\n\n- [Configuration](#configuration)\n  - [Environment Variables](#environment-variables)\n  - [Secrets](#secrets)\n- [Development environment set up](#development-environment-set-up)\n- [Run the API server](#run-the-api-server)\n- [Architecture](#architecture)\n- [Linting](#linting)\n- [Formatting](#formatting)\n- [Dependencies](#dependencies)\n- [Testing](#testing)\n- [Helper scripts](#helper-scripts)\n  - [`format.ps1`](#formatps1)\n  - [`extract_enums.py`](#extract_enumspy)\n  - [`update_ce_translation.py`](#update_ce_translationpy)\n  - [`load_rayshift_quest_list.py`](#load_rayshift_quest_listpy)\n  - [`get_test_data.py`](#get_test_datapy)\n  - [`niceexport.py`](#niceexportpy)\n\n### Configuration\n\nList of configuration variables for the main app. You can make a `config.json` file at the project root with the settings. Check the [`config.sample.json`](/config.sample.json) file for an example.\n\n**Required variables**\n- `DATA`: A JSON object with keys being region and values being gamedata's folder location and Postgresql DSN. Not all regions are required in the object. Any combination of regions is accepted.\n- `REDISDSN`: Redis DSN to a Redis server for caching.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eOptional variables\u003c/b\u003e (click to show)\u003c/summary\u003e\n\n- `REDIS_PREFIX`: default to `fgoapi`. Prefix for redis keys.\n- `CLEAR_REDIS_CACHE`: default to `True`. If set, will clear the redis cache on start and when the webhook above is used.\n- `RAYSHIFT_API_KEY`: default to `\"\"`. Rayshift.io API key to pull quest data.\n- `RAYSHIFT_API_URL`: default to https://rayshift.io/api/v1/. Rayshift.io API URL.\n- `QUEST_CACHE_LENGTH`: default to `3600`. How long to cache the quest and war endpoints in seconds. Because the rayshift data is updated continously, web and quest endpoints have lower cache time.\n- `DB_POOL_SIZE`: defaults to 3. Default pool size for SQLAlchemy connection pool. https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.QueuePool.params.pool_size\n- `DB_MAX_OVERFLOW`: defaults to 10. Max overflow for SQLAlchemy connection pool. https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.QueuePool.params.max_overflow\n- `WRITE_POSTGRES_DATA`: default to `True`. Overwrite the data in PostgreSQL when importing.\n- `WRITE_REDIS_DATA`: default to `True`. Overwrite the data in Redis when importing.\n- `ASSET_URL`: defaults to https://assets.atlasacademy.io/GameData/. Base URL for the game assets.\n- `OPENAPI_URL`: default to `None`. Set the server URL in the openapi schema export.\n- `EXPORT_ALL_NICE`: default to `False`. If set to `True`, at start the app will generate nice data of all servant and CE and serve them at the `/export` endpoint. It's recommended to serve the files in the `/export` folder using nginx or equivalent webserver to lighten the load on the API server.\n- `DOCUMENTATION_ALL_NICE`: default to `False`. If set to `True`, there will be links to the exported all nice files in the documentation.\n- `GITHUB_WEBHOOK_SECRET`: default to `\"\"`. If set, will add a webhook location at `/GITHUB_WEBHOOK_SECRET/update` that will pull and update the game data. If it's not set, the endpoint is not created.\n- `GITHUB_WEBHOOK_GIT_PULL`: default to `False`. If set, the app will do `git pull` on the gamedata repos when the webhook above is used.\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eOther ways to set the variables\u003c/b\u003e (click to show)\u003c/summary\u003e\n\n#### Environment Variables\n\nThe variables can also be set as environment variables.\n\n#### Secrets\n\nSecret variables can be put in the [secrets](secrets/) folder instead of being supplied as environment variable:\n```\n\u003e cat .\\secrets\\rayshift_api_key\neca334a9-3289-4ad7-9b92-1ec2bbc3fc19\n\u003e cat .\\secrets\\redisdsn\nredis://localhost\n```\n\nSettings at a higher level will override the settings at a lower level.\n1. Secrets variable\n2. Enviornment variable\n3. `.env` file\n4. `config.json`\n\u003c/details\u003e\n\n### Development environment set up\n\nMake sure poetry is installed: https://python-poetry.org/docs/#installation.\n\nDocker is recommended to set up the Postgres and redis servers but those can be set up manually as well. Postgres needs the [PGroonga](https://pgroonga.github.io/install/) extension installed.\n\n```sh\ngit clone --depth 1 --branch JP https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-jp\ngit clone --depth 1 --branch NA https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-na\n\n# It's not neccecary to clone the other regions.\ngit clone --depth 1 --branch CN https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-cn\ngit clone --depth 1 --branch KR https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-kr\ngit clone --depth 1 --branch TW https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-tw\n\ngit clone https://github.com/atlasacademy/fgo-game-data-api.git\ncd fgo-game-data-api\n\n# If you didn't clone other game data regions, remove them from the data field in config.json,\n# and the services key in docker-compose.sample.yaml\ncp config.sample.json config.json\ncp docker-compose.sample.yml docker-compose.yml\n\ndocker-compose up -d\n\n# Make sure you have the build prerequisites for psycopg2 installed if you are installing on Linux or macOS.\n# https://www.psycopg.org/docs/install.html#build-prerequisites\n# Debian/Ubuntu: sudo apt install libpq-dev python3-dev\n# CentOS: sudo yum install python-devel postgresql-devel\npoetry install\npoetry shell\n```\n\n### Run the API server\n\nRun at the project root to start the API server:\n\n```\n\u003e uvicorn app.main:app --reload --log-level debug --reload-dir app\n\nINFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\nINFO:     Started reloader process [16680] using watchgod\nINFO      fgoapi: Loading game data …\nINFO      fgoapi: Loaded game data in 15.14s.\nINFO:     Started server process [33312]\nINFO:     Waiting for application startup.\nINFO:     Application startup complete.\nDEBUG     fgoapi: Processed in 0.21ms.\nINFO:     127.0.0.1:56759 - \"GET / HTTP/1.1\" 307 Temporary Redirect\nDEBUG     fgoapi: Processed in 0.24ms.\nINFO:     127.0.0.1:56759 - \"GET /rapidoc HTTP/1.1\" 200 OK\n```\n\nGo to http://127.0.0.1:8000 for the API documentation.\n\nTips:\n- Change `write_postgres_data` to `false` after the first run to speed up reloading if it's not needed (schema doesn't change or data hasn't changed).\n\n### Architecture\n\n- `main.py`: Main entrypoint of the application.\n- `routers/`: Routers to deal with incoming requests. The routers call functions from `core` to get the response data.\n- `core/`: Build response data. Get raw data from either `db/helpers/` or `redis/helpers/`.\n- `data/`: Import translation data into memory. Preprocess data to be imported into db and redis.\n- `db/`: DB stuffs.\n  - `db/helpers/`: Functions to be used by `core` to get data from Postgres.\n- `redis/`: Redis stuffs.\n  - `redis/helpers/`: Functions to be used by `core` to get data from Redis.\n- `schemas/`: Response Pydantic models.\n- `models/`: SQLAlchemy Core Tables.\n\n### [Linting](scripts/lint.ps1)\n\n[ruff](https://github.com/charliermarsh/ruff) and [mypy](https://mypy.readthedocs.io/en/stable/) are used to lint the code. ruff's configuration and mypy's configuration are in [pyproject.toml](pyproject.toml).\n\n```\n./scripts/lint.ps1\n```\n\n### [Formatting](scripts/format.ps1)\n\n[isort](https://pycqa.github.io/isort/) and [black](https://black.readthedocs.io/en/stable/) are used to format the code. isort's configuration is in [pyproject.toml](pyproject.toml) and black uses default settings. [prettier](https://prettier.io/docs/en/) is used to format the json files.\n\n```\n./scripts/format.ps1\n```\n\n### Dependencies\n\nUse [poetry](https://python-poetry.org/docs/) to manage the dependencies. Run `poetry export` after adding a production dependency.\n\n```\npoetry export -f requirements.txt -o requirements.txt\n```\n\n### [Testing](scripts/test.ps1)\n\nRun pytest at project root to run the tests or use `coverage` to get coverage statistics.\n\n```\n./scripts/test.ps1\n```\n\n### Helper scripts\n\n#### [`format.ps1`](scripts/format.ps1)\n\nFormat all files with black, isort and prettier.\n\n```\n./scripts/format.ps1\n```\n\n#### [`extract_enums.py`](scripts/extract_enums.py)\n\nTake the `dump.cs` generated by [Il2CppDumper](https://github.com/Perfare/Il2CppDumper) and write the [`gameenums.py`](app/data/gameenums.py) file.\n\n```\npython scripts/extract_enums.py dump.cs_path app/schemas/gameenums.py\n```\n\n#### [`update_ce_translation.py`](scripts/update_ce_translation.py)\n\nUpdate translation files with new NA CEs translations. `--jp-master` and `--na-master` arguments are not needed if environment variables `JP_GAMEDATA` and `NA_GAMEDATA` are set, added to the `.env` file or set in `config.json`.\n\n```\npython scripts/update_ce_translation.py --jp-master jp_master_path --na-master na_master_path\n```\n\n#### [`load_rayshift_quest_list.py`](scripts/load_rayshift_quest_list.py)\n\nUpdate the `rayshiftQuest` tables with the list of available quests from Rayshift. This script should be run periodically to update the `rayshiftQuest` list.\n\n```\npython -m scripts.load_rayshift_quest_list\n```\n\n#### [`get_test_data.py`](tests/get_test_data.py)\n\nRun this script when the master data changed to update the tests or when new tests are added.\n\n```\npython -m tests.get_test_data --raw --nice --basic\n```\n\n#### [`niceexport.py`](export/niceexport.py)\n\nRun this script to update the static export files in `export/` folder.\n\n```\npython -m export.niceexport\n./scripts/format.ps1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatlasacademy%2Ffgo-game-data-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatlasacademy%2Ffgo-game-data-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatlasacademy%2Ffgo-game-data-api/lists"}