{"id":20550241,"url":"https://github.com/aisingapore/sealion-sampler","last_synced_at":"2026-04-20T02:34:26.775Z","repository":{"id":252192860,"uuid":"827273023","full_name":"aisingapore/sealion-sampler","owner":"aisingapore","description":"Lightweight HTML form with Python Flask app and accompanying scripts for swift testing of interactions with SEA-LION family of LLMs.","archived":false,"fork":false,"pushed_at":"2024-08-02T08:11:33.000Z","size":339,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-06T05:43:54.718Z","etag":null,"topics":["flask-application","large-language-models","ollama","tgi"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aisingapore.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-11T10:27:00.000Z","updated_at":"2024-08-08T05:59:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"ab43fdb6-08f1-4741-8de1-5c9361519d66","html_url":"https://github.com/aisingapore/sealion-sampler","commit_stats":null,"previous_names":["aisingapore/sealion-sampler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aisingapore/sealion-sampler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aisingapore%2Fsealion-sampler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aisingapore%2Fsealion-sampler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aisingapore%2Fsealion-sampler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aisingapore%2Fsealion-sampler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aisingapore","download_url":"https://codeload.github.com/aisingapore/sealion-sampler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aisingapore%2Fsealion-sampler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32030157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask-application","large-language-models","ollama","tgi"],"created_at":"2024-11-16T02:24:05.483Z","updated_at":"2026-04-20T02:34:26.729Z","avatar_url":"https://github.com/aisingapore.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SEA-LION Sampler App and Scripts\n\n## Table of Contents\n- [Introduction](#introduction)\n- [Specifications](#specifications)\n- [Quick Startup](#quick-startup)\n    - [Conda Installation](#conda-installation)\n    - [Creation of Virtual Environment](#creation-of-virtual-environment)\n    - [Test prompt via script](#test-prompt-via-script)\n    - [Configuring Server Address/Model Names](#configuring-server-addressmodel-names)\n    - [Retrieving Ollama Model](#retrieving-ollama-model)\n    - [Starting up the Flask App](#starting-up-the-flask-app)\n    - [Running the Flask App](#running-the-flask-app)\n- [For Further Exploration](#for-further-exploration)\n\n## Introduction\nThis repository contains basic scripts for running the SEA-LION base LLM locally, as well as sending requests to the SEA-LION Instruct models running on Ollama/TGI servers. It has been tested with the [SEA-LION-3B](https://huggingface.co/aisingapore/sea-lion-3b) base LLM, as well as quantized GGUF files for 7b/[8b instruct models](https://huggingface.co/aisingapore/llama3-8b-cpt-sea-lionv2-instruct-gguf) on the server side. The scripts available facilitate running prompts in the terminal, as well as running text completion/input, question and answer, and translation via a Flask app.\u003cbr\u003e\n\nDo check out the range of SEA-LION models available at https://huggingface.co/aisingapore/\u003cbr\u003e\nAlso available on Ollama at https://ollama.com/aisingapore\n\n## Specifications\nThe scripts provided have been tested in the following environments:\n### MacBook Pro\n- Processor: Apple M3 Max\n- Memory: 64GB\n- OS: MacOS Sonoma version 14.5\n- Chip Architecture: ARM64\n\n### MacBook Pro\n- Processor: 2.3GHz Quad-Core Intel Core i7\n- Memory: 32GB\n- OS: MacOS Sonoma version 14.5\n- Chip Architecture: x86-64\n\n### Debian GNU/Linux 11 (Bullseye) VM\n- Memory: 16GB\n- OS: Debian GNU/Linux 11 (Bullseye)\n- Chip Architecture: x86-64\n\n## Quick Startup\n\n### Conda Installation\n1. Download Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html).\n2. Follow the installation instructions for your operating system.\n\n### Creation of Virtual Environment\nOpen your terminal and execute the following commands:\n#### To create with `sealion_env.yml`\n```sh\n# Create a new conda environment\nconda env create -f sealion_env.yml\n\n# Activate the environment\nconda activate sealion_env\n```\n\n#### To create with `requirements.txt`\n```sh\n# Create a new conda environment\nconda create -n sealion_env python=3.12\n\n# Activate the environment\nconda activate sealion_env\n```\nEnsure you have requirements.txt in your project directory. Run:\n\n```sh\n# Install required packages\npip install -r requirements.txt\n```\n### Test prompt via script\nTo test the model with a simple prompt, run the following Python script via terminal:\n\n```sh\npython src.sealion_3b_prompt.py\n```\nIt will prompt the model with the string `The sea lion is a` and should return a continuation of this sentence.\n\n### Configuring Server Address/Model Names\nReferring to `.env.example` as an example, create a `.env` file in the same repository folder. \u003cbr\u003e\n[Model Selection](#model-selection) section will advise on how to configure the variables used.\n### Retrieving Ollama Model\nBefore running the Flask app, the model should be loaded on Ollama first.\u003cbr\u003e\nAfter [installation of Ollama](https://ollama.com/download), proceed to run the following command:\n```sh\nollama pull aisingapore/llama3-8b-cpt-sea-lionv2-instruct\n```\nFor specific quantizations, do add the available tag. For example:\n```sh\nollama pull aisingapore/llama3-8b-cpt-sea-lionv2-instruct:q4_k_m\n```\nYou should then see it when you run `ollama list`\n```sh\nollama list\n\u003e NAME                                                    ID              SIZE    MODIFIED\n\u003e aisingapore/llama3-8b-cpt-sea-lionv2-instruct:latest    648d5f2d7bbe    4.9 GB  23 hours ago\n\u003e aisingapore/llama3-8b-cpt-sea-lionv2-instruct:q4_k_m    648d5f2d7bbe    4.9 GB  23 hours ago\n```\nMake sure the full model name is updated under `OLL_API_MODEL` in your `.env` file.\n```\nOLL_API_MODEL=aisingapore/llama3-8b-cpt-sea-lionv2-instruct\n```\n\u003ci\u003eNote: Ollama should be running before starting up the Flask app, if not the server would not be up. Running any of the commands should suffice in starting up Ollama, e.g. `ollama list`\u003c/i\u003e\n\n### Starting up the Flask App\nTo start up the Flask app:\n\n```sh\nflask -A src.sealion_app run\n```\nTo make the app accessible from other machines by exposing ports:\n```sh\nflask -A src.sealion_app run --host '0.0.0.0'\n```\n\u003ci\u003eNote: Exposing your Flask app to '0.0.0.0' makes it accessible from any device on the network, which can introduce security risks. Ensure you have proper security measures in place.\u003c/i\u003e\u003cbr\u003e\n\nAlternatively: Running via Python script (Will run in debug mode and expose ports):\n```sh\npython src.sealion_app.py\n```\nOnce the terminal returns the following, you should proceed to access the app via [port 5000](http://127.0.0.1:5000):\n```sh\n * Serving Flask app 'src.sealion_app'\n * Debug mode: off\nWARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.\n * Running on http://127.0.0.1:5000\nPress CTRL+C to quit\n127.0.0.1 - - [DD/MMM/YYYY HH:MM:SS] \"GET / HTTP/1.1\" 200 -\n```\n#### Troubleshooting\nIn the event that the port is occupied, it may return such an error:\n```\nAddress already in use\nPort 5000 is in use by another program. Either identify and stop that program, or start the server with a different port.\nOn macOS, try disabling the 'AirPlay Receiver' service from System Preferences -\u003e General -\u003e AirDrop \u0026 Handoff.\n```\nTry assigning the Flask app to a different port instead:\n```sh\nflask -A src.sealion_app run --host '0.0.0.0' --port 5050\n```\nIf running via Python script, do update the port (located at the end of script):\n```python\nif __name__ == \"__main__\":\n    app.run(host=\"0.0.0.0\", port=5050, debug=False)\n```\n### Running the Flask App\n#### Model selection\n![dropdown list for model selection](graphics/modelselection.png)\u003cbr\u003e\nThere are 3 types of models made available for selection:\n- \u003cb\u003eLocally run model\u003c/b\u003e: Run directly from the Python script via HuggingFace Transformers, currently configured to run [`sea-lion-3b`](https://huggingface.co/aisingapore/sea-lion-3b) model under `LOCAL_MODEL` in `env.example`. Able to run on CPU.\n- \u003cb\u003eModel on Ollama server\u003c/b\u003e: This selection is configured to work with a model running on Ollama server. `OLL_API_URL` in `.env.example` is currently set to [default local address](https://github.com/ollama/ollama/blob/main/docs/api.md#request). `OLL_API_MODEL` should be set to the model name as per what was set in the Ollama [Modelfile](https://github.com/ollama/ollama/blob/0e4d653687f81db40622e287a846245b319f1fbe/docs/modelfile.md#basic-modelfile).\n- \u003cb\u003eMultiple models on online server (TGI)\u003c/b\u003e: This selection is configured to work with multiple models running on a TGI server with API Key authentication. `TGI_API_URL` should be set to the endpoint URL. `TGI_API_KEY` should be set to the API Key required, remove if no authentication used, it will default to `None`. Update the respective model names used in the server to `TGI_SEALION` and `TGI_LLAMA`. If only one model is used, remove `TGI_LLAMA`, it will default to `None`.\n\n#### Task type selection\n![dropdown list for task type selection](graphics/tasktype.png)\u003cbr\u003e\nCurrently there are three functions available, which prompt the base LLM differently:\n- Text Generation/Input: No additional template, LLM proceeds with text generation, continuing from where the input prompt ends. For base models, it should return the initial prompt, continued with the generated text. For chat/instruct-tuned models, they should return a response to the prompt.\n- Question and Answer: Prompt is input to a `Question: {prompt} Answer:` template\n- Translation: An additional `Language` option will be provided (Default: `English`). Prompt is input to a `'{prompt}' In {language}, this translates to:` template. \u003cbr\u003e\n![dropdown list for language selection](graphics/language.png)\n\n\nThere are two parameters provided for the app user to toggle:\n- Temperature (Default: `0.7`, Range:`0.0-1.0`): The value used to modulate the next token probabilities.\n- Max New Tokens (Default: `40` for local, `128` for server): The maximum numbers of tokens to generate, also known as num_predict in some frameworks.\n\n## For Further Exploration\nTo run the script for prompting via LangChain:\n\n```sh\npython src.sealion_3b_langchain.py\n```\nThis script is currently a work in progress, with different parts commented out to test different ways of prompting the SEA-LION base model. Feel free to experiment with the script!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faisingapore%2Fsealion-sampler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faisingapore%2Fsealion-sampler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faisingapore%2Fsealion-sampler/lists"}