{"id":20710027,"url":"https://github.com/oxylabs/python-script-service-guide","last_synced_at":"2025-10-24T01:33:55.467Z","repository":{"id":134336662,"uuid":"545426904","full_name":"oxylabs/python-script-service-guide","owner":"oxylabs","description":"A guide on running a Python script as a service on Windows \u0026 Linux.","archived":false,"fork":false,"pushed_at":"2024-04-19T10:51:22.000Z","size":15,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-17T21:07:25.938Z","etag":null,"topics":["amazon-scraper-python","github-python","python-ecommerce","python-image-scraper","python-script","python-web-crawler","python3","scraper-python","scraping","serp-api","serp-api-python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-04T10:55:30.000Z","updated_at":"2023-10-21T09:58:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"5773fbd6-0323-4a68-8ee3-c41e6e7570af","html_url":"https://github.com/oxylabs/python-script-service-guide","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpython-script-service-guide","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpython-script-service-guide/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpython-script-service-guide/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fpython-script-service-guide/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/python-script-service-guide/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242980784,"owners_count":20216285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-scraper-python","github-python","python-ecommerce","python-image-scraper","python-script","python-web-crawler","python3","scraper-python","scraping","serp-api","serp-api-python"],"created_at":"2024-11-17T02:09:35.222Z","updated_at":"2025-10-24T01:33:55.461Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","readme":"# Python Script As A Service\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=python-script-service-guide-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=python\u0026color=brightgreen\" /\u003e](https://github.com/topics/python)\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=service\u0026color=yellow\" /\u003e](https://github.com/topics/service)\n[\u003cimg src=\"https://img.shields.io/static/v1?label=\u0026message=Web%20Scraping\u0026color=important\" /\u003e](https://github.com/topics/web-scraping)\n\n- [Setting Up](#setting-up)\n- [Create A Systemd Service](#create-a-systemd-service)\n- [Create A Windows Service](#create-a-windows-service)\n- [Easier Windows Service Using NSSM](#easier-windows-service-using-nssm)\n\nA service (also known as a \"daemon\") is a process that performs tasks in the background and responds to system events.\n\nServices can be written using any language. We use Python in these examples as it is one of the most versatile languages out there.\n\nFor more information, be sure to read [our blogpost on the subject](https://oxylabs.io/blog/python-script-service-guide).\n\n## Setting Up\n\nTo run any of the examples, you will need Python 3. We also recommend [using a virtual environment](https://docs.python.org/3/library/venv.html).\n\n```bash\npython3 -m venv venv\n```\n\n## Create A Systemd Service\n\nFirst, create a script that scrapes a website. Make sure the script also handles OS signals so that it exits gracefully.\n\n**linux_scrape.py:**\n```python\nimport json\nimport re\nimport signal\nfrom pathlib import Path\n\nimport requests\nfrom bs4 import BeautifulSoup\n\nclass SignalHandler:\n    shutdown_requested = False\n\n    def __init__(self):\n        signal.signal(signal.SIGINT, self.request_shutdown)\n        signal.signal(signal.SIGTERM, self.request_shutdown)\n\n    def request_shutdown(self, *args):\n        print('Request to shutdown received, stopping')\n        self.shutdown_requested = True\n\n    def can_run(self):\n        return not self.shutdown_requested\n\n\nsignal_handler = SignalHandler()\nurls = [\n    'https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html',\n    'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html',\n    'https://books.toscrape.com/catalogue/sharp-objects_997/index.html',\n]\n\nindex = 0\nwhile signal_handler.can_run():\n    url = urls[index % len(urls)]\n    index += 1\n\n    print('Scraping url', url)\n    response = requests.get(url)\n\n    soup = BeautifulSoup(response.content, 'html.parser')\n    book_name = soup.select_one('.product_main').h1.text\n    rows = soup.select('.table.table-striped tr')\n    product_info = {row.th.text: row.td.text for row in rows}\n\n    data_folder = Path('./data')\n    data_folder.mkdir(parents=True, exist_ok=True)\n\n    json_file_name = re.sub('[\\': ]', '-', book_name)\n    json_file_path = data_folder / f'{json_file_name}.json'\n    with open(json_file_path, 'w') as book_file:\n        json.dump(product_info, book_file)\n```\n\nThen, create a systemd configuration file.\n\n**/etc/systemd/system/book-scraper.service:**\n```\n[Unit]\nDescription=A script for scraping the book information\nAfter=syslog.target network.target\n\n[Service]\nWorkingDirectory=/home/oxylabs/python-script-service/src/systemd\nExecStart=/home/oxylabs/python-script-service/venv/bin/python3 main.py\n\nRestart=always\nRestartSec=120\n\n[Install]\nWantedBy=multi-user.target\n```\nMake sure to adjust the paths based on your actual script location.\n\nA fully working example can be found [here](src/systemd/linux_scrape.py).\n\n## Create A Windows Service\n\nTo create a Windows service, you will need to implement methods such as `SvcDoRun` and `SvcStop` and handle events sent by the operating system.\n\n**windows_scrape.py:**\n```python\nimport sys\nimport servicemanager\nimport win32event\nimport win32service\nimport win32serviceutil\nimport json\nimport re\nfrom pathlib import Path\n\nimport requests\nfrom bs4 import BeautifulSoup\n\n\nclass BookScraperService(win32serviceutil.ServiceFramework):\n    _svc_name_ = 'BookScraperService'\n    _svc_display_name_ = 'BookScraperService'\n    _svc_description_ = 'Constantly updates the info about books'\n\n    def __init__(self, args):\n        win32serviceutil.ServiceFramework.__init__(self, args)\n        self.event = win32event.CreateEvent(None, 0, 0, None)\n\n    def GetAcceptedControls(self):\n        result = win32serviceutil.ServiceFramework.GetAcceptedControls(self)\n        result |= win32service.SERVICE_ACCEPT_PRESHUTDOWN\n        return result\n\n    def SvcDoRun(self):\n        urls = [\n'https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html',\n'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html',\n'https://books.toscrape.com/catalogue/sharp-objects_997/index.html',\n        ]\n\n        index = 0\n\n        while True:\n            result = win32event.WaitForSingleObject(self.event, 5000)\n            if result == win32event.WAIT_OBJECT_0:\n                break\n\n            url = urls[index % len(urls)]\n            index += 1\n\n            print('Scraping url', url)\n            response = requests.get(url)\n\n            soup = BeautifulSoup(response.content, 'html.parser')\n            book_name = soup.select_one('.product_main').h1.text\n            rows = soup.select('.table.table-striped tr')\n            product_info = {row.th.text: row.td.text for row in rows}\n\n            data_folder = Path('C:\\\\Users\\\\User\\\\Scraper\\\\dist\\\\scrape\\\\data')\n            data_folder.mkdir(parents=True, exist_ok=True)\n\n            json_file_name = re.sub('[\\': ]', '-', book_name)\n            json_file_path = data_folder / f'{json_file_name}.json'\n            with open(json_file_path, 'w') as book_file:\n                json.dump(product_info, book_file)\n\n    def SvcStop(self):\n        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)\n        win32event.SetEvent(self.event)\n\n\nif __name__ == '__main__':\n    if len(sys.argv) == 1:\n        servicemanager.Initialize()\n        servicemanager.PrepareToHostSingle(BookScraperService)\n        servicemanager.StartServiceCtrlDispatcher()\n    else:\n        win32serviceutil.HandleCommandLine(BookScraperService)\n```\n\nNext, install dependencies and run a post-install script. \n\n```\nPS C:\\\u003e cd C:\\Users\\User\\Scraper\nPS C:\\Users\\User\\Scraper\u003e .\\venv\\Scripts\\pip install pypiwin32\nPS C:\\Users\\User\\Scraper\u003e .\\venv\\Scripts\\pywin32_postinstall.py -install\n```\n\nBundle your script into an executable.\n```\nPS C:\\Users\\User\\Scraper\u003e venv\\Scripts\\pyinstaller --hiddenimport win32timezone -F scrape.py\n```\n\nAnd finally, install your newly-created service.\n```\nPS C:\\Users\\User\\Scraper\u003e .\\dist\\scrape.exe install\nInstalling service BookScraper\nChanging service configuration\nService updated\n\nPS C:\\Users\\User\\Scraper\u003e .\\dist\\scrape.exe start\nStarting service BookScraper\nPS C:\\Users\\User\\Scripts\u003e\n```\n\nA fully working example can be found [here](src/windows-service/windows_scrape.py).\n\n## Easier Windows Service Using NSSM\n\nInstead of dealing with the Windows service layer directly, you can use the NSSM (Non-Sucking Service Manager).\n\nInstall NSSM by visiting [the official website](https://nssm.cc/download). Extract it to a folder of your choice and add the folder to your PATH environment variable for convenience.\n\nOnce you have NSSM installed, simplify your script by getting rid of all Windows-specific methods and definitions.\n\n**simple_scrape.py:**\n```python\nimport json\nimport re\nfrom pathlib import Path\n\nimport requests\nfrom bs4 import BeautifulSoup\n\nurls = ['https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html',\n        'https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html',\n        'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ]\n\nindex = 0\n\nwhile True:\n    url = urls[index % len(urls)]\n    index += 1\n\n    print('Scraping url', url)\n    response = requests.get(url)\n\n    soup = BeautifulSoup(response.content, 'html.parser')\n    book_name = soup.select_one('.product_main').h1.text\n    rows = soup.select('.table.table-striped tr')\n    product_info = {row.th.text: row.td.text for row in rows}\n\n    data_folder = Path('C:\\\\Users\\\\User\\\\Scraper\\\\data')\n    data_folder.mkdir(parents=True, exist_ok=True)\n\n    json_file_name = re.sub('[\\': ]', '-', book_name)\n    json_file_path = data_folder / f'{json_file_name}.json'\n    with open(json_file_path, 'w') as book_file:\n        json.dump(product_info, book_file)\n```\n\nBundle your script into an executable.\n```\nPS C:\\Users\\User\\Scraper\u003e venv\\Scripts\\pyinstaller -F simple_scrape.py\n```\n\nAnd finally, install the script using NSSM.\n```\nPS C:\\\u003e nssm.exe install SimpleScrape C:\\Users\\User\\Scraper\\dist\\simple_scrape.exe\nPS C:\\Users\\User\\Scraper\u003e .\\venv\\Scripts\\pip install pypiwin32\n```\n\nA fully working script can be found [here](src/nssm/simple_scrape.py).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpython-script-service-guide","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fpython-script-service-guide","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fpython-script-service-guide/lists"}