{"id":23974393,"url":"https://github.com/dbaio/portsfallout","last_synced_at":"2025-09-07T00:35:00.493Z","repository":{"id":45815188,"uuid":"283910996","full_name":"dbaio/portsfallout","owner":"dbaio","description":"An easy way to search/query the FreeBSD pkg-fallout archive","archived":false,"fork":false,"pushed_at":"2025-03-11T10:09:03.000Z","size":456,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-01T19:54:19.270Z","etag":null,"topics":["django","django-application","django-rest-framework","hacktoberfest","python","python-script","python3","scraper","scraping","scraping-python","scraping-websites","scrapy"],"latest_commit_sha":null,"homepage":"https://portsfallout.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dbaio.png","metadata":{"files":{"readme":"README.rst","changelog":"changelog.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-31T01:13:13.000Z","updated_at":"2025-03-11T10:09:07.000Z","dependencies_parsed_at":"2025-01-11T18:26:55.885Z","dependency_job_id":"b07309a2-6a40-4e8b-95e5-bf546b48a704","html_url":"https://github.com/dbaio/portsfallout","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/dbaio/portsfallout","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbaio%2Fportsfallout","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbaio%2Fportsfallout/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbaio%2Fportsfallout/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbaio%2Fportsfallout/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dbaio","download_url":"https://codeload.github.com/dbaio/portsfallout/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbaio%2Fportsfallout/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273983049,"owners_count":25202092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","django-application","django-rest-framework","hacktoberfest","python","python-script","python3","scraper","scraping","scraping-python","scraping-websites","scrapy"],"created_at":"2025-01-07T05:35:47.427Z","updated_at":"2025-09-07T00:34:59.983Z","avatar_url":"https://github.com/dbaio.png","language":"Python","readme":"Ports Fallout\n=============\n\nhttps://portsfallout.com/\n\n- Django application\n- Web crawling (Scrapy)\n\nAn easy way to search the FreeBSD pkg-fallout reports.\n\nBe nice!\n\n\nRunning\n-------\n\nInstall all requirements:\n\n::\n\n    django\n    requests\n    scrapy\n    djangorestframework\n    python-dateutil\n    dnspython\n\n\nCopy the sample ``settings.py`` and configure your database access:\n\n::\n\n   $ cp portsfallout/settings_dev.py portsfallout/settings.py\n\n\nCreate initial database:\n\n::\n\n   $ python manage.py migrate\n   Operations to perform:\n     Apply all migrations: admin, auth, contenttypes, ports, sessions\n   Running migrations:\n     Applying contenttypes.0001_initial... OK\n     Applying auth.0001_initial... OK\n     Applying admin.0001_initial... OK\n     Applying admin.0002_logentry_remove_auto_add... OK\n     Applying admin.0003_logentry_add_action_flag_choices... OK\n     Applying contenttypes.0002_remove_content_type_name... OK\n     Applying auth.0002_alter_permission_name_max_length... OK\n     Applying auth.0003_alter_user_email_max_length... OK\n     Applying auth.0004_alter_user_username_opts... OK\n     Applying auth.0005_alter_user_last_login_null... OK\n     Applying auth.0006_require_contenttypes_0002... OK\n     Applying auth.0007_alter_validators_add_error_messages... OK\n     Applying auth.0008_alter_user_username_max_length... OK\n     Applying auth.0009_alter_user_last_name_max_length... OK\n     Applying auth.0010_alter_group_name_max_length... OK\n     Applying auth.0011_update_proxy_permissions... OK\n     Applying ports.0001_initial... OK\n     Applying sessions.0001_initial... OK\n\n\nPopulate database (ports and fallout info):\n\n::\n\n   $ ./scripts/cron-import-index.sh\n   $ ./scripts/cron-scrapy.sh\n\n\nStart web-server:\n\n::\n\n   $ python manage.py runserver\n\n\nYou can also fetch older fallouts:\n\n::\n\n   $ cd scripts\n\n   Crawling messages from an specific month / Verbose\n   $ scrapy runspider -O scrapy_output/2021-May.json \\\n      -a scrapydate=\"2021-May\" pkgfallout_scrapy_spider.py\n\n   Then import all .json files to database:\n   $ python import-scrapy.py\n\n\nMore info in ``scripts/pkgfallout_scrapy_spider.py``.\n\n\n\nCron jobs\n---------\n\nExecution for keeping the database always updated:\n\n::\n\n   # Update ports tree reference in the database\n   30  0  *  *  *  /portsfallout/scripts/cron-import-index.sh\n\n   # Fetch/import all pkg-fallout's reports from the Mlmmj archive of the\n   # current month. Requests are cached, only new fallouts are fetched.\n   45  0  *  *  *  /portsfallout/scripts/cron-scrapy.sh\n\n   # Fetch/import pkg-fallout's from the last month\n   30  10  *  *  *  /portsfallout/scripts/cron-scrapy.sh lastmonth\n\n   # Update DNS values of the pkg-fallout servers\n   45  3  *  *  *  python manage.py server_update\n   45  3  *  *  *  python manage.py server_update -v 0  # no output\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbaio%2Fportsfallout","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdbaio%2Fportsfallout","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbaio%2Fportsfallout/lists"}