{"id":21630059,"url":"https://github.com/mozilla-services/shavar","last_synced_at":"2025-04-11T13:51:38.014Z","repository":{"id":18626382,"uuid":"21832315","full_name":"mozilla-services/shavar","owner":"mozilla-services","description":"Tracking Protection update service for Firefox based on Safe Browsing protocol","archived":false,"fork":false,"pushed_at":"2023-08-24T20:12:00.000Z","size":495,"stargazers_count":21,"open_issues_count":50,"forks_count":10,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-25T10:04:56.757Z","etag":null,"topics":["privacy","python","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mozilla-services.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-14T19:21:35.000Z","updated_at":"2025-03-23T20:28:14.000Z","dependencies_parsed_at":"2023-01-11T20:29:27.816Z","dependency_job_id":null,"html_url":"https://github.com/mozilla-services/shavar","commit_stats":null,"previous_names":[],"tags_count":50,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Fshavar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Fshavar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Fshavar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-services%2Fshavar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mozilla-services","download_url":"https://codeload.github.com/mozilla-services/shavar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248411966,"owners_count":21099036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["privacy","python","security"],"created_at":"2024-11-25T02:09:40.726Z","updated_at":"2025-04-11T13:51:37.993Z","avatar_url":"https://github.com/mozilla-services.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"shavar - a service that speaks Google's safe browsing protocol\n\nFor more information on safe browsing and the wire protocol this code\nspeaks, see:\n\n  https://developers.google.com/safe-browsing/developers_guide\n\n\nRunning locally\n---------------\n\nFor dev testing, create and activate a virtual environment:\n\n    virtualenv -p python3.7 shavar\n    source shavar/bin/activate\n\nInstall the necessary dependencies:\n\n    pip install -r requirements-test.txt\n\nRun code style check:\n\n    flake8 --exclude ./shavar/lib,./shavar/bin,./build,./deactivate,./.local\n\nRun unit tests:\n\n    nosetests -s --nologcapture ./shavar/tests\n\nConfigure for running locally in a development environment:\n\n    python setup.py develop\n\nRun the service locally:\n\n    pserve shavar.testing.ini\n\nBy default the service listens on port the loopback interface, port 6543.  If\nyou want to change this, modify the values in the INI file's [server:main]\nsection.\n\n\nConfiguration\n-------------\n\nThe shavar service serves changes to a set of hashes of canonicalized URLs.\nBasic configuration consists of specifying the names of the lists to be served\nand a section for each of those lists declaring at least the two minimum\nrequired configuration directives for each list. Read shavar-server-list-config\nfor more examples of Shavar configurations.\n\nSince the tracking protection files (AKA block lists and entity lists) are hosted in S3 by [shavar-list-creation](https://github.com/mozilla-services/shavar-list-creation/), your Shavar dev environment will need to setup AWS keys to retrieve those files. For more information on configuration for `boto` see:\n\n  https://github.com/boto/boto/#getting-started-with-boto\n\nWhen an update is needed on a tracking protection file, Shavar responds with a CDN link to download the latest files. You will need to setup a CDN in front of your S3 bucket using AWS's CloudFront. For more information on setting up CloudFront on your S3 bucket see:\n\n  https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/GettingStarted.SimpleDistribution.html\n\nA commented example configuration:\n\n    [shavar]\n    # A newline separated list of the lists to be served.  The names given\n    # here will be used to locate the list specific configuration stanzas\n    # elsewhere in the INI file.\n    lists_served = mozpub-track-digest256\n                   moz-abp-shavar\n                   moz-bananas-shavar\n    # The default protocol version to speak.  As yet, we only speak version\n    # 2 of the protocol even though it has been superceded by Google.\n    # Default value: 2\n    default_proto_ver = 2.0\n    # The root directory for the data files for lists if absolute path names\n    # are not provided in the list specific stanzas.  Not necessary if you\n    # provide absolute paths.\n    lists_root = tests\n    sentry_dsn = \"\"\n    # The DSN from the \"Client Keys\" section in the project settings in Sentry\n    sentry_env = \"\"\n    # The environment to use for sentry, e.g. dev, stage, or prod\n\n    # This is the public host and scheme to reach the service\n    # like https://shavar.stage.mozaws.net\n    # when not provided, uses X-Forwarded-Host then fallsback to HTTP_HOST\n    # and X-Forwarded-Proto for the scheme\n    host = shavar.in.production.mozilla.com\n    scheme = https\n\n    [mozpub-track-digest256]\n    # The type of list data that will be shipped.  Presumably this has some\n    # greater impact on the client side but as yet, it isn't used for much\n    # other than making sure the list's name and type match.\n    #\n    # The technical difference between the shavar and digest256 list types\n    # is that digest256 formatted lists use the entire 32 byte SHA256 hash\n    # in the list data while shavar formatted list only send the first 4\n    # bytes of the 32 byte hash.  The client then queries the service if it\n    # encounters a match for a given hash prefix(the first 4 bytes) to\n    # retrieve the entire hash for a given prefix.\n    type = digest256\n    # URL or relative path to the source data for this list.  Possibilities\n    # at the moment include:\n    #\n    # relative/path/to/the/file\n    # /absolute/path/to/the/file\n    # file:///absolute/path/to/the/file\n    # s3+file:///s3_bucket_name/s3_key_name_which_can_include_slashes\n    #\n    # In this usage, \"my_s3_bukkit\" is the S3 bucket name and\n    # \"faux/path/to/file/moz-abp-shavar.data\" is the full key name.  This\n    # just permits slight simulation of a file name.\n    source = s3+file:///my_s3_bukkit/faux/path/to/file/mozpub-track-digest256.data\n\n    [moz-abp-shavar]\n    # Firefox currently (as of 2015-07-13) allows digest256 lists to get away\n    # with breaking the safe browsing wire protocol slightly.  The protocol\n    # actually states that the service's response to a client request for\n    # updated data be formatted as a list of URLs to data files to be\n    # downloaded.\n    #\n    # With Firefox, digest256 changes can be served inline in the initial\n    # response.  Naughty Firefox.  No cookie.\n    #\n    type = shavar\n    # delta_chunk_source is part of the unit test suite and should always be\n    # available to test against.\n    source = shavar/tests/delta_chunk_source\n    # As a result of this use of URLs(referred to as redirects in the protocol\n    # specification document given at the top of the README), it is necessary\n    # for the service to know where the data files will be publicly reachable.\n    # This setting provides the base URL that, when combined with the path\n    # portion of the URL given in the `source` directive above, will be the\n    # redirect served in the response.\n    #\n    # Best practice: make sure it ends in a /\n    redirect_url_base = http://localhost:6543/data/\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-services%2Fshavar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmozilla-services%2Fshavar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-services%2Fshavar/lists"}