{"id":20935759,"url":"https://github.com/ghostery/areweprivateyet","last_synced_at":"2025-07-31T00:40:42.391Z","repository":{"id":7319409,"uuid":"8638023","full_name":"ghostery/areweprivateyet","owner":"ghostery","description":"The crawler/analysis component of Are We Private Yet?","archived":false,"fork":false,"pushed_at":"2015-06-03T19:45:50.000Z","size":59186,"stargazers_count":47,"open_issues_count":1,"forks_count":10,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-07-17T21:00:07.044Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.areweprivateyet.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ghostery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-07T22:30:41.000Z","updated_at":"2025-04-10T11:50:42.000Z","dependencies_parsed_at":"2022-08-24T20:12:24.161Z","dependency_job_id":null,"html_url":"https://github.com/ghostery/areweprivateyet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ghostery/areweprivateyet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghostery%2Fareweprivateyet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghostery%2Fareweprivateyet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghostery%2Fareweprivateyet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghostery%2Fareweprivateyet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ghostery","download_url":"https://codeload.github.com/ghostery/areweprivateyet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghostery%2Fareweprivateyet/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267967720,"owners_count":24173566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T22:16:16.784Z","updated_at":"2025-07-31T00:40:42.342Z","avatar_url":"https://github.com/ghostery.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Are We Private Yet?\n-------------------------\n\nThis project is dedicated to automated recreation of Stanford's CIS [Tracking the Trackers: Self-Help Tools](http://cyberlaw.stanford.edu/blog/2011/09/tracking-trackers-self-help-tools) study.\nThis project extends certain feature counts as well, such as, total bandwidth, redirects, local storage, etc.\n\nThis project is separated broadly into three areas:\n- Firefox setup for the project\n- [FourthParty](http://fourthparty.info) extension\n- Crawler and Analysis utilities\n\nThe crawler is based on Selenium through Firefox WebDriver interface. Each run, while using an existing profile \nas a base is actually a separate anonymous profile for Firefox, and as such, the last thing crawler will do is\ncopy the FourthParty sqlite database into a path defined at runtime.\n\nAll required libraries for the test are stored in lib directory.\n\nResults\n=======\nThe results of several months' worth of crawling are available on the project's website: http://www.areweprivateyet.com/\n\nSetup\n=====\n\nTo set up the tool you will need Firefox (10 and up),\n[our FourthParty build](https://github.com/ghostery/fourthparty), and AreWeBetterYet analysis utilities.\nWe currently test the following extensions in addition to baseline:\n- Ghostery\n- DoNotTrackMe\n- Disconnect\n- Adblock Plus with [Fanboy Adblock list](http://www.fanboy.co.nz/fanboy-adblock.txt)\n- Adblock Plus with [EasyList](https://easylist-downloads.adblockplus.org/easylist.txt)\n- TrackerBlock\n- Firefox with third-party cookies disabled\n\nFirefox profiles need to be set up prior to running crawler from analysis utilities. The profiles must be named\nas follows: __\"ghostery\", \"dntme\", \"abp-fanboy\", \"abp-easylist\", \"trackerblock\", \"disconnect\", \"cookies-blocked\"__.\nEach profile needs to contain the FourthParty install (tho this could be force-installed on the profile\nthrough Selenium within crawler) and the extension that is being tested. You are also responsible for setting up\nthe extension in the way that you want to test, meaning that if you just install ABP and have no lists, obviously\nthe result would be different if it did contain EasyPrivacy or any other list.\n\nYou may add your own extensions by modifying the utilities array, or you may request that we add your extension for\nfuture testing by emailing us at \u003careweprivateyet@ghostery.com\u003e. Please remember that baseline is always the first profile to be \nexecuted.\n\n\nExecuting the crawl\n===================\n\nThe Crawler is a simple Selenium based utility that will use the top500.list (you may substitute it with your own) to \nload and then navigate to each website in the list. To execute you may load the project into your IDE of choice and\nsimply run Crawler class.  Alternatively, you may use the build provided and run it with your local installation of\nJava:\n\n```\njava -Dawby_path=/output_path/ -classpath \"apache-mime4j-0.6.jar:lib/bsh-1.3.0.jar:lib/cglib-nodep-2.1_3.jar:lib/commons-codec-1.6.jar:lib/commons-collections-3.2.1.jar:lib/commons-exec-1.1.jar:lib/commons-io-2.2.jar:lib/commons-jxpath-1.3.jar:lib/commons-lang3-3.1.jar:lib/commons-logging-1.1.1.jar:lib/cssparser-0.9.8.jar:lib/dom4j-1.6.1.jar:lib/guava-14.0.jar:lib/hamcrest-core-1.3.jar:lib/hamcrest-library-1.3.jar:lib/htmlunit-2.11.jar:lib/htmlunit-core-js-2.11.jar:lib/httpclient-4.2.1.jar:lib/httpcore-4.2.1.jar:lib/httpmime-4.2.1.jar:lib/ini4j-0.5.2.jar:lib/jcommander-1.29.jar:lib/jetty-websocket-8.1.8.jar:lib/jna-3.4.0.jar:lib/jna-platform-3.4.0.jar:lib/json-20080701.jar:lib/junit-dep-4.11.jar:lib/log4j-1.2.13.jar:lib/nekohtml-1.9.17.jar:lib/netty-3.5.7.Final.jar:lib/operadriver-1.2.jar:lib/phantomjsdriver-1.0.1.jar:lib/poi-3.9-20121203.jar:lib/protobuf-java-2.4.1.jar:lib/sac-1.3.jar:lib/selenium-java-2.31.0.jar:lib/serializer-2.7.1.jar:lib/sqlite-jdbc-3.7.2.jar:lib/stax-api-1.0.1.jar:lib/testng-6.8.jar:lib/xalan-2.7.1.jar:lib/xercesImpl-2.10.0.jar:lib/xml-apis-1.4.01.jar:lib/xmlbeans-2.3.0.jar:.\"  com.evidon.areweprivateyet.Crawler\n```\n\nawby_path is the local setting for location of the top500.list file as well as the input and output folder that will \nbe used.  This value is used in Crawler and Aggregator classes.\n\nAfter each extension's crawl is completed, Crawler will copy the FourthParty SQLite database to the output directory \n(awby_path) to be used in the Aggregator utility later on.  The file name of the copied FourthParty database is \nfourthparty-profileName.sqlite.\n\nCrawling may be done in any order and at any time prior to the running of analysis utilites. You may also use another\nautomation tool to produce the fourthparty output.\n\n\nRunning analysis utilities\n==========================\n\nThe Aggregator class is designed to query and collect information from multiple FourthParty databases\ninto a human-readable Excel spreadsheet as well as produce JSON output.\nTo run, either execute from your IDE or use the following command:\n\n```\njava -Dawby_path=/output_path/ -classpath \"apache-mime4j-0.6.jar:lib/bsh-1.3.0.jar:lib/cglib-nodep-2.1_3.jar:lib/commons-codec-1.6.jar:lib/commons-collections-3.2.1.jar:lib/commons-exec-1.1.jar:lib/commons-io-2.2.jar:lib/commons-jxpath-1.3.jar:lib/commons-lang3-3.1.jar:lib/commons-logging-1.1.1.jar:lib/cssparser-0.9.8.jar:lib/dom4j-1.6.1.jar:lib/guava-14.0.jar:lib/hamcrest-core-1.3.jar:lib/hamcrest-library-1.3.jar:lib/htmlunit-2.11.jar:lib/htmlunit-core-js-2.11.jar:lib/httpclient-4.2.1.jar:lib/httpcore-4.2.1.jar:lib/httpmime-4.2.1.jar:lib/ini4j-0.5.2.jar:lib/jcommander-1.29.jar:lib/jetty-websocket-8.1.8.jar:lib/jna-3.4.0.jar:lib/jna-platform-3.4.0.jar:lib/json-20080701.jar:lib/junit-dep-4.11.jar:lib/log4j-1.2.13.jar:lib/nekohtml-1.9.17.jar:lib/netty-3.5.7.Final.jar:lib/operadriver-1.2.jar:lib/phantomjsdriver-1.0.1.jar:lib/poi-3.9-20121203.jar:lib/protobuf-java-2.4.1.jar:lib/sac-1.3.jar:lib/selenium-java-2.31.0.jar:lib/serializer-2.7.1.jar:lib/sqlite-jdbc-3.7.2.jar:lib/stax-api-1.0.1.jar:lib/testng-6.8.jar:lib/xalan-2.7.1.jar:lib/xercesImpl-2.10.0.jar:lib/xml-apis-1.4.01.jar:lib/xmlbeans-2.3.0.jar:.\"  com.evidon.areweprivateyet.Aggregator\n```\n\nUsing the databases in the input folder, Aggregator collects results and outputs a final file named analysis.xls.\nThis should be a copy of the aforementioned study.\n\n\nLicense\n======= \nAreWePrivateYet uses Apache License 2.0 http://www.apache.org/licenses/LICENSE-2.0\n\nLicense information is stored in the LICENSE file.\n\n\nHow to file an issue\n====================\nYou may file an issue using GitHub's own issue tracker: https://github.com/ghostery/areweprivateyet/issues\n\n\nHow to submit a fix/pull-request\n================================\nYou may fork the project and modify at will. Your changes may be submitted back to us via a GitHub's pull\nrequest.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghostery%2Fareweprivateyet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fghostery%2Fareweprivateyet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghostery%2Fareweprivateyet/lists"}