{"id":51550304,"url":"https://github.com/gesiscss/gesis_surf_extension","last_synced_at":"2026-07-09T23:01:50.998Z","repository":{"id":331690546,"uuid":"1115953312","full_name":"gesiscss/gesis_surf_extension","owner":"gesiscss","description":"GESIS Surf Extension","archived":false,"fork":false,"pushed_at":"2026-05-07T20:40:54.000Z","size":1723,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-07T20:42:08.398Z","etag":null,"topics":["browser-extension","chrome-extension","extension","firefox","react","typescript"],"latest_commit_sha":null,"homepage":"https://www.gesis.org/info/surf","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gesiscss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-13T22:19:57.000Z","updated_at":"2026-05-07T19:59:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gesiscss/gesis_surf_extension","commit_stats":null,"previous_names":["gesiscss/gesis_surf_extension"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gesiscss/gesis_surf_extension","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2Fgesis_surf_extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2Fgesis_surf_extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2Fgesis_surf_extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2Fgesis_surf_extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gesiscss","download_url":"https://codeload.github.com/gesiscss/gesis_surf_extension/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gesiscss%2Fgesis_surf_extension/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35314872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser-extension","chrome-extension","extension","firefox","react","typescript"],"created_at":"2026-07-09T23:01:48.674Z","updated_at":"2026-07-09T23:01:50.990Z","avatar_url":"https://github.com/gesiscss.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ctable\u003e\u003ctr\u003e\u003ctd bgcolor=\"white\" style=\"padding: 20px;\"\u003e\n\u003cimg src=\"images/gesis.png\" alt=\"GESIS\" height=\"120\"\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\n# GESIS Surf\n\n**An Open-Source Infrastructure for Privacy-Preserving Longitudinal Web Browsing Data Collection**\n\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![Node Version](https://img.shields.io/badge/node-%3E%3D18.12.0-brightgreen.svg)](https://nodejs.org/)\n[![TypeScript](https://img.shields.io/badge/typescript-5.2.2-3178c6?logo=typescript\u0026logoColor=white)](https://www.typescriptlang.org/)\n[![React](https://img.shields.io/badge/react-18.2-61dafb?logo=react\u0026logoColor=white)](https://react.dev/)\n[![pnpm](https://img.shields.io/badge/pnpm-9.1.1-f69220?logo=pnpm\u0026logoColor=white)](https://pnpm.io/)\n\n---\n\n[![Quality Gate](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=coverage)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=bugs)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=code_smells)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=security_rating)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n[![Duplications](https://sonarcloud.io/api/project_badges/measure?project=gesiscss_gesis_surf_extension\u0026metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n\n[![SonarQube Cloud](https://sonarcloud.io/images/project_badges/sonarcloud-highlight.svg)](https://sonarcloud.io/summary/new_code?id=gesiscss_gesis_surf_extension)\n\n---\n\n[Features](#features) •\n[Installation](#installation) •\n[Usage](#development) •\n[API Documentation](#architecture) •\n[Contributing](#contributing) •\n[License](#license)\n\n\u003c/div\u003e\n\n---\n\nGESIS Surf is an open-source research infrastructure for privacy-preserving, longitudinal collection of web browsing behavioral data at scale — combining a browser extension, REST API backend, and hierarchical session modeling to enable reproducible passive panel studies created by [GESIS – Leibniz Institute for the Social Sciences](https://www.gesis.org/).\n\n\u003e 🔗 **Looking for the backend?** Check out [GESIS Surf Backend](https://github.com/geomario/gesis_surf_backend)\n\n## ✨ Features\n\n- � **Passive Longitudinal Data Collection** - Captures naturalistic browsing behavior over time without interrupting users, enabling large-scale panel studies\n- 🏗️ **Hierarchical Session Modeling** - Preserves the full structure of browsing behavior across windows, tabs, domains, and interactions\n- 📄 **Content-Level Capture** - Records clicks, scrolls, DOM changes, page metadata, and full HTML snapshots per observation\n- 🛡️ **Privacy-by-Design** - Strict opt-in participation, per-domain collection rules, and client-side data minimization at the point of collection\n- 🌐 **Cross-Browser Support** - Works on both Chrome and Firefox via WebExtension API\n- 🔐 **Secure Authentication** - Token-based authentication with secure session management\n- 💾 **Client-Side Storage** - IndexedDB for local data buffering before transmission\n- ♻️ **Reproducible Infrastructure** - Open-source, self-hostable backend with REST API for transparent and auditable research workflows\n\n## 📋 Requirements\n\n- **Node.js**: \u003e= 18.12.0\n- **Package Manager**: pnpm 9.1.1 or higher\n\n## 🚀 Installation\n\nClone the repository and install dependencies:\n\n```bash\ngit clone git@github.com:gesiscss/gesis_surf_extension.git\ncd gesis_surf_extension\nnpm install\n# or\npnpm install\n```\n\n## 🔨 Building the Extension\n\n### Firefox\n\nBuild the extension for Firefox (default):\n\n```bash\n pnpm run build:firefox\n```\n\nThe compiled files will be in the `dist/` directory.\n\nTo load the extension in Firefox:\n1. Navigate to `about:debugging#/runtime/this-firefox`\n2. Or go to **Firefox** \u003e **Preferences** \u003e **Extensions \u0026 Themes** \u003e **Debug Add-ons** \u003e **Load Temporary Add-on...**\n3. Locate and select the `dist/manifest.json` file\n\n### Chrome\n\nBuild the extension for Google Chrome:\n\n```bash\npnpm run build\n```\n\nThe compiled files will be in the `dist/` directory.\n\nTo load the extension in Chrome:\n1. Open `chrome://extensions/`\n2. Enable **Developer mode** (top-right corner)\n3. Click **Load unpacked**\n4. Select the `dist/` directory\n\n## 💻 Development\n\n### Start the Development Server\n\nFor Chrome (with HMR support):\n\n```bash\npnpm run dev\n```\n\nFor Firefox (with HMR support):\n\n```bash\npnpm run dev:firefox\n```\n\n### Available Scripts\n\n- `pnpm run clean` - Clean build artifacts and cache\n- `pnpm run build` - Build for Chrome\n- `pnpm run build:firefox` - Build for Firefox\n- `pnpm run dev` - Start development server (Chrome, with HMR)\n- `pnpm run dev:firefox` - Start development server (Firefox, with HMR)\n- `pnpm run test` - Run tests\n- `pnpm run type-check` - Type-check the entire project\n- `pnpm run lint` - Lint all files\n- `pnpm run lint:fix` - Fix linting issues\n- `pnpm run prettier` - Format code with Prettier\n- `pnpm run docs` - Generate TypeDoc documentation\n\n## 📁 Project Structure\n\n```\n├── chrome-extension/          # Chrome extension source code\n│   ├── lib/                   # Core extension logic\n│   │   ├── background/        # Service worker/background script\n│   │   ├── controllers/       # Core extension controller\n│   │   ├── db/                # Database service and configuration\n│   │   ├── events/            # Event managers (Tab, Window, Domain, Content)\n│   │   ├── handlers/          # Client and shared message handlers\n│   │   ├── messages/          # Message interfaces and handlers\n│   │   └── services/          # Auth, data collection, session, sync, policy\n│   ├── public/                # Static assets (icons, CSS)\n│   ├── utils/plugins/         # Vite manifest plugin\n│   └── manifest.js            # Extension manifest\n├── pages/                     # UI components and pages\n│   ├── content/               # Content script (clicks, scrolls, HTML capture)\n│   ├── popup/                 # Extension popup (React, MUI, Auth, PrivacyMode)\n│   └── utils/                 # Shared page assets and ConnectedPage HOC\n├── packages/                  # Shared packages and utilities\n│   ├── dev-utils/             # Manifest parser, logger, and dev utilities\n│   ├── hmr/                   # Hot module replacement (rollup-based)\n│   ├── shared/                # Shared React hooks, storages, HOCs, and services\n│   ├── tailwind-config/       # Shared Tailwind CSS configuration\n│   └── tsconfig/              # Shared TypeScript configurations\n└── docs/                      # Generated TypeDoc documentation\n```\n\n### Key Components\n\n- **Background Service Worker** (`lib/background/`) - Manages extension lifecycle, coordinates all events and services\n- **EventManager** (`lib/events/`) - Orchestrates Tab, Window, Domain, and Content event managers\n- **Content Script** (`pages/content/`) - Injected script capturing clicks, scrolls, and HTML snapshots per page\n- **Popup UI** (`pages/popup/`) - React interface for user authentication, privacy mode, and settings\n- **GlobalSessionService** (`lib/services/globalSession/`) - Builds and maintains the hierarchical session model across windows and tabs\n- **PolicyService** (`lib/services/policyService/`) - Enforces per-domain and per-content collection rules (privacy-by-design)\n- **AuthService** (`lib/services/authService/`) - Token-based authentication and session management\n- **DatabaseService** (`lib/db/`) - IndexedDB client-side data buffering before transmission\n- **DataCollectionService** (`lib/services/dataCollectionService/`) - Aggregates and processes collected interaction data\n- **SyncService** (`lib/services/syncService/`) - Handles periodic data synchronization to the backend API\n- **PrivateModeService** (`lib/services/privateModeService/`) - User-controlled privacy mode with timed activation\n- **MessageHandler** (`lib/messages/`) - Typed message passing between background, content, and popup scripts\n\n## 🏗️ Architecture\n\n### Extension Architecture\n\nThe extension follows a modular architecture:\n\n- **Background Script (Service Worker)** - Manages extension state and coordinates events\n- **Content Script** - Collects user interaction data from web pages\n- **Popup UI** - Provides user authentication and privacy controls\n- **Message Passing** - Secure communication between background, content, and popup scripts\n- **IndexedDB** - Local storage for data persistence\n\n### System Integration\n\n```\n┌─────────────────────┐\n│  Browser Extension  │\n├─────────────────────┤\n│ - Content Script    │  Collects: clicks, scrolls, HTML snapshots,\n│ - Background Worker │           domains, tab/window events,\n│ - Popup UI          │           session hierarchy, host policy\n│ - IndexedDB Storage │\n└──────────┬──────────┘\n           │ HTTPS/Secure\n           │ Authentication\n           ▼\n┌─────────────────────┐\n│  Django Backend     │\n├─────────────────────┤\n│ - REST API          │  Processes: user registration,\n│ - Token Auth        │  authentication, data aggregation,\n│ - Database          │  analysis \u0026 reporting\n│ - Celery/Redis      │\n│ - Elasticsearch     │\n└─────────────────────┘\n```\n\n**Data Flow:**\n1. On startup/install, `AuthService` validates the stored token against `/api/user/me/`\n2. If authenticated, `HostService` syncs the domain blocklist/allowlist from `/api/host/hosts/`\n3. `GlobalSessionService` creates a hierarchical session (global → window → tab → domain) and posts it to `/api/session/`\n4. `EventManager` starts `TabEventManager`, `WindowEventManager`, `DomainEventManager`, and `ContentEventManager`\n5. Content script captures **clicks**, **scrolls**, and **HTML snapshots** (with meta tags) and sends them via message passing to the background service worker\n6. Background worker writes events to **IndexedDB** via `DatabaseService` for local buffering\n7. Events are flushed to the backend API (`/api/clicks/`, `/api/scrolls/`, `/api/tab/tabs/`, `/api/domain/domains/`)\n8. `HeartbeatService` runs every 10 seconds to maintain extension liveness state\n9. `PrivateModeService` suspends data collection when the user activates privacy mode\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our **[Contributing Guide](CONTRIBUTING.md)** for detailed information on:\n\n- 🌿 **Branching Strategy** - `dev` → `main` → `prod` workflow\n- 📝 **Commit Conventions** - Using Commitizen with Conventional Commits\n- 🔍 **Code Quality** - Pre-commit hooks, linting, and formatting\n- 🔀 **Pull Request Process** - Guidelines and review workflow\n\n### Quick Start\n\n1. **Fork** the repository\n2. **Create** a feature branch from `dev`\n   ```bash\n   git checkout dev \u0026\u0026 git pull origin dev\n   git checkout -b feature/amazing-feature\n   ```\n3. **Install pre-commit hooks**\n   ```bash\n   pnpm install\n   pnpm run prepare\n   ```\n4. **Commit using Commitizen**\n   ```bash\n   git add .\n   pnpm cz\n   ```\n5. **Push and open a Pull Request** targeting `dev`\n\n## ✅ Code Quality\n\nThis project uses:\n\n- **ESLint** - For code linting\n- **Prettier** - For code formatting\n- **TypeScript** - For type safety\n- **Husky** - For pre-commit and commit-msg hooks\n- **lint-staged** - For running linters on staged files\n- **commitlint** - Enforces Conventional Commits format on every commit message\n\nRun quality checks:\n\n```bash\npnpm run lint\npnpm run lint:fix\npnpm run type-check\npnpm run prettier\n```\n\n## 📦 Technology Stack\n\n- **UI**: React 18, React Router v6, MUI v6 (Material UI), Emotion, Tailwind CSS\n- **Build Tools**: Vite 6, Turbo (monorepo task runner), Rollup (HMR package)\n- **Language**: TypeScript 5.9\n- **Storage**: IndexedDB via `idb` library\n- **Browser APIs**: WebExtension API with `webextension-polyfill`\n- **Unique IDs**: `uuid` v11 for session identifier generation\n- **Code Quality**: ESLint (Airbnb TypeScript config), Prettier, Husky, lint-staged, commitlint\n- **Commit Tooling**: Commitizen (`cz-conventional-changelog`), commitlint (`@commitlint/config-conventional`)\n- **Package Manager**: pnpm 9.1.1 (workspace monorepo)\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [`LICENSE`](LICENSE) file for details.\n\nCopyright © 2023-2025 [GESIS – Leibniz Institute for the Social Sciences](https://www.gesis.org/)\n\n## 🔗 Backend Integration\n\nThe GESIS Surf Extension works in conjunction with the **GESIS Surf Backend** for data processing and storage.\n\n### Related Repositories\n\n- **[GESIS Surf Backend](https://github.com/geomario/gesis_surf_backend)** - Django REST API for data collection, user management, and research analysis\n  - Built with Django 4.2 and Python 3.10+\n  - PostgreSQL for persistent storage\n  - Celery/Redis for async task processing\n  - Elasticsearch for fast data retrieval\n  - Docker-ready deployment\n\n### Data Collection Endpoints\n\nThe extension communicates with the backend API for:\n\n| Endpoint                    | Purpose                                              |\n| -------------------------- | ---------------------------------------------------- |\n| `/api/user/token/`         | Authentication token generation                      |\n| `/api/user/me/`            | User profile and data collection status              |\n| `/api/session/`            | Global session hierarchy submission                  |\n| `/api/tab/tabs/`           | Browser tab event tracking                           |\n| `/api/domain/domains/`     | Domain classification and event tracking             |\n| `/api/clicks/`             | Click event submission                               |\n| `/api/scrolls/`            | Scroll event submission                              |\n| `/api/host/hosts/`         | Host blocklist/allowlist sync (policy rules)         |\n| `/api/host/task-result/`   | Async host sync task polling                         |\n| `/api/selectors/`          | Dynamic LLM-based CSS selector retrieval             |\n| `/api/selectors/task-result/` | Async selector task polling                       |\n\n## 👥 Authors\n\n- **Mario Ramirez** - _Lead Research Software Engineer_ - [@geomario](https://github.com/geomario) [@MarioGesis](https://www.gesis.org/en/institute/about-us/staff/person/mario.ramirez)\n- **Fernando Guzman** - _Software Architect Consultant_ - [@Fernando](https://www.linkedin.com/in/fernando-guzman-9262801b/)\n- **Dr. Sebastian Stier** - _Department Director CSS_ [@Seb](https://www.gesis.org/en/institute/about-us/staff/person/sebastian.stier)\n- **Dr. Frank Mangold** - _Kommissarischer Teamleiter DDD_ [@Frank](https://www.gesis.org/institut/ueber-uns/mitarbeitendenverzeichnis/person/Frank.Mangold)\n\n## 🙏 Acknowledgments\n\n- [GESIS - Leibniz Institute for the Social Sciences](https://www.gesis.org/)\n- [Computational Social Science Department](https://www.gesis.org/en/institute/about-us/departments/computational-social-science)\n\n## 🔒 Privacy Notice\n\nThis extension is designed with privacy in mind. Data collection is:\n- **Transparent** - Users know what data is being collected\n- **Ethical** - Complies with research ethics standards\n- **Secure** - Uses secure authentication and storage mechanisms\n- **User-Controlled** - Includes privacy mode and user controls\n\nFor detailed privacy information, please refer to the project's privacy documentation or contact GESIS directly.\n\n## 📧 Contact\n\nQuestions or feedback? Reach out!\n\n- **GitHub Issues**: [Create an issue](https://github.com/gesiscss/gesis_surf_extension/issues)\n- **Backend Issues**: [Backend Repository](https://github.com/geomario/gesis_surf_backend/issues)\n- **GESIS**: https://www.gesis.org/\n\n## 📝 Citation\n\nIf you use this software in your research, please cite:\n\n```bibtex\n@article{ramirez2025gesis,\n  title = {GESIS Surf Extension},\n  author = {Ramirez, Mario and Guzman, Fernando and Stier, Sebastian and Mangold, Frank},\n  journal = {SoftwareX},\n  volume = {XX},\n  pages = {XXXXXX},\n  year = {2026},\n  publisher = {Elsevier},\n  doi = {10.1016/j.softx.2025.xxxxxx}\n}\n```\n\nSee [`CITATION.cff`](CITATION.cff) for more citation formats.\n\n---\n\n\u003cdiv align=\"center\"\u003e\nMade with ❤️ at GESIS\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgesiscss%2Fgesis_surf_extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgesiscss%2Fgesis_surf_extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgesiscss%2Fgesis_surf_extension/lists"}