{"id":13537854,"url":"https://github.com/Lookyloo/lookyloo","last_synced_at":"2025-04-02T04:31:58.727Z","repository":{"id":37095613,"uuid":"98115418","full_name":"Lookyloo/lookyloo","owner":"Lookyloo","description":"Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.","archived":false,"fork":false,"pushed_at":"2024-08-07T11:55:08.000Z","size":5871,"stargazers_count":678,"open_issues_count":47,"forks_count":83,"subscribers_count":19,"default_branch":"main","last_synced_at":"2024-08-11T17:10:01.430Z","etag":null,"topics":["capture","dfir","information-security","lookyloo","privacy","scraping","web-security"],"latest_commit_sha":null,"homepage":"https://www.lookyloo.eu","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lookyloo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"code_of_conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-23T17:53:24.000Z","updated_at":"2024-08-13T15:06:31.918Z","dependencies_parsed_at":"2023-02-18T06:00:52.395Z","dependency_job_id":"62a56aa6-3e63-48ec-a80e-7ff8572369d7","html_url":"https://github.com/Lookyloo/lookyloo","commit_stats":null,"previous_names":["circl/lookyloo"],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lookyloo%2Flookyloo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lookyloo%2Flookyloo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lookyloo%2Flookyloo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lookyloo%2Flookyloo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lookyloo","download_url":"https://codeload.github.com/Lookyloo/lookyloo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246757525,"owners_count":20828914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capture","dfir","information-security","lookyloo","privacy","scraping","web-security"],"created_at":"2024-08-01T09:01:04.327Z","updated_at":"2025-04-02T04:31:53.712Z","avatar_url":"https://github.com/Lookyloo.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e新添加的","\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e工具","Python","Cybersecurity"],"sub_categories":["\u003ca id=\"31185b925d5152c7469b963809ceb22d\"\u003e\u003c/a\u003e新添加的"],"readme":"[![Lookyloo icon](website/web/static/lookyloo.jpeg)](https://www.lookyloo.eu/docs/main/index.html)\n\n*[Lookyloo](https://lookyloo.circl.lu/)* is a web interface that captures a webpage and then displays a tree of the domains, that call each other.\n\n\n[![Gitter](https://badges.gitter.im/Lookyloo/community.svg)](https://gitter.im/Lookyloo/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\n\n\n* [What is Lookyloo?](#whats-in-a-name)\n* [REST API](#rest-api)\n* [Install Lookyloo](#installation)\n* [Lookyloo Client](#python-client)\n* [Contributing to Lookyloo](#contributing-to-lookyloo)\n  * [Code of Conduct](#code-of-conduct)\n* [Support](#support)\n  * [Security](#security)\n  * [Credits](#credits)\n  * [License](#license)\n\n\n\n## What's in a name?!\n\n```\nLookyloo ...\n\nSame as Looky Lou; often spelled as Looky-loo (hyphen) or lookylou\n\n1. A person who just comes to look.\n2. A person who goes out of the way to look at people or something, often causing crowds and disruption.\n3. A person who enjoys watching other people's misfortune. Oftentimes car onlookers that stare at a car accidents.\n\nIn L.A., usually the lookyloos cause more accidents by not paying full attention to what is ahead of them.\n```\nSource: [Urban Dictionary](https://www.urbandictionary.com/define.php?term=lookyloo)\n\n\n## No, really, what is Lookyloo?\n\nLookyloo is a web interface that allows you to capture and map the journey of a website page.\n\nFind all you need to know about Lookyloo on our [documentation website](https://www.lookyloo.eu/docs/main/index.html).\n\nHere's an example of a Lookyloo capture of the site **github.com**\n![Screenshot of Lookyloo capturing Github](https://www.lookyloo.eu/docs/main/_images/sample_github.png)\n\n# REST API\n\nThe API is self documented with swagger. You can play with it [on the demo instance](https://lookyloo.circl.lu/doc/).\n\n# Installation\n\nPlease refer to the [install guide](https://www.lookyloo.eu/docs/main/install-lookyloo.html).\n\n\n# Python client\n\n`pylookyloo` is the recommended client to interact with a Lookyloo instance.\n\nIt is avaliable on PyPi, so you can install it using the following command:\n\n```bash\npip install pylookyloo\n```\n\nFor more details on `pylookyloo`, read the overview [docs](https://www.lookyloo.eu/docs/main/pylookyloo-overview.html), the [documentation](https://pylookyloo.readthedocs.io/en/latest/) of the module itself, or the code in this [GitHub repository](https://github.com/Lookyloo/PyLookyloo).\n\n# Notes regarding using S3FS for storage\n\n## Directory listing\n\nTL;DR: it is slow.\n\nIf you have namy captures (say more than 1000/day), and store captures in a s3fs bucket mounted with s3fs-fuse,\ndoing a directory listing in bash (`ls`) will most probably lock the I/O for every process\ntrying to access any file in the whole bucket. The same will be true if you access the\nfilesystem using python methods (`iterdir`, `scandir`...))\n\nA workaround is to use the python s3fs module as it will not access the filesystem for listing directories.\nYou can configure the s3fs credentials in `config/generic.json` key `s3fs`.\n\n**Warning**: this will not save you if you run `ls` on a directoy that contains *a lot* of captures.\n\n## Versioning\n\nBy default, a MinIO bucket (backend for s3fs) will have versioning enabled, wich means it\nkeeps a copy of every version of every file you're storing. It becomes a problem if you have a lot of captures\nas the index files are updated on every change, and the max amount of versions is 10.000.\nSo by the time you have \u003e 10.000 captures in a directory, you'll get I/O errors when you try\nto update the index file. And you absolutely do not care about that versioning in lookyloo.\n\nTo check if versioning is enabled (can be either enabled or suspended):\n\n```\nmc version info \u003calias_in_config\u003e/\u003cbucket\u003e\n```\n\nThe command below will suspend versioning:\n\n```bash\nmc version suspend \u003calias_in_config\u003e/\u003cbucket\u003e\n```\n\n### I'm stuck, my file is raising I/O errors \n\nIt will happen when your index was updated 10.000 times and versioning was enabled.\n\nThis is how to check you're in this situation: \n\n* Error message from bash (unhelpful):\n\n```bash\n$ (git::main) rm /path/to/lookyloo/archived_captures/Year/Month/Day/index\nrm: cannot remove '/path/to/lookyloo/archived_captures/Year/Month/Day/index': Input/output error\n```\n\n* Check with python\n\n```python\nfrom lookyloo.default import get_config\nimport s3fs\n\ns3fs_config = get_config('generic', 's3fs')\ns3fs_client = s3fs.S3FileSystem(key=s3fs_config['config']['key'],\n                                secret=s3fs_config['config']['secret'],\n                                endpoint_url=s3fs_config['config']['endpoint_url'])\n\ns3fs_bucket = s3fs_config['config']['bucket_name']\ns3fs_client.rm_file(s3fs_bucket + '/Year/Month/Day/index')\n```\n\n* Error from python (somewhat more helpful):\n\n```\nOSError: [Errno 5] An error occurred (MaxVersionsExceeded) when calling the DeleteObject operation: You've exceeded the limit on the number of versions you can create on this object\n```\n\n* **Solution**: run this command to remove all older versions of the file \n\n```bash\nmc rm --non-current --versions --recursive --force \u003calias_in_config\u003e/\u003cbucket\u003e/Year/Month/Day/index\n```\n\n# Contributing to Lookyloo\nTo learn more about contributing to Lookyloo, see our [contributor guide](https://www.lookyloo.eu/docs/main/contributing.html).\n\n### Code of Conduct\nAt Lookyloo, we pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. You can access our Code of Conduct [here](https://github.com/Lookyloo/lookyloo/blob/main/code_of_conduct.md) or on the [Lookyloo docs site](https://www.lookyloo.eu/docs/main/code-conduct.html).\n\n\n# Support\n * To engage with the Lookyloo community contact us on [Gitter](https://gitter.im/lookyloo-app/community).\n * Let us know how we can improve Lookyloo by opening an [issue](https://github.com/Lookyloo/lookyloo/issues/new/choose).\n * Follow us on [Twitter](https://twitter.com/lookyloo_app).\n\n### Security\nTo report vulnerabilities, see our [Security Policy](lookyloo/SECURITY.md).\n\n### Credits\nThank you very much [Tech Blog @ willshouse.com](https://techblog.willshouse.com/2012/01/03/most-common-user-agents/) for the up-to-date list of UserAgents.\n\n### License\nSee our [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLookyloo%2Flookyloo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLookyloo%2Flookyloo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLookyloo%2Flookyloo/lists"}