{"id":29224562,"url":"https://github.com/sing-group/warcprocessor","last_synced_at":"2025-07-03T06:07:19.099Z","repository":{"id":34433514,"uuid":"38366822","full_name":"sing-group/WARCProcessor","owner":"sing-group","description":"WARCProcessor is a platform independent integrative tool providing specific support to scientists that need to perform experiments in the field of web spam research.","archived":false,"fork":false,"pushed_at":"2024-02-06T12:00:25.000Z","size":32039,"stargazers_count":0,"open_issues_count":8,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-02-06T13:58:28.916Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sing-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-01T11:12:02.000Z","updated_at":"2020-10-14T08:03:20.000Z","dependencies_parsed_at":"2023-01-15T07:15:28.645Z","dependency_job_id":null,"html_url":"https://github.com/sing-group/WARCProcessor","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/sing-group/WARCProcessor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2FWARCProcessor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2FWARCProcessor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2FWARCProcessor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2FWARCProcessor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sing-group","download_url":"https://codeload.github.com/sing-group/WARCProcessor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing-group%2FWARCProcessor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263271501,"owners_count":23440396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-03T06:07:18.420Z","updated_at":"2025-07-03T06:07:19.078Z","avatar_url":"https://github.com/sing-group.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WARCProcessor\n\n## Usage\njava -jar WarcProject-4.X.X-X.jar [--help] [--nogui] [--config \u003cpath\u003e]\n\n* --help: Show the startup options\n* --nogui: Executes WARCProcessor without gui\n* --config \u003cpath\u003e: Starts WARCProcessor using the specified configuration file\n\n## Updates\n\n### Version 4\n\n* Modifications in the view to include links to the directory of the \n  datasource and to the output dir. A keyboard control is included to \n  move through the menu tree and there is a link from the main menu to create\n  datasources\n* Datasources can be disables through gui\n* A corpus can be created from CSV and multiple WARC or WARC.gz files\n* Executing without gui (using GetOpt)\n* Filtering data using idiom\n* A simple executable that can be executed through java -jar\n* Unified property names in CSV and ARFF datasources.\n* The SpamValue is now included in ARFF file\n* Save the config with a own extension for filenames\n\n### Version 3\n\n* Graphical interface added\n* Added DataSource for Wark directories (WarcDS). In contrast to CorpusDS, \n  this datasource search warc files in a directory and let user to indicate whether\n  the warc files included in the directory are ham or spam\n* Solved issue Cod. 2.01. When the dept is greater than 0 in the crawler, the searched \n  links are the ones that belongs to the original websitees.\n* Solved issue Cod. 2.02. In the datasource we should be able to \n  stablish the parameter \"spamCol\" with the value of a CSV field indicating whether\n  the URL of this row belongs to a ham or spam website. Additionally, in the parameter\n  \"spamColValue\" we should speficy the value for the field when the ROW is spam \n  (distinct values will be considered as ham)\n* Solved issue Cod. 2.03. In the datasource CSVDS we have included the parameter\n \"fieldSeparator\" where we can configure the filed used to keep columns separated.\n\n### Version 2\n\n* Added support for new input formats (datasources): Arff, CSV, Corpus Warc\n* Modified the configuration file of the application to adjust the Datasources\n  \n\n## Third party files\n\nIcons: www.aha-soft.com - Creative Commons Attribution-Share Alike 3.0 License\nThanks for the work.\n\n## Dependencies\n```\n# WarcProject dependencies\ncommons-beanutils-1.9.2.jar\ncommons-validator-1.4.0.jar\n\n# crawler4j dependencies\nhttpcore-4.2.2.jar\ncommons-logging-1.1.1.jar\nhttpclient-4.2.3.jar\ncommons-codec-1.6.jar\nje-4.0.92.jar\ntika-parsers-1.0.jar\nboilerpipe-1.1.0.jar\ntagsoup-1.2.1.jar\nmetadata-extractor-2.4.0-beta-1.jar\nasm-3.1.jar\ngeronimo-stax-api_1.0_spec-1.0.1.jar\ncommons-compress-1.3.jar\napache-mime4j-dom-0.7.jar\napache-mime4j-core-0.7.jar\ntika-core-1.0.jar\nlog4j-1.2.14.jar\n\n# heritrix dependencies\n# http://sourceforge.net/projects/archive-crawler/files/heritrix3\nheritrix-commons-3.1.0.jar\nfastutil-5.0.7.jar\ncommons-io-1.4.jar\n\n# Kryo dependencies\n# https://code.google.com/p/kryo/\n# Used to serialization for Warc reader\nkryo-1.04-all.jar\n\n# Weka dependencies\n# http://weka.wikispaces.com/\nweka.jar\n\n# Lucene dependencies\nluceneTrigramsLanguageGuesser.jar\n\n# Jsoup dependencies\n# http://jsoup.org/\n# Used to extract text from html\njsoup-1.8.1.jar\n\n# GetOpt dependencies\n# www.urbanophile.com/arenn/hacking/download.html\n# Used to get options from command line\njava-getopt-1.0.13.jar\n```\n\n## License\n\nSee LICENSE\n\n## Authors\n* Miguel Cayon Álvarez\n* Ivan Ferrant\n* Rosalía Laza Fidalgo\n* Reyes Pavón Rial\n* José Ramón Méndez Reboredo\n* David Ruano Ordás\n* José Ramón Méndez Reboredo\n* Florentino Fernández Riverola\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing-group%2Fwarcprocessor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsing-group%2Fwarcprocessor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing-group%2Fwarcprocessor/lists"}