{"id":13649266,"url":"https://github.com/doodyparizada/word2vec-spam-filter","last_synced_at":"2025-04-12T05:52:12.399Z","repository":{"id":87834599,"uuid":"113837606","full_name":"doodyparizada/word2vec-spam-filter","owner":"doodyparizada","description":"Using word vectors to classify spam messages","archived":false,"fork":false,"pushed_at":"2017-12-18T13:38:13.000Z","size":3248,"stargazers_count":150,"open_issues_count":3,"forks_count":12,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-12T05:52:04.759Z","etag":null,"topics":["antispam","flask","glove-vectors","numpy","react","schematics","stylus","typescript","webpack","word2vec"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doodyparizada.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-11T09:16:55.000Z","updated_at":"2024-11-22T19:56:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"6d8f46c9-9e86-4844-8669-e75bc4e48022","html_url":"https://github.com/doodyparizada/word2vec-spam-filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doodyparizada%2Fword2vec-spam-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doodyparizada%2Fword2vec-spam-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doodyparizada%2Fword2vec-spam-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doodyparizada%2Fword2vec-spam-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doodyparizada","download_url":"https://codeload.github.com/doodyparizada/word2vec-spam-filter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525156,"owners_count":21118616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["antispam","flask","glove-vectors","numpy","react","schematics","stylus","typescript","webpack","word2vec"],"created_at":"2024-08-02T01:04:54.351Z","updated_at":"2025-04-12T05:52:12.373Z","avatar_url":"https://github.com/doodyparizada.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# word2vec-spam-filter\n\nThis is a project done during the [Kik](https://github.com/kikinteractive/) hackathon 2017.\n\nIn this project we demonstrate a way to classify spam messages on the client while protecting user privacy.\n\nA client generates a \"hash\" from the message sending it to the server. The server then compares the \"hash\" to a bank of known reported messages.\n\nThe bank of known reported messages is created from spam reports. The server compares a given reported message to the previous bank of reported messages. If the message is similar to a previously reported message, a report count is incremented. Otherwise the message is added to the bank with a count of 1.\n\nA message in the bank of reported messages is considered a spam message once it was reported more than 3 times.\n\n## Preview\n\n\u003cimg src=\"clip.gif\" width=\"700\"/\u003e\n\n## Corpus downloads\nWe used 2 datasets for creating sentence vectors:\n1. word vectors taken from: https://github.com/stanfordnlp/GloVe\n2. word frequencies from: https://github.com/IlyaSemenov/wikipedia-word-frequency/blob/master/results/enwiki-20150602-words-frequency.txt\n\n## Configurable parameters (Hyper-Parameters)\nWe played around with a few configurations to get the best results for short user messages:\n\n* Confidence Threshold - a number between 0.0 - 1.0 to determine when 2 messages are considered the same\n* Distance Function - we used vector dot product\n* Normalization - how to deal with words we don't have in our corpus, punctuation marks, non english words\n* Vector Size - the longer the vector the higher the accuracy but heavier in memory\n* Weight Function - given a word frequency, how to create the vector weights (`the` should weigh less than `camera`)\n* Custom Corpus - creating the word vectors and frequencies from real user message data might yield better results\n* Random Indices - how many random indices should the client send to the server to mask the original message indices\n\n## Running the code\nThis project includes a single makefile to help with the initialization, dependency installation and corpus download.\nYou can invoke a help message by running:\n\n```\nmake\n```\n\nOr you can manually run the server and client apps:\n\n### server\nIn the `server` directory install the pip dependencies in a `virtualenv`:\n\n```\npip install -r requirements.txt\n```\n\nand run the server:\n```\npython app.py\n```\n\n### web client\nTo use the web client go into the `webclient` directory in your terminal and then:\n```\nnpm install\nnpm run dev\n```\n\nThat should install all dependencies and kick start the project, if it all works you should see something like:\n \u003e Project is running at http://localhost:3333/\n \u003e webpack output is served from /\n\nNow load http://localhost:3333/ in your browser\n\nThere are 3 different \"view modes\" which can be switched using the select box at the top right corner of the page.  \nThe 3 views are:\n * Standalone Tester: A textarea in which one can input a message and then either report it as spam or check whether it is classified as spam.\n * IM Sender: A textarea in which the user can input a message (or select a message from a bunch of existing ones) and then \"send\" the message to another client.\n * IM Receiver: A view which displays a list of received messages (using the `IM Sender`) and the ability to report each message as spam.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoodyparizada%2Fword2vec-spam-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoodyparizada%2Fword2vec-spam-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoodyparizada%2Fword2vec-spam-filter/lists"}