{"id":18383593,"url":"https://github.com/sashido/content-moderation-automations","last_synced_at":"2025-07-20T11:32:34.641Z","repository":{"id":91749138,"uuid":"267062502","full_name":"SashiDo/content-moderation-automations","owner":"SashiDo","description":"An NSFW Image Classifier including an Automation Engine for fast deletion \u0026 moderation built with Node.js, TensorFlow, and Parse Server","archived":false,"fork":false,"pushed_at":"2020-08-05T10:45:58.000Z","size":135,"stargazers_count":22,"open_issues_count":0,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-06T23:34:39.854Z","etag":null,"topics":["example","nodejs","nsfw","nsfw-app","nsfw-classifier","nsfw-detection","parse-community","parse-platform","parse-sdk","parse-server","tensorflow","tensorflow-js","tutorial"],"latest_commit_sha":null,"homepage":"https://blog.sashido.io/content-moderation-service-with-nodejs-and-tensorflow-part-2-moderation-automation/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SashiDo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-26T14:16:56.000Z","updated_at":"2025-01-25T16:10:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"e66c36f3-072d-4f38-bd76-ea68742e4bfa","html_url":"https://github.com/SashiDo/content-moderation-automations","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SashiDo/content-moderation-automations","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SashiDo%2Fcontent-moderation-automations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SashiDo%2Fcontent-moderation-automations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SashiDo%2Fcontent-moderation-automations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SashiDo%2Fcontent-moderation-automations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SashiDo","download_url":"https://codeload.github.com/SashiDo/content-moderation-automations/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SashiDo%2Fcontent-moderation-automations/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266117509,"owners_count":23879076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["example","nodejs","nsfw","nsfw-app","nsfw-classifier","nsfw-detection","parse-community","parse-platform","parse-sdk","parse-server","tensorflow","tensorflow-js","tutorial"],"created_at":"2024-11-06T01:11:54.964Z","updated_at":"2025-07-20T11:32:29.632Z","avatar_url":"https://github.com/SashiDo.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ready-to-use Node.JS REST API and Automation Engine for moderation of indecent images.\n\nWith the vastly growing number of apps on the market, Content Moderation is becoming a heavy task that takes too much time and effort. That combined with the fact that advances in technologies brought Machine Learning closer to single developers inspired our team to build an **fully functional Open-Source Content Moderation service with ReactJS based Admin Panel** and make content moderation a piece of cake!\n\nWe've put the thoughts into a realization and built up three components, each one more advanced than the previous. No matter if you only need [Image Classification REST API](https://github.com/SashiDo/content-moderation-image-api), the REST API + Automation logic from this repo or the full-fledged product with ready-to-use React-based Admin panel(coming soon) - all you need to do is clone the respective repository, maybe customize a bit and deploy to your production app. The solution is simple, effective and can be easily integrated into every project even if this is your first encounter with Machine Learning. \n\n\u003cbr /\u003e\n\n# Examples \u0026 Demos\n\n## Automation Engine \n\nAn example of how the Automation Engine makes a decision based on the Moderation Preferences we've set and the REST API Image Classification.\n\n![](https://media-blog.sashido.io/content/images/2020/07/nsfw-4.png)\n\n\n## Image Classifiaction REST API \n\nThese are the examples and the demos how the REST API classifies images for some of the classes. For the other classes we think you should experiment by yourself ... you know what I mean ;).\n\n\u003ctable align=\"center\"\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth align=\"center\"\u003eImage Source\u003c/th\u003e\n      \u003cth align=\"center\"\u003eImage Source\u003c/th\u003e\n      \u003cth align=\"center\"\u003eImage Source\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003ca\n          href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/41a516cc1141cdaf775a5cfa083dcb26_thumbnail.png\"\u003e\n          \u003cimage\n            src=\"https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/41a516cc1141cdaf775a5cfa083dcb26_thumbnail.png\" /\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003ca\n          href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/a3115074f603c99770d376f3e6d0f72e_thumbnail.png\"\u003e\n          \u003cimage\n            src=\"https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/a3115074f603c99770d376f3e6d0f72e_thumbnail.png\" /\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003ca\n          href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/355a4b37b8a657dd09f3f50816481cca_thumbnail.png\"\u003e\n          \u003cimage\n            src=\"https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/355a4b37b8a657dd09f3f50816481cca_thumbnail.png\" /\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth align=\"center\"\u003eClassification Result\u003c/th\u003e\n      \u003cth align=\"center\"\u003eClassification Result\u003c/th\u003e\n      \u003cth align=\"center\"\u003eClassification Result\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\n\u003cpre\u003e[{\n  \"className\": \"Neutral\",\n  \"probability\": 0.93821\n}, {\n  \"className\": \"Drawing\",\n  \"probability\": 0.05473\n}, {\n  \"className\": \"Sexy\",\n  \"probability\": 0.00532\n}, {\n  \"className\": \"Hentai\",\n  \"probability\": 0.00087\n}, {\n  \"className\": \"Porn\",\n  \"probability\": 0.00085\n}]\u003c/pre\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n\u003cpre\u003e[{\n  \"className\": \"Sexy\",\n  \"probability\": 0.99394\n}, {\n  \"className\": \"Neutral\",\n  \"probability\": 0.00432\n}, {\n  \"className\": \"Porn\",\n  \"probability\": 0.00164\n}, {\n  \"className\": \"Drawing\",\n  \"probability\": 0.00006\n}, {\n  \"className\": \"Hentai\",\n  \"probability\": 0.00001\n}]\u003c/pre\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n\u003cpre\u003e[{\n  \"className\": \"Drawing\",\n  \"probability\": 0.96063\n}, {\n  \"className\": \"Neutral\",\n  \"probability\": 0.03902\n}, {\n  \"className\": \"Hentai\",\n  \"probability\": 0.00032\n}, {\n  \"className\": \"Sexy\",\n  \"probability\": 0.00001\n}, {\n  \"className\": \"Porn\",\n  \"probability\": 0.00005\n}]\u003c/pre\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/41a516cc1141cdaf775a5cfa083dcb26_thumbnail.png\"\u003eNeutral Demo\u003c/\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/a3115074f603c99770d376f3e6d0f72e_thumbnail.png\"\u003eSexy Demo\u003c/\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://nsfw-demo.sashido.io/api/image/classify?url=https://mqfy379g6xncpc3va4epdzkfsxf0vs.files-sashido.cloud/355a4b37b8a657dd09f3f50816481cca_thumbnail.png\"\u003eDrawing Demo\u003c/\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cbr /\u003e\n\n\n# How it works\n\nThis service is built in Node.JS with Mongo DB and Parse Server. You can use it in a standard Express app, but keep in mind that the file structure of the repo is Parse specific. The code is organized in a ```src``` folder and ```src/cloud/main.js``` is the root file for the service.\n\n## REST API\n\nThe REST API works with [NSFW.JS](https://github.com/infinitered/nsfwjs) classification, which uses [Tensorflow](https://www.tensorflow.org/js) pre-trained ML models. Given an URL, it returns predictions how likely the image falls into each of the classes - Drawing, Neutral, Sexy, Porn and Hentai. More details on the usage and logic behind you can fin in [this blog post](https://blog.sashido.io/content-moderation-service-with-nodejs-tensorflowjs-and-reactjs-part-1-restful-api-service/). \n\n## Automation Engine\n\nThe Automation Engine's purpose is to check how the classification of a certain image corresponds to the parameters you’ve set as safe for your project. \n\n### 1. Moderation Preferences\n\nIn the beginning, it is essential to define which of the five NSFW classes and values can contain disturbing images and require moderation, i.e. which prognoses are considered safe and which toxic for your users. As there are no specific standards and Moderation Preferences are strictly individual for different projects, we'll advise you to think carefully. \n\nTo illustrate what's the idea and setup, let's imagine we need to set preferences for Clothes Styling app. Users upload outfits and exchange fashion ideas. We can assume the type of photos should be mainly Neutral...and some sexy pics during beach season. So we will add all other classes to our moderation preferences. Something like:\n\n```JSON\n{  \n  \"Sexy\": { \"min\": 0.6, \"max\": 1 },\n  \"Drawing\": { \"min\": 0.4, \"max\": 0.8 },\n  \"Porn\": { \"min\": 0.4, \"max\": 0.8 },\n  \"Hentai\": { \"min\": 0.2, \"max\": 0.8 }\n}\n```\n**The Automation Engine will auto-reject all images that are classified above the `max` limit set in our preferences and approve all that are below the `min` value.** More details on how to fine-tune the parameters for your project you can find in the article [here](https://blog.sashido.io/content-moderation-service-with-nodejs-and-tensorflow-part-2-moderation-automation/).\n\nThe moderation preferences will be saved into a moderationScores config parameter for the production application, as that will allow you to modify them on the fly if needed. \n\n### 2. Automation Business Logic\n\nThe Automation Engine is implemented with [Parse Server Cloud Code Trigger](https://docs.parseplatform.org/cloudcode/guide/#aftersave-triggers). An afterSave trigger, hooked to the user-generated collection automatically checks newly uploaded photos and marks them as either safe, deleted or for moderation. The afterSafe contains logic for getting the predictions from the API and info if an image is safe or toxic for your users, according to the parameters that you define. Based on all data passed, the decision is made and the result is saved to your database.\n\n![](https://media-blog.sashido.io/content/images/2020/07/auto_readme2.gif)\n\nN.B.! The afterSave automation is hooked to a collection UserImage by default. To use straight away after deployment, you should either keep the same class name or change with the respective one [here](https://github.com/SashiDo/content-moderation-automations/blob/master/src/cloud/triggers.js#L3).\n\n### 3. DB Schema\n\nTo store automation results, you need to add the following columns to the DB collection that stores users’ images(UserImage in our case). \n\n**isSafe(Boolean)** - if an image is safe according to the moderation preferences, the Automation Engine will mark ```isSafe - true```. \n\n**deleted(Boolean)** - all inappropriate images that are over the ```max``` values of the moderation preferences, are marked ```deleted - true```. Those pictures won’t be automatically deleted from the file storage. If you do not want to keep track of disturbing images, implement the deletion logic into the afterSave.   \n\n**moderationRequired(Boolean)** - All images that are neither toxic nor safe require manual moderation!\n\n**NSFWPredictions(Array)** - stores the NSFW classification for this image.\n\nThe automation engine will take care to fill those respectively, once a photo is uploaded. 🙂\n\n\n# Installation \u0026 Configuration\n\n## Requirements:\n\n- Node.JS \u003e= 10.2.1\n\n- Mongo DB\n\n- Parse Server\n\n## Download the project\n\nClone the repo:\n\n```\ngit clone https://github.com/SashiDo/content-moderation-automations.git\n```\n\n## Set Environment Variables\n\nCopy the env.example to .env file and set the environment variables for your local environment with your favorite editor:\n\n```\ncp env.example .env\n```\n\nPlace your MongoDB URI. If your app is hosted at SashiDo, you can use the database URI of your SashiDo project. Find the connection string from the app's Dashboard -\u003e App Settings. Same goes for the files URL.\n\n## Install Dependencies\n\nAs this is a full-featured example, all dependencies are present to the package.json. You only need to run:\n```\nnpm install\n```\n\n## Start the project\n\n```\nnpm run dev\n```\n\n**If everything is okay you should see an output similar to this one**:\n\n```\n[nodemon] 2.0.4\n...\n[nodemon] starting `node index index.js`\n✨  Built in 2.55s.\nnode-pre-gyp ...\n...\nRunning on http://localhost:1337\n⠙ Building index.js...The NSFW Model was loaded successfuly!\n✨  Built in 16.41s.\n```\n\nIf you see the output above, you are ready to play with the Automation Engine! 🙂\n\n# Usage\n\nFor seamless integration on any platform, the project offers clear Express routes communication. Also, you can easily query pictures based on their status from any of the Parse Server SDKs. \n\n## API Routes\n\nWe’ve created **express endpoints, which works with the Image Classiffication and Moderation Automation** and facilitates the integration into a project through the REST API.\n\n- Get NSFW predictions at /api/image/classify with the following cURL request.\n\n```sh\ncurl http://YOUR_PARSE_SERVER_URL/api/image/classify?url=https://nsfw-demo.sashido.io/sexy.png\n```\n\n- Check if an image is safe or toxic, NSFW predictions are also included in the response :), at /api/image/is_safe.\n\n```sh\ncurl http://YOUR_PARSE_SERVER_URL/api/image/is_safe?url=https://nsfw-demo.sashido.io/sexy.png\n```\n\n## Query User Images\n\nCheck the examples below on how to query images from Parse SDKs.\n\n### Parse JS SDK\n\n- GET Images that require **manual moderation**:\n\n```js\nconst query = new Parse.Query(\"UserPicture\");\nquery.equalTo(\"moderationRequired\", true);\nquery.find().then((results) =\u003e {\n console.log(results);\n});\n```\n\n- GET Images that are **non-disturbing** for your audience:\n\n```js\nconst query = new Parse.Query(\"UserPicture\");\nquery.equalTo(\"isSafe\", true);\nquery.find().then((results) =\u003e {\n console.log(results);\n});\n```\n\n- GET Images that **may be not suitable** for your customers:\n\n```js\nconst query = new Parse.Query(\"UserPicture\");\nquery.equalTo(\"isSafe\", false);\nquery.find().then((results) =\u003e {\n console.log(results);\n});\n```\n\n- GET Images that are **toxic** for your users:\n\n```js\nconst query = new Parse.Query(\"UserPicture\");\nquery.equalTo(\"deleted\", true);\nquery.find().then((results) =\u003e {\n console.log(results);\n});\n```\nBasically these are the filters you will need. As they can be queried from the client-side thorugh any of the Parse SDKs, we'll share some examples how to get images that are for manual moderation from Android, iOS and Parse REST API.\n\n### Android SDK\n\n```java\nParseQuery\u003cParseObject\u003e query = ParseQuery.getQuery(\"UserPicture\");\nquery.whereEqualTo(\"moderationRequired\", true);\nquery.findInBackground(new FindCallback\u003cParseObject\u003e() {\n    public void done(List\u003cParseObject\u003e UserPicture, ParseException e) {\n        if (e == null) {\n            Log.d(\"isSafe\", \"Safe images retrieved\");\n        } else {\n            Log.d(\"isSafe\", \"Error: \" + e.getMessage());\n        }\n    }\n});\n```\n\nMore information about how to work with the Android SDK can be found in the [official docs](https://docs.parseplatform.org/android/guide/#queries).\n\n\n### iOS SDK\n```swift\nlet query = PFQuery(className:\"UserImage\")\nquery.whereKey(\"moderationRequired\", equalTo:true)\nquery.findObjectsInBackground { (objects: [PFObject]?, error: Error?) in\n    if let error = error {\n        // Log details of the failure\n        print(error.localizedDescription)\n    } else if let objects = objects {\n        // The find succeeded.\n        print(\"Successfully retrieved images for moderation\")\n        }\n    }\n}\n```\nMore information about how to work with the Parse iOS SDK can be found in the [official docs](https://docs.parseplatform.org/ios/guide/#queries).\n\n\n### REST API\n```sh\ncurl -X GET \\\n  -H \"X-Parse-Application-Id: ${APPLICATION_ID}\" \\\n  -H \"X-Parse-REST-API-Key: ${REST_API_KEY}\" \\\n  -G \\\n  --data-urlencode 'where={\"moderationRequired\": true}' \\\n  http://localhost:1337/1/classes/UserImage\n});\n```\n\nMore details on REST Queries you can find in the oficial [Parse REST API Guide](https://docs.parseplatform.org/rest/guide/#queries). And SashiDo users can test REST requests from a super-friendly [API Console](https://blog.sashido.io/introducing-the-api-console/) that’s built in the Dashboard.\n\n\n# Deployment\n\n## Parse.Configs for production\n\nSet the following [Parse.Configs](https://parseplatform.org/Parse-SDK-JS/api/master/Parse.Config.html) for your production server. \n\n- **moderationScores** object should be saved as a Parse.Config, so preferences can be updated on the fly. \n\n- **moderationAutomation option** of boolean type that allows enabling/disabling content moderation automation. \n\n\n## Environment Variables for production\n\n   For production, you need to set the **NSFW model URL**, **NSFW Model Shape size** and variable for **Automation Configs caching**.\n   \n   - SashiDo stores three NSFW models, each one you can set easily using the following URLs:\n\n     \n| Model URL                                                   | Size   | Shape Size | Accuracy |\n| :---------------------------------------------------------- | :----: | :--------: | :------: |\n| https://ml.files-sashido.cloud/models/nsfw_inception_v3/    | Huge   | 299        | 93%      |\n| https://ml.files-sashido.cloud/models/nsfw_mobilenet_v2/90/ | 2.6 MB | 224        | 90%      |\n| https://ml.files-sashido.cloud/models/nsfw_mobilenet_v2/93/ | 4.2 MB | 224        | 93%      |\n     \n *Please note the Inception_v3 model used for this projects has high RAM/CPU consumption. While the two mobilenet models are far more lightweight.*\n\n### Choose the model and set the following environment variables for your live server:\n\n```sh\nTF_MODEL_URL =  MODEL_URL\nTF_MODEL_INPUT_SHAPE_SIZE = MODEL_SHAPE_SIZE\nCONFIG_CACHE_MS = CONFIG_CAHE_IN_MILISECONDS\n\n# Example\nTF_MODEL_URL = \"https://ml.files-sashido.cloud/models/nsfw_mobilenet_v2/93/\"\nTF_MODEL_INPUT_SHAPE_SIZE = 224\nCONFIG_CACHE_MS = 10000\n```\n\n## Code Deployment\n\n   - **In SashiDo** - This is probably the simplest way to deploy the code in production. At SashiDo we have implemented an automatic git deployment process following the [The Twelve-Factor App](https://12factor.net/) principle.\n\nConnect your [SashiDo app with GitHub](https://blog.sashido.io/how-to-start-using-github-with-sashido-for-beginners/) and next the code can be easily deployed with two simple commands for adding a remote branch and pushing changes.\n\n    git remote add production git@github.com:parsegroundapps/\u003cyour-pg-app-your-app-repo\u003e.git\n    git push -f production master\n\n\n   - **On other providers** - Basically, you need to follow the same steps as for SashiDo Deployment. Simply follow the requirements of your hosting provider when setting environment variables for production and deploying the code.\n   \n\n# What's next?\n\n   - [**Admin Panel**](https://github.com/SashiDo/content-moderation-application) - The final touch of our moderation system, where all images in need of moderation are stacked up in a beautiful interface that allows you to make decisions with just a click. Check the demo [here](https://nsfw-demo.sashido.io/moderator).\n   \n# Contribution\n\n   Thanks for looking at this section. We’re open to any cool ideas, so if you have one and are willing to share - fork the repo, apply changes and open a pull request.:)\n\n# License\n\nCopyright © 2020, CloudStrap AD. See [LICENSE](https://github.com/SashiDo/content-moderation-automations/blob/master/LICENSE) for further details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsashido%2Fcontent-moderation-automations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsashido%2Fcontent-moderation-automations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsashido%2Fcontent-moderation-automations/lists"}