https://github.com/softmarshmallow/inked-engine

🤖 natural language processing out of the box
https://github.com/softmarshmallow/inked-engine

django nlp nltk python

Last synced: about 1 month ago
JSON representation

🤖 natural language processing out of the box

• Host: GitHub
• URL: https://github.com/softmarshmallow/inked-engine
• Owner: softmarshmallow
• Created: 2018-07-13T11:33:22.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2022-11-22T02:47:41.000Z (almost 2 years ago)
• Last Synced: 2024-10-03T13:24:07.235Z (about 2 months ago)
• Topics: django, nlp, nltk, python
• Language: Jupyter Notebook
• Homepage:
• Size: 87.7 MB
• Stars: 9
• Watchers: 2
• Forks: 0
• Open Issues: 9
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# inked engine 🤖🤖

| 뉴스 분석을 위한 툴킷입니다. 

Main features

* news data indexing

* news data processing

* provide api for service server

  

Inked-news-crawler 에서 새로운 뉴스데이터를 받아온후, 인덱싱과 pre-proccessing 을 합니다. 서비스 서버에서 요청하는 정보를 분석하여 서비스 서버로 전달하며, 서비스 서버에서 클라이언트로 뉴스 정보를 제공합니다.

## News data model

- tags : { company : [], namedEntities: [], keywords: []}

- content

- origin

- title

- time

# How to install virtualenv:

### Install **pip** first

    sudo apt-get install python3-pip

### Then install **virtualenv** using pip3

    sudo pip3 install virtualenv

### Now create a virtual environment

    virtualenv venv

## KoNlPy setup

http://konlpy.org/en/v0.4.4/install/

`sudo apt-get install g++ openjdk-8-jdk`

`bash <(curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh)`

## start the engine server

`daphne server.asgi:application`

## supervisor ctrl

restart server

`sudo supervisorctl restart asgi_daphne`

## IMPORTANT:: seed credential files

you can see 

```gitignore

server/settings/production.py

credentials/db-connection.json

```

from `.gitignore` which two files you will have to provide manually to run this project.

## modules

- duplicate news checker ✅

- spam news detector 🚫

- word2vec ✅ (wiki) 🚫 (news)

## used by

* [wor.io](https://github.com/softmarshmallow/wor.io)

* [inked-server](https://github.com/softmarshmallow/inked-server)

## developed by

develped by softmarshmallow