https://github.com/sr-murthy/doc_classifier

A simple experimental document classification tool based on a domain-dependent, keywords-based document class map and a keyword frequency score
https://github.com/sr-murthy/doc_classifier

document-classification

Last synced: 5 months ago
JSON representation

A simple experimental document classification tool based on a domain-dependent, keywords-based document class map and a keyword frequency score

• Host: GitHub
• URL: https://github.com/sr-murthy/doc_classifier
• Owner: sr-murthy
• License: gpl-3.0
• Created: 2020-01-21T12:48:12.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2024-04-09T18:13:50.000Z (almost 2 years ago)
• Last Synced: 2024-12-27T06:43:56.288Z (over 1 year ago)
• Topics: document-classification
• Language: Python
• Homepage:
• Size: 74.2 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
Document Classifier

===================

Document classification tool based on a domain-dependent, keywords-based document class map and a simple keyword frequency score.

Currently only financial statements (in CSV format) can be classified. A keyword-based class map for a given document type (stored as a JSON file in `static`) is used to create a frequency score for keywords occurring in a user-specified list of columns in the CSV document.

Scoring & Classification Functions

----------------------------------

The scoring function, for a given CSV document and a list of keywords, is given by

$$

score(W, C) = \frac{\sum_{w_i \in W} \sum_{c_{j,k} \in C} I_{i,j,k}}{r \cdot m \cdot n}

$$

where $W$ is the set of $r$ keywords $w_1,\ldots,w_r$ to search for, $C$ is the user-defined set of $n$ columns $C_1,\ldots,C_n$ in which to perform the keyword search - the $j$-th column containing $m$ strings $c_{1,j},\ldots,c_{m,j}$ - and $I_{i,j,k}$ is an indicator function for the presence of keywords in the $m \cdot n$ column entries $c_{j,k}$ given by

$$

I_{i,j,k} = \begin{cases}

            1, \hskip{3em} w_i \in c_{j,k} \\

            0, \hskip{3em} w_i \in c_{j,k}

            \end{cases}

$$

Note: the scoring function is guaranteed to be a value between 0 and 1 (inclusive) as the frequency score (numerator in the scoring function) can be a maximum of $r \cdot m \cdot n$.

Given a CSV document $\mathcal{D}$ of type $\mathcal{T}(\mathcal{D})$, with $t$ classes $L_1,\ldots,L_t$ defined in its class keywords map (a JSON file with keys being the class names/IDs $L_t,\ldots,L_t$ and values being lists of keywords associated with the classes), and $C$ being the user-defined set of $n$ columns $C_1,\ldots,C_n$ in which to perform the keywords search, the classification function is given by

$$

classify(D, C) = argmax_{L_i \in L} \hskip{1em} score(W(L_i), C)

$$

Usage

-----

Here's a simple example of classifying a sample income statement with only a single keywords-search column .

    [path/to/doc_classifier/src]$ ./classify.py -t 'financial statements' -f ../sample_data/income_statement/microsoft.csv --verbose

    Classification: income

    Keywords score map: {

        "income": 0.03428571428571429,

        "cash flow": 0.013333333333333334,

        "balance sheet": 0.0

    }

This is an example of classifying a sample income statement with multiple keywords-search columns.

    [/path/to/doc_classifier/src]$ ./classify.py -t 'financial statements' -f ../sample_data/income_statement/microsoft2.csv  -c 'line item 1, line item 2' --verbose

    Classification: income

    Keywords score map: {

        "income": 0.025714285714285714,

        "cash flow": 0.006666666666666667,

        "balance sheet": 0.0

    }