{"id":13856873,"url":"https://github.com/andreaferretti/charade","last_synced_at":"2025-09-10T17:41:02.327Z","repository":{"id":38430940,"uuid":"232312390","full_name":"andreaferretti/charade","owner":"andreaferretti","description":"A server for multilanguage, composable NLP API in Python","archived":false,"fork":false,"pushed_at":"2022-12-08T07:27:36.000Z","size":2682,"stargazers_count":29,"open_issues_count":7,"forks_count":2,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-13T15:06:06.146Z","etag":null,"topics":["nlp","nlp-apis","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreaferretti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-07T11:42:28.000Z","updated_at":"2025-02-20T01:18:18.000Z","dependencies_parsed_at":"2023-01-25T07:15:46.669Z","dependency_job_id":null,"html_url":"https://github.com/andreaferretti/charade","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andreaferretti/charade","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaferretti%2Fcharade","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaferretti%2Fcharade/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaferretti%2Fcharade/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaferretti%2Fcharade/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreaferretti","download_url":"https://codeload.github.com/andreaferretti/charade/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaferretti%2Fcharade/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274496675,"owners_count":25296431,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-10T02:00:12.551Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nlp","nlp-apis","python"],"created_at":"2024-08-05T03:01:16.879Z","updated_at":"2025-09-10T17:41:02.273Z","avatar_url":"https://github.com/andreaferretti.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# 1. Charade\n\n![logo](./charade.png)\n\nA server for multilanguage, composable NLP API in Python.\n\n\u003c!-- TOC --\u003e\n\n- [Charade](#charade)\n  - [Philosophy](#philosophy)\n  - [What Charade is and is not](#what-charade-is-and-is-not)\n  - [Installing](#installing)\n    - [Using Pipenv (recommended)](#using-pipenv-recommended)\n      - [Common errors](#common-errors)\n    - [Using Conda and Pip](#using-conda-and-pip)\n  - [Running](#running)\n  - [Docker running](#docker-running)\n  - [Endpoints](#endpoints)\n  - [Architecture](#architecture)\n  - [Requests](#requests)\n    - [Resumable requests](#resumable-requests)\n  - [Describing services](#describing-services)\n  - [Services](#services)\n    - [Parsing](#parsing)\n    - [NER](#ner)\n    - [Date extraction](#date-extraction)\n    - [Codes extraction](#codes-extraction)\n    - [Fiscal codes](#fiscal-codes)\n    - [Extractive summarization](#extractive-summarization)\n    - [Keyword extraction](#keyword-extraction)\n    - [Sentiment detection](#sentiment-detection)\n    - [Names](#names)\n    - [Topic modeling](#topic-modeling)\n    - [Classification](#classification)\n  - [How to create a new service](#how-to-create-a-new-service)\n  - [Testing](#testing)\n  - [Style guide](#style-guide)\n  - [Organization](#organization)\n\n\u003c!-- /TOC --\u003e\n\n## 1.1. Philosophy\n\nCharade was born as a container where multiple independent natural language\nservices can coexist and interact with each other. In order to develop on\nCharade, it may be useful to understand the reasons behind its implementation.\n\n* multiple analyses can be run over a single text - for instance named entity\n  recognition and sentiment detection - so a request from a user should be\n  able to specify what kind of tasks should be performed on the provided text\n* to avoid repeting work and ensure consistency, one task may be dependent on\n  another: for instance, if both the NER and sentiment analysis rely on the same\n  parsing stage, they will get to see the same tokens, something which would not\n  be guaranteed if the two analyses performed tokenization internally\n* a single task could have many coexisting implementations, so that a developer\n  would be free to experiment with new models without having to interfere with\n  existing ones. The user consuming the service could then be able to request\n  a particular implementation of a task by specifying its name\n* multiple implementations of a single task should offer a consistent\n  interface, in order to ensure that clients or other downstream services can\n  switch between them freely\n* the server should not be restricted to a single (natural) language, and\n  various implementations should be free to decide what languages to support\n* developers implementing various models should be able to choose freely what\n  technology to use, so various services can be implemented on top of NLTK,\n  spaCy, pyTorch, TensorFlow, GenSim... Charade should make it easy to use any\n  of these libraries to implement a particular model, without forcing other\n  developers to adopt the same library\n* one should be able to implement as many tasks and models as desired, while\n  choosing at deploy time which one are supported by the server - i.e. the\n  server should be composable from Lego pieces\n\nTherefore, the process of deploying Charade servers works as follows. The\ndevelopers write various models to perform some tasks, possibly trying\ncompeting implementations in parallel. Various kind of models are already\nprovided with Charade, but you should not shy from writing your own.\n\nOnce the models are ready, one writes an entry point script that actually\nloads only the ones that will be used in production. At every point of the\nprocess, one has available an API offering the existing models, and a user\ninterface to try them.\n\n## 1.2. What Charade is and is not\n\nCharade is a framework that helps teams experimenting with multiple approaches\nto tackle some custom NLP task. It is meant to leverage existing NLP libraries,\nsuch as NLTK or spaCy, and not to replace them. A team using Charade can develop\nand evolve a suite of NLP capabilities - say NER, sentiment analysis and so on -\nwhile maintaining the possibility to customize them on particular datasets, and\ncompose servers where only the relevant capabilities are deployed.\n\nCharade is not itself a library for NLP tasks, although it provides some examples\nof models developed using various libraries. It is not a ready-made component\neither: while some of the models provided can be useful, we expect that teams\nusing Charade will develop and customize their own models. The provided ones\ncan serve as example, or can provide some capabilities in a larger deployment.\n\n## 1.3. Installing\n\n**NB** If you are on MacOS Mojave, make sure to have the XCode headers installed\n\n```\nxcode-select --install\nopen /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg\n```\n\nAlso, OpenMP is required by PyTorch, on MacOS it can be installed by\n\n```\nbrew install libomp\n```\n\n### 1.3.1. Using Pipenv (recommended)\n\nInstall Pipenv if needed (`pip install pipenv`). An introduction to Pipenv\ncan be found [here](https://realpython.com/pipenv-guide/).\n\nCreate a virtual environment related to this project by running `pipenv shell`\nfrom inside the top directory in the project.\n\n1. If you want to **develop Charade**, you can install dependencies with this command:\n\n```\npipenv install --dev\n```\n\nIf you also make the iPython kernel for Charade visible to other environments,\nyou can use\n\n```\npython -m ipykernel install --user --name=\"charade\"\n```\n\nIn this way, you can use any installation of Jupyter to launch the `charade`\nkernel.\n\n2. If instead you want to **try Charade** without developing, then run\n\n```\npipenv install --ignore-pipfile\n```\n\nto install all dependencies.\n\n3. In **both cases**, download the models for `spacy`, `allen-nlp` and `nltk` via\n\n```\npython -m spacy download en\npython -m spacy download it\npython -m spacy download de\n\npython -m nltk.downloader averaged_perceptron_tagger\npython -m nltk.downloader maxent_ne_chunker\npython -m nltk.downloader words\n\nmkdir -p models/allen/pretrained\nwget https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.12.18.tar.gz -O models/allen/pretrained/ner-model-2018.12.18.tar.gz\n```\n\n#### 1.3.1.1. Common errors\n\n**NB** If you get an error that you don't have the right version of Pyhton,\nyou can manage that through PyEnv.  To install PyEnv, see\n[the installation instructions](https://github.com/pyenv/pyenv#installation).\nOn MacOS just run `brew install pyenv`. After having install PyEnv, install\nthe required version of Pyhton, for instance `pyenv install 3.6.8`.\n\nAfter this step, `pipenv` should detect the version of pyenv automatically.\n\n### 1.3.2. Using Conda and Pip\n\nIf you **don't need to develop Charade** itself, you can create a virtual environment\nin Conda by running something like `conda create -n charade python=3.6`, then\nactivate it with `source activate charade` (any other name will do). Then install\ndependencies with Pip inside the environment:\n\n```\npip install -r requirements.txt\n```\n\nFinally, update `spacy` models via\n\n```\npython -m spacy download en\npython -m spacy download it\n```\n\n**NB** The `requirements.txt file` is autogenerated by Pipenv with the command\n`pipenv lock --requirements \u003e requirements.txt` - do not edit this file by hand.\n\n## 1.4. Running\n\nJust define the server in `src/main.py`, then run\n\n```\npython src/main.py\n```\n\nThe existing `main.py` file only contains those models that do not require\na custom training step. The other models are commented. You can launch any of\nthe traning scripts - they are ready, but may be trained on toy datasets, so be\nready to adjust them to your needs - and then uncomment the resulting models in\nthe main script.\n\nOnce you have a running server, you can try some queries. An example query can be\nsent using `examples/request.sh`. You can pass a parameter to select a particular\nrequest, for instance\n\n```bash\nexamples/request.sh reprise\n```\n\nYou can see available examples with `ls examples`.\n\nAlso, there is a frontend available at `http://localhost:9000/app`.\n\n## 1.5. Docker running\n\nThe docker can be built by using `scripts/build-docker.sh`. Then, to run the\ndocker container simply do\n\n```bash\ndocker-compose up\n```\n\n**NB** Since both `uwsgi` and some services (e.g. pytorch) make use of multiple\nthreads, this can cause deadlocks. To avoid them, we need to run the uwsgi command\nwith the option `--lazy-apps` as specified in the `Dockerfile` (see\nhttps://engineering.ticketea.com/uwsgi-preforking-lazy-apps/ for an explanation\nof this mechanism).\nNote that if the uwsgi option `--processes` is \u003e 1, each worker will load the full\napplication and thus the server startup may require **a lot** of time and memory.\nBy employing multiple threads and a single process instead (e.g. `--processes 1 --threads 4`)\nthe server startup is fast enough.\n\n## 1.6. Endpoints\n\nA Charade server had just two endpoints:\n\n* GET `/`: returns a JSON describing the available services\n* POST `/`: post a request with a text and some services to be performed\n\n## 1.7. Architecture\n\nA Charade server is defined by instantiating and putting together various\nservices. Each service is defined by\n\n* a **task**\n* a **service name**\n* optional **dependencies**\n* an actual **implementation**.\n\n**Tasks** are used to denote interchangeable services. For instance, there may\nexist various NER models, possibly using different libraries and technologies.\nIn this case, we will define a `ner` task, with the only requirement that if\nthere are various implementations of `ner`, they need to abide to the same\ninterface.\n\n**Names** are used to distinguish different implementations of the same task.\nThe task/name pair should identify a unique service. For instance, one could\nhave deployed `ner` services named `allen`, `nltk`, `pytorch-crf`, `pytorch-crf-2`.\n\n**Dependencies** can be used to avoid repeating the same task over and over.\nFor instance, a `ner` implementation may (or may not) depend on some implementation\nof the `parse` task, which takes care of tokenization. At runtime, the server\nwill ensure that the `parse` task is executed before `ner`.\n\nThe precise mechanism is as follows. The user request contains a field called\n`tasks`, which contains the list of tasks to be executed on the given chunk of\ntext. For instance:\n\n```json\n  \"tasks\": [\n    {\"task\": \"parse\", \"name\": \"spacy\"},\n    {\"task\": \"ner\", \"name\": \"allen\"},\n    {\"task\": \"dates\", \"name\": \"misc\"}\n  ]\n```\n\nTasks are executed in the order requested by the user. The objects returned by\nthe various tasks populate corresponding fields in a `response` dictionary. For\ninstance, for this request, the `response` object will have the shape\n\n```json\n{\n  \"parse\": ...,\n  \"ner\": ...,\n  \"dates\": ...\n}\n```\n\nEach service can look at the `request` object and the `response` object (the\npart that has been populated so far). In this way, a service can look at the\noutput produced by other services that come before.\n\nIf a dependency for a service has not been requested explicitly by the user,\nthe server will choose *any* implementation of the dependency task and execute\nit *before* the dependent task. For instance, say one has a `ner` service called\n`custom` which depends on `parse`. If the user request contains\n\n```json\n  \"tasks\": [\n    {\"task\": \"ner\", \"name\": \"custom\"},\n    {\"task\": \"dates\", \"name\": \"misc\"}\n  ]\n```\n\nthen the server will choose *any* implementation of `parse` and perform it\nbefore `ner`. This has two advantages:\n\n* duplication is reduced, for instance the parsing and tokenization of the text\n  can be done just once and many other services can consume it\n* one has the guarantee that all services rely on the same tokenization, giving\n  a better consistency.\n\n**Implementations** are defined by writing a class that inherits from\n`services.Service`. The methods to override are `Service.run(request, response)`\nand `Service.describe()` (optional, but recommended).\nThe former has access to\n\n* the user request\n* the part of the response constructed so far\n\nand has to return a dictionary containing the service output. This method can\nraise `services.MissingLanguage` if the language of the request is not\nsupported in the given service. The class should load any needed model in its\nconstructor, to avoid reloading models for each request.\n\nFor instance, a trivial parser that just splits sentences on period and tokens\non whitespace may look like this:\n\n```python\nfrom services import Service\n\nclass SimpleParser(Service):\n    def __init__(self):\n        pass\n\n    def run(self, request, response):\n        text = request['text']\n        debug = request.get('debug', False)\n        result = []\n        start = 0\n        end = 0\n        for sentence in text.split('\\.'):\n            tokens = []\n            for token in sentence.split(' '):\n                start = end + 1\n                end += start + len(token)\n                if debug:\n                    tokens.append({\n                        'text': token,\n                        'start': start,\n                        'end': end\n                    })\n                else:\n                    tokens.append({\n                        'start': start,\n                        'end': end\n                    })\n            result.append(tokens)\n        return result\n```\n\n## 1.8. Requests\n\nThe user requests have the following fields:\n\n* `text`: required, the text to be analyzed\n* `debug`: optional flag, default False. Services can use this flag to decide to\n  include additional information. Also, when this flag is set, the response\n  contains an additional field `debug` with general information, such as\n  timing of the services and the resolved ordering among tasks.\n* `lang`: 2 letter language of the text, optional. Default: autodetect\n* `previous`: see Resumable requests\n* `tasks`: a list of requested tasks, with the shape\n\n```json\n  \"tasks\": [\n    {\"task\": \"parse\", \"name\": \"spacy\"},\n    {\"task\": \"ner\", \"name\": \"allen\"},\n    {\"task\": \"dates\", \"name\": \"misc\"}\n  ]\n```\n\nplus possibly other service-dependent fields.\n\n### 1.8.1. Resumable requests\n\nSay there are two tasks, task A and task B. Task A has a dependency on B, which\nis much slower. When trying various implementations for A, it does not make\nsense to recompute the result of task B again and again. In this case, one\nmay want to issue a request for task B, and then a second request for task A,\n*passing the result* of the previous request. In this way, there will be no\nneed to recompute the result of task B.\n\nIn this case, one can put a field called `previous` in the request. The\ncontent of the field must match the response for the previous request. In this\ncase, the server will resume computation from that point. For instance, a\nuser request may look like this:\n\n```json\n{\n  \"text\": \"Ulisse Dini (Pisa, 14 novembre 1845 ...\",\n  \"tasks\": [\n    {\"task\": \"names\", \"name\": \"misc\"}\n  ],\n  \"previous\": {\n    \"ner\": [\n      {\n        \"text\": \"Ulisse Dini\",\n        \"start\": 0,\n        \"end\": 11,\n        \"label\": \"PER\"\n      },\n      ...\n    ]\n  }\n}\n```\n\nIn this example, the `ner` step is already computed, and does not need to be\nrecomputed again.\n\n## 1.9. Describing services\n\nEach service can be self describing by ovverriding the method `describe(self)`\nof the `Service` class. This can be used to report information about\nsupported languages, dependencies, additional parameters needed in the request,\ntrained models and so on. The class `Service` already defines a basic\nimplementation, while services can add more specific information. Some\nstandard keys to use for this purpose are:\n\n* `langs`: the supported languages; use `['*']` if any languages are supported\n* `extra-params`: an optional list of additional parameters of the request\n  accepted by the service (see example)\n* `models`: a dictionary containing the information about the models used\n  by the service\n\nFor each models, the following parameters are standardized:\n\n* `pretrained`: indicates that the model is included in the library\n* `trained-at`: datetime in ISO format\n* `training-time`: as format `HH:mm:ss`\n* `datasets`: list of datasets on which the model is trained\n* `metrics`: a dictionary of metrics that measure the performance of the model\n* `params`: a dictionary of parameters that were used to train the model\n\nA complete example of response could look like this:\n\n```python\n{\n  'task': 'some-task',\n  'name': 'my-name',\n  'deps': ['parse'],\n  'optional_deps': ['ner'],\n  'langs': ['it', 'en'],\n  'extra-params': [\n    {\n      'name': 'some-param1',\n      'type': 'string',\n      'required': False\n    },\n    {\n      'name': 'some-param2',\n      'type': 'int',\n      'required': True\n    },\n    {\n      'name': 'some-param3',\n      'type': 'string',\n      'choices': ['value1', 'value2'],\n      'required': True\n    }\n  ],\n  'models': {\n    'it': {\n      'pretrained': False,\n      'trained-at': '2019-03-27T16:00:49',\n      'training-time': '02:35:23',\n      'datasets': ['some-dataset'],\n      'metrics': {\n        'accuracy': 0.935,\n        'precision': 0.87235,\n        'recall': 0.77253\n      },\n      'params': {\n        'learning-rate': 0.001,\n        'momentum': 0.8,\n        'num-epochs': 50\n      },\n    },\n    'en': {\n      'pretrained': True\n    }\n  }\n}\n```\n\nYou can use the `extra-params` field to describe additional parameters that\nare required (or optional) for a specific service. Each extra parameter can\ntake the shape\n\n```python\n{\n  'name': \u003cstring\u003e,\n  'type': \u003cstring\u003e,\n  'choices': \u003cstring list?\u003e,\n  'required': \u003cbool\u003e\n}\n```\n\nwhere `type` can take the values `\"string\"` or `\"int\"`, and `choices` can be used\nto optionally constrain the valid values for the parameter.\n\n## 1.10. Services\n\nThe following services are defined. To read the interface: output types\nare written inside `\u003c\u003e`. A trailing `?` denotes that the field is only present\nwhen `debug` is `True` in the user request.\n\n### 1.10.1. Parsing\n\nSplits the text into sentences and the sentences into tokens. The interface\nrequires that the output has the shape\n\n```python\n[\n  [\n    {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'text': \u003cstring?\u003e},\n    ...\n  ]\n]\n```\n\n### 1.10.2. NER\n\nFinds people, organizations, dates, places and other entities in the text.\nThe interface requires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'text': \u003cstring?\u003e, 'label': \u003cstring\u003e},\n  ...\n]\n```\n\n### 1.10.3. Date extraction\n\nFinds and parses dates in the text. The interface requires that the output has\nthe shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'text': \u003cstring?\u003e, 'date': \u003cstring\u003e},\n  ...\n]\n```\n\nwhere `date` is formatted as `yyyy-MM-dd`.\n\n### 1.10.4. Codes extraction\n\nFinds common codes in the text. The interface requires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'text': \u003cstring\u003e, 'type': \u003cstring\u003e, 'lang': \u003clang code\u003e},\n  ...\n]\n```\n\n### 1.10.5. Fiscal codes\n\nExtracts information from fiscal codes. The interface requires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e,\n   'end': \u003cint\u003e,\n   'text': \u003cstring\u003e,\n   'type': \u003cstring\u003e,\n   'lang': \u003clang code\u003e,\n   'correct': \u003cbool\u003e, # if the fiscal code is formally correct\n   'sex': \u003csex code\u003e,\n   'birthdate' \u003cstring\u003e\n  }\n]\n```\n\n### 1.10.6. Extractive summarization\n\nExtracts the sentences from the text that best summarize it. The interface\nrequires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'text': \u003cstring?\u003e},\n  ...\n]\n```\n\nwhere the sentences are in order from most informative to least informative.\n\nIt can require additional (optional) parameters in the request:\n\n* `num-extractive-sentences`: the number of sentences to extract\n\n### 1.10.7. Keyword extraction\n\nExtracts the most relevant keywords from the text. The interface\nrequires that the output has the shape\n\n```python\n[\n  {'text': \u003cstring\u003e},\n  ...\n]\n```\n\nwhere the keywords are in order from most to least relevant. Here we do not\nuse spans, since the important information is the keyword, which is probably\nrepeated many times across the text.\n\nIt can require additional (optional) parameters in the request:\n\n* `num-keywords`: the number of keywords to extract\n\n### 1.10.8. Sentiment detection\n\nDetects the sentiment used in various sentences of the text. The interface\nrequires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'sentiment': \u003cfloat\u003e, 'text': \u003cstring?\u003e},\n  ...\n]\n```\n\nwhere there is an entry for each sentence, and `sentiment` ranges from 0\n(extremely negative) to 1 (extremely positive).\n\n### 1.10.9. Names\n\nExtract names and surnames of people mentioned in the text. It is a more refined\nversion of NER, which just retrieves entities of type PER.\n\nThe interface requires that the output has the shape\n\n```python\n[\n  {'start': \u003cint\u003e, 'end': \u003cint\u003e, 'name': \u003cstring?\u003e, 'surname': \u003cstring?\u003e},\n  ...\n]\n```\n\n### 1.10.10. Topic modeling\n\nDoes a soft clustering of text (for instance using\n[LDA](http://ethen8181.github.io/machine-learning/clustering_old/topic_model/LDA.html)\nor similar techniques). This means that the text is associated to a\ndistribution over topics. Topics themselves are discovered as a word mixture\nfrom the training data. The interface requires that the output has the shape\n\n```python\n{\n  'distribution': \u003carray[float]\u003e,\n  'best-topic': \u003cint\u003e,\n  'best-score': \u003cfloat\u003e,\n  'topics': \u003carray[array[string]]?\u003e,\n}\n```\n\nwhere each topic is represented with the arrary of its most representative\nwords. The `topics` field is only present in debug mode.\n\nIt can require additional (optional) parameters in the request:\n\n* `lda-model`: the name of a pretrained LDA model\n\n### 1.10.11. Classification\n\nDoes a classification of the text in a pre-trained and finite set of possible\nclasses. This means that the text is associated to a distribution over possible\nclasses, of which we only output the most fitting. The interface requires that\nthe output has the shape\n\n```python\n{\n  'category': \u003cstring\u003e,\n  'category_probability': \u003cfloat\u003e,\n  'distribution': \u003cmap[string, float]?\u003e\n}\n```\n\nThe `distribution` field is only present in debug mode.\n\n\n## 1.11. How to create a new service\n\nCreate a new class in a file inside `src/services` which inherits from\n`services.Service`. In this class, make sure to call the `Service` constructor\nto register the service, like this:\n\n```python\nclass SomeService(Service):\n    def __init__(self, langs):\n        Service.__init__(self, 'some-task', 'some-name', [], []) # first required deps, then optional deps\n        ...\n```\nOverride the method `def run(self, request, response)` which implements the\nlogic for your service. The return type for the service should be any\ndictionary.\n\nAlso, override the method `describe(self)` to return information about the\nservice itself. A basic implementation of `describe` is in the `Service`\nclass, so a standard implementation would look like:\n\n```python\ndef describe(self):\n    result = super().describe()\n    result['key'] = value\n    # more keys\n    return result\n```\n\nFor the common keys, see the section on Describing services.\n\nBe sure to check out the following things:\n\n* The return type of `run` should be JSON serializable\n* If your service defines a new task, make sure to document it in the README\n* Otherwise, follow the type convention of existing services for the same task\n* If your service requires some previous step (e.g. parsing), try to add it as\n  a dependency and do not hardcode it inside the service\n* If your service may benefit of some previous step (e.g. extra hints), you can\n  add it as optional dependency; the main task will be performed whether or\n  not the optional dependency is already scheduled, but if the optional dependency is\n  scheduled anyway, it will be executed first.\n* If your service requires an optional parameter in the request, add it\n  in the schema validator in `src/server.py`\n* If you cannot handle a certain language, raise `services.MissingLanguage`\n* If you have a model that needs a training step, follow the conventions under\n  `Organization`\n* If you need an additional library, `pipenv install the-library`, then\n  commit the new `Pipfile` and `Pipfile.lock`. Also remember to keep the\n  requirements file up to date with `pipenv lock --requirements \u003e requirements.txt`.\n* Add tests as needed\n\n## 1.12. Testing\n\nTests are written with [nose](https://nose.readthedocs.io/en/latest/). If you\nhave installed Charade in development mode (`pipenv install --dev`), you can run\ntests with the `nosetests` command.\n\nTests for a particular service should put under `tests/services/test_the_service.py`.\nThe naming convention is so that Nose autodiscovery will find them when\nrunning `nosetests`. Classes and methods should also follow this naming\nconvention:\n\n```python\nclass TestTheThing(TestCase):\n    def test_something(self):\n        ...\n```\n\nYou can also test here classes and functions under `common`. If you need to\ntest something which is only used in training, put it under `common` as well.\n\nTests for Charade itself are placed under `tests` without further nesting.\n\n## 1.13. Style guide\n\n* Follow PEP-8\n* Prefer long names such as `request`, `result`, `token` over `req`, `res`, `tok`\n* But be consistent with libraries: for instance, `spacy` defines `document.ents`\n  Iterate over that as `for ent in documents.ents:`\n* Do not use trailing commas\n* Do not commit models or data - commit scripts to retrieve them\n* All bash scripts use `set -e`, `set -u`\n* Make sure that bash scripts can be called from anywhere (see the existing one\n  for examples)\n\n## 1.14. Organization\n\nFollow a tree similar to the following\n\n```\n.\n├── Pipfile\n├── Pipfile.lock\n├── README.md\n├── TODO.md\n├── data\n│   └── ner\n│       └── ...\n├── examples\n│   ├── request.json\n│   ├── request.sh\n│   ├── request2.json\n│   └── request3.json\n├── models\n│   └── pytorch\n│       └── ner\n│           └── ...\n├── requirements.txt\n├── resources\n│   ├── names\n│   │   └── it.txt\n│   ├── stopwords\n│   │   └── en.txt\n│   └── surnames\n│       └── it.txt\n├── scripts\n│   └── pytorch\n│       └── ner\n│           └── it\n│               ├── 1-get-data.sh\n│               ├── 2-prepare-data.sh\n│               └── 3-train.sh\n├── src\n│   ├── __init__.py\n│   ├── common\n│   │   ├── __init__.py\n│   │   └── pytorch\n│   │       ├── __init__.py\n│   │       └── ner\n│   │           ├── __init__.py\n│   │           └── model.py\n│   ├── main.py\n│   ├── server.py\n│   ├── services\n│   │   ├── __init__.py\n│   │   ├── allen.py\n│   │   ├── misc.py\n│   │   ├── pytorch.py\n│   │   ├── regex.py\n│   │   ├── spacy.py\n│   │   └── textrank.py\n│   └── training\n│       └── pytorch\n│           └── ner\n│               ├── generate_wikiner_vectors.py\n│               └── train.py\n└── tests\n    ├── __init__.py\n    ├── services\n    │   ├── __init__.py\n    │   └── test_textrank.py\n    └── test_server.py\n```\n\nIt should be clear what goes where: `data`, `models`, `resources`, `training`\nand so on. When in doubt, follow existing conventions. The directory `common`\nholds code that should be shared at inference and training time.\n\nUnder `data`, only put data that is needed at training time - everything that\nis needed at inference time goes under `models`. If some data file is needed\nalso at inference time, either\n\n* store the content of the file as a field inside the model, or\n* make sure that the training scripts copy the necessary files from `data` to `models`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreaferretti%2Fcharade","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreaferretti%2Fcharade","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreaferretti%2Fcharade/lists"}