{"id":37084322,"url":"https://github.com/jchenghu/sacreeos","last_synced_at":"2026-01-14T10:18:41.012Z","repository":{"id":169184188,"uuid":"560124347","full_name":"jchenghu/sacreeos","owner":"jchenghu","description":"SacreEOS is a signature generator and implementation helper for the Self Critical Sequence Training","archived":false,"fork":false,"pushed_at":"2024-04-21T23:56:32.000Z","size":117,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-26T03:24:26.613Z","etag":null,"topics":["cider","computer-vision","deep-learning","image-captioning","reinforcement-learning","self-critical-sequence-training"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2305.12254","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jchenghu.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-10-31T19:37:50.000Z","updated_at":"2024-04-21T23:56:36.000Z","dependencies_parsed_at":"2023-07-07T03:15:45.580Z","dependency_job_id":"2d2bdd43-e646-4be8-9b78-46ccecbb9ba3","html_url":"https://github.com/jchenghu/sacreeos","commit_stats":null,"previous_names":["jchenghu/sacreeos"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jchenghu/sacreeos","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchenghu%2Fsacreeos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchenghu%2Fsacreeos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchenghu%2Fsacreeos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchenghu%2Fsacreeos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jchenghu","download_url":"https://codeload.github.com/jchenghu/sacreeos/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchenghu%2Fsacreeos/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28416857,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:18:03.274Z","status":"ssl_error","status_checked_at":"2026-01-14T10:16:11.865Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cider","computer-vision","deep-learning","image-captioning","reinforcement-learning","self-critical-sequence-training"],"created_at":"2026-01-14T10:18:40.545Z","updated_at":"2026-01-14T10:18:41.004Z","avatar_url":"https://github.com/jchenghu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n### SacreEOS\n\nSacreEOS [ [ICIAP2023](https://link.springer.com/chapter/10.1007/978-3-031-43148-7_4) ] [ [Arxiv](https://arxiv.org/abs/2305.12254) ] is a signature generator and \nimplementation helper for the Self Critical Sequence Training [ [Arxiv](https://arxiv.org/abs/1612.00563) ]. \u003cbr\u003e\n\n### Motivation\n\nSacreEOS main goal consists of spreading awareness about easy-to-overlook aspects\nof the Self-Critical Squence Training (SCST), in particular over the `End-of-Sequence (EOS)`\ntoken. Currently most Image Captioning projects follow two different approaches, we call for simplicity, \nStandard and No\u0026lt;Eos\u0026gt;. The two versions differ in few subtle implementation details that significantly \nimpact the outputs and performances, ultimately posing an obstacle to the evaluation and comparison of models.\n\n\nThe generation of shareable signatures serves the purpose of spreading awareness about the issue,\nincrase transparency over previous works and guiding future works into an informed implementation.\nTo this end, the package provides also useful classes in the hope of saving time and preventing headaches \nto those implementing SCST for the first time.\n\n\n### Features\n\n##### Main feature\n\n\u003cul\u003e\n    \u003cli\u003e SacreEOS signature generation.\n    The package provides an easy interface for the generation of signatures\n    that inform regarding the key aspects of the SCST implementation. This can be done\n    either manually or automatically using the implementation helper.\u003cbr\u003e\u003cbr\u003e\n    \u003c/li\u003e\n\u003c/ul\u003e\n\n##### Optional features\n\nThe adoption of SacreEOS as an implementation helper is completely optional.\nEstablished projects are only invited to manually generate the SacreEOS signature.\n\n\n\u003cul\u003e\n    \u003cli\u003e Mode and metric selection. \n    The user can customize the target SCST configuration according to its needs. The package currently \n    supports 2 SCST modes, 4 metrics and 2 bases. \u003cbr\u003e\u003cbr\u003e\n    \u003c/li\u003e\n    \u003cli\u003e Implementation helper. \n    Through a collection of exceptions and input conditions the user is guided toward an informed selection of SCST implementation. \n    \u003cbr\u003e\u003cbr\u003e\u003c/li\u003e\n    \u003cli\u003e Efficiency. \n    For each metric, originally proposed in Python, an additional and optional C implementation \n    is provided. It can be toggled by setting one of the loss computation arguments. In case of \n    platform incompatibilities, the portability is still preserved thanks to the Python implementation. \u003cbr\u003e\u003cbr\u003e\n    \u003c/li\u003e\n\u003c/ul\u003e\n\n\n- - -\n\n### Installation\n\nRequirements:\n\u003cul\u003e\n    \u003cli\u003e torch, numpy, typing.\n    \u003c/li\u003e\n    \u003cli\u003e Python \u003e= 3.7\n    \u003c/li\u003e\n\u003c/ul\u003e\n\n\nYou can install the package using pip:\n```\npython -m pip install sacreeos\n```\nor clone the repository:\n```\ngit clone https://github.com/jchenghu/sacreeos\n```\n\n- - -\n\n### Usage\n\n\n#### Signature generation\n\n##### Manual \n\nThe SacreEOS signature can be generated in two ways.\nIn case you installed the package using `pip` simply write: \n```\n$ python3.7 -m sacreeos\n```\notherwise in case you cloned the repository:\n```\n$ cd sacreeos\n$ python sacreeos\n```\n\nIt will ask few information regarding the SCST implementation. A default\nanswer is provided in most cases based on the most popular configurations.\n\nExamples of the two most common signatures: \u003cbr\u003e\n`STANDARD_wInit+Cider-D[n4,s6.0]+average[nspi5]+1.0.0` \u003cbr\u003e\n`NO\u003cEOS\u003e_wInit+Cider-D[n4,s6.0]+average[nspi5]+1.0.0` \u003cbr\u003e\n\n\n\n##### Automatic\n\nThe signature is automatically generated in case the package is adopted as \nimplementation helper. Once the `Scst` class is constructed, the \nsignature can be generated using the method `get_signature()`.  We refer to the sections below.\n\n#### Implementation helper\n\nSacreEOS provide the `Scst` class which computes the SCST loss over the inputs using the\nselected metrics and performs a series of assersions and input checks based on the selected\nclass.\n\n`Scst` Constructor: \n```\nclass: Scst(scst_class, metric_class, base_class, eos_token, corpus_refss=None,\n            verbose=True, metric_args=None, base_args=None):\n--------------------------------------------------------------------------------------\nArguments:\n\n    \u003c\u003e scst_class: scst mode class\n              - Choices: Scst.SCST_CONFIG_STANDARD, Scst.SCST_CONFIG_NO_EOS                \n\n    \u003c\u003e metric_class: metric class to be selected \n              - Choices: Scst.METRIC_BLEU: requires, Scst.METRIC_CIDER, Scst.METRIC_CIDER_D, Scst.METRIC_CIDER_R.\n    \n    \u003c\u003e base_class: base reward type \n              - Choices: Scst.BASE_AVERAGE, Scst.BASE_GREEDY.\n    \n    \u003c\u003e eos_token: str : end of sequence token\n    \n    \u003c\u003e corpus_refss: list of list of sentences. \n                     The document frequencies calculation won't be affected by how sentences \n                     are placed inside the list of list, in fact, the format is required only\n                     for sake of consistency with the one required by the loss computation\n\n    \u003c\u003e verbose: bool\n    \n    Further Customizations: \n        leave these arguments to None to keep the standard configurations.\n\n    \u003c\u003e metric_args: dictionary with metric arg name as key and custom arg values as value\n            - Scst.METRIC_BLEU: requires no args\n            - Scst.METRIC_CIDER:  {'n': int}                # max number of ngram   \n            - Scst.METRIC_CIDER_D : {'n': int,  \n                                     'sigma': float}        # gaussial length penalty deviance\n            - Scst.METRIC_CIDER_R: {'n': int,               \n                                    'repeat_coeff': float,  # length and repeatition penalty weights, the two must sum to 1\n                                    'length_coeff': float,  \n                                    'alpha': float}         # gaussian length penalty deviance\n\n    \u003c\u003e base_args: dictionary with base arg name as key and custom arg values as value\n            - {'nspi': int}  # number of samples per input/image.\n\n\n\n```\n\nScst loss computation method:\n```\nmethod compute_scst_loss(sampled_preds: List[List[str]], sampled_logprobs: torch.tensor,\n                         refss: List[List[str]], base_preds=None, eos_pos_upthresh=None,\n                         reduction='mean', get_stat_data=False):\n--------------------------------------------------------------------------------------\nARGUMENTS:\n\n    \u003c\u003e sampled_preds: List[List[str]] of shape (batch_size, nspi): is the list of predicted sequences.    \n\n    \u003c\u003e sampled_logprobs: tensor of shape (batch size, nspi, seq_len): refers to the log-probabilities of the predicted sequences.           \n            -  The function does not perform padding on `sampled_logprobs`. The value of padding elements must be zero.\n             Because of the popular practice of sub-word tokenization (such as BPE), this method cannot perform padding\n             safely by itself by aligning the elements in `sampled_logprobs` and `sampled_preds`.\n\n    \u003c\u003e base_preds: None or List[List[str]] of shape (batch_size, nspi): if not None, it contains the base sequences.\n\n    \u003c\u003e refss: List[List[str]] of shape (batch_size, int): is the list of references list.\n            - refss[i] are expected to be the references of sampled_preds[i]\n            - * can be any number \n    \n    \u003c\u003e eos_pos_upthresh (end of sequence position upper threshold) defines the length up until which the\n     method will ensure the `eos_token` termination. Since SCST is a learning process, in some cases,\n     especially in the early iterations, sampled sequences may not end with the `eos_token` because they\n     reached the maximum sequence length defined by the model. In case of `None`, is set to the last dimension of `sampled_logprobs` argument.\n\n    \u003c\u003e It should preferably be set to the model's max_len if no sub-word techniques are applied, but\n       None (hence the `sampled_logprobs` last dimension) should ensure a wide enough error catching web.\n\n    \u003c\u003e reduction: str: reduction method (None, 'sum', 'mean').\n    \u003c\u003e get_stat_data: bool: if true, return not only the loss, but also reward and base reward\n    \nRETURN:\n      (r-b) * (-sampled_logprobs.sum(dim=-1))\n    Where:\n        \u003c\u003e r represents the reward of `sampled_preds`\n        \u003c\u003e b is the reward calculated on base sentences.\n```\n\nFor automatic signature generation, call `get_signature()` on the class instance.\n\nAn example of usage can be found in the demo.\n\n\n- - - \n\n### Demo\n\nThe demo sub-package provide an usage example of SacreEOS helper functionality. \u003cbr\u003e\nIt implements a small image captioning system to be trained with SCST.\n\nThe demo requires a pre-trained model and data sampled from COCO. They\ncan be found in this [drive](https://drive.google.com/drive/folders/1dCFLY0zBRKlV3QQlv6AiadzKaatnQHG8?usp=share_link)\n(~170MB), all contents are expected to be placed in `/demo/data/`. \u003cbr\u003e\n\nThe directories and files in `/demo/` should look like this:\n```\ndemo\n├── data\n│   ├── demo_data.pickle            -\u003e model pre-trained on XE and training samples\n│   ├── coco_train_refss.pickle     -\u003e COCO corpus references\n│   ├── dataset.py                  \n│   ├── sample_coco_refss.py        -\u003e additional samples for testing\n│   └── sample_coco_test.py\n├── demo_launcher.py                -\u003e training script\n├── __init__.py\n├── net\n    ├── layers.py                   -\u003e model sub-layers\n    ├── model.py                    -\u003e model definition and sampling procedures\n    └── utils.py                    -\u003e naive utility functions\n\n```\n\nThe training script does not require arguments and can be launched as follows:\n```\npython demo_launcher.py\n```\n\nThis is the expected result, although it may be different because of stochastic factors and\nsmall dataset sample:\n\n```\nStandard Scst training started...\nStandard Scst signature: STANDARD_wInit+Cider-D[n4,s6.0]+average[nspi5]+1.0.0\nTraining epoch 49: 100%|████████████████████████| 50/50 [45:20\u003c00:00, 54.42s/it]\nNo artifacts.\nNote: because of the poor train set, the standard case may present few artifacts as well\nNoEos Scst training started...\nNoEos scst signature: NO\u003cEOS\u003e_wInit+Cider-D[n4,s6.0]+average[nspi5]+1.0.0\nTraining epoch 49: 100%|████████████████████████| 50/50 [42:36\u003c00:00, 51.12s/it]\nThere are 531 artifacts.\nEnd.\n```\n\n\n- - -\n\n### Credits\n\nThis project is based on the work of [Rennie et al., 2017](https://arxiv.org/abs/1612.00563)\nand inspired by [SacreBLEU (Matt Post, 2018)](https://arxiv.org/abs/1804.08771).\n\nReference:\n\n```\n@InProceedings{hu2023sacreeos,\nauthor=\"Hu, Jia Cheng\nand Cavicchioli, Roberto\nand Capotondi, Alessandro\",\neditor=\"Foresti, Gian Luca\nand Fusiello, Andrea\nand Hancock, Edwin\",\ntitle=\"A Request for Clarity over the End of Sequence Token in the Self-Critical Sequence Training\",\nbooktitle=\"Image Analysis and Processing -- ICIAP 2023\",\nyear=\"2023\",\npublisher=\"Springer Nature Switzerland\",\naddress=\"Cham\",\npages=\"39--50\",\nabstract=\"The Image Captioning research field is currently compromised by the lack of transparency and awareness over the End-of-Sequence token (\u003cEos\u003e) in the Self-Critical Sequence Training. If the \u003cEos\u003e token is omitted, a model can boost its performance up to +4.1 CIDEr-D using trivial sentence fragments. While this phenomenon poses an obstacle to a fair evaluation and comparison of established works, people involved in new projects are given the arduous choice between lower scores and unsatisfactory descriptions due to the competitive nature of the research. This work proposes to solve the problem by spreading awareness of the issue itself. In particular, we invite future works to share a simple and informative signature with the help of a library called SacreEOS. Code available at: https://github.com/jchenghu/sacreeos.\",\nisbn=\"978-3-031-43148-7\"\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjchenghu%2Fsacreeos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjchenghu%2Fsacreeos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjchenghu%2Fsacreeos/lists"}