{"id":15026464,"url":"https://github.com/symfony-doge/veslo","last_synced_at":"2026-03-16T21:32:13.587Z","repository":{"id":57063673,"uuid":"130989672","full_name":"symfony-doge/veslo","owner":"symfony-doge","description":":rowboat: A vacancy aggregation engine","archived":false,"fork":false,"pushed_at":"2021-02-09T10:24:24.000Z","size":8667,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-19T20:27:28.247Z","etag":null,"topics":["aggregator","php73","vacancy"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/symfony-doge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-25T10:26:59.000Z","updated_at":"2022-12-12T16:33:23.000Z","dependencies_parsed_at":"2022-08-24T10:10:17.044Z","dependency_job_id":null,"html_url":"https://github.com/symfony-doge/veslo","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symfony-doge%2Fveslo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symfony-doge%2Fveslo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symfony-doge%2Fveslo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symfony-doge%2Fveslo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/symfony-doge","download_url":"https://codeload.github.com/symfony-doge/veslo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243330320,"owners_count":20274039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregator","php73","vacancy"],"created_at":"2024-09-24T20:04:31.248Z","updated_at":"2025-12-28T01:23:50.745Z","avatar_url":"https://github.com/symfony-doge.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cpre\u003e\n                                                  ________________________  \n                                                 /                        \\ \n,--.                                            /                          |\n|  |___________________________________________/_____________________      |\n|  |_________________________________________________________________)     |\n|  |                                           \\                           |\n`--'                                            \\                          |\n                                                 \\________________________/ \n        Veslo: a vacancy aggregation engine\n\u003c/pre\u003e\n\n[![Packagist Version](https://img.shields.io/packagist/v/symfony-doge/veslo.svg)](https://packagist.org/packages/symfony-doge/veslo)\n[![Quality Score](https://img.shields.io/scrutinizer/g/symfony-doge/veslo.svg)](https://scrutinizer-ci.com/g/symfony-doge/veslo)\n[![Code Climate technical debt](https://img.shields.io/codeclimate/tech-debt/symfony-doge/veslo.svg)](https://codeclimate.com/github/symfony-doge/veslo)\n[![Packagist](https://img.shields.io/packagist/l/symfony-doge/veslo.svg?color=blue)](https://github.com/symfony-doge/veslo/blob/master/LICENSE)\n\n## Installation\n\n### Docker\n\nThe preferred way to install is through [docker-compose](https://docs.docker.com/compose).\nYou need to have a [Docker](https://docs.docker.com/install) daemon at least [17.05.0-ce](https://docs.docker.com/engine/release-notes/#17050-ce)\n(with build-time `ARG` in `FROM`) to successfully cook all containers.\n\nRun an automated deploy script for local development with Docker.\n\n```\n$ bin/deploy_dev.sh\n```\n\n![installation asciicast](.github/images/installation.gif)\n\nRoadmap configurations were not automatically loaded by the script,\nyou need to insert them manually with a separate command.\n\n```\n$ docker-compose run --rm app bin/console doctrine:fixtures:load \\\n    --group roadmap.configuration.parameters \\\n    --append\n```\n\nUp to 120 sec may be required for starting up the vacancy parsing process.\n\n### Manual\n\nYou can clone and deploy the application with your own environment\nby providing `.env` file using `.env.dev.dist` as a template.\nBelow are steps to prepare your application before accessing through a web server.\n\n```\n$ git clone git@github.com:symfony-doge/veslo.git veslo \u0026\u0026 cd \"$_\"\n\n// setup your own connection parameters\n$ cp .env.dev.dist .env\n\n$ cp app/config/parameters.yml.dist app/config/parameters.yml\n$ cp webpack.env.dev.js.dist webpack.env.js\n```\n\nCheck your environment for Symfony requirements\nand install PHP dependencies via [Composer](https://getcomposer.org/download).\nYou also need the [php-ds](https://github.com/php-ds/ext-ds) extension\nfor some services working with efficient PHP 7 data structures.\n\n```\n$ bin/symfony_requirements\n$ composer install\n```\n\nInstall Javascript dependencies and compile assets via [Yarn](https://yarnpkg.com/lang/en/docs/install)\n([Node.js](https://nodejs.org/en/download) 10.x is required).\n\n```\n$ yarn install\n$ yarn run build:dev\n```\n\nApply database migrations.\n\n```\n$ bin/console doctrine:migrations:migrate latest\n```\n\nLoad roadmap configurations.\n\n```\n$ bin/console doctrine:fixtures:load --group roadmap.configuration.parameters --append\n```\n\n## Testing\n\nLoading fixtures.\n\n```\n$ docker-compose run --rm app bin/console doctrine:fixtures:load --group test\n```\n\nApplying [Codeception](https://codeception.com/docs/reference/Commands) parameters.\n\n```\n$ cp parameters.codeception.dev.yml.dist parameters.codeception.yml\n```\n\nExecuting tests.\n\n```\n$ docker-compose run --rm app bin/codecept run --steps\n```\n\n### Workflow\n\n```\nFOUND (initial place)\n|\n| to_parse\n ------------\u003e PARSED\n               |\n               | to_collect\n                ------------\u003e COLLECTED\n                              |\n                              | to_index\n                               ------------\u003e INDEXED\n```\n\nThere is a set of log files suited for debugging an each workflow transition.\n`var/logs` contains:\n\n| Log file | Description |\n| :--- | :--- |\n| `dev.anthill.digging-YYYY-mm-dd.log` | Search process (to_parse) |\n| `dev.anthill.parsing-YYYY-mm-dd.log` | Parsing process (to_collect) |\n| `dev.anthill.collecting-YYYY-mm-dd.log` | Saving in local storage (to_index) |  \n| `dev.sanity.indexing-YYYY-mm-dd.log` | Analysis |\n| `dev.app.workflow-YYYY-mm-dd.log` | Queue get/push events\u003cbr /\u003e (data distribution between workers) |  \n| `dev.app.http-YYYY-mm-dd.log` | Dumping all HTTP requests/responses\u003cbr /\u003e (works if `app.http_client.logging` is `true`) |\n| `dev-YYYY-mm-dd.log` | A common Symfony log file |\n\n## API\n\n```\n$ docker-compose run --rm app bin/console veslo:anthill:digging hh php --iterations=1\n```\n\nFinds a vacancy from job site by roadmap (search plan) and configuration.\nSearch result will be offered to parsing queue,\naccording to current workflow (default `veslo.app.workflow.vacancy_research.to_parse`).\n\nAvailable roadmaps are defined in `AnthillBundle/Resources/config/roadmaps.yml`\nand configurations in `AnthillBundle/Resources/fixtures/roadmap/`.\n\n```\n$ docker-compose run --rm app bin/console veslo:anthill:parsing --iterations=1\n```\n\nPolls a new raw vacancy data (html/json etc.) for parsing.\nResult will be offered to collecting queue\n(default `veslo.app.workflow.vacancy_research.to_collect`).\n\n```\n$ docker-compose run --rm app bin/console veslo:anthill:collecting --iterations=1\n```\n\nGrabs a parsed vacancy data (instance of `AnthillBundle/Dto/Vacancy/RawDto`) \nand decides whether should it be collected for analysis or not.\nResult will be persisted in the local storage and offered to the indexing queue\n(default `veslo.app.workflow.vacancy_research.to_index`)\n\n```\n$ docker-compose run --rm app bin/console veslo:sanity:indexing --iterations=1\n```\n\nSends an accepted vacancy to the microservice for analysis (see `ANALYSER_HOST`, `ANALYSER_PORT`)\nand persists received metadata in the local storage. Remote service implements\nAPI defined by [symfony-doge/ministry-of-truth-client](https://github.com/symfony-doge/ministry-of-truth-client) bridge.\nExample (Go + Gin): [symfony-doge/ministry-of-truth-cis](https://github.com/symfony-doge/ministry-of-truth-cis).\n\n### Currently supported roadmaps:\n\n* [hh](https://hh.ru) (configurations: `php`, `javascript`, `golang`, `python`, `java`, `c++`)\n\n## Usage example\n\n[www.veslo.it](https://veslo.it)\n\n## See also\n\n- [symfony-doge/management-kit](https://github.com/symfony-doge/management-kit) — A set of monitoring \u0026 management GUI tools to manage data \u0026 processes of this project.\n\n## Changelog\nAll notable changes to this project will be documented in [CHANGELOG.md](CHANGELOG.md).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsymfony-doge%2Fveslo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsymfony-doge%2Fveslo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsymfony-doge%2Fveslo/lists"}