{"id":20217827,"url":"https://github.com/matrix-org/sytest","last_synced_at":"2025-04-04T23:07:41.068Z","repository":{"id":20165482,"uuid":"23436169","full_name":"matrix-org/sytest","owner":"matrix-org","description":"Black-box integration testing for Matrix homeservers","archived":false,"fork":false,"pushed_at":"2025-03-18T13:29:52.000Z","size":4189,"stargazers_count":73,"open_issues_count":93,"forks_count":59,"subscribers_count":25,"default_branch":"develop","last_synced_at":"2025-03-28T22:13:02.153Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matrix-org.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-08-28T17:05:43.000Z","updated_at":"2025-02-24T10:35:27.000Z","dependencies_parsed_at":"2023-11-16T13:41:31.838Z","dependency_job_id":"4560233e-27c2-46d7-bd6f-1b82a1a432bb","html_url":"https://github.com/matrix-org/sytest","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrix-org%2Fsytest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrix-org%2Fsytest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrix-org%2Fsytest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrix-org%2Fsytest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matrix-org","download_url":"https://codeload.github.com/matrix-org/sytest/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247261603,"owners_count":20910108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-11-14T06:35:51.047Z","updated_at":"2025-04-04T23:07:41.044Z","avatar_url":"https://github.com/matrix-org.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"SyTest\n======\n\nSyTest is an integration testing system for Matrix homeserver implementations;\nprimarily at present the Synapse server. It performs \"black-box\" testing, by\nstarting up multiple home server instances and testing the interaction using\nregular HTTP interaction much as any other Matrix client would do. It can\noutput test results either to an interactive terminal, or as a TAP-style test\nreport, for continuous-integration test harnesses such as Jenkins.\n\nSyTest is written in Perl, and is the original integration testing system for\nMatrix.  New tests tend to be written in Complement_, a Go-based integration\ntesting system.\n\n.. _Complement: https://github.com/matrix-org/complement\n\nInstalling\n----------\n\nSyTest requires a number of dependencies that are easiest installed from CPAN.\n\n1. If these are not being installed directly into the system perl (as root),\n   then you will first have to arrange that ``cpan`` can install to somewhere\n   writable as the non-root user you are running SyTest as, and that ``perl``\n   can see that.\n\n   Personally I arrange this by adding three lines to ``.profile``::\n\n    export PERL5LIB=$HOME/lib/perl5\n    export PERL_MB_OPT=--install_base=$HOME\n    export PERL_MM_OPT=INSTALL_BASE=$HOME\n\n   Alternatively, see https://metacpan.org/pod/local::lib#The-bootstrapping-technique\n\n   If you have edited your ``.profile``, don't forget to ``source`` it again\n   into your running shell.\n\n#. If you have not run ``cpan`` before, it will prompt for answers to several\n   questions when it performs the initial setup. Running it once with no\n   arguments will give you a chance to answer these questions. Most likely you\n   can just let it configure \"automatically\"::\n\n    $ cpan\n\n    CPAN.pm requires configuration, but most of it can be done automatically.\n    If you answer 'no' below, you will enter an interactive dialog for each\n    configuration option instead.\n\n    Would you like to configure as much as possible automatically? [yes] \n\n    ...\n\n    cpan[1]\u003e exit\n    Lockfile removed.\n\n#. Fetch the ``sytest`` source and install its dependencies::\n\n    git clone https://github.com/matrix-org/sytest\n    cd sytest\n    ./install-deps.pl\n    cd ..\n\n#. As ``sytest`` is intended for testing the synapse home server\n   implementation, it is likely you'll want to fetch the source of that as\n   well. By default SyTest will expect to find it in a sibling directory called\n   ``synapse``::\n\n    pip install pynacl --user # Work around pynacl directory sort bug.\n    git clone https://github.com/matrix-org/synapse\n    cd synapse\n    git checkout develop\n    python setup.py develop --user\n    python setup.py test\n    cd ..\n\n   Synapse does not need to be installed, as SyTest will run it directly from\n   its source code directory.\n\nAdditionally, a number of native dependencies are required. To install these\ndependencies on an Ubuntu/Debian-derived Linux distribution, run the following::\n\n    sudo apt install libpq-dev build-essential\n\nInstalling on OS X\n------------------\nDependencies can be installed on OS X in the same manner, except that packages\nusing NaCl / libsodium may fail. This can be worked around by:\n\nInstalling libsodium manually, eg.::\n\n    $ brew install libsodium\n\nand confirm it is installed correctly and visible to pkg-config. It should give\nsome configuration output, rather than an error::\n\n    $ pkg-config --libs libsodium\n    -L/usr/local/Cellar/libsodium/1.0.8/lib -lsodium\n\nThen force an install of Crypt::NaCl::Sodium::\n\n    $ cpan\n    cpan\u003e force install Crypt::NaCl::Sodium\n\nYou may also need to force install Shell::Guess, and manually install\nDBI before DBD::Pg, otherwise DBD::Pg will fail with::\n\n    No rule to make target '.../auto/DBI/Driver_xst.h'\n\nThen run install-deps.pl as normal.\n\nRunning\n-------\n\nTo run SyTest with its default settings, simply invoke the ``run-tests.pl``\nscript with no additional arguments::\n\n    cd sytest\n    ./run-tests.pl\n\nIf the Synapse source is checked out in a different location, this can be set\nwith ``--synapse-directory``::\n\n    ./run-tests.pl --synapse-directory /home/synapse/synapse\n\nIf it is necessary to run the synapse server with a particular python\ninterpreter (for example, to run it within a virtualenv), this can be done\nusing ``--python``::\n\n    ./run-tests.pl --python ../synapse/env/bin/python\n\nIf you're seeing the following error when trying to run against Synapse,\nmake sure you have your Synapse Python virtualenv(venv) activated: ``source ../synapse/env/bin/Activate``::\n\n    $ ./run-tests.pl\n    ...\n    ** Error starting server-0 (on port 8800): python exited 1: /Users/myuser/.pyenv/versions/3.8.6/bin/python: Error while finding module specification for 'synapse.app.homeserver' (ModuleNotFoundError: No module named 'synapse')\n    ❌\n\n    $ source ../synapse/env/bin/Activate\n    (synapse) $ ./run-tests.pl\n    ...\n    ✅\n\nTo obtain greater visibility on why a particular test is failing, two\nadditional options can be passed to print extra information. The\n``--client-log`` flag (shortened to ``-C``) will print HTTP requests made and\nresponses received::\n\n    ./run-tests.pl -C\n\nThe ``--server-log`` flag (shortened to ``-S``) will print lines from the\nSynapse server's standard error stream::\n\n    ./run-tests.pl -S\n\nTo run only a subset of tests in certain files, name the files as additional\narguments::\n\n    ./run-tests.pl tests/20profile-events.pl\n\nTo run synapse with a specific logging configuration, create a YAML file\nsuitable for dictConfig_ called ``log.config`` (it can be copied from a running\nsynapse) and place it within the homeserver configuration directory\n(``localhost-\u003cport\u003e``).\n\nA blacklist file can be used to mark certain tests as expected to fail. These\ntests will still be run, but their failure will not cause the entire test suite\nto fail. To use a blacklist file pass the ``--test-blacklist-file`` option\n(shortened to ``-B``). For example, if you are testing Synapse, you will\nprobably want to use Synapse's sytest-blacklist_::\n\n    ./run-tests.pl -B synapse/sytest-blacklist\n\n.. _dictConfig: https://docs.python.org/2/library/logging.config.html#logging.config.dictConfig\n.. _sytest-blacklist: https://github.com/matrix-org/synapse/blob/develop/sytest-blacklist\n\nPlugins\n~~~~~~~\n\nSytest supports plugins. Plugins follow the same project structure as sytest and can be placed\nin the ``plugins`` directory. They should contain the ``lib/SyTest/HomeserverFactory`` and\n``lib/SyTest/Homeserver``, or ``lib/SyTest/Output`` directories, similar to the root of the sytest repository.\nThe path of the plugins directory can be overridden via the ``SYTEST_PLUGINS`` environment variable.\n\nCurrently only ``Homeserver`` and ``Output`` implementations are supported in plugins.\n\nSee https://github.com/valkum/sytest_conduit for an example of a plugin.\n\nDeveloping\n----------\n\nFor more information on developing SyTest itself (maintaining or writing new\ntests) see the `DEVELOP.rst` file.\n\n\nPostgres Template Database\n--------------------------\n\nWhen testing with postgres SyTest will check if there is a database named\n`sytest_template` and will create the test databases using that as a template.\nThis can be used to greatly reduce the time to create databases as they don't\nneed to be created from scratch.\n\nThe easiest way to create the template database is to start a HS pointing at\nthe database and stop it once the database has been created.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatrix-org%2Fsytest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatrix-org%2Fsytest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatrix-org%2Fsytest/lists"}