Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/release-engineering/kobo

A pile of python modules used by Red Hat release engineering to build their tools.
https://github.com/release-engineering/kobo

Last synced: 8 months ago
JSON representation

A pile of python modules used by Red Hat release engineering to build their tools.

Awesome Lists containing this project

README 

          
kobo

====



A collection of Python utilities.



[![Build Status](https://travis-ci.org/release-engineering/kobo.svg?branch=master)](https://travis-ci.org/release-engineering/kobo)

[![Coverage Status](https://coveralls.io/repos/github/release-engineering/kobo/badge.svg?branch=master)](https://coveralls.io/github/release-engineering/kobo?branch=master)



Development

===========



Use a virtualenv for development.

For example, install kobo and dependencies in editable mode:



    virtualenv ~/kobo-dev

    . ~/kobo-dev/bin/activate

    pip install --editable ~/src/kobo

    pip install -rtest-requirements.txt



To run the test suite:



- Run `py.test` to run tests against installed versions

  of python and dependencies, or...

- Install and run `tox` to run test suite against a matrix of supported

  python and Django versions (more thorough, slower).



Please submit pull requests against https://github.com/release-engineering/kobo.



Changelog

=========



kobo 0.41.0

-----------



### FEATURES & IMPROVEMENTS



- hub: improved Django 5.2 compatibility by modernizing middleware code



### BUG FIXES



- hub: fixed enable-worker command to properly refresh ready state when workers are re-enabled

- hub: added missing Django migration to avoid migration warnings



kobo 0.40.0

-----------



### FEATURES & IMPROVEMENTS



- hub: Task model now records the user who cancelled a task.

- client: `HubProxy` now retries PermissionDenied errors to mitigate false

  negatives that might appear immediately after login.



kobo 0.39.0

-----------



### FEATURES & IMPROVEMENTS



- client: `HubProxy` now accepts `transport_args` to customize the

  instantiation of the default transport

- `kobo.pkgset.FileCache` now implements `__contains__` for improved

  performance



kobo 0.38.0

-----------



### FEATURES & IMPROVEMENTS



- hub: introduced `VIEW_RAW_LOG_EXTENSIONS` setting



kobo 0.37.0

-----------



### FEATURES & IMPROVEMENTS



- hub: get_worker_tasks XMLRPC function now has an option to also return ASSIGNED tasks



kobo 0.36.2

-----------



### BUG FIXES



- worker: fixed another logging deadlock introduced in kobo 0.36.0



kobo 0.36.1

-----------



### BUG FIXES



- hub: fixed display of permissions in user details template

- worker: fixed a logging deadlock introduced in kobo 0.36.0

- worker: fixed a locked TaskManager sometimes opening tasks



kobo 0.36.0

-----------



### FEATURES & IMPROVEMENTS



- worker: on `SIGHUP`, will now gracefully await task completion and re-exec



### BUG FIXES



- client: the `--hub` option to `watch-log` command is no longer ignored

- worker: logging thread is more robust, recovers from many errors

- hub: minor fixes to worker detail UI

- hub: respond with 404 error on missing logs, rather than 500

- other minor fixes



kobo 0.35.1

-----------



### BUG FIXES



- client: fixed support for proxies (`https_proxy`) in XML-RPC client



kobo 0.35.0

-----------



### FEATURES & IMPROVEMENTS



- Support Django 5.0



### BUG FIXES



- hub: restricted login redirects to same site only

- hub: fixed assignment of tasks to workers with incompatible arch

- hub: fixed display of arches/channels on worker detail page

- hub: avoid SELinux relabelling issue during file upload

- Minor cleanups of obsolete or deprecated code



kobo 0.34.0

-----------



### FEATURES & IMPROVEMENTS



- hub: the `/` character may now be used within usernames



### BUG FIXES



- hub: uploads of the same name but different content are no

  longer considered duplicates



kobo 0.33.0

-----------



### FEATURES & IMPROVEMENTS



- hub: added a page for listing failed tasks, alongside the existing

  Running/Finished task lists.



kobo 0.32.0

-----------



### FEATURES & IMPROVEMENTS



- hub: introduced `inherit_worker` argument to `create_subtask` API, allowing workers

  to ensure created subtasks are assigned to themselves



kobo 0.31.0

-----------



### FEATURES & IMPROVEMENTS



- hub: introduced `settings.USERS_ACL_PERMISSION` to adjust access to user list



### BUG FIXES



- hub: minor fixes to page titles



kobo 0.30.1

-----------



### FEATURES & IMPROVEMENTS



- hub: deleting a worker no longer deletes associated tasks



kobo 0.30.0

-----------



### BUG FIXES



- hub: decorate Task.save with transaction.atomic

- hub: do not call save twice in Task.save()

- hub: fix Task.cancel_subtasks

- client: allow file upload size to be sent a string

- django: use BigIntegerField for file upload sizes

- hub: return error 404 for non-existent logs

- hub: move filename resolution to _streamed_log_response

- worker: do not repeatedly remove already awaited tasks from subtask list

- django: reimplement django_version_ge in pure Python

- hub: do not use workers without ids in a filter

- hub: replace pipes with shlex



### FEATURES & IMPROVEMENTS



- Make xmlrpc timeout configurable

- django: remove support for Django 1.10 and older

- hub: update `Worker.update_worker` documentation



kobo 0.29.1

-----------



### BUG FIXES



- Fixed XML-RPC authentication regression introduced in kobo 0.29.0



kobo 0.29.0

-----------



### FEATURES & IMPROVEMENTS



- Introduced `create-worker` command

- Improved compatibility with Django 4.2

- Improved output from admin commands on XML-RPC errors

- Cleaned up unused code



kobo 0.28.0

-----------



### BUG FIXES



- Renamed deprecated get_query_set to get_queryset

- Changed repr to properly deserialize tracebacks sent over XML-RPC

- Fixed remaining Django 3 deprecations

- Fixed reporting of failed uploads

- Changed behavior that log decoding failure doesn't raise an error

- Made it possible to interrupt assigned tasks



### FEATURES & IMPROVEMENTS



- Retired python 2 support

- Added Client Credentials Flow OIDC authentication



kobo 0.27.0

-----------



### FEATURES & IMPROVEMENTS



- Added logging for authN/authZ events

- Added `--hub` option to client



kobo 0.26.0

-----------



### BUG FIXES



- Fixed calculating the number of task's subtasks in new Django versions

- Fixed cli/help_RST compatibility with Python 3

- fixed encoding errors by forcing the usage of UTF-8 in str and bytes functions



### FEATURES & IMPROVEMENTS



- Added an option to make subtask inherit parents priority and make it configurable



kobo 0.25.0

-----------



### FEATURES & IMPROVEMENTS



- Improved compatibility with Django 4.1

- Improved compatibility with python 3



kobo 0.24.2

-----------



### BUG FIXES



- Fixed a crash from Django dumpdata command when using Django 2 and later.



kobo 0.24.1

-----------



### BUG FIXES



- Fixed an encoding error when using a python 2 XML-RPC client 

  against a python 3 kobo hub.



kobo 0.24.0

-----------



### FEATURES & IMPROVEMENTS



- It is now possible to provide a custom task manager implementation into the kobo

  worker's main loop. This can be used to alter the worker's task scheduling or

  execution behavior.



kobo 0.23.0

-----------



### FEATURES & IMPROVEMENTS



- Introduced `AUTH_METHOD=oidc` for XML-RPC clients. If this authentication method

  is enabled, the OIDC endpoint introduced by kobo 0.22.0 will be used.

- `HubProxy` now respects the system configuration for kerberos realm when

  `KRB_REALM` has not been set.



kobo 0.22.0

-----------



### FEATURES & IMPROVEMENTS



- hub: added an endpoint for OIDC login. Note that kobo itself does not

  implement any OIDC workflow, which may instead be implemented by the

  hosting web server.

- `HubProxy` will now retry on additional cases of XML-RPC error.



kobo 0.21.0

-----------



### BUG FIXES



- Fixed a crash bug in `LoggingThread` when mixing bytes and text.



### FEATURES & IMPROVEMENTS



- Added methods to `HubProxy` for explicitly logging into hub.



kobo 0.20.3

-----------



### BUG FIXES



- Many compatibility fixes for modern Python and Django versions.

- Fixed loss of error details when `LoggingThread` fails.



kobo 0.20.2

-----------



### FEATURES & IMPROVEMENTS



- Improved Django 4.x compatibility



kobo 0.20.1

-----------



### BUG FIXES



- `kobo.rpmlib.get_keys_from_header` now correctly handles RPMs using header-only signing.

  Also fixes `pkgset.SimpleRpmWrapper.signature`.



kobo 0.20.0

-----------



### FEATURES & IMPROVEMENTS



- kobo workers now maintain a `last_seen` field which may be used to monitor the

  availability of workers.



kobo 0.19.0

-----------



### FEATURES & IMPROVEMENTS



- The `resubmit-tasks` now supports adjusting the priority of the resubmitted task.

  The client and server must both upgrade to kobo 0.19.0 in order to use this feature.



### BUG FIXES



- Fixed backwards-incompatible change in kobo 0.18.0: if a kobo-derived client

  provided its own `--profile` argument, this could clash with the new argument

  in kobo.



kobo 0.18.0

-----------



### FEATURES & IMPROVEMENTS



- The kobo client now supports a `--profile` argument to select an alternative

  configuration file. To enable this functionality, users of kobo should provide

  a `default_profile` and `configuration_directory` when creating a

  `CommandOptionParser`.



### BUG FIXES



- Fixed a python3 compatibility issue in kobo.http.

- Fixed a crash bug for projects using the default `XMLRPC_TEMPLATE` for the

  XML-RPC help page with Django > 1.



kobo 0.17.0

-----------



### FEATURES & IMPROVEMENTS



- Introduced the concept of global variables to `kobo.conf.PyConfigParser`.

  This feature allows a variable to be declared in one file and overridden

  in a subsequently imported file.



### CHANGES



- The `client.get_worker_info` XML-RPC method no longer requires authentication.

  It was formerly restricted to superusers only.



kobo 0.16.0

-----------



### REMOVED



- The HubProxy auto_logout feature was removed. This feature was unreliable and

  could occasionally result in deadlocks, due to the usage of finalizers.

  It is recommended to call `logout()` from a `finally` block in cases where it's

  important to ensure a logout.



kobo 0.15.1

-----------



### BUG FIXES



- Fixed a Django 1.x compatibility issue introduced in kobo 0.15.0



kobo 0.15.0

-----------



### FEATURES & IMPROVEMENTS



- Improved Django 3.x compatibility



kobo 0.14.0

-----------



### FEATURES & IMPROVEMENTS



- Introduced `gssapi` login method, for improved Python3 compatibility.

  This login method requires the hub's `krb5login` endpoint to be enabled.

  The default login method is unchanged.



kobo 0.13.0

-----------



### BUG FIXES



- Fixed a Python3 compatibility issue in hub migrations

- Improved Django 1.11 compatibility



kobo 0.12.0

-----------



### BUG FIXES



- Improved Django 1.11 and 2.x compatibility



kobo 0.11.0

-----------



### FEATURES & IMPROVEMENTS



- The resubmit-tasks command now accepts a `--nowait` argument



### BUG FIXES



- Fixed usage of `shortcuts.run` with text mode in Python 3

  ([#133](https://github.com/release-engineering/kobo/issues/133))

- Fixed decoding crashes in `shortcuts.run` when an incomplete multibyte

  sequence is read

  ([#119](https://github.com/release-engineering/kobo/issues/119))

- Fixed rpmlib attempting to decode binary data in RPM headers

- Fixed various Python 3 compatibility issues



kobo 0.10.0

-----------



### FEATURES & IMPROVEMENTS



- Improved Python 3 compatibility, particularly in xmlrpc and rpmlib modules

- Improved Django 1.11 and 2.x compatibility



### REMOVED



- Testing of Python 2.6 has been dropped; some kobo functionality still works

  with Python 2.6, but may break without warning in later releases

- Support for Django 1.6 has been removed



kobo 0.9.0

----------



### BUG FIXES



- Fixed LoggingThread on Python 3 ([#66](https://github.com/release-engineering/kobo/issues/66))

- Fixed `kobo.django.xmlrpc` migrations for Django 2.x

- Fixed some exceptions discarded without logging ([#32](https://github.com/release-engineering/kobo/issues/32))

- Fixed some reliability issues in `kobo.xmlrpc`



kobo 0.8.0

----------



### FEATURES & IMPROVEMENTS



- Improved Python 3 compatibility

- Improved Django 2.0 compatibility

- Improved tests coverage

- Header produced by kobo.shortcuts.run(show_cmd=True) is now limited to 79 characters length



### BUG FIXES



- Fixed handling of string SERVER_PORT in wsgi requests

- Fixed Worker.timeout_task wrongly setting subtasks to INTERRUPTED ([#72](https://github.com/release-engineering/kobo/issues/72))

- Fixed Worker.set_task_weight always crashing ([#75](https://github.com/release-engineering/kobo/issues/75))



kobo 0.7.0

----------



### FEATURES & IMPROVEMENTS



- Improved Python 3 compatibility

- Improved tests coverage



kobo 0.6.0

----------



### FEATURES & IMPROVEMENTS



- kobo worker name no longer needs to match host FQDN

- improved error reporting when loading configuration

- improved error reporting from kobo.shortcuts.run

- reduced memory usage when handling large log files

- models now respect settings.AUTH_USER_MODEL



### BUG FIXES



- fixed crash on xml-rpc client in python <= 2.7.9

- fixed spurious whitespace from kobo.shortcuts.run (#40)

- fixed missing migration for User model



kobo 0.5.0

----------



### FEATURES & IMPROVEMENTS



- kobo.shortcuts.run now supports all Popen keyword arguments

- resubmit-tasks has a --force argument to resubmit successful tasks

- new watch-log command for watching a log from CLI

- admin UI now covers user model



### BUG FIXES



- kobo.shortcuts.run now resumes on interrupted system calls

- worker load no longer includes assigned but unstarted tasks



kobo 0.4.0

----------



### FEATURES & IMPROVEMENTS



- pkgset.SimpleRpmWrapper has now checksum_type member

- threads.run_in_threads helper function



### Django 1.5 update



Django part of kobo was udpated to be compatible with 1.5 release. Lower

version are no more supported. As a side-effect, only python 2.6+ is

supported by django part.



From same reason there will be no builds of kobo-django package for RHEL 5

and lower as it lacks required python version.



- RemoteUserMiddleware is now used for KrbV authentication

- LimitedRemoreUserMiddleware can be used to authenticate only on entry

pages.

- LongnameUser model is used for auth backend. It allows 255 characters long

user names (as they come from KrbV)

- Class-based generic views ExtraListView and ExtraDetailView. object_list

helper stays for compatibility, but it is deprecated now and will be removed

in future.



kobo 0.3.0

----------



### FEATURES & IMPROVEMENTS



- State machine implementation - StateEnum, db field, form fields

- Brand new HTML template, media, views, urls and menu

- Menu supports django.root

- Task logs, javascript log watcher, threaded worker stdout logger

- XML-RPC help pages display list of contents

- Kerberos support in CookieTransport

- JSONField to store dicts and lists in database

- Add relative_path() and split_path() functions to shortcuts



### kobo.pkgset.FileWrapper



file_name attribute renamed to file_path.

file_name is now a property which returns actual file name.



Action:

Change file_name to file_path in your code.



### Username hack



Username hack is enabled by default now (when kobo.django.auth is used).

It changes username to 255 characters and also overrides validation RE.



Action:

On postgresql run: ALTER TABLE auth_user ALTER username TYPE VARCHAR(255);

Sqlite users have to use db_update-0.2.0-0.3.0 script.



### Worker FQDN checking



Each worker's name must match it's FQDN now.

This prevents cut&paste configuration errors when tasks end in INTERRUPTED state.



Action:

Change worker names to FQDN.

Change related usernames as well.



### Changes in kobo.plugins



Removed 'lower_case' attribute.

Plugins are now subclassed when a container is created.

Each plugin now contains 'container' attribute.



Action:

Remove 'lower_case' attribute from plugins, do whatever is necessary in 'normalize_name() class method instead.



- Improve PluginContainer inheritance. Also add 'container' attribute to each plugin class obtained from a container instance. (Daniel Mach)

- Plugins are now subclassed when a container is created. (Daniel Mach)



### kobo.hub.models.Task refactoring



Field 'traceback' moved to a file (traceback.log).

Field 'result' content dumped to a file (stdout.log), it is supposed to contain actual task result.

Field 'args' changed to JSONField and data is directly available without any conversion.



Action:

Run db_update-0.2.0-0.3.0 script.



### Configuration handling in kobo.client and kobo.worker



Configuration no longer uses os.environ to get config file path.

ClientCommandContainer, HubProxy and TaskManager constructor has a new mandatory 'conf' argument.



Action:

    config_file = os.environ.get("_CONFIG_FILE", "/etc/.conf")

    conf = kobo.conf.PyConfigParser()

    conf.load_from_file(config_file)

    ... and pass conf to ClientCommandContainer, HubProxy or TaskManager



### Configuration passed to tasks



TaskBase constructor has now a mandatory argument 'conf', which is automatically set in TaskManager.



Action:

Add 'conf' argument to tasks classes with custom constructor.