https://github.com/fedora-infra/datagrepper

HTTP API for datanommer and the fedmsg bus
https://github.com/fedora-infra/datagrepper

data data-analysis data-science fedora fedora-project postgres postgresql python

Last synced: 26 days ago
JSON representation

HTTP API for datanommer and the fedmsg bus

• Host: GitHub
• URL: https://github.com/fedora-infra/datagrepper
• Owner: fedora-infra
• License: gpl-2.0
• Created: 2013-01-19T22:19:42.000Z (almost 12 years ago)
• Default Branch: develop
• Last Pushed: 2024-04-15T22:52:47.000Z (7 months ago)
• Last Synced: 2024-04-22T00:42:04.854Z (7 months ago)
• Topics: data, data-analysis, data-science, fedora, fedora-project, postgres, postgresql, python
• Language: Python
• Homepage: https://apps.fedoraproject.org/datagrepper/
• Size: 5.38 MB
• Stars: 41
• Watchers: 13
• Forks: 33
• Open Issues: 36
• Metadata Files:
  • Readme: README.md
  • License: COPYING

Awesome Lists containing this project

README

        
# datagrepper

Datagrepper is a web application and JSON API to retrieve historical messages sent via Fedora Messaging. [Datanommer](https://github.com/fedora-infra/datanommer/) is a seperate project and service that consumes messages from the Fedora Messaging queue and puts them in a database. These messages is what datagrepper queries. 

Datagrepper is curently running in production at https://apps.fedoraproject.org/datagrepper/

## Development Environment

Vagrant allows contributors to get quickly up and running with a datagrepper development environment by automatically configuring a virtual machine. 

The datagrepper Vagrant environment configures configures and enables a datanommer service and database. The datanommer instance is configured to be empty when first provisioned, but to consume messages from the stage Fedora Messaging queue.

### Install vagrant

To get started, run the following commands to install the Vagrant and Virtualization packages needed, and start the libvirt service:

    $ sudo dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager

    $ sudo systemctl enable libvirtd

    $ sudo systemctl start libvirtd

### Checkout and Provision

Next, check out the datagrepper code and run vagrant up:

    $ git clone https://github.com/fedora-infra/datagrepper

    $ cd datanommer

    $ vagrant up

### Interacting with your development datagrepper

After successful provisioning of the Datagrepper vagrant setup, the datagrepper web application will be accessible from your host machine's web browser at

http://datagrepper.test:5000/

### Using the development environment

SSH into your newly provisioned development environment:

    $ vagrant ssh

The vagrant setup also defines 4 handy commands to interact with the service that runs the datagrepper flask application: 

    $ datagrepper-start

    $ datagrepper-stop

    $ datagrepper-restart

    $ dataprepper-logs

Additionally, the following commands are also available for interacting with the datanommer service:

    $ datanommer-consumer-start

    $ datanommer-consumer-stop

    $ datanommer-consumer-restart

    $ datanommer-consumer-logs

### Running the tests

Datanommer is comprised of 3 seperate modules in this single repository. There is a handy script in the top directory of this repo to run the tests on all 3 modules:

    $ ./runtests.sh

However, tests can also be run on a single module by invotking tox in that modules' directory. For example:

    $ cd datanommer.models/

    $ tox

Note, that the tests use virtual environments that are not created from scratch with every subsequent run of the tests. Therefore, **when changes happen to dependencies, the tests may fail to run correctly**. To recreate the virtual envrionments,  run the tests commands with the `-r` flag, for example:

    $ ./runtests.sh -r

or

    $ cd datanommer.models/

    $ tox -r