https://github.com/cloudfoundry-community/kibana-me-logs
The application hosts Kibana 3 and a proxy that binds to your Logstash/Elastic Search backend service. You can now see your logs in Kibana!
https://github.com/cloudfoundry-community/kibana-me-logs
Last synced: 9 months ago
JSON representation
The application hosts Kibana 3 and a proxy that binds to your Logstash/Elastic Search backend service. You can now see your logs in Kibana!
- Host: GitHub
- URL: https://github.com/cloudfoundry-community/kibana-me-logs
- Owner: cloudfoundry-community
- Created: 2015-03-10T16:13:04.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2016-11-29T21:19:57.000Z (over 9 years ago)
- Last Synced: 2024-04-14T22:47:40.897Z (about 2 years ago)
- Language: HTML
- Homepage:
- Size: 981 KB
- Stars: 17
- Watchers: 32
- Forks: 13
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Kibana Me Logs
==============
Draining logs from your Cloud Foundry hosted application to a backend Logstash/Elastic Search is great - it drains them all day long. Except, you can't see them. They are stored in Elastic Search and you have no way to access either Elastic Search nor a Kibana UI to view the logs.
**Events over time:**
**Line-by-line logs**
The application hosts Kibana 3 and a proxy that binds to your Logstash/Elastic Search backend service. You can now see your logs in Kibana!
This is a work-in-progress and/or a stop-gap until a better multi-tenant solution exists. It's primary weakness is it lacks authentication.
It assumes that users are getting Logstash via the Docker/Logstash Service Broker. See below for details.
For users, there is the complimentary `cf kibana-me-logs` CLI plugin https://github.com/cloudfoundry-community/cf-plugin-kibana-me-logs
For administrators, there is `./bin/upgrade-all.sh` to systematically upgrade all kibana-me-logs applications on your Cloud Foundry.
View logs for a specific app
----------------------------
If you are binding one logstash service to many applications in a space, then you might want a way to only see the logs for one app at a time.
`http://kibana.DOMAIN/#/dashboard/file/app-logs-01a4ad6a-51b1-450b-ab8d-ef5b836bb8cb.json`
Put the GUID for the application into the URL above.
Authentication
--------------
If you provide `kibana-me-logs` the `KIBANA_USERNAME` and `KIBANA_PASSWORD` environment variables
it will use them as HTTP Basic Auth credentials to help secure your log data.
Security weaknesses
-------------------
If you do not use SSL to connect to kibana-me-logs and are using the HTTP Basic Auth,
your credentials will not be encrypted.
It requires the running of a shared proxy that grants users access to any backend service. Currently it doesn't have any authentication pass-thru.
But, it might provide you something useful in the meantime. Users can see and search their logs. And each other's logs.
Assumptions
-----------
It is assumed that:
- you already have an application running on your own Cloud Foundry
- you are using the Docker/Logstash Service Broker (see below) or a service broker that provides matching credentials schema
- you have already bound your application to your logstash service instance and Cloud Foundry is already draining logs into it
- you have the Go programming language installed on your machine
For example:
```
cf create-service logstash14 free my-logstash-service
cf bind-service my-app my-logstash-service
cf restart my-app
```
You can confirm that your application is bound to the service:
```
$ cf services
Getting services in org system / space dev as admin...
OK
name service plan bound apps status
my-logstash-service logstash14 free my-app available
```
Usage
-----
To view your application's logs in Kibana you need to deploy the `kibana-me-logs` application and also bind it to the same `my-logstash-service` service instance as above:
```
cd /tmp; rm -rf kibana-me-logs
git clone https://github.com/cloudfoundry-community/kibana-me-logs
cd kibana-me-logs
cf push kibana-myapp --no-start --random-route -b https://github.com/cloudfoundry/go-buildpack
cf bs kibana-myapp my-logstash-service
cf start kibana-myapp
```
Now view your Kibana UI in your browser. It should redirect to a url like `http://kibana-myapp.apps.1.2.3.4.xip.io/#/dashboard/file/logstash.json` automatically and start showing your logs.
If you are a regular Go user, you can also fetch the application using:
```
go get github.com/cloudfoundry-community/kibana-me-logs
cd $GOPATH/src/github.com/cloudfoundry-community/kibana-me-logs
```
Docker/Logstash Service Broker
------------------------------
A requirement for this application is that the service binding credentials to logstash/elasticsearch fit a certain schema. This is the schema that comes from the Docker/Logstash Service Broker.
The easiest way to deploy this service broker is with the docker-services-boshworkspace.
See its README for detailed setup instructions for administrators.
When running `bosh setup deployment` choose "Logstash 1.4" as the service to be deployed.
Logstash Service Brokers
------------------------
If you are building your own service broker for Logstash/Elastic Search, then this application assumes that the binding credentials look like:
```json
{
"name": "my-logstash-service",
"credentials": {
"hostname": "10.10.5.251",
"ports": {
"514/tcp": "49160",
"9200/tcp": "49161",
"9300/tcp": "49159"
}
},
"syslog_drain_url": "http://10.10.5.251:49160",
...
}
```
The `9200/tcp` port is the port for the Elastic Search API.
If your Elastic Search API is hosted on a different hostname than the remainder of the ELK cluster then please submit a PR/issue so we can discuss what to do next. Happy to help.
Your service bindings should also include a `syslog_drain_url` URI. Cloud Foundry will use this to automatically setup a continuous syslog drain of your application's logs into your logstash/elastic search service instance.