Upgrades (#37)

* Update elastic stack to 6.0.0

Author: webwurst

README.md

@@ -2,15 +2,10 @@
 
 See [docs](docs/index.md) for full recipe content.
 
-Feature comparison of the log shippers in this repo:
-
-| Log Shipper | fluentd | fluent-bit | filebeat |
-| ----------- | ------- | ---------- | -------- |
-| rbac        | [x](manifests/fluentd/rbac.yaml) | tbd | tbd |
-| metadata    | [x](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) | tbd | tbd |
-
 
+This setup is similar to the [`Full Stack Example`](https://github.com/elastic/examples/tree/master/Miscellaneous/docker/full_stack_example), but adopted to be run on a Kubernetes cluster.
 
+There is no access control for the Kibana web interface. If you want to run this in public you need to secure your setup. The provided manifests here are for demonstration purposes only.
 
 
 # Local Setup
@@ -21,41 +16,35 @@ Feature comparison of the log shippers in this repo:
 
 ```bash
 minikube start --memory 4096
-# --vm-driver kvm
 
 minikube dashboard
 # maybe wait a bit and retry
 kubectl get --all-namespaces services,pods
 ```
 
-## Extra configuration for Filebeat
+## Logging with Elasticsearch and fluentd
 
 ```bash
-minikube ssh
-
-sudo sh -c "sed -i 's/^ExecStart=\/usr\/bin\/docker daemon.*$/& --log-opt labels=io.kubernetes.container.hash,io.kubernetes.container.name,io.kubernetes.pod.name,io.kubernetes.pod.namespace,io.kubernetes.pod.uid/' /etc/systemd/system/docker.service"
+kubectl apply \
+  --filename https://raw.githubusercontent.com/giantswarm/kubernetes-elastic-stack/master/manifests-all.yaml
 
-sudo systemctl daemon-reload
-sudo systemctl restart docker.service
+minikube service kibana
 ```
 
-## Logging with Elasticsearch and filebeat, fluentd or fluent-bit
+For the index pattern in Kibana choose `fluentd-*`, then switch to the "Discover" view.
+Every log line by containers running within the Kubernetes cluster is enhanced by meta data like `namespace_name`, `labels` and so on. This way it is easy to group and filter down on specific parts.
 
-```bash
-kubectl apply \
-  --filename https://raw.githubusercontent.com/giantswarm/kubernetes-elastic-stack/master/manifests-all.yaml
-minikube service --namespace logging kibana
-  # for index pattern in Kibana choose `filebeat-*` and `@json.time` for Time-field name
-  # or `fluentd-*`
-  # or `fluent-bit-*`
-```
 
 ## Turn down all logging components
 
 ```bash
-kubectl delete namespace logging
+kubectl delete \
+  --filename https://raw.githubusercontent.com/giantswarm/kubernetes-elastic-stack/master/manifests-all.yaml
 ```
 
+FIXME alternatively
+--selector stack=logging
+
 To delete the whole local Kubernetes cluster use this:
 
 ```bash

docs/index.md

@@ -1,15 +1,15 @@
 +++
 title = "Logging with the Elastic Stack"
 description = "The Elastic stack, also known as the ELK stack, has become a wide-spread tool for aggregating logs. This recipe helps you to set it up in Kubernetes."
-date = "2016-11-21"
+date = "2017-10-30"
 type = "page"
 weight = 50
 tags = ["recipe"]
 +++
 
 # Logging with the Elastic Stack
 
-The Elastic stack, most prominently know as the ELK stack, in this recipe is the combination of Filebeat, Elasticsearch, and Kibana. This stack helps you get all logs from your containers into a single searchable data store without having to worry about logs disappearing together with the containers. With Kibana you get a nice analytics and visualization platform on top.
+The Elastic stack, most prominently know as the ELK stack, in this recipe is the combination of Fluentd, Elasticsearch, and Kibana. This stack helps you get all logs from your containers into a single searchable data store without having to worry about logs disappearing together with the containers. With Kibana you get a nice analytics and visualization platform on top.
 
 ![Kibana](kibana.png)
 
@@ -27,22 +27,14 @@ kubectl apply \
 Now we need to open up Kibana. As we have no authentication set up in this recipe (you can check out [Shield](https://www.elastic.co/products/x-pack/security) for that), we access Kibana through
 
 ```nohighlight
-$ POD=$(kubectl get pods --namespace logging --selector component=kibana \
+$ POD=$(kubectl get pods --selector component=kibana \
     -o template --template '{{range .items}}{{.metadata.name}} {{.status.phase}}{{"\n"}}{{end}}' \
     | grep Running | head -1 | cut -f1 -d' ')
 $ kubectl port-forward --namespace logging $POD 5601:5601
 ```
 
 Now you can open up your browser at `http://localhost:5601/app/kibana/` and access the Kibana frontend.
 
-Now set `filebeat-*` for `index pattern`.
-
-Then, we can choose `json.time` for `time-field name` below.
+Now set `fluentd-*` for `index pattern`.
 
 All set! You can now use Kibana to access your logs including filtering logs based on pod names and namespaces.
-
-## Configuring Curator to change log retention
-
-Included in this recipe, there is a Scheduled Job running [Curator](https://github.com/elastic/curator) once a day to clean up your logs. The pod is set to run at 1 minute past midnight and delete indices that are older than 3 days.
-
-You can change this by editing the ConfigMap named `curator-config`. The definition of the `action_file.yaml` is quite self-explaining for simple set ups. For more advanced configuration options, please consult the [Curator Documentation](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html).