Skip to content

Commit 0bb7748

Browse files
anapsixanapsix
committed
initial implemnation with lame bash script
0 parents  commit 0bb7748

25 files changed

+8403
-0
lines changed

‎.dockerignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.git/
2+
helm/
3+
logs/
4+
*.md

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
logs/

‎Dockerfile

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
FROM alpine:3.11
2+
ENV FILEBEAT_VERSION=7.6.2
3+
ENV FILEBEAT_HOME=/opt/filebeat
4+
ENV FILEBEAT_USER=filebeat
5+
RUN apk upgrade -U && \
6+
apk add dumb-init bash jq curl coreutils libc6-compat && \
7+
rm -rf /var/cache/apk/* && \
8+
addgroup -g 1000 ${FILEBEAT_USER} && \
9+
adduser -h ${FILEBEAT_HOME} -H -D -u 1000 -G ${FILEBEAT_USER} -s /bin/false ${FILEBEAT_USER} && \
10+
mkdir /logs && \
11+
curl -sS https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -o /tmp/filebeat.tar.gz && \
12+
tar -C /opt -xzf /tmp/filebeat.tar.gz && \
13+
rm /tmp/filebeat.tar.gz && \
14+
mv /opt/filebeat-${FILEBEAT_VERSION}-linux-x86_64 ${FILEBEAT_HOME} && \
15+
cp ${FILEBEAT_HOME}/fields.yml ${FILEBEAT_HOME}/fields.yml.reference && \
16+
mkdir ${FILEBEAT_HOME}/data && \
17+
chown -R ${FILEBEAT_USER}:${FILEBEAT_USER} ${FILEBEAT_HOME} /logs && \
18+
ln -s ${FILEBEAT_HOME}/filebeat /usr/local/bin/filebeat && \
19+
filebeat version
20+
COPY config/filebeat.yml ${FILEBEAT_HOME}/
21+
COPY config/fields.yml ${FILEBEAT_HOME}/
22+
COPY config/templates/index-template.json.tpl ${FILEBEAT_HOME}/
23+
COPY scripts/get_cloudflare_logs.sh /usr/local/bin/
24+
COPY scripts/docker-entrypoint.sh /usr/local/bin/
25+
VOLUME ["/logs"]
26+
USER ${FILEBEAT_USER}
27+
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
28+
CMD ["/usr/local/bin/docker-entrypoint.sh"]

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2020 Anastas Dancha
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
Retrieving Cloudflare logs via [Logpull API][logpull], and pushing them
2+
into Elasticsearch with [Filebeat][filebeat].
3+
4+
5+
## Local Development
6+
7+
### Build
8+
9+
```sh
10+
docker build -t get-logs .
11+
```
12+
13+
### Launch
14+
15+
> Before launching, make sure to set your CF credentials as environment variables
16+
> ```
17+
> export CF_ZONE_ID=51e241f08e014feb95d1b2760228d12a
18+
> export [email protected]
19+
> export CF_AUTH_KEY=51e241f08e014feb95d1b2760228d12a2df50
20+
> ```
21+
> or modify [`docker-compose.yaml`][docker-compose.yaml] appropriately (see docs on [`env_file`][compose-env-file], and [`environment`][compose-environment] usage)
22+
23+
After launching local environment, access Kibana via http://localhost:5601/app/kibana#/discover.
24+
25+
26+
#### With Docker Compose
27+
28+
```sh
29+
# launch Elasticsearch, Kibana, and get-logs container instances
30+
docker-compose up -d
31+
32+
# keep an eye on the logs
33+
docker-compose logs -f get-logs
34+
```
35+
36+
#### Launch manually
37+
38+
```sh
39+
# launch Elasticsearch container instance
40+
docker run -d \
41+
--name es \
42+
-p 9200:9200 \
43+
-e "discovery.type=single-node" \
44+
docker.elastic.co/elasticsearch/elasticsearch:7.6.2
45+
46+
# launch Kibana container instance
47+
docker run -d \
48+
--name ki \
49+
-p 5601:5601 \
50+
--link es:elasticsearch \
51+
docker.elastic.co/kibana/kibana:7.6.2
52+
53+
# launch Cloudflare Logpull container instance
54+
docker run -it --rm \
55+
-e CF_AUTH_EMAIL \
56+
-e CF_AUTH_KEY \
57+
-e CF_ZONE_ID \
58+
-e SAMPLE_RATE="0.01" \
59+
-e ES_HOST="http://elasticsearch:9200" \
60+
-e ES_INDEX="cloudflare-test" \
61+
-e ES_INDEX_SHARD=6 \
62+
-e ES_INDEX_REPLICAS=0 \
63+
-e ES_INDEX_REFRESH=10s \
64+
--link es:elasticsearch \
65+
get-logs
66+
```
67+
68+
[link reference]::
69+
[logpull]: https://developers.cloudflare.com/logs/logpull-api/
70+
[filebeat]: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-overview.html
71+
[compose-env-file]: https://docs.docker.com/compose/compose-file/#env_file
72+
[compose-environment]: https://docs.docker.com/compose/compose-file/#environment
73+
[docker-compose.yaml]: ./docker-compose.yaml

0 commit comments

Comments
 (0)