acquisition/victorialogs: add new datasource

[zekker6]

Related issue: #3307

Data source supports:

• cat mode with automatic adjustment of poll interval (same as one at Loki datasource)
• tail mode by using tailing API

Most of the logic is exactly the same as Loki datasource.
The only difference is related to adding support of live dailing and different response formats.
VictoriaLogs uses newline delimeted JSON for both reqular query and tail modes.

Tested locally with the following config:

source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: tail
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx
---
source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: cat
since: 1m
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx

[github-actions]

@zekker6: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@zekker6: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[codecov]

Codecov Report

Attention: Patch coverage is 21.90476% with 410 lines in your changes missing coverage. Please review.

Project coverage is 54.93%. Comparing base (51f762c) to head (b36d918).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...odules/victorialogs/internal/vlclient/vl_client.go	4.79%	257 Missing and 1 partial ⚠️
...g/acquisition/modules/victorialogs/victorialogs.go	40.07%	143 Missing and 8 partials ⚠️
pkg/acquisition/victorialogs.go	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3310      +/-   ##
==========================================
- Coverage   59.30%   54.93%   -4.38%     
==========================================
  Files         358      361       +3     
  Lines       38428    38953     +525     
==========================================
- Hits        22791    21399    -1392     
- Misses      13742    15722    +1980     
+ Partials     1895     1832      -63     

Flag	Coverage Δ
bats	41.65% <0.19%> (-0.61%)	⬇️
unit-linux	?
unit-windows	28.39% <21.90%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mmetc]

this should be listed in the Makefile too -- the only reason to "register" datasources is to optionally exclude from the builds with a tag, and still list them in "cscsli version"

[zekker6]

Added this to Makefile in f4603e1, thanks for noticing

[denisgolius]

AFAIK it requires to install curl every time when mostly Linux Distributions have installed wget out-from-box. Do you think curl will be better decision?

[blotus]

AFAIK it requires to install curl every time when mostly Linux Distributions have installed wget out-from-box. Do you think curl will be better decision?

This change only impacts the CI, and AFAIK curl is installed by default on the images provided by github.

I made the change just to test/debug why the victoria logs container seems to fail to start (or at least, github/docker thinks it has failed, I cannot reproduce the issue locally).

[zekker6]

Hello @blotus, I've updated healthcheck so it should work now. This can be tested locally with:

docker run --rm -it --name=victorialogs1 \
                            --health-cmd "wget -q -O - http://0.0.0.0:9428" \
                            --health-interval 30s \
                            --health-timeout 10s \
                            --health-retries 5 \
                            --health-start-period 30s \
                  victoriametrics/victoria-logs:v1.5.0-victorialogs

Previously, it was using localhost which does not work in alpine based images.

[zekker6]

@blotus Hopefully I've fixed the last reason for the tests to fail at 2b42a43, could you please re-run tests?

[mmetc]

@blotus should this method be called or removed

Error: pkg/acquisition/modules/victorialogs/internal/vlclient/vl_client.go:69:21: func (*VLClient).resetFailStart is unused (unused)
func (lc *VLClient) resetFailStart() {

[zekker6]

@mmetc I've removed unused method as it was not needed.