[APP-6612] Add Filters for Downloading Logs #4673

JosephBorodach · 2025-01-03T17:57:10Z

This PR implements additional filters to the machines logs CLI command, specifically: keyword, start time, end time and log level. Below are the details and test cases for the command:

See ticket here

Invalid: Without any arguments

go run cli/viam/main.go machines logs

With Keyword Filter

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --keyword="stack"

With Error Level

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --levels="error"

Start Time

Start from after the last log

go run cli/viam/main.go machines logs --machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-10T21:30:00Z"

The most recent few logs

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-13T07:20:31.000Z"

Logs from a slightly earlier start time

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-12T17:37:26.985Z"

Invalid: Cannot use start and count without end.

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-12T17:37:26.985Z" --count=10

End Time

Notice how in the second screenshot, we skip the logs passed 15:16.

Start & End Time

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-12T17:37:26.985Z" --end="2025-01-13T04:24:08.928Z"

Start + Count + End Time

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --start="2025-01-13T08:19:18.865Z" --end="2025-01-13T21:03:37.372Z" --count=5

Edge Case

Do we not want to output something prettier if there are no logs to display?

Without providing `count`, defaults to 100 logs

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --output="logs.txt" --format="text"

`count` of 10,000 works

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --output="logs.txt" --format="text" --count=10000

`count` of 10,001 does not work

go run cli/viam/main.go machines logs -machine="1" --organization="viam-dev" --location="office temp" --output="logs.txt" --format="text" --count=10001

@jr22 - I personally do not love that when surpassing the maxLogs we still display the following:

…gs to text or json file

…ng the logs, instead of retrieving all of the logs and then writing them all to the file

…ck when writing to file

…n object

…t arguments to be paired - either both are provided, or an error will be displated

…ds called on the *viamClient

… logs cli command

purplenicole730

LGTM! There were some changes in flag names that were recently merged in, but I don't suspect it should affect your PR too much

purplenicole730 · 2025-01-07T15:02:31Z

cli/app.go

+						&cli.StringFlag{
+							Name:  logsFlagOutputFile,
+							Usage: "path to output file",
+						},
+						&cli.StringFlag{
+							Name:  logsFlagFormat,
+							Usage: "file format (text or json)",
+						},


maybe mention needs format flag, needs output flag for the users in the usage text? Not sure if necessary, but a suggestion, since I feel like I would read that as being able to format it when printing to console as well

You can decide the format when printing to the console! The command will default to text if the format flag is not included. Are you suggesting something else?

Oh, you can also format it for the console! Ignore me then :)

stuqdog

A couple small things but generally lgtm

stuqdog · 2025-01-07T16:02:11Z

cli/app.go

+	logsFlagFormat     = "format"
+	logsFlagOutputFile = "output"
+	logsFlagKeyword    = "keyword"
+	logsFlagLevels     = "levels"
+	logsFlagStartTime  = "start"
+	logsFlagEndTime    = "end"
+	logsFlagErrors     = "errors"
+	logsFlagTail       = "tail"
+	logsFlagCount      = "count"


it looks like some of these were added in a separate PR already. Would you mind pulling down main so we can get a more up-to-date look at what the diff is precisely?

stuqdog · 2025-01-07T16:04:28Z

cli/app.go

+	logsFlagStartTime  = "start"
+	logsFlagEndTime    = "end"


(minor) I'm gonna ask you to do a bit of bookkeeping for us here if you don't mind! We already have a "start" flag and an "end" flag (dataFlagStart and dataFlagEnd). Would you be willing to rename those to generalFlagStart and generalFlagEnd and then update call sites so that there's only the single instance of a --start or --end flag?

…s-For-Downloading-Logs

…Time to generalFlagStart and generalFlagEnd

JosephBorodach · 2025-01-14T14:24:33Z

cli/client.go

@jr22 - Is there a reason we don't have unit tests for this function's helpers? Should I add unit tests?

not a reason that im aware of, feel free to add test coverage if you want!

JosephBorodach added 11 commits December 31, 2024 15:41

Support downloading logs

386db8c

Restructure RobotsLogsAction to extend functionality for outputing lo…

232f143

…gs to text or json file

Refactor saveLogsToDisk to handle large log files efficiently

6a02466

Restructure fetchAndSaveLogs to write logs to the file while retrievi…

e14e9be

…ng the logs, instead of retrieving all of the logs and then writing them all to the file

Add nolint for not checking the error returned when closing the file

761d6fe

add headers to log output for file downloads

05dc2c2

Add constant variable for FormatJSON to appease linter. Add error che…

921a908

…ck when writing to file

Make the json format consecutive json object instead of one large jso…

690b887

…n object

Make error handling more comprehensive for requiring Output and Forma…

e2786a5

…t arguments to be paired - either both are provided, or an error will be displated

Make fetchAndSaveLogs, streamLogsForPart and printLogsToConsole metho…

225b5cb

…ds called on the *viamClient

Add filters including keyword, levels, and start and end times to the…

b0698af

… logs cli command

JosephBorodach requested a review from a team as a code owner January 3, 2025 17:57

JosephBorodach requested review from stuqdog and purplenicole730 and removed request for a team January 3, 2025 17:57

Base automatically changed from APP-6612-Support-Downloading-Logs to main January 7, 2025 14:17

purplenicole730 approved these changes Jan 7, 2025

View reviewed changes

stuqdog reviewed Jan 7, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into APP-6612-Add-Filter…

b3e00d3

…s-For-Downloading-Logs

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Jan 8, 2025

Replace dataFlagStart, dataFlagEnd, logsFlagStartTime and logsFlagEnd…

bf0df5f

…Time to generalFlagStart and generalFlagEnd

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Jan 8, 2025

Add logic to streamLogsForPart for the filters

7abd091

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Jan 8, 2025

Remove validation checks

0c6c6f9

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Jan 8, 2025

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Jan 13, 2025

JosephBorodach requested review from jr22 and removed request for jr22 January 13, 2025 22:58

JosephBorodach commented Jan 14, 2025

View reviewed changes

Fix useless print with f usage found Errorf

2f273e2