Skip to content

Commit 9a65837

Browse files
Feature/improve parser (#16)
* Restructure pkg/ folder in preparation of new parser * Implement Setters for ParsedLog * Rework parser to have uniform interface for parse functions * Remove test that doesn't apply anymore * Rework Staus to Created/NotCreated/Failed * Implement correct parsing of Modify/Destroy * Add tests for new parsers * Remove legend and make only Failed red * Fix go vet issues * Make Created the default for BeforeStatus * Parse refreshes as well * Add more State stats to the stats command * Only export the necessary functions for other packages * Adapt docs for new parser * Various small docs changes after review * Remove redundant command * Small improvements to README * Small improvements to docs * Small improvements to docs * Rewrite outdated comment * Improve comments and clean up * go fmt
1 parent 99e29f8 commit 9a65837

33 files changed

+1567
-738
lines changed

.github/stats.png

-45.3 KB
Loading

.github/table.png

-33 KB
Loading

README.md

+37-30
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@
55
CLI tool to profile Terraform runs, written in Go.
66

77
Main features:
8-
- Modern CLI ([cobra](https://github.com/spf13/cobra)-based), including autocomplete
8+
- Modern CLI ([cobra](https://github.com/spf13/cobra)-based) with autocomplete
99
- Read logs straight from your Terraform process (using pipe) or a log file
1010
- Can generate global stats, resource-level stats or visualizations
11-
- Provides many levels of granularity and aggregation, customizable outputs
11+
- Provides many levels of granularity and aggregation and customizable outputs
1212

1313
## Installation
1414

@@ -23,7 +23,7 @@ Main features:
2323
If you want to try `tf-profile` without installing anything, you can run it using Docker (or similar).
2424

2525
```bash
26-
❱ cat my_log_file.log | docker run -i qbruynseraede/tf-profile:0.0.1 stats
26+
❱ cat my_log_file.log | docker run -i qbruynseraede/tf-profile:0.2.0 stats
2727

2828
Key Value
2929
Number of resources created 1510
@@ -37,7 +37,7 @@ Longest apply resource time_sleep.foo[*]
3737
Optionally, define an alias:
3838

3939
```bash
40-
alias tf-profile=docker run -i qbruynseraede/tf-profile:0.0.1
40+
alias tf-profile=docker run -i qbruynseraede/tf-profile:0.2.0
4141
❱ cat my_log_file.log | tf-profile
4242
```
4343

@@ -68,7 +68,7 @@ Usage:
6868
Three major commands are supported:
6969
- [🔗](#anchor_stats) `tf-profile stats`: provide general statistics about a Terraform run
7070
- [🔗](#anchor_table) `tf-profile table`: provide detailed, resource-level statistics about a Terraform run
71-
- [🔗](#anchor_graph) `tf-profile graph`: generate visual overview of a Terraform run.
71+
- [🔗](#anchor_graph) `tf-profile graph`: generate a visual overview of a Terraform run.
7272

7373

7474
## `tf-profile stats`
@@ -79,27 +79,33 @@ Three major commands are supported:
7979
```bash
8080
❱ terraform apply -auto-approve > log.txt
8181
❱ tf-profile stats log.txt
82-
❱ tf-profile stats test/many_modules.log
8382

8483
Key Value
8584
-----------------------------------------------------------------
86-
Number of resources created 1510
85+
Number of resources in configuration 1510
8786

88-
Cumulative duration 36m19s
89-
Longest apply time 7m18s
90-
Longest apply resource time_sleep.foo[*]
91-
92-
No. resources in state AllCreated 800
93-
No. resources in state Created 695
94-
No. resources in state Started 15
95-
96-
Number of top-level modules 13
97-
Largest top-level module module.core[2]
98-
Size of largest top-level module 170
99-
Deepest module module.core[2].module.role[47]
100-
Deepest module depth 2
101-
Largest leaf module module.dbt[4]
102-
Size of largest leaf module 40
87+
Cumulative duration 36m19s
88+
Longest apply time 7m18s
89+
Longest apply resource time_sleep.foo[*]
90+
91+
Resources marked for operation Create 892
92+
Resources marked for operation None 18
93+
Resources marked for operation Replace 412
94+
95+
Resources in state AllCreated 800
96+
Resources in state Created 695
97+
Resources in state Started 15
98+
99+
Resources in desired state 1492 out of 1510 (98.8%)
100+
Resources not in desired state 18 out of 1510 (0.01%)
101+
102+
Number of top-level modules 13
103+
Largest top-level module module.core[2]
104+
Size of largest top-level module 170
105+
Deepest module module.core[2].module.role[47]
106+
Deepest module depth 2
107+
Largest leaf module module.dbt[4]
108+
Size of largest leaf module 40
103109
```
104110

105111
For more information, refer to the [reference](./docs/stats.md) for the `stats` command.
@@ -112,12 +118,13 @@ For more information, refer to the [reference](./docs/stats.md) for the `stats`
112118
❱ terraform apply -auto-approve > log.txt
113119
❱ tf-profile table log.txt
114120

115-
resource n tot_time idx_creation idx_created status
116-
--------------------------------------------------------------------------------------
117-
time_sleep.count[*] 5 11s 0 13 AllCreated
118-
time_sleep.foreach[*] 3 7s 4 11 AllCreated
119-
module.test[1].time_sleep.count[*] 3 5s 3 9 AllCreated
120-
module.test[0].time_sleep.count[*] 3 4s 9 7 AllCreated
121+
resource n tot_time modify_started modify_ended desired_state operation final_state
122+
aws_ssm_parameter.p6 1 0s 6 7 Created Replace Created
123+
aws_ssm_parameter.p1 1 0s 7 5 Created Replace Created
124+
aws_ssm_parameter.p3 1 0s 5 6 Created Replace Created
125+
aws_ssm_parameter.p4 1 0s / 1 NotCreated Destroy NotCreated
126+
aws_ssm_parameter.p5 1 0s 4 4 Created Modify Created
127+
aws_ssm_parameter.p2 1 0s / / Created None Created
121128
```
122129

123130
For a full description of the options, see the [reference](./docs/table.md) page.
@@ -148,9 +155,9 @@ _Disclaimer:_ Terraform's logs do not contain any absolute timestamps. We can on
148155
## Roadmap
149156

150157
- [x] Release v0.0.1 as binary and as a Docker image
151-
- [ ] Improve parser
158+
- [x] Improve parser
152159
- [x] Detect failed resources (see [#13](https://github.com/QuintenBruynseraede/tf-profile/pull/13))
153-
- [ ] Use plan and refresh phase to discover more resources
160+
- [x] Use plan and refresh phase to discover more resources
154161
- [x] Implement a basic Gantt chart in `tf-profile graph` (see [#14](https://github.com/QuintenBruynseraede/tf-profile/pull/14))
155162
- [ ] Implement a single-resource view in `tf-profile detail <resource>`
156163
- This command should filter logs down to 1 single resource (i.e. refresh, plan, changes, and result)

docs/stats.md

+12-5
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,29 @@
99

1010
**Arguments:**
1111

12-
Log_file: Optional. Instruct `tf-profile` to read input from a text file instead of stdin.
12+
- log_file: _Optional_. Instruct `tf-profile` to read input from a text file instead of stdin.
1313

1414
## Description
1515

1616
The following statistics will be printed:
1717

1818
General:
19-
- **Number of resources created**: Number of resources detected in your log.
19+
- **Number of resources created**: Number of resources detected in your log. Depending on which phases (refresh, plan, apply) were present in the log, this number can differ and may not always match what is defined in your code. For example, doing `terraform apply my_plan_file` will not include a resource that is not to be modified in this plan.
2020

2121
Duration:
2222
- **Cumulative duration**: Cumulative duration of modifications. This is the sum of the duration of all modifications in the logs. Because Terraform modifies resources in parallel, this will typically be more than the actual wall time.
23-
- **Longest apply time**: Lngest time it took to modify a single resource. The next metric shows which resource that was.
24-
- **Longest apply resource**: the name of the resource that took the longest to modify.
23+
- **Longest apply time**: Longest time it took to modify a single resource. The next metric shows which resource that was.
24+
- **Longest apply resource**: The name of the resource that took the most time to modify.
25+
26+
Operations:
27+
- **Resources marked for operation \<OPERATION\>**: The amount of resources marked for a certain operation. An Operation can be any of: Create, Destroy, Modify, Replace, None. Resources that are consistent with the state, will be marked for operation None.
2528

2629
Resource status:
27-
- **No. resources in state \<STATE\>**: This statistic shows per state how many resources are in that state after the modifications.
30+
- **Resources in state \<STATE\>**: This statistic shows per state how many resources are in that state after the modifications. In general, resources can be in three states after a Terraform run: Created, NotCreated or Failed.
31+
32+
Desired state:
33+
- **Resources in desired state**: The amount of resources whose `final_state` is equal to their `desired_state`. In a fully applied configuration, this number should be 100%.
34+
- **Resources not in desired state**: Resources whose desired state was not achieved after this run. This can be due to failed creation, failed deletion or because "upstream" resources were not able to get to their desired state.
2835

2936
Modules:
3037
- **Number of top-level modules**: Number of modules called in the root module.

docs/table.md

+33-25
Original file line numberDiff line numberDiff line change
@@ -6,52 +6,60 @@
66

77
**Options:**
88
- -t, --tee: print logs while parsing them. Shorthand for `terraform apply | tee >(tf-profile stats)`
9-
- -d, --max_depth: aggregate resources nested deeper than `-d` levels into a resource that represents the module at depth `-d`.
9+
- -d, --max_depth: aggregate resources nested deeper than `-d` levels into a resource that represents the module at depth `-d`. **Not implented yet**
1010
- -s, --sort: comma-separated key-value pairs that instruct how to sort the output table. Valid values follow the format `column1:(asc|desc),column2:(asc|desc):...`. By default, `tot_time=desc,resource=asc` is used: sort first by descending modification time, second by resource name in alphabetical order.
1111

1212
**Arguments:**
1313

14-
Log_file: Optional. Instruct `tf-profile` to read input from a text file instead of stdin.
14+
- log_file: _Optional_. Instruct `tf-profile` to read input from a text file instead of stdin.
1515

1616
## Description
1717

18-
A table generated based on the log file or input, sorted according to `-s / --sort` and printed to the terminal.
18+
This command prints a table based on the log file or input, sorted according to `-s / --sort` and printed to the terminal. Useful to inspect properties about individual resources.
1919

2020
```
21-
resource n tot_time idx_creation idx_created status
22-
--------------------------------------------------------------------------------------
23-
time_sleep.count[*] 5 15s 0 13 AllCreated
24-
time_sleep.foreach[*] 3 1m30s 4 11 AllCreated
25-
module.test[1].time_sleep.count[*] 3 6 3 9 AllCreated
26-
module.test[0].time_sleep.count[*] 3 3s 9 7 AllCreated
21+
resource n tot_time modify_started modify_ended desired_state operation final_state
22+
---------------------------------------------------------------------------------------------------------
23+
aws_ssm_parameter.p6 1 0s 6 7 Created Replace Created
24+
aws_ssm_parameter.p1 1 0s 7 5 Created Replace Created
25+
aws_ssm_parameter.p3 1 0s 5 6 Created Replace Created
26+
aws_ssm_parameter.p4 1 0s / 1 NotCreated Destroy NotCreated
27+
aws_ssm_parameter.p5 1 0s 4 4 Created Modify Created
28+
aws_ssm_parameter.p2 1 0s / / Created None Created
2729
```
2830

2931
The column names are lowercase and separated by underscores to allow for easy referencing in the `--sort` option. The meaning of each column is:
3032

31-
- **resource**: Name of the resource. In case a resource is created by a `for_each` or `count` statement, resources are aggregated and individual resource identifiers are replaced by an asterisk (*). See
33+
- **resource**: Name of the resource. In case a resource is created by a `for_each` or `count` statement, resources are aggregated and individual resource identifiers are replaced by an asterisk (*).
3234
- **n**: Number of resources represented by this resource name. For regular resources, this will be 1. For resourced created with `for_each` or `count`, this number represents the number of resources created in that loop.
3335
- **tot_time**: Total cumulative time of all resources identified by this resource name. This is typically higher than the actual wall time, as Terraform can modify multiple resources at the same time.
34-
- **idx_creation**: order in which resource creation _started_. This means that Terraform started by creation the resource with `idx_creation = 0`. That does not guarantee the creation of this resource finished first as well (see `idx_created`).
35-
- **idx_created**: order in which resource creation _ended_. this means that the resource with `idx_created = 0` was the first resource to be fully creatd.
36-
- **status**: For single resources, status can be any of: `Started|NotStarted|Created|Failed`. For aggregated resources, status can be any of: `AllCreated|AllFailed|SomeFailed|NoneStarted|AllStarted|SomeStarted`.
37-
38-
With resource aggregation, more informative statuses have precedence over less informative statuses. For example, `AllCreated` will be shown over `AllStarted`.
36+
- **modify_started**: order in which resource modification _started_. This means that Terraform started by modifying the resource with `modify_started = 0`. It does not guarantee the changes to this resource finished first as well (see `modify_ended`). Resources that were already consistent with the desired state do not have this property.
37+
- **modify_ended**: order in which resource modifications _ended_. This means that the resource with `modify_ended = 0` was the first resource to finish its modifications (either a creation, deletion, modification or replacement). Resources that were already consistent with the desired state do not have this property.
38+
- **desired_state**: state (Created, NotCreated) that Terraform will try to achieve with this run. For resources to be modified, created or replaced, Created is the desired state. For resources to be destroyed, NotCreated is the desired state.
39+
- **operation**: the name of the operation the Terraform will use to reconcile the current and desired situation. Operations can be: Create, Destroy, Replace, Modify, None. Resources in the state that are already consistent with the configuration, the operation will be None.
40+
- **final_state**: Final state of the resource after this run. In addition to Created and NotCreated, Failed is used to indicate the operation failed.
3941

4042
## Sorting
4143

4244
Any of the columns above can be used to sort the output table, by means of the `--sort` (shorthand `-s`) option. This option follows the format `column1:(asc|desc),column2:(asc|desc):...`. For example:
4345
- `tot_time=desc,resource=asc`: sort first by total modification time (showing the highest first). For entries with the same modification time, sort alphabetically.
44-
- `idx_creation=asc`: sort in order of creation, showing the resources that Terraform finished modifying first.
46+
- `modify_started=asc`: sort in order modifications, showing the resources that Terraform started modifying first.
47+
- `final_state=asc`: sort by the final state. See below for the sort order.
4548

46-
When sorting on resource status (`status`), statuses are mapped onto integers before sorting.
49+
When sorting on resource status (`desired_state` or `final_state`), statuses are mapped onto integers before sorting.
4750

48-
- NotStarted: 0
49-
- Started: 1
51+
- Unknown: 0
52+
- NotCreated: 1
5053
- Created: 2
5154
- Failed: 3
52-
- SomeStarted: 4
53-
- AllStarted: 5
54-
- NoneStarted: 6
55-
- SomeFailed: 7
56-
- AllFailed: 8
57-
- AllCreated: 9
55+
- Tainted: 4
56+
- Multiple (for aggregated resources): 5
57+
58+
When sorting on resource operations (`operation`), these are mapped onto integers as well:
59+
60+
- None: 0
61+
- Create: 1
62+
- Modify: 2
63+
- Replace: 3
64+
- Destroy: 4
65+
- Multiple (for aggregated resources): 5

0 commit comments

Comments
 (0)