Skip to content

Commit

Permalink
Move jq subprocess code into cli package
Browse files Browse the repository at this point in the history
Signed-off-by: Josh Dolitsky <[email protected]>
  • Loading branch information
jdolitsky committed Mar 26, 2024
1 parent b8472f6 commit 7bf5f5c
Show file tree
Hide file tree
Showing 7 changed files with 91 additions and 73 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ Go library which is used to parse and generate .tf.

## Caveats

- There is not currently no support for comments, meaning any comments on the original input are removed from the resulting HCL output
- There is not currently support for comments, meaning any comments on the original input are removed from the resulting HCL output
- In order to maintain deterministic output, all attributes (module arguments etc.) are alphabetically sorted, which may result in unexpected diffs
- All attribute values are strings (the `attributes` field is a `map[string]string`). This is required due to how the underlying libraries are used. For this purpose, real strings must be double quoted (e.g. `"\"t2.micro\""`) and non-strings (of any type) should be single quoted (e.g. `"local.ami"`)
- In order to provide total parity `jq` functionality, `tq` just shells out to `jq`. We could have probably somehow imported [`gojq`](https://github.com/itchyny/gojq), but did not want to sacrifice functionality for the sake of having a standalone binary when most development environments contain `jq` anyway
Expand Down Expand Up @@ -314,5 +314,5 @@ a reliable way to convert bidirectionally between Terraform to JSON.

The following alternatives were considered prior to `tq` being made:

- [`hlcdec`](https://github.com/hashicorp/hcl/tree/main/cmd/hcldec): This is extremely powerful as it is closely tied into the rest of the Terraform codebase. However, it relies on a pre-existing spec file. This does not work if you want to operate dynamically on any .tf file. You can only also only convert in one direction using the CLI (Go libraries can be used to convert from JSON to Terraform if you have a spec file)
- [`hcldec`](https://github.com/hashicorp/hcl/tree/main/cmd/hcldec): This is extremely powerful as it is closely tied into the rest of the Terraform codebase. However, it relies on a pre-existing spec file. This does not work if you want to operate dynamically on any .tf file. You can only also only convert in one direction using the CLI (Go libraries can be used to convert from JSON to Terraform if you have a spec file)
- [`hcl2json`](https://github.com/tmccombs/hcl2json): This is a third-party tool which will convert any HCL file to JSON. The JSON format is nice and readable (and could be easily piped into `jq`), but it does not provide any way to reliably convert the JSON back to .tf
9 changes: 6 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
module github.com/jdolitsky/tq

go 1.22.0
go 1.22.1

require (
github.com/google/go-cmp v0.3.1
github.com/hashicorp/hcl/v2 v2.19.1
github.com/google/go-cmp v0.6.0
github.com/hashicorp/hcl/v2 v2.20.0
github.com/lithammer/dedent v1.1.0
)

Expand All @@ -14,5 +14,8 @@ require (
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7 // indirect
github.com/zclconf/go-cty v1.13.0 // indirect
golang.org/x/mod v0.8.0 // indirect
golang.org/x/sys v0.5.0 // indirect
golang.org/x/text v0.11.0 // indirect
golang.org/x/tools v0.6.0 // indirect
)
16 changes: 12 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-test/deep v1.0.3 h1:ZrJSEWsXzPOxaZnFteGEfooLba+ju3FYIbOrS+rQd68=
github.com/go-test/deep v1.0.3/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/hashicorp/hcl/v2 v2.19.1 h1://i05Jqznmb2EXqa39Nsvyan2o5XyMowW5fnCKW5RPI=
github.com/hashicorp/hcl/v2 v2.19.1/go.mod h1:ThLC89FV4p9MPW804KVbe/cEXoQ8NZEh+JtMeeGErHE=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/hashicorp/hcl/v2 v2.20.0 h1:l++cRs/5jQOiKVvqXZm/P1ZEfVXJmvLS9WSVxkaeTb4=
github.com/hashicorp/hcl/v2 v2.20.0/go.mod h1:WmcD/Ym72MDOOx5F62Ly+leloeu6H7m0pG7VBiU6pQk=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
Expand All @@ -26,5 +26,13 @@ github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/zclconf/go-cty v1.13.0 h1:It5dfKTTZHe9aeppbNOda3mN7Ag7sg6QkBNm6TkyFa0=
github.com/zclconf/go-cty v1.13.0/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4=
golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
3 changes: 1 addition & 2 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,14 @@ import (
"log"

"github.com/jdolitsky/tq/pkg/cli"
"github.com/jdolitsky/tq/pkg/tq"
)

func main() {
buf, query, err := cli.CommandLineArgsBuffer(true)
if err != nil {
log.Fatal(err)
}
tqb, err := tq.TQ(query, buf.Bytes())
tqb, err := cli.Query(query, buf.Bytes())
if err != nil {
log.Fatal(err)
}
Expand Down
67 changes: 67 additions & 0 deletions pkg/cli/query.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package cli

import (
"bytes"
"encoding/json"
"io"
"os"
"os/exec"

"github.com/hashicorp/hcl/v2/hclwrite"
"github.com/jdolitsky/tq/pkg/tq"
)

// Query does the following:
// 1. Converts incoming Terraform bytes to JSON
// 2. Runs it through jq (via subprocess) to apply the query
// 3. Converts it back to Terraform (Note: if this fails, just return the JSON)
func Query(query string, in []byte) ([]byte, error) {
// Convert original input to JSON
tfFile, err := tq.ParseTerraform(in)
if err != nil {
return nil, err
}
jb, err := json.Marshal(tfFile)
if err != nil {
return nil, err
}

// Run jq directly, passing the parsed JSON as STDIN
cmd := exec.Command("jq", "-r", query)
var jqbuff bytes.Buffer
cmd.Stdout = &jqbuff
cmd.Stderr = os.Stderr
stdin, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
if err := cmd.Start(); err != nil {
return nil, err
}
if _, err := io.WriteString(stdin, string(jb)); err != nil {
return nil, err
}
stdin.Close()
if err := cmd.Wait(); err != nil {
return nil, err
}

// Now attempt to convert it back to tf
jqb := jqbuff.Bytes()
file, err := tq.ParseJSON(jqb)
if err != nil {
// TODO: add option to fail here
// If there was some issue converting back to terraform,
// just print out the JSON (assume what the user wants here)
return jqb, nil
}
tfb := hclwrite.Format(file.Bytes())
if tfb == nil {
// TODO: add option to fail here
// Like above, maybe we somehow passed conversion, but
// the result is empty. Return the jq raw output.
return jqb, nil
}

return tfb, nil
}
6 changes: 3 additions & 3 deletions pkg/tq/tq_test.go → pkg/cli/query_test.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package tq
package cli

import (
"strings"
Expand All @@ -8,7 +8,7 @@ import (
"github.com/lithammer/dedent"
)

func TestTQ(t *testing.T) {
func TestQuery(t *testing.T) {
// Note: when adding tests here, be mindful of tabs (\t) vs. spaces
tests := []struct {
name string
Expand Down Expand Up @@ -220,7 +220,7 @@ func TestTQ(t *testing.T) {
}

for _, test := range tests {
got, err := TQ(test.query, []byte(test.input))
got, err := Query(test.query, []byte(test.input))
if err != nil {
if test.shouldErr {
continue
Expand Down
59 changes: 0 additions & 59 deletions pkg/tq/tq.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
package tq

import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"sort"
"strings"

Expand All @@ -31,61 +27,6 @@ type (
}
)

// TQ does the following:
// 1. Converts incoming Terraform bytes to JSON
// 2. Runs it through jq (via subprocess) to apply the query
// 3. Converts it back to Terraform (Note: if this fails, just return the JSON)
func TQ(query string, in []byte) ([]byte, error) {
// Convert original input to JSON
tfFile, err := ParseTerraform(in)
if err != nil {
return nil, err
}
jb, err := json.Marshal(tfFile)
if err != nil {
return nil, err
}

// Run jq directly, passing the parsed JSON as STDIN
cmd := exec.Command("jq", "-r", query)
var jqbuff bytes.Buffer
cmd.Stdout = &jqbuff
cmd.Stderr = os.Stderr
stdin, err := cmd.StdinPipe()
if err != nil {
return nil, err
}
if err := cmd.Start(); err != nil {
return nil, err
}
if _, err := io.WriteString(stdin, string(jb)); err != nil {
return nil, err
}
stdin.Close()
if err := cmd.Wait(); err != nil {
return nil, err
}

// Now attempt to convert it back to tf
jqb := jqbuff.Bytes()
file, err := ParseJSON(jqb)
if err != nil {
// TODO: add option to fail here
// If there was some issue converting back to terraform,
// just print out the JSON (assume what the user wants here)
return jqb, nil
}
tfb := hclwrite.Format(file.Bytes())
if tfb == nil {
// TODO: add option to fail here
// Like above, maybe we somehow passed conversion, but
// the result is empty. Return the jq raw output.
return jqb, nil
}

return tfb, nil
}

func ParseTerraform(b []byte) (*TerraformFile, error) {
file, diags := hclwrite.ParseConfig(b, "", hcl.InitialPos)
if diags.HasErrors() {
Expand Down

0 comments on commit 7bf5f5c

Please sign in to comment.