Json decoding function

[purpleidea]

We'd like to implement a function that decodes an mcl string of json data, and turns it into the correct mcl type. This is a difficult challenge, but definitely doable.

Design

mcl is a statically typed language (like golang) and since it's impossible to guarantee the "shape" (type) of the input data, then we need to employ some tricks. Let's start off with the function signature and an example:

$mcl_value = json_decode("[]str", $data)

The first arg is a string in the mcl type format. This can be parsed with types.NewType(...)
$data should be a string of json data.

Internal API's

Since we build functions of different types depending on the input type, we'll need the fancier function signature of:

FuncInfer(partialType *types.Type, partialValues []types.Value) (*types.Type, []*interfaces.UnificationInvariant, error)

which, before type unification, lets you look at what we happen to already know statically, and build a function that fits that expectation! Of course if the type isn't known, or if it's an invalid type, we just error early, and your program won't compile!

Bad input data?

If the $data doesn't match the expected type, or if it changes at runtime, to a new version that doesn't match the expected type, then it's perfectly legal to fail. As a fancy bonus, if it happens that the data is known statically at compile time, then we should definitely parse it early. This will happen for more and more data as the compiler gets more clever about knowing what data is static.

Hints

A rough POC to show this is possible is here. This should hint that the function should be recursive. If it's possible to combine it with one of the existing functions in lang/types/ then that's even better.

// one example:
//	{
//	  "foo": [
//		"a",
//		"b",
//		"c"
//	  ],
//	  "bar": [
//		"a",
//		"b",
//		"c"
//	  ]
//	}

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"io"
	"reflect"

	"github.com/purpleidea/mgmt/lang/types"
	//"github.com/purpleidea/mgmt/lang/funcs"
	"github.com/purpleidea/mgmt/lang/interfaces"
)

type JSONDecodeFunc struct {

}

func (obj *JSONDecodeFunc) FuncInfer(partialType *types.Type, partialValues []types.Value) (*types.Type, []*interfaces.UnificationInvariant, error) {
	panic("not implemented")
}

func (obj *JSONDecodeFunc) Build(typ *types.Type) (*types.Type, error) {
	panic("not implemented")
}

func main() {
	fmt.Printf("hello\n")
	f, err := os.Open("f.json")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	in, err := io.ReadAll(f)
	if err != nil {
		panic(err)
	}
	//var out map[string]interface{}
	var out interface{}
	if err := json.Unmarshal([]byte(in), &out); err != nil {
		panic(err)
	}

	fmt.Printf("type: %T\n", out)
	fmt.Printf("value: %+v\n", out)

	is, ok := out.(map[string]interface{})
	fmt.Printf("is: %+v\n", ok)
	if !ok {
		panic("not ok")
	}

	out2 := is["foo"]
	is2, ok := out2.([]interface{})
	fmt.Printf("is2: %+v\n", ok)
	if !ok {
		panic("not ok2")
	}

	// TODO: loop...
	out3 := is2[0]
	is3, ok := out3.(string)
	fmt.Printf("is3: %+v\n", ok)
	if !ok {
		panic("not ok3")
	}

	rvalue := reflect.ValueOf(is3)

	opts := []types.TypeOfOption{
		//StructTagOpt(StructTag),
		//StrictStructTagOpt(false),
		//SkipBadStructFieldsOpt(false),
		types.AllowInterfaceTypeOpt(true),
	}
	typ, err := types.ConfigurableTypeOf(rvalue.Type(), opts...)
	if err != nil {
		panic(err)
	}

	fmt.Printf("typ: %+v\n", typ)


//	v, err := types.ValueOfGolang(out)
//	if err != nil {
//		panic(err)
//	}

//	fmt.Printf("mcl: %+v\n", v)

}

Testing

You'll need a few tests for this patch, including the obvious cases listed here:

https://pkg.go.dev/encoding/json

    bool, for JSON booleans
    float64, for JSON numbers
    string, for JSON strings
    []interface{}, for JSON arrays
    map[string]interface{}, for JSON objects
    nil for JSON null

Future

If it turns out to be possible, we could also consider accepting a single arg variant of the function which requires that it can statically read the json data at compile time to determine the type. I don't know if this is possible with the golang library, we'd have to figure that out first.

Sugar

Obviously if the type argument changes at runtime, then we would have to error/shutdown. For this reason, it may be worth adding this with some compiler sugar so that it's not possible to do so.

Questions?

Don't be shy, let me know if you have questions.

[brian-villa]

Is there any situation where inferring types dynamically would be acceptable in mcl?

[purpleidea]

Is there any situation where inferring types dynamically would be acceptable in mcl?

Need more information about your question:

I think the function graph needs to have all static types, but there are situations where we decide a type at compile time, but allow a different type to be a runtime error... For example, theoretically we might allow this to be func(str, str) str but the actual type represented by the first arg (string representation) and the json, could be variable and it works as long as they match.