[wip] new FP-like intermediate representation #10

vilterp · 2018-03-28T05:25:59Z

Queries are planned into expressions in a small functional programming language,

e.g. the TreeSQL query

MANY blog_posts {
  id,
  comments: MANY comments {
    id,
    blog_post_id
  }
}

becomes

map(
  scan(tables.blog_posts.id),
  (row1) => {
    id: row1.id
    comments: map(
      filter(
        scan(tables.comments.id),
        (row2) => strEq(row2.blog_post_id, row1.id)
      ),
      (row2) => {
        blog_post_id: row2.blog_post_id,
        id: row2.id
      }
    ),
  }
)

…which is then executed to generate the result. Hopefully this will serve as a basis for expanding the set of queries TreeSQL can execute, and be a good place to do index selection, etc.

Still working on the language basics and figuring out how to incrementalize this XD

I don't set out to write a code generator in the Format function of PlanNode, but it appears that's what's happened. Next step? Transform this into another intermediate representation which is executed by a VM?

messed up Not sure how far to go with this

there's about to be a lot of stuff to pretty print y'all

check in basic types, values, and expressions

it compiles!

not actually deserializing and returning an object yet

- add concept of type vars - new function for matching types There are a few panics though, so I'm afraid it's fragile.

things are getting wonky.

once we have a parser, can just parse these

i.e. still return usable traces even in the presence of errors

also add support to the test harness for Map nodes in the trace tree

plus clean up shell's use of channels a bit

oof

by namespacing tables under a "tables" record

no joins yet

need to do this when I'm less sleep-deprived. closures not working.

they still don't execute correctly because we don't create a new iterator for each scan yet.

now, a lambda which does a scan can be invoked multiple times, and the scan will start from the beginning each time.

golangcibot · 2019-04-19T08:27:14Z

pkg/client.go

-
-		case incomingMsg := <-conn.IncomingMessages:
-			channel := conn.Channels[incomingMsg.StatementID]
+			conn.webSocketConn.WriteMessage(websocket.TextMessage, []byte(request.Statement))


Error return value of conn.webSocketConn.WriteMessage is not checked (from errcheck)

golangcibot · 2019-04-19T08:27:14Z

pkg/client.go

-		err := conn.WebSocketConn.ReadJSON(&parsedMessage)
+		parsedMessage := &BasicChannelMessage{}
+		_, rawMsg, err := conn.webSocketConn.ReadMessage()
+		json.Unmarshal(rawMsg, parsedMessage)


Error return value of json.Unmarshal is not checked (from errcheck)

golangcibot · 2019-04-19T08:27:15Z

pkg/lang/expr.go

+				fc.funcName, idx, argType.Format(), param.Typ.Format(),
+			)
+		}
+		bindings.extend(argBindings)


Error return value of bindings.extend is not checked (from errcheck)

golangcibot · 2019-04-19T08:27:15Z

pkg/lang/scope.go

+		if !matches {
+			return false, nil
+		}
+		bindings.extend(paramBindings)


Error return value of bindings.extend is not checked (from errcheck)

golangcibot · 2019-04-19T08:27:15Z

pkg/lang/type.go

+	if !paramsMatch {
+		return false, nil
+	}
+	bindings.extend(paramBindings)


Error return value of bindings.extend is not checked (from errcheck)

golangcibot · 2019-04-19T08:27:19Z

pkg/table_iterator.go


 // schema columns iterator

-type SchemaColumnsIterator struct {
+type schemaColumnsIterator struct {


U1000: type schemaColumnsIterator is unused (from unused)

golangcibot · 2019-04-19T08:27:19Z

pkg/table_iterator.go

-	for _, table := range db.Schema.Tables {
-		for _, column := range table.Columns {
-			columnDoc := column.ToRecord(table.Name, db)
+func newColumnsIterator(db *Database) (*schemaColumnsIterator, error) {


U1000: func newColumnsIterator is unused (from unused)

golangcibot · 2019-04-19T08:27:20Z

pkg/table_iterator.go


 // record listeners iterator

-type RecordListenersIterator struct {
+type recordListenersIterator struct {


U1000: type recordListenersIterator is unused (from unused)

golangcibot · 2019-04-19T08:27:20Z

pkg/table_iterator.go

-func newRecordListenersIterator(db *Database) (*RecordListenersIterator, error) {
-	listenersTable := db.Schema.Tables["__record_listeners__"]
-	listeners := make([]*Record, 0)
+func newRecordListenersIterator(db *Database) (*recordListenersIterator, error) {


U1000: func newRecordListenersIterator is unused (from unused)

golangcibot · 2019-04-19T08:27:20Z

pkg/util/test_util.go

+	var o1 interface{}
+	var o2 interface{}
+
+	var err error


S1021: should merge variable declaration with assignment on next line (from gosimple)

vilterp changed the title ~~new FP-like intermediate representation~~ [wip] new FP-like intermediate representation Mar 28, 2018

vilterp force-pushed the fp branch from 97debac to d115922 Compare April 1, 2018 23:48

Pete Vilter added 28 commits April 19, 2019 04:21

create a bucket for each referencestable column as well as the primary

ecf5d42

insert into the correct bucket

d107c79

I seem to have written some kind of code generator

8d46343

I don't set out to write a code generator in the Format function of PlanNode, but it appears that's what's happened. Next step? Transform this into another intermediate representation which is executed by a VM?

flatten things out with the indent buffer, but var names are still

aec4dde

messed up Not sure how far to go with this

different var naming strategy

dc1da4f

better string rep. oooh generators

3f879d4

add pretty printing library

4f3c434

there's about to be a lot of stuff to pretty print y'all

get rid of indent buffer

f8baf59

initial commit of language / interpreter

8a269bb

check in basic types, values, and expressions

add functions and function calls; build out interpreter

ef6f811

it compiles!

interpreter tests: variables, builtins, lambdas

36a964e

tests for write to json

13bad7f

add some Value#GetType tests

50a6250

[wip] start getting table iterator from schema

505872c

iterate through a table!

d1bf8d7

not actually deserializing and returning an object yet

successfully return table results as JSON!

fb5e7ed

lower-case type names

780921c

map iterator basically works!

8cb1b20

- add concept of type vars - new function for matching types There are a few panics though, so I'm afraid it's fragile.

add isConcrete function; clean up printlns

a8d9b03

try to check types of lambda bodies against their stated return type.

a1f00d8

things are getting wonky.

ok somehow got everything to pass

a476a89

add pretty printed exprs to tests

44b3132

once we have a parser, can just parse these

pretty print lib: String => Debug; Render => String

85532ff

object => record

3fa0ab2

add table names to type scope; pretty print scopes

1913610

basic grammar structs and formatting

1033520

first basic parse yaaas

1855142

implement regex matcher

b3cf91b

Pete Vilter added 27 commits April 19, 2019 04:21

make parser recover better!

b2ad8e1

i.e. still return usable traces even in the presence of errors

server error handling tweaks

b7c51e1

add basic parser for FP expressions

5ffc61d

move system tables to their own file

e2675cb

start mapping parse trees to FP exprs

0a0a6c2

also add support to the test harness for Map nodes in the trace tree

parse record literals!

e7e2fae

parse func calls

7d423db

actually parse func calls; parse lambdas!

7fd469d

parse return types, sort of

ce66082

parse param types (sort of)

a42e18a

delete type expr

6be2fcf

leave a todo

4fe9e37

package => pkg

f8a9782

first TreeSQL => FP query execution!

ca0e444

plus clean up shell's use of channels a bit

get tests to pass, by commenting out some things

d99a95e

oof

un-export a bunch of stuff

2ef9450

prevent table names from colliding with builtin names

6fb1881

by namespacing tables under a "tables" record

add plan tests

c0e40e7

pretty_print => prettyprint

37e9c61

fix tests

45036c6

add strEq and intEq

3b81a44

plan queries with a WHERE

23777a1

no joins yet

get a query with a filter to work

2f41e41

wip

4815f71

need to do this when I'm less sleep-deprived. closures not working.

plan one-many joins correctly!

bbb276f

they still don't execute correctly because we don't create a new iterator for each scan yet.

introduce Index type and scan function

7f1c1b9

now, a lambda which does a scan can be invoked multiple times, and the scan will start from the beginning each time.

remove printlns; uncomment test that now works

a15d7b1

vilterp force-pushed the fp branch from de91591 to a15d7b1 Compare April 19, 2019 08:23

golangcibot reviewed Apr 19, 2019

View reviewed changes

get parserlib test harness to work again

f874714

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] new FP-like intermediate representation #10

[wip] new FP-like intermediate representation #10

vilterp commented Mar 28, 2018 •

edited

Loading

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

golangcibot Apr 19, 2019

[wip] new FP-like intermediate representation #10

Are you sure you want to change the base?

[wip] new FP-like intermediate representation #10

Conversation

vilterp commented Mar 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vilterp commented Mar 28, 2018 •

edited

Loading