Load a new WCON

[ver228]

I am trying to export the output produced by our lab (Andre's) into the WCON format. I produce the following test file. I think I am following the specifications for the format, and it does pass the schema validation. I am saving all the features, coordinates and time as arrays, so I do not have to repeat long feature names thousand of times. Most of the features have the same elements as the time array, but some
are event features that are an array with variable length per worm trajectory.

In the Python implementation the file is read correctly, once I remove 'degree' as units (see issue). However the feature files are not loaded in the WCONWorms object. I do not know if it is a bug or it hasn't being implemented.

I couldn't run the scala implementation (I'll send another issue here or rather a request for help), but I managed to run the R wrapper. The wrapper example works fine as long as one set the working directory as the one where 'wcon-ready.r' is located. However, when I try to log the file I generated I get the errors below. The description is a bit cryptical and I have not idea what is the problem.

java.util.NoSuchElementException: Either.right.value on Left
at scala.util.Either$RightProjection.get(Either.scala:453)
at $line82.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$.(:33)
at $line82.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$.()
at $line82.$eval$.$result$lzycompute(:5)
at $line82.$eval$.$result(:5)
at $line82.$eval.$result()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.callEither(IMain.scala:790)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.evalEither(IMain.scala:805)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.eval(IMain.scala:624)
at scala.tools.nsc.interpreter.IMain.eval(IMain.scala:1074)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at org.ddahl.rscala.ScalaInterpreterAdapter.eval(ScalaInterpreterAdapter.scala:6)
at org.ddahl.rscala.ScalaServer.doEval(ScalaServer.scala:107)
at org.ddahl.rscala.ScalaServer.org$ddahl$rscala$ScalaServer$$heart(ScalaServer.scala:559)
at org.ddahl.rscala.ScalaServer.run(ScalaServer.scala:599)
at $line3.$read$$iw$$iw$.(:13)
at $line3.$read$$iw$$iw$.()
at $line3.$eval$.$print$lzycompute(:7)
at $line3.$eval$.$print(:6)
at $line3.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1$$anonfun$apply$5$$anonfun$apply$6.apply(ILoop.scala:427)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1$$anonfun$apply$5$$anonfun$apply$6.apply(ILoop.scala:423)
at scala.reflect.io.Streamable$Chars$class.applyReader(Streamable.scala:111)
at scala.reflect.io.File.applyReader(File.scala:50)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1$$anonfun$apply$5.apply(ILoop.scala:423)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1$$anonfun$apply$5.apply(ILoop.scala:423)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1.apply(ILoop.scala:422)
at scala.tools.nsc.interpreter.ILoop$$anonfun$interpretAllFrom$1.apply(ILoop.scala:422)
at scala.tools.nsc.interpreter.ILoop.savingReader(ILoop.scala:96)
at scala.tools.nsc.interpreter.ILoop.interpretAllFrom(ILoop.scala:421)
at scala.tools.nsc.interpreter.ILoop$$anonfun$run$3$1.apply(ILoop.scala:577)
at scala.tools.nsc.interpreter.ILoop$$anonfun$run$3$1.apply(ILoop.scala:576)
at scala.tools.nsc.interpreter.ILoop.withFile(ILoop.scala:570)
at scala.tools.nsc.interpreter.ILoop.run$3(ILoop.scala:576)
at scala.tools.nsc.interpreter.ILoop.loadCommand(ILoop.scala:583)
at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$8.apply(ILoop.scala:207)
at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$8.apply(ILoop.scala:207)
at scala.tools.nsc.interpreter.LoopCommands$LineCmd.apply(LoopCommands.scala:62)
at scala.tools.nsc.interpreter.ILoop.colonCommand(ILoop.scala:688)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:679)
at scala.tools.nsc.interpreter.ILoop.loadFiles(ILoop.scala:835)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:87)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Exception message: Either.right.value on Left

[Ichoran]

This is weird. It probably has to do with the problems running the Scala version, though, since the R version just calls Scala.

[MichaelCurrie]

the feature files are not loaded in the WCONWorms object. I do not know if it is a bug or it hasn't being implemented.

The reason for this is you have to write a class that inherits from WCONWorms to handle the custom features. I will work on you with this to make this happen if you like.

I did not anticipate that we would be loading the feature file as a WCON file - this format I thought would be used for interoperability and so I thought we would just use WCON for the skeleton and not for the features. After all, we can get a features object loaded into the open worm analysis toolbox by calculating the features from a basicworm object.

[JimHokanson]

When I try and load the test file it fails because the lab has an address field, but the spec is looking for location.

https://github.com/openworm/tracker-commons/blob/master/WCON_format.md#metadata-fields-in-detail

[JimHokanson]

Also not currently valid in the test file is 'commit_hash' for software. Address vs location is a naming issue that needs to be worked out. This on the other hand is a completely new entry which seems useful. Also of note, the Python implementation should have been choking on these things .... The Scala implementation choked, but failed with perhaps a too cryptic error message "NoSuchElementException"

[JimHokanson]

All, can we get some clarification on the following:

1. Whether or not arbitrary fields are allowed in specified types, i.e. whether software can contain commit_hash
2. Where in the documentation we are specifying this
3. If (1) is allowed, how we expect generic software to handle this (drop values on loading, carry values on loading)
4. Whether or not (1) is conditioned on specific levels of the data (i.e. arbitrary may be allowed in units, data, and metadata, but not in specific metadata fields?)

[Ichoran]

@JimHokanson - I propose that

1. Arbitrary fields are allowed anywhere by readers. Writers will not by default write arbitrary fields. Fields with the wrong structure are never allowed. For instance, "metadata" cannot be a string instead of an object.
2. Once we agree what the rules are, we should state them clearly in appropriate places (including the introduction) since it's quite important to know what's going to happen to your data.
3. It is up to the reader whether they remain accessible in some way or are simply dropped. If they are not dropped, a writer may have an option to write them out again.
4. There is no context. The same rules apply everywhere. (This makes it harder to get wrong.)
5. Unit conversions will not apply to arbitrary fields that are ignored but maybe kept, even if they have the right names. Unit conversions apply only to fields defined by the specification and to specifically-named fields inside @ blocks. The point of a custom @ field is that the reader is to retain the contents and the writer is to write them.

I am very reluctant to try to specify a second kind of custom field (maybe with % instead of @) where no conversions are done and everything is just carried along verbatim. Maybe the solution is to have conversions not happen by default but allow them to be invoked on request. But that is an issue for a different ticket.

@MichaelCurrie - What do you think of these proposals?