applicative arrows

[rebcabin]

How about an arrow that chains "apply", as in

(-@>> ([1 2] [3 4]) concat + ) => (apply + (apply concat '([1 2] [3 4]))) => 10

[rplevy]

Interesting idea. Maybe I am missing some context, but in what situations do you think this would be useful?

To me it seems like a pretty specific use rather than a common pattern.

And I personally would have to mentally convert

  (-@>> a b c)

to

  (->> a (apply b) (apply c))

And would prefer the latter for that reason...

[rebcabin]

Since I noticed that the "nil-shortcutting diamond wand" acts like the Maybe monad, I started getting the feeling that the swiss arrows could be generalized over all monads. Since the archetype of monads is the sequence monad and the mother operator, bind, for sequence is (apply concat (map my-foo your-sequence-monad)) I started to see chaining of apply-concat as a start toward monadic swiss arrows :)

Also, Wolfram / Mathematica have a host of operators that thread and merge Apply around expressions (see http://reference.wolfram.com/mathematica/ref/Apply.html). Mathematica was designed before monads were formalized in programming languages, but their precursors are all over Mathematica, for instance in the frequent use of Apply. [edit: I should add, contextually, that I am a big admirer of Mathematica just as a programming language, never mind its huge knowledge base of math. I often refer to it for ideas to bring to Clojure and other languages.]

As for the first argument being special, that didn't seem out-of-step with the other arrows, none of which, for instance, can take an expression with an angle-hole "<>" in the first position, and all of which take expressions-with-angle-holes in every slot except the first, modulo the defaults. The defaults are abundant and require the same kind of mental substitution that -@>> etc. would require, so the overall design has established the precedent of "implicits."

[rplevy]

Cool, those are great reasons, I agree that this is a good idea. Let's add it.

I also like the name '-@>>' because the @ char is also used for splicing in macros, which is a kind of apply.

[rplevy]

Update: file under "oh yeah, right, duh" but @ is not allowed by the Clojure reader to be in a symbol name.

I think apply->> is a good name.

It occurred to me for a second that this may have been done already by https://github.com/LonoCloud/synthread but that is different, it is not a threading macro, just a helper for the threading macros.

[rplevy]

See code, tests, and README changes. I'd appreciate code review and possible added tests describing any edge cases, and relevant code changes / additions (pull request preferably) to flesh out this idea.

[rebcabin]

Awesome. Just made a fork and I should be able to have a deep look late Monday :)

[rebcabin]

Another little comment: I think what this exercise will do is reveal the even more general monadic arrows. Once we get this under our belts and can play a bit with them, the ultimate truth will be revealed and the veils will be lifted from our eyes :)

[rplevy]

Cool! :)

[rebcabin]

Hi, RP. I'm not ready to submit a pull-request yet, but I noticed a couple of things. First, my attempt at part of inner product, namely

(apply->> [[2 3] [5 6]] (partial map *))

does not produce the same as

(apply (partial map *) [[2 3] [5 6]])

I haven't yet had time to figure out why not. This was a step along my way to an inner product

(apply->> [[2 3] [5 6]] (partial map *) +)

I think once I get to inner product, I will be able to generalize it.

EDIT: My stupid

(apply->> [[2 3] [5 6]] ((partial map *)) +)

works fine.

Another thing I noticed is that midje does not work as expected. On one of my two machines, "lein midje" ran the midje tests; on another of my two machines, after synching all with Github, it didn't work. On both machines, my independent tests of swiss arrows (minus the apply's) from https://github.com/rebcabin/ClojureProjects/tree/working/monads/clojure-dot-net/midje-motivation work as expected, so I have a bunch of midje stuff working. I spent some time comparing your midje specs to my midje specs, but I could not find a significant difference.

Finally, the midje repl business works in my project, but not in yours on either of my two machines. That business is summarized in my test file https://github.com/rebcabin/ClojureProjects/blob/working/monads/clojure-dot-net/midje-motivation/test/midje_motivation/t_core.clj, namely

;;; To run this, add {:user {:plugins [[lein-midje "3.0.0"]]}} to your
;;; ~/.lein/profiles.clj. Then type "lein repl" at a command
;;; prompt. Then type "(use 'midje.repl)" and "autotest" in the
;;; repl. Every time you save either this file or the corresponding
;;; source file, all the tests will run again.

[rebcabin]

I'm working on a workaround for the non-running midje repl, but the workaround may be more difficult than just solving the problem in the swiss-arrows repo. The workaround is to make my working-midje project (refer above) access a local build of swiss-arrows. The basic technique is outlined here https://gist.github.com/stuartsierra/3062743 , but a quick shot at it did not work (maven & classpath do not respond well to wishful thinking :). Still searching for the path of least resistance. May have more time on Wednesday. Will definitely have more time around Christmas and New Years.

[rplevy]

I think I will switch over to clojure.test. Midje works for me, but other people have problems with it.

[rebcabin]

Ah, ok. I like Midje, too, but I am on-board with going back to clojure.test.

[rplevy]

Made the move to clojure.test, see commit 39ff6c

Yeah I like the various features midje has for expressing common kinds of tests, and mocking/stubbing. However I have increasingly come over to the side that thinks the midje project has been too sloppy and buggy for too long. Ultimately if your test framework introduces uncertainty it is doing the opposite of what a test framework should be doing.

[rebcabin]

Just as a quick fyi, here is a Mathematica design of the example I am working towards with arrows: https://www.dropbox.com/s/v9htegavui9dyhe/OnlineIncrementalStatistics.nb.pdf . Will keep you posted of my progress.

[rebcabin]

some progress being made: note the solution proposed here APPLIES Composition to a list of functions to accomplish monadic chaining iteratively, reinforcing my initial guess that apply-> was going to be enabling for monadic patterns.

http://mathematica.stackexchange.com/questions/39249/writing-fold-in-terms-of-map-or-mapthread

[rplevy]

Just now reading your posts after being away for a few days on a vacation. That's really interesting stuff!

[rebcabin]

ooh, glad you approve. Many improvements coming, arrows still the inspiration :)

[rebcabin]

nearing lift-off of monadic tests. Preview here : https://github.com/rebcabin/swiss-arrows/blob/master/test/swiss/arrows/test.clj

[rebcabin]

I've gotten to a certain point and I'm stuck. I wonder if you can see a way out. For monadic chaining, an alternative (in this case, to the nil-propagating arrow) is the following. It's generalizable to other monads, of course, so it should just work out-of-the-box for the state monad and incremental stats and other magic. But here goes the current problem:

(defmacro =<>
  "the 'monadic diamond wand': top-level threading of monadic values
   through expressions"
  [monad x & forms]
  `(with-monad ~monad
     ((m-chain [~@forms]) ~x)))

with a test like the following

(is nil? (=<> maybe-m "abc"
              (fn [s] s)
              (fn [s] (if (string? "adf") nil s))
              (fn [s] (str s " + more"))
              ))

of course, what I really want is

(is nil? (=<> maybe-m "abc"
              <>
              (if (string? "adf") nil <>)
              (str <> " + more")
              ))

So I begin by modifying a copy of -<>* as follows: the idea is to replace each form with a function of a fresh argument s# wherein each appearance of <> is replaced by s#. Otherwise, it's the same as your original. The problem is that this is attempting to evaluate <> and I don't know why. I'll exhibit a test at the bottom of this message.

(defmacro ^:internal =<>*
  "TODO"
  [form default-position]
  (let [substitute-pos (fn [x form'] (replace {'<> x} form'))
        count-pos (fn [form'] (count (filter (partial = '<>) form')))
        c (cond
           (or (seq? form) (vector? form)) (count-pos form)
           (map? form) (count-pos (mapcat concat form))
           :otherwise 0)]
    (cond
     (> c 1)              (throw
                           (Exception.
                            "No more than one position per form is allowed."))
     (or (symbol? form)
         (keyword? form)) `(fn [s#] (~form s#))
     (= 0 c)              (cond (vector? form)
                                (if (= :first default-position)
                                  `(fn [s#] (vec (cons s# ~form)))
                                  `(fn [s#] (conj ~form s#))) ,
                                (coll? form)
                                (if (= :first default-position)
                                  `(fn [s#] (~(first form) s# ~@(next form)))
                                  `(fn [s#] (~(first form) ~@(next form) s#))) ,
                                :otherwise `(fn [s#] ~form))
     (vector? form)       `(fn [s#] (substitute-pos s# form))
     (map? form)          `(fn [s#] (apply hash-map
                                          (mapcat
                                           (partial substitute-pos s#)
                                           form)))
     (= 1 c)              `(fn [s#] (~(first form)
                                    (~substitute-pos s# ~(next form)))))))

a test

((=<>* (+ 40 <>) :first) 2)

produces

CompilerException java.lang.RuntimeException: Unable to resolve symbol: <> in this context, compiling:(/private/var/folders/cv/cxp762pj6t728007t2kyvfnh0000gp/T/form-init5499451975087797637.clj:1:2) 

Can you spot my mistake?

[rplevy]

Apologies for not responding. I've been busy moving to a new place and starting a new job. I will take a look sometime soon, or maybe you will figure it out before I get to it.

[rebcabin]

no hurry here -- i'm recovering from h1n1 flu, which became pneumonia and
has completely knocked me out for two solid weeks -- somehow i managed to
crawl up from the depths to do the little i have done. i expect to be
better in another week -- but note to self -- get the damn flu shot!

On Mon, Jan 6, 2014 at 5:12 PM, Robert Levy [email protected]:

Apologies for not responding. I've been busy moving to a new place and
starting a new job. I will take a look sometime soon, or maybe you will
figure it out before I get to it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-31704842
.

[rplevy]

wow that sounds terrible, get well soon!

[rebcabin]

Have begun some experiments with the continuation monad -- definitely arrowable through m-chain :) Next challenge is arrowing delimited continuations as with https://github.com/swannodette/delimc. More from me as time permits. Thanks for your past and future patience :) My calendar is absolutely jam-packed, but this is fun and important long-term.

[moea]

I think that extending the diamonds to replace <...> with the value in transit, in the context of an apply in the surrounding form is a cleaner approach than introducing a set of top-level macros for this. The example then becomes:

(-<>> [[1 2] [3 4]]
   (concat [5 6] <...>)
   (+ <...>)

What do you think of that?

[moea]

Here's a toy implementation if you want to mess around. Turns out that <...> isn't the best placeholder, because it gets interpreted as a class name if unbound, but I left it in for now. I can throw a branch together and fix the innumerable bugs in the below if it doesn't seem like a terrible idea.

(defmacro ^:internal -<>*
  [form x default-position]
  (let [substitute-pos (fn [form' & {:keys [op] :or {op '<>}}] (replace {op x} form'))
        count-pos (fn [form' & {:keys [op] :or {op '<>}}] (count (filter (partial = op) form')))
        [should-apply? c] (cond
                           (vector? form) [false (count-pos :op form)]
                           (seq? form) (let [c (count-pos form)]
                                         (if (zero? c)
                                           (let [apply-c (count-pos form :op '<...>)]
                                             [(< 0 apply-c) apply-c])
                                           [false c]))
                           (map? form) [false (count-pos (mapcat concat form))]
                           :otherwise [false 0])]
    (cond
     (> c 1)              (throw
                           (Exception.
                            "No more than one position per form is allowed."))
     (or (symbol? form)
         (keyword? form)) `(~form ~x)
         (= 0 c)              (cond (vector? form)
                                    (if (= :first default-position)
                                      `(vec (cons ~x ~form))
                                      `(conj ~form ~x)) ,
                                    (coll? form)
                                    (if (= :first default-position)
                                      `(~(first form) ~x ~@(next form))
                                      `(~(first form) ~@(next form) ~x)) ,
                                    :otherwise form)
         (vector? form)       (substitute-pos form)
         (map? form)          (apply hash-map (mapcat substitute-pos form))
         (= 1 c)              (if should-apply?
                                `(apply ~(first form) ~@(substitute-pos (next form) :op '<...>))
                                `(~(first form) ~@(substitute-pos (next form)))))))