Pipe matching in Clojure YAP (metaprogramming in Lisp for beginners)

the

Introduction

A few days ago I discovered a wonderful JAPANESE Clojure — a modern Lisp dialects, a feature which is a good implementation means of multi-threading, compiling to bytecode the jvm, respectively the possibility of using the java libraries, jit compilation, etc. About Clojure, you can read for example here. But in this article we will talk about metaprogramming. Lisp is designed so that the data and code in it is one and the same. Function declarations, macros, function calls, deployment of macros in Lisp it's all just a list, possibly nested in each other.

the

(defn square [foo] (* foo foo))
(defmacro show-it [foo] `(println ~foo))

Such unity of code and data offers powerful capabilities for metaprogramming — code that writes code that writes code that writes code etc. — it is the most common thing for programming in Lisp. In compile-time us available all the functionality of the language, we can call a function to expand macros, possibly recursively. For example, if we define such a macro:

the

(defmacro recurs [foo bar]
(println "hello from compiler" foo)
(case (<= bar 0)
true `(defn foo [] ~foo)
false `(recurs ~(- foo 1) ~(- bar 1))))

And in your code insert a

the

(2 recurs 1)

Then at compile time we will see

the

hello from compiler 2
hello from compiler 1

And the macro in this case will unfold in the definition of the function foo with arity 1 and returns a value of 1, i.e.,

the

(def user/foo (clojure.core/fn ([] 1)))

If we write (recurs 3 1) the function foo will return the value 2, etc., Macros in Lisp is a great tool to conceal any complex logic or control structures beyond a simple syntax and therefore to the enhancement of the expressive means of the language itself. Many language constructs of the type "defn", "->>", "->" in reality too, just the macros.

Macros Lisp's normal functions, except that they are executed at compile time, and to master the technique of writing is in principle enough to know how to operate 4 the following "special forms"

the

`(expr) '(expr) ~(expr) ~@(expr)

This can be read in detail here. If very briefly — design quote ( ' and ` ) is just a notation to the compiler: "this expression should not try to run, and should be returned as is, i.e. in the form of code" and the unquote (~ and ~@) means roughly "to execute the given expression and insert the result into this place of code". Naturally unqoute-design have meaning only within a quote-designs. Thus a macro is a function that takes as arguments a quote-design that returns as values quote-design and execute at compile-time.

the

Why you need a matching pipe?

To demonstrate the power of Lisp macros will write the implementation of pipe matching to Lisp. What is a matching pipe? As you can tell from the name is a composition of two control structures: pipe and pattern matching.

Pipes in YAP Clojure is just a macro that accepts as arguments the n-Noe number of expressions and pop-up a certain way in the composition of these expressions, in very simple language, the following happens:
(-> expr1 expr2 expr3... ) payp "->" insert expr1 as the first argument in expr2 ( all other arguments are shifted by 1 position to the right ), thus expr12 inserted as the first argument in expr3, etc. Example

the

(-> (func1 "foo") 
(func2 "bar") 
(func3 123))

the

(func3 (func2 (func1 "foo") "bar") 123)

These two expressions are one and the same. The question of readability is a matter of taste, but personally to me it is obvious that the pipe needed. There are other pipes, for example ->> which is not difficult to guess this is similar to -> not only inserts the first and last argument. In General, you can write any kind of pipe, but as a rule most often used these two.
Pattern Matching is an obvious thing for example for erlang / elixir, haskell, ml developer, but in a nutshell there is not so easy to explain, it is rather necessary to feel. Very vaguely p.m. can be defined as a composition assignment and comparison. The uninitiated can refer to Wikipedia or to watch my previous article where I in a few words painted using p.m. YAP in elixir / erlang. In Clojure there is no native pattern matching, but as you might guess there are libraries that implement with macros p.m. almost identical to p.m. in erlang. We will not invent a Bicycle and use the match macro library in the project.

Now back to the question — why do you need to connect p.m. and pipe in a single entity matching pipe? When we write a program of level "hello world", where all functions are pure, no side — effects and everything is determined, it may not mean much. But in real production without the dirty functions are not enough. For example, we need to upload the image to an album in the social network VK. According to documentation for this we need the id of the album, the access key and the actual data that should be loaded. The General algorithm of loading the following

the

to read the file
to obtain the upload url (get http request, parsing the json)
to upload data (post-http request, parsing the json)
save the downloaded data (get-http request, parsing the json)

Reading the file, parsing json, http requests are all dirty function. During their call, anything can happen — there is no file at this address not valid json, valid json without the required fields, disconnection, server responded with 200 and something and so and so. In each of these functions is still very bad. And all anything, but each feature requires correct results from the previous one. In each function we can write clauses for the bad occasions and avoid accessnow, but you have to make it all work together, and if something goes wrong, we want to know what and where went wrong. Perhaps many of the readers will want to write something like this.



But it's a shame) Which is not that difficult to maintain and debajit, but just to read / write / understand. But in such situations (which agree very often) pipe matching can make code very easy, hiding all complex logic. Example use of macros pipe_matching and pipe_not_matching

the

(defn foo bar baz nested_process
(pipe_matching {:ok some_data}
(simple_func1 foo)
(simple_func2)
(simple_func3 bar)
(simple_func4 baz)))

What's going on here: nested_process function takes arguments foo, bar and baz. And then start posledovatelna calls may dirty simple_func1 functions with arity 1, simple_func2 with arity 1, simple_func3 with arity 2 and simple_func4 with arity 2, as in conventional pipes the result of the previous expression is the first argument, next. And now the most important thing is the first argument of the macro pipe_matching we asked pattern {:ok some_data}. Under this pattern will fit the map, where there is the key :ok with any value. And while simple_func functions return values that fit this pattern — it calls the following function (as in a normal pipe). But as soon as some of the functions simple_func return value is not suitable for this pattern — it will return as the function value nested_process and will not be passed further along the chain. For example, if simple_func3 then will return {:error "on server 500 ans simple_func3"}, function simple_func4 generally will not be called, and nested_process will return {:error "on server 500 ans simple_func3"}. You can also make a similar macro pipe_not_matching that can be used so

the

(defn foo bar baz nested_process
(pipe_not_matching {:error some_error}
(simple_func1 foo)
(simple_func2)
(simple_func3 bar) 
(simple_func4 baz)))

I think how it works is clear from the context. If simple_func3 then will return {:error "on server 500 ans simple_func3"}, and 1st and 2nd function to return the value not matching the pattern, the result will be the same as in the previous example. In practice I prefer pipe_not_matching. In the end we have almost a literal serialization logic problem in the code without the if / else / elseif / case / switch etc. In General, everything what we love the OP — code is the very task without the extra abstract entities. Cool? Now let's see how to write such macros in Lisp just a couple of lines.
the

Write pipe matching for Lisp

The first thing prescribe according to namespace our library — we just need one macro "match"

the

(ns pmclj.core
(:use [clojure.core.match :only (match)]))

Define in General terms 2 macro, which is our goal.

the

(defmacro pipe_matching [pattern init_expression & other_expressions ]
(pipe_matching_inner {:pattern pattern :result init_expression, :expressions other_expressions, :continue_on_match true}))
(defmacro pipe_not_matching [pattern init_expression & other_expressions ]
(pipe_matching_inner {:pattern pattern :result init_expression, :expressions other_expressions, :continue_on_match false}))

The macro will accept as the first argument pattern, the rest of the arguments — the expression for p itself.m. It is logical that we need at least one expression, so let's call it init_expression and put a mandatory second argument. Please note on the sign & here it means that other_expressions — list of any length (possibly empty) respectively consisting of the possible arguments (third, fourth, etc.). Thus a macro can be deployed with any number of arguments greater than 1.

Then just for the convenience of serializable arguments in a map with keys :pattern, :result, :expressions, :continue_on_match. result is the result obtained in the previous step expressions — the rest are not deployed in the resulting macro code expression continue_on_match — true / false: indicates what to do if result matched with the sample — to continue the chain of calls or return a value.

The obtained map pass to a recursive function which will return the pipe_matching_inner we need code. Yes, that's the beauty of Lisp, compile-time functions can be invoked, if only they were to the point of invocation is already compiled. The function looks in the following way.

the

(defn pipe_matching_inner [{pattern :pattern, result :result, expressions :expressions, continue_on_match :continue_on_match}]
(case (or (= expressions nil) (= expressions ()))
true result
false (let [to_pipe (first expressions) rest_expr (rest expressions)]
`(let [~'res ~result]
(case (= ~continue_on_match (check_match ~'res ~pattern))
true ~(pipe_matching_inner {:pattern pattern :result `(-> ~'res ~to_pipe), :expressions rest_expr, :continue_on_match continue_on_match})
false ~'res)))))

She takes a map, which we sformirovali in the body of the macro. Here by the way you can see what native p.m. in some form, and with a rather strange syntax in Clojure is still there: [{pattern :pattern, result :result, expressions :expressions, continue_on_match :continue_on_match}].
Case — expression here says the following: if we already processed all the expression remains nothing how to return the result. Otherwise, processed. Next comes a very important special form of the Lisp language — let. Global variables and assignment there is not and can not be, but we can do a local binding in the spirit of the expression => symbol. The functions first and rest return the first and in fact all but the first element of the list, then everything is transparent — we take the following expression to process.
Further the most interesting — the actual code which we dynamically generated at compile-time. There are already have to apply a little mental effort to understand what is happening. We are a locally bintim result from the previous expressions the symbol res (in runtime, not to execute the expression more than once). Note ~ ' is a small hack due to the fact that when compiling the symbols attached to the corresponding namespace, the combination of ~' them so to speak undocks from him, we will not in the context of this narrative to delve into these questions. Then follows the case expression, which honestly executed at runtime — we will check the suitability of the standard pattern our res. check_match is actually also the macro is very simple, based on a library macro match

the

(defmacro check_match [obj pattern]
`(match ~obj
~pattern true
:else false))

Further, depending on whether the expression check_match expected true to pipe_matching and false pipe_not_matching will either continue the call chain, or come back res. The subsequent call chain is as beautiful draw recursive function calls pipe_matching_inner.

Now let's see how it will look.
Here is a nice matching pipe
the

(pipe_matching {:ok some}
(func1 "foo")
(func2 "bar")
(func3 123))

Actually takes place here in a local ad

the

(clojure.core/let [res (func1 "foo")] (clojure.core/case (clojure.core/= true (pmclj.core/check_match res {:ok some})) true (clojure.core/let [res (clojure.core/-> res (func2 "bar"))] (clojure.core/case (clojure.core/= true (pmclj.core/check_match res {:ok some})) true (clojure.core/-> res (func3 123)) false res)) res, false))

I want to draw your attention that to build the same logic without matching pipe would have to write all this hell with his own hands)

Full code can be viewed here.

In conclusion, I want to say that at first glance, the syntax of Lisp is certainly not very user-friendly, but to build large / complex systems IMHO it fits well. Of course, after the erlang / elixir feel without otp without hands, but I'm sure it's a matter of time. Anyway this is my first experience with the jvm, and I think it will be pretty good)

the

UPD:

Did some refactoring after which the macros themselves don't look so scared. Listened to the advice and added another 4 macro:

pred_matching / pred_not_matching the same thing, only takes the first argument of the lambda with arity 1.
key_matching / key_not_matching the same thing, only takes the first argument to any key, and in runtime searches the call results values for a given key if value is nil or missing, this is equivalent to false otherwise true.

usage example

the

(defn func1 [arg]
{:ok, arg})
(defn func2 [arg1 arg2]
{:fail (+ (get arg1 :ok) arg2)})
(defn func3 [arg1 arg2]
{:ok (+ (get arg1 :ok) arg2)})

(defn example_pred []
(pred_matching #(contains? % :ok) 
(1, func1) 
(func2 2) 
(func3 3)))

(defn example_pm []
(pipe_matching {:ok some}
(1, func1) 
(func2 2) 
(func3 3)))

(defn example_key []
(key_matching :ok
(1, func1) 
(func2 2) 
(func3 3)))

That way you have an example of code where you need the correct return from the previous function: spot (+ (get arg1 :ok) arg2) potentially contains eksepsi, for example if the get function returns nil.
As expected all three functions return example in this case the same value.

the

(example_pred)
{:fail 3}
(example_pm)
{:fail 3}
(example_key)
{:fail 3}

Also added macros rkey_matching / rkey_not_matching work similarly, just looking for occurrences of a key in the structure, recursively, and find if — will stop chain of calls to return and found a mistake. As shown is the most convenient option in terms of ease of use / versatility.

The versatility of control structures then naturally such pred_matching > pipe_matching > rkey_matching > key_matching. What to use — depends on personal taste and the complexity of the data with which we work.

The resulting code here.

Article based on information from habrahabr.ru