Marcel

Marcel now supports JSON input and output.

To convert a JSON document into a Python structure, use the `parse_json()` function.
A contrived example:

shell
bash ps | jc --ps | (p: json_parse(p)) | expand | (j: (j.pid, j.cmd))

* `bash ps | jc --ps` runs the bash `ps` command and then uses `jc --ps` to convert the output into a JSON document.
* `(p: json_parse(p))` converts the JSON into a Python structure.
* `expand` is a marcel operator that breaks a list into its elements. I.e., the input stream has one list with all the process info, and the output stream has a stream of structures each describing one process. Each structure has an object with fields describing process properties, e.g. `pid` and `cmd`.
* `(j: (j.pid, j.cmd))` extracts the `pid` and `cmd` of each process.

(This example is contrived because it can be done far more simply in marcel without JSON,
e.g. `ps | (p: (p.pid, p.command))`)

The mapping from JSON to Python is done by the Python `json` module. Consult documentation on
that module for details of the mapping.

Conversion of Python structures to JSON is done by the `json_format()` function. In this example,
maps of the form `{x: [x, x, x]}` are generated, for x = 0 ... 4. The structure is converted to a
JSON-formatted string by passing the stream, as a single list, to `json_format`. (By declaring the
parameter as `*m` instead of `m`, the stream of dicts is concatenated into a list of dicts.)

shell
({x: [x, x, x] for x in range(5)}) | (*m: json_format(m))

Marcel aims to be familiar to bash users.
In particular, it should be possible to issue simple bash commands via marcel.
One impediment to this goal was
that some characters have different interpretations in bash and marcel. For example, prior to this release,
marcel used `[...]` delimit a pipeline.
But in bash, the same symbols are used in glob patterns, e.g. `ls [0-9]*`.

With this release, pipelines are delimited by `(|...|)`. This change is necessary because
it did not seem possible to completely disambiguate `[...]`. For example, suppose a command contains

shell
... [grep foobar a-z] ...


One possibility is that grep is searching a file named `a-z` for lines containing `foobar`. But that's
a very odd filename, and odd usage of grep. Perhaps there is a typo, and a glob was intended: `[a-z]`, and the
actual pipeline terminating `]` appears later.

Changing the pipeline delimiter to `(|...|)` avoids these problems.

This release adds commands to upload local files to all nodes of a cluster, and to download
files from all nodes of a cluster.

Suppose that cluster `lab` has nodes `node1`, `node2`, `node3`, `node4`.
This would have been configured in the marcel startup script, e.g.

shell
lab = cluster(user='qa',
identity='/home/qa/.ssh/id_rsa',
hosts=['node1', 'node2', 'node3', 'node4'])

You could then upload the `scripts` directory to `/usr/local/foobar` in each of the nodes as follows:

shell
upload scripts lab /usr/local/foobar


You could download log files from all nodes, to the `logs` directory as follows:

shell
download logs lab /var/log/foobar/log*


In the `logs` target directory, the `download` command will create one directory for each node
of the cluster. So after this command, `logs` might contain the following (e.g.,
`ls -fr dest | (f: f.relative_to('dest'))`)

shell
node1/log.1
node1/log.2
node1/log.3
node2/log.1
node2/log.2
node2/log.3
node2/log.4
node2/log.5

The Linux `tee` command copies an
input stream to multiple files. Marcel now has a `tee` operator. For example:

shell
ls | tee [> f] [select (f: now() - f.mtime < days(1)) > r]


* `ls` lists the contents of the current directory.

* The `tee` command sends each received `File` produced by `ls` to each pipeline
and to the output stream, which causes the files to be printed.

* `[> f]` is the first pipeline, which simply writes the `File`s to the variable
`f`.

* `[select ... > r]` is the second pipeline. It locates those `File`s that have been
modified in the past day, and writes those `File`s to variable `r`.

The output from this command is the result from the `ls` command, but in addition,
the `f` and `r` variables have been set, and can be examined.
E.g., to print out the contents of `r`:

shell
r >

Bug fix. Assignment to a variable whose name matches that of an operator caused the parser to complain.

Bug fixes.