Command reference¶

mario¶

Mario: Python pipelines for your shell.

Docs: https://python-mario.readthedocs.org

Addons: https://mario-addons.readthedocs.org

Configuration:

Declarative config: /home/docs/.config/mario/config.toml

Python modules: /home/docs/.config/mario/m/*.py

mario [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options

--max-concurrent <max_concurrent>¶

--exec-before <exec_before>¶: Python source code to be executed before any stage.

--base-exec-before <base_exec_before>¶: Python source code to be executed before any stage; typically set in the user config file. Combined with –exec-before value.

--version¶: Show the version and exit.

Traversals¶

apply¶

Apply code to the iterable of items.

The code should take an iterable and it will be called with the input items. The items iterable will be converted to a list before the code is called, so it doesn’t work well on very large streams.

For example,

$ mario map int apply sum <<EOF
10
20
30
EOF
60

mario apply [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

chain¶

Concatenate a sequence of input iterables together into one long iterable.

Converts an iterable of iterables of items into an iterable of items, like itertools.chain.from_iterable.

For example,

$ mario eval '[[1,2]]'
[[1, 2]]


$ mario eval '[[1, 2]]' chain
[1, 2]

mario chain [OPTIONS]

eval¶

Evaluate a Python expression.

No input items are used.

For example,

$ mario eval 1+1
2

mario eval [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

filter¶

Keep input items that satisfy a condition.

Order of input items is retained in the output.

For example,

$ mario filter 'x > "c"' <<EOF
a
b
c
d
e
f
EOF
d
e
f

mario filter [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

map¶

Run code on each input item.

Each item is handled in the order it was received, and also output in the same order. For less strict ordering and asynchronous execution, see async-map and async-map-unordered.

For example,

$ mario map 'x*2' <<EOF
a
b
c
EOF
aa
bb
cc

mario map [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

reduce¶

Reduce input items with code that takes two arguments, similar to functools.reduce.

For example,

$ mario reduce map int operator.mul <<EOF
1
2
3
4
5
EOF
120

mario reduce [OPTIONS] FUNCTION_NAME

Options

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

FUNCTION_NAME¶: Required argument

walk¶

[EXPERIMENTAL] Walk a tree from the bottom up, transforming objects as you go.

For example,

$ mario read-json walk -t'int=x*2' write-json  <<EOF
[
        {"name": "Alice", "age": 21},
        {"name": "Bob", "age": 22}
]
EOF
[
    {
        "name": "Alice",
        "age": 42
    },
    {
        "name": "Bob",
        "age": 44
    }
]

mario walk [OPTIONS]

Options

-t <t>¶: Transformers for types. Pass a type-function pair. Each instance of the type will be converted with the function. For example, use -tdict=collections.OrderedDict to convert all dict objects to OrderedDict. The right-hand side of the = is Python code which will be executed. It may use x as a reference to the input value.

Commands for calling code on data.

Async traversals¶

async-apply¶

Apply code to an async iterable of items.

The code should take an async iterable.

mario async-apply [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

async-chain¶

Concatenate a sequence of input async iterables into one long async iterable.

Converts an async iterable of async iterables of items into an async iterable of items, like itertools.chain.from_iterable for async iterables.

mario async-chain [OPTIONS]

async-filter¶

Keep input items that satisfy an asynchronous condition.

For example,

$ mario async-filter '(await asks.get(x)).json()["url"].endswith(("1", "3"))'  <<EOF
http://httpbin.org/delay/5
http://httpbin.org/delay/1
http://httpbin.org/delay/2
http://httpbin.org/delay/3
http://httpbin.org/delay/4
EOF
http://httpbin.org/delay/1
http://httpbin.org/delay/3

mario async-filter [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

async-map¶

Run code on each input item asynchronously.

The order of inputs is retained in the outputs. However, the order of inputs does not determine the order in which each input is handled, only the order in which its result is emitted. To keep the order in which each input is handled, use the synchronous version, map.

In this example, we make requests that have a server-side delay of specified length. The input order is retained in the output by holding each item until its precedents are ready.

$ mario async-map 'await asks.get ! x.json()["url"]'  <<EOF
http://httpbin.org/delay/5
http://httpbin.org/delay/1
http://httpbin.org/delay/2
http://httpbin.org/delay/3
http://httpbin.org/delay/4
EOF
https://httpbin.org/delay/5
https://httpbin.org/delay/1
https://httpbin.org/delay/2
https://httpbin.org/delay/3
https://httpbin.org/delay/4

mario async-map [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

async-map-unordered¶

Run code on each input item asynchronously, without retaining input order.

Each result is emitted in the order it becomes ready, regardless of input order. Input order is also ignored when determining in which order to start handling each item. Results start emitting as soon as the first one is ready. It also saves memory because it doesn’t require accumulating results while waiting for previous items to become ready. For stricter ordering, see map or async_map.

In this example, we make requests that have a server-side delay of specified length. The input order is lost but the results appear immediately as they are ready (the delay length determines the output order):

$ mario async-map-unordered 'await asks.get ! x.json()["url"]'  <<EOF
http://httpbin.org/delay/5
http://httpbin.org/delay/1
http://httpbin.org/delay/2
http://httpbin.org/delay/3
http://httpbin.org/delay/4
EOF
https://httpbin.org/delay/1
https://httpbin.org/delay/2
https://httpbin.org/delay/3
https://httpbin.org/delay/4
https://httpbin.org/delay/5

mario async-map-unordered [OPTIONS] CODE

Options

--autocall, --no-autocall¶: Automatically call the function if “x” does not appear in the expression. Allows map len instead of map len(x).

--exec-before <exec_before>¶: Execute code in the function’s global namespace.

Arguments

CODE¶: Required argument

Commands for asynchronously calling code on data.

Read¶

read-csv-dicts¶

Read a csv file into Python dicts. Given a csv like this:

try:

$ mario read-csv-dicts <<EOF
name,age
Alice,21
Bob,22
EOF
{'name': 'Alice', 'age': '21'}
{'name': 'Bob', 'age': '22'}

mario read-csv-dicts [OPTIONS]

Options

--dialect <dialect>¶

CSV dialect (See https://docs.python.org/3/library/csv.html)

Options:	excel\|excel-tab\|unix

read-csv-tuples¶

Read a csv file into Python tuples. Given a csv like this:

try:

$ mario read-csv-tuples <<EOF
Alice,21
Bob,22
Carol,23
EOF
('Alice', '21')
('Bob', '22')
('Carol', '23')

mario read-csv-tuples [OPTIONS]

Options

--dialect <dialect>¶

CSV dialect (See https://docs.python.org/3/library/csv.html)

Options:	excel\|excel-tab\|unix

read-json¶

Read a single json string into a Python object.

For example,

$ mario read-json  <<EOF
[
    {"name": "Alice", "age": 21},
    {"name": "Bob", "age": 22}
]
EOF
[{'name': 'Alice', 'age': 21}, {'name': 'Bob', 'age': 22}]

mario read-json [OPTIONS]

read-json-array¶

Read a single json string into a Python object.

For example,

$ mario read-json-array  <<EOF
[
    {"name": "Alice", "age": 21},
    {"name": "Bob", "age": 22}
]
EOF
{'name': 'Alice', 'age': 21}
{'name': 'Bob', 'age': 22}


$ mario read-json-array map 'x["age"]' <<EOF
[
    {"name": "Alice", "age": 21},
    {"name": "Bob", "age": 22}
]
EOF
21
22

mario read-json-array [OPTIONS]

read-jsonl¶

Read a sequence of json entities into Python objects.

For example,

$ mario read-jsonl  <<EOF
{"a":1, "b":2}
{"a": 5, "b":9}
EOF
{'a': 1, 'b': 2}
{'a': 5, 'b': 9}

mario read-jsonl [OPTIONS]

read-text¶

Read input lines as a block of text, joining lines with a line separator.

For example,

$ mario read-text <<EOF
Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
EOF
Lorem ipsum dolor sit amet,
consectetur adipiscing elit,

$ mario read-text map len <<EOF
Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
EOF
56

mario read-text [OPTIONS]

Options

--sep <sep>¶: Separator to join input lines with

read-toml¶

Read a toml document into a Python object.

For example,

$ mario read-toml  <<EOF
[[persons]]
name = "Alice"
age = 21

[[persons]]
name = "Bob"
age = 22
EOF
{'persons': [{'name': 'Alice', 'age': 21}, {'name': 'Bob', 'age': 22}]}

mario read-toml [OPTIONS]

read-xml¶

Read xml into a Python object.

For example,

$ mario read-xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<message>
    <warning>
         Hello World
    </warning>
</message>
EOF
{'message': {'warning': 'Hello World'}}

mario read-xml [OPTIONS]

Options

--process-namespaces¶

read-yaml¶

Read a yaml document into a Python object.

For example,

$ mario read-yaml <<EOF
- Cat: "foo"
- Dog: "bar"
- Goldfish: "baz"
EOF
[{'Cat': 'foo'}, {'Dog': 'bar'}, {'Goldfish': 'baz'}]

mario read-yaml [OPTIONS]

read-yaml-array¶

Read a yaml document into a Python object.

For example,

$ mario read-yaml-array <<EOF
- Cat: "foo"
- Dog: "bar"
- Goldfish: "baz"
EOF
{'Cat': 'foo'}
{'Dog': 'bar'}
{'Goldfish': 'baz'}

mario read-yaml-array [OPTIONS]

Write¶

write-csv-dicts¶

Write a list of dicts to csv.

For example,

$ mario read-json write-csv-dicts --no-header <<EOF
[
      {
          "name": "Alice",
          "age": 21
      },
      {
          "name": "Bob",
          "age": 22
      }
]
EOF
Alice,21
Bob,22

mario write-csv-dicts [OPTIONS]

Options

--header, --no-header¶: Whether to write the dict keys as the first row

--dialect <dialect>¶

CSV dialect (See https://docs.python.org/3/library/csv.html)

Options:	excel\|excel-tab\|unix

write-csv-tuples¶

Write a list of tuples to csv.

For example,

$ mario read-json write-csv-tuples  <<EOF
[
        ["name", "age"],
        ["Alice", 21],
        ["Bob", 22]
]
EOF
name,age
Alice,21
Bob,22

mario write-csv-tuples [OPTIONS]

Options

--dialect <dialect>¶

CSV dialect (See https://docs.python.org/3/library/csv.html)

Options:	excel\|excel-tab\|unix

write-json¶

Serialize each input item to its json representation.

For example,

$ mario eval "[1, 2, 'foo']" write-json --no-pretty
[1, 2, "foo"]

Use the --indent option to set the indentation level:

mario write-json [OPTIONS]

Options

--pretty, --no-pretty¶

write-json-array¶

Write the input sequence into a json array.

$ mario read-json-array write-json-array map str.rstrip  <<EOF
[
        {
            "name": "Alice",
            "age": 21
        },
        {
            "name": "Bob",
            "age": 22
        }
]
EOF
[
    {
        "name": "Alice",
        "age": 21
    },
    {
        "name": "Bob",
        "age": 22
    }
]

mario write-json-array [OPTIONS]

Options

--pretty, --no-pretty¶

write-jsonl¶

Write a sequence to newline-separated json.

$ mario read-json write-jsonl <<EOF
[
        {"name": "Alice", "age": 21},
        {"name": "Bob", "age": 22}
]
EOF
{"name": "Alice", "age": 21}
{"name": "Bob", "age": 22}

mario write-jsonl [OPTIONS]

write-toml¶

Write each input item to its toml representation.

For example,

$ mario read-json write-toml map str.rstrip <<EOF
{
    "persons": [
        {
            "name": "Alice",
            "age": 21
        },
        {
            "name": "Bob",
            "age": 22
        }
    ]
}
EOF
[[persons]]
name = "Alice"
age = 21

[[persons]]
name = "Bob"
age = 22

mario write-toml [OPTIONS]

write-xml¶

Write a mapping to xml string.

For example,

$ mario eval '{"foo": {"bar": 1}}' write-xml
<?xml version="1.0" encoding="utf-8"?>
<foo>
    <bar>1</bar>
</foo>

mario write-xml [OPTIONS]

Options

--pretty, --no-pretty¶: Pretty-print the output

write-yaml¶

Write a yaml document.

$ mario read-json write-yaml map str.rstrip  <<EOF
[
        {
            "name": "Alice",
            "age": 21
        },
        {
            "name": "Bob",
            "age": 22
        }
]
EOF
- age: 21
  name: Alice
- age: 22
  name: Bob

mario write-yaml [OPTIONS]

Custom¶

xpath¶

Pull data out of xml documents using xpath.

For example,

$ mario xpath '//'  map 'x.text' <<EOF
      <slide type="all">
        <title>Overview</title>
          <item>Anything<em>can be</em> in here</item>
          <item>Or<em>also</em> in here</item>
      </slide>
EOF
Overview
Anything
can be
Or
also

mario xpath [OPTIONS] QUERY

Arguments

QUERY¶: Required argument

Custom defined commands

meta¶

Commands about using mario.

mario meta [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

pip¶

Run pip in the environment that mario is installed into.

Arguments are forwarded to pip.

mario meta pip [OPTIONS] [PIP_ARGS]...

Arguments

PIP_ARGS¶: Optional argument(s)

test¶

Run all declarative command tests from plugins and config.

Executes each test in the command.tests field with pytest.

Default pytest args: -vvv --tb=short

mario meta test [OPTIONS] [PYTEST_ARGS]...

Arguments

PYTEST_ARGS¶: Optional argument(s)

Commands about using mario.

mario meta [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Mario: Python pipelines for your shell.

Docs: https://python-mario.readthedocs.org

Addons: https://mario-addons.readthedocs.org

Configuration:

Declarative config: /home/docs/.config/mario/config.toml

Python modules: /home/docs/.config/mario/m/*.py

mario [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options

--max-concurrent <max_concurrent>¶

--exec-before <exec_before>¶: Python source code to be executed before any stage.

--base-exec-before <base_exec_before>¶: Python source code to be executed before any stage; typically set in the user config file. Combined with –exec-before value.

--version¶: Show the version and exit.