Lesson 3: Gather

The gather operation is often used implicitly and unconsciously, but its details are worth understanding. gather() converts non-symbolic values to symbolic values. For example, it will convert an integer to an integer literal, and it will convert a list of symbolic nodes into a symbolic node whose value is a list. To achieve this latter behavior, it must recursively search structured values for symbolic nodes. When doing this, it only recognizes Python’s general purpose built-in containers: dict, list, set, and tuple.

call() automatically gathers its arguments, and uberjob.run() automatically gathers its output.

Example

Implicit gathering. This is the preferred way. Note that literals are created explicitly here for illustration purposes; implicit gathering is the preferred way to create them too.

>>> import uberjob
>>> plan = uberjob.Plan()
>>> x = plan.lit(1)
>>> y = plan.lit(2)
>>> z = plan.call(sum, [x, y])
>>> uberjob.run(plan, output=z)
3
>>> uberjob.render(plan)

More complex structured values can be gathered as well:

>>> import json
>>> plan = uberjob.Plan()
>>> x = plan.lit(1)
>>> y = plan.lit(2)
>>> z = plan.call(json.dumps, {'x': x, 'y': y, 'w': [x, (y, y)]})
>>> uberjob.run(plan, output=z)
'{"x": 1, "y": 2, "w": [1, [2, 2]]}'
>>> uberjob.render(plan)

The output argument of run() is also implicitly gathered:

>>> plan = uberjob.Plan()
>>> x = plan.lit(1)
>>> y = plan.lit(2)
>>> uberjob.run(plan, output=[x, y])
[1, 2]

Gather can be called explicitly, but this is usually unnecessary:

>>> plan = uberjob.Plan()
>>> x = plan.lit(1)
>>> y = plan.lit(2)
>>> w = plan.gather([x, y])
>>> z = plan.call(sum, w)
>>> uberjob.run(plan, output=z)
3

Warning

gather will raise a stack overflow exception if passed a structured value that contains a cycle, such as a list that contains itself. This will also happen with call because it gathers all of its arguments. To resolve this, explicitly convert the offending value to a literal using lit.