uberjob¶

uberjob is a Python package for building and running call graphs.

Its powerful primitives can be combined to build massive and sophisticated pipelines. Previous outputs are automatically reused when possible, providing incremental builds for free and enabling seamless recovery from errors. This eliminates the need for a heavyweight scheduling platform, and makes development and production environments simpler and more consistent.

The package is built around two core classes: uberjob.Plan and uberjob.Registry. A Plan represents a call graph and a Registry maintains how and where values are stored.

The lessons below introduce important concepts in uberjob.