data:processing:pipeline

About

A pipeline is a finite or infinite automata known as a stream) where:

the data
transition (pipe)
from one state to another
via a series of transformations (work)

A pipeline creates a composition relationship.

A pipeline is also known as:

Compose
Chain (for instance Chain of command) - Daisy Chain ;)

Pipeline follows a compositional structure known as cascade of operations.

Flow vs Pipeline

A dataflow (data workflow) is:

a directed graph structure (loop may be present)
of pipeline (ie the nodes are pipelines)

Model

A pipeline is a combination of:

an input source
an output destination.
and a sequence of data (that may be a byte of an object). In the general sense, it's called a message.
that may be chained to form a pipeline

Type

Imperative

The pipeline is executed step by step

Declarative

The pipeline is executed only when the terminal operation is called.

All steps are building a composite type known as algebraic data type.

Example

Shell

In an OS Shell (Dos, bash), a serie of command connected by the pipe operators forms a pipeline. See Pipeline

Code

By returning the calling object from a function, you can compose (or chain) functions. See Builder. When we compose (chain) an operation, the output of one operation becomes the input for the next operation, and operations are applied from left to right.

MapReduce

Pipeline

Library

Jva Apache Common Pipeline

Documentation / Reference

wiki/Pipeline_(computing)

Table of Contents