Pipelines Overview

You can use CMS Pipelines to solve a large problem by breaking it up into a series of several smaller, less complex programs. These programs, called stages, then can be hooked together to form a pipeline to give you the results you want.

Some advantages of using CMS Pipelines are:

  • Several stages can be connected to accomplish a complex function making CMS Pipelines a more efficient approach to programming
  • Numerous commands provide the ability to transform data in many different ways
  • CMS Pipelines provides an interface that lets you write your own stage commands in the REXX or Assembler language
  • Code is reused each time a stage is run in a pipeline and requires no modification or recompilation
  • CMS Pipelines applications are simpler to create and debug because there are no device interface dependencies
  • CMS Pipelines supports most CP and CMS devices and interfaces, many of which are not available to REXX programs using standard CMS interfaces
  • Built-in interface to SQL/DS and ISPF.

Use the PIPE command to call CMS Pipelines. The operand of the PIPE command consists of at least one pipeline. A pipeline is a series of stages, and stages are separated by a character called a stage separator. The default stage separator character has a value of X'4F', which is usually the solid vertical bar (|).

A stage reads data, processes the data, and writes the data to the next stage. The output from one stage is the input to the next stage. Each stage reads its input and writes its output through a device-independent interface without concern for other stages in the pipeline. Because data enters a pipeline and moves through a pipeline's stages, the data is said to flow through a pipeline. The flow of data records into a stage and out of a stage is called a stream.

The PIPE command supports an extensive set of stage commands you can specify for a stage. Using stage commands, you can:

  • Enter CP and CMS commands
  • Read from or write to your terminal
  • Read, create, or append to a CMS file
  • Access BFS data
  • Read a CMS file backwards
  • Copy records
  • Select a subset of records
  • Manipulate multiple records
  • Rearrange the contents of records
  • Sort records
  • Discard or retain duplicate records
  • Truncate records
  • Count lines, blank-delimited words, and bytes
  • Block data
  • Deblock external data formats.

The preceding list shows just a few examples of CMS Pipelines functions.

Filters and device drivers are two types of stage commands. Filters transform data by performing operations such as sorting, truncating, and selecting records. Device drivers read data from or write data to a device, CMS, CP, or other host environments, such as reading a file from disk or writing a file to tape.

The following example of a PIPE command shows the structure of a pipeline:

pipe < PROFILE EXEC A | count lines | console This example contains three stages. The stages are delimited by the solid vertical bar, which is the default stage separator character. < PROFILE EXEC A is the first stage of the pipeline. It reads the records from the file PROFILE EXEC A into the pipeline.

count lines is the second stage of the pipeline. It counts the number of lines in the file PROFILE EXEC A.

console is the third stage of the pipeline. It writes the number generated by the second stage to your terminal.

When CMS Pipelines does not provide the necessary commands to perform a particular function, you can write your own commands. These user-written commands are exec procedures that must be written in the REXX or Assembler language. The file type of a user-written stage command should be REXX or ASSEMBLE. You use user-written stage commands in a pipeline the same way you use the stage commands provided by CMS Pipelines.

In addition, a pipeline that consists of several stages can be cumbersome and time-consuming to retype each time at your terminal. In cases like this, it is more efficient to put the pipeline in a REXX exec where it can be modified more easily and reused.

To further extend the versatility of CMS Pipelines, an unlimited number of interconnected pipelines are supported. One PIPE command can specify multiple pipelines.