Pipelining Package Tasks

In traditional monorepo task runners, like lerna or even yarn's own built-in workspaces run command, each npm lifecycle script like build or test is run topologically (which is the mathematical term for "dependency-first" order) or in parallel individually. Depending on the dependency graph of the monorepo, CPU cores might be left idle—wasting valuable time and resources.

Turborepo gives developers a way to specify task relationships explicitly and conventionally. The advantage here is twofold.

1. Incoming new developers can look at the Turborepo pipeline and understand how tasks are related.
2. turbo can use this explicit declaration to perform an optimized and scheduled execution based on the abundant availability of multi-core processors.

To give you a sense of how powerful this can be, the below diagram compares the turbo vs lerna task execution timelines:

Notice that turbo is able to schedule tasks efficiently--collapsing waterfalls--whereas lerna can only execute one task a time. The results speak for themselves.

Defining a pipeline#

To define your project's task dependency graph, use the pipeline key in the turbo.json configuration file at the root of your project. turbo interprets this configuration and conventions to properly schedule, execute, and cache the outputs of the tasks in your project.

Each key in the pipeline object is the name of a task that can be executed by turbo run. You can specify its dependencies with the dependsOn key beneath it as well as some other options related to caching.

Packages that do not have have this task defined in their package.json's list of scripts will be ignored and skipped over during turbo run.

An example Pipeline configuration in turbo.json:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]
    },
    "test": {
      "dependsOn": ["build"],
      "outputs": [],
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "test/**/*.ts"]
    },
    "lint": {
      "outputs": []
    },
    "deploy": {
      "dependsOn": ["build", "test", "lint"],
      "outputs": []
    }
  }
}

The rough execution order for a given package based on the dependsOn keys above will be:

1. build once its dependencies have run their build commands
2. test once its own build command is finished
3. lint whenever
4. deploy once its own build, test, and lint commands have finished

The full pipeline can then be run:

npx turbo run build test lint deploy

Turborepo will then efficiently schedule execution minimizing idle CPUs.

Task Dependency Format#

What you are declaring here in the pipeline object of the turbo configuration is a dependency graph of tasks. In the above example, in plain english, the configuration translates to the following conventions:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      // A package's `build` task depends on that package's
      // topological dependencies' and devDependencies'
      // `build` tasks  being completed first
      // (that's what the `^` symbol signifies).
      "dependsOn": ["^build"]
    },
    "test": {
      // A package's `test` task depends on that package's
      // own `build` task being completed first.
      "dependsOn": ["build"],
      "outputs": [],
      // A package's `test` task should only be rerun when
      // either a `.tsx` or `.ts` file has changed.
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "test/**/*.ts", "test/**/*.tsx"]
    },
    "lint": {
      // A package's `lint` task has no dependencies and
      // can be run whenever.
      "outputs": []
    },
    "deploy": {
      // A package's `deploy` task depends on the `build`,
      // `test`, and `lint` tasks of the same package
      // being completed.
      "dependsOn": ["build", "test", "lint"],
      "outputs": []
    }
  }
}

Topological Dependency#

The ^ symbol explicitly declares that the task has a package-topological dependency on another task.

A common pattern in many TypeScript monorepos is to declare that a package's build task (e.g. tsc) should only run once the build tasks of all of its dependencies in the monorepo have run their own build tasks. This type of relationship can be expressed as follows:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      // "A package's `build` command depends on its dependencies'
      // and devDependencies' `build` commands being completed first"
      "dependsOn": ["^build"]
    }
    // ... omitted for brevity
  }
}

Empty Dependency List#

An empty dependency list (dependsOn is either undefined or []) means that a task can start executing whenever it can! After all, it has NO dependencies.

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    // ... omitted for brevity

    "lint": {
      // A package's `lint` command has no dependencies and can be run at
      // whenever.
      "outputs": []
    }
  }
}

Restricting Inputs to a Task#

A package is considered to have been updated when any of the files in that package have changed. However, for some tasks, we only want to rerun that task when relevant files have changed. Specifying inputs lets us define which files are relevant for a particular task:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    // ... omitted for brevity

   "test": {
      // A package's `test` task depends on that package's
      // own `build` task being completed first.
      "dependsOn": ["build"],
      "outputs": [],
      // A package's `test` task should only be rerun when
      // either a `.tsx` or `.ts` file has changed.
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "test/**/*.ts"]
    }
  }
}

In this scenario, the test will only be rerun if a .tsx or .ts file has changed in the package.

Tasks that are in the pipeline but not in SOME package.json#

Sometimes tasks declared in the pipeline are not present in all packages' package.json files. turbo will automatically and gracefully ignore those. No problem!

pipeline tasks are the only ones that turbo knows about#

turbo will only account for tasks declared in the pipeline configuration. If it's not listed there, turbo will not know how to run them.

Implicit Dependencies and Specific Package Tasks#

Sometimes you need to manually place a package-task dependency on another package-task. This can occur especially in repos that are just coming off of a lerna or rush repository where the tasks are traditionally run in separate phases. Sometimes assumptions are made for those repositories that are not expressible in the simple task pipeline configuration, as seen above. Additionally, you may need to express implicit dependencies or sequences between applications or microservices when using turbo in CI/CD.

For these cases, you can express these relationships in your pipeline configuration like this:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]
    },
    "test": {
      "dependsOn": ["^build"],
      "outputs": [],
    },
    "deploy": {
      "dependsOn": ["test", "build"],
      "outputs": []
    },
    "frontend#deploy": {
      "dependsOn": ["ui#test", "backend#deploy", "backend#health-check"],
      "outputs": []
    }
  }
}

In this example, we illustrate a deploy script of a frontend application depends on both the deploy and health-check npm scripts of backend as well as the test script of a ui package. The syntax is <package>#<task>.

This seems like it goes against the "test": { "dependsOn": ["build"] } and "deploy": { "dependsOn": ["test"] }, but it does not. Since test and deploy scripts do not have topological dependencies (e.g. ^<task>), they theoretically can get triggered anytime once their own package's build and test scripts have finished! Unless they are being used for applications/services for deployment orchestration, the general guidance is to get rid of these specific package-task to package-task dependencies in the pipeline as quickly as possible (so the builds can be optimized better).

Running tasks defined in the root package.json#

turbo can run tasks that exist in the package.json file at the root of the monorepo. These must be added to the pipeline configuration using the form "//#<task>": {...}. This is true even for tasks that already have their own entry. For example, if you already have "build": {...} in your pipeline, but want to include the build script defined in the root package.json when running turbo run build, you must opt the root into it by also including "//#build": {...} in your configuration. Conversely, you do not need to define a generic "my-task": {...} entry if all you need is "//#my-task": {...}.

A sample pipeline that defines the root task format and opts the root into test might look like:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]
    },
    "test": {
      "dependsOn": ["^build"],
      "outputs": [],
    },
    // This will cause the "test" script to be included when
    // "turbo run test" is run
    "//#test": {
      "dependsOn": [],
      "outputs": []
    },
    // This will cause the "format" script in the root package.json
    // to be run when "turbo run format" is run. Since the general
    // "format" task is not defined, only the root's "format" script
    // will be run.
    "//#format": {
      "dependsOn": [],
      "outputs": ["dist/**/*"],
      "inputs": ["version.txt"]
    },
  }
}

A note on recursion: Scripts defined in the root package.json often call turbo themselves. For example, the build script might be turbo run build. In this situation, including //#build in turbo run build will cause infinite recursion. It is for this reason that tasks run from the root must be explicitly opted into via including //#<task> in the pipeline configuration. turbo includes some best-effort checking to produce an error in the recursion situations, but it is up to you to to only opt in those tasks which don't themselves trigger a turbo run that would recurse.