Runbooks Overview

This page explains the feature that runs and manages routine procedures, such as incident response and routine operations, automatically as a series of steps that you define in advance.

Overview

Runbooks in Lorel.ai let you automate operations workflows by combining command execution, script execution, external API calls, cloud operations, approval flows, conditional branches, and more.

A Runbook you create has one of three statuses: [Draft], [Active], or [Inactive]. Only an Active Runbook can run.

A Runbook can start in several ways: a user runs it manually, or it starts automatically, for example from an incident policy.

• Step definitions. Combine shell commands, scripts such as Python and Bash, external APIs, cloud operations, notifications, and approvals.
• Execution control flow. Runbooks support conditional branches, retry loops, loops over a list, parallel execution, and waiting for a condition.
• Safe operations by design. Runbooks have built-in manual approval steps, automatic recovery when a step fails (jump, skip, or stop), and limits that prevent runaway execution.

What you can do

The following table lists the main features and operations of Runbooks, and the project roles that you need.

The lifecycle of a Runbook

A Runbook is managed with three statuses.

[Draft] → [Active] ⇄ [Inactive], [Inactive] → [Draft]

• Draft. The Runbook is being created or edited and can’t run yet (the state saved with [Save as Draft]).
• Active. The Runbook is available. It can be run manually or started automatically (the state saved with [Save and Activate]).
• Inactive. The Runbook is temporarily disabled. You can’t start a new execution (past execution history is kept). From Inactive, you can activate it again or move it back to draft.

You can’t delete an Active Runbook. To delete it, deactivate it first.

How an execution starts

A Runbook execution starts from one of the following.

How to read and use the Runbooks list

When you open [Runbooks] in the sidebar, the list shows every Runbook registered in the project.

The operation area at the top of the screen

Use the bar at the top of the screen to filter and search Runbooks and to create a new one.

The columns of the list

When you select a row in the list, the detail screen of that Runbook opens. Each column shows the following information.

The number of executions, the success rate, and the last executed date are calculated from completed executions only. They exclude test runs, child executions started from a sub-Runbook or parallel execution, and cancelled or failed executions.

The [Last Run] badge, on the other hand, shows the state of the most recent execution as it is. This includes test runs, child executions, and failed or cancelled executions. The [Cancelled] badge is the same state that the Execution Detail screen shows as “Cancelled.”

Note: To sort by a sortable column, select the ↑↓ icon in the column header to switch between ascending and descending order.

How to create a Runbook

To create a new Runbook, open [Runbooks] in the sidebar, and then select [Create Runbook] in the upper-right corner of the screen.

The creation screen has three tabs: [Basic Info], [Knowledge], and [Steps]. You can select [Save as Draft] or [Save and Activate].

Set the basic information (required)

Set the basics of the Runbook, such as its name, description, time constraints, and tags.

How to write a clear description (a guide for AI agents and operators)

Select [Writing guide] next to the description field to see a guide on how to write it.

• Purpose & action. Write what the procedure is for and what it does. (For example, it restarts the target to restore the service; when the disk is running out of space, it moves old logs aside to free up space.)
• Target & environment. Identify the target system and environment by their specific names. (For example, the production Aurora (MySQL) cluster; the web servers (EC2) in the staging environment.)
• Trigger & thresholds. Write when to use it, with concrete numbers. (For example, when CPU utilization stays above 90% for 10 minutes or more; when replica lag continues for 5 minutes or more.)
• When not to use it. Write the conditions under which not to use it, and how it differs from similar procedures. (For example, it isn’t used when the lag is less than 5 minutes; it doesn’t apply to the staging environment.)
• Results & side effects. Write what happens after it runs, including effects that can’t be undone. (For example, the service is unreachable for a few minutes during the restart; deleted data can’t be restored.)

Write in the active voice and present tense, with a clear subject, as objective facts (avoid imperatives). Don’t write commands or individual work steps in the description. Put them in [Steps] or [Knowledge].

Set the knowledge (optional)

You can record detailed operating procedures, troubleshooting notes, and configuration notes for the Runbook in Markdown (up to 20,000 characters).

You can switch between two editing modes at any time from the link below the input area.

• Switch to rich editor. Edit as rich text, where you can see formatting such as headings, bold text, and lists as you type.
• Switch to simple editor. Edit directly as plain Markdown text without formatting.

Set the steps (at least one is required to activate)

Define the series of processing steps that the Runbook actually runs. If you only save it as a draft, you can save it with no steps.

When a new Runbook doesn’t have any steps yet, select [Add Step] in the center of the screen to add one. After you add a step, tabs to switch between [Step list] and [Flow diagram] appear at the top of the screen, and you can switch to the view you prefer to check the structure at any time.

Working with step cards and reordering

On each step card that you add, you can do the following.

• Reorder the steps. Hover over the drag handle on the left of the card, and then drag and drop it up or down.
• Collapse and expand. Select the chevron icon on the right of the card (pointing down when expanded, pointing right when collapsed) to open or close its settings.
• Delete a step. Select the trash icon in the upper-right corner of the card.

Basic fields common to all steps

• Type (required). Select the type of step you want to run from the drop-down list.
• Step Name (required). Enter a name for the step. You can use only letters (a-z, A-Z), digits, underscores, and hyphens, and the name must be unique in the Runbook (for example, check_db_process). Later steps use it as the key when they reference the output. Avoid hyphens, because an expression treats a hyphen as a minus sign.
• Description (optional). Enter what the step does and why.

Execution control common to all steps

When you expand “Execution control” under “Advanced settings” on a step card, you can set a time limit for each step and automatic recovery for errors.

Rules for jumping to a step

When you select [Jump to step] for [On Failure], the behavior depends on where the jump target is.

• When the target is after the current step. All the steps that haven’t run yet between the failed step and the target are skipped automatically, and processing continues from the target step.
• When the target is the current step or before it. The range from the target to the current step runs again as a recovery loop.
• When the target step can’t be found (a safety measure). If the target setting is invalid, Lorel.ai automatically treats it as [Stop], to prevent unexpected behavior.

List of step types

In Runbooks, you build a workflow by freely combining steps that fit your purpose and role. The step type drop-down on the screen shows all the types in a single list. This page groups them by purpose.

Action and integration steps

These steps perform actual operations and checks on EC2 instances, cloud environments, and external services.

Communication and approval steps

These steps notify the people who operate the system, and make a decision before an important step runs.

Control flow and variable steps

These steps control branching, repetition, parallel processing, and passing variables between steps. Retry Loop, For Each, and Parallel Group steps run other steps (inner steps) that you place inside their frame. The types of steps you can place inside have the following limits.

• Retry Loop and For Each. You can place only Command, Script, API Call, Set Variable, Notification, Cloud Action, Call Sub-Runbook, and Condition steps.
• Parallel Group branches. You can place only Command, Script, API Call, Notification, Delay, and Wait For Condition steps.

Using outputs and variables between steps

You can insert the execution results of earlier steps and the variables you set in advance into input fields of later steps, such as message bodies and conditional expressions, and use them dynamically.

Later steps can reference the execution result of each step in the form steps.<step name>.<item name>, and the variables you set so far in the form ctx.<variable name>. The two differ in nature.

• Step output (steps.*). Once it is recorded, the value is kept and isn’t overwritten later. However, when a jump on failure goes back to that step and runs it again, the previous output is discarded and the new result is recorded.
  • In a notification body, write it like {{ steps.check_process.stdout }}.
  • In a condition, write it like steps.check_db.exit_code == 0.
• Variables (ctx.*). By running a Set Variable step several times, you can assign a new value to a variable with the same name and overwrite or update it repeatedly.
  • In a retry condition, write it like ctx.retry_count < 3.

Rules for passing outputs and variables out of control flow steps

For control flow steps that have inner steps inside a frame, whether later steps can use the execution results and variables of the inner steps depends on the type.

• For Each. Access the item being processed with {item} (the current value) and {index} (the zero-based position). If you use {item} in a Command or Script step, the value can contain only letters, digits, and the characters _ - . / : = + @ , (no spaces); otherwise the step fails. If you use {item} in the parameters of a Cloud Action, Notification, or Call Sub-Runbook step, the value can’t contain curly braces. If you use it in the parameters of an API Call step, the value can’t contain line breaks. In either case, the step fails if it does. The execution results of the inner steps and the variables (ctx.*) that are set inside are both discarded for each iteration. Steps after the loop, and the next iteration, can’t use them (even the result of the last iteration isn’t kept).
• Retry Loop. If a retry succeeds, only the execution results of the inner steps and the variables (ctx.*) set inside in the successful attempt are passed on to later steps.
• Parallel Group. If all branches finish normally, the execution results (steps.*) of the inner steps of each branch are passed on to later steps together. You can’t set variables inside a parallel branch.
• Call Sub-Runbook (synchronous). Variables that are set in the called Runbook are passed on together as the execution result of the calling step itself. However, the calling side defines the name execution_mode as a system-reserved name. If the called Runbook outputs a variable with the same name, execution_mode, the reserved value overwrites it.

The Runbook detail screen

On the detail screen of a Runbook that you created and saved, you can check and edit the definition, and also check the operational metrics, the linked items, and the past execution history.

Operations at the top of the screen

You can do the following from the top of the detail screen.

• Run. Runs an Active Runbook right away. When you select [Run], a confirmation dialog opens. [Test run] is turned on by default, so to run the Runbook live, clear the check box. A test run doesn’t change your actual environment in principle (for the exceptions, see the test-run item under Limitations later on this page).
• Edit. Opens the edit screen for the settings and steps.
• Clone. Creates a new draft Runbook based on an existing one.
• Activate or deactivate. Use the switch next to the title to switch the status between [Active] and [Inactive].
• Delete. Deletes a Runbook that you no longer need from the project. It is in the [More actions] menu. You can’t delete an Active Runbook, so deactivate it first.
• Move back to draft. Moves an Inactive Runbook back to a draft. It is in the [More actions] menu.
• Execution access. For a Runbook that includes a cloud action, sets whether Workers can run it ([Allow execution by WORKER]).

What each section shows

Managing executions and troubleshooting

You can check the status of a Runbook execution and the progress of its steps in real time in “Execution History” on the Runbook detail screen, or on “Execution Detail,” which opens when you select an execution in the history. Execution Detail has two tabs, [Steps & log] and [Flow diagram]. In Execution History, the [Mode] column shows [Dry-run] for a test run and [Live] for a real run ([Test run] in the run dialog is the same as a dry-run). When an execution pauses or hits a problem, you can recover manually from the screen.

The status of the whole execution

The whole execution of a Runbook moves through the following statuses.

The status of each step

Separately from the status of the whole execution, each step has one of the following statuses. On Execution Detail, a step’s status appears mainly as an icon and a color, and the step row has no dedicated status label such as Completed or Failed. The following table explains what each status means; the names aren’t labels shown on the screen.

Manual recovery (manual retry and manual skip)

When an execution is stopped in Paused or Awaiting Approval, you can retry or skip the step that caused it manually, so that the execution continues.

This differs from the automatic retry and automatic skip on failure in the step settings. It is a manual operation that an operator performs for an individual step on the screen.

• Manual retry ([Retry] button on the step row). For an approval step whose approval was rejected, the button is named [Re-request approval]. You can run it when the whole execution is Paused and the target step is Failed. The target step returns to Pending, the whole execution returns to Running, and processing continues (the retry count is incremented).
• Manual skip ([Skip] button on the step row). You can run it when the whole execution is Paused or Awaiting Approval, and the target step is Failed or Awaiting Approval. The target step becomes Skipped, the whole execution returns to Running, and processing moves on to the next step.
• Cancel the execution ([Cancel Execution] button). Use this when the execution can’t continue. It ends the whole execution safely as Cancelled. Select [Cancel Execution] in the confirmation dialog to confirm. Cancelling doesn’t roll back the results (side effects, such as command execution or API calls) of steps that already ran.

Conditions that make an execution fail automatically (expire)

If any of the following applies, a running Runbook ends automatically as “Failed,” without waiting for a user to step in.

• The execution stays paused for a certain period of time or longer. This is a safety measure for a paused execution that is left alone. It applies to a pause caused by a step failure. It doesn’t apply to a pause caused by a rejected approval or by an approval that was rejected automatically after it timed out, so those need a manual retry, skip, or cancel.
• The execution exceeds the time limit for the whole execution (the maximum run time). This applies only if you set a maximum run time in [Basic Info] in advance (if you didn’t, there is no timeout for the whole execution, and it is unlimited).

When an execution that started automatically from an incident, from Tasks, or from a sub-Runbook call reaches an approval step that has no timeout set, a default timeout of 60 minutes applies (this default doesn’t apply to a run started by approving an AI agent’s proposal, or to a manual run). When an approval times out, by default it is rejected automatically and the execution becomes paused.

Roles and permissions

Each Runbook operation needs the permissions of a project role.

Note: In a Runbook whose steps reference an Access that was activated by manual approval, only Project Administrators can add or change steps and clone the Runbook. Activating it is also limited to Project Administrators, including when the Access is reached through a sub-Runbook.

Note: In a test run, by default a Project Administrator or Editor can approve or reject even if they aren’t the designated approver (for details, see Limitations later on this page).

Note: A person who ran a Runbook manually can approve or reject the approval step of that run, if they are designated as an approver. Lorel.ai records who approved and who ran it.

Note: You can allow Workers to run only a Runbook that includes a cloud action (including through a sub-Runbook). Workers can run a Runbook that doesn’t include a cloud action without any additional setting.

Note: Turning on [Allow execution by WORKER] for a Runbook that reaches an Access activated by manual approval (including through a sub-Runbook) is limited to Project Administrators. Turning it off is always allowed.

Note: If a change to the definition alters whether the Runbook includes cloud actions, or the set of sub-Runbooks that it calls, the execution permission for Workers is revoked automatically.

For details, see Roles and Permissions.

Limits and safeguards against runaway execution

To prevent excessive resource use and infinite loops, the following constraints apply when you create and run a Runbook.

Input limits

Limits for control flow and execution (preventing runaway execution)

Timeout constraints

• Timeout for each step. You can set it on Command, Script, API Call, and Cloud Action steps. The range is 1 to 300 seconds (up to 5 minutes), and 30 to 300 seconds for Command and Script steps, to make sure the processing finishes safely.
• Timeout for control flow steps. Retry Loop, For Each, Parallel Group, Wait For Condition, and Call Sub-Runbook steps don’t use the per-step timeout field (1 to 300 seconds). Each step type has its own settings for time and count (such as the maximum polling wait time and the number of loops).

Limitations

These are the specifications and limitations to understand in advance when you design and operate Runbooks.

• Targets of Command and Script steps. You can specify only Amazon EC2 instances (with the SSM Agent installed). On-premises servers and ECS containers can’t be specified as targets.
• Cancelling an execution and what happens on the target instance. When you cancel an execution while a Command, Script, or API Call step is running, it goes through “Cancelling…” and then becomes “Cancelled.” If none is running, it becomes “Cancelled” immediately. A Command, Script, or API Call step that was running is recorded as “Interrupted,” and the other steps that hadn’t finished (including steps awaiting approval or waiting for children, and running Cloud Action and Notification steps) are recorded as “Skipped.” Steps that already failed, completed, or were skipped stay as they are. A process that has already started on the target instance can run to completion, but its result isn’t reflected in the Runbook’s log. Also, cancelling doesn’t roll back the results (side effects, such as command execution or API calls) of steps that ran before the cancellation.
• Security and confidential information restrictions for cloud actions. To protect security, operations such as the following are always restricted, regardless of the permissions of the account on the cloud side.
  • Directly retrieving secrets from a secret management service (such as AWS Secrets Manager)
  • Issuing temporary credentials (such as AWS STS)
  • Creating a new IAM access key or changing login information
  • Retrieving an encrypted parameter in decrypted form (such as a SecureString in AWS Systems Manager Parameter Store; retrieving an unencrypted parameter is allowed)
  • Running helper functions that depend on client-side libraries, such as generating a presigned URL
• Scope of cloud actions. For an Access tagged [Allowed operations only], a cloud action runs only within the scope of operations that you selected. A manually approved Access (tagged [Arbitrary execution]) isn’t limited to a selected scope. The actions that you declare in a step (“Declaration required” and “Declared actions”) can be used only with a manually approved Access. Declaring them doesn’t lift the limits on an [Allowed operations only] Access.
  • Operations on IAM, STS, Organizations, Account, and billing-related services (such as Billing), and operations that start with delete, terminate, create, or run, are denied by default. A manually approved Access can run them when you declare them in the step. However, the operations that are always restricted above can’t be run even if you declare them.
  • STS operations can’t be declared. They can never run.
  • One step can declare up to 16 actions, across up to 12 kinds of services.
• Communication limits for external API calls. To keep communication safe, you can’t call an unencrypted HTTP URL or a URL for an internal or private IP address (only HTTPS communication is supported).
• Steps that don’t expand variable references. In the parameters of Command and Script steps, variable references ({{ ... }}) aren’t expanded and are treated as literal text. You can use variable references in notification bodies, in the arguments of cloud actions, in the input mapping of a sub-Runbook, and in conditional expressions.
• Outputs and variables of the inner steps of a For Each. The output values of the steps that ran inside a For Each step, and the variables (ctx.*) set inside, aren’t available to steps after the processing finishes or to the next iteration (even the result of the last iteration isn’t kept).
• Setting variables in parallel execution and the number of branches that run at the same time.
  • To prevent variable conflicts, you can’t place a Set Variable step inside a parallel branch.
  • The number of branches in a parallel group is the maximum you can define. Depending on the system load and other conditions, the number of branches that actually run at the same time can be higher or lower.
• When the API runs in Wait For Condition. The API Call step that a Wait For Condition step references runs once in the normal flow of execution, and then runs again at every poll. The referenced API call must use GET, and the polling interval is 5 to 900 seconds.
• Asynchronous sub-Runbook calls. If you call a sub-Runbook with [Async (don’t wait)], the status of the calling (parent) execution isn’t affected even if the called (child) Runbook fails later.
• The run time of a sub-Runbook’s child execution. The time spent waiting for a synchronously called sub-Runbook counts toward the maximum run time of the calling (parent) execution. The maximum run time that you set on the child Runbook doesn’t apply to an execution that starts from a sub-Runbook.
• Behavior of each step type in test runs. A test run doesn’t change your actual environment in principle. Each step type behaves as follows.
  • Command, Script, and Cloud Action. Command and Script steps aren’t run, but a read-only reachability check of the target instance is actually performed, and if it fails, the test run fails. Cloud Action steps send nothing and are simulated (mock behavior).
  • Notification. A step that you newly add on the screen doesn’t send in a test run by default. Turn on [Send notifications on test runs] in the step to make it send. A step whose [Notify] toggle is off never sends.
  • Delay. It really waits in a test run by default (you can skip the wait in the step’s settings).
  • API Call. It is skipped in a test run by default. Turn on [Send the real request during test runs] to send it.
  • Approval. It waits for a real approval, even in a test run. By default, a Project Administrator or Editor can approve or reject even if they aren’t the designated approver. Turn on [Enforce approver restrictions on test runs] in the step to limit this to the designated approvers.

Related features

• Rules Overview (detecting incidents, and the incident policies that start a Runbook automatically)
• AI Agents Overview (approving a Runbook that was proposed as a result of analyzing an incident)
• Incidents / Tasks Overview (linking a Runbook to a task and checking the execution status)
• Integration Overview (the Access that cloud actions use)
• Roles and Permissions (details of the Project Administrator, Editor, Worker, and Viewer permissions)
• Data Handling (Runbook execution history is kept for 90 days on all plans)