How to create a conditional task in Airflow

I would like to create a conditional task in Airflow as described in the schema below. The expected scenario is the following:

  • Task 1 executes
  • If Task 1 succeed, then execute Task 2a
  • Else If Task 1 fails, then execute Task 2b
  • Finally execute Task 3

Conditional Task All tasks above are SSHExecuteOperator. I'm guessing I should be using the ShortCircuitOperator and / or XCom to manage the condition but I am not clear on how to implement that. Could you please describe the solution?

You have to use airflow trigger rules

All operators have a trigger_rule argument which defines the rule by which the generated task get triggered.

The trigger rule possibilities:

    ALL_SUCCESS = 'all_success'
    ALL_FAILED = 'all_failed'
    ALL_DONE = 'all_done'
    ONE_SUCCESS = 'one_success'
    ONE_FAILED = 'one_failed'
    DUMMY = 'dummy'

Here is the idea to solve your problem:

    from airflow.operators.ssh_execute_operator import SSHExecuteOperator
    from airflow.utils.trigger_rule import TriggerRule
    from airflow.contrib.hooks import SSHHook

    sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>)

    task_1 = SSHExecuteOperator(
            task_id='task_1',
            bash_command=<YOUR COMMAND>,
            ssh_hook=sshHook,
            dag=dag)

    task_2 = SSHExecuteOperator(
            task_id='conditional_task',
            bash_command=<YOUR COMMAND>,
            ssh_hook=sshHook,
            dag=dag)

    task_2a = SSHExecuteOperator(
            task_id='task_2a',
            bash_command=<YOUR COMMAND>,
            trigger_rule=TriggerRule.ALL_SUCCESS,
            ssh_hook=sshHook,
            dag=dag)

    task_2b = SSHExecuteOperator(
            task_id='task_2b',
            bash_command=<YOUR COMMAND>,
            trigger_rule=TriggerRule.ALL_FAILED,
            ssh_hook=sshHook,
            dag=dag)

    task_3 = SSHExecuteOperator(
            task_id='task_3',
            bash_command=<YOUR COMMAND>,
            trigger_rule=TriggerRule.ONE_SUCCESS,
            ssh_hook=sshHook,
            dag=dag)


    task_2.set_upstream(task_1)
    task_2a.set_upstream(task_2)
    task_2b.set_upstream(task_2)
    task_3.set_upstream(task_2a)
    task_3.set_upstream(task_2b)

From: stackoverflow.com/q/43678408