Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.1
What happened and how to reproduce it?
Pools that include deferred tasks do not count tasks in the scheduled state. For the first scheduled state over a task lifetime, I imagine this is not problematic, but whenever there are subsequent occurrences of the scheduled state in a deferred task's lifetime, any pool tracking that task is at risk of undercounting how many slots are occupied. This reduces the efficacy of pools as a semaphore protecting some external resource, which was the motivation of #32709, the PR that enabled pools to track deferred tasks.
See the maintainer comment on a related issue: #29416 (comment).
Given that this is a matter of pool implementation, reproduction is trivial. Pools will undercount by definition: source.
What you think should happen instead?
Per the maintainer comment, and given that deferred tasks can transition into and out of the scheduled state arbitrarily many times, tasks in their second or greater scheduled state should count toward pool totals, provided the pool includes deferred tasks.
Operating System
No response
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.1
What happened and how to reproduce it?
Pools that include deferred tasks do not count tasks in the scheduled state. For the first scheduled state over a task lifetime, I imagine this is not problematic, but whenever there are subsequent occurrences of the scheduled state in a deferred task's lifetime, any pool tracking that task is at risk of undercounting how many slots are occupied. This reduces the efficacy of pools as a semaphore protecting some external resource, which was the motivation of #32709, the PR that enabled pools to track deferred tasks.
See the maintainer comment on a related issue: #29416 (comment).
Given that this is a matter of pool implementation, reproduction is trivial. Pools will undercount by definition: source.
What you think should happen instead?
Per the maintainer comment, and given that deferred tasks can transition into and out of the scheduled state arbitrarily many times, tasks in their second or greater scheduled state should count toward pool totals, provided the pool includes deferred tasks.
Operating System
No response
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct