feat(pd): improve request scheduling and overload recovery

docs/CN/source/tutorial/api_server_args.rst

@@ -92,26 +92,52 @@ PD 分离模式参数
     推理进度健康检查：当仍有在途请求，且整个 PD Master 连续 ``HEALTH_TIMEOUT`` 秒
     没有任何请求成功返回 token 时，接口将返回 HTTP 503。
 
-.. option:: --enable_pd_node_self_request_limit
-
-    在 Prefill/Decode 节点上启用本地请求限流。PD Master 当前不执行请求准入限流。
-    HTTP server 申请本地 ``shm_req`` 对象的超时时间由
-    ``LIGHTLLM_PD_NODE_SHM_REQ_ALLOC_TIMEOUT_SECONDS`` 控制（默认 20 秒）；请求进入 Router 后等待
-    进入推理系统的超时时间由 ``LIGHTLLM_PD_NODE_ROUTER_WAIT_TIMEOUT_SECONDS`` 控制（默认 20 秒）。
-    超时会导致 ``Server is busy``；其中已进入 Router 但仍未进入推理系统的请求会主动标记为 aborted，
-    由 PD Master 转换为 HTTP 429。未开启限流时请求会持续等待资源；PD 高优先级请求
-    （分段续跑请求，或预计输入 cache 命中率高于 0.8 且命中记录仍然新鲜的请求）由 PD Master 通过
-    ``pd_high_priority_request_time_out_seconds`` 下发一个统一的超时时间下限。P/D 节点分别取
-    该值与本地 ``shm_req``、Router 超时的较大值；该字段为 0 时不延长本地超时。PD Master 下发值由
-    ``LIGHTLLM_PD_HIGH_PRIORITY_REQUEST_TIMEOUT_SECONDS`` 控制，默认 60 秒。cache 命中记录允许提升优先级的
-    最大年龄由 ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MAX_AGE_SECONDS`` 控制，默认 16 秒。本地请求限流默认关闭。
+.. option:: --disable_pd_node_self_request_limit
+
+    P/D 节点资源等待限流默认启用，并由 PD Master 统一管理。该参数只在需要关闭此功能时设置，且只需添加到
+    PD Master 的启动参数中，不需要在 Prefill/Decode 节点上设置。默认情况下，PD Master 通过
+    ``pd_node_resource_wait_timeout_seconds`` 为所有请求下发统一的资源等待上限；P/D 节点只负责按下发值
+    控制本地 ``shm_req`` 申请和 Router 等待进入推理系统，不读取本地限流开关或超时配置。首段的等待上限由
+    PD Master 上的
+    ``LIGHTLLM_PD_NODE_RESOURCE_WAIT_TIMEOUT_SECONDS`` 控制，默认 10 秒；设置为 -1 表示永久等待。
+    ``segment_index > 0`` 的续跑分段使用独立的等待上限，该值由
+    ``LIGHTLLM_PD_NODE_CONTINUATION_RESOURCE_WAIT_TIMEOUT_SECONDS`` 控制，默认 60 秒，以提高已产生部分结果的
+    请求最终完成的成功率。
+    设置为非负数时，超时会导致 ``Server is busy``；
+    其中已进入 Router 但仍未进入推理系统的请求会主动标记为 aborted，由 PD Master 转换为 HTTP 429。
+    本功能启用时，PD Master 收到 ``Server is busy`` 会重新选择 P/D 节点并重试；最长探测周期由
+    ``LIGHTLLM_PD_NODE_BUSY_RETRY_TIMEOUT_SECONDS`` 控制，默认 120 秒。若请求已经向客户端输出 token，
+    则不再从头重试，以免产生重复内容。设置 ``--disable_pd_node_self_request_limit`` 后，PD Master 不再下发
+    有限的资源等待时间；P/D 节点永久等待，其他原因产生的 ``Server is busy`` 也会直接返回，不触发重试。
+    多机 TP 场景仅由 master 节点执行超时判断，slave 节点永久等待。cache 命中记录允许提升优先级的最大年龄由
+    ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MAX_AGE_SECONDS`` 控制，默认 36 秒。cache 命中提权还要求输入
+    token 数达到 ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MIN_PROMPT_TOKENS`` 配置的门槛（默认 4096），避免短请求仅因
+    cache 命中率高而提升优先级。
+
+    启动示例：
+
+    .. code-block:: bash
+
+        LIGHTLLM_PD_NODE_RESOURCE_WAIT_TIMEOUT_SECONDS=10 \
+            LIGHTLLM_PD_NODE_CONTINUATION_RESOURCE_WAIT_TIMEOUT_SECONDS=60 \
+            LIGHTLLM_PD_NODE_BUSY_RETRY_TIMEOUT_SECONDS=120 \
+            python -m lightllm.server.api_server --run_mode pd_master ...
 
 .. option:: --disable_pd_cache_high_priority
 
-    禁止 PD Master 将预计输入 cache 命中率高且命中记录仍然新鲜的首段请求提升为高优先级。
+    禁止 PD Master 将输入足够长、预计输入 cache 命中率高且命中记录仍然新鲜的首段请求提升为高优先级。
     该参数不影响 PD Decode 容量不足后的分段续跑请求；续跑请求仍保持高优先级。默认不启用，
     即默认允许新鲜高 cache 命中请求提升优先级。
 
+    建议只在 PD Master 上配置该参数。当单个 P 节点的 GPU cache、CPU cache 和 disk cache 总容量相对于
+    请求工作集较小时，高负载下后到的请求容易快速淘汰已有 cache，使原本可以命中 cache 的请求退化为
+    重新执行 Prefill，进而显著降低 Prefill 效率。此时建议保留默认的高优先级策略，让预计 cache 命中率高的
+    请求提前进入推理，尽量在 cache 被淘汰前完成复用。
+
+    该策略会改变排队顺序，因此普通请求（未达到 cache 命中率、cache 年龄或最小 prompt token 数门槛的请求）
+    的首字延迟可能升高。如果 P 节点 cache 容量充足、系统负载较低，或者业务更重视调度公平性和普通请求的
+    首字延迟，可以设置 ``--disable_pd_cache_high_priority`` 关闭该策略。
+
 .. option:: --config_server_host
 
     配置服务器模式下的主机地址

docs/EN/source/tutorial/api_server_args.rst

@@ -95,29 +95,60 @@ PD disaggregation Mode Parameters
     the endpoints return HTTP 503 if no request on the PD Master successfully returns a token for
     ``HEALTH_TIMEOUT`` consecutive seconds.
 
-.. option:: --enable_pd_node_self_request_limit
-
-    Enable local request limiting on Prefill/Decode nodes. PD Master does not currently perform request admission
-    limiting. The local ``shm_req`` allocation timeout is controlled by
-    ``LIGHTLLM_PD_NODE_SHM_REQ_ALLOC_TIMEOUT_SECONDS`` (20 seconds by default), while the timeout from Router entry
-    to inference entry is controlled by ``LIGHTLLM_PD_NODE_ROUTER_WAIT_TIMEOUT_SECONDS`` (20 seconds by default).
-    A timeout reports ``Server is busy``; a request that has entered the Router but not inference is proactively
-    marked aborted, and PD Master converts this to HTTP 429. Requests continue waiting when local admission is
-    disabled. For PD high-priority requests (segmented continuation requests, or requests whose estimated input
-    cache hit rate is above 0.8 and whose cache record is still fresh), PD Master supplies a shared timeout floor through
-    ``pd_high_priority_request_time_out_seconds``. Each P/D node uses the greater of this value and its local
-    ``shm_req`` or Router timeout; zero does not extend the local timeout. The value supplied by PD Master is controlled by
-    ``LIGHTLLM_PD_HIGH_PRIORITY_REQUEST_TIMEOUT_SECONDS`` and defaults to 60 seconds. The maximum cache-record age
-    eligible for promotion is controlled by ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MAX_AGE_SECONDS`` and defaults to
-    16 seconds. Local request limiting is disabled by default.
+.. option:: --disable_pd_node_self_request_limit
+
+    P/D-node resource wait limiting is enabled by default and managed centrally by PD Master. Set this option only
+    when disabling the feature, and only when starting PD Master; it is not needed on Prefill or Decode nodes.
+    By default, PD Master supplies
+    ``pd_node_resource_wait_timeout_seconds`` for every request. P/D nodes only enforce the received value for local
+    ``shm_req`` allocation and the wait from Router entry to inference entry; they do not read local limiting switches
+    or timeout settings. The first segment's timeout is
+    controlled on PD Master by ``LIGHTLLM_PD_NODE_RESOURCE_WAIT_TIMEOUT_SECONDS`` and defaults to 10 seconds; set it
+    to -1 to wait indefinitely. Continuation segments with ``segment_index > 0`` use a separate timeout controlled by
+    ``LIGHTLLM_PD_NODE_CONTINUATION_RESOURCE_WAIT_TIMEOUT_SECONDS`` and defaults to 60 seconds, improving the chance
+    that requests which have already produced partial results complete successfully. When set to a non-negative value,
+    a timeout reports ``Server is busy``; a request that has
+    entered the Router but not inference is proactively marked aborted, and PD Master converts this to HTTP 429.
+    While this feature is enabled, PD Master selects P/D nodes again and retries after receiving ``Server is busy``.
+    The maximum probing period is controlled by ``LIGHTLLM_PD_NODE_BUSY_RETRY_TIMEOUT_SECONDS`` and defaults to
+    120 seconds. Once response tokens have been streamed to the client, the request is not restarted because doing so
+    would duplicate output. With ``--disable_pd_node_self_request_limit``, PD Master no longer supplies a finite
+    resource wait timeout; all P/D nodes wait indefinitely, and a ``Server is busy`` raised for another reason is
+    returned immediately without retrying.
+    In multi-node TP deployments, only the master node evaluates the timeout; slave nodes wait indefinitely.
+    The maximum cache-record age eligible for promotion is controlled by
+    ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MAX_AGE_SECONDS`` and defaults to
+    36 seconds. Cache-hit promotion also requires at least the number of input tokens configured by
+    ``LIGHTLLM_PD_CACHE_HIGH_PRIORITY_MIN_PROMPT_TOKENS`` (4096 by default), so short requests do not gain priority
+    solely from a high cache-hit rate.
+
+    Startup example:
+
+    .. code-block:: bash
+
+        LIGHTLLM_PD_NODE_RESOURCE_WAIT_TIMEOUT_SECONDS=10 \
+            LIGHTLLM_PD_NODE_CONTINUATION_RESOURCE_WAIT_TIMEOUT_SECONDS=60 \
+            LIGHTLLM_PD_NODE_BUSY_RETRY_TIMEOUT_SECONDS=120 \
+            python -m lightllm.server.api_server --run_mode pd_master ...
 
 .. option:: --disable_pd_cache_high_priority
 
-    Disable PD Master from promoting first-segment requests whose estimated input cache hit rate is high and whose
-    cache record is still fresh. This does not affect segmented continuation requests after PD Decode capacity
-    exhaustion; continuation requests remain high priority. Disabled by default, so fresh high-cache-hit requests
+    Disable PD Master from promoting sufficiently long first-segment requests whose estimated input cache hit rate
+    is high and whose cache record is still fresh. This does not affect segmented continuation requests after PD
+    Decode capacity exhaustion; continuation requests remain high priority. Disabled by default, so eligible requests
     are promoted unless this option is set.
 
+    Configure this option only on PD Master. When a Prefill node's combined GPU, CPU, and disk cache capacity is small
+    relative to its request working set, later requests can quickly evict reusable cache entries under high load.
+    Requests that could otherwise hit the cache must then repeat Prefill computation, which can significantly reduce
+    Prefill efficiency. In this situation, keep the default high-priority policy enabled so requests with a high
+    estimated cache hit rate can run earlier and reuse their cache entries before eviction.
+
+    This policy changes queue ordering and may increase time to first token (TTFT) for ordinary requests that do not
+    meet the cache-hit-rate, cache-age, or minimum-prompt-token thresholds. Consider setting
+    ``--disable_pd_cache_high_priority`` when Prefill cache capacity is sufficient and cache churn is low, or when
+    scheduling fairness and ordinary-request TTFT are more important than preserving cache-hit efficiency.
+
 .. option:: --config_server_host
 
     Host address in configuration server mode

lightllm/server/api_cli.py

@@ -77,19 +77,26 @@ def add_cli_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
         ),
     )
     parser.add_argument(
-        "--enable_pd_node_self_request_limit",
+        "--disable_pd_node_self_request_limit",
         action="store_true",
         help=(
-            "Enable local request limiting on Prefill/Decode nodes by enforcing shm_req allocation and Router "
-            "scheduling wait timeouts. PD Master admission limiting is not currently enabled. Default: disabled."
+            "Disable PD Master-managed resource wait limiting and retries for requests rejected as server busy. "
+            "Configure this option only on PD Master. By default, PD Master sends timeout details to P/D nodes, "
+            "which only enforce the received values, and retries busy requests."
         ),
     )
     parser.add_argument(
         "--disable_pd_cache_high_priority",
         action="store_true",
         help=(
-            "Disable promoting first-segment PD Master requests with a fresh high cache-hit estimate. "
-            "Segmented continuation requests remain high priority. Default: disabled."
+            "Disable PD Master's high-priority scheduling for first-segment requests with a fresh, high "
+            "cache-hit estimate. Keep this policy enabled when a Prefill node's combined GPU, CPU, and disk "
+            "cache is small relative to its workload: under high load, ordinary scheduling can evict reusable "
+            "cache entries before they are consumed and significantly reduce Prefill efficiency. The policy "
+            "lets eligible cache-hit requests run earlier, but may increase TTFT for ordinary requests. "
+            "Consider disabling it only when scheduling fairness or ordinary-request latency is more important, "
+            "or when cache capacity is sufficient and cache churn is low. Segmented continuation requests remain "
+            "high priority. Configure this option only on PD Master. The policy is enabled by default."
         ),
     )
     parser.add_argument(

lightllm/server/core/objs/sampling_params.py

@@ -297,9 +297,9 @@ class SamplingParams(ctypes.Structure):
         # 由 PD Master 为分段续跑或预计 cache 命中率较高的请求设置，表示请求需
         # 以高优先级插入 Router 调度队列。
         ("pd_high_priority_request", ctypes.c_bool),
-        # PD 高优先级请求在开启本地限流的 P/D 节点上的等待时间下限。节点分别取
-        # 该值与本地超时的较大值，用于 shm_req 申请和 Router 等待进入推理系统。
-        ("pd_high_priority_request_time_out_seconds", ctypes.c_int),
+        # P/D 节点的资源等待超时，由 PD Master 下发。非负值用于控制 shm_req 申请和
+        # Router 等待进入推理系统的时限；负数表示永久等待。
+        ("pd_node_resource_wait_timeout_seconds", ctypes.c_int),
         ("suggested_dp_index", ctypes.c_int),  # suggest dp index, deepseekv2 dp mode, use to suggest used dp_index
         # in pd split mode, use to keep the id of pd master
         ("pd_master_node_id", NodeUUId),
@@ -345,7 +345,7 @@ def init(self, tokenizer, **kwargs):
         self.group_request_id = kwargs.get("group_request_id", -1)
         # 这两个字段是 PD Master 的内部调度信息，不能由外部请求参数开启或修改。
         self.pd_high_priority_request = False
-        self.pd_high_priority_request_time_out_seconds = 0
+        self.pd_node_resource_wait_timeout_seconds = -1
         self.suggested_dp_index = kwargs.get("suggested_dp_index", -1)
 
         self.skip_special_tokens = kwargs.get("skip_special_tokens", SKIP_SPECIAL_TOKENS)
@@ -513,7 +513,7 @@ def to_dict(self):
             "invalid_token_ids": self.invalid_token_ids.to_list(),
             "group_request_id": self.group_request_id,
             "pd_high_priority_request": self.pd_high_priority_request,
-            "pd_high_priority_request_time_out_seconds": self.pd_high_priority_request_time_out_seconds,
+            "pd_node_resource_wait_timeout_seconds": self.pd_node_resource_wait_timeout_seconds,
             "skip_special_tokens": self.skip_special_tokens,
             "add_special_tokens": self.add_special_tokens,
             "add_spaces_between_special_tokens": self.add_spaces_between_special_tokens,

lightllm/server/core/objs/start_args_type.py

@@ -23,7 +23,7 @@ class StartArgs:
     pd_master_ip: str = field(default="0.0.0.0")
     pd_master_port: int = field(default=1212)
     pd_master_mode: str = field(default="elastic")
-    enable_pd_node_self_request_limit: bool = field(default=False)
+    disable_pd_node_self_request_limit: bool = field(default=False)
     disable_pd_cache_high_priority: bool = field(default=False)
     pd_trans_mode: str = field(default="nccl", metadata={"choices": ["nccl", "nixl"]})
     config_server_host: str = field(default=None)