Skip to content

feat: add EdgeKV driver with Edge Function proxy for EdgeOne Pages - #41

Merged
PIKACHUIM merged 18 commits into
mainfrom
feat/edgeone-kv-proxy
Sep 13, 2026
Merged

PIKACHUIM merged 18 commits into
mainfrom
feat/edgeone-kv-proxy

Conversation

@PIKACHUIM

@PIKACHUIM PIKACHUIM commented Sep 11, 2026

Copy link
Copy Markdown
Member

EdgeOne KV 代理 + 存储驱动加固 + 配置精简

PR 分支feat/edgeone-kv-proxymain
对比基准04b25d8(merge-base)
提交数:11 | 规模:65 文件,+3463 / −1600
类型feat + fix + refactor⚠️ 含破坏性配置变更)


一、摘要

一句话概括

让 OpenList-TSWorker 在 EdgeOne Node 云函数上可用(此前完全不可用),同时修正存储驱动的若干正确性缺陷、精简并统一配置变量。

修了什么

类别 内容
新能力 EdgeOne Node 云函数的 KV HTTP 代理(issue #46 场景)
内容修复 8 项:内存回退、检测顺序、RESP 陷阱、KV 键名、MySQL 方言、sshkeys 数据丢失、密钥就绪竞态、CF 绑定声明
配置治理 环境变量从 19 个精简/重命名,密钥三合一为 JWT_SECRET
部署修复 一键部署不再因绑定声明失败;Secret 通过模板声明并可持久保留
可观测性 新增 /public/env_check 环境自检 + 前端三步初始化向导

不修什么(刻意保留)

  • map 格式(宽表)保留 —— 删的是「宽表塞进 SQL 表」的旧实现,不是宽表本身
  • memory 驱动保留 —— 仅本地/容器允许,serverless 一律拒绝
  • JWT_SECRET 的加密共存 —— 未拆分为独立加密密钥(见「设计取舍」)

二、背景

2.1 平台架构边界(本 PR 的起点)

EdgeOne 有两种运行时,KV 绑定只注入其中一种

运行时 代码位置 KV binding 实测错误特征
边缘函数 Edge Functions ✅ 注入
Node 云函数 cloud-functions/[[default]].js 不注入 栈含 node:internal/process/task_queues

因此在 EdgeOne Node 云函数上:

  • context.env.KV 永远拿不到 → 报 KV binding not found
  • 这不是配置问题,是平台架构限制

2.2 相关 Issue

Issue 问题
#46 自动检测落到 memory → serverless 上数据随实例消亡
#34 一键部署无法自动绑定存储;二次部署报错

2.3 部署机制限制

Cloudflare 官方文档确认:

  • Deploy 按钮只读取仓库内的配置文件预配资源,不支持 URL 传参
  • 要求「为每个 binding 提供默认值(资源名 + 资源 ID)」
  • 由此产生三个互斥需求:
    1. 一键部署要自动绑定资源
    2. 不要强制创建用不到的资源
    3. 二次部署不能因已有资源报错

根因(#34 用户反馈「二次部署找不到已创建的 KV」):首次部署时 Cloudflare 把生成的 id 回填到了**「新建的 Git 仓库」**,而非源仓库。二次从源仓库部署时 id 仍为空 → 冲突。


三、修改方案

3.1 KV HTTP 代理(新能力)

方案:Node 云函数经 HTTP 调用 Edge Function 代理来访问 KV。

Node 云函数 (DB_DRIVER=kv)
  └─ createProxyBinding(origin)  →  HTTP  →  Edge Function
                                              ├─ kv-get
                                              ├─ kv-put
                                              ├─ kv-delete
                                              └─ kv-list
                                              (_kv-proxy.js 共享逻辑)

触发条件:显式 DB_DRIVER=kv 且无原生 binding 且存在请求来源。

三层鉴权_kv-proxy.js authorize()):

通道 方式
内部调用(Node 云函数) X-Internal-Call 常量时间比对密钥前 16 位
用户调用 真实校验 HS256 JWT(签名 + exp + nbf + admin 角色)
其他 401

强制 HS256,拒绝 alg=none 降级攻击。

密钥不缓存:KV 密钥可能被 Node 侧随时写入/轮换,任何模块级缓存(哪怕带 TTL)都会让 Edge 实例持有旧值导致鉴权失败 → 每次直读 env.JWT_SECRET → 回退 KV

前置条件checkProxyConfig() 在代理模式强制要求 JWT_SECRET(≥16 字符),缺失直接 503,避免「静默不可用」。

3.2 Serverless 禁止内存存储

双层防护

行为
autoDetectDriver() serverless 探测不到任何驱动 → 抛错(本地/容器才回退 memory)
resolveDriver() serverless 永不接受 memory 驱动

isServerlessRuntime() 仅依据运行时特征(不依赖用户配置):

平台 特征
通用 注入型请求上下文
EdgeOne EDGEONE_BLOBEdgeOne 全局、TENCENTCLOUD_SCF_FUNCTIONNAME
Cloudflare WebSocketPaircaches.default
阿里云 ESA ESA_BLOBESA 全局

3.3 检测顺序修正

顺序
修改前 blob → cfkv → kv → d1domysql
修改后 mysql → d1 → kv → cfkv → blob → do

设计要点

  • mysql 仅在显式配置 MYSQL_URL / MYSQL_HOST 时参与探测 —— 网络连接,无条件尝试会给每个请求增加 TCP 建连开销
  • kv 优先于 cfkv —— 本地 binding 比 REST API 更直接

修复的真实缺陷:CF Worker 仅绑 DO 时报「no storage available」拒绝初始化(doDriver 不在候选列表),尽管 DO 完全可用。

3.4 绑定探测加固

问题:EdgeOne Node 云函数注入名为 KV 的绑定,但它走 RESP/Redis 协议(TCP socket),非 Web KV API。原代码接受任何真值 → 调用 put/getcannot find the collection by name

方案:探测必须校验接口形态

驱动 要求
kv / blob get() 且(put()set()
d1 prepare()
do idFromName()

同时修复(env && env[key]) || g[key] 形式在 env[key] 存在时短路globalThis 上可用的绑定永远探测不到。现改为 env 与 globalThis 独立检查

所有属性访问包 try/catch(抛异常的 getter 不得让探测崩溃)。

3.5 KV-safe 键名

问题:EdgeOne KV 限制键名只能含 [A-Za-z0-9_]DB_FORMAT=key 原先用 : 分隔并内嵌 UUID(含 -)→ 每次写入被拒。

方案:新增 keycodec.ts

  • 分隔符 _(本身合法,无需转义)
  • 其他字符按 UTF-8 逐字节转义为 xHH
  • 按 Unicode 码点遍历,避免拆散代理对(emoji)
  • 空字符串映射 0(避免 users_ 与表前缀混淆)

无需反向解析:主键值取自记录 JSON,键名只作存储地址。

3.6 列式表重构 + 删除宽表死代码

背景:旧版 store/d1.tsstore/mysql.tsstore/kv.ts 是「SQL 表壳 + data JSON 列」的宽表实现 —— Go 后端读不了,与「共享物理库」目标冲突。

方案:删除这三个旧实现,统一走新的 driver/ + format/ 架构;schema.ts 采用列式表(每字段独立成列),表名对齐 Go 的 GORM:

逻辑表 SQL 表名
settings x_setting_items
storages x_storages
users x_users
shares x_sharing_dbs
metas x_metas
plugins x_plugins(TS 独有)

3.7 sshkeys 表数据丢失修复

问题:TS 侧 SSH 公钥存在 user.ssh_keys,从不写顶层 db.sshkeys;Go 后端存在独立的 x_ssh_public_keys。若 sshkeys 参与往返,sqlFormat.save()DELETE FROM x_ssh_public_keys 再写空数组 → 清空 Go 的公钥

方案:拆为两个列表

列表 数量 用途
TABLE_NAMES 6 参与 load/save 往返
DDL_TABLE_NAMES 7 仅建表(含 sshkeys),保证与 Go 共享库结构一致

3.8 MySQL 方言修复

问题format/sql.ts 用 SQLite 专属的 INSERT OR REPLACE,MySQL 上必报语法错误。

方案:新增 upsertSql()driver.name 分支

方言 语句
SQLite(d1 / do) INSERT OR REPLACE
MySQL ON DUPLICATE KEY UPDATE

3.9 加密密钥就绪竞态(修复「初始化成功但密码认证失败」)

故障链路(KV 最终一致性):

setup(实例 I₁):生成密钥 A → 写入 KV(尚未传播)→ 用 A 加密密码落盘 ✅
紧接着登录(可能落在实例 I₂):
  缓存为空 → 重新读 KV → 读到 null(A 还没传播)
  → getEncryptionKey 返回 null
  → unsealDb 跳过解密
  → password 保持 "enc:v1:..." 密文
  → verifyUserPassword 判定非 64 位 hex → false
  → ❌「密码不正确」
等待数十秒后:KV 传播完成 → ✅ 又能登录(「过一会就好了」)

方案ensureEncryptionSecret 写入后主动回读确认,读不到则指数退避重试(最多 6 次,总等待约 1.9s);读到并发的不同密钥时采纳它,保持加解密对称。

配套

  • getEncryptionKey() 只读不生成(瞬时失败不轮换密钥)
  • 进程内单飞(inflight 合并)消除同实例并发重复生成
  • 新增 isEncryptionReady()绕过缓存直查真实来源),经 /public/init_status 暴露 ready

3.10 一键部署与 Secret 保留

方案wrangler.tomlwrangler.jsonc刻意不声明任何存储绑定

{
  "vars": {
    "ENVIRONMENT": "production",
    "DB_FORMAT": "map",
    "DB_DRIVER": "auto"
  }
  // ▶ KV / D1 / DO / Blob+MySQL 全部以注释形式提供模板
}

选用 .jsonc 原因:Wrangler v3.91.0+ 支持、可写注释、官方文档示例均用 jsonc。

Secret 声明机制(官方确认):

环节 机制
① 声明 Deploy 按钮读取 .dev.vars.example / .env.example,识别需用户填写的 Secret
② 输入 部署页生成输入项,用户填真实值 → 保存为该 Worker 的 Secret
③ 保留 Secrets 不被 wrangler deploy 覆盖或删除,仅 wrangler secret delete 才移除

vars 的分工

类型 声明位置 部署行为
JWT_SECRET 等敏感项 .dev.vars.example Secret → 设置后永久保留
DB_FORMAT / DB_DRIVER wrangler.jsoncvars 每次覆盖 → 正是默认值语义 ✅

3.11 环境自检接口

新增 免鉴权GET /public/env_check(初始化页未登录就需要),绝不回显密钥或 DSN 原文

就绪判定规则

字段 规则
storage.available hasDriver && !isMemory && !hasConfigError && driverHealthy
jwt.ready isEncryptionReady()(绕过缓存直查真实来源)
ready 二者皆真

两个易漏边界(已修复):memory 不算可用;驱动「配置齐全但不可达」(如 KV 代理 401)报 STORAGE_UNHEALTHY

3.12 前端三步初始化向导

独立分支 feat/init-storage-progress

步骤 内容
① Environment 自检面板(格式 / 驱动 / 运行时 / 存储 / JWT)+ 问题清单 + 文档链接;未就绪则禁用「继续」
② Account 用户名 / 密码 / 确认密码 / 站点标题
③ Finishing 进度条 + Spinner;完成后展示管理员用户名站点地址,提供「打开首页」「前往登录」

严格完成判定:仅当 ready=true 才进入完成态;超时显示「仍在同步」+ 重试,不谎报成功

Go 兼容:面板仅对 TS Worker 渲染;Go 侧跳过就绪等待(强一致存储,且无 ready 字段)。


四、改动文件

4.1 新增(9)

文件 作用
functions/_kv-proxy.js Edge Function 代理共享逻辑(KV 解析、密钥、JWT 校验)
functions/kv-get/index.js KV 读取代理
functions/kv-put/index.js KV 写入代理
functions/kv-delete/index.js KV 删除代理
functions/kv-list/index.js KV 列举代理
src/backend/internal/model/store/keycodec.ts KV-safe 键名编解码
wrangler.jsonc CF 配置(替代 toml,不声明绑定)
.dev.vars.example 部署按钮的 Secret 声明模板
scripts/env-check.mjs 本地环境自检诊断脚本
scripts/_regress.mjs 回归测试套件(50+ 断言)

4.2 删除(5)

文件 原因
wrangler.toml wrangler.jsonc 替代
src/backend/internal/model/store/d1.ts 宽表旧实现,零引用
src/backend/internal/model/store/mysql.ts 宽表旧实现,零引用
src/backend/internal/model/store/kv.ts 宽表旧实现,零引用
scripts/d1-schema.sql schema.ts 已脱节的过时脚本

4.3 主要修改(按模块)

存储核心

文件 改动
store/backend.ts 检测顺序、memory 防护、运行时识别、错误信息
store/schema.ts 列式表、TABLE_NAMES / DDL_TABLE_NAMES 拆分、固定 x_ 前缀
store/types.ts EnvContext;删 StoreDriverTABLE_EXTRA_COLUMNS
store/json.ts 绑定探测加固、密钥读写、代理支持、删 DB_JSON_BACKEND
store/format/sql.ts 方言兼容 UPSERT
store/format/key.ts KV-safe 键名
store/driver/kv.ts 代理绑定、双签名 get()、绑定名收敛
store/driver/{blob,d1,do,mysql,cfkv}.ts 绑定形态校验、绑定名收敛

服务层

文件 改动
internal/model/db.ts 密钥统一、写后回读校验、isEncryptionReady()、删 DATABASE_JSON
server/public.ts 新增 /public/env_checkinit_status.ready
server/middlewares.ts CRON_SECRET 合并入 JWT_SECRET
server/router.ts ALLOWED_ORIGINSALLOW_URLS
server/auth.ts ADMIN_PASSWORDADMIN_PASS
server/{assets,fs,seed,task}.ts 变量重命名
index.ts 503 + STORAGE_CONFIG_ERROR

测试(5)

store.test.tsstorage.test.tshealthz.test.tscrypto_security.test.tsdefault_credentials.test.ts

脚本/配置/文档(14 + 12 readmes)

scripts/{deploy,test-deploy,security-check}.jspackage.json.env.example.gitignoreserverless.ymlesa-entry.tsmiddleware.jscloud-functions/[[default]].jsREADME.md + 12 个 readmes/*.md


五、配置变化

5.1 删除的变量(5)

变量 替代方案
ENCRYPTION_SECRET JWT_SECRET
CRON_SECRET JWT_SECRET
DATABASE_JSON 已移除(仅测试用途,改用公开 API)
TABLE_PREFIX 表前缀固定 x_
DB_JSON_BACKEND DB_DRIVER
DATABASE_URL MYSQL_URL

5.2 重命名的变量(11)

旧名 新名
ADMIN_PASSWORD ADMIN_PASS
ALLOWED_ORIGINS ALLOW_URLS
MAX_UPLOAD_SIZE MAX_UPLOAD
MAX_PART_SIZE MAX_UPPART
CDN_URL ASSET_URLS
SEED_SOURCE_ALLOWED_HOSTS ALLOW_SEED
MYSQL_PASSWORD MYSQL_PASS
MYSQL_DATABASE MYSQL_NAME
CF_ACCOUNT_ID CF_ACCOUNT
CF_KV_NAMESPACE_ID CF_KV_UUID
CF_API_TOKEN CF_API_KEY

5.3 绑定名收敛

原先识别 现在
EDGEONE_KV / EO_KV / KV / CF_KV / DATABASE_KV / EDGEONE_KV_NAME / KV_NAMESPACE / KV_NAME KV
DO_BINDING / DO DO
SCF_FUNCTIONNAME / TENCENTCLOUD_SCF_FUNCTIONNAME TENCENTCLOUD_SCF_FUNCTIONNAME

5.4 最终变量清单

变量 默认 必需性
JWT_SECRET 推荐(签名 + 加密 + 定时鉴权)
DB_DRIVER auto 可选
DB_FORMAT map 可选
ADMIN_PASS 可选
ALLOW_URLS 可选
MAX_UPLOAD 26214400 可选
MAX_UPPART 16777216 可选
ASSET_URLS 可选
ALLOW_SEED 可选
MYSQL_URL / MYSQL_HOST / MYSQL_PORT / MYSQL_USER / MYSQL_PASS / MYSQL_NAME mysql 驱动时
CF_ACCOUNT / CF_KV_UUID / CF_API_KEY cfkv 驱动时

5.5 存储格式与驱动对照

格式 说明 键名/表名
map(默认) 整库序列化为单个 JSON openlist_config
key 一条记录一个键 users_1
sql 每字段一列 x_users

驱动检测顺序mysql → d1 → kv → cfkv → blob → do


六、迁移方法

⚠️ 本节含破坏性变更,请勿跳过。

6.1 【必做】密钥变量合并

若你设置了 ENCRYPTION_SECRET

# 旧配置
ENCRYPTION_SECRET=abc123...
JWT_SECRET=xyz789...

# 迁移后(关键:让 JWT_SECRET 保持为原 ENCRYPTION_SECRET 的值)
JWT_SECRET=abc123...
# 删除 ENCRYPTION_SECRET

为什么? 字段加密密钥来源已统一为 JWT_SECRET。你已有的网盘凭据、OTP 密钥等是用 ENCRYPTION_SECRET 加密的。若直接把 JWT_SECRET 保留为原值,这些数据将无法解密

安全兜底:若解密失败,unsealValue保留原始密文不丢数据(敏感字段显示为 enc:v1:...)。把 JWT_SECRET 改回原 ENCRYPTION_SECRET 的值即可恢复。

你的情况 需要做什么
只设了 JWT_SECRET 无需操作
设了 ENCRYPTION_SECRET ⚠️JWT_SECRET 改为原 ENCRYPTION_SECRET 的值,删除 ENCRYPTION_SECRET
两个都没设 ✅ 系统自动生成并持久化,无需操作

6.2 【必做】删除废弃变量

变量 处理
CRON_SECRET 删除;调度请求改用 ?cron_secret=<JWT_SECRET>
DATABASE_JSON 删除
TABLE_PREFIX 删除;表前缀固定 x_(非 x_ 前缀需迁移表名,见 6.5)
DB_JSON_BACKEND 改用 DB_DRIVER
DATABASE_URL 改用 MYSQL_URL

6.3 【必做】变量重命名

见 §5.2 的 11 项对照表,逐项替换。

6.4 【按需】KV / DO 绑定名

绑定名固定为 KVDO。若用了自定义绑定名,请改为 KV / DO

6.5 【按需】表前缀迁移(仅 DB_FORMAT=sql 且自定义过前缀)

RENAME TABLE custom_setting_items TO x_setting_items;
RENAME TABLE custom_sharing_dbs   TO x_sharing_dbs;
RENAME TABLE custom_storages      TO x_storages;
RENAME TABLE custom_users         TO x_users;
RENAME TABLE custom_metas         TO x_metas;
RENAME TABLE custom_plugins       TO x_plugins;

若从未设置过 TABLE_PREFIX(一直用默认 x_),无需操作

6.6 【按需】EdgeOne Node 云函数 + KV 用户

  1. 必须配置 JWT_SECRET(≥16 字符)—— 代理鉴权依赖它
  2. KV namespace 必须绑定到「边缘函数」(不是 Node 云函数)
  3. 若无法满足 → 建议改用 DB_DRIVER=blob(EdgeOne Blob 零配置)

平台架构边界:EdgeOne KV 绑定不会注入到 Node 云函数(SCF)。错误栈含 node:internal/process/task_queues 即可判定。

6.7 【按需】从 wrangler.toml 迁到 wrangler.jsonc

  • 本 PR 已用 wrangler.jsonc 替换 wrangler.toml,并不再声明任何存储绑定
  • 若你原来在 wrangler.toml 中声明了 [[kv_namespaces]] 等:
    1. 迁移配置到 wrangler.jsonc(或继续用 wrangler.toml,二者不冲突)
    2. 推荐:改为在 Cloudflare 控制台手动绑定(避免二次部署冲突)
  • Secrets 无需迁移JWT_SECRET 等已配置的 Secret 会保留

6.8 迁移速查表

你的旧配置 需要做什么
只设 JWT_SECRET ✅ 无需操作
设了 ENCRYPTION_SECRET ⚠️ JWT_SECRET 改为原值,删除 ENCRYPTION_SECRET
设了 CRON_SECRET ⚠️ 删除;调度改用 ?cron_secret=<JWT_SECRET>
设了 ADMIN_PASSWORD ⚠️ 重命名为 ADMIN_PASS
设了 ALLOWED_ORIGINS ⚠️ 重命名为 ALLOW_URLS
设了 TABLE_PREFIX ⚠️ 删除;非 x_ 需迁移表名
设了 DATABASE_JSON ⚠️ 删除
设了 DB_JSON_BACKEND ⚠️ 改用 DB_DRIVER
MYSQL_DSN/SQL_DSN/DATABASE_URL ⚠️ 改用 MYSQL_URL
自定义 KV/DO 绑定名 ⚠️ 改为 KV / DO
wrangler.toml 声明绑定 ⚠️ 迁到 jsonc 或改控制台手动绑定
EdgeOne Node 云函数 + KV ⚠️ 必须配 JWT_SECRET,KV 绑到边缘函数
其他变量名 ⚠️ 见 §5.2

6.9 【验证】首次部署清单

Secret 保留机制依赖 Cloudflare Deploy 按钮行为,需一次真实部署确认:

# 步骤 预期结果
1 点击 Deploy 按钮 部署页显示来自 .dev.vars.example7 个输入项
2 JWT_SECRET,执行部署 成功;无 KV/D1/DO 被自动创建
3 查看 Worker → Settings → Variables JWT_SECRET 显示为 Secret 类型(值不可见)
4 再次部署(不改配置) 成功,JWT_SECRET 仍存在且未被清空
5 打开前端初始化页 显示「存储 Not ready」,提示绑定 KV
6 控制台绑定 KV 后刷新 变为「Ready」,可进入第 2 步

第 4 步是核心验证点。


七、设计取舍

决策 理由
不拆分加密密钥 当前 JWT_SECRET 兼作加密密钥。拆分需数据迁移,且用户须管理两个密钥;先用合并方案,风险见 §6.1
保留 map 格式 宽表在 KV/Blob 上简单可靠;删的是「宽表塞进 SQL 表」的旧实现
保留 memory 驱动 本地开发必需;serverless 由双层防护拒绝
mysql 需显式配置才探测 避免边缘环境每次 auto 都尝试 TCP 建连
kv 优先于 cfkv 本地 binding 比 REST API 更直接
不声明 CF 绑定 牺牲「自动绑定」,换取「任何情况下部署都成功」
wrangler.jsonc 而非 .json 可写注释,便于提供「取消注释即启用」的绑定模板

八、验证结果

tsc --noEmit              → 0 错误
store.test.ts             → 9/9
storage.test.ts           → 4/4
healthz.test.ts           → 6/6(含 2 个 env_check 用例)
scripts/_regress.mjs      → ALL PASS(50+ 断言)
scripts/test-deploy.js    → 全部通过
wrangler deploy --dry-run → 仅绑定 ASSETS + 3 个 vars,无存储绑定
prettier --check          → 全部合规

回归套件覆盖重点

分组 内容
auto 驱动检测 CF 仅绑 DO → do;仅绑 D1 → d1;KV+D1 → d1;KV+CF_REST → kv;无 MYSQL 配置不选 mysql;serverless 无存储抛错
绑定形态 RESP 客户端 / 字符串 / 空对象不算 binding;Web KV 算
密钥就绪 延迟 KV 下的写后回读、幂等复用、isEncryptionReady
格式往返 map / key / sql 三格式 roundtrip;MySQL 方言 UPSERT
密钥安全 env_check 响应不含密钥原文

已知预存问题default_credentials.test.ts 有 2 个用例失败,经 git stash 验证在本分支改动前即已失败getDb 按 env 对象缓存导致的测试隔离问题),与本 PR 无关。


九、提交清单

# 提交 说明
1 17ec8cf feat: KV proxy, persistent secrets, no memory fallback, KV-safe keys
2 13094df fix(kv): init table name issue
3 3365fc7 refactor(config): unify secrets into JWT_SECRET and collapse binding names
4 39ad759 refactor(config): normalize env var names to ≤10 chars
5 d884941 chore: untrack local docs and refine wrangler.toml
6 5750563 fix(db): verify encryption key is readable before setup returns
7 cbe59e0 feat(public): add env readiness self-check for the setup page
8 b5af83c fix(store): correct auto-detection order and stop declaring CF bindings
9 8774c7e chore(config): declare secrets via .dev.vars.example and extend wrangler.jsonc
10 d35d7e0 docs(config): list every env var in wrangler.jsonc vars, commented by default
11 bdb378d fix(store): probe kv before cfkv in auto detection

十、关联 PR

仓库 分支 内容
OpenList-Worker feat/edgeone-kv-proxy 本文档(后端)
OpenList-Frontend feat/init-storage-progress 三步初始化向导 + 环境自检面板(4 提交)
OpenList-Docs docs/worker-env-rename 文档同步(2 提交)

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 11, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
openlist-tsworkers 29cf3a3 Sep 13 2026, 02:43 AM

@PIKACHUIM
PIKACHUIM force-pushed the feat/edgeone-kv-proxy branch 15 times, most recently from fbf1255 to 4784b61 Compare September 11, 2026 14:36
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 11, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
openlist-work 29cf3a3 Sep 13 2026, 02:42 AM

@PIKACHUIM
PIKACHUIM force-pushed the feat/edgeone-kv-proxy branch 3 times, most recently from dca5b32 to 007e8a5 Compare September 11, 2026 15:32
Binding resolution is now hardened across every driver. EdgeOne Node
Functions inject a binding named 'KV' that speaks RESP/Redis over a TCP
socket rather than the Web KV API, so calling put/get on it failed with
'cannot find the collection by name' (stack through processResponses /
Socket / TCP.onStreamRead). Any truthy value at that name was previously
accepted, so the RESP client was used as a KV binding and the proxy branch
was skipped. Resolution now requires the expected interface shape:

- kv.ts     get() and (put() or set())
- blob.ts   get() and (put() or set())
- d1.ts     prepare()
- do.ts     idFromName()

All probes also check env and globalThis independently. The previous form
'(env && env[key]) || g[key]' short-circuits when env[key] exists, so a
usable globalThis binding was never reached when env held a trap value.

Property access is wrapped in try/catch because proxy objects and lazily
initialised SDKs can throw from getters; a throwing binding must not crash
probing.

Verified against trap shapes for every driver: RESP client, plain string
(binding name), number, boolean, null, undefined, empty object, get-only,
put-only, array, and throwing getters. Correct shapes are still detected.

Earlier fixes retained in this branch:

1. KV-safe keys. EdgeOne KV restricts keys to [A-Za-z0-9_], but DB_FORMAT=key
   used ':' as a separator and embedded raw UUIDs (with '-'), so every write
   was rejected with 'Key can only contain letters, numbers, and underscores'.
   store/keycodec.ts encodes keys: '_' as separator, '_' escaped to 'xx',
   other characters escaped per UTF-8 byte as xHH, code-point iteration so
   surrogate pairs (emoji) are not split, empty input mapped to '0'. Used by
   format/key.ts and store/kv.ts. No reverse parsing is needed because entity
   primary keys always come from the record JSON.

2. Serverless runtimes must not silently fall back to in-memory storage.
   autoDetectDriver() throws when nothing is available in a serverless
   environment; local and container deployments keep the fallback.

3. Missing or unusable storage returns an actionable error
   (NO_STORAGE_MESSAGE) listing what to configure. isServerlessRuntime()
   detects the runtime from behaviour only (WebSocketPair, caches.default,
   injected request context, EdgeOne bindings).

4. The error surfaces on the init page and the home page: global middleware
   rejects non-static, non-navigation requests with 503 and
   { code, message, data: { error: 'STORAGE_CONFIG_ERROR' } }; static assets
   and HTML shells pass through; /healthz reports 503 with
   mode='unavailable'; getStoreStatus() returns driver='none' plus
   configError instead of throwing.

5. Storage backend cache key included only ':', so under
   DB_DRIVER=auto a local env (memory) and a serverless env (error) could
   share an entry. The key now includes a runtime tag.

6. Legacy path supports the KV HTTP proxy (store/json.ts): proxy branch in
   getKvBinding tried before Blob when DB_DRIVER=kv and no usable native
   binding, createProxyBinding() over functions/kv-*, so all 16 existing call
   sites work unchanged. This fixes 'configured kv but kv not used':
   DB_DRIVER only affected store/backend.ts, while the JWT secret, revocation
   list and login-failure counters went through store/json.ts.

7. Blob probe no longer caches failure permanently (_blobChecked was set
   before the probe, so one early failure disabled storage for the instance
   lifetime and produced the contradictory log pair 'No persistence backend
   configured' followed by 'using EdgeOne Blob storage').

8. Secrets are generated only during setup and never overwritten
   (readPersistedSecret / writePersistedSecret / generateSecret /
   ensureEncryptionSecret, existence-gated). getEncryptionKey() reads env
   then storage and never generates, so a transient read failure cannot
   rotate the key and make existing encrypted data undecryptable.

9. The KV proxy verifies callers: HS256 signature, exp and nbf via Web
   Crypto, admin role required, constant-time comparison of X-Internal-Call
   against the first 16 chars of the secret.

Local tests: binding shape hardening 68/68, end-to-end regression 32/32 (KV
proxy with strict key validation plus a RESP trap, key legality and
round-trip, secret lifecycle, config enforcement, serverless guard, error
surfacing), build artifact audit 19/19, healthz 4/4. The 6 failures in the
full suite pre-date this branch.
@PIKACHUIM
PIKACHUIM force-pushed the feat/edgeone-kv-proxy branch from 007e8a5 to 17ec8cf Compare September 11, 2026 15:41
…names

- Merge ENCRYPTION_SECRET / CRON_SECRET into JWT_SECRET (sign + encrypt + cron)

- Rename ADMIN_PASSWORD -> ADMIN_PASS, ALLOWED_ORIGINS -> ALLOW_URLS

- Remove DATABASE_JSON, TABLE_PREFIX (fixed to x_), DB_JSON_BACKEND

- Collapse KV/DO binding names to fixed KV / DO; drop SCF_FUNCTIONNAME alias

- Remove Edge KV proxy secret cache entirely (always read fresh)

- Fix MySQL upsert dialect (ON DUPLICATE KEY UPDATE), sshkeys round-trip guard

- Sync README/readmes/docs/wrangler.toml/serverless.yml; add optional vars
- Rename >10-char vars: MAX_UPLOAD_SIZE->MAX_UPLOAD, MAX_PART_SIZE->MAX_UPPART,

  CDN_URL->ASSET_URLS, SEED_SOURCE_ALLOWED_HOSTS->ALLOW_SEED,

  MYSQL_PASSWORD->MYSQL_PASS, MYSQL_DATABASE->MYSQL_NAME,

  CF_ACCOUNT_ID->CF_ACCOUNT, CF_KV_NAMESPACE_ID->CF_KV_UUID, CF_API_TOKEN->CF_API_KEY

- Remove DATABASE_URL alias; keep MYSQL_URL

- Rewrite .env.example to match real variables

- Beautify wrangler.toml: uniform separators, concise comments, add kv_namespaces id

- Sync README/readmes/docs/PR migration guide
- Remove docs/storage-architecture.md and docs/PR-config-unification.md from git (kept on disk, now gitignored)

- Align driver comments and simplify ASSET_URLS example in wrangler.toml
- ensureEncryptionSecret now re-reads the key after writing, retrying with

  exponential backoff until it is actually visible to other instances.

  This fixes 'setup succeeds but password auth fails' on EdgeOne/Cloudflare KV,

  caused by eventual-consistency write propagation delay.

- Adopt an existing different key if a concurrent setup wins the race.

- Add isEncryptionReady() and expose it via /public/init_status as 'ready'.

- Add regression coverage with a delayed KV.
Adds GET /public/env_check (unauthenticated, never echoes secrets) so the
initialization page can show, before any account exists:

- config: DB_FORMAT / DB_DRIVER as configured, plus the resolved values
- storage: availability, memory fallback flag, health, platform
- jwt: whether the signing/encryption key is readable from its real source
- ready: combined verdict (storage usable AND key ready)
- issues: actionable list with code / level / message / docUrl

Readiness treats in-memory storage as NOT ready on serverless (writes would
vanish) and flags a configured-but-unreachable driver (e.g. KV proxy 401)
as STORAGE_UNHEALTHY instead of silently reporting success.

Also adds scripts/env-check.mjs to dump the check locally, and covers the
endpoint with tests (unready env, healthy env, no secret leakage, and the
serverless-with-key-but-no-storage case).
flyfeel pushed a commit to flyfeel/openList-worker that referenced this pull request Sep 12, 2026
fix: 勾选记住登录后 token 应持久化到 localStorage
Issue #46 - auto detection could reject usable deployments:
- Detect in order mysql -> d1 -> kv -> blob -> do -> cfkv. `do` was missing
  from the candidate list, so a Cloudflare Worker bound only to Durable
  Objects reported "no storage available" and refused to initialize.
- Only probe mysql when MYSQL_URL / MYSQL_HOST is configured. MySQL needs a
  network connection; probing it unconditionally added a TCP attempt to every
  auto-detection on edge runtimes.
- In-memory fallback stays forbidden on Workers / EdgeOne / ESA, so data can
  never silently land in RAM.

Issue #34 - one-click deploy could not bind resources:
- Replace wrangler.toml with wrangler.jsonc, declaring NO storage bindings.
  Declaring them forced the deploy button to provision all of them, and a
  second deploy from the source repo conflicted with the already-created
  resources (the ids live in the generated repo, not here).
- Guide binding from the console instead, with per-option instructions in the
  file and in scripts/deploy.js.
- Add package.json `cloudflare.bindings` descriptions so the deploy form
  explains JWT_SECRET / DB_DRIVER / DB_FORMAT at the point of entry.
- Default vars: DB_FORMAT=map, DB_DRIVER=auto.

Also rewrite scripts/test-deploy.js (which still parsed wrangler.toml and
would crash) to test the wrangler output parsers, and cover the new detection
order in the regression suite.
…ler.jsonc

Secret retention:
- Add .dev.vars.example (names only, values empty). The Deploy to Cloudflare
  button reads it (or .env.example) to generate inputs for the required
  secrets; the values entered there are stored as Worker Secrets.
- Align .env.example with the same variable set.
- Worker Secrets survive redeploys - they are only removed by an explicit
  `wrangler secret delete` - so an empty template value never clobbers a
  configured secret. In contrast, `vars` are overwritten on every deploy,
  which is exactly what we want for DB_FORMAT / DB_DRIVER defaults.
- Ignore real .dev.vars files while keeping the *.example templates tracked.

wrangler.jsonc:
- Port every variable and comment that used to live in wrangler.toml, all
  optional ones commented out by default (MAX_UPLOAD, MAX_UPPART, ASSET_URLS,
  ALLOW_SEED), plus the KV / D1 / DO binding templates.
- Document the auto-detection order: mysql -> d1 -> kv -> blob -> do -> cfkv.

Verified with `wrangler deploy --dry-run` (wrangler 4.118.0): only ASSETS plus
the three vars are bound; no storage bindings are declared.
… default

Mirror the full variable set from the retired wrangler.toml so the file stays
the single reference for what can be configured:

- secrets: JWT_SECRET, ADMIN_PASS, ALLOW_URLS
- optional: MAX_UPLOAD, MAX_UPPART, ASSET_URLS, ALLOW_SEED
- KV REST API: CF_KV_UUID, CF_ACCOUNT, CF_API_KEY
- MySQL: MYSQL_URL, or MYSQL_HOST / MYSQL_PORT / MYSQL_USER / MYSQL_PASS /
  MYSQL_NAME

All are commented out except ENVIRONMENT / DB_FORMAT / DB_DRIVER, so they
never override anything until the operator opts in. The same names are
declared (values empty) in .dev.vars.example / .env.example for the deploy
button. Verified with `wrangler deploy --dry-run`: only ASSETS plus the three
active vars are bound.
Both are KV-semantic drivers. A local KV binding is more direct and faster
than the Cloudflare REST API, so it should win when both are available.

New order: mysql -> d1 -> kv -> cfkv -> blob -> do

Updates the inline comments, wrangler.jsonc, README and storage-architecture
docs, and adds a regression assertion that kv beats cfkv when both are
present. Also drops the retired wrangler.toml (replaced by wrangler.jsonc)
and rebuilds the EdgeOne bundle.
Addresses findings from the code review of this branch.

P1 (major) - internal call used a truncated secret:
- The X-Internal-Call header was compared against the first 16 characters of
  JWT_SECRET, cutting entropy from 256 to 64 bits. Passing that check also
  bypassed the admin-role enforcement entirely, so a guessed value granted
  unrestricted KV read/write/delete.
- Now the full secret is required on both sides (_kv-proxy.js, json.ts,
  driver/kv.ts).

P2 (major) - kv-list pagination could repeat pages:
- If the platform's cursor is not "the last key of this page", the loop kept
  re-fetching the same page until the 1000-iteration guard.
- Adds repeated-page detection (same first key as the previous page stops
  pagination) and de-duplicates the returned key set.

P3 (major) - proxy origin was trusted as-is:
- __requestOrigin derives from the request Host, so a forged Host could point
  the Node side at an attacker server, leaking the (now full) JWT_SECRET via
  the X-Internal-Call header and allowing forged KV responses.
- Adds sanitizeProxyOrigin(): only http/https, must parse with a hostname,
  and request-derived origins must be HTTPS unless localhost. Used by both
  json.ts and driver/kv.ts.

Minor:
- env_check now redacts error text (first line, credentials masked, truncated)
  since the endpoint is unauthenticated.
- driver/kv.ts get() distinguishes signature errors (TypeError) from real
  failures so network/auth errors are no longer swallowed as "key not found".
- format/sql.ts save() switches from "DELETE all + INSERT all" to
  "UPSERT each row + DELETE ... NOT IN (keys)", removing the empty-table
  window and avoiding clobbering concurrent writes from the Go backend.
- package.json gains test:store / test:regress / test:deploy / env:check.
- Documents the concurrency limits of map/key and the constraints of sharing
  the sql database with the Go backend.

Regression suite grows to 71 assertions, including the truncated-secret
rejection and origin-validation matrix.
The code-review report slipped into the previous commit because .gitignore only listed storage-architecture.md and PR-config-unification.md by name. Broaden the rule to docs/code-review-*.md so all local notes follow the same untracked policy; files stay on disk.
The env_check issues were too verbose to read in the setup UI: the JWT
one ran to four lines of explanation, and the storage one dumped
NO_STORAGE_MESSAGE (an eleven line, terminal oriented configuration
guide) straight into the page.

Reduce each to a single actionable sentence and let the docUrl link
carry the detail. The storage config error no longer echoes the raw
backend message, which also keeps internal DSNs and hostnames off this
unauthenticated endpoint.
@PIKACHUIM PIKACHUIM added bug Something isn't working enhancement New feature or request labels Sep 12, 2026
…e checks

Follow-up to the code review of this branch. Removes duplicated proxy
implementations, tightens variable naming and makes the storage readiness
checks share a single source of truth.

KV proxy de-duplication (was two independent implementations):
- store/json.ts::createProxyBinding re-implemented the exact same HTTP
  proxy protocol as store/driver/kv.ts (same /kv-get|kv-put|kv-delete|kv-list
  endpoints, same X-Internal-Call header). Both paths are live at runtime:
  business data goes through kvDriver, while the JWT secret, revoked-token
  blacklist and login-failure counters go through json.ts. Any drift between
  them would silently make the two subsystems disagree.
- createProxyBinding now delegates to kvDriver and only adapts the
  Driver string[] list() contract to the binding {name,key}[] shape.
- sanitizeProxyOrigin moves to a new store/proxy.ts so neither module has to
  import the other; json.ts still re-exports it for existing callers.

Variable naming:
- MYSQL_URL -> MYSQL_URLS (the code read MYSQL_URLS in hasMysqlConfig but
  mysql.ts read MYSQL_URL, so following the error message produced a
  deployment that reported "configured" yet could not connect).
- EDGE_KV_BASE_URL -> EO_KV_URLS, and it is now declared in both example
  templates plus wrangler.jsonc (it was previously code-only, so users hit
  the runtime error before ever seeing the variable name).

Remove dead branches and misleading names:
- isEdgeOneNodeEnv() was `getKvBinding(env) === null` and every call site was
  already guarded by `if (kv) return`, so it was always true. Deleted it and
  the two now-unreachable branches it guarded (the "KV binding not found"
  throw and the mode:"unknown" health result).
- isAvailable() and health() each issued their own __health__ probe with
  different verdicts on HTTP 401. Both now share probeProxy(); isAvailable
  stays lenient (401 still means the proxy is deployed, only the secret is
  wrong) while health stays strict (401 is not usable), preserving the
  existing behaviour that /env_check depends on.

Single source of truth for storage readiness:
- Dropped the configErrorCache WeakMap from getStoreConfigError. It was
  redundant: getStorageBackend already caches driver resolution per env
  fingerprint, and checkProxyConfig is a pure synchronous env read. It also
  never worked as documented, since a Worker gets a fresh env object per
  request, so the single-reference fast path always missed.
- isPersistentStorageAvailable and getStoreConfigError now share
  isPersistentStatus(), so /env_check, init_status and the 503 middleware
  can no longer reach contradictory conclusions.
- getStoreConfigError keeps reporting the actionable "JWT_SECRET is missing"
  hint first when DB_DRIVER=kv is explicit.

Other:
- keycodec: KEY_SEP is no longer exported (internal only); encodeKeyPart and
  decodeKeyPart stay exported because the regression suite uses them to
  assert round-trip encoding.
- db.ts: corrected the comment claiming signature and field encryption share
  one key. They share the JWT_SECRET env var when set, but fall back to two
  separate KV slots, so without an explicit secret they are different values.
- .gitignore: ignore .tmp-dryrun/ (dry-run output written by the deploy
  constraint test); the existing rule only covered .wrangler-dryrun/.

Verified: tsc --noEmit clean, test:regress 71/71, test:store 9/9,
test:deploy 4/4, healthz 6/6. The three test:server failures (F-11 password
reset, F-11 random password, CAS codec field names) are pre-existing and
reproduce identically on the parent commit.
Follow-up hardening after reviewing the previous refactor. Three findings,
all reproduced before fixing.

GetStoreConfigError could return null for an unavailable backend:
- The function decided availability via isPersistentStatus() but then looked
  up the reason in a chain (configError -> memory -> healthError) that could
  exhaust without a match. A driver reporting { available: false } without an
  error string fell through to the final `return null`, so the 503 middleware
  let the request proceed against a backend that was known to be unusable.
- Added a terminal fallback derived from the driver name, so every path that
  isPersistentStatus() rejects also produces an error string. Verified that
  getStoreConfigError and isPersistentStorageAvailable now agree in all
  probed cases (empty env, serverless without storage, explicit kv without a
  secret, explicit d1 without a binding).

HTTP 401 from the KV proxy surfaced as an unactionable error:
- isAvailable() treats 401 as "proxy deployed" so the kv driver gets selected
  during auto-detection, while health() treats it as unusable. The combination
  is intended (selectable vs. actually usable), but the resulting message was
  the raw "HTTP 401" even though the real cause is a JWT_SECRET mismatch with
  the Edge Functions serving the proxy.
- getStoreConfigError now also inspects the resolved driver, not just an
  explicit DB_DRIVER=kv. When the kv driver is selected in proxy mode and the
  probe comes back 401, it explains that the deployment's JWT_SECRET does not
  match the proxy's and suggests checking EO_KV_URLS, which is what an
  operator actually needs to fix.

kvDriver.get() could return undefined instead of null:
- The proxy branch trusted { value } from the Edge Function, but kv-get returns
  { error } with HTTP 500 on failure, leaving data.value undefined. The Driver
  contract is string | null, and callers that test `=== null` would have missed
  it. Now normalised explicitly.

Also documented why sanitizeProxyOrigin does not block private/metadata IPs:
string-based checks cannot stop DNS rebinding (which needs resolution the edge
runtime cannot reliably do), would not protect against an attacker who already
controls Host, and would break legitimate internal deployments that point
EO_KV_URLS at a private origin.

Verified: tsc --noEmit clean, test:regress 71/71, test:store 9/9, healthz 6/6.
jyxjjj
jyxjjj previously approved these changes Sep 13, 2026
@PIKACHUIM
PIKACHUIM merged commit 5cac7eb into main Sep 13, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EdgeOne Makers · 国际站部署OpenList 發現老是初始化狀態 Cloudflar 部署OpenList 發現多個問題 deploy脚本的2个问题

2 participants