Compare commits

..

167 Commits

Author SHA1 Message Date
134
365bad8539 fix(canvas): distinguish provider balance from image rate limits 2026-10-01 07:45:39 +08:00
134
08eecb0190 docs: record the UUID validation regression that broke detail replacement 2026-09-27 19:19:20 +08:00
134
f75d99baf6 fix(agent): repair the UUID check that made every detail replacement request 400
新增的批次接口把一个 UUID 正则写成了 8-4-4-12(少一组 -{4}),标准 UUID 是
8-4-4-4-12。结果是任何真实项目编号和请求编号都过不了校验:查询接口恒返 400
「项目编号无效」,提交接口恒返 400「任务编号或项目编号无效」,详情替换完全无法使用。

- 校验抽成 isDetailReplacementUuid 并修正为规范的 8-4-4-4-12,路由不再自带正则
- 回归测试覆盖真实线上项目编号、浏览器生成的请求编号,以及导致故障的 8-4-4-12 形状
- 增加源码守卫,防止该路由再次出现自带的错误 UUID 正则
- 前端在项目尚未保存时给出明确提示,而不是把空编号丢给接口
2026-09-27 19:14:56 +08:00
134
db3a038b00 docs: record the detail replacement durable batch incident and rollout 2026-09-26 02:53:01 +08:00
134
0907aaff67 fix(agent): make detail replacement results survive page and server restarts
详情替换原来把整批生图挂在浏览器保持的对话流上,切换标签页、断网或连接中断后
结果就再也回不来,前端还会继续播放按时间推算的 96% 假进度。

现在改为服务器端后台批次:提交时先落任务,再按参考图逐张生成并把每张结果写进
项目,页面刷新、切页或服务重启后都能按任务状态恢复,不再依赖前端连接。

- 新增 /api/agents/detail-replacement/batches(提交/查询、幂等去重、重试上限)
- 前端只在服务端确认批次后显示处理中,卡片显示排队/生成/完成/失败真实状态
- 去掉假进度动画;没有产品输入图时提交前直接拦截
- 服务重启后自动恢复未完成批次,长批次不再被通用 30 分钟清理误判
- 单张失败只落到该卡片并继续后续图片,已完成的图片不重复生成和扣费
- 详情替换默认模型由持续 404 的 gpt-image-2.5-flare 改回 gpt-image-2.5
2026-09-26 02:39:11 +08:00
134
1c2acee547 Preserve uploaded media aspect ratios in video canvas 2026-09-24 14:36:22 +08:00
134
439af60c2a Update video canvas home and Aliyun analysis flow 2026-09-24 13:36:31 +08:00
134
e97296a9e8 Add share entry to the canvas top bar
The share popover (publish to site, copy link) and publish modal had
no reachable entry; add a share button next to the save status,
matching the reference product's top bar.
2026-09-19 17:09:39 +08:00
134
94b49b7fad Document shortcut system rollout and server-side verification 2026-09-19 14:57:29 +08:00
134
ea4f34bf89 Re-enable the video toolbar HD entry
The P40 upscale flow (popover, HD node creation, task submission) is
fully wired; the toolbar entry had been left disabled and mislabeled
as not connected.
2026-09-19 14:42:52 +08:00
134
1141ecc43a Bound separation input WAV duration to the probed length
The unbounded apad filter made ffmpeg pad the 3-second test clip
endlessly (a 22.9GB WAV) until the disk filled, which was the real
cause of the voice separation failures. Trim to the probed duration
and drop apad.
2026-09-19 14:33:06 +08:00
134
8b2955eb0b Add canvas undo/redo and full shortcut system
Snapshot-based undo/redo (max 50 steps) with Cmd+Z / Cmd+Shift+Z,
ignoring volatile generation progress fields. Bind the remaining
reference shortcuts: Cmd+D duplicate, Cmd+Enter generate selection,
Tab new-node palette, Cmd+F canvas node search panel, Cmd+0 fit view,
Cmd+=/- zoom, V pan-tool toggle, H hold-to-pan, plain G to group.
Expand the shortcut help panel to the full four-group layout.
2026-09-19 12:53:07 +08:00
134
94b3ac9cb3 Restyle story script nodes to dark theme
Align the story script generator table and script text node with the
reference product: dark cards with subtle white borders, muted header
row, dark status pills, white primary button and dark textarea card,
replacing the previous light table chrome.
2026-09-18 22:57:07 +08:00
134
8a1876e016 Render canvas image nodes with object-contain
Show the full picture inside image resource nodes instead of cropping
with object-cover; matches the reference product where node frames
adapt to the image and no content is clipped, and fixes existing
nodes created before aspect fitting was added.
2026-09-18 18:45:39 +08:00
134
b4b6e1a738 Size dropped image nodes to intrinsic aspect ratio
Dropped images landed in fixed 420x280 nodes and rendered with
object-cover, cropping portraits and wide photos. Read the image's
natural size during ingestion and fit the node frame to the aspect
ratio (landscape caps height, portrait caps width) so the full
picture stays visible like the reference product.
2026-09-18 18:34:23 +08:00
134
885cab9fca Document voice separation rollout 2026-09-18 18:21:47 +08:00
134
6ed8879b28 Wire canvas voice/background separation to remote HTDemucs service
The private FunASR/HTDemucs audio service was already live and used by
the smart edit pipeline, but the canvas video toolbar still claimed
voice/background separation was unavailable. Add a dedicated
/api/canvas/audio-separate route (download, ffmpeg stem extraction,
remote separation, storage upload) and route the toolbar's voice and
background modes through it, replacing the stale not-connected notice.
2026-09-18 18:16:44 +08:00
134
c1975942ac Document video canvas shell cleanup and media upload fix rollout 2026-09-18 18:04:51 +08:00
134
a4354dc535 Remove video canvas demo shells and fix media upload timeouts
- Starter/toolbox presets no longer seed fake generated results; demo
  fallback media URLs in tool actions now fail with honest guidance
- Delete dead demo scene components, fake director console, demo
  history constants and unreachable submit toast fallbacks
- Smart edit dialog accepts desktop drag-and-drop video files
- Raise media upload client timeout to 10 minutes and surface
  friendly timeout errors instead of raw 'signal timed out'
2026-09-18 17:40:49 +08:00
134
3002704d77 Document video canvas production rollout 2026-09-18 15:24:10 +08:00
134
fe1c9b60f3 Fix video canvas media imports and node layout 2026-09-18 15:09:34 +08:00
134
4f9453b8b2 Add durable remote audio inference and verified offline speech models 2026-09-18 14:12:28 +08:00
134
229f4d72cb Record verified production rollout of canvas groups and video upscaling 2026-09-18 13:37:54 +08:00
134
12576c565f Complete durable canvas groups and add isolated P40 video upscaling 2026-09-18 13:18:48 +08:00
134
b17b8b7689 Make canvas connections resolve live inputs and respect disconnects 2026-09-18 13:18:48 +08:00
134
0d5558c50c Fix handwritten storyboard entry and generic grid presets; record GPU allocation audit 2026-09-18 13:18:48 +08:00
134
c6533e92f3 Complete smart edit audio processing and font catalogue with main app validation 2026-09-18 13:18:48 +08:00
134
17e17b9ad5 Fix smart edit captions, audio gain and timeline parity 2026-09-18 13:18:48 +08:00
134
689fa615e8 Audit smart edit parity and fix caption timing and outline previews 2026-09-18 13:18:47 +08:00
134
725c1ec3d2 Match smart edit transition previews and centered effects to reference 2026-09-18 13:18:47 +08:00
134
84bfd6b9f4 Support local preview transforms and detached smart edit audio tracks 2026-09-18 13:18:47 +08:00
134
6869bd8c18 Align smart edit canvas and timeline UI with reference workflow 2026-09-17 18:05:29 +08:00
134
a5f4ad0385 Bound per-input decoder and scene detection threads 2026-09-17 16:42:06 +08:00
134
6eff0a7332 Use static import in remote rendering helper test 2026-09-17 16:39:51 +08:00
134
a00a910227 Verify remote render recovery identity and endpoint binding 2026-09-17 16:39:51 +08:00
134
721de36692 Remove duplicate canvas definitions introduced by release merge 2026-09-17 16:34:47 +08:00
134
be8bdb4dab Merge verified video canvas updates with production hotfix history 2026-09-17 16:33:36 +08:00
134
bbac6289b3 Add smart editing workflow and isolated durable video rendering service 2026-09-17 16:32:04 +08:00
134
cb42f27ee9 Add official Alibaba audio generation and editable director object tracks 2026-09-17 15:43:32 +08:00
134
292eddc5a6 fix(video-canvas): resume provider jobs and recover editing and 3d state 2026-09-17 15:01:02 +08:00
134
69dcf9b745 fix(video-canvas): persist story analysis and resume accepted enhance tasks 2026-09-17 13:51:15 +08:00
134
86b66172b1 fix(video-canvas): persist image operations and story subtasks 2026-09-17 13:30:32 +08:00
134
0d17638a8d fix(video-canvas): persist generation tasks and repair workflow controls 2026-09-17 12:32:47 +08:00
134
23f86dfce1 fix(video-canvas): repair audited controls and document remaining validation gaps 2026-09-17 11:22:49 +08:00
134
f10e6b7c8d fix(video-canvas): connect generation controls and repair script media director workflows 2026-09-17 10:28:18 +08:00
134
e5b2de3c1b docs(video-canvas): audit 231 nested LibTV feature checkpoints 2026-09-17 09:48:11 +08:00
134
d5593ac3f0 feat(video-canvas): implement camera motion timeline and real MP4 export 2026-09-17 00:31:36 +08:00
134
38be09de9c feat(video-canvas): connect real 3d director and validate media chain 2026-09-17 00:02:03 +08:00
134
598f5f617a feat(video-canvas): add tested Aliyun Wan first-frame adapter 2026-09-16 23:03:37 +08:00
134
f7550251ec fix(video-canvas): preserve story length and archive generated videos 2026-09-16 22:12:35 +08:00
134
2ba093b714 feat(video-canvas): connect prompt tools and durable media processing 2026-09-16 21:37:07 +08:00
134
5b0d75a738 fix(video-canvas): persist edits and record live provider regression results 2026-09-16 18:56:03 +08:00
134
027f0a4201 docs: record official Qwen3.8 Max canvas configuration 2026-09-16 18:10:25 +08:00
134
96b8e1662c docs: record official Qwen3.8 Max canvas configuration 2026-09-16 18:10:05 +08:00
134
290f6d2c99 docs: record video canvas production rollout and verification 2026-09-16 16:35:48 +08:00
134
7c276fe43a docs: record video canvas production rollout and verification 2026-09-16 16:35:27 +08:00
134
741fe3663e fix: exclude build artifacts from canvas stylesheet scanning 2026-09-16 16:29:31 +08:00
134
924c8065b6 fix: match video canvas models to real capabilities and remove simulated processing 2026-09-16 16:29:31 +08:00
134
50d782850f fix: exclude build artifacts from canvas stylesheet scanning 2026-09-16 15:11:50 +08:00
134
60b7900bf9 fix: match video canvas models to real capabilities and remove simulated processing 2026-09-16 15:06:05 +08:00
134
0c1a2e6fee docs: record Claw 3D loading rollout verification 2026-09-15 16:24:43 +08:00
134
73593051fe fix: render Claw models without waiting for external HDR lighting 2026-09-15 16:21:06 +08:00
134
3df44f1f8c fix: render Claw models without waiting for external HDR lighting 2026-09-15 16:20:23 +08:00
134
54c857d714 docs: record Claw history recovery rollout checks 2026-09-15 14:39:08 +08:00
134
3bf0fec097 fix: restore Claw history before waiting for active tasks 2026-09-15 14:37:26 +08:00
134
d31079e26d fix: restore Claw history before waiting for active tasks 2026-09-15 14:34:58 +08:00
134
2bf5f6232d fix: separate Claw markdown table columns 2026-09-15 13:36:52 +08:00
134
c024ae31b9 fix: separate Claw markdown table columns 2026-09-15 13:34:41 +08:00
134
8f0dff35ed docs: record Claw production rollout verification 2026-09-14 09:16:01 +08:00
134
f3cded51fd fix: stop Claw artifact request loops and restore CAD actions 2026-09-14 09:10:16 +08:00
134
b45d9ee314 fix: stop Claw artifact request loops and restore CAD actions 2026-09-14 09:09:31 +08:00
134
0aff83f61a docs: record VIP98K live matrix and deployment evidence 2026-09-11 15:10:20 +08:00
134
b26e265083 fix: verify VIP98K sizes and recover replacement results 2026-09-11 14:56:00 +08:00
134
aaa650265c fix: verify VIP98K sizes and recover replacement results 2026-09-11 14:52:38 +08:00
134
a2a9e7d8b2 fix: stabilize detail replacement generation 2026-09-11 11:36:53 +08:00
134
dcff190c62 fix(claw): prevent new chat history rebound 2026-09-11 11:19:04 +08:00
134
a7d994d93b fix: advertise verified vip98k image sizes 2026-09-10 16:32:25 +08:00
134
a1973ae202 fix: honor vip98k image contracts 2026-09-10 16:19:49 +08:00
134
f9d1b46d81 fix: bind image extension to Alibaba provider key 2026-09-10 12:48:27 +08:00
134
821b8658a8 feat: add verified VIP98K multi-image models 2026-09-10 10:08:13 +08:00
134
6f5a444756 fix: preserve detail replacement product geometry 2026-09-09 19:50:03 +08:00
134
2e340c004a fix: restore bananaimage models 2026-09-08 10:14:29 +08:00
134
2a9b05de80 fix: hide unsupported vip98k image editors 2026-09-08 10:00:19 +08:00
134
08ed9ac3ef perf: cache vip98k reference staging 2026-09-07 20:50:20 +08:00
134
2fb53b4279 test: accept synchronous vip98k image results 2026-09-07 20:28:21 +08:00
134
42359fbf0f fix: support vip98k image array contracts 2026-09-07 20:20:07 +08:00
134
6ce7c31a1e fix: poll qwen image generation asynchronously 2026-09-07 19:46:18 +08:00
134
e49b0707e9 fix: enforce vip98k multi-image contracts 2026-09-07 18:17:01 +08:00
134
77f8774a55 fix: restore vip98k text image models 2026-09-07 15:06:40 +08:00
134
3a1c445c0b fix: stabilize vip98k image generation uploads 2026-09-07 14:45:06 +08:00
134
cfd3418bb7 fix: accept large canvas references for vip98k 2026-09-07 13:17:39 +08:00
134
faf3a2dbb2 fix: retry vip98k reconciliation responses 2026-09-07 13:14:50 +08:00
134
002ab8b98d fix: bypass vip98k media fetch cache on retry 2026-09-07 12:41:41 +08:00
134
c69ddb1bb3 fix: retry vip98k while staged media propagates 2026-09-07 12:32:38 +08:00
134
01b3f6eec1 fix: use public upload path for vip98k staging 2026-09-07 12:24:25 +08:00
134
6a30795bb8 fix: stage vip98k media on reachable storage 2026-09-07 12:18:26 +08:00
134
0a364f537f fix: normalize vip98k references for provider limits 2026-09-07 12:04:46 +08:00
134
d133a48866 fix: stabilize vip98k reference image delivery 2026-09-07 11:48:40 +08:00
134
434d0a63cd fix: keep verified vip98k image models available 2026-09-07 11:16:52 +08:00
134
608bd25679 fix: trust vip98k published image contracts 2026-09-07 00:32:33 +08:00
134
47909e39a9 fix: update vip98k image task integration 2026-09-07 00:22:45 +08:00
134
740fc90dfd feat: support updated OpenClaw 3D task protocol 2026-09-06 12:45:39 +08:00
134
272592488b fix(claw): persist 3d preview and restore chat history 2026-09-06 09:19:09 +08:00
134
6269aefee5 fix(ai): preserve relational image references 2026-09-05 18:46:25 +08:00
134
9717196264 fix(claw): assemble legacy 3d models by default 2026-09-03 17:43:01 +08:00
134
2817aab197 Merge Claw interactive 3D preview support 2026-09-02 20:22:41 +08:00
134
bbfece5720 feat(claw): support interactive 3D previews 2026-09-02 20:21:40 +08:00
134
e0501bbbd1 Merge remote-tracking branch 'origin/codex/bbyy-production-156-20260820' into codex/deploy-48-bloub-20260902 2026-09-02 19:14:09 +08:00
134
6ab68e8ac9 fix(canvas): show Bloub in chat and image placeholders 2026-09-02 19:12:58 +08:00
134
0ed21b1b76 Merge remote-tracking branch 'origin/codex/bbyy-production-156-20260820' into codex/deploy-48-bloub-20260902 2026-09-02 18:08:52 +08:00
134
02017a2531 feat(ui): add official Bloub assistant states 2026-09-02 18:06:46 +08:00
134
0a375de55c feat(claw): preview 3D artifacts in chat 2026-09-02 12:46:05 +08:00
134
4d8c984d08 feat(claw): preview 3D artifacts in chat 2026-09-02 12:42:02 +08:00
134
c39d7260fb fix(model3d): restore archived world assets and previews 2026-08-28 22:17:20 +08:00
134
cb9a90d9ac fix(ai): restore models after successful circuit probe 2026-08-27 17:10:09 +08:00
134
412fa85a81 fix(ai): restore models after successful circuit probe 2026-08-27 17:09:33 +08:00
134
ac1e586156 fix(ai): fail over forbidden image requests 2026-08-27 16:32:47 +08:00
134
e1c155be9e fix(ai): fail over forbidden image requests 2026-08-27 16:31:27 +08:00
134
2a4f8f631b feat(ai): support Qwen Image 3.0 2026-08-26 16:10:47 +08:00
134
ba232f3385 feat(ai): support Qwen Image 3.0 2026-08-26 16:10:00 +08:00
134
5f6341a318 fix: preserve manually disabled image models 2026-08-26 13:44:42 +08:00
134
d005b3477a fix: preserve manually disabled image models 2026-08-26 13:43:58 +08:00
134
07a9a61743 fix(ai): fail over failed VIP98K image tasks 2026-08-25 17:04:50 +08:00
root
86c8e4995d Merge remote-tracking branch 'origin/codex/48-ai-template-integration-20260813' into codex/48-ai-template-integration-20260813 2026-08-25 17:04:04 +08:00
134
e5dc650443 fix(ai): fail over failed VIP98K image tasks 2026-08-25 17:00:38 +08:00
134
9596b0a667 fix(canvas): honor text edit failover exclusions 2026-08-25 15:44:25 +08:00
134
4a30cad91f fix(canvas): honor text edit failover exclusions 2026-08-25 15:43:50 +08:00
134
7752ec42ee fix(canvas): restore reliable text editing 2026-08-25 14:46:01 +08:00
134
3a88fb0df0 fix(canvas): restore reliable text editing 2026-08-25 14:44:50 +08:00
134
f95c6b2b2b fix(canvas): decouple text detection from image provider 2026-08-25 14:03:51 +08:00
134
a0999bfcde fix(canvas): decouple text detection from image provider 2026-08-25 14:03:25 +08:00
134
4b5540f3c2 fix(ai): route vip98k image tools correctly 2026-08-25 13:30:33 +08:00
134
d120e0ddb6 fix(ai): route vip98k image tools correctly 2026-08-25 13:27:31 +08:00
134
b73afcdfdc fix(ai): mark vip98k images as async tasks 2026-08-25 13:11:40 +08:00
134
c21a86beea fix(ai): mark vip98k images as async tasks 2026-08-25 13:07:58 +08:00
134
bcb21344ba fix(ai): update vip98k image task contract 2026-08-25 13:02:39 +08:00
134
e87724499b fix(ai): update vip98k image task contract 2026-08-25 12:59:53 +08:00
134
42e1175830 fix: persist generated images before gallery save 2026-08-22 17:25:34 +08:00
134
afac47add2 fix(home): harden Qiniu featured previews 2026-08-20 22:41:01 +08:00
134
d1582f33e8 fix(bbyy): isolate production branding from E2E defaults 2026-08-20 22:12:33 +08:00
134
1d84b1b1c0 fix(ops): wait for production container startup 2026-08-20 19:52:05 +08:00
134
f0a24a1a99 fix(canvas): optimize Qiniu image previews 2026-08-20 19:47:02 +08:00
134
3d12c508d3 fix: stop retrying expired image archives forever 2026-08-20 19:34:56 +08:00
134
88dec71a41 ops: make production builds atomic 2026-08-20 19:32:49 +08:00
134
6180c5fc49 fix: restore production model availability helpers 2026-08-20 19:28:51 +08:00
root
dfb50c83fa fix: make chat layout dynamic for stable homepage bundles 2026-08-19 16:01:52 +08:00
134
3ad353b97f fix: make chat layout dynamic to avoid broken home chunk artifacts 2026-08-19 16:00:38 +08:00
134
e05b42942d fix(world): fallback status polling from failed external task result 2026-08-19 13:48:07 +08:00
134
e601f8079f fix(world): harden world task create external-marble fallback 2026-08-19 13:45:10 +08:00
134
fd49843c5a fix(world): align vip98k world task mode support 2026-08-19 13:17:37 +08:00
134
a9a9f8e138 fix(world): guard unsupported modes for external world task provider 2026-08-19 12:34:13 +08:00
134
15c3b519a7 Fix image proxy public access and model avatar fallback 2026-08-19 09:27:10 +08:00
134
c27a707776 fix world labs task polling fallback for dynamic response payloads 2026-08-18 18:43:17 +08:00
134
7602365bb8 fix(canvas3d): retry transient poll failures before fail state 2026-08-18 16:53:28 +08:00
134
1fa126cecf fix(canvas): allow repeated layout source picking 2026-08-14 12:10:23 +08:00
134
398a4bbefd feat(canvas): add direct canvas image picking 2026-08-14 11:47:42 +08:00
134
3010e51f72 feat(canvas): add durable batch template replacement 2026-08-14 10:20:01 +08:00
134
71b3ce56cf fix(canvas): restore AI layout replacement 2026-08-14 00:18:15 +08:00
134
bf6bd01fcb feat(canvas): add batch template replacement 2026-08-13 23:29:08 +08:00
134
3009727ec1 chore(git): ignore local runtime data 2026-08-13 18:38:28 +08:00
134
e2806578c0 fix(release): align template flow with production baseline 2026-08-13 18:34:46 +08:00
134
231650dc00 chore(git): enforce deployment branch and feature guards 2026-08-13 18:31:44 +08:00
134
675966a783 feat(agent): add AI template reuse to main results 2026-08-13 18:31:19 +08:00
134
9ff3dc68ce test(canvas): guard PSD template toolbar entry 2026-08-13 18:28:53 +08:00
134
01722af2fb fix(canvas): restore PSD template replacement entry 2026-08-13 18:28:53 +08:00
134
46a28f77f0 fix(claw): restore spreadsheet attachment badges 2026-08-11 21:24:44 +08:00
134
aafc970ae8 fix(claw): preserve per-task credentials on reconnect 2026-08-11 18:22:32 +08:00
446 changed files with 99719 additions and 11950 deletions

27
.gitignore vendored
View File

@ -50,6 +50,7 @@ yarn-error.log*
/playwright-report/
/blob-report/
/playwright/*
/output/
deploy.tar.gz
libvips-8.14.5-linux-x64.tar.br
.temp-packages/
@ -70,6 +71,10 @@ migration-*.txt
artifacts/liblib-*/
analysis/qiaodou-extract/
# 本地运行时数据库与上传文件
.local/
public/uploads/local/
# 模型文件
*.model
*.pth
@ -100,3 +105,25 @@ public/vendor/zoo-modeling-app/
# Drizzle migrations must be versioned (override the global *.sql ignore above)
!lib/db/migrations/*.sql
# Private smart edit runtime and media/model caches
.venv-smart-edit/
.smart-edit-models/
.cache/smart-edit/
services/video-render/dist/
# Downloaded and normalized subtitle fonts are runtime assets.
.cache/smart-edit-fonts/
services/smart-edit/__pycache__/
# Local verification artefacts (also excluded from Tailwind's automatic source scan).
.qa/
.next-video-*/
services/video-upscale/dist/
services/video-upscale/__pycache__/
services/video-upscale/vendor/__pycache__/
/services/audio-processing/dist/
/services/audio-processing/__pycache__/

View File

@ -94,7 +94,7 @@ pnpm db:generate
推荐提交前至少运行:
```bash
git fetch github main --prune
git fetch origin --prune
pnpm check:branch
pnpm check:quality
pnpm audit:security

View File

@ -665,7 +665,7 @@ export function LocalAiImagePipelineSections({
</p>
<p>• 支持的图片格式:JPG, JPEG, PNG, BMP, WEBP, TIFF等</p>
<p>
• 图片大小:不超过50MB,不限制分辨率(建议按业务需求控制尺寸)
• 图片大小:不超过20MB,不限制分辨率(建议按业务需求控制尺寸)
</p>
<p>• 返回的链接有效期为1小时,系统会自动下载并存储到S3</p>
</div>
@ -784,7 +784,7 @@ export function LocalAiImagePipelineSections({
<p>• 支持2x、4x、8x放大倍数</p>
<p>• 处理时间:通常1-10秒,取决于图片大小</p>
<p>• 支持格式:JPG, PNG, JPEG等常见图片格式</p>
<p>• 文件限制:最大50MB</p>
<p>• 文件限制:最大20MB</p>
</div>
</div>
</>
@ -2377,13 +2377,10 @@ export function LocalAiImagePipelineSections({
items={[
{
label:
extendConfig.provider === "vip98k" || extendConfig.apiKey
? "Key 已配置"
: "Key 待配置",
tone:
extendConfig.provider === "vip98k" || extendConfig.apiKey
? "enabled"
: "warning",
extendConfig.provider === "vip98k"
? "VIP98K 服务商"
: "阿里云服务商",
tone: "enabled",
},
{
label: `${extendConfig.xScale}x / ${extendConfig.yScale}x`,
@ -2400,22 +2397,11 @@ export function LocalAiImagePipelineSections({
VIP98K 服务商读取,仅在这里维护扩展比例。
</div>
) : (
<div className="space-y-2">
<Label htmlFor="extendApiKey">API密钥 *</Label>
<Input
id="extendApiKey"
onChange={(e) =>
setExtendConfig({ ...extendConfig, apiKey: e.target.value })
}
placeholder="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
type="password"
value={extendConfig.apiKey}
/>
<div className="text-muted-foreground text-xs">
请输入阿里云Dashscope API密钥,例如:
sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
<div className="rounded-lg border bg-muted/40 p-4 text-sm">
使用阿里云 DashScope 的 image-out-painting 模型。API Key
从“AI 模型设置”中已启用的阿里云官方生图服务商读取,
仅在这里维护拓展比例。
</div>
</div>
)}
<div className="grid grid-cols-2 gap-4">
<div className="space-y-2">
@ -2470,7 +2456,7 @@ export function LocalAiImagePipelineSections({
https://dashscope.aliyuncs.com/api/v1/services/aigc/image2image/out-painting
</p>
<p>• 支持的图片格式:JPG, PNG, WebP等常见格式</p>
<p>• 最大图片大小:4MB</p>
<p>• 最大图片大小:10MB</p>
<p>• 处理时间:约5-15秒/张</p>
<p>• 放大倍数范围:1.0-2.0(推荐1.5倍等比例扩展)</p>
</div>

View File

@ -275,7 +275,6 @@ export type EnhanceConfig = {
export type ExtendConfig = {
provider: "vip98k" | "dashscope";
apiKey: string;
xScale: string;
yScale: string;
};

View File

@ -38,6 +38,7 @@ import {
ImagePlus,
Key,
Layers,
LayoutTemplate,
Layers3,
Loader2,
type LucideIcon,
@ -1061,6 +1062,7 @@ const CANVAS_TOOLBAR_FALLBACK_ICONS: Record<CanvasToolbarItemKey, LucideIcon> =
imageRemoveBackground: Scissors,
imageBoxCutout: Scan,
imageLocalEdit: Wand2,
imageLayoutReplace: LayoutTemplate,
imageRemoveWatermark: Eraser,
imageResize: Maximize2,
imageTextEdit: Type,
@ -2180,7 +2182,7 @@ export function AdminSettingsClient() {
usageTimeEnabled: false,
usageTimeStart: "08:00",
usageTimeEnd: "20:00",
usageTimeMessage: "由壹点累了也是需要休息的",
usageTimeMessage: "BuYi 累了也是需要休息的",
defaultTheme: "system",
bannerEnabled: true,
bannerText: "限时特惠:升级会员最高立享 45% OFF!",
@ -2222,7 +2224,7 @@ export function AdminSettingsClient() {
agentWorkflowDefaultModelId: null,
canvasDetailDefaultModelId: null,
canvasDetailAgentModelId: null,
lovartComposerPrefix: "让「由壹点AI」",
lovartComposerPrefix: "让「𝗕𝘂𝗬𝗶」",
lovartComposerPreviewTexts: [
"打造引人注目的社交媒体视觉",
"帮我把这张产品图做成高级电商海报",
@ -2238,8 +2240,8 @@ export function AdminSettingsClient() {
lovartPromoText:
"现在升级,立享 Kling 3.0 与 Nano Banana Pro,最高 5 折优惠",
lovartPromoActionText: "立即升级",
lovartHeroTitle: "由壹点AI 让设计更简单",
lovartHeroSubtitle: "懂你的设计代理,帮你搞定一切",
lovartHeroTitle: "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle: "下达指令,即刻呈现",
surpriseCategories: ["全部", "精选"],
membershipConfig: defaultMembershipConfig,
membershipUserOverrides: defaultMembershipUserOverrides,
@ -2300,7 +2302,7 @@ export function AdminSettingsClient() {
usageTimeStart: siteSettingsData.usageTimeStart || "08:00",
usageTimeEnd: siteSettingsData.usageTimeEnd || "20:00",
usageTimeMessage:
siteSettingsData.usageTimeMessage || "由壹点累了也是需要休息的",
siteSettingsData.usageTimeMessage || "BuYi 累了也是需要休息的",
defaultTheme: siteSettingsData.defaultTheme || "system",
bannerEnabled: siteSettingsData.bannerEnabled ?? true,
bannerText:
@ -2396,7 +2398,7 @@ export function AdminSettingsClient() {
(siteSettingsData as any).canvasDetailAgentModelId || null,
lovartComposerPrefix:
String((siteSettingsData as any).lovartComposerPrefix || "").trim() ||
"让「由壹点AI」",
"让「𝗕𝘂𝗬𝗶」",
lovartComposerPreviewTexts: Array.isArray(
siteSettingsData.lovartComposerPreviewTexts
)
@ -2498,9 +2500,9 @@ export function AdminSettingsClient() {
lovartPromoActionText:
siteSettingsData.lovartPromoActionText || "立即升级",
lovartHeroTitle:
siteSettingsData.lovartHeroTitle || "由壹点AI 让设计更简单",
siteSettingsData.lovartHeroTitle || "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle:
siteSettingsData.lovartHeroSubtitle || "懂你的设计代理,帮你搞定一切",
siteSettingsData.lovartHeroSubtitle || "下达指令,即刻呈现",
surpriseCategories: Array.isArray(siteSettingsData.surpriseCategories)
? siteSettingsData.surpriseCategories
.map((item: any) => String(item).trim())
@ -2631,8 +2633,7 @@ export function AdminSettingsClient() {
// 图像拓展API配置状态
const [extendConfig, setExtendConfig] = useState({
provider: "vip98k" as "vip98k" | "dashscope",
apiKey: "",
provider: "dashscope" as "vip98k" | "dashscope",
xScale: "1.5", // X轴放大倍数
yScale: "1.5", // Y轴放大倍数
});
@ -2951,13 +2952,11 @@ export function AdminSettingsClient() {
if (extendConfigData?.setting?.value) {
const config = extendConfigData.setting.value as {
provider?: "vip98k" | "dashscope";
apiKey?: string;
xScale?: number;
yScale?: number;
};
setExtendConfig({
provider: config.provider === "vip98k" ? "vip98k" : "dashscope",
apiKey: config.apiKey || "",
xScale: config.xScale ? String(config.xScale) : "1.5",
yScale: config.yScale ? String(config.yScale) : "1.5",
});
@ -5554,11 +5553,6 @@ export function AdminSettingsClient() {
// --- 图像拓展API配置逻辑 ---
const handleSaveExtendConfig = async () => {
try {
if (extendConfig.provider !== "vip98k" && !extendConfig.apiKey.trim()) {
toast.error("请输入API密钥");
return;
}
const xScale = Number.parseFloat(extendConfig.xScale);
const yScale = Number.parseFloat(extendConfig.yScale);
@ -5579,7 +5573,6 @@ export function AdminSettingsClient() {
key: "extend_api_config",
value: {
provider: extendConfig.provider,
apiKey: extendConfig.apiKey.trim(),
xScale,
yScale,
},
@ -8998,7 +8991,7 @@ export function AdminSettingsClient() {
key: "extend",
label: "图像拓展",
group: "图像处理",
tone: extendConfig.apiKey ? "enabled" : "warning",
tone: "enabled",
},
{
key: "watermark-remove",
@ -10522,7 +10515,7 @@ export function AdminSettingsClient() {
lovartComposerPrefix: e.target.value.slice(0, 40),
})
}
placeholder="让「由壹点AI」"
placeholder="让「𝗕𝘂𝗬𝗶」"
value={siteSettings.lovartComposerPrefix}
/>
<div className="text-muted-foreground text-xs">
@ -10724,7 +10717,7 @@ export function AdminSettingsClient() {
lovartHeroTitle: e.target.value,
})
}
placeholder="由壹点AI 让设计更简单"
placeholder="𝗕𝘂𝗬𝗶 让设计触手可及"
value={siteSettings.lovartHeroTitle}
/>
</div>
@ -10738,7 +10731,7 @@ export function AdminSettingsClient() {
lovartHeroSubtitle: e.target.value,
})
}
placeholder="懂你的设计代理,帮你搞定一切"
placeholder="下达指令,即刻呈现"
value={siteSettings.lovartHeroSubtitle}
/>
</div>

View File

@ -1,4 +1,5 @@
"use client";
import { filterCanvasModels } from "@/lib/video-canvas/model-matching";
import Link from "next/link";
import { ArrowLeft, Plus, Trash2 } from "lucide-react";
@ -24,10 +25,6 @@ import {
} from "@/components/ui/select";
import { Switch } from "@/components/ui/switch";
import { Textarea } from "@/components/ui/textarea";
import {
isImageGenerationModelDescriptor,
isVideoModelDescriptor,
} from "@/lib/ai/model-classification";
import { fetcher } from "@/lib/utils";
import {
getVideoCanvasFunctionBillingCopy,
@ -87,56 +84,23 @@ function getModelOptionLabel(model: AiModelOption) {
return `${name}${id}`;
}
function isImageModelOption(model: AiModelOption) {
return isImageGenerationModelDescriptor({
providerType: model.providerType,
modelId: model.modelId,
capabilities: model.capabilities,
supportsImageGen: model.supportsImageGen,
});
}
function isVideoModelOption(model: AiModelOption) {
return isVideoModelDescriptor({
providerType: model.providerType,
modelId: model.modelId,
capabilities: model.capabilities,
supportsImageGen: model.supportsImageGen,
});
}
function filterModelsForFunctionKind(
models: AiModelOption[],
kind: VideoCanvasFunctionDefinition["kind"],
) {
const enabled = models.filter((model) => model.isEnabled !== false && getModelOptionValue(model));
let filtered = enabled;
if (kind === "image" || kind === "image-edit") {
filtered = enabled.filter(isImageModelOption);
} else if (kind === "video") {
filtered = enabled.filter(isVideoModelOption);
} else if (kind === "multimodal") {
filtered = enabled.filter(
(model) => model.isMultimodal || model.supportsVision || (!isImageModelOption(model) && !isVideoModelOption(model)),
);
}
return filtered.length ? filtered : enabled;
}
function ModelSelect({
allowInherit = false,
kind,
functionKey,
models,
onChange,
value,
}: {
allowInherit?: boolean;
kind: VideoCanvasFunctionDefinition["kind"];
functionKey?: string;
models: AiModelOption[];
onChange: (value: string) => void;
value: string;
}) {
const filteredModels = filterModelsForFunctionKind(models, kind);
const modelKind = functionKey === "audio-to-video" ? "audio-video" : kind === "image" || kind === "image-edit" || kind === "video" || kind === "multimodal" ? kind : "text";
const filteredModels = filterCanvasModels(models, modelKind);
const normalizedValue = value?.trim() || (allowInherit ? INHERIT_MODEL_VALUE : "auto");
const hasCurrent =
normalizedValue === "auto" ||
@ -851,13 +815,14 @@ export function VideoCanvasSettingsClient() {
<div className="grid gap-4 md:grid-cols-1">
<div className="space-y-2">
<Label htmlFor={`model-${item.key}`}>模型覆盖</Label>
<ModelSelect
{["canvas-generate-image", "canvas-generate-video", "canvas-workflow-analyze", "pick-object"].includes(item.route) ? <ModelSelect
allowInherit
functionKey={item.key}
kind={item.kind}
models={modelOptions}
onChange={(value) => updateFunction(item.key, { model: value })}
value={item.model || ""}
/>
/> : <p className="text-sm text-muted-foreground">{item.route === "local" ? "本地处理,无需生成模型" : "使用此功能的专用服务接口"}</p>}
</div>
<div className="rounded-lg border border-dashed border-border/70 bg-muted/20 px-3 py-2">
<div className="text-xs font-medium text-foreground">

View File

@ -2,9 +2,31 @@ import { createLogger } from "@/lib/observability/logger";
import { generateText } from "ai";
import { fetchImageBufferFromUrl } from "@/lib/ai/rmbg/service";
import { getLanguageModelById } from "@/lib/ai/dynamic-provider";
import {
DETAIL_REPLACEMENT_IMAGE_MODEL_CONFIG_KEY,
normalizeDetailReplacementImageModelConfig,
} from "@/lib/ai/detail-replacement-image-model-config";
import {
compositeReplacementWithReferenceLocks,
parseReplacementLayoutLockRegions,
type NormalizedReplacementBox,
} from "@/lib/ai/replacement-layout-lock";
import {
buildReplacementProductIdentityAnalysisPrompt,
buildReplacementProductStructureLockLines,
formatReplacementProductIdentityProfile,
normalizeReplacementProductImageIndexes,
parseReplacementProductIdentityProfile,
type ReplacementProductIdentityProfile,
} from "@/lib/ai/replacement-product-identity";
import { resolveDetailAgentLocalMoonshotModel } from "@/lib/ai/detail-agent-model-runtime";
import { resolveDetailReplacementVisionRuntime } from "@/lib/ai/detail-replacement-vision-runtime";
import { inlineVisionImages } from "@/lib/ai/vision-inline";
import { getEnabledAiModelsWithProvider, getSiteSettings } from "@/lib/db/queries";
import {
getEnabledAiModelsWithProvider,
getSiteSettings,
getSystemSetting,
} from "@/lib/db/queries";
import { uploadFile } from "@/lib/storage/service";
import { bufferToArrayBuffer } from "./persistence";
import { isVideoProviderType } from "./video-models";
@ -234,6 +256,8 @@ export async function normalizeReplacementResultImage(params: {
referenceImageUrl: string;
referenceGeometry?: ReplacementReferenceGeometry | null;
scenePlan?: ReplacementScenePlan | null;
preserveReferenceOutsideSubject?: boolean;
preserveReferenceText?: boolean;
uploadPrefix: string;
}): Promise<{
originalUrl: string;
@ -242,6 +266,11 @@ export async function normalizeReplacementResultImage(params: {
}> {
const sharp = (await import("sharp")).default;
const generatedBuffer = await fetchImageBufferFromUrl(params.generatedImageUrl);
let referenceBuffer: Buffer | null = null;
const getReferenceBuffer = async () => {
referenceBuffer ||= await fetchImageBufferFromUrl(params.referenceImageUrl);
return referenceBuffer;
};
let targetWidth = params.referenceGeometry?.width;
let targetHeight = params.referenceGeometry?.height;
const generatedMetadata = await sharp(generatedBuffer, { failOn: "none" })
@ -250,8 +279,8 @@ export async function normalizeReplacementResultImage(params: {
if (!targetWidth || !targetHeight) {
try {
const referenceBuffer = await fetchImageBufferFromUrl(params.referenceImageUrl);
const referenceMetadata = await sharp(referenceBuffer, { failOn: "none" })
const resolvedReferenceBuffer = await getReferenceBuffer();
const referenceMetadata = await sharp(resolvedReferenceBuffer, { failOn: "none" })
.rotate()
.metadata();
targetWidth = referenceMetadata.width;
@ -298,7 +327,7 @@ export async function normalizeReplacementResultImage(params: {
: shouldPreserveFullGeneratedCanvas
? "contain"
: "cover";
const normalizedBuffer =
let normalizedBuffer =
targetWidth && targetHeight
? await imagePipeline
.resize(targetWidth, targetHeight, {
@ -312,6 +341,82 @@ export async function normalizeReplacementResultImage(params: {
.png()
.toBuffer()
: await imagePipeline.png().toBuffer();
if (
targetWidth &&
targetHeight &&
(params.preserveReferenceOutsideSubject || params.preserveReferenceText)
) {
let layoutLocks = {
subjectBoundingBox: params.scenePlan?.subjectBoundingBox || null,
textBoundingBoxes: params.scenePlan?.textBoundingBoxes || [],
analyzed: params.scenePlan?.layoutLockAnalyzed === true,
};
const needsLayoutLockFallback =
!layoutLocks.analyzed ||
(params.preserveReferenceOutsideSubject && !layoutLocks.subjectBoundingBox);
if (needsLayoutLockFallback) {
const fallbackLocks = await resolveReplacementLayoutLockRegions({
referenceImageUrl: params.referenceImageUrl,
referenceBuffer: await getReferenceBuffer(),
});
layoutLocks = {
subjectBoundingBox:
layoutLocks.subjectBoundingBox || fallbackLocks.subjectBoundingBox,
textBoundingBoxes:
layoutLocks.textBoundingBoxes.length > 0
? layoutLocks.textBoundingBoxes
: fallbackLocks.textBoundingBoxes,
analyzed: layoutLocks.analyzed || fallbackLocks.analyzed,
};
}
const shouldUseSafeReferenceFallback = Boolean(
(params.preserveReferenceOutsideSubject && !layoutLocks.subjectBoundingBox) ||
(params.preserveReferenceText && !layoutLocks.analyzed),
);
normalizedBuffer = shouldUseSafeReferenceFallback
? await sharp(await getReferenceBuffer(), { failOn: "none" })
.rotate()
.resize(targetWidth, targetHeight, { fit: "fill" })
.png()
.toBuffer()
: await compositeReplacementWithReferenceLocks({
generatedBuffer: normalizedBuffer,
referenceBuffer: await getReferenceBuffer(),
width: targetWidth,
height: targetHeight,
preserveReferenceOutsideSubject: Boolean(
params.preserveReferenceOutsideSubject && layoutLocks.subjectBoundingBox,
),
subjectBoundingBox: layoutLocks.subjectBoundingBox,
textBoundingBoxes: params.preserveReferenceText
? layoutLocks.textBoundingBoxes
: [],
});
if (shouldUseSafeReferenceFallback) {
chatApiLogger.warn(
"[Chat API] Replacement layout lock analysis unavailable; preserved the reference image instead of returning a layout-damaged result",
{
preserveReferenceOutsideSubject: params.preserveReferenceOutsideSubject,
preserveReferenceText: params.preserveReferenceText,
hasSubjectBoundingBox: Boolean(layoutLocks.subjectBoundingBox),
layoutLockAnalyzed: layoutLocks.analyzed,
},
);
}
chatApiLogger.info("[Chat API] Replacement reference layout locks applied:", {
preserveReferenceOutsideSubject: Boolean(
params.preserveReferenceOutsideSubject && layoutLocks.subjectBoundingBox,
),
preservedTextRegionCount: params.preserveReferenceText
? layoutLocks.textBoundingBoxes.length
: 0,
usedFallbackAnalysis: needsLayoutLockFallback,
usedSafeReferenceFallback: shouldUseSafeReferenceFallback,
});
}
const thumbnailBuffer = await createAgentImageThumbnailBuffer(normalizedBuffer);
const uniqueSuffix = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
@ -399,6 +504,12 @@ export type ReplacementScenePlan = {
generationPrompt: string;
selectedProductImageIndexes: number[];
productSelectionReason: string;
productIdentitySummary?: string;
productStructureSummary?: string;
productStructuralRisks?: string;
subjectBoundingBox?: NormalizedReplacementBox | null;
textBoundingBoxes?: NormalizedReplacementBox[];
layoutLockAnalyzed?: boolean;
analysisModelName?: string | null;
analysisSource: "local_moonshot" | "configured_vision" | "heuristic";
};
@ -439,6 +550,188 @@ export type ReplacementScenePlanningRuntime =
const QWEN_VL_MODEL_MATCHER = /(qwen|千问|通义).*(vl|vision)|qwen3-vl|qwen-vl/i;
const replacementProductIdentityProfileCache = new Map<
string,
{
expiresAt: number;
promise: Promise<ReplacementProductIdentityProfile | null>;
}
>();
async function resolveReplacementProductIdentityProfile(params: {
productImageUrls: string[];
planningRuntime: Exclude<ReplacementScenePlanningRuntime, null>;
}) {
const productImageUrls = Array.from(
new Set(
params.productImageUrls
.map((url) => String(url || "").trim())
.filter((url) => /^https?:\/\//i.test(url)),
),
).slice(0, 8);
if (productImageUrls.length === 0) return null;
const cacheKey = [
params.planningRuntime.kind,
params.planningRuntime.modelName,
...productImageUrls,
].join("|");
const now = Date.now();
const cached = replacementProductIdentityProfileCache.get(cacheKey);
if (cached && cached.expiresAt > now) {
return cached.promise;
}
if (replacementProductIdentityProfileCache.size >= 100) {
const oldestKey = replacementProductIdentityProfileCache.keys().next().value;
if (oldestKey) replacementProductIdentityProfileCache.delete(oldestKey);
}
const promise = (async () => {
try {
const { inlineDataUrls } = await inlineVisionImages(productImageUrls, {
maxImages: productImageUrls.length,
maxSide: 800,
timeoutMs: 12_000,
retries: 2,
loggerPrefix: "[Chat API] Replacement product identity",
trimTransparentPadding: true,
});
const effectiveImages =
inlineDataUrls.length === productImageUrls.length
? inlineDataUrls
: productImageUrls;
const { text } = await generateText({
model: params.planningRuntime.model,
messages: [
{
role: "user",
content: [
{
type: "text",
text: buildReplacementProductIdentityAnalysisPrompt(
productImageUrls.length,
),
},
...effectiveImages.map((image) => ({
type: "image" as const,
image,
})),
],
},
],
system:
"你是高精度商品结构分析器。多张图片属于同一商品;请跨视角还原唯一结构身份,并明确每个视角最清楚的构件。",
temperature: params.planningRuntime.kind === "local_moonshot" ? 1 : 0.1,
maxTokens: 1800,
abortSignal: AbortSignal.timeout(
params.planningRuntime.kind === "local_moonshot" ? 90_000 : 60_000,
),
} as any);
const profile = parseReplacementProductIdentityProfile(
text,
productImageUrls.length,
);
if (!profile) {
throw new Error("商品结构画像缺少必需字段");
}
return profile;
} catch (error) {
chatApiLogger.warn("[Chat API] Replacement product identity analysis failed:", {
productImageCount: productImageUrls.length,
modelName: params.planningRuntime.modelName,
error: error instanceof Error ? error.message : "unknown",
});
return null;
}
})();
replacementProductIdentityProfileCache.set(cacheKey, {
expiresAt: now + 30 * 60 * 1000,
promise,
});
return promise;
}
const replacementLayoutLockCache = new Map<
string,
{
expiresAt: number;
promise: Promise<ReturnType<typeof parseReplacementLayoutLockRegions>>;
}
>();
async function resolveReplacementLayoutLockRegions(params: {
referenceImageUrl: string;
referenceBuffer: Buffer;
}) {
const now = Date.now();
const cached = replacementLayoutLockCache.get(params.referenceImageUrl);
if (cached && cached.expiresAt > now) {
return cached.promise;
}
if (replacementLayoutLockCache.size >= 200) {
const oldestKey = replacementLayoutLockCache.keys().next().value;
if (oldestKey) {
replacementLayoutLockCache.delete(oldestKey);
}
}
const promise = (async () => {
try {
const sharp = (await import("sharp")).default;
const inlineBuffer = await sharp(params.referenceBuffer, { failOn: "none" })
.rotate()
.resize(1600, 1600, {
fit: "inside",
withoutEnlargement: true,
})
.flatten({ background: "#ffffff" })
.jpeg({ quality: 88 })
.toBuffer();
const { callQwen3Vl } = await import("@/lib/ai/qwen3-vl");
const result = await callQwen3Vl({
imageUrl: `data:image/jpeg;base64,${inlineBuffer.toString("base64")}`,
prompt: [
"你是电商详情图的布局区域定位器。",
"只分析当前图像,返回严格 JSON,不要解释。",
"所有坐标使用 0到1000 的归一化坐标,格式必须是 [x1,y1,x2,y2],绝对不是 x/y/宽/高。",
"subjectBoundingBox:只框出图中当前旧商品主体的可视外轮廓区域,包含完整商品和它的自然阴影,不要包含文字;没有商品则为 null。",
"textBoundingBoxes:框出所有可见文案块,包括中英文标题、卖点、参数、标签、数字、品牌字样、按钮文字和紧邻的图标;没有则返回空数组。",
'{"subjectBoundingBox":[0,0,1000,1000],"textBoundingBoxes":[[0,0,1000,1000]]}',
].join("\n"),
temperature: 0,
maxTokens: 1600,
timeoutMs: 60_000,
retries: 1,
});
const parsedLocks = parseReplacementLayoutLockRegions(result.text);
if (!parsedLocks.analyzed) {
throw new Error("布局区域定位结果缺少必需字段");
}
return parsedLocks;
} catch (error) {
replacementLayoutLockCache.delete(params.referenceImageUrl);
chatApiLogger.warn("[Chat API] Replacement layout lock fallback analysis failed:", {
referenceImageUrl: params.referenceImageUrl,
error: error instanceof Error ? error.message : "unknown",
});
return {
subjectBoundingBox: null,
textBoundingBoxes: [],
analyzed: false,
};
}
})();
replacementLayoutLockCache.set(params.referenceImageUrl, {
expiresAt: now + 30 * 60 * 1000,
promise,
});
return promise;
}
export function normalizeDetailReplacementStrategy(
value: unknown,
): AgentDetailReplacementStrategy {
@ -1441,7 +1734,27 @@ export function parseReplacementScenePlan(
visualProfile: params.visualProfile,
});
const productImageSelection = buildDefaultProductImageSelection(params.productImageCount);
const defaultProductImageSelection = buildDefaultProductImageSelection(
params.productImageCount,
);
const selectedProductImageIndexes = normalizeReplacementProductImageIndexes(
parsed.selectedProductImageIndexes ??
parsed.selectedProductIndexes ??
parsed.productImageIndexes,
params.productImageCount,
);
const productImageSelection =
selectedProductImageIndexes.length > 0
? {
selectedProductImageIndexes,
productSelectionReason:
truncateReplacementAnalysisText(
parsed.productSelectionReason ?? parsed.selectionReason,
240,
) || "按当前详情底板机位选择最匹配的产品视角。",
}
: defaultProductImageSelection;
const layoutLocks = parseReplacementLayoutLockRegions(parsed);
let scenePlan: ReplacementScenePlan = {
pageKind,
generationRoute,
@ -1486,6 +1799,9 @@ export function parseReplacementScenePlan(
generationPrompt,
selectedProductImageIndexes: productImageSelection.selectedProductImageIndexes,
productSelectionReason: productImageSelection.productSelectionReason,
subjectBoundingBox: layoutLocks.subjectBoundingBox,
textBoundingBoxes: layoutLocks.textBoundingBoxes,
layoutLockAnalyzed: layoutLocks.analyzed,
analysisModelName: null,
analysisSource: "heuristic",
};
@ -1844,6 +2160,27 @@ export async function resolveDirectReplacementModelRef(params: {
finalModelId?: string | null;
models?: Awaited<ReturnType<typeof getEnabledAiModelsWithProvider>>;
}) {
const models = params.models || (await getEnabledAiModelsWithProvider(false));
const imageModels = models.filter(
({ model, provider }) =>
model.supportsImageGen &&
!model.isSubModel &&
!isVideoProviderType(provider?.type, model),
);
const configuredModelSetting = await getSystemSetting({
key: DETAIL_REPLACEMENT_IMAGE_MODEL_CONFIG_KEY,
}).catch(() => null);
const configuredModelId = normalizeDetailReplacementImageModelConfig(
configuredModelSetting?.value,
).modelId;
const configuredModel = imageModels.find(
({ model }) =>
model.id === configuredModelId || model.modelId === configuredModelId,
);
if (configuredModel) {
return configuredModel.model.id;
}
const explicitModel = String(params.selectedChatModel || "").trim();
if (explicitModel && explicitModel !== "auto") {
return explicitModel;
@ -1853,13 +2190,6 @@ export async function resolveDirectReplacementModelRef(params: {
return routedModel;
}
const models = params.models || (await getEnabledAiModelsWithProvider(false));
const imageModels = models.filter(
({ model, provider }) =>
model.supportsImageGen &&
!model.isSubModel &&
!isVideoProviderType(provider?.type, model),
);
const preferred =
imageModels.find(({ model }) => model.isDefault) ||
imageModels.find(({ provider }) => String(provider?.type || "").includes("image")) ||
@ -2121,6 +2451,16 @@ export async function analyzeReplacementReferenceImage(params: {
export async function resolveReplacementScenePlanningRuntime(params: {
models?: Awaited<ReturnType<typeof getEnabledAiModelsWithProvider>>;
}): Promise<ReplacementScenePlanningRuntime> {
try {
const dedicatedVisionRuntime = await resolveDetailReplacementVisionRuntime();
if (dedicatedVisionRuntime) return dedicatedVisionRuntime;
} catch (error) {
chatApiLogger.warn(
"[Chat API] Failed to resolve dedicated detail replacement vision model:",
error,
);
}
try {
const siteSettings = await getSiteSettings().catch(() => null);
const localDetailAgentResolved = await resolveDetailAgentLocalMoonshotModel({
@ -2191,6 +2531,13 @@ export async function analyzeReplacementExecutionPlan(params: {
const referenceVisualProfile = isDetailReplacement
? await resolveReplacementReferenceVisualProfile(params.referenceImageUrl)
: null;
const productIdentityProfile =
params.planningRuntime && params.productImageUrls.length > 0
? await resolveReplacementProductIdentityProfile({
productImageUrls: params.productImageUrls,
planningRuntime: params.planningRuntime,
})
: null;
if (params.planningRuntime) {
try {
const replacementScenePlanningTimeoutMs =
@ -2221,7 +2568,10 @@ export async function analyzeReplacementExecutionPlan(params: {
? "你是电商详情参考图背景替换流程中的“版式与背景区域规划器”。"
: "你是电商详情替换流程中的“场景理解规划器”。",
`当前要分析的参考图:${params.referenceLabel}`,
"你现在只分析这一张参考图本身,不分析产品图,不分析模特图。",
"当前视觉输入只包含这一张详情参考图;产品图已经单独做过跨视角结构分析,请根据下面的产品结构画像选择最匹配当前机位的产品视角,不要臆测产品结构。",
productIdentityProfile
? `【产品结构画像】\n${formatReplacementProductIdentityProfile(productIdentityProfile)}`
: "产品结构画像暂不可用:仍需按当前参考图机位选择产品图,但不得把旧产品的腿型、支架或构件拓扑当成新产品结构。",
referenceVisualProfile
? `像素初筛:nearWhite=${referenceVisualProfile.nearWhitePixelRatio.toFixed(2)}, lightLowSaturation=${referenceVisualProfile.lightLowSaturationPixelRatio.toFixed(2)}, topLight=${referenceVisualProfile.topLightLowSaturationPixelRatio.toFixed(2)}, borderLight=${referenceVisualProfile.borderLightLowSaturationPixelRatio.toFixed(2)}, dark=${referenceVisualProfile.darkPixelRatio.toFixed(2)}, likelyWhiteLayout=${referenceVisualProfile.likelyWhiteLayout ? "true" : "false"}, likelyWhiteScene=${referenceVisualProfile.likelyWhiteScene ? "true" : "false"}。${referenceVisualProfile.reason}`
: "",
@ -2231,7 +2581,7 @@ export async function analyzeReplacementExecutionPlan(params: {
!referenceVisualProfile?.likelyWhiteLayout && referenceVisualProfile?.likelyWhiteScene
? "像素初筛已经命中白/浅色低饱和真实场景:这不是白底说明页。若画面存在真实空间、人物或产品照片,promptPackPolicy 应为 soft 或 allow,必须使用前置工作流词包重建图片框内产品、空间、软装和灯光,使差异清晰可见;但最终仍保持白/浅色低饱和基调,不要改造成深色厨房、黑墙、复杂吊灯或强软装。"
: "",
"产品图和模特图会在后续生图阶段单独提供给生图模型使用,你这里不要尝试理解它们的具体外观,也不要要求必须使用所有输入图。",
"产品图和模特图会在后续生图阶段单独提供给生图模型。你必须依据产品结构画像输出 selectedProductImageIndexes,选择最多两张最能匹配当前参考图机位并共同说明完整结构的产品图;不得固定选择前两张,也不要要求强行使用全部产品图。",
params.modelImageUrl && !isProductOnlyDetailReplacement
? "你只需要根据当前参考图判断:这一屏是否存在真实的人物展示需求;如果没有人物需求,useModelImage 必须为 false。"
: "本轮没有模特图,useModelImage 必须为 false。",
@ -2278,12 +2628,18 @@ export async function analyzeReplacementExecutionPlan(params: {
isDetailReplacement
? "5. 判断 textPolicy:画面中存在营销标题、卖点、编号、参数、标签且与图匹配时必须 lock 或 preserve,避免后续生图改字、混色或重排。"
: "",
isDetailReplacement
? "5a. 必须输出 subjectBoundingBox 和 textBoundingBoxes。所有坐标为 0到1000 归一化坐标 [x1,y1,x2,y2],不是 x/y/宽/高。subjectBoundingBox 紧密框住旧商品主体及自然阴影,不含文字,无商品时为 null;textBoundingBoxes 覆盖所有中英文文案、标题、卖点、参数、标签、数字、品牌字样、按钮文字及紧邻图标,无文字时为 []。"
: "",
isDetailReplacement
? "6. 判断 peoplePolicy:有人物/多人/自拍/聚会/围坐/聚餐/模特互动/拿手机/用餐/试坐/承重动作时必须 preserve_relation;如果文字明确写了“多人围坐、聚餐、合照、用餐、单侧承重、横向受力测试、拿手机”等动作,generationPrompt 必须把这个动作写成硬约束;如果只提供 1 张模特图但参考图多人,仍要保留参考图人数和关系,额外人物按参考图生成相近人群。"
: "",
isDetailReplacement
? "7. 提炼 sceneSummary、cameraSummary、placementSummary、preserveSummary、styleSummary、avoidSummary;cameraSummary 必须写出产品主体的视角特征,例如正侧视/三分之四/俯视/平视、倾斜方向、可见面数量、主体朝向和透视消失方向。"
: "",
isDetailReplacement
? "7a. 输出 selectedProductImageIndexes:根据当前详情图机位,从产品结构画像的图片编号中选择最多两张最相关的产品图;第一张应最接近当前机位,第二张补足腿、支架、横梁、侧框或其他容易变形的结构。输出 productSelectionReason 说明选择依据。"
: "",
isProductOnlyDetailReplacement
? "4. 输出 generationPrompt:这是一段给后续生图模型的本屏生成指令,要求只替换产品主体,保留其余背景、文案、排版、装饰和光影完全不变。"
: isDetailReplacement
@ -2309,7 +2665,7 @@ export async function analyzeReplacementExecutionPlan(params: {
: "",
"只返回严格 JSON,不要任何额外解释。",
`JSON 格式:
{"shouldReplace":true,"containsFurnitureProduct":true,"reason":"不超过30字","pageKind":"prompt_pack_scene|white_scene|white_layout|copy_locked|people_scene|action_poster|dimension_spec|material_detail|unknown","generationRoute":"${isDetailReplacement ? "prompt_pack_scene|layout_preserve_replace|copy_locked_regenerate|people_relation_locked" : "prompt_pack_scene|layout_preserve_replace|copy_locked_regenerate|people_relation_locked|direct_passthrough"}","promptPackPolicy":"allow|soft|forbid","textPolicy":"free|preserve|lock","peoplePolicy":"none|preserve_relation|require_user_model","whiteBackgroundLikely":false,"copyImageCoupled":false,"containsPeople":false,"containsMultiplePeople":false,"routeReason":"不超过50字","useModelImage":false,"modelImageReason":"不超过40字","sceneSummary":"不超过80字","cameraSummary":"不超过60字","placementSummary":"不超过60字","preserveSummary":"不超过80字","styleSummary":"不超过80字","avoidSummary":"不超过80字","generationPrompt":"不超过300字"}`,
{"shouldReplace":true,"containsFurnitureProduct":true,"reason":"不超过30字","pageKind":"prompt_pack_scene|white_scene|white_layout|copy_locked|people_scene|action_poster|dimension_spec|material_detail|unknown","generationRoute":"${isDetailReplacement ? "prompt_pack_scene|layout_preserve_replace|copy_locked_regenerate|people_relation_locked" : "prompt_pack_scene|layout_preserve_replace|copy_locked_regenerate|people_relation_locked|direct_passthrough"}","promptPackPolicy":"allow|soft|forbid","textPolicy":"free|preserve|lock","peoplePolicy":"none|preserve_relation|require_user_model","whiteBackgroundLikely":false,"copyImageCoupled":false,"containsPeople":false,"containsMultiplePeople":false,"subjectBoundingBox":[0,0,1000,1000],"textBoundingBoxes":[[0,0,1000,1000]],"selectedProductImageIndexes":[1,2],"productSelectionReason":"不超过80字","routeReason":"不超过50字","useModelImage":false,"modelImageReason":"不超过40字","sceneSummary":"不超过80字","cameraSummary":"不超过60字","placementSummary":"不超过60字","preserveSummary":"不超过80字","styleSummary":"不超过80字","avoidSummary":"不超过80字","generationPrompt":"不超过300字"}`,
]
.filter(Boolean)
.join("\n");
@ -2334,7 +2690,7 @@ export async function analyzeReplacementExecutionPlan(params: {
system:
"你是高精度商品详情图场景规划器。你只分析当前参考图本身,负责理解这一屏的场景、机位、主体位置和人物需求,不要臆测或依赖其他输入图片。",
temperature: params.planningRuntime.kind === "local_moonshot" ? 1 : 0.1,
maxTokens: 1200,
maxTokens: 1600,
abortSignal: AbortSignal.timeout(replacementScenePlanningTimeoutMs),
} as any);
@ -2368,6 +2724,9 @@ export async function analyzeReplacementExecutionPlan(params: {
: "configured_vision",
scenePlan: {
...normalizedPlan.scenePlan,
productIdentitySummary: productIdentityProfile?.identitySummary || "",
productStructureSummary: productIdentityProfile?.structureSummary || "",
productStructuralRisks: productIdentityProfile?.structuralRisks || "",
analysisModelName: params.planningRuntime.modelName,
analysisSource:
params.planningRuntime.kind === "local_moonshot"
@ -2413,7 +2772,12 @@ export async function analyzeReplacementExecutionPlan(params: {
);
return {
...normalizedFallbackPlan.decision,
scenePlan: normalizedFallbackPlan.scenePlan,
scenePlan: {
...normalizedFallbackPlan.scenePlan,
productIdentitySummary: productIdentityProfile?.identitySummary || "",
productStructureSummary: productIdentityProfile?.structureSummary || "",
productStructuralRisks: productIdentityProfile?.structuralRisks || "",
},
};
}
@ -2537,11 +2901,17 @@ export function buildAgentReplacementPrompt(params: {
]
: [];
const productAngleLockLines = [
"【产品角度锁定规则】最终商品/主体的拍摄角度以最后一张参考图为准,而不是以前面的产品输入图为准;前面的产品图只提供款式、材质、颜色、结构和细节。",
"必须逐项对齐最后一张参考图中的产品姿态:朝向、正侧视/三分之四/俯视/平视关系、桌面或面板倾斜方向、可见侧面数量、支架/腿部/把手/转轴位置、主体外轮廓、占比、重心和透视消失方向。",
"【产品角度锁定规则】最终商品/主体的拍摄角度、朝向、占比、重心和透视消失方向以最后一张参考图为准;商品身份、外轮廓、腿型、支架拓扑、构件数量和连接方式以前面的产品输入图为准。",
"必须对齐最后一张参考图的正侧视/三分之四/俯视/平视关系、桌面或面板倾斜方向和可见侧面数量,但不得照搬旧商品的腿部、横梁、侧框、把手、转轴或内部构件设计。",
"如果最后一张参考图的产品是侧视或低视角,不允许改成俯视三分之四;如果是俯视三分之四,不允许改成正侧视;不允许镜像、旋转、翻转或把产品重画成另一个角度。",
"若参考图含功能标签、箭头、尺寸线、编号或引导线,它们指向的实际结构部位必须保持一致,不得因产品转角改变而让标签指向错误位置。",
];
const productStructureLockLines = buildReplacementProductStructureLockLines({
productImageCount: params.productImageCount,
identitySummary: params.scenePlan?.productIdentitySummary,
structureSummary: params.scenePlan?.productStructureSummary,
structuralRisks: params.scenePlan?.productStructuralRisks,
});
const normalizedBasePrompt = String(params.basePrompt || "").trim();
const promptPackInstructionLines =
params.modeType === "detail_replacement" &&
@ -2641,6 +3011,7 @@ export function buildAgentReplacementPrompt(params: {
...promptPackInstructionLines,
...scenePlanLines,
...marketplaceBrandRemovalLines,
...productStructureLockLines,
...productAngleLockLines,
"你现在做的是‘仅替换产品’,不是重建场景,也不是重新设计整张详情图。",
"忽略用户补充提示词里的场景风格、氛围、空间改造要求;这一次只服务于产品主体替换。",
@ -2670,6 +3041,7 @@ export function buildAgentReplacementPrompt(params: {
...promptPackInstructionLines,
...scenePlanLines,
...marketplaceBrandRemovalLines,
...productStructureLockLines,
...productAngleLockLines,
"你现在做的是“详情参考图背景/场景图替换”,不是重新设计整张详情图。",
"【版式锁层规则】把最后一张参考图当成 PSD 模板:文字、标题、编号、卖点标签、按钮、图标、白底/浅色底、留白、分割线、原有圆角照片区域和原有裁切范围都属于锁定版式层,不允许移动、缩放、重排或重新设计;不要把这些版式边界画成新的白色线框、描边或卡片。",
@ -2785,6 +3157,7 @@ export function buildAgentReplacementPrompt(params: {
"【图片顺序规则】",
...imageOrderLines,
...scenePlanLines,
...productStructureLockLines,
...productAngleLockLines,
params.pairedProductIndex !== undefined
? `请优先使用第 ${params.pairedProductIndex + 1} 张输入商品图作为本张参考图的一对一替换主体。`

View File

@ -1,6 +1,6 @@
import { createLogger } from "@/lib/observability/logger";
import { NextResponse } from "next/server";
import { getRequestErrorMessage } from "@/lib/request-error";
import { getRequestErrorMessage, isProviderBalanceError, isRateLimitError } from "@/lib/request-error";
import { generateUUID } from "@/lib/utils";
const chatApiLogger = createLogger("chat.errors");
@ -20,6 +20,14 @@ export function getChatStreamErrorMessage(error: unknown): string {
error instanceof Error ? String(error.message || "").trim() : String(error || "").trim();
const normalized = rawMessage.toLowerCase();
if (isProviderBalanceError(error)) {
return "当前模型服务商线路余额或额度不足,请切换模型或联系管理员;与您的积分余额无关。";
}
if (isRateLimitError(error)) {
return "当前模型服务请求过于频繁,请稍后重试或切换模型。";
}
if (
normalized.includes("token quota is not enough") ||
normalized.includes("pre_consume_token_quota_failed") ||
@ -47,6 +55,16 @@ function resolveChatApiFailure(error: unknown) {
const detailMessage = getRequestErrorMessage(error);
const normalized = String(detailMessage || "").toLowerCase();
if (isProviderBalanceError(error)) {
return {
code: "server_error:chat",
message: "模型服务商线路余额或额度不足,请切换模型或联系管理员。",
status: 503,
category: "provider_balance",
detailMessage,
};
}
if (
normalized.includes("payload too large") ||
normalized.includes("request entity too large") ||

View File

@ -4,6 +4,8 @@ import { getLanguageModelById } from "@/lib/ai/dynamic-provider";
import { modelLacksReferenceImageInputSupport } from "@/lib/ai/reference-image-capability";
import type { ChatMessage } from "@/lib/types";
import { convertMessagesWithImages } from "./image-media";
import { isProviderBalanceError } from "@/lib/request-error";
import { isImageFallbackModelBalanceCoolingDown } from "@/lib/ai/runtime/image-fallback-balance-cooldown";
const chatApiLogger = createLogger("chat.imageFailover");
@ -57,7 +59,21 @@ export function resolveImageModelRuntimeId(
export function normalizeFailoverErrorMessage(error: unknown): string {
if (!error) return "";
if (typeof error === "string") return error;
if (error instanceof Error) return error.message || String(error);
if (error instanceof Error) {
const record = error as Error & {
status?: unknown;
statusCode?: unknown;
httpStatus?: unknown;
response?: { status?: unknown };
};
const status =
record.status ??
record.statusCode ??
record.httpStatus ??
record.response?.status;
const message = error.message || String(error);
return status ? `[HTTP ${String(status)}] ${message}` : message;
}
try {
return JSON.stringify(error);
} catch {
@ -68,6 +84,7 @@ export function normalizeFailoverErrorMessage(error: unknown): string {
export function shouldCooldownImageFallbackModel(errorMessage: string): boolean {
const msg = errorMessage.toLowerCase();
return (
isProviderBalanceError(errorMessage) ||
msg.includes("model not found") ||
msg.includes("model_not_found") ||
msg.includes("no available channel for model") ||
@ -76,6 +93,13 @@ export function shouldCooldownImageFallbackModel(errorMessage: string): boolean
msg.includes("api key is missing") ||
msg.includes("invalid api key") ||
msg.includes("unauthorized") ||
/(?:^|\D)401(?:\D|$)/.test(msg) ||
msg === "forbidden" ||
msg.includes("access forbidden") ||
msg.includes("access denied") ||
msg.includes("permission denied") ||
/(?:^|\D)403(?:\D|$)/.test(msg) ||
msg.includes("vip98k task failed") ||
msg.includes("cannot convert argument to a bytestring") ||
msg.includes("model maintenance") ||
msg.includes("maintenance") ||
@ -302,9 +326,9 @@ export function buildImageFallbackCandidates(
if (!sameProvider && !IMAGE_FAILOVER_PROVIDER_ALLOWLIST.has(providerType)) {
return false;
}
if (isImageFallbackModelCoolingDown(item.model.id)) {
if (isImageFallbackModelCoolingDown(item.model.id) || isImageFallbackModelBalanceCoolingDown(item.model)) {
const cooldown = imageFailoverModelCooldown.get(item.model.id);
chatApiLogger.info(`[Chat API] ⏭ Skip configured image fallback on cooldown: ${item.model.displayName} (${providerType}) - ${cooldown?.reason || "cooldown"}`);
chatApiLogger.info(`[Chat API] ⏭ Skip configured image fallback on cooldown: ${item.model.displayName} (${providerType}) - ${cooldown?.reason || "provider balance cooldown"}`);
return false;
}
return true;
@ -464,9 +488,9 @@ export function getImageCrossProviderCandidates(
}
if (m.model.id === options.currentModelId) return false;
if (options.currentProviderId && m.provider.id === options.currentProviderId) return false;
if (isImageFallbackModelCoolingDown(m.model.id)) {
if (isImageFallbackModelCoolingDown(m.model.id) || isImageFallbackModelBalanceCoolingDown(m.model)) {
const cooldown = imageFailoverModelCooldown.get(m.model.id);
chatApiLogger.info(`[Chat API] ⏭ Skip image failover model on cooldown: ${m.model.displayName} (${m.provider.type}) - ${cooldown?.reason || "cooldown"}`);
chatApiLogger.info(`[Chat API] ⏭ Skip image failover model on cooldown: ${m.model.displayName} (${m.provider.type}) - ${cooldown?.reason || "provider balance cooldown"}`);
return false;
}
return true;

View File

@ -179,6 +179,7 @@ import { ChatSDKError } from "@/lib/errors";
import {
getFriendlyRateLimitMessage,
getRequestErrorMessage,
getUserFacingImageGenerationErrorMessage,
isRateLimitError,
} from "@/lib/request-error";
import {
@ -197,6 +198,7 @@ import {
recordAiModelHealthFailure,
recordAiModelHealthSuccess,
shouldFailoverAiModelError,
shouldFailoverVip98kRequestRejection,
} from "@/lib/ai/runtime/model-health";
import type { FalImageEditTaskMetadata } from "@/lib/ai/runtime/fal-image-edit-task";
import { saveAssistantTextOnlyMessage } from "@/lib/ai/runtime/assistant-message-persistence";
@ -363,6 +365,10 @@ import {
resolveReplacementReferenceGeometry,
resolveReplacementScenePlanningRuntime,
} from "./_lib/detail-replacement";
import {
selectDetailReplacementGenerationInputs,
shouldIncludeDetailReplacementModelImage,
} from "@/lib/ai/detail-replacement-inputs";
import {
normalizeAgentReportForDisplay,
sanitizeAgentOutputSettingsSection,
@ -417,8 +423,6 @@ const chatApiLogger = createLogger("chat.route");
const INTERNAL_IMAGE_GENERATION_FAILURE_PATTERN =
/生成图片失败|图片生成失败|未找到图片数据|未解析到图片|模型未返回生成图片|图片数据不可用|未返回可用图片/i;
const DETAIL_REPLACEMENT_MODEL_TEXT_HINT_PATTERN =
/自拍|合照|聚会|聚餐|围坐|用餐|试坐|承重|受力|拿手机|手持|人物场景|人物关系|真人演示|人物展示|人像展示|固定模特|用户模特|same\s+model|fixed\s+model|person|people|human/i;
function assertGeneratedImageMarkdownHasImage(markdown: string, modelLabel: string) {
const normalizedMarkdown = String(markdown || "").trim();
@ -435,41 +439,13 @@ function assertGeneratedImageMarkdownHasImage(markdown: string, modelLabel: stri
function getPublicImageGenerationErrorText(error: unknown) {
const message = error instanceof Error ? error.message : String(error || "");
if (!message || INTERNAL_IMAGE_GENERATION_FAILURE_PATTERN.test(message)) {
const publicMessage = getUserFacingImageGenerationErrorMessage(error);
if (!message || (publicMessage === message && INTERNAL_IMAGE_GENERATION_FAILURE_PATTERN.test(message))) {
return "图片生成失败,请稍后重试";
}
return message;
return publicMessage;
}
function replacementScenePlanShouldIncludeModelImage(scenePlan: any) {
if (!scenePlan) return false;
if (scenePlan.useModelImage === true) return true;
if (
scenePlan.peoplePolicy &&
scenePlan.peoplePolicy !== "none"
) {
return true;
}
if (scenePlan.generationRoute === "people_relation_locked") return true;
if (scenePlan.pageKind === "people_scene" || scenePlan.pageKind === "action_poster") {
return true;
}
const text = [
scenePlan.modelImageReason,
scenePlan.routeReason,
scenePlan.sceneSummary,
scenePlan.cameraSummary,
scenePlan.placementSummary,
scenePlan.preserveSummary,
scenePlan.styleSummary,
scenePlan.avoidSummary,
scenePlan.generationPrompt,
]
.filter(Boolean)
.join(" ");
return DETAIL_REPLACEMENT_MODEL_TEXT_HINT_PATTERN.test(text);
}
const AGENT_REPLACEMENT_PARALLEL_WORKER_COUNT = Math.max(
1,
@ -5723,12 +5699,15 @@ export async function POST(request: Request) {
sourceImageUrls: string[];
streamLabel: string;
promptForPersistence?: string | null;
isFallback?: boolean;
}) => {
const persistedImageSize =
String(params.imageSizeLabel || "").trim() || resolvedImageSize;
const assistantText = prependImageModelBadge(
[
`<!-- AGENT_REPLACEMENT_CARD_ID:${String(params.reference.id || "").trim()} -->`,
`<!-- AGENT_REPLACEMENT_RUN_ID:${message.id} -->`,
`<!-- AGENT_REPLACEMENT_STATUS:${params.isFallback ? "failed" : "completed"} -->`,
`### ${params.reference.label}`,
String(params.bodyText || "").trim(),
buildAgentToolImagesMarkdown([
@ -5764,26 +5743,28 @@ export async function POST(request: Request) {
],
});
await saveGeneratedImages(
assistantText,
session.user.id,
id,
assistantMessageId,
params.promptForPersistence || baseReplacementPrompt,
resolvedModelRef,
{
sourceAssetIds,
sourceImageUrls: params.sourceImageUrls,
relationType: "edited_from",
sourceType: "detail_replacement",
projectId: contextPayload?.projectId || null,
userPrompt: requestTextContent,
optimizedPrompt: params.promptForPersistence || baseReplacementPrompt,
requestedImageCount: 1,
aspectRatio: resolvedAspectRatio,
imageSize: persistedImageSize,
},
);
if (!params.isFallback) {
await saveGeneratedImages(
assistantText,
session.user.id,
id,
assistantMessageId,
params.promptForPersistence || baseReplacementPrompt,
resolvedModelRef,
{
sourceAssetIds,
sourceImageUrls: params.sourceImageUrls,
relationType: "edited_from",
sourceType: "detail_replacement",
projectId: contextPayload?.projectId || null,
userPrompt: requestTextContent,
optimizedPrompt: params.promptForPersistence || baseReplacementPrompt,
requestedImageCount: 1,
aspectRatio: resolvedAspectRatio,
imageSize: persistedImageSize,
},
);
}
} catch (persistError) {
chatApiLogger.warn("[Chat API] Failed to persist direct detail replacement result; streaming response anyway:", {
chatId: id,
@ -5933,20 +5914,18 @@ export async function POST(request: Request) {
const shouldIncludeModelImage =
detailReplacementStrategy !== "product_only" &&
Boolean(modelInputImageUrl) &&
(replacementScenePlanShouldIncludeModelImage(
(shouldIncludeDetailReplacementModelImage(
replacementExecutionPlan.scenePlan,
) ||
selectedProductContextImageUrls.length === 0);
const sourceImageUrls = Array.from(
new Set([
...selectedProductContextImageUrls,
...(shouldIncludeModelImage && modelInputImageUrl
? [modelInputImageUrl]
: []),
...rerunReferenceImageUrls,
reference.imageUrl,
]),
);
const generationInputs = selectDetailReplacementGenerationInputs({
productImageUrls: selectedProductContextImageUrls,
modelImageUrl: modelInputImageUrl,
includeModelImage: shouldIncludeModelImage,
rerunReferenceImageUrls,
referenceImageUrl: reference.imageUrl,
});
const sourceImageUrls = generationInputs.imageUrls;
chatApiLogger.info("[Chat API] Direct detail replacement image generation started:", {
referenceIndex: referenceIndex + 1,
totalReferences: replacementReferences.length,
@ -5954,10 +5933,10 @@ export async function POST(request: Request) {
pageKind: replacementExecutionPlan.scenePlan.pageKind,
generationRoute: replacementExecutionPlan.scenePlan.generationRoute,
packPolicy: replacementExecutionPlan.scenePlan.promptPackPolicy,
selectedProductImages: selectedProductContextImageUrls.length,
selectedProductImages: generationInputs.productImageUrls.length,
sourceProductImages: productInputImageUrls.length,
includeModelImage: shouldIncludeModelImage,
rerunReferenceImages: rerunReferenceImageUrls.length,
includeModelImage: Boolean(generationInputs.modelImageUrl),
rerunReferenceImages: generationInputs.rerunImageUrls.length,
sourceImageCount: sourceImageUrls.length,
whiteLayout: replacementExecutionPlan.scenePlan.whiteBackgroundLikely,
copyCoupled: replacementExecutionPlan.scenePlan.copyImageCoupled,
@ -5972,10 +5951,10 @@ export async function POST(request: Request) {
referenceLabel: reference.label,
referenceIndex,
totalReferences: replacementReferences.length,
hasModelImage: shouldIncludeModelImage,
productImageCount: selectedProductContextImageUrls.length,
hasModelImage: Boolean(generationInputs.modelImageUrl),
productImageCount: generationInputs.productImageUrls.length,
originalProductImageCount: productInputImageUrls.length,
rerunReferenceCount: rerunReferenceImageUrls.length,
rerunReferenceCount: generationInputs.rerunImageUrls.length,
templateReferenceCount: 0,
scenePlan: replacementExecutionPlan.scenePlan,
referenceGeometry,
@ -6007,6 +5986,10 @@ export async function POST(request: Request) {
imageSize: resolvedImageSize,
imageCount: 1,
referenceImages: sourceImageUrls,
promptExtend: false,
providerImageParams: {
inputImageRoleByUrl: generationInputs.inputImageRoleByUrl,
},
debugLabel: "agent-detail-replacement",
debugMeta: {
referenceLabel: reference.label,
@ -6054,6 +6037,9 @@ export async function POST(request: Request) {
referenceImageUrl: reference.imageUrl,
referenceGeometry,
scenePlan: replacementExecutionPlan.scenePlan,
preserveReferenceOutsideSubject:
detailReplacementStrategy === "product_only",
preserveReferenceText: true,
uploadPrefix: "agent-detail-replacement",
});
finalImage = {
@ -6136,8 +6122,8 @@ export async function POST(request: Request) {
modelDisplayName: null,
sourceImageUrls: referenceSourceImageUrls,
streamLabel: "agent-mode:image:detail-replacement:item-fallback",
isFallback: true,
});
successCount += 1;
}
} catch (replacementPlanError) {
chatApiLogger.error("[Chat API] Direct detail replacement item failed before generation:", {
@ -6160,8 +6146,8 @@ export async function POST(request: Request) {
modelDisplayName: null,
sourceImageUrls: fallbackSourceImageUrls,
streamLabel: "agent-mode:image:detail-replacement:item-fallback",
isFallback: true,
});
successCount += 1;
}
});
} finally {
@ -6489,9 +6475,20 @@ export async function POST(request: Request) {
...(selectedAgentPreset.defaultParams || {}),
...(replacementSlot?.defaultParams || {}),
} as Record<string, any>;
const resolvedModelRef =
String(selectedChatModel || "").trim() &&
String(selectedChatModel || "").trim() !== "auto"
const replacementVisionModels = await getEnabledAiModelsWithProvider().catch(
(error) => {
chatApiLogger.warn("[Chat API] Failed to load configured vision models for replacement analysis:", error);
return undefined;
},
);
const resolvedModelRef = isDetailReplacementPreset
? await resolveDirectReplacementModelRef({
selectedChatModel,
finalModelId: presetModelId,
models: replacementVisionModels,
})
: String(selectedChatModel || "").trim() &&
String(selectedChatModel || "").trim() !== "auto"
? String(selectedChatModel || "").trim()
: presetModelId;
const configuredReplacementAspectRatio =
@ -6513,12 +6510,6 @@ export async function POST(request: Request) {
configuredAspectRatio: configuredReplacementAspectRatio,
});
}
const replacementVisionModels = await getEnabledAiModelsWithProvider().catch(
(error) => {
chatApiLogger.warn("[Chat API] Failed to load configured vision models for replacement analysis:", error);
return undefined;
},
);
const replacementScenePlanningRuntime =
await resolveReplacementScenePlanningRuntime({
models: replacementVisionModels,
@ -6816,24 +6807,36 @@ export async function POST(request: Request) {
!(isDetailReplacementPreset &&
detailReplacementStrategy === "product_only") &&
Boolean(modelInputImageUrl) &&
(replacementScenePlanShouldIncludeModelImage(
(shouldIncludeDetailReplacementModelImage(
replacementExecutionPlan.scenePlan,
) ||
(isDetailReplacementPreset &&
selectedProductContextImageUrls.length === 0));
const sourceImageUrls = Array.from(
new Set(
[
...selectedProductContextImageUrls,
...(shouldIncludeModelImage && modelInputImageUrl
? [modelInputImageUrl]
: []),
...replacementTemplateReferenceImageUrls,
...rerunReferenceImageUrls,
reference.imageUrl,
].filter(Boolean),
),
);
const detailGenerationInputs = isDetailReplacementPreset
? selectDetailReplacementGenerationInputs({
productImageUrls: selectedProductContextImageUrls,
modelImageUrl: modelInputImageUrl,
includeModelImage: shouldIncludeModelImage,
templateReferenceImageUrls: replacementTemplateReferenceImageUrls,
rerunReferenceImageUrls,
referenceImageUrl: reference.imageUrl,
})
: null;
const sourceImageUrls = detailGenerationInputs
? detailGenerationInputs.imageUrls
: Array.from(
new Set(
[
...selectedProductContextImageUrls,
...(shouldIncludeModelImage && modelInputImageUrl
? [modelInputImageUrl]
: []),
...replacementTemplateReferenceImageUrls,
...rerunReferenceImageUrls,
reference.imageUrl,
].filter(Boolean),
),
);
const replacementPrompt = buildAgentReplacementPrompt({
basePrompt:
baseReplacementPrompt ||
@ -6850,11 +6853,19 @@ export async function POST(request: Request) {
referenceLabel: reference.label,
referenceIndex,
totalReferences: replacementReferences.length,
hasModelImage: shouldIncludeModelImage,
productImageCount: selectedProductContextImageUrls.length,
hasModelImage: detailGenerationInputs
? Boolean(detailGenerationInputs.modelImageUrl)
: shouldIncludeModelImage,
productImageCount: detailGenerationInputs
? detailGenerationInputs.productImageUrls.length
: selectedProductContextImageUrls.length,
originalProductImageCount: productInputImageUrls.length,
rerunReferenceCount: rerunReferenceImageUrls.length,
templateReferenceCount: replacementTemplateReferenceImageUrls.length,
rerunReferenceCount: detailGenerationInputs
? detailGenerationInputs.rerunImageUrls.length
: rerunReferenceImageUrls.length,
templateReferenceCount: detailGenerationInputs
? detailGenerationInputs.templateImageUrls.length
: replacementTemplateReferenceImageUrls.length,
pairedProductIndex,
scenePlan: replacementExecutionPlan.scenePlan,
referenceGeometry,
@ -6875,11 +6886,20 @@ export async function POST(request: Request) {
sourceImageCount: sourceImageUrls.length,
generationAspectRatio,
imageOrder: {
productImages: selectedProductContextImageUrls.length,
productImages:
detailGenerationInputs !== null
? detailGenerationInputs.productImageUrls.length
: selectedProductContextImageUrls.length,
originalProductImages: productInputImageUrls.length,
hasModelImage: shouldIncludeModelImage,
templateReferenceImages: replacementTemplateReferenceImageUrls.length,
rerunReferenceImages: rerunReferenceImageUrls.length,
hasModelImage: detailGenerationInputs
? Boolean(detailGenerationInputs.modelImageUrl)
: shouldIncludeModelImage,
templateReferenceImages: detailGenerationInputs
? detailGenerationInputs.templateImageUrls.length
: replacementTemplateReferenceImageUrls.length,
rerunReferenceImages: detailGenerationInputs
? detailGenerationInputs.rerunImageUrls.length
: rerunReferenceImageUrls.length,
finalReferenceImageIndex: sourceImageUrls.length,
},
});
@ -6897,6 +6917,17 @@ export async function POST(request: Request) {
imageSize: resolvedImageSize,
imageCount: 1,
referenceImages: sourceImageUrls,
promptExtend: isDetailReplacementPreset
? false
: typeof mergedImageParams.promptExtend === "boolean"
? mergedImageParams.promptExtend
: undefined,
providerImageParams: detailGenerationInputs
? {
inputImageRoleByUrl:
detailGenerationInputs.inputImageRoleByUrl,
}
: undefined,
debugLabel: `agent-${replacementSourceType}`,
debugMeta: {
referenceLabel: reference.label,
@ -6943,6 +6974,10 @@ export async function POST(request: Request) {
referenceImageUrl: reference.imageUrl,
referenceGeometry,
scenePlan: replacementExecutionPlan.scenePlan,
preserveReferenceOutsideSubject:
isDetailReplacementPreset &&
detailReplacementStrategy === "product_only",
preserveReferenceText: isDetailReplacementPreset,
uploadPrefix: isDetailReplacementPreset
? "agent-detail-replacement"
: "agent-sku-replacement",
@ -9635,6 +9670,11 @@ export async function POST(request: Request) {
alibabaModelConfig?.fallbackModelId2,
].filter(Boolean) as string[];
const allModels = await getEnabledAiModelsWithProvider();
const currentModelEntry = resolveModelEntryByRuntimeModelId(
allModels,
actualModelId,
finalModelId,
);
const fallbackCandidates = buildImageFallbackCandidates(allModels, {
currentProviderId: alibabaModelConfig?.providerId,
currentModelId: alibabaModelConfig?.id,
@ -9880,6 +9920,7 @@ export async function POST(request: Request) {
let imageMarkdown = "";
let usedModelId = actualModelId;
const generationStartedAt = Date.now();
// 🆕 图片生成真正开始时发出标记(用于前端显示占位图)
writeAssistantTextChunk(
@ -9906,14 +9947,43 @@ export async function POST(request: Request) {
} catch (primaryError) {
chatApiLogger.error("[Chat API] Alibaba primary model failed:", primaryError);
if (!shouldFailoverAiModelError(primaryError)) {
await recordAiModelHealthFailure({
model: currentModelEntry?.model || alibabaModelConfig,
provider: currentModelEntry?.provider,
error: primaryError,
context: "chat_image_primary",
elapsedMs: Date.now() - generationStartedAt,
metadata: {
runtimeModelId: actualModelId,
finalModelId,
providerType: (languageModel as any).provider,
aspectRatio: effectiveImageGenerationParams.aspectRatio,
imageSize: effectiveImageGenerationParams.imageSize,
imageCount:
effectiveImageGenerationParams.imageCount ||
defaultImageQualityParams.imageCount ||
1,
},
}).catch((healthError) => {
chatApiLogger.warn("[Chat API] Record image primary health failure failed:", healthError);
});
if (
!shouldFailoverAiModelError(primaryError) &&
!shouldFailoverVip98kRequestRejection(
primaryError,
(languageModel as any).provider,
)
) {
chatApiLogger.info("[Chat API] Alibaba primary error is not eligible for image failover; returning original error.");
throw primaryError;
}
let fallbackSuccess = false;
const triedFallbackModelIds = new Set<string>();
for (const candidate of fallbackCandidates) {
try {
triedFallbackModelIds.add(String(candidate.model.id || ""));
chatApiLogger.info(`[Chat API] 🔄 Alibaba fallback model: ${candidate.model.displayName} (${candidate.provider.type})`);
const fallbackResult = await generateImageWithFailoverCandidate({
candidate,
@ -9938,6 +10008,22 @@ export async function POST(request: Request) {
if (shouldCooldownImageFallbackModel(fallbackErrorMessage)) {
setImageFallbackModelCooldown(candidate.model.id, fallbackErrorMessage);
}
await recordAiModelHealthFailure({
model: candidate.model,
provider: candidate.provider,
error: fallbackError,
context: "chat_image_fallback",
elapsedMs: Date.now() - generationStartedAt,
metadata: {
primaryModelId: actualModelId,
fallbackModelId: candidate.model.modelId,
providerType: candidate.provider.type,
aspectRatio: effectiveImageGenerationParams.aspectRatio,
imageSize: effectiveImageGenerationParams.imageSize,
},
}).catch((healthError) => {
chatApiLogger.warn("[Chat API] Record image fallback health failure failed:", healthError);
});
chatApiLogger.error(`[Chat API] ❌ Alibaba fallback model ${candidate.model.displayName} (${candidate.provider.type}) failed:`, fallbackError);
}
}
@ -9966,10 +10052,11 @@ export async function POST(request: Request) {
currentModelKey: actualModelId,
currentProviderId: alibabaModelConfig?.providerId,
providerPriority,
});
}).filter((candidate) => !triedFallbackModelIds.has(String(candidate.model.id || "")));
for (const candidate of crossProviderCandidates) {
try {
triedFallbackModelIds.add(String(candidate.model.id || ""));
chatApiLogger.info(`[Chat API] 🔁 Cross-provider failover attempt: ${candidate.model.displayName} (${candidate.provider.type})`);
if (
candidate.provider.type === "fal-image" &&
@ -10053,6 +10140,22 @@ export async function POST(request: Request) {
if (shouldCooldownImageFallbackModel(crossErrorMessage)) {
setImageFallbackModelCooldown(candidate.model.id, crossErrorMessage);
}
await recordAiModelHealthFailure({
model: candidate.model,
provider: candidate.provider,
error: crossError,
context: "chat_image_cross_fallback",
elapsedMs: Date.now() - generationStartedAt,
metadata: {
primaryModelId: actualModelId,
fallbackModelId: candidate.model.modelId,
providerType: candidate.provider.type,
aspectRatio: effectiveImageGenerationParams.aspectRatio,
imageSize: effectiveImageGenerationParams.imageSize,
},
}).catch((healthError) => {
chatApiLogger.warn("[Chat API] Record image cross-provider health failure failed:", healthError);
});
chatApiLogger.error(`[Chat API] ❌ Cross-provider failover failed: ${candidate.model.displayName} (${candidate.provider.type})`, crossError);
}
}
@ -10066,6 +10169,19 @@ export async function POST(request: Request) {
chatApiLogger.info("[Chat API] Alibaba generated, length:", imageMarkdown.length);
chatApiLogger.info("[Chat API] Alibaba used model:", usedModelId);
const usedModelEntry = resolveModelEntryByRuntimeModelId(
allModels,
usedModelId,
usedModelId === actualModelId ? finalModelId : undefined,
);
await recordAiModelHealthSuccess({
model:
usedModelEntry?.model ||
(usedModelId === actualModelId ? alibabaModelConfig : null),
}).catch((healthError) => {
chatApiLogger.warn("[Chat API] Record image health success failed:", healthError);
});
const usedModelDisplayName =
resolveModelDisplayNameByModelId(allModels, usedModelId) ||
alibabaModelConfig?.displayName ||

View File

@ -13,6 +13,10 @@ import { z } from "zod";
import { auth } from "@/app/(auth)/auth";
import { upsertMediaAsset } from "@/lib/db/queries";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
import { createLogger } from "@/lib/observability/logger";
import {
buildS3PublicUrl,
@ -160,7 +164,10 @@ function isAllowedDirectUpload(
);
}
function getPurposeMaxBytes(purpose?: string) {
function getPurposeMaxBytes(purpose?: string, contentType?: string) {
if (String(contentType || "").toLowerCase().startsWith("image/")) {
return IMAGE_UPLOAD_MAX_BYTES;
}
if (purpose === OPENCLAW_CHAT_VIDEO_UPLOAD_PURPOSE) {
return OPENCLAW_CHAT_VIDEO_MAX_BYTES;
}
@ -281,9 +288,14 @@ export async function POST(request: Request) {
{ status: 400 }
);
}
if (payload.size > getPurposeMaxBytes(payload.purpose)) {
const maxBytes = getPurposeMaxBytes(payload.purpose, contentType);
if (payload.size > maxBytes) {
return NextResponse.json(
{ error: `文件不能超过 ${getPurposeMaxBytes(payload.purpose) / 1024 / 1024}MB` },
{
error: contentType.startsWith("image/")
? IMAGE_UPLOAD_TOO_LARGE_MESSAGE
: `文件不能超过 ${maxBytes / 1024 / 1024}MB`,
},
{ status: 413 }
);
}
@ -390,8 +402,15 @@ export async function POST(request: Request) {
if (!isAllowedDirectUpload(contentType, payload.purpose, payload.filename)) {
return NextResponse.json({ error: "不支持续传该文件类型" }, { status: 400 });
}
if (payload.size > getPurposeMaxBytes(payload.purpose)) {
return NextResponse.json({ error: "文件超过允许的上传大小" }, { status: 413 });
if (payload.size > getPurposeMaxBytes(payload.purpose, contentType)) {
return NextResponse.json(
{
error: contentType.startsWith("image/")
? IMAGE_UPLOAD_TOO_LARGE_MESSAGE
: "文件超过允许的上传大小",
},
{ status: 413 }
);
}
const expectedPartSize = computePartSize(payload.size);
if (payload.partSize !== expectedPartSize) {
@ -505,8 +524,15 @@ export async function POST(request: Request) {
{ status: 400 }
);
}
if (payload.size > getPurposeMaxBytes(payload.purpose)) {
return NextResponse.json({ error: "文件超过允许的上传大小" }, { status: 413 });
if (payload.size > getPurposeMaxBytes(payload.purpose, contentType)) {
return NextResponse.json(
{
error: contentType.startsWith("image/")
? IMAGE_UPLOAD_TOO_LARGE_MESSAGE
: "文件超过允许的上传大小",
},
{ status: 413 }
);
}
if (payload.uploadId) {

View File

@ -1,5 +1,9 @@
import { NextResponse } from "next/server";
import { z } from "zod";
import {
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
isImageUploadTooLarge,
} from "@/lib/image-upload-limits";
import { auth } from "@/app/(auth)/auth";
import { createAgentWorkflowPreviewImageBuffer } from "@/lib/agents/workflow-preview-images";
@ -260,6 +264,12 @@ export async function POST(request: Request) {
const clientUploadId =
String(formData.get("clientUploadId") || "").trim() || null;
const isSvgUpload = contentType === "image/svg+xml";
if (contentType.startsWith("image/") && isImageUploadTooLarge(file.size)) {
return NextResponse.json(
{ error: IMAGE_UPLOAD_TOO_LARGE_MESSAGE },
{ status: 413 }
);
}
if (
isSvgUpload &&
(uploadPurpose !== "admin-svg-icon" || !isSiteAdminType(session.user.type))

View File

@ -102,8 +102,8 @@ export async function GET(request: Request) {
},
{
id: "local-e2e-video-model",
modelId: "kling-3-local",
displayName: "可灵 3.0",
modelId: "veo-3.1-fast",
displayName: "Veo 3.1 Fast(本地布局测试)",
avatarUrl: null,
description: "本地视频画布测试视频模型",
supportsVision: true,
@ -111,18 +111,18 @@ export async function GET(request: Request) {
isMultimodal: true,
isDefault: false,
providerName: "Local E2E",
providerType: "local",
providerType: "vip98k",
isSubModel: false,
parentModelId: null,
capabilities: { video: true },
capabilities: { kind: "video", supportedModes: ["text-to-video", "first-frame", "first-last-frame", "reference-images"], supportedRatios:["16:9"], supportedResolutions:["720p","1080p"], durationOptions:[8], maxReferenceImages:3 },
videoToolCapability: {
supported: true,
durations: [5, 10],
durations: [8],
resolutions: ["720p", "1080p"],
},
imageGenConfig: {
creditCostPerVideo: 20,
videoDurationOptions: [5, 10],
videoDurationOptions: [8],
},
sortOrder: 1,
selectable: true,

View File

@ -1,5 +1,6 @@
import { Suspense } from "react";
import { ChatShell } from "@/components/chat-shell";
import { CanvasLogoLoadingView } from "@/components/canvas/canvas-logo-loading";
import { DataStreamProvider } from "@/components/data-stream-provider";
import { auth } from "../(auth)/auth";
import {
@ -8,10 +9,14 @@ import {
getTutorialVideos,
} from "@/lib/db/queries";
export const dynamic = "force-dynamic";
export default function Layout({ children }: { children: React.ReactNode }) {
return (
<DataStreamProvider>
<Suspense fallback={<div className="flex h-dvh" />}>
<Suspense
fallback={<CanvasLogoLoadingView label="正在加载页面..." />}
>
<SidebarWrapper>{children}</SidebarWrapper>
</Suspense>
</DataStreamProvider>

View File

@ -1,79 +1,75 @@
import { randomUUID } from "node:crypto";
import { redirect } from "next/navigation";
import { auth } from "@/app/(auth)/auth";
import { db } from "@/lib/db";
import { canvasProject } from "@/lib/db/schema";
import { and, desc, eq } from "drizzle-orm";
import { auth } from "@/app/(auth)/auth";
import { VideoCanvasHome, type VideoCanvasHomeProject } from "@/components/canvas/VideoCanvasHome";
import { db } from "@/lib/db";
import { getSiteSettings } from "@/lib/db/queries";
import { canvasProject } from "@/lib/db/schema";
type VideoCanvasRedirectPageProps = {
searchParams?: Promise<Record<string, string | string[] | undefined>>;
};
type Props = { searchParams?: Promise<Record<string, string | string[] | undefined>> };
function getSearchParam(
params: Record<string, string | string[] | undefined>,
key: string
) {
const value = params[key];
function first(value: string | string[] | undefined) {
return Array.isArray(value) ? value[0] : value;
}
function buildVideoCanvasUrl(
canvasId: string,
params: Record<string, string | string[] | undefined>
) {
const query = new URLSearchParams();
query.set("mode", "video");
query.set("workspaceType", "video");
for (const [key, value] of Object.entries(params)) {
if (
key === "new" ||
key === "mode" ||
key === "workspaceType" ||
value === undefined
) {
continue;
}
if (Array.isArray(value)) {
value.forEach((item) => query.append(key, item));
continue;
}
query.set(key, value);
function canvasHref(id: string, incoming: Record<string, string | string[] | undefined>) {
const params = new URLSearchParams({ mode: "video", workspaceType: "video" });
for (const [key, value] of Object.entries(incoming)) {
if (["new", "mode", "workspaceType"].includes(key) || value === undefined) continue;
for (const item of Array.isArray(value) ? value : [value]) params.append(key, item);
}
return `/canvas/${canvasId}?${query.toString()}`;
return `/canvas/${id}?${params.toString()}`;
}
async function getLatestVideoCanvasId(userId: string) {
const [latestProject] = await db
.select({ id: canvasProject.id })
.from(canvasProject)
.where(
and(
eq(canvasProject.userId, userId),
eq(canvasProject.workspaceType, "video")
)
)
.orderBy(desc(canvasProject.updatedAt))
.limit(1);
return latestProject?.id || null;
function projectPreview(thumbnail: string | null, canvasData: unknown) {
const nodes = canvasData && typeof canvasData === "object" && "nodes" in canvasData
? (canvasData as { nodes?: unknown }).nodes : null;
const candidates = [thumbnail];
if (Array.isArray(nodes)) for (const node of [...nodes].reverse()) {
if (!node || typeof node !== "object") continue;
const item = node as { inputType?: string; draftValues?: Record<string, unknown> };
const draft = item.draftValues || {};
candidates.push(
String(draft.poster || draft.previewImageUrl || draft.thumbnailUrl || draft.resultImage || (item.inputType === "image" ? draft.mediaUrl || (Array.isArray(draft.url) ? draft.url[0] : "") : "") || ""),
);
}
return candidates.find(value => typeof value === "string" && (value.startsWith("https://") || value.startsWith("/"))) || null;
}
export default async function VideoCanvasRedirectPage({
searchParams,
}: VideoCanvasRedirectPageProps) {
const params = (await searchParams) || {};
const shouldCreateNew = getSearchParam(params, "new") === "1";
if (shouldCreateNew) {
redirect(buildVideoCanvasUrl(randomUUID(), params));
}
export default async function VideoCanvasHomePage({ searchParams }: Props) {
const session = await auth();
const latestCanvasId = session?.user?.id
? await getLatestVideoCanvasId(session.user.id).catch(() => null)
: null;
if (!session?.user?.id) redirect("/login?callbackUrl=%2Fvideo-canvas");
redirect(buildVideoCanvasUrl(latestCanvasId || randomUUID(), params));
const params = (await searchParams) || {};
if (first(params.new) === "1") {
const [created] = await db.insert(canvasProject).values({
userId: session.user.id,
title: "未命名视频画布",
workspaceType: "video",
canvasData: { nodes: [], edges: [], viewport: { x: 0, y: 0, zoom: 1 } },
}).returning({ id: canvasProject.id });
redirect(canvasHref(created.id, params));
}
const [rows, siteSettings] = await Promise.all([db.select({
id: canvasProject.id,
title: canvasProject.title,
thumbnail: canvasProject.thumbnail,
canvasData: canvasProject.canvasData,
createdAt: canvasProject.createdAt,
updatedAt: canvasProject.updatedAt,
}).from(canvasProject).where(and(
eq(canvasProject.userId, session.user.id),
eq(canvasProject.workspaceType, "video"),
)).orderBy(desc(canvasProject.updatedAt)).limit(25), getSiteSettings()]);
const projects: VideoCanvasHomeProject[] = rows.slice(0, 24).map(row => ({
id: row.id,
title: row.title,
thumbnail: projectPreview(row.thumbnail, row.canvasData),
createdAt: row.createdAt.toISOString(),
updatedAt: row.updatedAt.toISOString(),
}));
return <VideoCanvasHome initialHasMore={rows.length > 24} initialProjects={projects} siteLogo={siteSettings.siteLogo} siteLogoDark={siteSettings.siteLogoDark} siteName={siteSettings.siteName} userName={session.user.name || ""} />;
}

View File

@ -94,6 +94,8 @@ export async function POST(request: Request) {
providerId,
modelId: preset.modelId,
...values,
isEnabled: true,
isHidden: false,
isDefault: false,
isSubModel: false,
parentModelId: null,

View File

@ -76,8 +76,6 @@ export async function POST(request: Request) {
description: preset.description,
supportsVision: preset.supportsVision,
supportsImageGen: preset.supportsImageGen,
isEnabled: true,
isHidden: false,
sortOrder: preset.sortOrder,
capabilities: preset.capabilities || null,
defaultParams: preset.defaultParams || null,

View File

@ -0,0 +1,295 @@
import { and, desc, eq, inArray, sql } from "drizzle-orm";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { db } from "@/lib/db";
import { canvasImageTask, canvasProject, message } from "@/lib/db/schema";
import { extractAgentProjectStateFromCanvasData, isAgentProjectRow } from "@/lib/agents/agent-project-compat";
import {
createCanvasImageTask,
enqueueCanvasImageTask,
ensureCanvasImageTaskActive,
getCanvasImageTaskForUser,
} from "@/lib/canvas/image-task-queue";
import {
checkUserImageQuota,
getChatById,
getEnabledAiModelsWithProvider,
getPointsSystemConfig,
getUserById,
saveChat,
} from "@/lib/db/queries";
import { getModelGenerationCreditCost } from "@/lib/ai/image-model-pricing";
import { resolveDirectReplacementModelRef } from "@/app/(chat)/api/chat/_lib/detail-replacement";
import {
buildAgentSlotPrompt,
normalizeAgentSupplementPrompt,
} from "@/app/(chat)/api/chat/_lib/agent-scene-workflow";
import {
detailReplacementSubmissionKey,
isDetailReplacementBatchRequest,
isDetailReplacementUuid,
type DetailReplacementBatchReference,
type DetailReplacementBatchRequest,
type DetailReplacementBatchResult,
} from "@/lib/agents/detail-replacement-batch";
export const maxDuration = 1200;
function scopedChatId(projectId: string) {
const normalized = projectId.toLowerCase().replace(/-/g, "");
const scoped = `${normalized.slice(0, 20)}a93e17a60001`;
return [scoped.slice(0, 8), scoped.slice(8, 12), scoped.slice(12, 16), scoped.slice(16, 20), scoped.slice(20, 32)].join("-");
}
function taskResponse(task: Awaited<ReturnType<typeof getCanvasImageTaskForUser>>) {
const result = task?.result as DetailReplacementBatchResult | null;
const request = task?.request as DetailReplacementBatchRequest | null;
return {
id: task?.id,
status: task?.status,
attempts: task?.attempts,
error: task?.error || null,
cards: Array.isArray(result?.cards) ? result.cards : [],
currentCardId: result?.currentCardId || null,
requestedCardIds: request?.references?.map((item) => item.id) || [],
createdAt: task?.createdAt || null,
queueStatus: task?.queueStatus || null,
};
}
async function findActiveBatchId(params: {
userId: string;
projectId: string;
submissionKey: string;
}) {
const [row] = await db.select({ id: canvasImageTask.id })
.from(canvasImageTask)
.where(and(
eq(canvasImageTask.userId, params.userId),
inArray(canvasImageTask.status, ["pending", "running"]),
sql`${canvasImageTask.request}->>'taskKind' = 'agent-detail-replacement-batch'`,
sql`${canvasImageTask.request}->>'projectId' = ${params.projectId}`,
sql`${canvasImageTask.request}->>'submissionKey' = ${params.submissionKey}`,
))
.orderBy(desc(canvasImageTask.createdAt)).limit(1);
return row?.id || null;
}
function isUniqueViolation(error: unknown) {
for (let cursor: unknown = error; cursor && typeof cursor === "object"; ) {
if ((cursor as { code?: unknown }).code === "23505") return true;
cursor = (cursor as { cause?: unknown }).cause;
}
return false;
}
export async function GET(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
const params = new URL(request.url).searchParams;
const projectId = String(params.get("projectId") || "").trim();
const taskId = String(params.get("taskId") || "").trim();
if (!isDetailReplacementUuid(projectId)) {
return NextResponse.json({ error: "项目编号无效" }, { status: 400 });
}
const [project] = await db.select({ id: canvasProject.id }).from(canvasProject).where(and(
eq(canvasProject.id, projectId), eq(canvasProject.userId, session.user.id),
)).limit(1);
if (!project) return NextResponse.json({ error: "项目不存在" }, { status: 404 });
const [candidate] = await db.select({ id: canvasImageTask.id, request: canvasImageTask.request })
.from(canvasImageTask)
.where(and(
eq(canvasImageTask.userId, session.user.id),
...(isDetailReplacementUuid(taskId) ? [eq(canvasImageTask.id, taskId)] : []),
sql`${canvasImageTask.request}->>'taskKind' = 'agent-detail-replacement-batch'`,
sql`${canvasImageTask.request}->>'projectId' = ${projectId}`,
))
// A batch that is still working matters more than a newer finished one, so the
// workspace keeps showing live progress after a rerun of a single card.
.orderBy(
sql`case when ${canvasImageTask.status} in ('pending', 'running') then 0 else 1 end`,
desc(canvasImageTask.createdAt),
).limit(1);
if (!candidate || !isDetailReplacementBatchRequest(candidate.request)) {
return NextResponse.json({ task: null });
}
let task = await getCanvasImageTaskForUser({ taskId: candidate.id, userId: session.user.id });
if (task && (task.status === "failed" ||
(task.status === "running" && Date.now() - task.updatedAt.getTime() > 90_000))) {
await ensureCanvasImageTaskActive({ taskId: task.id, userId: session.user.id });
task = await getCanvasImageTaskForUser({ taskId: candidate.id, userId: session.user.id });
}
return NextResponse.json({ task: taskResponse(task) });
}
export async function POST(request: Request) {
try {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
const body = await request.json();
const projectId = String(body.projectId || "").trim();
const taskId = String(body.requestId || "").trim();
if (!isDetailReplacementUuid(projectId) || !isDetailReplacementUuid(taskId)) {
return NextResponse.json({ error: "任务编号或项目编号无效" }, { status: 400 });
}
const [project] = await db.select().from(canvasProject).where(and(
eq(canvasProject.id, projectId), eq(canvasProject.userId, session.user.id),
)).limit(1);
if (!project || !isAgentProjectRow(project)) {
return NextResponse.json({ error: "风格化项目不存在" }, { status: 404 });
}
const state = extractAgentProjectStateFromCanvasData(project.canvasData);
const knownReferences = new Map(
(state.workflowState.agentModeState?.detailReplacement?.references || [])
.map((item) => [String(item.id), String(item.imageUrl)]),
);
const requested = Array.isArray(body.references) ? body.references : [];
const references: DetailReplacementBatchReference[] = requested.map((raw: unknown) => {
const item = raw && typeof raw === "object" ? raw as Record<string, unknown> : {};
return {
id: String(item.id || "").slice(0, 128),
label: String(item.label || "详情参考图").slice(0, 255),
imageUrl: String(item.imageUrl || "").trim(),
candidateId: String(item.candidateId || "").slice(0, 128) || null,
passthrough: item.passthrough === true,
rerunReferenceImages: Array.isArray(item.rerunReferenceImages)
? item.rerunReferenceImages.slice(0, 6).map((entry: { url?: string }) => ({ url: String(entry?.url || "") }))
: [],
};
});
if (
references.length === 0 || references.length > 40 ||
new Set(references.map((item) => item.id)).size !== references.length ||
references.some((item) => !item.id || knownReferences.get(item.id) !== item.imageUrl)
) {
return NextResponse.json({ error: "详情参考图尚未保存或已发生变化,请刷新后重试" }, { status: 400 });
}
const productImageUrls = Array.isArray(body.productImageUrls)
? Array.from(new Set(body.productImageUrls.map((url: unknown) => String(url || "").trim())
.filter((url: string) => /^https?:\/\//i.test(url)))).slice(0, 16) as string[]
: [];
if (productImageUrls.length === 0 && !references.every((item) => item.passthrough)) {
return NextResponse.json({ error: "请先在输入图区选择至少一张产品图" }, { status: 400 });
}
const strategy = body.replacementStrategy === "product_only" ? "product_only" : "scene_replace";
const prompt = String(body.prompt || "").trim().slice(0, 24_000);
if (strategy !== "product_only" && !prompt) {
return NextResponse.json({ error: "请先配置风格化提示词" }, { status: 400 });
}
const models = await getEnabledAiModelsWithProvider();
const model = await resolveDirectReplacementModelRef({
selectedChatModel: String(body.model || "auto"),
finalModelId: String(body.model || "auto"),
models,
});
if (!model || model === "auto" || !models.some((entry) => entry.model.id === model)) {
return NextResponse.json({ error: "详情替换模型不可用,请检查模型配置" }, { status: 400 });
}
const quota = await checkUserImageQuota(session.user.id);
if (!quota.allowed) {
return NextResponse.json({ error: "本月图片生成配额已用完" }, { status: 429 });
}
const requestedImageCount = references.filter((item) => !item.passthrough).length;
if (quota.limit != null && quota.used + requestedImageCount > quota.limit) {
return NextResponse.json({ error: `本月剩余生图配额不足 ${requestedImageCount} 张` }, { status: 429 });
}
const pointsConfig = await getPointsSystemConfig();
if (pointsConfig.enabled && pointsConfig.blockOnInsufficient !== false) {
const selectedModel = models.find((entry) => entry.model.id === model)?.model;
const unitCost = getModelGenerationCreditCost(
selectedModel as Parameters<typeof getModelGenerationCreditCost>[0], "image", {
imageSize: ["1K", "2K", "4K"].includes(body.imageSize) ? body.imageSize : "2K",
});
const user = await getUserById({ id: session.user.id });
const expectedCost = unitCost * requestedImageCount;
if (!user?.isHighVip && expectedCost > Number(user?.pointsBalance || 0)) {
return NextResponse.json({ error: `积分不足,本批次预计消耗 ${expectedCost} 积分` }, { status: 402 });
}
}
const chatId = scopedChatId(projectId);
const existingChat = await getChatById({ id: chatId });
if (existingChat && existingChat.userId !== session.user.id) {
return NextResponse.json({ error: "无权访问项目对话" }, { status: 403 });
}
if (!existingChat) await saveChat({
id: chatId, userId: session.user.id, title: project.title || "详情替换", visibility: "private",
});
const taskRequest: DetailReplacementBatchRequest = {
taskKind: "agent-detail-replacement-batch",
projectId,
chatId,
model,
prompt: strategy === "product_only"
? "仅替换参考详情图中的产品主体"
: buildAgentSlotPrompt({
workflowPrompt: prompt,
slotPrompt: "",
userSupplementPrompt: normalizeAgentSupplementPrompt(
String(body.userPrompt || ""), "详情参考图替换",
),
includeSlotTitle: false,
modeType: "detail_replacement",
}),
userPrompt: String(body.userPrompt || "").slice(0, 12_000),
imageSize: ["1K", "2K", "4K"].includes(body.imageSize) ? body.imageSize : "2K",
replacementStrategy: strategy,
productImageUrls,
modelImageUrl: /^https?:\/\//i.test(String(body.modelImageUrl || ""))
? String(body.modelImageUrl) : null,
references,
};
const submissionKey = detailReplacementSubmissionKey(taskRequest);
taskRequest.submissionKey = submissionKey;
const duplicateId = await findActiveBatchId({
userId: session.user.id, projectId, submissionKey,
});
if (duplicateId) {
await ensureCanvasImageTaskActive({ taskId: duplicateId, userId: session.user.id });
const duplicate = await getCanvasImageTaskForUser({ taskId: duplicateId, userId: session.user.id });
return NextResponse.json({ task: taskResponse(duplicate) });
}
let task: Awaited<ReturnType<typeof createCanvasImageTask>>;
try {
task = await createCanvasImageTask({
taskId, userId: session.user.id, request: taskRequest as unknown as Record<string, unknown>,
});
} catch (error) {
if (!isUniqueViolation(error)) throw error;
// Two identical submissions raced past the lookup above; the database
// unique index kept one of them, so answer with that batch.
const racedId = await findActiveBatchId({
userId: session.user.id, projectId, submissionKey,
});
if (!racedId) throw error;
const raced = await getCanvasImageTaskForUser({ taskId: racedId, userId: session.user.id });
return NextResponse.json({ task: taskResponse(raced) });
}
const existingTask = await getCanvasImageTaskForUser({ taskId: task.id, userId: session.user.id });
const storedRequest = existingTask?.request as DetailReplacementBatchRequest | null;
if (
!isDetailReplacementBatchRequest(storedRequest) ||
storedRequest.projectId !== projectId ||
(storedRequest.submissionKey && storedRequest.submissionKey !== submissionKey)
) {
return NextResponse.json({ error: "任务编号已被其他请求使用" }, { status: 409 });
}
await db.insert(message).values({
id: taskId,
chatId,
role: "user",
parts: [{ type: "text", text: taskRequest.userPrompt || `批量替换 ${references.length} 张详情图` }],
attachments: [],
createdAt: new Date(),
}).onConflictDoNothing({ target: message.id });
await enqueueCanvasImageTask(task.id);
const current = await getCanvasImageTaskForUser({ taskId, userId: session.user.id });
return NextResponse.json({ task: taskResponse(current) });
} catch (error) {
console.error("[Detail Replacement Batch] Failed to start", error);
return NextResponse.json({ error: error instanceof Error ? error.message : "提交详情替换任务失败" }, { status: 500 });
}
}

View File

@ -1,385 +1,10 @@
import { NextRequest, NextResponse } from "next/server";
import {
createMediaAssetRelation,
getMediaAssetsByUserAndUrls,
getPointsSystemConfig,
getSystemSetting,
upsertMediaAsset,
} from "@/lib/db/queries";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { uploadFile } from "@/lib/storage/service";
import { db } from "@/lib/db";
import { generatedImage } from "@/lib/db/schema";
import { createLogger } from "@/lib/observability/logger";
import { getImageToolCreditCost } from "@/lib/points/config";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
const logger = createLogger("angle-edit-api");
// 角度编辑API配置类型
type AngleEditConfig = {
enabled?: boolean;
falApiKey?: string;
loraScale?: number;
inferenceSteps?: number;
guidanceScale?: number;
maxImageSize?: number;
timeout?: number;
};
// 设置API路由最大执行时间为3分钟(180秒)
import { executeAngleEditPOST } from "@/lib/canvas/operations/angle-edit";
export const runtime = "nodejs";
export const maxDuration = 180;
/**
* 压缩图片生成缩略图
*/
async function createThumbnail(buffer: Buffer): Promise<Buffer> {
try {
const sharp = await import("sharp");
const thumbnailBuffer = await sharp.default(buffer)
.resize(500, null, {
fit: "inside",
withoutEnlargement: true,
})
.jpeg({ quality: 80 })
.toBuffer();
logger.debug("thumbnail_created", {
originalBytes: buffer.length,
thumbnailBytes: thumbnailBuffer.length,
});
return thumbnailBuffer;
} catch (error) {
logger.warn("thumbnail_creation_failed", error);
return buffer;
}
}
/**
* 将 base64 图片上传到 S3
*/
async function uploadBase64ToS3(base64Data: string, prefix: string = "angle-edit"): Promise<{ thumbnail: string; original: string }> {
try {
const buffer = Buffer.from(base64Data, "base64");
const timestamp = Date.now();
const randomStr = Math.random().toString(36).slice(2, 8);
const thumbnailFilename = `${prefix}-thumb-${timestamp}-${randomStr}.png`;
const originalFilename = `${prefix}-orig-${timestamp}-${randomStr}.png`;
logger.debug("source_image_prepared", { originalBytes: buffer.length });
const thumbnailBuffer = await createThumbnail(buffer);
const thumbnailArrayBuffer = thumbnailBuffer.buffer.slice(
thumbnailBuffer.byteOffset,
thumbnailBuffer.byteOffset + thumbnailBuffer.byteLength
) as ArrayBuffer;
const originalArrayBuffer = buffer.buffer.slice(
buffer.byteOffset,
buffer.byteOffset + buffer.byteLength
) as ArrayBuffer;
logger.debug("storage_upload_started", {
hasOriginal: originalArrayBuffer.byteLength > 0,
hasThumbnail: thumbnailArrayBuffer.byteLength > 0,
prefix,
});
const [thumbnailResult, originalResult] = await Promise.all([
uploadFile(thumbnailFilename, thumbnailArrayBuffer),
uploadFile(originalFilename, originalArrayBuffer),
]);
logger.info("storage_upload_completed", {
originalUrl: originalResult.url,
thumbnailUrl: thumbnailResult.url,
});
return {
thumbnail: thumbnailResult.url,
original: originalResult.url,
};
} catch (error) {
logger.error("storage_upload_failed", error);
throw new Error(`图片上传失败: ${error instanceof Error ? error.message : "未知错误"}`);
}
}
export async function POST(request: NextRequest) {
const startTime = Date.now();
try {
// 1. 验证用户登录
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ error: "未登录" }, { status: 401 });
}
logger.info("request_received", { authenticated: true });
// 2. 从数据库获取配置
const angleEditConfig = await getSystemSetting({ key: "angle_edit_config" });
const config = (angleEditConfig?.value as AngleEditConfig) || {};
// 验证功能是否启用
if (!config.enabled) {
return NextResponse.json(
{ error: "角度编辑功能未启用,请联系管理员在系统设置中开启" },
{ status: 400 }
);
}
// 验证 API Key
if (!config.falApiKey) {
return NextResponse.json(
{ error: "fal.ai API Key 未配置,请联系管理员配置" },
{ status: 500 }
);
}
logger.debug("config_loaded", {
enabled: Boolean(config.enabled),
hasApiKey: Boolean(config.falApiKey),
loraScale: config.loraScale || 1,
});
// 3. 解析请求
const body = await request.json();
const {
image_url,
image_base64,
// 数值参数 (官方 API 格式)
horizontal_angle, // 0-360 度
vertical_angle, // -30 到 90 度
zoom, // 0-10
lora_scale,
} = body;
if (!image_url && !image_base64) {
return NextResponse.json({ error: "缺少图片数据" }, { status: 400 });
}
// 使用请求中的参数或配置中的默认值
const finalLoraScale = lora_scale ?? config.loraScale ?? 1;
const finalHorizontalAngle = horizontal_angle ?? 0;
const finalVerticalAngle = vertical_angle ?? 0;
const finalZoom = zoom ?? 5;
logger.info("request_parameters_normalized", {
hasBase64Input: Boolean(image_base64),
hasUrlInput: Boolean(image_url),
horizontalAngle: finalHorizontalAngle,
loraScale: finalLoraScale,
verticalAngle: finalVerticalAngle,
zoom: finalZoom,
});
const pointsConfig = await getPointsSystemConfig();
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, "angle-edit"),
reason: "image_angle_edit",
modelId: "qwen-image-edit-2511-multiple-angles",
note: "多角度编辑消耗积分",
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
// 4. 准备图片输入
let imageInput: string;
if (image_url) {
imageInput = image_url;
} else {
const mimeType = image_base64.startsWith("/9j/") ? "image/jpeg" : "image/png";
imageInput = `data:${mimeType};base64,${image_base64}`;
}
// 5. 动态导入 fal.ai SDK 并调用官方多角度 API
logger.info("provider_call_started", {
modelId: "fal-ai/qwen-image-edit-2511-multiple-angles",
});
const fal = await import("@fal-ai/serverless-client");
fal.config({
credentials: config.falApiKey,
});
// 使用官方多角度 API 端点
// 文档: https://fal.ai/models/fal-ai/qwen-image-edit-2511-multiple-angles
const result = await fal.subscribe("fal-ai/qwen-image-edit-2511-multiple-angles", {
input: {
image_urls: [imageInput], // 官方使用 image_urls (数组)
horizontal_angle: finalHorizontalAngle, // 水平旋转角度 0-360
vertical_angle: finalVerticalAngle, // 垂直角度 -30 到 90
zoom: finalZoom, // 缩放 0-10
lora_scale: finalLoraScale, // LoRA 强度
num_inference_steps: config.inferenceSteps || 28,
guidance_scale: config.guidanceScale || 4.5, // 官方默认 4.5
enable_safety_checker: true,
output_format: "png",
},
logs: true,
onQueueUpdate: (update) => {
if (update.status === "IN_PROGRESS") {
logger.debug("provider_progress", { status: update.status });
}
},
});
// 6. 处理结果
const apiResult = result as { images?: Array<{ url?: string }>; prompt?: string };
const outputImage = apiResult.images?.[0];
if (!outputImage?.url) {
throw new Error("API 未返回有效图片");
}
const outputImageUrl = outputImage.url;
logger.info("provider_call_completed", {
hasGeneratedPrompt: Boolean(apiResult.prompt),
imageCount: apiResult.images?.length ?? 0,
outputImageUrl,
});
// 7. 下载图片并上传到我们的 S3
const imageResponse = await fetch(outputImageUrl);
if (!imageResponse.ok) {
throw new Error(`下载输出图片失败: ${imageResponse.status}`);
}
const imageBuffer = Buffer.from(await imageResponse.arrayBuffer());
const imageBase64Result = imageBuffer.toString("base64");
const s3Result = await uploadBase64ToS3(imageBase64Result, "angle-edit");
logger.info("result_image_persisted", {
originalUrl: s3Result.original,
thumbnailUrl: s3Result.thumbnail,
});
// 8. 保存到数据库
const processingTime = (Date.now() - startTime) / 1000;
const [savedImage] = await db
.insert(generatedImage)
.values({
userId: session.user.id,
thumbnailUrl: s3Result.thumbnail,
originalUrl: s3Result.original,
prompt: `角度编辑: H${finalHorizontalAngle}° V${finalVerticalAngle}° Z${finalZoom}`,
modelId: "qwen-image-edit-2511-multiple-angles",
createdAt: new Date(),
})
.returning();
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume({
note: `多角度编辑消耗积分(${savedImage.id})`,
});
} catch (pointsError) {
logger.error("points_consume_failed", pointsError, {
imageId: savedImage.id,
userId: session.user.id,
});
}
}
try {
const sourceAssets =
typeof image_url === "string" && image_url.trim()
? await getMediaAssetsByUserAndUrls({
userId: session.user.id,
urls: [image_url],
})
: [];
const media = await upsertMediaAsset({
userId: session.user.id,
mediaType: "image",
sourceType: "edited",
status: "ready",
originalUrl: s3Result.original,
previewUrl: s3Result.original,
thumbnailUrl: s3Result.thumbnail,
displayName: `角度编辑 ${finalHorizontalAngle}/${finalVerticalAngle}`.slice(0, 80),
prompt: `角度编辑: H${finalHorizontalAngle}° V${finalVerticalAngle}° Z${finalZoom}`,
modelId: "qwen-image-edit-2511-multiple-angles",
sourceTable: "GeneratedImage",
sourceRowId: savedImage.id,
primaryParentAssetId: sourceAssets[0]?.id ?? null,
metadata: {
horizontalAngle: finalHorizontalAngle,
verticalAngle: finalVerticalAngle,
zoom: finalZoom,
},
});
if (media) {
for (const sourceAsset of sourceAssets) {
await createMediaAssetRelation({
userId: session.user.id,
fromAssetId: sourceAsset.id,
toAssetId: media.id,
relationType: "edited_from",
metadata: {
horizontalAngle: finalHorizontalAngle,
verticalAngle: finalVerticalAngle,
zoom: finalZoom,
},
});
}
}
} catch (assetError) {
logger.warn("media_asset_registration_failed", assetError);
}
logger.info("request_completed", {
processingSeconds: Number(processingTime.toFixed(2)),
});
// 9. 返回结果
return NextResponse.json({
success: true,
image_url: s3Result.original,
thumbnail_url: s3Result.thumbnail,
image_base64: imageBase64Result,
processing_time: processingTime,
angles: {
horizontal_angle: finalHorizontalAngle,
vertical_angle: finalVerticalAngle,
zoom: finalZoom,
},
prompt: apiResult.prompt,
});
} catch (error) {
logger.error("request_failed", error);
// 输出详细的错误信息
if (error && typeof error === "object" && "status" in error) {
const apiError = error as { status?: number; body?: unknown; message?: string };
logger.warn("provider_error_summary", {
bodyType: typeof apiError.body,
hasBody: apiError.body != null,
message: apiError.message,
status: apiError.status,
});
if (apiError.status === 403) {
return NextResponse.json(
{
error: "API 访问被拒绝 (403 Forbidden)。请检查 API Key 是否正确,以及是否有权限访问该模型。",
details: apiError.body,
},
{ status: 403 }
);
}
}
return NextResponse.json(
{
error: error instanceof Error ? error.message : "角度编辑处理失败",
},
{ status: 500 }
);
}
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
return executeAngleEditPOST(request, session.user.id);
}

View File

@ -14,6 +14,7 @@ const ALLOWED_MODEL_HOST_DOMAINS = [
"worldlabs.ai",
"s3.1080p.hk",
"img.yydd.ai",
"scqny.jiansun.vip",
"blob.vercel-storage.com",
"aliyuncs.com",
"amazonaws.com",

View File

@ -102,7 +102,7 @@ export async function GET(
// 更新画布(支持 upsert:如果不存在则创建)
export async function PUT(
request: NextRequest,
{ params }: { params: Promise<{ id: string }> }
{ params }: { params: Promise<{ id: string }> },
) {
try {
const { id } = await params;
@ -116,29 +116,33 @@ export async function PUT(
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ success: false, error: '未登录' }, { status: 401 });
return NextResponse.json(
{ success: false, error: "未登录" },
{ status: 401 },
);
}
const { title, description, thumbnail, canvasData } = body;
const baseCanvasRevision = body?.baseCanvasUpdatedAt || body?.baseCanvasRevision;
const baseCanvasRevision =
body?.baseCanvasUpdatedAt || body?.baseCanvasRevision;
const workspaceType =
body && Object.prototype.hasOwnProperty.call(body, "workspaceType")
? resolveWorkspaceType(body.workspaceType)
: undefined;
// 先检查画布是否存在
const [existingProject] = await db
.select()
.from(canvasProject)
.where(
and(
eq(canvasProject.id, id),
eq(canvasProject.userId, session.user.id)
return await db.transaction(async (tx) => {
const [existingProject] = await tx
.select()
.from(canvasProject)
.where(
and(
eq(canvasProject.id, id),
eq(canvasProject.userId, session.user.id),
),
)
);
if (existingProject) {
.for("update");
if (
existingProject &&
canvasData !== undefined &&
baseCanvasRevision &&
isStaleCanvasSaveBaseRevision({
@ -149,101 +153,66 @@ export async function PUT(
) {
return NextResponse.json(
buildStaleCanvasSaveConflictPayload(existingProject.updatedAt),
{ status: 409 }
{ status: 409 },
);
}
if (!existingProject) {
// The parent must exist before node/asset foreign keys are synchronized.
const [created] = await tx
.insert(canvasProject)
.values({
id,
userId: session.user.id,
createdAt: new Date(),
updatedAt: new Date(),
title: title || getDefaultWorkspaceTitle(workspaceType || "canvas"),
workspaceType: workspaceType || "canvas",
canvasData: {
nodes: [],
edges: [],
viewport: { x: 0, y: 0, zoom: 1 },
},
})
.onConflictDoNothing({ target: canvasProject.id })
.returning();
if (!created)
return NextResponse.json(
{ success: false, error: "画布已被创建,请重新加载后保存" },
{ status: 409 },
);
}
const syncedCanvasData =
canvasData !== undefined
? await syncCanvasNodesForProject({
projectId: id,
userId: session.user.id,
canvasData,
})
? await syncCanvasNodesForProject(
{ projectId: id, userId: session.user.id, canvasData },
tx,
)
: undefined;
// 更新现有画布
const updateData: any = { updatedAt: new Date() };
const updateData: any = { updatedAt: new Date(Math.max(Date.now(), (existingProject?.updatedAt.getTime() || 0) + 1)) };
if (title !== undefined) updateData.title = title;
if (description !== undefined) updateData.description = description;
if (thumbnail !== undefined) updateData.thumbnail = thumbnail;
if (workspaceType !== undefined) updateData.workspaceType = workspaceType;
if (syncedCanvasData !== undefined) updateData.canvasData = syncedCanvasData;
const [updatedProject] = await db
if (syncedCanvasData !== undefined)
updateData.canvasData = syncedCanvasData;
const [saved] = await tx
.update(canvasProject)
.set(updateData)
.where(
and(
eq(canvasProject.id, id),
eq(canvasProject.userId, session.user.id)
)
eq(canvasProject.userId, session.user.id),
),
)
.returning();
return NextResponse.json({ success: true, data: updatedProject });
} else {
const syncedCanvasData =
canvasData !== undefined
? await syncCanvasNodesForProject({
projectId: id,
userId: session.user.id,
canvasData,
})
: undefined;
const nextWorkspaceType = workspaceType || "canvas";
const nextCanvasData = syncedCanvasData || {
nodes: [],
edges: [],
viewport: { x: 0, y: 0, zoom: 1 },
};
const nextTitle = title || getDefaultWorkspaceTitle(nextWorkspaceType);
const insertData: any = {
id, // 使用指定的 ID
userId: session.user.id,
title: nextTitle,
workspaceType: nextWorkspaceType,
canvasData: nextCanvasData,
};
const conflictUpdateData: any = {
title: nextTitle,
workspaceType: nextWorkspaceType,
canvasData: nextCanvasData,
updatedAt: new Date(),
};
if (description !== undefined) {
insertData.description = description;
conflictUpdateData.description = description;
}
if (thumbnail !== undefined) {
insertData.thumbnail = thumbnail;
conflictUpdateData.thumbnail = thumbnail;
}
// 创建新画布(使用指定 ID),并兼容同一用户并发创建同一画布的场景。
const [newProject] = await db
.insert(canvasProject)
.values(insertData)
.onConflictDoUpdate({
target: canvasProject.id,
set: conflictUpdateData,
where: eq(canvasProject.userId, session.user.id),
})
.returning();
if (!newProject) {
return NextResponse.json(
{ success: false, error: '画布ID已被占用' },
{ status: 409 }
);
}
return NextResponse.json({ success: true, data: newProject });
}
return NextResponse.json({ success: true, data: saved });
});
} catch (error) {
console.error('更新/创建画布失败:', error);
return NextResponse.json({ success: false, error: '更新画布失败' }, { status: 500 });
console.error("更新/创建画布失败:", error);
return NextResponse.json(
{ success: false, error: "更新画布失败" },
{ status: 500 },
);
}
}

View File

@ -0,0 +1,112 @@
import { createHash, randomUUID } from "node:crypto";
import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { uploadFile } from "@/lib/storage/service";
import { ffmpeg, mediaSafety, probeMedia } from "@/lib/video-canvas/smart-edit/local";
import {
remoteAudio,
remoteAudioConfig,
} from "@/lib/video-canvas/smart-edit/remote-audio";
export const runtime = "nodejs";
export const maxDuration = 600;
const running = new Set<string>();
const allowedStems = ["vocals", "background"] as const;
export async function POST(request: Request) {
const session = await auth();
const userId = session?.user?.id;
if (!userId)
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
const body = (await request.json().catch(() => null)) as {
url?: unknown;
stem?: unknown;
} | null;
const url = typeof body?.url === "string" ? body.url.trim() : "";
const stem = typeof body?.stem === "string" ? body.stem : "";
if (!/^https?:\/\//i.test(url) || !allowedStems.includes(stem as (typeof allowedStems)[number]))
return NextResponse.json({ success: false, error: "参数不完整" }, { status: 400 });
if (!remoteAudioConfig())
return NextResponse.json(
{ success: false, error: "音频分离服务未配置,请联系管理员" },
{ status: 503 }
);
const lockKey = `${userId}:${url}:${stem}`;
if (running.has(lockKey) || running.size >= 3)
return NextResponse.json(
{ success: false, error: "已有分离任务在进行,请稍候" },
{ status: 429 }
);
running.add(lockKey);
let dir = "";
try {
dir = await mkdtemp(path.join(tmpdir(), "canvas-separate-"));
const input = path.join(dir, "input");
const response = await fetch(url, {
signal: AbortSignal.timeout(180000),
cache: "no-store",
});
if (!response.ok) throw new Error("素材下载失败,请确认素材仍可访问");
const bytes = Buffer.from(await response.arrayBuffer());
if (!bytes.length || bytes.length > 150 * 1024 * 1024)
throw new Error("素材必须非空且不超过 125 MiB");
await writeFile(input, bytes);
const info = await probeMedia(input, 600);
if (!info.hasAudio) throw new Error("当前视频没有音轨");
const inputAudio = path.join(dir, "separation-input.wav");
const durationSeconds = Math.round(info.duration / 1e6);
await ffmpeg(
[
...mediaSafety,
"-i",
input,
"-vn",
"-t",
String(durationSeconds),
"-ac",
"2",
"-ar",
"44100",
"-c:a",
"pcm_f32le",
inputAudio,
],
undefined,
240000
);
const requestKey = createHash("sha256")
.update(`${url}:${stem}:${(await stat(inputAudio)).size}`)
.digest("hex");
await remoteAudio("separate", inputAudio, userId, requestKey, dir);
const stemFile = path.join(dir, `${stem}.wav`);
const wav = await readFile(stemFile);
if (wav.length < 1024) throw new Error("分离结果异常,请稍后重试");
const uploaded = await uploadFile(
`canvas-separate-${randomUUID()}.wav`,
await new Blob([wav]).arrayBuffer()
);
return NextResponse.json({
success: true,
data: {
url: uploaded.url,
duration: Math.round(info.duration / 1e6),
},
});
} catch (error) {
return NextResponse.json(
{
success: false,
error:
error instanceof Error ? error.message : "人声分离失败,原音轨已保留",
},
{ status: 500 }
);
} finally {
running.delete(lockKey);
if (dir) await rm(dir, { recursive: true, force: true });
}
}

View File

@ -0,0 +1,24 @@
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { listCanvasAudioModels } from "@/lib/video-canvas/audio-generation";
import { audioContract } from "@/lib/video-canvas/audio-contract";
export async function GET() {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
try {
const models = await listCanvasAudioModels();
return NextResponse.json(
{
models: models.map(({ model }) => ({
id: model.id,
modelId: model.modelId,
name: model.displayName,
...audioContract(model.modelId),
})),
},
{ headers: { "Cache-Control": "no-store" } },
);
} catch {
return NextResponse.json({ error: "音频模型读取失败,请稍后重试" }, { status: 503 });
}
}

View File

@ -0,0 +1,266 @@
import { lookup } from "node:dns/promises";
import { isIP } from "node:net";
import { NextResponse } from "next/server";
import { z } from "zod";
import { auth } from "@/app/(auth)/auth";
import { getChatById, getMessageById, getSystemSetting } from "@/lib/db/queries";
import {
normalizeOpenClawChatConfig,
OPENCLAW_CHAT_CONFIG_KEY,
} from "@/lib/openclaw-chat/config";
import { normalizeClawdTaskMetadata } from "@/lib/openclaw-chat/task-state";
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export const maxDuration = 120;
const querySchema = z.object({
chatId: z.string().uuid(),
messageId: z.string().uuid(),
index: z.coerce.number().int().min(0).max(127),
download: z.enum(["0", "1"]).optional().default("0"),
});
const MAX_REDIRECTS = 4;
const RESPONSE_HEADERS = [
"accept-ranges",
"content-length",
"content-range",
"etag",
"last-modified",
] as const;
function isPrivateIpv4(address: string) {
const octets = address.split(".").map(Number);
if (octets.length !== 4 || octets.some((value) => !Number.isInteger(value))) {
return true;
}
const [a, b] = octets;
return (
a === 0 ||
a === 10 ||
a === 127 ||
(a === 100 && b >= 64 && b <= 127) ||
(a === 169 && b === 254) ||
(a === 172 && b >= 16 && b <= 31) ||
(a === 192 && b === 168) ||
a >= 224
);
}
function isPrivateIpAddress(address: string) {
if (isIP(address) === 4) return isPrivateIpv4(address);
if (isIP(address) !== 6) return true;
const normalized = address.toLowerCase();
if (normalized.startsWith("::ffff:")) {
return isPrivateIpv4(normalized.slice("::ffff:".length));
}
return (
normalized === "::" ||
normalized === "::1" ||
normalized.startsWith("fc") ||
normalized.startsWith("fd") ||
normalized.startsWith("fe8") ||
normalized.startsWith("fe9") ||
normalized.startsWith("fea") ||
normalized.startsWith("feb") ||
normalized.startsWith("ff")
);
}
async function validateTargetUrl(target: URL, trustedOrigin: string) {
if (target.username || target.password) return false;
if (target.origin === trustedOrigin) {
return target.protocol === "http:" || target.protocol === "https:";
}
if (target.protocol !== "https:") return false;
const hostname = target.hostname.toLowerCase().replace(/^\[|\]$/g, "");
if (
hostname === "localhost" ||
hostname.endsWith(".localhost") ||
hostname.endsWith(".local")
) {
return false;
}
if (isIP(hostname)) return !isPrivateIpAddress(hostname);
const addresses = await lookup(hostname, { all: true, verbatim: true }).catch(
() => []
);
return (
addresses.length > 0 &&
addresses.every((entry) => !isPrivateIpAddress(entry.address))
);
}
function normalizeRangeHeader(value: string | null) {
const range = String(value || "").trim();
if (!range) return null;
const match = range.match(/^bytes=(\d*)-(\d*)$/);
if (!match || (!match[1] && !match[2])) return "";
if (match[1] && match[2] && Number(match[1]) > Number(match[2])) return "";
return range;
}
function inferContentType(filename: string, declaredType: string) {
if (declaredType) return declaredType;
const lower = filename.toLowerCase();
if (lower.endsWith(".glb")) return "model/gltf-binary";
if (lower.endsWith(".gltf")) return "model/gltf+json";
if (lower.endsWith(".png")) return "image/png";
if (lower.endsWith(".webp")) return "image/webp";
if (lower.endsWith(".jpg") || lower.endsWith(".jpeg")) return "image/jpeg";
return "application/octet-stream";
}
function safeDownloadName(value: string) {
const normalized = value.replace(/[\r\n"\\/]/g, "_").trim() || "artifact";
return normalized.slice(0, 180);
}
async function fetchArtifact(params: {
request: Request;
initialUrl: URL;
method: "GET" | "HEAD";
range: string | null;
token: string;
trustedOrigin: string;
}) {
let target = params.initialUrl;
for (let redirects = 0; redirects <= MAX_REDIRECTS; redirects += 1) {
if (!(await validateTargetUrl(target, params.trustedOrigin))) {
throw new Error("不允许访问该文件地址");
}
const headers = new Headers({ Accept: "*/*" });
if (params.range) headers.set("Range", params.range);
if (target.origin === params.trustedOrigin && params.token) {
headers.set("Authorization", `Bearer ${params.token}`);
}
const response = await fetch(target, {
method: params.method,
headers,
cache: "no-store",
redirect: "manual",
signal: params.request.signal,
});
if (![301, 302, 303, 307, 308].includes(response.status)) {
return { response, target };
}
if (redirects === MAX_REDIRECTS) throw new Error("文件重定向次数过多");
const location = response.headers.get("location");
if (!location) throw new Error("文件重定向地址无效");
target = new URL(location, target);
}
throw new Error("文件地址无效");
}
async function handleRequest(request: Request, method: "GET" | "HEAD") {
const session = await auth().catch(() => null);
if (!session?.user?.id) {
return NextResponse.json({ error: "未登录" }, { status: 401 });
}
const url = new URL(request.url);
const parsed = querySchema.safeParse({
chatId: url.searchParams.get("chatId"),
messageId: url.searchParams.get("messageId"),
index: url.searchParams.get("index"),
download: url.searchParams.get("download") || undefined,
});
if (!parsed.success) {
return NextResponse.json({ error: "请求参数不正确" }, { status: 400 });
}
const chat = await getChatById({ id: parsed.data.chatId }).catch(() => null);
if (!chat || chat.userId !== session.user.id) {
return NextResponse.json({ error: "Forbidden" }, { status: 403 });
}
const rows = await getMessageById({ id: parsed.data.messageId }).catch(
() => []
);
const message = rows.find((row) => row.chatId === parsed.data.chatId);
const metadata = normalizeClawdTaskMetadata(message?.videoMetadata);
const artifact = metadata?.artifacts?.[parsed.data.index];
if (!message || !artifact) {
return NextResponse.json({ error: "文件不存在" }, { status: 404 });
}
const setting = await getSystemSetting({ key: OPENCLAW_CHAT_CONFIG_KEY }).catch(
() => null
);
const config = normalizeOpenClawChatConfig(setting?.value);
let target: URL;
let trustedOrigin: string;
try {
target = new URL(artifact.downloadUrl);
trustedOrigin = new URL(config.baseUrl).origin;
} catch {
return NextResponse.json({ error: "文件地址无效" }, { status: 400 });
}
const range = normalizeRangeHeader(request.headers.get("range"));
if (range === "") {
return NextResponse.json({ error: "Range 请求格式无效" }, { status: 400 });
}
try {
const { response, target: finalTarget } = await fetchArtifact({
request,
initialUrl: target,
method,
range,
token: config.token,
trustedOrigin,
});
if (!response.ok && response.status !== 206) {
return NextResponse.json(
{ error: `文件读取失败(HTTP ${response.status})` },
{ status: response.status >= 400 && response.status < 600 ? response.status : 502 }
);
}
const headers = new Headers();
for (const name of RESPONSE_HEADERS) {
const value = response.headers.get(name);
if (value) headers.set(name, value);
}
headers.set(
"content-type",
response.headers.get("content-type") ||
inferContentType(artifact.filename || finalTarget.pathname, artifact.mimeType)
);
headers.set("cache-control", "private, max-age=3600, must-revalidate");
headers.set("x-content-type-options", "nosniff");
if (parsed.data.download === "1") {
const filename = safeDownloadName(artifact.filename || artifact.displayName);
headers.set(
"content-disposition",
`attachment; filename*=UTF-8''${encodeURIComponent(filename)}`
);
}
return new Response(method === "HEAD" ? null : response.body, {
status: response.status,
headers,
});
} catch (error) {
console.warn("[Clawd Artifact] Proxy failed:", {
chatId: parsed.data.chatId,
messageId: parsed.data.messageId,
artifactIndex: parsed.data.index,
error: error instanceof Error ? error.message : String(error),
});
return NextResponse.json(
{ error: error instanceof Error ? error.message : "文件读取失败" },
{ status: 502 }
);
}
}
export async function GET(request: Request) {
return handleRequest(request, "GET");
}
export async function HEAD(request: Request) {
return handleRequest(request, "HEAD");
}

View File

@ -25,7 +25,7 @@ export async function GET(request: Request) {
const url = new URL(request.url);
const limit = Math.max(
1,
Math.min(100, Math.round(Number(url.searchParams.get("limit") || 50)))
Math.min(500, Math.round(Number(url.searchParams.get("limit") || 100)))
);
const chats = await getClawdChatsByUserId({

View File

@ -29,6 +29,7 @@ import {
OPENCLAW_CHAT_MEDIA_MAX_ATTACHMENTS,
type OpenClawChatMediaAttachment,
} from "@/lib/openclaw-chat/media-attachments";
import { extractOpenClawArtifacts } from "@/lib/openclaw-chat/artifacts";
import {
buildOpenClawDisplayMessage,
buildOpenClawMessageWithSpreadsheetMarkdown,
@ -298,7 +299,7 @@ async function executeClawdTask(params: ExecuteClawdTaskParams) {
params.spreadsheetMarkdownTruncated || undefined,
...params.metadata,
},
stream_progress: false,
stream_progress: !params.sensitiveInput,
sensitive_input: params.sensitiveInput,
};
@ -337,6 +338,9 @@ async function executeClawdTask(params: ExecuteClawdTaskParams) {
assistantMessageId: params.assistantMessageId,
answer,
requestId: getRequestId(payload),
artifacts: extractOpenClawArtifacts(payload, {
baseUrl: params.config.baseUrl,
}),
});
clearClawdRequestCancellation(params.cancelKey);
return;
@ -370,6 +374,7 @@ async function executeClawdTask(params: ExecuteClawdTaskParams) {
...target,
answer: outcome.answer,
requestId: outcome.requestId,
artifacts: outcome.artifacts,
});
} else if (outcome.status === "canceled") {
await markClawdTaskCanceled({
@ -576,7 +581,7 @@ export async function POST(request: Request) {
: "image/jpeg"),
}));
await persistPendingClawdTask({
const persistedTask = await persistPendingClawdTask({
chatId: parsed.id,
userMessageId,
assistantMessageId,
@ -587,6 +592,22 @@ export async function POST(request: Request) {
skillTagName,
processingHint,
});
if (
persistedTask &&
persistedTask.assistantMessageId !== assistantMessageId
) {
return NextResponse.json(
buildTaskResponse({
chatId: parsed.id,
userMessageId,
assistantMessageId: persistedTask.assistantMessageId,
status: persistedTask.status,
processingHint:
persistedTask.processingHint || DEFAULT_PROCESSING_HINT,
}),
{ status: 202, headers: { "Retry-After": "2" } }
);
}
const userProfile = await getUserById({ id: session.user.id }).catch(
() => null

View File

@ -75,6 +75,7 @@ async function reconcileDetachedOpenClawTask(params: {
...target,
answer: outcome.answer,
requestId: outcome.requestId,
artifacts: outcome.artifacts,
});
} else if (outcome.status === "failed") {
await markClawdTaskFailed({ ...target, error: outcome.error });

View File

@ -0,0 +1,45 @@
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { uploadFile } from "@/lib/storage/service";
import {
DIRECTOR_MAX_GLB_BYTES,
validateDirectorGlb,
} from "@/lib/video-canvas/director-glb";
export const runtime = "nodejs";
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ error: "未登录" }, { status: 401 });
try {
const form = await request.formData();
const file = form.get("file");
if (
!(file instanceof File) ||
!/\.glb$/i.test(file.name) ||
file.size > DIRECTOR_MAX_GLB_BYTES
)
return NextResponse.json(
{ error: "请上传 25 MiB 以内的 GLB 模型" },
{ status: 400 },
);
const buffer = await file.arrayBuffer();
try {
validateDirectorGlb(buffer);
} catch (error) {
return NextResponse.json(
{ error: error instanceof Error ? error.message : "无效 GLB" },
{ status: 400 },
);
}
const result = await uploadFile(
`director-${crypto.randomUUID()}.glb`,
buffer,
);
return NextResponse.json({ success: true, data: result });
} catch {
return NextResponse.json(
{ error: "模型上传失败,请重试" },
{ status: 500 },
);
}
}

View File

@ -1,62 +1,10 @@
import { executeCanvasImageResponse } from "@/lib/canvas/generate-image-response";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import {
createCanvasImageTask,
enqueueCanvasImageTask,
getCanvasImageTaskForUser,
} from "@/lib/canvas/image-task-queue";
import {
generateCanvasImage,
type CanvasGenerateImageRequest,
} from "@/lib/canvas/generate-image";
import { getPointsSystemConfig } from "@/lib/db/queries";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
import {
getImageToolCreditCost,
type ImageToolCreditFeature,
} from "@/lib/points/config";
import { getCanvasImageTaskForUser } from "@/lib/canvas/image-task-queue";
export const maxDuration = 1200;
interface GenerateImageRequest extends CanvasGenerateImageRequest {
async?: boolean;
}
function normalizeBillingFeature(
value: unknown
): Extract<ImageToolCreditFeature, "multi-view" | "repaint"> | null {
if (value === "multi-view" || value === "repaint") return value;
return null;
}
function toGenerateImagePayload(body: GenerateImageRequest): CanvasGenerateImageRequest {
return {
prompt: String(body.prompt || "").trim(),
negativePrompt:
typeof body.negativePrompt === "string" ? body.negativePrompt : undefined,
model: typeof body.model === "string" ? body.model : undefined,
aspectRatio:
typeof body.aspectRatio === "string" && body.aspectRatio.trim()
? body.aspectRatio.trim()
: "auto",
imageSize:
typeof body.imageSize === "string" && body.imageSize.trim()
? body.imageSize.trim()
: undefined,
imageCount:
typeof body.imageCount === "number" && Number.isFinite(body.imageCount)
? body.imageCount
: 1,
promptExtend:
typeof body.promptExtend === "boolean" ? body.promptExtend : undefined,
referenceImage:
typeof body.referenceImage === "string" ? body.referenceImage : undefined,
referenceImages: Array.isArray(body.referenceImages)
? body.referenceImages.map((item) => String(item || "").trim()).filter(Boolean)
: [],
};
}
export async function GET(request: Request) {
try {
const session = await auth();
@ -67,10 +15,7 @@ export async function GET(request: Request) {
const { searchParams } = new URL(request.url);
const taskId = String(searchParams.get("taskId") || "").trim();
if (!taskId) {
return NextResponse.json(
{ success: false, error: "缺少 taskId 参数" },
{ status: 400 }
);
return NextResponse.json({ success: false, error: "缺少 taskId 参数" }, { status: 400 });
}
const task = await getCanvasImageTaskForUser({ taskId, userId: session.user.id });
@ -99,105 +44,15 @@ export async function GET(request: Request) {
success: false,
error: error instanceof Error ? error.message : "查询任务失败",
},
{ status: 500 }
{ status: 500 },
);
}
}
export async function POST(request: Request) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json(
{ success: false, error: "未登录" },
{ status: 401 }
);
}
const body: GenerateImageRequest = await request.json();
const payload: CanvasGenerateImageRequest = {
...toGenerateImagePayload(body),
ownerKey: session.user.id,
};
if (!payload.prompt) {
return NextResponse.json(
{ success: false, error: "请输入提示词" },
{ status: 400 }
);
}
const billingFeature = normalizeBillingFeature(body.billingFeature);
const pointsCharge = billingFeature
? await (async () => {
const pointsConfig = await getPointsSystemConfig();
return prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, billingFeature),
reason:
billingFeature === "multi-view"
? "image_multi_view"
: "image_repaint",
modelId: "canvas-generate-image",
note:
billingFeature === "multi-view"
? "图片多视角生成消耗积分"
: "图片重绘消耗积分",
pointsConfig,
});
})()
: null;
if (pointsCharge && !pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
const asyncMode = body.async === true;
if (asyncMode) {
const task = await createCanvasImageTask({
userId: session.user.id,
request: payload as unknown as Record<string, unknown>,
});
await enqueueCanvasImageTask(task.id);
const currentTask = await getCanvasImageTaskForUser({
taskId: task.id,
userId: session.user.id,
});
if (pointsCharge?.chargeable) {
try {
await pointsCharge.consume({
note:
billingFeature === "multi-view"
? `图片多视角生成消耗积分(${task.id})`
: `图片重绘消耗积分(${task.id})`,
});
} catch (pointsError) {
console.error("[Points] Failed to consume canvas image task points:", pointsError);
}
}
return NextResponse.json({
success: true,
taskId: task.id,
status: currentTask?.status || task.status,
queueStatus: currentTask?.queueStatus,
});
}
const result = await generateCanvasImage(payload);
if (pointsCharge?.chargeable) {
try {
await pointsCharge.consume();
} catch (pointsError) {
console.error("[Points] Failed to consume canvas image points:", pointsError);
}
}
return NextResponse.json({
success: true,
data: result,
});
} catch (error) {
console.error("[Canvas Generate Image] Error:", error);
return NextResponse.json(
{ success: false, error: error instanceof Error ? error.message : "生成失败" },
{ status: 500 }
);
}
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
const body = await request.json().catch(() => ({}));
return executeCanvasImageResponse(body, session.user.id);
}

View File

@ -1,7 +1,8 @@
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { executeDirectVideoGeneration } from "@/lib/ai/tools/generate-video";
import { filterReachableRemoteUrls } from "@/lib/net/url-access";
import { probeRemoteUrl } from "@/lib/net/url-access";
import { validateVideoReferences } from "@/lib/video-canvas/reference-input";
import type { VideoGenerationMode, VideoRatio, VideoResolution } from "@/types/ai";
export const maxDuration = 300;
@ -10,6 +11,8 @@ interface GenerateVideoRequest {
prompt?: string;
mode?: VideoGenerationMode;
images?: string[];
audioUrls?: string[];
generateAudio?: boolean;
resolution?: VideoResolution;
ratio?: VideoRatio;
duration?: number;
@ -46,23 +49,18 @@ export async function POST(request: Request) {
);
}
const inputImages = Array.isArray(body.images)
? body.images.map((item) => String(item || "").trim()).filter(Boolean).slice(0, 4)
: [];
const { kept: reachableImages, dropped: unreachableImages } =
await filterReachableRemoteUrls(inputImages, { timeoutMs: 3000 });
if (inputImages.length > 0 && reachableImages.length === 0) {
return NextResponse.json(
{ success: false, error: "参考图片链接不可访问,请重新上传后再试" },
{ status: 400 }
);
const referenceCheck = await validateVideoReferences(body.images, url => probeRemoteUrl(url, { timeoutMs: 8000 }));
if (referenceCheck.error) {
return NextResponse.json({ success: false, error: referenceCheck.error }, { status: referenceCheck.status });
}
const reachableImages = referenceCheck.images;
const mode = body.mode || inferMode(reachableImages);
const result = await executeDirectVideoGeneration({
prompt,
mode,
audioUrls: Array.isArray(body.audioUrls) ? body.audioUrls.filter(url => typeof url === "string" && /^https?:\/\//i.test(url)).slice(0, 3) : undefined,
generateAudio: body.generateAudio,
images: reachableImages,
resolution: body.resolution,
ratio: body.ratio,
@ -90,13 +88,13 @@ export async function POST(request: Request) {
success: true,
data: {
url: result.videoUrl,
metadata: result.metadata || null,
metadata: { ...result.metadata, temporaryVideoUrl: result.videoUrl, storageStatus: result.metadata?.permanentVideoUrl === result.videoUrl ? "archived" : "temporary" },
modelDisplayName: result.modelDisplayName,
modelId: result.modelId,
modelRef: result.modelRef,
mode,
images: reachableImages,
droppedImages: unreachableImages,
droppedImages: [],
},
});
} catch (error) {

View File

@ -0,0 +1,128 @@
import { videoUpscaleSchema } from "@/lib/video-canvas/video-upscale-contract";
import { smartEditRequestSchema } from "@/lib/video-canvas/smart-edit/contract";
import { audioRequestSchema } from "@/lib/video-canvas/audio-contract";
import { canResumeCanvasTask } from "@/lib/video-canvas/task-recovery";
import { imageOperationSchema } from "@/lib/video-canvas/image-operation";
import { after, NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { z } from "zod";
import { canvasTaskRequestSchema } from "@/lib/video-canvas/task-contract";
import {
cancelCanvasGenerationTask,
createCanvasGenerationTask,
listCanvasGenerationTasks,
ownsTaskProject,
} from "@/lib/video-canvas/task-store";
import { runCanvasGenerationQueue } from "@/lib/video-canvas/task-worker";
export const runtime = "nodejs";
export const maxDuration = 900;
async function owner(projectId: unknown) {
const session = await auth();
if (!session?.user?.id) throw Object.assign(Error("未登录"), { status: 401 });
if (!z.string().uuid().safeParse(projectId).success)
throw Object.assign(Error("画布 ID 无效"), { status: 400 });
if (!(await ownsTaskProject(session.user.id, projectId as string)))
throw Object.assign(Error("画布不存在"), { status: 404 });
return session.user.id;
}
const fail = (error: unknown) =>
NextResponse.json(
{ success: false, error: error instanceof Error ? error.message : "任务操作失败" },
{ status: Number((error as any)?.status) || (error instanceof SyntaxError ? 400 : 500) },
);
const kick = (userId: string, projectId: string) =>
after(() =>
runCanvasGenerationQueue(userId, projectId).catch((error) =>
console.error("[Canvas tasks]", error),
),
);
export async function POST(request: Request) {
try {
const text = await request.text();
if (new TextEncoder().encode(text).byteLength > 262144)
throw Object.assign(Error("任务数据过大,请使用已上传的素材链接"), { status: 413 });
const parsed = canvasTaskRequestSchema.safeParse(JSON.parse(text));
if (!parsed.success) throw Object.assign(Error("任务参数不合法"), { status: 400 });
if(parsed.data.workflow)throw Object.assign(Error("编组任务必须使用整组提交接口"),{status:400});
if (
parsed.data.kind === "image-operation" &&
!imageOperationSchema.safeParse(parsed.data.params).success
)
throw Object.assign(Error("图片处理参数不合法"), { status: 400 });
if (
parsed.data.attachment &&
((parsed.data.attachment.type === "story-asset" && parsed.data.kind !== "image") ||
(parsed.data.attachment.type === "story-prompts" && parsed.data.kind !== "analysis"))
)
throw Object.assign(Error("脚本结果类型不匹配"), { status: 400 });
if (parsed.data.kind === "audio" && !audioRequestSchema.safeParse(parsed.data.params).success)
throw Object.assign(Error("音频参数不合法"), { status: 400 });
if (
parsed.data.kind === "smart-edit" &&
!smartEditRequestSchema.safeParse(parsed.data.params).success
)
throw Object.assign(Error("剪辑参数无效"), { status: 400 });
if (parsed.data.kind === "video-upscale" && !videoUpscaleSchema.safeParse(parsed.data.params).success) throw Object.assign(Error("视频高清参数无效"), {status:400});
const userId = await owner(parsed.data.projectId);
const task = await createCanvasGenerationTask(userId, parsed.data);
kick(userId, parsed.data.projectId);
return NextResponse.json({ success: true, task }, { status: 202 });
} catch (error) {
return fail(error);
}
}
export async function GET(request: Request) {
try {
const projectId = new URL(request.url).searchParams.get("projectId") || "";
const userId = await owner(projectId);
const tasks = await listCanvasGenerationTasks(userId, projectId);
if (tasks.some((task) => task.status === "pending" || canResumeCanvasTask(task)))
kick(userId, projectId);
return NextResponse.json(
{ success: true, tasks },
{ headers: { "Cache-Control": "no-store" } },
);
} catch (error) {
return fail(error);
}
}
export async function DELETE(request: Request) {
try {
const { projectId, taskId } = await request.json();
const userId = await owner(projectId);
if (!z.string().uuid().safeParse(taskId).success)
throw Object.assign(Error("任务 ID 无效"), { status: 400 });
const result = await cancelCanvasGenerationTask(userId, projectId, taskId);
if (!result.task)
return NextResponse.json({ success: false, error: "任务不存在" }, { status: 404 });
if (!result.ok)
return NextResponse.json(
{
success: false,
error:
result.task.status === "running"
? "任务已提交供应商,当前线路不支持中止。结果会继续保存,请勿重复生成。"
: "任务已结束或状态待核实,请先刷新任务状态。",
task: result.task,
},
{ status: 409 },
);
return NextResponse.json({ success: true, task: result.task });
} catch (error) {
return fail(error);
}
}
export async function PATCH(request: Request) {
try {
const { projectId, taskId, nodeId } = await request.json();
const userId = await owner(projectId);
if (!z.string().uuid().safeParse(taskId).success || typeof nodeId !== "string")
return NextResponse.json({ success: false }, { status: 400 });
const { hideCanvasGenerationNode } = await import("@/lib/video-canvas/task-store");
const success = await hideCanvasGenerationNode(userId, projectId, taskId, nodeId);
return NextResponse.json({ success }, { status: success ? 200 : 404 });
} catch (error) {
return fail(error);
}
}

View File

@ -0,0 +1,39 @@
import { auth } from "@/app/(auth)/auth";
import { after, NextResponse } from "next/server";
import { canvasTaskRequestSchema } from "@/lib/video-canvas/task-contract";
import {
createCanvasGroupTasks,
ownsTaskProject,
} from "@/lib/video-canvas/task-store";
import { runCanvasGenerationQueue } from "@/lib/video-canvas/task-worker";
export const runtime = "nodejs",
maxDuration = 900;
export async function POST(request: Request) {
try {
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ error: "未登录" }, { status: 401 });
const text = await request.text();
if (Buffer.byteLength(text) > 1048576)
return NextResponse.json({ error: "编组数据过大" }, { status: 413 });
const parsed = canvasTaskRequestSchema
.array()
.min(1)
.max(40)
.safeParse(JSON.parse(text));
if (!parsed.success)
return NextResponse.json({ error: "编组参数无效" }, { status: 400 });
const projectId = parsed.data[0].projectId,
userId = session.user.id;
if (!(await ownsTaskProject(userId, projectId)))
return NextResponse.json({ error: "画布不存在" }, { status: 404 });
const tasks = await createCanvasGroupTasks(userId, parsed.data);
after(() => runCanvasGenerationQueue(userId, projectId));
return NextResponse.json({ success: true, tasks }, { status: 202 });
} catch (e) {
return NextResponse.json(
{ error: e instanceof Error ? e.message : "编组执行失败" },
{ status: Number((e as any)?.status) || 400 }
);
}
}

View File

@ -0,0 +1,40 @@
import { auth } from "@/app/(auth)/auth";
import { db } from "@/lib/db";
import { canvasProject } from "@/lib/db/schema";
import { eq, and, sql, desc } from "drizzle-orm";
import { NextResponse } from "next/server";
export async function GET() {
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ error: "未登录" }, { status: 401 });
const rows = await db
.select({
projectId: canvasProject.id,
title: canvasProject.title,
templates: sql<any[]>`${canvasProject.canvasData}->'groupTemplates'`,
})
.from(canvasProject)
.where(
and(
eq(canvasProject.userId, session.user.id),
eq(canvasProject.workspaceType, "video"),
sql`${canvasProject.canvasData} ? 'groupTemplates'`
)
)
.orderBy(desc(canvasProject.updatedAt))
.limit(100);
return NextResponse.json(
{
templates: rows.flatMap((row) =>
Array.isArray(row.templates)
? row.templates.map((template) => ({
...template,
projectId: row.projectId,
projectTitle: row.title,
}))
: []
),
},
{ headers: { "Cache-Control": "no-store" } }
);
}

View File

@ -0,0 +1,143 @@
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import {
getCanvasLayoutReplaceBatchActiveTaskId,
getCanvasLayoutReplaceBatchForUser,
markCanvasLayoutReplaceBatchItemsAdded,
retryCanvasLayoutReplaceBatchItem,
} from "@/lib/canvas/layout-replace-batch";
import {
enqueueCanvasImageTask,
ensureCanvasImageTaskActive,
} from "@/lib/canvas/image-task-queue";
function normalizeUuid(value: unknown) {
const id = String(value || "").trim();
return /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(id)
? id
: "";
}
export async function GET(
_request: Request,
{ params }: { params: Promise<{ batchId: string }> }
) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
}
const batchId = normalizeUuid((await params).batchId);
if (!batchId) {
return NextResponse.json({ success: false, error: "批次ID无效" }, { status: 400 });
}
let batch = await getCanvasLayoutReplaceBatchForUser({
batchId,
userId: session.user.id,
});
if (!batch) {
return NextResponse.json({ success: false, error: "批次不存在" }, { status: 404 });
}
const activeTaskId = getCanvasLayoutReplaceBatchActiveTaskId(batch);
if (activeTaskId) {
await ensureCanvasImageTaskActive({
taskId: activeTaskId,
userId: session.user.id,
});
batch = await getCanvasLayoutReplaceBatchForUser({
batchId,
userId: session.user.id,
});
}
return NextResponse.json({ success: true, batch });
} catch (error) {
console.error("[Canvas Layout Replace Batch][Detail] Error:", error);
return NextResponse.json(
{
success: false,
error: error instanceof Error ? error.message : "查询批次失败",
},
{ status: 500 }
);
}
}
export async function PATCH(
request: Request,
{ params }: { params: Promise<{ batchId: string }> }
) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
}
const batchId = normalizeUuid((await params).batchId);
const body = await request.json().catch(() => ({}));
if (body?.action === "retry") {
const itemId = normalizeUuid(body?.itemId);
if (!batchId || !itemId) {
return NextResponse.json(
{ success: false, error: "缺少需要重试的结果" },
{ status: 400 }
);
}
const taskId = await retryCanvasLayoutReplaceBatchItem({
batchId,
itemId,
userId: session.user.id,
});
if (!taskId) {
return NextResponse.json(
{ success: false, error: "该结果当前无法重试" },
{ status: 409 }
);
}
await enqueueCanvasImageTask(taskId);
return NextResponse.json({ success: true, taskId });
}
const items = Array.isArray(body?.items)
? body.items
.map((item: unknown) => {
const record =
item && typeof item === "object" && !Array.isArray(item)
? (item as Record<string, unknown>)
: {};
return {
itemId: normalizeUuid(record.itemId),
canvasNodeId: String(record.canvasNodeId || "").trim(),
};
})
.filter(
(item: { itemId: string; canvasNodeId: string }) =>
Boolean(item.itemId && item.canvasNodeId)
)
: [];
if (!batchId || items.length === 0) {
return NextResponse.json(
{ success: false, error: "缺少已添加的结果信息" },
{ status: 400 }
);
}
const updated = await markCanvasLayoutReplaceBatchItemsAdded({
batchId,
userId: session.user.id,
items,
});
if (!updated) {
return NextResponse.json({ success: false, error: "批次不存在" }, { status: 404 });
}
return NextResponse.json({ success: true });
} catch (error) {
console.error("[Canvas Layout Replace Batch][PATCH] Error:", error);
return NextResponse.json(
{
success: false,
error: error instanceof Error ? error.message : "更新批次失败",
},
{ status: 500 }
);
}
}

View File

@ -0,0 +1,201 @@
import { and, eq } from "drizzle-orm";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import {
CANVAS_LAYOUT_REPLACE_BATCH_LIMIT,
createCanvasLayoutReplaceBatch,
getCanvasLayoutReplaceBatchActiveTaskId,
getLatestCanvasLayoutReplaceBatchForUser,
type CanvasLayoutReplaceBatchSource,
} from "@/lib/canvas/layout-replace-batch";
import {
enqueueCanvasImageTask,
ensureCanvasImageTaskActive,
} from "@/lib/canvas/image-task-queue";
import { db } from "@/lib/db";
import { canvasProject } from "@/lib/db/schema";
import { getPointsSystemConfig } from "@/lib/db/queries";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
import { getImageToolCreditCost } from "@/lib/points/config";
export const maxDuration = 1200;
function normalizeHttpUrl(value: unknown) {
const url = String(value || "").trim();
return /^https?:\/\//i.test(url) ? url : "";
}
function normalizeUuid(value: unknown) {
const id = String(value || "").trim();
return /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(id)
? id
: "";
}
function normalizeSources(value: unknown): CanvasLayoutReplaceBatchSource[] {
if (!Array.isArray(value)) return [];
const seen = new Set<string>();
const sources: CanvasLayoutReplaceBatchSource[] = [];
for (const item of value) {
if (!item || typeof item !== "object" || Array.isArray(item)) continue;
const record = item as Record<string, unknown>;
const sourceImageUrl = normalizeHttpUrl(record.sourceImageUrl);
if (!sourceImageUrl || seen.has(sourceImageUrl)) continue;
seen.add(sourceImageUrl);
sources.push({
sourceImageUrl,
sourceKind: record.sourceKind === "upload" ? "upload" : "canvas",
sourceNodeId: String(record.sourceNodeId || "").trim() || undefined,
sourceTitle: String(record.sourceTitle || "").trim().slice(0, 256) || undefined,
});
if (sources.length >= CANVAS_LAYOUT_REPLACE_BATCH_LIMIT) break;
}
return sources;
}
async function getOwnedProject(projectId: string, userId: string) {
const [project] = await db
.select({ id: canvasProject.id })
.from(canvasProject)
.where(and(eq(canvasProject.id, projectId), eq(canvasProject.userId, userId)))
.limit(1);
return project || null;
}
export async function GET(request: Request) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
}
const { searchParams } = new URL(request.url);
const projectId = normalizeUuid(searchParams.get("projectId"));
const sourceNodeId = String(searchParams.get("sourceNodeId") || "").trim();
if (!projectId) {
return NextResponse.json(
{ success: false, error: "缺少有效的画布项目" },
{ status: 400 }
);
}
if (!(await getOwnedProject(projectId, session.user.id))) {
return NextResponse.json({ success: false, error: "画布不存在" }, { status: 404 });
}
let batch = await getLatestCanvasLayoutReplaceBatchForUser({
projectId,
userId: session.user.id,
sourceNodeId: sourceNodeId || undefined,
});
const activeTaskId = batch
? getCanvasLayoutReplaceBatchActiveTaskId(batch)
: null;
if (activeTaskId) {
await ensureCanvasImageTaskActive({
taskId: activeTaskId,
userId: session.user.id,
});
batch = await getLatestCanvasLayoutReplaceBatchForUser({
projectId,
userId: session.user.id,
sourceNodeId: sourceNodeId || undefined,
});
}
return NextResponse.json({ success: true, batch: batch || null });
} catch (error) {
console.error("[Canvas Layout Replace Batch][GET] Error:", error);
return NextResponse.json(
{
success: false,
error: error instanceof Error ? error.message : "查询批次失败",
},
{ status: 500 }
);
}
}
export async function POST(request: Request) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
}
const body = await request.json().catch(() => ({}));
const projectId = normalizeUuid(body?.projectId);
const referenceImageUrl = normalizeHttpUrl(body?.referenceImageUrl);
const sources = normalizeSources(body?.sources);
if (!projectId || !(await getOwnedProject(projectId, session.user.id))) {
return NextResponse.json({ success: false, error: "画布不存在" }, { status: 404 });
}
if (sources.length === 0) {
return NextResponse.json(
{ success: false, error: "请至少添加一张待处理图片" },
{ status: 400 }
);
}
if (!referenceImageUrl) {
return NextResponse.json(
{ success: false, error: "请上传目标模版图" },
{ status: 400 }
);
}
const pointsConfig = await getPointsSystemConfig();
const perImageCost = getImageToolCreditCost(pointsConfig, "repaint");
const totalCost = perImageCost * sources.length;
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: totalCost,
reason: "image_repaint",
modelId: "canvas-layout-replace-batch",
note: `批量排版替换 ${sources.length} 张`,
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
const created = await createCanvasLayoutReplaceBatch({
userId: session.user.id,
projectId,
sources,
referenceImageUrl,
supplementalPrompt:
typeof body?.supplementalPrompt === "string"
? body.supplementalPrompt
: undefined,
imageSize: typeof body?.imageSize === "string" ? body.imageSize : "2K",
model: typeof body?.model === "string" ? body.model : undefined,
});
await enqueueCanvasImageTask(created.firstTaskId);
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume({
note: `批量排版替换 ${sources.length} 张(${created.batchId})`,
});
} catch (error) {
console.error("[Points] Failed to consume layout batch points:", error);
}
}
return NextResponse.json({
success: true,
batchId: created.batchId,
firstTaskId: created.firstTaskId,
sourceCount: sources.length,
pointsCost: totalCost,
});
} catch (error) {
console.error("[Canvas Layout Replace Batch][POST] Error:", error);
return NextResponse.json(
{
success: false,
error: error instanceof Error ? error.message : "创建批次失败",
},
{ status: 500 }
);
}
}

View File

@ -0,0 +1,124 @@
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import {
createCanvasImageTask,
enqueueCanvasImageTask,
} from "@/lib/canvas/image-task-queue";
import { getPointsSystemConfig } from "@/lib/db/queries";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
import { getImageToolCreditCost } from "@/lib/points/config";
import type { CanvasLayoutReplaceTaskRequest } from "@/lib/canvas/layout-replace-task";
export const maxDuration = 1200;
function normalizeHttpUrl(value: unknown) {
const url = String(value || "").trim();
return /^https?:\/\//i.test(url) ? url : "";
}
function normalizeUuid(value: unknown) {
const uuid = String(value || "").trim();
return /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(uuid)
? uuid
: undefined;
}
export async function POST(request: Request) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json(
{ success: false, error: "未登录" },
{ status: 401 },
);
}
const body = await request.json().catch(() => ({}));
const sourceImageUrl = normalizeHttpUrl(body?.sourceImageUrl);
const referenceImageUrl = normalizeHttpUrl(body?.referenceImageUrl);
if (!sourceImageUrl) {
return NextResponse.json(
{ success: false, error: "缺少当前产品图片" },
{ status: 400 },
);
}
if (!referenceImageUrl) {
return NextResponse.json(
{ success: false, error: "缺少排版参考图" },
{ status: 400 },
);
}
const pointsConfig = await getPointsSystemConfig();
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, "repaint"),
reason: "image_repaint",
modelId: "canvas-layout-replace",
note: "排版替换消耗积分",
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, {
status: pointsCharge.status,
});
}
const taskRequest: CanvasLayoutReplaceTaskRequest = {
taskKind: "canvas-layout-replace",
sourceImageUrl,
referenceImageUrl,
supplementalPrompt:
typeof body?.supplementalPrompt === "string"
? body.supplementalPrompt
: undefined,
model: typeof body?.model === "string" ? body.model : undefined,
imageSize:
typeof body?.imageSize === "string" && body.imageSize.trim()
? body.imageSize.trim()
: "2K",
sourceNodeId:
typeof body?.sourceNodeId === "string" ? body.sourceNodeId : undefined,
agentProjectId: normalizeUuid(body?.agentProjectId),
referenceLabel:
typeof body?.referenceLabel === "string" ? body.referenceLabel : undefined,
debugMeta:
body?.debugMeta &&
typeof body.debugMeta === "object" &&
!Array.isArray(body.debugMeta)
? body.debugMeta
: undefined,
};
const task = await createCanvasImageTask({
userId: session.user.id,
request: taskRequest as unknown as Record<string, unknown>,
});
await enqueueCanvasImageTask(task.id);
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume({
note: `排版替换消耗积分(${task.id})`,
});
} catch (pointsError) {
console.error("[Points] Failed to consume canvas layout replace points:", pointsError);
}
}
return NextResponse.json({
success: true,
taskId: task.id,
status: task.status,
});
} catch (error) {
console.error("[Canvas Layout Replace] Error:", error);
return NextResponse.json(
{
success: false,
error: error instanceof Error ? error.message : "排版替换提交失败",
},
{ status: 500 },
);
}
}

View File

@ -0,0 +1,65 @@
import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { promisify } from "node:util";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { uploadFile } from "@/lib/storage/service";
export const runtime = "nodejs";
export const maxDuration = 180;
const probe = promisify(execFile);
const running = new Set<string>();
const formats: Record<string, string> = {
"video/mp4": "mp4", "video/quicktime": "mov", "video/webm": "webm",
"audio/mpeg": "mp3", "audio/mp3": "mp3", "audio/mp4": "m4a", "audio/x-m4a": "m4a",
"audio/wav": "wav", "audio/x-wav": "wav", "audio/ogg": "ogg", "audio/flac": "flac", "audio/webm": "webm",
};
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
if (running.has(session.user.id) || running.size >= 4) return NextResponse.json({ success: false, error: "正在上传,请稍后重试" }, { status: 429 });
running.add(session.user.id);
let dir = "";
try {
const form = await request.formData();
const file = form.get("file");
if (!(file instanceof File) || !formats[file.type]) return NextResponse.json({ success: false, error: "请上传 MP4、MOV、WebM 视频,或 MP3、M4A、WAV、OGG、FLAC 音频" }, { status: 400 });
if (!file.size || file.size > 125 * 1024 * 1024) return NextResponse.json({ success: false, error: "素材必须非空且不超过 125 MiB" }, { status: 413 });
dir = await mkdtemp(join(tmpdir(), "canvas-upload-"));
const input = join(dir, "input");
const buffer = await file.arrayBuffer();
await writeFile(input, Buffer.from(buffer));
let stdout: string;
try {
({ stdout } = await probe("ffprobe", ["-v", "error", "-protocol_whitelist", "file", "-format_whitelist", "mov,matroska,webm,mp3,wav,ogg,flac", "-show_streams", "-show_format", "-of", "json", input], { timeout: 15000, maxBuffer: 1024 * 1024 }));
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
return NextResponse.json({ success: false, error: "服务器缺少 ffprobe 视频解析组件,请联系管理员安装 FFmpeg" }, { status: 503 });
}
return NextResponse.json({ success: false, error: "视频文件无法解析,请确认素材完整且格式受支持" }, { status: 400 });
}
const info = JSON.parse(stdout);
const kind = file.type.startsWith("video/") ? "video" : "audio";
const stream = info.streams?.find((item: { codec_type: string }) => item.codec_type === kind);
const duration = Number(info.format?.duration);
if (!stream || !Number.isFinite(duration) || duration <= 0 || duration > 600) return NextResponse.json({ success: false, error: "请上传内容完整、10 分钟以内的音视频" }, { status: 400 });
let uploaded;
try {
uploaded = await uploadFile(`canvas-${randomUUID()}.${formats[file.type]}`, buffer);
} catch (error) {
console.error("[Canvas media upload] Storage unavailable:", error);
return NextResponse.json({ success: false, error: "媒体存储暂不可用,请稍后重试" }, { status: 503 });
}
return NextResponse.json({ success: true, data: { url: uploaded.url, duration, width: stream.width, height: stream.height } });
} catch (error) {
console.error("[Canvas media upload] Failed:", error);
return NextResponse.json({ success: false, error: "素材上传失败,请确认文件完整或稍后重试" }, { status: 400 });
} finally {
running.delete(session.user.id);
if (dir) await rm(dir, { recursive: true, force: true });
}
}

View File

@ -0,0 +1,61 @@
import { generateText } from "ai";
import { createLogger } from "@/lib/observability/logger";
const logger = createLogger("canvas-prompt-tools");
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { getLanguageModelById } from "@/lib/ai/dynamic-provider";
import { getEnabledAiModelsWithProvider, getSystemSetting } from "@/lib/db/queries";
import { VIDEO_CANVAS_FUNCTION_CONFIG_KEY, normalizeVideoCanvasFunctionConfig } from "@/lib/video-canvas/config";
import { promptToolErrorMessage, promptToolFunctionKey, promptToolSchema, selectPromptToolModel } from "@/lib/video-canvas/prompt-tools";
export const runtime = "nodejs";
export const maxDuration = 120;
const running = new Set<string>();
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
const parsed = promptToolSchema.safeParse(await request.json().catch(() => null));
if (!parsed.success) return NextResponse.json({ success: false, error: parsed.error.issues[0]?.message || "参数无效" }, { status: 400 });
if (running.has(session.user.id)) return NextResponse.json({ success: false, error: "提示词处理中,请稍候" }, { status: 429 });
running.add(session.user.id);
try {
const [setting, available] = await Promise.all([
getSystemSetting({ key: VIDEO_CANVAS_FUNCTION_CONFIG_KEY }),
getEnabledAiModelsWithProvider(false),
]);
const config = normalizeVideoCanvasFunctionConfig(setting?.value);
const definition = config.functions[promptToolFunctionKey(parsed.data.mode)];
if (!config.enabled || !definition?.enabled) return NextResponse.json({ success: false, error: "此提示词功能已停用" }, { status: 403 });
const requested = parsed.data.model && parsed.data.model !== "auto"
? parsed.data.model : definition.model && definition.model !== "auto"
? definition.model : config.defaultMultimodalModel;
let selected;
try {
selected = selectPromptToolModel(available
.filter(item => item.provider.isEnabled && item.model.isEnabled && !item.model.isSubModel)
.map(item => ({ ...item.model, providerType: item.provider.type })), requested);
} catch (error) {
return NextResponse.json({ success: false, error: (error as Error).message }, { status: 400 });
}
const model = await getLanguageModelById(selected.id);
if (!model) return NextResponse.json({ success: false, error: "提示词模型初始化失败" }, { status: 502 });
const { text } = await generateText({
model,
system: definition.prompt,
prompt: parsed.data.prompt,
temperature: parsed.data.mode === "translate" ? 0.1 : 0.4,
maxOutputTokens: 4000,
maxRetries: 0,
abortSignal: AbortSignal.any([request.signal, AbortSignal.timeout(90_000)]),
});
if (!text.trim()) throw new Error("empty model output");
return NextResponse.json({ success: true, data: { text: text.trim(), model: selected.id, modelName: selected.displayName || selected.modelId } });
} catch (error) {
const detail = error as { name?: string; statusCode?: number; cause?: { code?: string }; data?: { error?: { code?: string } } };
logger.warn("Prompt tool request failed", { name: detail?.name, statusCode: detail?.statusCode, code: detail?.data?.error?.code || detail?.cause?.code });
return NextResponse.json({ success: false, error: promptToolErrorMessage(error) }, { status: 502 });
} finally {
running.delete(session.user.id);
}
}

View File

@ -100,6 +100,8 @@ export async function POST(request: NextRequest) {
.insert(canvasProject)
.values({
userId: session.user.id,
createdAt: new Date(),
updatedAt: new Date(),
title: title || getDefaultWorkspaceTitle(workspaceType),
description,
workspaceType,

View File

@ -0,0 +1,68 @@
import { remoteAudioCapabilities } from "@/lib/video-canvas/smart-edit/remote-audio";
import { auth } from "@/app/(auth)/auth";
import { NextResponse } from "next/server";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import path from "node:path";
import { getEnabledAiModelsWithProvider } from "@/lib/db/queries";
import { isCanvasVisionModel } from "@/lib/video-canvas/model-matching";
import { remoteRenderConfig } from "@/lib/video-canvas/smart-edit/remote-render";
export async function GET() {
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ error: "未登录" }, { status: 401 });
try {
let capabilities = {
separation: false,
scenes: false,
beats: false,
asr: false,
ffmpeg: false,
ffprobe: false,
};
try {
const { stdout } = await promisify(execFile)(
process.env.SMART_EDIT_PYTHON || "python3",
[path.resolve("services/smart-edit/analyze.py"), "--check"],
{ timeout: 10000, maxBuffer: 65536 }
);
capabilities = JSON.parse(stdout);
} catch {}
const audio = await remoteAudioCapabilities();
if (audio) capabilities = { ...capabilities, ...audio };
let renderAvailable = capabilities.ffmpeg;
const remote = remoteRenderConfig();
if (remote) {
try {
const response = await fetch(remote.base + "/health", {
headers: { authorization: "Bearer " + remote.token },
signal: AbortSignal.timeout(3000),
redirect: "error",
cache: "no-store",
});
renderAvailable =
response.ok && (await response.json()).status === "ok";
} catch {
renderAvailable = false;
}
}
const models = (await getEnabledAiModelsWithProvider(false))
.filter((item) =>
isCanvasVisionModel({ ...item.model, providerType: item.provider.type })
)
.map(({ model }) => ({
id: model.id,
name: model.displayName,
modelId: model.modelId,
}));
return NextResponse.json(
{ capabilities: { ...capabilities, render: renderAvailable }, models },
{ headers: { "Cache-Control": "no-store" } }
);
} catch {
return NextResponse.json(
{ error: "剪辑服务配置读取失败" },
{ status: 503 }
);
}
}

View File

@ -0,0 +1,22 @@
import { auth } from "@/app/(auth)/auth";
import { captionFont } from "@/lib/video-canvas/smart-edit/fonts";
import { captionFontFile } from "@/lib/video-canvas/smart-edit/font-files";
import { readFile } from "node:fs/promises";
export const runtime = "nodejs";
export const maxDuration = 180;
export async function GET(_request: Request, { params }: { params: Promise<{ id: string }> }) {
if (!(await auth())?.user?.id) return Response.json({ error: "未登录" }, { status: 401 });
const { id } = await params;
if (!captionFont(id)) return Response.json({ error: "字体不存在" }, { status: 404 });
try {
const data = await readFile(await captionFontFile(id));
return new Response(new Uint8Array(data), {
headers: {
"Content-Type": "font/ttf",
"Cache-Control": "private, max-age=31536000, immutable",
},
});
} catch {
return Response.json({ error: "字体加载失败,请检查字体服务配置后重试" }, { status: 503 });
}
}

View File

@ -29,8 +29,9 @@ export const runtime = "nodejs";
export const maxDuration = 1200;
const logger = createLogger("canvas-text-edit");
const TEXT_EDIT_SOURCE_ARCHIVE_TIMEOUT_MS = 20_000;
const TEXT_EDIT_SOURCE_ARCHIVE_TIMEOUT_MS = 120_000;
const TEXT_EDIT_MODEL_FALLBACK_LIMIT = 3;
const TEXT_EDIT_GENERATION_TIMEOUT_MS = 180_000;
const textItemSchema = z.object({
id: z.string().min(1),
@ -68,6 +69,13 @@ const SUPPORTED_TEXT_EDIT_PROVIDER_TYPES = new Set([
type TextEditModelCandidate = Awaited<ReturnType<typeof getEnabledAiModelsWithProvider>>[number];
function isExplicitTextEditModel(candidate: TextEditModelCandidate) {
const capabilities = candidate.model.capabilities as Record<string, unknown> | null;
const kind = String(capabilities?.kind || "").toLowerCase();
const modelId = String(candidate.model.modelId || "").toLowerCase();
return kind === "image-edit" || /(?:^|[/_-])edit(?:$|[/_-])/.test(modelId);
}
function getTextEditModelCandidates(
models: Awaited<ReturnType<typeof getEnabledAiModelsWithProvider>>
) {
@ -82,16 +90,19 @@ function getTextEditModelCandidates(
}) &&
item.model.isEnabled &&
item.provider.isEnabled &&
(!item.model.excludeFromFailover || item.model.isDefaultTextEditModel) &&
SUPPORTED_TEXT_EDIT_PROVIDER_TYPES.has(item.provider.type),
)
.sort((a, b) => {
const aScore =
(a.model.isDefaultTextEditModel ? 100 : 0) +
(isExplicitTextEditModel(a) ? 30 : 0) +
(a.model.isDefault ? 10 : 0) +
(a.model.modelId?.toLowerCase().includes("seedream") ? 5 : 0) +
(a.model.displayName?.includes("火山") ? 3 : 0);
const bScore =
(b.model.isDefaultTextEditModel ? 100 : 0) +
(isExplicitTextEditModel(b) ? 30 : 0) +
(b.model.isDefault ? 10 : 0) +
(b.model.modelId?.toLowerCase().includes("seedream") ? 5 : 0) +
(b.model.displayName?.includes("火山") ? 3 : 0);
@ -220,6 +231,13 @@ async function resolveTextEditSourceImageUrl(imageUrl: string) {
} catch (error) {
logger.warn("source_image_archive_failed", {
error: error instanceof Error ? error.message : String(error),
sourceHost: (() => {
try {
return new URL(imageUrl).host;
} catch {
return "invalid-url";
}
})(),
});
return imageUrl;
}
@ -495,6 +513,7 @@ async function generateTextEditImageWithFailover(params: {
],
imageParams: {
imageCount: 1,
timeoutMs: TEXT_EDIT_GENERATION_TIMEOUT_MS,
},
};
if (selected.provider.type === "atlascloud" || selected.provider.type === "vip98k") {

View File

@ -0,0 +1,39 @@
import { auth } from "@/app/(auth)/auth";
import { upscaleConfig } from "@/lib/video-canvas/video-upscale-remote";
import { NextResponse } from "next/server";
export async function GET() {
if (!(await auth())?.user?.id)
return NextResponse.json({ error: "未登录" }, { status: 401 });
try {
const config = upscaleConfig();
if (!config)
return NextResponse.json({
available: false,
message: "视频高清服务尚未接通",
});
const response = await fetch(config.base + "/health", {
headers: { authorization: "Bearer " + config.token },
signal: AbortSignal.timeout(5000),
redirect: "error",
cache: "no-store",
});
if (!response.ok) throw Error("unavailable");
const data = await response.json();
return NextResponse.json(
{
available: data.status === "ok",
engine: "Real-ESRGAN",
resolutions: ["1080P", "2K", "4K"],
maxScale: 4,
maxSeconds: 120,
queued: data.queued,
},
{ headers: { "Cache-Control": "no-store" } }
);
} catch {
return NextResponse.json(
{ available: false, message: "视频高清服务暂时不可用,请稍后重试" },
{ headers: { "Cache-Control": "no-store" } }
);
}
}

View File

@ -0,0 +1,23 @@
import { NextResponse } from "next/server";
import { z } from "zod";
import { auth } from "@/app/(auth)/auth";
import { archiveCanvasVideo } from "@/lib/video-canvas/archive-video";
export const runtime = "nodejs";
export const maxDuration = 300;
const running = new Set<string>();
const schema = z.object({ url: z.string().url().max(12000) });
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
const parsed = schema.safeParse(await request.json().catch(() => null));
if (!parsed.success) return NextResponse.json({ success: false, error: "视频地址无效" }, { status: 400 });
if (running.has(session.user.id)) return NextResponse.json({ success: false, error: "视频正在保存,请稍候" }, { status: 429 });
running.add(session.user.id);
try {
const url = await archiveCanvasVideo(parsed.data.url);
return NextResponse.json({ success: true, data: { url, storageStatus: "archived", permanentVideoUrl: url } });
} catch {
return NextResponse.json({ success: false, error: "视频转存失败,原视频仍保留;请稍后重试保存,无需重新生成" }, { status: 502 });
} finally { running.delete(session.user.id); }
}

View File

@ -0,0 +1,96 @@
import { auth } from "@/app/(auth)/auth";
import { NextResponse } from "next/server";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { randomUUID } from "node:crypto";
import { uploadFile } from "@/lib/storage/service";
import { downloadCanvasMedia } from "@/lib/video-canvas/download-media";
import { processVideoFile } from "@/lib/video-canvas/media-processing";
import { videoProcessSchema } from "@/lib/video-canvas/media-contract";
export const runtime = "nodejs";
export const maxDuration = 300;
const running = new Set<string>();
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id)
return NextResponse.json({ success: false, error: "未登录" }, { status: 401 });
const parsed = videoProcessSchema.safeParse(await request.json().catch(() => null));
if (!parsed.success)
return NextResponse.json({ success: false, error: "视频处理参数无效" }, { status: 400 });
if (running.has(session.user.id) || running.size >= 2)
return NextResponse.json(
{ success: false, error: "视频处理中,请稍后再提交" },
{ status: 429 },
);
running.add(session.user.id);
let dir = "";
try {
dir = await mkdtemp(join(tmpdir(), "canvas-video-"));
const input = join(dir, "input");
await downloadCanvasMedia(parsed.data.url, input);
const save = async (name: string) => {
const buffer = await readFile(join(dir, name));
return (
await uploadFile(`canvas-video-${randomUUID()}-${name}`, new Uint8Array(buffer).buffer)
).url;
};
if (parsed.data.operation === "frame" || parsed.data.operation === "slow") {
const name = parsed.data.operation === "frame" ? "frame.png" : "slow.mp4";
const metadata = await processVideoFile(
input,
join(dir, name),
parsed.data.operation,
undefined,
undefined,
parsed.data,
);
return NextResponse.json({ success: true, data: { url: await save(name), ...metadata } });
}
if (parsed.data.operation === "trim") {
const metadata = await processVideoFile(
input,
join(dir, "clip.mp4"),
"trim",
undefined,
parsed.data.range,
);
return NextResponse.json({
success: true,
data: { url: await save("clip.mp4"), ...metadata },
});
}
if (parsed.data.operation === "crop") {
const metadata = await processVideoFile(
input,
join(dir, "crop.mp4"),
"crop",
parsed.data.rect,
);
return NextResponse.json({
success: true,
data: { url: await save("crop.mp4"), ...metadata },
});
}
const metadata = await processVideoFile(input, join(dir, "audio.m4a"), "extract-audio");
await processVideoFile(input, join(dir, "silent.mp4"), "silent-video");
return NextResponse.json({
success: true,
data: { audioUrl: await save("audio.m4a"), url: await save("silent.mp4"), ...metadata },
});
} catch (error) {
const message = error instanceof Error ? error.message : "视频处理失败";
return NextResponse.json(
{
success: false,
error: message.startsWith("Command failed")
? "视频处理失败,请确认文件完整且格式受支持"
: message,
},
{ status: 400 },
);
} finally {
running.delete(session.user.id);
if (dir) await rm(dir, { recursive: true, force: true });
}
}

File diff suppressed because it is too large Load Diff

View File

@ -190,6 +190,48 @@ function assertArchivedAssetBuffer(params: {
}
}
async function verifyArchivedAssetUpload(params: {
url: string;
expectedBytes: number;
validateAs?: ArchiveValidationKind;
}) {
const headResponse = await fetch(params.url, {
method: "HEAD",
cache: "no-store",
});
if (!headResponse.ok) {
throw new Error(`归档文件校验失败(HTTP ${headResponse.status})`);
}
const archivedLength = Number(headResponse.headers.get("content-length"));
if (
Number.isFinite(archivedLength) &&
archivedLength > 0 &&
archivedLength !== params.expectedBytes
) {
throw new Error(
`归档文件大小不一致(源文件 ${params.expectedBytes},归档文件 ${archivedLength})`
);
}
if (!params.validateAs) return;
const rangeResponse = await fetch(params.url, {
method: "GET",
headers: { Range: "bytes=0-63" },
cache: "no-store",
});
if (!rangeResponse.ok) {
throw new Error(`归档文件头校验失败(HTTP ${rangeResponse.status})`);
}
const headerBuffer = await rangeResponse.arrayBuffer();
assertArchivedAssetBuffer({
buffer: headerBuffer,
contentType: rangeResponse.headers.get("content-type"),
validateAs: params.validateAs,
});
}
async function archiveRemoteAssetIfNeeded(params: {
cacheKey: string;
sourceUrl?: string | null;
@ -249,6 +291,11 @@ async function archiveRemoteAssetIfNeeded(params: {
);
const filename = `${params.filenamePrefix}.${ext}`;
const uploaded = await uploadFile(filename, assetBuffer);
await verifyArchivedAssetUpload({
url: uploaded.url,
expectedBytes: assetBuffer.byteLength,
validateAs: params.validateAs,
});
cache.set(params.cacheKey, {
sourceUrl,
archivedUrl: uploaded.url,
@ -299,7 +346,8 @@ async function archiveRemoteAssetMapIfNeeded(params: {
)
).filter((entry): entry is readonly [string, string] => Boolean(entry));
return archivedEntries.length > 0 ? Object.fromEntries(archivedEntries) : null;
if (archivedEntries.length !== entries.length) return null;
return Object.fromEntries(archivedEntries);
}
function getStringValue(...values: unknown[]) {
@ -368,33 +416,96 @@ async function updatePersistedWorldAsset(params: {
function extractWorldId(operation: WorldLabsOperation) {
const response = getObjectValue(operation.response);
const metadata = getObjectValue(operation.metadata);
return getStringValue(
response?.world_id,
response?.worldId,
metadata?.world_id,
metadata?.worldId
);
const nestedWorldId =
getStringValue(response?.world_id, response?.worldId) ||
findNestedString(response, ["world_id", "worldId"]);
const metadataWorldId =
getStringValue(metadata?.world_id, metadata?.worldId) ||
findNestedString(metadata, ["world_id", "worldId"]);
return getStringValue(nestedWorldId, metadataWorldId);
}
function isWorldLikePayload(value: unknown): value is WorldLabsWorld {
const record = getObjectValue(value);
if (!record) return false;
const hasCoreWorldField =
typeof record.world_id === "string" ||
typeof record.world_marble_url === "string" ||
typeof record.display_name === "string" ||
typeof record.worldId === "string";
const hasPossibleWorldContainerFields = record.assets || record.permission || record.tags || record.world_prompt;
const hasModelAliasOnly =
typeof record.model === "string" && /world/i.test(record.model);
return Boolean(hasCoreWorldField || hasPossibleWorldContainerFields || hasModelAliasOnly);
}
function extractWorldCandidate(value: unknown, depth = 0): WorldLabsWorld | null {
if (depth > 8) return null;
const record = getObjectValue(value);
if (!record) return null;
if (isWorldLikePayload(record)) return record;
const childCandidates: Array<Record<string, unknown> | null> = [
getObjectValue(record.world),
getObjectValue(record.result),
getObjectValue(record.output),
getObjectValue(record.data),
getObjectValue(record.payload),
];
for (const child of childCandidates) {
if (!child) continue;
if (isWorldLikePayload(child)) return child;
const nested = extractWorldCandidate(child, depth + 1);
if (nested) return nested;
}
for (const child of Object.values(record)) {
const nested = extractWorldCandidate(child, depth + 1);
if (nested) return nested;
}
return null;
}
function extractWorldFromOperation(operation: WorldLabsOperation) {
const response = operation.response;
if (response && typeof response === "object" && "world_marble_url" in (response as Record<string, unknown>)) {
return response as WorldLabsWorld;
}
return null;
return extractWorldCandidate(operation.response);
}
function hasUsableWorldAssets(world?: WorldLabsWorld | null) {
const assets = world?.assets || null;
if (!assets) return false;
const spzUrls = assets.splats?.spz_urls;
const meshRecord = getObjectValue(assets.mesh);
const imageryRecord = getObjectValue(assets.imagery);
const splatsRecord = getObjectValue(assets.splats);
const spzUrls =
splatsRecord?.spz_urls || splatsRecord?.spzUrls || splatsRecord?.splat_urls;
return Boolean(
getStringValue(assets.thumbnail_url) ||
getStringValue(assets.imagery?.pano_url) ||
getStringValue(assets.mesh?.collider_mesh_url) ||
getStringValue(
imageryRecord?.pano_url,
imageryRecord?.panoUrl,
imageryRecord?.panorama_url,
imageryRecord?.panoramaUrl
) ||
getStringValue(
meshRecord?.collider_mesh_url,
meshRecord?.colliderMeshUrl,
meshRecord?.collider_mesh,
meshRecord?.mesh_url,
meshRecord?.meshUrl,
meshRecord?.model_url,
meshRecord?.modelUrl
) ||
(spzUrls &&
typeof spzUrls === "object" &&
Object.values(spzUrls).some((value) => typeof value === "string" && value.trim()))
Object.values(spzUrls).some(
(value) => typeof value === "string" && value.trim()
))
);
}
@ -667,185 +778,224 @@ export async function GET(request: Request, { params }: RouteParams) {
});
providerForLog = config.providerId || null;
const modelAliasConfig = await getThreeDModelAliasConfigForRuntime();
const persistedAssetPayload =
let persistedAssetPayload =
getObjectValue(persistedWorldAsset?.assetPayload) || {};
const shouldUseExternalTaskResultApi =
let shouldUseExternalTaskResultApi =
isWorldLabsExternalTaskApi(config.baseUrl) &&
persistedAssetPayload.externalTaskApi !== false &&
persistedAssetPayload.marbleApi !== true;
if (shouldUseExternalTaskResultApi) {
const externalPayload = await worldLabsRequest<
WorldLabsExternalTaskResponse<WorldLabsExternalTaskResultData>
>(config, {
method: "GET",
path: `/api/external/task/result?taskId=${encodeURIComponent(normalizedTaskId)}`,
timeoutMs: resolveWorldLabsStatusTimeoutMs(),
});
const externalData =
unwrapExternalTaskData<WorldLabsExternalTaskResultData>(externalPayload);
const externalStatusText = getStringValue(externalData.status) || "running";
const externalStatus = normalizeExternalTaskStatus(
externalStatusText,
externalData.done === true
);
const externalAssetPayloadBase = {
...persistedAssetPayload,
providerId: config.providerId || null,
providerName: config.providerName || null,
externalTaskApi: true,
externalTaskStatus: externalStatusText,
requestId: getStringValue(externalPayload.requestId, externalPayload.request_id),
};
if (externalStatus === "running") {
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "running",
progress: null,
output: externalData,
assetPayload: externalAssetPayloadBase,
},
try {
const externalPayload = await worldLabsRequest<
WorldLabsExternalTaskResponse<WorldLabsExternalTaskResultData>
>(config, {
method: "GET",
path: `/api/external/task/result?taskId=${encodeURIComponent(normalizedTaskId)}`,
timeoutMs: resolveWorldLabsStatusTimeoutMs(),
});
}
if (externalStatus === "failed") {
const errorMessage = extractExternalTaskErrorMessage(
externalData,
externalPayload
const externalData =
unwrapExternalTaskData<WorldLabsExternalTaskResultData>(externalPayload);
const externalStatusText = getStringValue(externalData.status) || "running";
const externalStatus = normalizeExternalTaskStatus(
externalStatusText,
externalData.done === true
);
const externalAssetPayloadBase = {
...persistedAssetPayload,
providerId: config.providerId || null,
providerName: config.providerName || null,
externalTaskApi: true,
externalTaskStatus: externalStatusText,
requestId: getStringValue(externalPayload.requestId, externalPayload.request_id),
};
if (externalStatus === "running") {
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "running",
progress: null,
output: externalData,
assetPayload: externalAssetPayloadBase,
},
});
}
if (externalStatus === "failed") {
const errorMessage = extractExternalTaskErrorMessage(
externalData,
externalPayload
);
await updatePersistedWorldAsset({
id: persistedWorldAsset?.id,
status: "failed",
progress: null,
error: errorMessage,
assetPayload: externalAssetPayloadBase,
});
logger.warn("external task failed status", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
error: errorMessage,
});
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "failed",
progress: null,
errorMessage,
output: externalData,
assetPayload: externalAssetPayloadBase,
},
});
}
const extractedAssetPayload = extractExternalWorldAssetPayload(externalData);
const rawMeshUrl = extractedAssetPayload.meshUrl;
const rawThumbnailUrl = extractedAssetPayload.thumbnailUrl;
const rawPanoUrl = extractedAssetPayload.panoUrl;
const stableWorldUrl: string | null = null;
const stableModelUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:mesh`,
sourceUrl: rawMeshUrl,
filenamePrefix: `worldlabs-mesh-${normalizedTaskId}`,
fallbackExtension: "glb",
validateAs: "glb",
});
const stableThumbnailUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:thumb`,
sourceUrl: rawThumbnailUrl,
filenamePrefix: `worldlabs-thumb-${normalizedTaskId}`,
fallbackExtension: "jpg",
});
const stablePanoUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:pano`,
sourceUrl: rawPanoUrl,
filenamePrefix: `worldlabs-pano-${normalizedTaskId}`,
fallbackExtension: "jpg",
});
const stableSpzUrls = await archiveRemoteAssetMapIfNeeded({
cacheKeyPrefix: `worldlabs:${normalizedTaskId}:spz`,
sourceMap: extractedAssetPayload.spzUrls,
filenamePrefix: `worldlabs-splat-${normalizedTaskId}`,
fallbackExtension: "spz",
validateAs: "spz",
});
const sourceSpzEntryCount = Object.keys(
extractedAssetPayload.spzUrls || {}
).length;
const stableSpzEntryCount = Object.keys(stableSpzUrls || {}).length;
const assetsArchivingPending = Boolean(
(rawMeshUrl && !stableModelUrl) ||
(rawThumbnailUrl && !stableThumbnailUrl) ||
(rawPanoUrl && !stablePanoUrl) ||
(sourceSpzEntryCount > 0 && stableSpzEntryCount !== sourceSpzEntryCount)
);
const storageArchivedAt = assetsArchivingPending
? null
: new Date().toISOString();
const assetPayload = {
...externalAssetPayloadBase,
worldId: extractedAssetPayload.worldId,
worldUrl: stableWorldUrl,
displayName: extractedAssetPayload.displayName,
model: resolveThreeDModelAliasId(
extractedAssetPayload.model,
modelAliasConfig.worldModels
),
caption: extractedAssetPayload.caption,
thumbnailUrl: stableThumbnailUrl || extractedAssetPayload.thumbnailUrl,
panoUrl: stablePanoUrl || extractedAssetPayload.panoUrl,
meshUrl: stableModelUrl,
spzUrls: stableSpzUrls,
semanticsMetadata: extractedAssetPayload.semanticsMetadata,
spzArchiveValidatedAt: stableSpzUrls ? storageArchivedAt : null,
storageArchivedAt,
assetsArchivingPending,
sourceWorldUrl: extractedAssetPayload.worldUrl || null,
sourceMeshUrl:
rawMeshUrl && stableModelUrl && rawMeshUrl !== stableModelUrl
? rawMeshUrl
: null,
sourceThumbnailUrl:
rawThumbnailUrl && stableThumbnailUrl && rawThumbnailUrl !== stableThumbnailUrl
? rawThumbnailUrl
: null,
sourcePanoUrl:
rawPanoUrl && stablePanoUrl && rawPanoUrl !== stablePanoUrl
? rawPanoUrl
: null,
sourceSpzUrls:
extractedAssetPayload.spzUrls &&
stableSpzUrls !== extractedAssetPayload.spzUrls
? extractedAssetPayload.spzUrls
: null,
};
await updatePersistedWorldAsset({
id: persistedWorldAsset?.id,
status: "failed",
progress: null,
error: errorMessage,
assetPayload: externalAssetPayloadBase,
status: "success",
progress: 100,
modelUrl: stableModelUrl,
previewImageUrl: stableThumbnailUrl || stablePanoUrl || null,
error: null,
assetPayload,
});
logger.warn("external task failed status", {
logger.info("external task success status", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
error: errorMessage,
hasMesh: Boolean(stableModelUrl),
hasPano: Boolean(stablePanoUrl),
hasThumbnail: Boolean(stableThumbnailUrl),
hasSplat: Boolean(stableSpzUrls && Object.keys(stableSpzUrls).length > 0),
assetsArchivingPending,
});
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "failed",
progress: null,
errorMessage,
status: "success",
progress: 100,
output: externalData,
assetPayload: externalAssetPayloadBase,
modelUrl: stableModelUrl,
previewImageUrl: stableThumbnailUrl || stablePanoUrl || null,
worldId: assetPayload.worldId,
worldUrl: stableWorldUrl,
panoUrl: stablePanoUrl,
caption: assetPayload.caption,
assetPayload,
},
});
} catch (error) {
const fallbackAssetPayload = {
...persistedAssetPayload,
providerId: config.providerId || null,
providerName: config.providerName || null,
marbleApi: true,
externalTaskApi: false,
};
shouldUseExternalTaskResultApi = false;
persistedAssetPayload = fallbackAssetPayload;
await updatePersistedWorldAsset({
id: persistedWorldAsset?.id,
status: "running",
assetPayload: fallbackAssetPayload,
}).catch(() => {});
logger.warn("external task result failed, fallback to marble status", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
error: error instanceof Error ? error.message : String(error),
});
}
const extractedAssetPayload = extractExternalWorldAssetPayload(externalData);
const rawMeshUrl = extractedAssetPayload.meshUrl;
const rawThumbnailUrl = extractedAssetPayload.thumbnailUrl;
const rawPanoUrl = extractedAssetPayload.panoUrl;
const stableWorldUrl: string | null = null;
const stableModelUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:mesh`,
sourceUrl: rawMeshUrl,
filenamePrefix: `worldlabs-mesh-${normalizedTaskId}`,
fallbackExtension: "glb",
validateAs: "glb",
});
const stableThumbnailUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:thumb`,
sourceUrl: rawThumbnailUrl,
filenamePrefix: `worldlabs-thumb-${normalizedTaskId}`,
fallbackExtension: "jpg",
});
const stablePanoUrl = await archiveRemoteAssetIfNeeded({
cacheKey: `worldlabs:${normalizedTaskId}:pano`,
sourceUrl: rawPanoUrl,
filenamePrefix: `worldlabs-pano-${normalizedTaskId}`,
fallbackExtension: "jpg",
});
const stableSpzUrls = await archiveRemoteAssetMapIfNeeded({
cacheKeyPrefix: `worldlabs:${normalizedTaskId}:spz`,
sourceMap: extractedAssetPayload.spzUrls,
filenamePrefix: `worldlabs-splat-${normalizedTaskId}`,
fallbackExtension: "spz",
validateAs: "spz",
});
const assetPayload = {
...externalAssetPayloadBase,
worldId: extractedAssetPayload.worldId,
worldUrl: stableWorldUrl,
displayName: extractedAssetPayload.displayName,
model: resolveThreeDModelAliasId(
extractedAssetPayload.model,
modelAliasConfig.worldModels
),
caption: extractedAssetPayload.caption,
thumbnailUrl: stableThumbnailUrl || extractedAssetPayload.thumbnailUrl,
panoUrl: stablePanoUrl || extractedAssetPayload.panoUrl,
meshUrl: stableModelUrl,
spzUrls: stableSpzUrls,
semanticsMetadata: extractedAssetPayload.semanticsMetadata,
storageArchivedAt: new Date().toISOString(),
sourceWorldUrl: extractedAssetPayload.worldUrl || null,
sourceMeshUrl:
rawMeshUrl && stableModelUrl && rawMeshUrl !== stableModelUrl
? rawMeshUrl
: null,
sourceThumbnailUrl:
rawThumbnailUrl && stableThumbnailUrl && rawThumbnailUrl !== stableThumbnailUrl
? rawThumbnailUrl
: null,
sourcePanoUrl:
rawPanoUrl && stablePanoUrl && rawPanoUrl !== stablePanoUrl
? rawPanoUrl
: null,
sourceSpzUrls:
extractedAssetPayload.spzUrls && stableSpzUrls !== extractedAssetPayload.spzUrls
? extractedAssetPayload.spzUrls
: null,
};
await updatePersistedWorldAsset({
id: persistedWorldAsset?.id,
status: "success",
progress: 100,
modelUrl: stableModelUrl,
previewImageUrl: stableThumbnailUrl || stablePanoUrl || null,
error: null,
assetPayload,
});
logger.info("external task success status", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
hasMesh: Boolean(stableModelUrl),
hasPano: Boolean(stablePanoUrl),
hasThumbnail: Boolean(stableThumbnailUrl),
hasSplat: Boolean(stableSpzUrls && Object.keys(stableSpzUrls).length > 0),
});
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "success",
progress: 100,
output: externalData,
modelUrl: stableModelUrl,
previewImageUrl: stableThumbnailUrl || stablePanoUrl || null,
worldId: assetPayload.worldId,
worldUrl: stableWorldUrl,
panoUrl: stablePanoUrl,
caption: assetPayload.caption,
assetPayload,
},
});
}
const operation = await worldLabsRequest<WorldLabsOperation>(config, {
@ -908,22 +1058,68 @@ export async function GET(request: Request, { params }: RouteParams) {
let world = extractWorldFromOperation(operation);
const worldId = extractWorldId(operation);
if ((!world || !hasUsableWorldAssets(world)) && worldId) {
const detailedWorld = await worldLabsRequest<WorldLabsWorld>(config, {
method: "GET",
path: `/marble/v1/worlds/${encodeURIComponent(worldId)}`,
timeoutMs: resolveWorldLabsStatusTimeoutMs(),
});
world = world
? {
...world,
...detailedWorld,
assets: detailedWorld.assets || world.assets || null,
}
: detailedWorld;
try {
const detailedWorld = await worldLabsRequest<WorldLabsWorld>(config, {
method: "GET",
path: `/marble/v1/worlds/${encodeURIComponent(worldId)}`,
timeoutMs: resolveWorldLabsStatusTimeoutMs(),
});
world = world
? {
...world,
...detailedWorld,
assets: detailedWorld.assets || world.assets || null,
}
: detailedWorld;
} catch (error) {
logger.warn("world detail fetch failed, retry with running status", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
worldId,
error: error instanceof Error ? error.message : "unknown",
});
}
}
if (!world) {
throw new Error("World Labs 任务已完成,但未返回 world 数据");
const pendingPayload = {
...(getObjectValue(persistedWorldAsset?.assetPayload) || {}),
providerId: config.providerId || null,
providerName: config.providerName || null,
worldId: worldId || null,
worldUrl: null,
displayName: null,
model: null,
assetsPending: true,
operationMetadata: operation.metadata ?? null,
};
await updatePersistedWorldAsset({
id: persistedWorldAsset?.id,
status: "running",
progress: 95,
assetPayload: pendingPayload,
});
logger.warn("task completed but world payload is missing", {
elapsedMs: Date.now() - startedAt,
taskIdLength: normalizedTaskId.length,
providerId: config.providerId || null,
worldId: worldId || null,
});
return NextResponse.json({
success: true,
data: {
taskId: normalizedTaskId,
type: "world",
status: "running",
progress: 95,
output: operation.response ?? null,
worldId: worldId || null,
worldUrl: null,
previewImageUrl: null,
assetPayload: pendingPayload,
},
});
}
if (!hasUsableWorldAssets(world)) {
@ -1039,24 +1235,15 @@ export async function GET(request: Request, { params }: RouteParams) {
: 0;
const hasMissingSpzArchive =
rawSpzUrls.length > 0 && stableSpzEntryCount < rawSpzUrls.length;
const storageArchivedAt =
[
[rawWorldUrl, stableWorldUrl],
[rawMeshUrl, stableModelUrl],
[rawThumbnailUrl, stableThumbnailUrl],
[rawPanoUrl, stablePanoUrl],
].some(([rawUrl, stableUrl]) => rawUrl && stableUrl && rawUrl !== stableUrl) ||
Boolean(
stableSpzUrls &&
assetPayload.spzUrls &&
Object.entries(stableSpzUrls).some(
([key, value]) =>
typeof assetPayload.spzUrls?.[key] === "string" &&
assetPayload.spzUrls[key] !== value
)
)
? new Date().toISOString()
: null;
const assetsArchivingPending = Boolean(
(rawMeshUrl && !stableModelUrl) ||
(rawThumbnailUrl && !stableThumbnailUrl) ||
(rawPanoUrl && !stablePanoUrl) ||
hasMissingSpzArchive
);
const storageArchivedAt = assetsArchivingPending
? null
: new Date().toISOString();
logger.info("task success status", {
elapsedMs: Date.now() - startedAt,
@ -1068,10 +1255,7 @@ export async function GET(request: Request, { params }: RouteParams) {
hasThumbnail: Boolean(stableThumbnailUrl),
hasSplat: Boolean(stableSpzUrls && Object.keys(stableSpzUrls).length > 0),
storageArchivedAt,
assetsArchivingPending: Boolean(
(rawMeshUrl && !stableModelUrl) ||
hasMissingSpzArchive
),
assetsArchivingPending,
});
const responseAssetPayload = {
@ -1081,12 +1265,9 @@ export async function GET(request: Request, { params }: RouteParams) {
panoUrl: stablePanoUrl || assetPayload.panoUrl,
meshUrl: stableModelUrl,
spzUrls: stableSpzUrls,
spzArchiveValidatedAt: stableSpzUrls ? new Date().toISOString() : null,
spzArchiveValidatedAt: stableSpzUrls ? storageArchivedAt : null,
storageArchivedAt,
assetsArchivingPending: Boolean(
(rawMeshUrl && !stableModelUrl) ||
hasMissingSpzArchive
),
assetsArchivingPending,
sourceWorldUrl: rawWorldUrl || null,
sourceMeshUrl:
rawMeshUrl && stableModelUrl && rawMeshUrl !== stableModelUrl

View File

@ -142,7 +142,27 @@ function shouldUseExternalTaskCreateApi(
config: Awaited<ReturnType<typeof getWorldLabsProviderConfig>>,
mode: WorldTaskMode
) {
return isWorldLabsExternalTaskApi(config.baseUrl) && mode === "text";
return isWorldLabsExternalTaskApi(config.baseUrl) && mode === "image";
}
function isWorldLabsModeUnsupportedByExternalProvider(
config: Awaited<ReturnType<typeof getWorldLabsProviderConfig>>,
mode: WorldTaskMode
) {
return isWorldLabsExternalTaskApi(config.baseUrl) && mode !== "image";
}
function assertWorldModeSupportedByCandidate(
config: Awaited<ReturnType<typeof getWorldLabsProviderConfig>>,
mode: WorldTaskMode,
) {
if (isWorldLabsModeUnsupportedByExternalProvider(config, mode)) {
const normalizedBaseUrl = String(config.baseUrl || "").trim() || "未知服务地址";
const normalizedMode = String(mode || "").trim() || "未知模式";
throw new Error(
`${normalizedBaseUrl} 的 world-generation 当前仅支持图生世界(image)。当前提交的是 ${normalizedMode},请先切换供应商或使用图生世界。`
);
}
}
function resolveOfficialMarbleModel(model?: string | null) {
@ -153,6 +173,49 @@ function resolveOfficialMarbleModel(model?: string | null) {
return normalized;
}
function buildExternalWorldTaskPayload(
mode: WorldTaskMode,
worldPrompt: Record<string, unknown>,
prompt: string | null,
displayName: string,
) {
const normalizedPrompt = readOptionalTrimmedText(prompt);
if (!normalizedPrompt && mode === "text") {
throw new Error("请输入世界描述");
}
const payload: Record<string, unknown> = {
model: WORLDLABS_EXTERNAL_TASK_MODEL,
displayName,
};
if (normalizedPrompt) {
payload.prompt = normalizedPrompt;
} else if (mode !== "text") {
payload.prompt = "图生世界";
}
if (mode === "image" || mode === "multi-image") {
payload.world_prompt = worldPrompt;
}
return payload;
}
function buildOfficialWorldTaskPayload(
displayName: string,
worldPrompt: Record<string, unknown>,
officialMarbleModel: string,
body: CreateWorldTaskRequest
) {
return {
display_name: displayName,
model: officialMarbleModel,
permission: {
public: body.makePublic === true,
},
world_prompt: worldPrompt,
};
}
function buildWorldPrompt(body: CreateWorldTaskRequest, mode: WorldTaskMode) {
const textPrompt = readOptionalTrimmedText(body.prompt);
@ -356,41 +419,73 @@ export async function POST(request: Request) {
for (const candidate of candidates) {
try {
const useExternalTaskApi = shouldUseExternalTaskCreateApi(candidate, mode);
assertWorldModeSupportedByCandidate(candidate, mode);
let requestUsedExternalTaskApi = false;
if (useExternalTaskApi) {
const prompt = readOptionalTrimmedText(body.prompt);
if (!prompt) {
throw new Error("请输入世界描述");
const payload = buildExternalWorldTaskPayload(
mode,
worldPrompt,
readOptionalTrimmedText(body.prompt),
displayName,
);
try {
const externalResult = await worldLabsRequest<
WorldLabsExternalTaskResponse<WorldLabsExternalTaskCreateData>
>(candidate, {
method: "POST",
path: "/api/external/task/generate",
timeoutMs: resolveWorldLabsCreateTimeoutMs(),
json: payload,
});
result = normalizeExternalTaskCreateOperation(externalResult);
requestUsedExternalTaskApi = true;
} catch (error) {
const externalError = error instanceof Error ? error.message : String(error);
logger.warn("external world create failed, fallback to marble endpoint", {
providerId: candidate.providerId || null,
error: externalError,
});
if (isCredentialOrBillingProviderError(error)) {
await reportWorldLabsProviderRuntimeFailure(
candidate.providerId,
externalError
);
}
result = await worldLabsRequest<WorldLabsOperation>(candidate, {
method: "POST",
path: "/marble/v1/worlds:generate",
timeoutMs: resolveWorldLabsCreateTimeoutMs(),
json: buildOfficialWorldTaskPayload(
displayName,
worldPrompt,
officialMarbleModel,
body
),
});
requestUsedExternalTaskApi = false;
}
const externalResult = await worldLabsRequest<
WorldLabsExternalTaskResponse<WorldLabsExternalTaskCreateData>
>(candidate, {
method: "POST",
path: "/api/external/task/generate",
timeoutMs: resolveWorldLabsCreateTimeoutMs(),
json: {
model: WORLDLABS_EXTERNAL_TASK_MODEL,
displayName,
prompt,
},
});
result = normalizeExternalTaskCreateOperation(externalResult);
} else {
requestUsedExternalTaskApi = false;
result = await worldLabsRequest<WorldLabsOperation>(candidate, {
method: "POST",
path: "/marble/v1/worlds:generate",
timeoutMs: resolveWorldLabsCreateTimeoutMs(),
json: {
display_name: displayName,
model: officialMarbleModel,
permission: {
public: body.makePublic === true,
},
world_prompt: worldPrompt,
},
json: buildOfficialWorldTaskPayload(
displayName,
worldPrompt,
officialMarbleModel,
body
),
});
}
if (!result?.operation_id) {
throw new Error("创建 World Labs 世界任务返回缺少 operation_id");
}
usedProvider = candidate;
usedExternalTaskApi = useExternalTaskApi;
usedExternalTaskApi = requestUsedExternalTaskApi;
providerForLog = candidate.providerId || null;
break;
} catch (error) {

File diff suppressed because it is too large Load Diff

View File

@ -1,234 +1,10 @@
import { type NextRequest, NextResponse } from "next/server";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import { ExtendServiceError, executeExtendJob } from "@/lib/ai/extend/service";
import { getPointsSystemConfig } from "@/lib/db/queries";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
import { getImageToolCreditCost } from "@/lib/points/config";
import {
getFriendlyRateLimitMessage,
getUserFacingRequestErrorMessage,
isRateLimitError,
} from "@/lib/request-error";
import { executeExtendPOST } from "@/lib/canvas/operations/extend";
export const runtime = "nodejs";
export const maxDuration = 180;
function parseIncludeBase64(value: unknown): boolean {
if (typeof value === "boolean") return value;
const normalized = String(value ?? "").trim().toLowerCase();
return normalized === "true" || normalized === "1" || normalized === "yes";
}
function buildExtendSuccessPayload(result: Awaited<ReturnType<typeof executeExtendJob>>) {
return {
success: true,
original_url: result.original_url,
thumbnail_url: result.thumbnail_url,
...(result.image_base64 ? { base64: result.image_base64 } : {}),
processing_time: result.processing_time,
};
}
export async function POST(request: NextRequest) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ error: "未授权,请先登录" }, { status: 401 });
}
const contentType = request.headers.get("content-type") || "";
let imageUrl: string | null = null;
let imageBase64: string | null = null;
let prompt = "图像拓展处理";
let chatId: string | null = null;
let messageId: string | null = null;
let includeBase64 = false;
const requestId = `extend-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
if (contentType.includes("application/json")) {
const body = await request.json();
imageUrl = body.image_url ? String(body.image_url).trim() : null;
imageBase64 = body.image_base64 ? String(body.image_base64).trim() : null;
prompt = body.prompt || prompt;
chatId = body.chatId || null;
messageId = body.messageId || null;
includeBase64 = parseIncludeBase64(body.includeBase64);
if (!imageUrl && !imageBase64) {
return NextResponse.json(
{ error: "缺少image_base64或image_url参数" },
{ status: 400 },
);
}
console.log("[Extend API] Request received:", {
requestId,
source: imageUrl ? "url" : "base64",
includeBase64,
chatId,
messageId,
xScale: body.xScale,
yScale: body.yScale,
});
const pointsConfig = await getPointsSystemConfig();
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, "extend"),
reason: "image_extend",
modelId: "image-extend",
note: "图片扩图消耗积分",
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
const result = await executeExtendJob({
userId: session.user.id,
imageUrl,
imageBase64,
prompt,
chatId,
messageId,
xScale: body.xScale ? Number(body.xScale) : undefined,
yScale: body.yScale ? Number(body.yScale) : undefined,
includeBase64,
requestId,
});
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume();
} catch (pointsError) {
console.error("[Points] Failed to consume extend points:", pointsError);
}
}
console.log("[Extend API] Request completed:", {
requestId,
includeBase64,
processingTime: result.processing_time,
originalUrl: result.original_url.slice(0, 120),
});
return NextResponse.json(buildExtendSuccessPayload(result));
}
if (contentType.includes("multipart/form-data")) {
const formData = await request.formData();
const file = formData.get("file") as File | null;
if (!file) {
return NextResponse.json({ error: "缺少file参数" }, { status: 400 });
}
const arrayBuffer = await file.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
const multipartChatId = String(formData.get("chatId") || "") || null;
const multipartMessageId = String(formData.get("messageId") || "") || null;
includeBase64 = parseIncludeBase64(formData.get("includeBase64"));
console.log("[Extend API] Multipart request received:", {
requestId,
includeBase64,
chatId: multipartChatId,
messageId: multipartMessageId,
bytes: buffer.length,
});
const pointsConfig = await getPointsSystemConfig();
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, "extend"),
reason: "image_extend",
modelId: "image-extend",
note: "图片扩图消耗积分",
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
const result = await executeExtendJob({
userId: session.user.id,
imageBase64: buffer.toString("base64"),
prompt: String(formData.get("prompt") || prompt),
chatId: multipartChatId,
messageId: multipartMessageId,
includeBase64,
requestId,
});
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume();
} catch (pointsError) {
console.error("[Points] Failed to consume extend points:", pointsError);
}
}
console.log("[Extend API] Multipart request completed:", {
requestId,
includeBase64,
processingTime: result.processing_time,
originalUrl: result.original_url.slice(0, 120),
});
return NextResponse.json(buildExtendSuccessPayload(result));
}
return NextResponse.json(
{ error: "不支持的Content-Type,请使用application/json或multipart/form-data" },
{ status: 400 },
);
} catch (error) {
if (error instanceof ExtendServiceError) {
if (error.status === 429 || isRateLimitError(error)) {
return NextResponse.json(
{
error: getFriendlyRateLimitMessage("当前操作过于频繁,请稍后再试"),
detail: error.detail || error.message,
retryAfter: 10,
},
{
status: 429,
headers: {
"Retry-After": "10",
},
},
);
}
return NextResponse.json(
{
error: getUserFacingRequestErrorMessage(error, "图像拓展失败"),
detail: error.detail || error.message,
},
{ status: error.status },
);
}
console.error("[Extend API] 错误:", error);
if (isRateLimitError(error)) {
return NextResponse.json(
{
error: getFriendlyRateLimitMessage("当前操作过于频繁,请稍后再试"),
detail: error instanceof Error ? error.message : String(error),
retryAfter: 10,
},
{
status: 429,
headers: {
"Retry-After": "10",
},
},
);
}
return NextResponse.json(
{
error: getUserFacingRequestErrorMessage(error, "图像拓展失败"),
detail: error instanceof Error ? error.message : String(error),
},
{ status: 500 },
);
}
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
return executeExtendPOST(request, session.user.id);
}

View File

@ -1,7 +1,7 @@
import { NextRequest, NextResponse } from "next/server";
import { db } from "@/lib/db";
import { generatedImage } from "@/lib/db/schema";
import { eq, desc, sql } from "drizzle-orm";
import { generatedImage, imageArchiveTask } from "@/lib/db/schema";
import { and, desc, eq, isNull, sql } from "drizzle-orm";
import { auth } from "@/app/(auth)/auth";
export async function GET(request: NextRequest) {
@ -19,12 +19,23 @@ export async function GET(request: NextRequest) {
const page = Number.parseInt(searchParams.get("page") || "1");
const pageSize = Number.parseInt(searchParams.get("pageSize") || "24");
const offset = (page - 1) * pageSize;
const visibleImageFilter = and(
eq(generatedImage.userId, userId),
isNull(imageArchiveTask.id)
);
// 查询总数
const countResult = await db
.select({ count: sql<number>`count(*)::int` })
.from(generatedImage)
.where(eq(generatedImage.userId, userId));
.leftJoin(
imageArchiveTask,
and(
eq(imageArchiveTask.sourceUrl, generatedImage.originalUrl),
eq(imageArchiveTask.status, "abandoned")
)
)
.where(visibleImageFilter);
const total = countResult[0]?.count || 0;
@ -42,7 +53,14 @@ export async function GET(request: NextRequest) {
modelId: generatedImage.modelId, // 添加 modelId 字段
})
.from(generatedImage)
.where(eq(generatedImage.userId, userId))
.leftJoin(
imageArchiveTask,
and(
eq(imageArchiveTask.sourceUrl, generatedImage.originalUrl),
eq(imageArchiveTask.status, "abandoned")
)
)
.where(visibleImageFilter)
.orderBy(desc(generatedImage.createdAt))
.limit(pageSize)
.offset(offset);

View File

@ -14,6 +14,10 @@ import {
updateMessageById,
} from "@/lib/db/queries";
import type { DBMessage } from "@/lib/db/schema";
import {
buildOpenClawArtifactProxyUrl,
normalizeOpenClawArtifacts,
} from "@/lib/openclaw-chat/artifacts";
import {
getFalImageTaskStatus,
resolveFalImageTaskToMarkdown,
@ -32,12 +36,29 @@ const FAL_IMAGE_TASK_POLL_INTERVAL_MS = 2_000;
const falTaskLastPolledAt = new Map<string, number>();
const falTaskSyncInFlight = new Set<string>();
function toPublicVideoMetadata(value: DBMessage["videoMetadata"]) {
function toPublicVideoMetadata(params: {
message: DBMessage;
requestOrigin: string;
}) {
const value = params.message.videoMetadata;
if (!value || typeof value !== "object" || Array.isArray(value)) return value;
const metadata = value as Record<string, unknown>;
if (metadata.taskKind !== "clawd_chat") return value;
const publicMetadata = { ...metadata };
delete publicMetadata.upstreamTaskKey;
publicMetadata.artifacts = normalizeOpenClawArtifacts(metadata.artifacts).map(
(artifact, artifactIndex) => ({
...artifact,
downloadUrl: new URL(
buildOpenClawArtifactProxyUrl({
chatId: params.message.chatId,
messageId: params.message.id,
artifactIndex,
}),
params.requestOrigin
).toString(),
})
);
return publicMetadata;
}
@ -286,7 +307,11 @@ export async function GET(request: NextRequest) {
createdAt: msg.createdAt,
parts: parts,
attachments: msg.attachments || [],
videoMetadata: toPublicVideoMetadata(msg.videoMetadata) || null,
videoMetadata:
toPublicVideoMetadata({
message: msg,
requestOrigin: request.nextUrl.origin,
}) || null,
};
});

View File

@ -1,138 +1,10 @@
import { NextRequest, NextResponse } from "next/server";
import { NextResponse } from "next/server";
import { auth } from "@/app/(auth)/auth";
import {
executeRmbgJob,
fetchImageBufferFromUrl,
RmbgServiceError,
} from "@/lib/ai/rmbg/service";
import { getPointsSystemConfig } from "@/lib/db/queries";
import { prepareFeaturePointsCharge } from "@/lib/points/feature-charge";
import { getImageToolCreditCost } from "@/lib/points/config";
import { executeRemoveBackgroundPOST } from "@/lib/canvas/operations/rmbg";
export const runtime = "nodejs";
export const maxDuration = 60;
export async function POST(request: NextRequest) {
try {
const session = await auth();
if (!session?.user?.id) {
return NextResponse.json({ error: "未授权,请先登录" }, { status: 401 });
}
const contentType = request.headers.get("content-type") || "";
let imageBuffer: Buffer;
let imageBase64: string | undefined;
let prompt = "抠图处理";
let chatId: string | null = null;
let messageId: string | null = null;
let skipGallerySave = false;
let sourceImageUrl: string | null = null;
if (contentType.includes("application/json")) {
const body = await request.json();
prompt = body.prompt || "抠图处理";
chatId = body.chatId || null;
messageId = body.messageId || null;
skipGallerySave = body.skipGallerySave === true || body.skip_gallery_save === 1;
const imageUrl = body.imageUrl || body.image_url;
const explicitSourceImageUrl = body.sourceImageUrl || body.source_image_url;
if (imageUrl) {
sourceImageUrl = String(imageUrl);
imageBuffer = await fetchImageBufferFromUrl(sourceImageUrl);
} else if (body.image_base64) {
imageBase64 = String(body.image_base64);
imageBuffer = Buffer.from(imageBase64, "base64");
if (explicitSourceImageUrl) {
sourceImageUrl = String(explicitSourceImageUrl);
}
} else {
return NextResponse.json(
{ error: "缺少 imageUrl 或 image_base64 参数" },
{ status: 400 }
);
}
} else if (contentType.includes("multipart/form-data")) {
const formData = await request.formData();
const file = formData.get("file") as File | null;
if (!file) {
return NextResponse.json({ error: "缺少file参数" }, { status: 400 });
}
const arrayBuffer = await file.arrayBuffer();
imageBuffer = Buffer.from(arrayBuffer);
imageBase64 = imageBuffer.toString("base64");
prompt = (formData.get("prompt") as string) || "抠图处理";
chatId = (formData.get("chatId") as string) || null;
messageId = (formData.get("messageId") as string) || null;
skipGallerySave =
formData.get("skipGallerySave") === "true" || formData.get("skip_gallery_save") === "1";
sourceImageUrl =
(formData.get("sourceImageUrl") as string) ||
(formData.get("source_image_url") as string) ||
null;
} else {
return NextResponse.json(
{ error: "不支持的Content-Type,请使用application/json或multipart/form-data" },
{ status: 400 }
);
}
const pointsConfig = await getPointsSystemConfig();
const pointsCharge = await prepareFeaturePointsCharge({
userId: session.user.id,
unitCost: getImageToolCreditCost(pointsConfig, "remove-background"),
reason: "image_remove_background",
modelId: "rmbg",
note: "图片抠图消耗积分",
pointsConfig,
});
if (!pointsCharge.ok) {
return NextResponse.json(pointsCharge.payload, { status: pointsCharge.status });
}
const result = await executeRmbgJob({
userId: session.user.id,
imageBuffer,
imageBase64,
prompt,
chatId,
messageId,
skipGallerySave,
sourceImageUrl,
});
if (pointsCharge.chargeable) {
try {
await pointsCharge.consume();
} catch (pointsError) {
console.error("[Points] Failed to consume RMBG points:", pointsError);
}
}
return NextResponse.json({
success: true,
...result,
});
} catch (error) {
if (error instanceof RmbgServiceError) {
return NextResponse.json(
{
error: error.message,
detail: error.detail || error.message,
},
{ status: error.status }
);
}
console.error("[RMBG API] 处理错误:", error);
return NextResponse.json(
{ error: "服务器内部错误", detail: error instanceof Error ? error.message : "未知错误" },
{ status: 500 }
);
}
export async function POST(request: Request) {
const session = await auth();
if (!session?.user?.id) return NextResponse.json({ error: "未登录" }, { status: 401 });
return executeRemoveBackgroundPOST(request, session.user.id);
}

View File

@ -399,7 +399,7 @@ function normalizeLovartComposerPreviewTexts(value: unknown): string[] {
function normalizeLovartComposerPrefix(value: unknown): string {
const normalized = String(value || "").trim();
if (!normalized) {
return "让「由壹点AI」";
return "让「𝗕𝘂𝗬𝗶」";
}
return normalized.slice(0, 40);
}
@ -573,7 +573,7 @@ const PUBLIC_SITE_SETTINGS_SCOPES = new Set<string>([
]);
const HOME_SITE_SETTINGS_FALLBACK: Record<string, unknown> = {
siteName: "AI Chatbot",
siteName: "𝗕𝘂𝗬𝗶",
siteLogo: null,
siteLogoDark: null,
favicon: null,
@ -583,11 +583,11 @@ const HOME_SITE_SETTINGS_FALLBACK: Record<string, unknown> = {
bannerText: "限时特惠:升级会员最高立享 45% OFF!",
bannerLink: null,
bannerActionText: "立即查看",
lovartHeroTitle: "由壹点AI 让设计更简单",
lovartHeroSubtitle: "懂你的设计代理,帮你搞定一切",
lovartHeroTitle: "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle: "下达指令,即刻呈现",
surpriseCategories: ["全部", "精选"],
canvasDefaultModelId: null,
lovartComposerPrefix: "让「由壹点AI」",
lovartComposerPrefix: "让「𝗕𝘂𝗬𝗶」",
lovartComposerPreviewTexts: [],
lovartPromptTagVisibility: {},
contactConfig: normalizeContactConfig(undefined),
@ -664,7 +664,7 @@ async function readScopedSiteSettings(keys: string[]) {
function createBrandingSettings(settings: Record<string, unknown> | undefined) {
return {
siteName: String(settings?.siteName || "AI Chatbot"),
siteName: String(settings?.siteName || "𝗕𝘂𝗬𝗶"),
siteLogo: settings?.siteLogo || null,
siteLogoDark: settings?.siteLogoDark || null,
favicon: settings?.favicon || null,
@ -822,7 +822,7 @@ async function getScopedPublicSiteSettings(
usageTimeStart: typedSettings?.usageTimeStart || "08:00",
usageTimeEnd: typedSettings?.usageTimeEnd || "20:00",
usageTimeMessage:
typedSettings?.usageTimeMessage || "由壹点累了也是需要休息的",
typedSettings?.usageTimeMessage || "BuYi 累了也是需要休息的",
});
}
@ -953,9 +953,9 @@ async function getScopedPublicSiteSettings(
typedSettings?.lovartPromoText ||
"现在升级,立享 Kling 3.0 与 Nano Banana Pro,最高 5 折优惠",
lovartPromoActionText: typedSettings?.lovartPromoActionText || "立即升级",
lovartHeroTitle: typedSettings?.lovartHeroTitle || "由壹点AI 让设计更简单",
lovartHeroTitle: typedSettings?.lovartHeroTitle || "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle:
typedSettings?.lovartHeroSubtitle || "懂你的设计代理,帮你搞定一切",
typedSettings?.lovartHeroSubtitle || "下达指令,即刻呈现",
surpriseCategories: normalizeSurpriseCategories(
valueByKey.get(HOME_SURPRISE_CATEGORIES_KEY)
),
@ -1476,7 +1476,7 @@ export async function GET(request: NextRequest) {
usageTimeEnabled: false,
usageTimeStart: "08:00",
usageTimeEnd: "20:00",
usageTimeMessage: "由壹点累了也是需要休息的",
usageTimeMessage: "BuYi 累了也是需要休息的",
bannerEnabled: true,
bannerText: "限时特惠:升级会员最高立享 45% OFF!",
bannerLink: null,
@ -1525,8 +1525,8 @@ export async function GET(request: NextRequest) {
canvasDetailAgentModelId: null,
lovartPromoText: "现在升级,立享 Kling 3.0 与 Nano Banana Pro,最高 5 折优惠",
lovartPromoActionText: "立即升级",
lovartHeroTitle: "由壹点AI 让设计更简单",
lovartHeroSubtitle: "懂你的设计代理,帮你搞定一切",
lovartHeroTitle: "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle: "下达指令,即刻呈现",
surpriseCategories,
aiDetailGenerationEnabled,
lovartComposerPrefix,
@ -1546,8 +1546,8 @@ export async function GET(request: NextRequest) {
baseSettings.lovartPromoText ||
"现在升级,立享 Kling 3.0 与 Nano Banana Pro,最高 5 折优惠",
lovartPromoActionText: baseSettings.lovartPromoActionText || "立即升级",
lovartHeroTitle: baseSettings.lovartHeroTitle || "由壹点AI 让设计更简单",
lovartHeroSubtitle: baseSettings.lovartHeroSubtitle || "懂你的设计代理,帮你搞定一切",
lovartHeroTitle: baseSettings.lovartHeroTitle || "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle: baseSettings.lovartHeroSubtitle || "下达指令,即刻呈现",
notification,
surpriseCategories,
aiDetailGenerationEnabled,
@ -3119,7 +3119,7 @@ export async function PUT(request: NextRequest) {
usageTimeEnabled: siteSettingsPayload.usageTimeEnabled ?? false,
usageTimeStart: siteSettingsPayload.usageTimeStart ?? "08:00",
usageTimeEnd: siteSettingsPayload.usageTimeEnd ?? "20:00",
usageTimeMessage: siteSettingsPayload.usageTimeMessage ?? "由壹点累了也是需要休息的",
usageTimeMessage: siteSettingsPayload.usageTimeMessage ?? "BuYi 累了也是需要休息的",
bannerEnabled: siteSettingsPayload.bannerEnabled ?? true,
bannerText: siteSettingsPayload.bannerText ?? "限时特惠:升级会员最高立享 45% OFF!",
bannerLink: siteSettingsPayload.bannerLink,
@ -3144,8 +3144,8 @@ export async function PUT(request: NextRequest) {
siteSettingsPayload.lovartPromoText ??
"现在升级,立享 Kling 3.0 与 Nano Banana Pro,最高 5 折优惠",
lovartPromoActionText: siteSettingsPayload.lovartPromoActionText ?? "立即升级",
lovartHeroTitle: siteSettingsPayload.lovartHeroTitle ?? "由壹点AI 让设计更简单",
lovartHeroSubtitle: siteSettingsPayload.lovartHeroSubtitle ?? "懂你的设计代理,帮你搞定一切",
lovartHeroTitle: siteSettingsPayload.lovartHeroTitle ?? "𝗕𝘂𝗬𝗶 让设计触手可及",
lovartHeroSubtitle: siteSettingsPayload.lovartHeroSubtitle ?? "下达指令,即刻呈现",
updatedBy: session.user.id,
})
.returning();

View File

@ -9374,6 +9374,7 @@ export default function CanvasEditorLegacyPage({
// 键盘快捷键
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
if (useCanvasStore.getState().canvasImageSelectionSession) return;
// 如果正在输入则不触发快捷键(包括 input、textarea 和 contentEditable 元素)
if (isCanvasTextEntryTarget(e.target)) {
return;

View File

@ -365,6 +365,62 @@ function resolveWorldTaskPreviewUrl(task?: TaskRecord | null) {
);
}
function getTaskPreviewCandidates(task?: TaskRecord | null) {
if (!task) return [];
const candidates = [
task.previewImageUrl,
task.assetPayload?.thumbnailUrl,
task.assetPayload?.panoUrl,
task.assetPayload?.sourcePreviewImageUrl,
task.assetPayload?.sourceThumbnailUrl,
task.assetPayload?.sourcePanoUrl,
];
const seen = new Set<string>();
return candidates
.map((value) => String(value || "").trim())
.filter((value) => {
if (!value || seen.has(value)) return false;
seen.add(value);
return true;
});
}
function isPersistableTaskPreviewUrl(value?: string | null) {
return /^https?:\/\//i.test(String(value || "").trim());
}
function TaskAssetPreview({ task }: { task: TaskRecord }) {
const candidates = getTaskPreviewCandidates(task);
const signature = candidates.join("|");
const [candidateIndex, setCandidateIndex] = useState(0);
useEffect(() => {
setCandidateIndex(0);
}, [signature]);
const previewUrl = candidates[candidateIndex] || null;
if (!previewUrl) {
return (
<div className="flex h-full w-full items-center justify-center text-zinc-700">
{task.status === "queued" || task.status === "running" ? (
<Loader2 className="h-7 w-7 animate-spin text-yellow-400/70" />
) : (
<Box className="h-8 w-8" />
)}
</div>
);
}
return (
<img
alt="3D 资产预览"
className="h-full w-full object-cover transition-transform duration-300 group-hover:scale-[1.035]"
onError={() => setCandidateIndex((index) => index + 1)}
src={previewUrl}
/>
);
}
function canImportTaskToCanvas(task?: TaskRecord | null) {
if (!task || task.status !== "success") return false;
if (task.provider !== "worldlabs-3d") return Boolean(resolveTaskModelUrl(task));
@ -850,9 +906,21 @@ export default function CanvasModel3DPage() {
);
const pollAttemptsRef = useRef(0);
const pollErrorCountRef = useRef(0);
const multiViewSlotsRef = useRef<MultiViewSlot[]>([]);
const batchImageSlotsRef = useRef<MultiViewSlot[]>([]);
const MAX_RETRYABLE_TASK_POLL_FAILURES = 12;
const isRetryableTaskPollFailure = (message: string) => {
const normalized = String(message || "").toLowerCase();
return (
/http[^0-9]*5\d\d/.test(normalized) ||
normalized.includes("超时") ||
normalized.includes("timeout")
);
};
// Fetch site settings for Logo
const { data: siteSettingsData } = useSWR(
"/api/site-settings?scope=branding",
@ -1114,7 +1182,9 @@ export default function CanvasModel3DPage() {
modelVersion: task.modelVersion,
progress: task.progress ?? null,
modelUrl: task.modelUrl ?? null,
previewImageUrl: task.previewImageUrl ?? null,
previewImageUrl: isPersistableTaskPreviewUrl(task.previewImageUrl)
? task.previewImageUrl
: null,
error: task.error ?? null,
sourceTaskId: task.sourceTaskId ?? null,
exportFormat: task.exportFormat ?? null,
@ -1434,6 +1504,33 @@ export default function CanvasModel3DPage() {
});
if (!result.success || !result.data) {
const failMessage = result.error || "任务查询失败";
if (isRetryableTaskPollFailure(failMessage)) {
pollErrorCountRef.current += 1;
if (pollErrorCountRef.current === 1) {
console.warn("[Model3DPage] 任务查询临时异常,继续重试", {
taskId: currentTask.taskId,
provider: currentTask.provider,
error: failMessage,
attempt: pollAttemptsRef.current,
});
toast.info(`任务查询异常,正在重试...`);
}
if (pollErrorCountRef.current >= MAX_RETRYABLE_TASK_POLL_FAILURES) {
const failedRecord: TaskRecord = {
...currentTask,
status: "failed",
error: `${failMessage}(重试仍异常)`,
};
setCurrentTask(failedRecord);
upsertTaskHistory(failedRecord);
}
return;
}
pollErrorCountRef.current = 0;
const failedRecord: TaskRecord = {
...currentTask,
status: "failed",
@ -1479,6 +1576,8 @@ export default function CanvasModel3DPage() {
error:
status === "failed" ? result.data.errorMessage || "任务失败" : null,
};
pollErrorCountRef.current = 0;
if (taskRecordChanged(currentTask, nextRecord)) {
setCurrentTask(nextRecord);
upsertTaskHistory(nextRecord);
@ -1652,6 +1751,7 @@ export default function CanvasModel3DPage() {
sourceTaskId?: string | null;
exportFormat?: string | null;
assetPayload?: Model3DTaskAssetPayload | null;
previewImageUrl?: string | null;
}) => {
const normalizedPayload =
params.provider === "tripo-3d" &&
@ -1706,6 +1806,7 @@ export default function CanvasModel3DPage() {
createdAt: Date.now(),
sourceTaskId: params.sourceTaskId || null,
exportFormat: params.exportFormat || null,
previewImageUrl: params.previewImageUrl || null,
assetPayload:
params.provider === "worldlabs-3d"
? worldCreateResultData?.assetPayload ?? params.assetPayload ?? null
@ -1821,6 +1922,7 @@ export default function CanvasModel3DPage() {
providerId: uploadResult.data.providerId || null,
providerName: uploadResult.data.providerName || null,
},
previewImageUrl: referencePreviewUrl,
});
toast.success("已提交图生3D任务");
return;
@ -1855,6 +1957,8 @@ export default function CanvasModel3DPage() {
providerId: uploadResult.data.providerId || null,
providerName: uploadResult.data.providerName || null,
},
previewImageUrl:
batchImageSlots.find((slot) => slot.file === file)?.previewUrl || null,
});
successCount += 1;
}
@ -1910,6 +2014,7 @@ export default function CanvasModel3DPage() {
providerId: boundTripoProviderId,
providerName: boundTripoProviderName,
},
previewImageUrl: multiViewSlots.find((slot) => slot.previewUrl)?.previewUrl || null,
});
toast.success("已提交多视图3D任务");
} catch (error) {
@ -1992,6 +2097,7 @@ export default function CanvasModel3DPage() {
mode: "world-image",
promptText: normalizedPrompt || "图生世界",
modelLabel: worldModel,
previewImageUrl: sourceUrl || worldImagePreviewUrl,
});
toast.success("已提交图生世界任务");
return;
@ -2076,6 +2182,10 @@ export default function CanvasModel3DPage() {
mode: "world-multi-image",
promptText: normalizedPrompt || "多图世界",
modelLabel: worldModel,
previewImageUrl:
mediaAssets.find((item) => "sourceUrl" in item)?.sourceUrl ||
selectedWorldImages.find((slot) => slot.previewUrl)?.previewUrl ||
null,
});
toast.success("已提交多图世界任务");
} catch (error) {
@ -4177,8 +4287,6 @@ export default function CanvasModel3DPage() {
const canImportWorld =
task.provider === "worldlabs-3d" && canImportTaskToCanvas(task);
const isSelectedForCanvas = selectedWorldCanvasTaskIds.includes(task.taskId);
const previewUrl = resolveWorldTaskPreviewUrl(task) || task.previewImageUrl;
return (
<div
className={cn(
@ -4195,17 +4303,7 @@ export default function CanvasModel3DPage() {
onClick={() => handleSelectTask(task)}
type="button"
>
{previewUrl ? (
<img
alt="preview"
className="h-full w-full object-cover transition-transform duration-300 group-hover:scale-[1.035]"
src={previewUrl}
/>
) : (
<div className="flex h-full w-full items-center justify-center text-zinc-700">
<Box className="h-8 w-8" />
</div>
)}
<TaskAssetPreview task={task} />
</button>
<div className="border-white/10 border-t bg-zinc-950/95 px-1.5 py-2">
<div className="flex items-center justify-center gap-1">

View File

@ -3,6 +3,9 @@
/* include utility classes in streamdown */
@source '../node_modules/streamdown/dist/index.js';
@source '../components/canvas/**/*.tsx';
@source not '../.next*/**';
@source not '../.qa/**';
@source not '../services/pascal-editor';
/* custom variant for setting dark mode programmatically */
@ -405,6 +408,15 @@
}
/* Dense, shared system-settings language. Scoped so canvas tools keep their own scale. */
.video-canvas-workspace {
/* Match the compact typography used by the reference canvas. */
--canvas-font-caption: 12px;
--canvas-font-body: 13px;
--canvas-font-heading: 14px;
font-size: var(--canvas-font-body);
line-height: 1.45;
}
.settings-workspace {
font-size: 13px;
line-height: 1.45;

View File

@ -0,0 +1,315 @@
"use client";
import { useEffect, useId, useRef, useState } from "react";
import { LayoutTemplate, Loader2, Upload } from "lucide-react";
import { toast } from "sonner";
import { AgentPreviewImage } from "@/components/agents/AgentPreviewImage";
import { Button } from "@/components/ui/button";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea";
import { fetchUploadWithRetry } from "@/lib/client/upload-with-retry";
import { imageGenerationService } from "@/services/canvas";
const POLL_INTERVAL_MS = 2500;
const POLL_LIMIT = 240;
export type AgentAiTemplateResult = {
imageUrl: string;
sourceImageUrl: string;
referenceImageUrl: string;
};
type AgentAiTemplateDialogProps = {
open: boolean;
sourceImageUrl?: string | null;
model?: string | null;
imageSize?: string | null;
projectId: string;
onOpenChange: (open: boolean) => void;
onProcessingChange?: (processing: boolean, sourceImageUrl: string) => void;
onResultCreated: (result: AgentAiTemplateResult) => void | Promise<void>;
};
function isHttpImageUrl(value?: string | null) {
return /^https?:\/\//i.test(String(value || "").trim());
}
function sleep(ms: number) {
return new Promise((resolve) => window.setTimeout(resolve, ms));
}
async function pollAiTemplateTask(taskId: string) {
for (let attempt = 0; attempt < POLL_LIMIT; attempt += 1) {
await sleep(POLL_INTERVAL_MS);
const taskResult = await imageGenerationService.getImageTask(taskId);
const task = taskResult.data?.task;
if (!taskResult.success || !task) {
continue;
}
const status = String(task.status || "").toLowerCase();
if (status === "succeeded" || status === "completed") {
const image = task.result?.images?.[0];
const imageUrl = String(image?.originalUrl || image?.thumbnailUrl || "").trim();
if (!isHttpImageUrl(imageUrl)) {
throw new Error("AI套模版已完成但未返回结果图片");
}
return imageUrl;
}
if (["failed", "cancelled", "expired", "banned"].includes(status)) {
throw new Error(task.error || "AI套模版生成失败");
}
}
throw new Error("AI套模版任务等待超时,请稍后在结果区查看");
}
export function AgentAiTemplateDialog({
open,
sourceImageUrl,
model,
imageSize,
projectId,
onOpenChange,
onProcessingChange,
onResultCreated,
}: AgentAiTemplateDialogProps) {
const referenceUploadId = useId();
const [referenceImageUrl, setReferenceImageUrl] = useState("");
const [supplementalPrompt, setSupplementalPrompt] = useState("");
const [isUploading, setIsUploading] = useState(false);
const [isSubmitting, setIsSubmitting] = useState(false);
const normalizedSourceUrl = String(sourceImageUrl || "").trim();
const onProcessingChangeRef = useRef(onProcessingChange);
const onResultCreatedRef = useRef(onResultCreated);
useEffect(() => {
onProcessingChangeRef.current = onProcessingChange;
onResultCreatedRef.current = onResultCreated;
}, [onProcessingChange, onResultCreated]);
useEffect(() => {
if (!open || isSubmitting) return;
setReferenceImageUrl("");
setSupplementalPrompt("");
setIsUploading(false);
}, [isSubmitting, open, normalizedSourceUrl]);
const handleReferenceUpload = async (file?: File) => {
if (!file) return;
if (!file.type.startsWith("image/")) {
toast.error("请上传图片文件");
return;
}
setIsUploading(true);
try {
const formData = new FormData();
formData.append("file", file);
const response = await fetchUploadWithRetry(formData);
const payload = await response.json().catch(() => null);
const uploadedUrl = String(payload?.url || "").trim();
if (!response.ok || !isHttpImageUrl(uploadedUrl)) {
throw new Error(payload?.error || "参考图上传失败");
}
setReferenceImageUrl(uploadedUrl);
toast.success("版式参考图已上传");
} catch (error) {
toast.error(error instanceof Error ? error.message : "参考图上传失败");
} finally {
setIsUploading(false);
}
};
const handleSubmit = async () => {
const normalizedReferenceUrl = referenceImageUrl.trim();
if (!isHttpImageUrl(normalizedSourceUrl)) {
toast.error("当前主图结果地址不可用");
return;
}
if (!isHttpImageUrl(normalizedReferenceUrl)) {
toast.error("请先上传或填写版式参考图");
return;
}
if (isSubmitting) return;
setIsSubmitting(true);
onProcessingChangeRef.current?.(true, normalizedSourceUrl);
try {
const submitResult = await imageGenerationService.layoutReplace({
sourceImageUrl: normalizedSourceUrl,
referenceImageUrl: normalizedReferenceUrl,
supplementalPrompt: supplementalPrompt.trim() || undefined,
model: model && model !== "auto" ? model : undefined,
imageSize: imageSize || "2K",
agentProjectId: projectId,
referenceLabel: "风格化主图 AI 套模版参考图",
debugMeta: {
operation: "agent-main-ai-template",
source: "agent-main-result-card",
},
});
const taskId = String(submitResult.data?.taskId || "").trim();
if (!submitResult.success || !taskId) {
throw new Error(submitResult.error || "AI套模版提交失败");
}
onOpenChange(false);
toast.success("AI套模版已进入生图队列");
const resultImageUrl = await pollAiTemplateTask(taskId);
await onResultCreatedRef.current({
imageUrl: resultImageUrl,
sourceImageUrl: normalizedSourceUrl,
referenceImageUrl: normalizedReferenceUrl,
});
toast.success("AI套模版结果已加入主图结果区");
} catch (error) {
toast.error(error instanceof Error ? error.message : "AI套模版生成失败");
} finally {
setIsSubmitting(false);
onProcessingChangeRef.current?.(false, normalizedSourceUrl);
}
};
return (
<Dialog
open={open}
onOpenChange={(nextOpen) => {
if (!isUploading) {
onOpenChange(nextOpen);
}
}}
>
<DialogContent className="flex max-h-[calc(100dvh-1rem)] w-[calc(100vw-1rem)] max-w-[780px] flex-col gap-0 overflow-hidden border-white/10 bg-[#09090b] p-0 text-white sm:max-h-[calc(100dvh-2rem)] sm:w-[calc(100vw-2rem)]">
<DialogHeader className="shrink-0 border-b border-white/10 px-4 py-3 pr-12 text-left sm:px-6 sm:py-4">
<DialogTitle className="flex items-center gap-2 text-base">
<LayoutTemplate className="h-4 w-4" />
AI套模版
</DialogTitle>
<DialogDescription className="text-white/55">
保留参考图的版式、文字和图标,仅将产品主体替换为当前主图。
</DialogDescription>
</DialogHeader>
<div className="min-h-0 flex-1 overflow-y-auto overscroll-contain px-4 py-4 sm:px-6">
<div className="grid gap-4 md:grid-cols-2">
<div className="min-w-0 space-y-2">
<Label>当前主图</Label>
<div className="flex h-[clamp(180px,34dvh,340px)] items-center justify-center overflow-hidden rounded-lg border border-white/10 bg-zinc-900">
{isHttpImageUrl(normalizedSourceUrl) ? (
<AgentPreviewImage
alt="当前主图结果"
className="max-h-full max-w-full object-contain"
eager
previewQuality={86}
previewWidth={720}
src={normalizedSourceUrl}
/>
) : null}
</div>
</div>
<div className="min-w-0 space-y-2">
<Label htmlFor={referenceUploadId}>版式参考图</Label>
<div className="flex gap-2">
<Input
className="min-w-0 border-white/10 bg-white/[0.04]"
onChange={(event) => setReferenceImageUrl(event.currentTarget.value)}
placeholder="填写图片链接或上传"
value={referenceImageUrl}
/>
<Button
asChild
className="shrink-0 border-white/10 bg-white/[0.06] text-white hover:bg-white/[0.12]"
variant="outline"
>
<label htmlFor={referenceUploadId}>
{isUploading ? (
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
) : (
<Upload className="mr-2 h-4 w-4" />
)}
上传
</label>
</Button>
<input
accept="image/jpeg,image/png,image/webp,image/heic,image/heif,.heic,.heif"
className="sr-only"
disabled={isUploading || isSubmitting}
id={referenceUploadId}
onChange={(event) => {
const file = event.currentTarget.files?.[0];
event.currentTarget.value = "";
void handleReferenceUpload(file);
}}
type="file"
/>
</div>
<div className="flex h-[clamp(180px,34dvh,340px)] items-center justify-center overflow-hidden rounded-lg border border-dashed border-white/10 bg-zinc-900">
{isUploading ? (
<Loader2 className="h-5 w-5 animate-spin text-white/45" />
) : isHttpImageUrl(referenceImageUrl) ? (
<AgentPreviewImage
alt="AI套模版参考图"
className="max-h-full max-w-full object-contain"
eager
previewQuality={86}
previewWidth={720}
src={referenceImageUrl}
/>
) : (
<LayoutTemplate className="h-7 w-7 text-white/20" />
)}
</div>
</div>
</div>
<div className="mt-4 space-y-2">
<Label htmlFor={`${referenceUploadId}-prompt`}>补充要求</Label>
<Textarea
className="min-h-[84px] max-h-[180px] border-white/10 bg-white/[0.04]"
id={`${referenceUploadId}-prompt`}
onChange={(event) => setSupplementalPrompt(event.currentTarget.value)}
placeholder="可选:补充产品材质、颜色或替换细节"
value={supplementalPrompt}
/>
</div>
</div>
<DialogFooter className="shrink-0 gap-2 border-t border-white/10 bg-black/25 px-4 py-3 sm:px-6">
<Button
className="border-white/10 bg-white/[0.04] text-white hover:bg-white/[0.10] hover:text-white"
disabled={isUploading}
onClick={() => onOpenChange(false)}
type="button"
variant="outline"
>
取消
</Button>
<Button
disabled={isSubmitting || isUploading || !isHttpImageUrl(referenceImageUrl)}
onClick={() => void handleSubmit()}
type="button"
>
{isSubmitting ? (
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
) : (
<LayoutTemplate className="mr-2 h-4 w-4" />
)}
{isSubmitting ? "提交中" : "开始AI套模版"}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}

View File

@ -1,6 +1,8 @@
"use client";
import { useChat } from "@ai-sdk/react";
import { appendGenerationMessage, mergeGenerationHistory, parseAppendedGenerationMessage, recoverReplacementRun } from "@/lib/agents/replacement-run-recovery";
import type { DetailReplacementBatchCard } from "@/lib/agents/detail-replacement-batch";
import { DefaultChatTransport } from "ai";
import {
Fragment,
@ -26,6 +28,7 @@ import {
Download,
GitCompareArrows,
ImagePlus,
LayoutTemplate,
Loader2,
Maximize2,
Sparkles,
@ -37,6 +40,10 @@ import {
Settings2,
} from "lucide-react";
import { toast } from "sonner";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
import { Button } from "@/components/ui/button";
import { ClientErrorBoundary } from "@/components/ui/client-error-boundary";
import {
@ -64,6 +71,10 @@ import { ImageParamsBar, type ImageGenerationParams } from "@/components/image-p
import { AgentWorkflowShowcaseBanner } from "@/components/agents/AgentWorkflowShowcaseBanner";
import { AgentDetailReferenceStitchDialog } from "@/components/agents/AgentDetailReferenceStitchDialog";
import { AgentImageTextEditDialog } from "@/components/agents/AgentImageTextEditDialog";
import {
AgentAiTemplateDialog,
type AgentAiTemplateResult,
} from "@/components/agents/AgentAiTemplateDialog";
import { AgentPsdTemplateEditorDialog } from "@/components/agents/AgentPsdTemplateEditorDialog";
import { AgentPreviewImage } from "@/components/agents/AgentPreviewImage";
import {
@ -300,7 +311,7 @@ const MAX_CHAT_HIDDEN_PROMPT_LENGTH = 4000;
const MAX_CHAT_ATTACHMENT_NAME_LENGTH = 100;
const MAX_AGENT_MODE_QUERY_LENGTH = 64;
const MAX_AGENT_MODE_REFERENCE_COUNT = 40;
const DETAIL_LOCAL_EDIT_REFERENCE_MAX_UPLOAD_BYTES = 30 * 1024 * 1024;
const DETAIL_LOCAL_EDIT_REFERENCE_MAX_UPLOAD_BYTES = IMAGE_UPLOAD_MAX_BYTES;
const REPLACEMENT_RERUN_MAX_REFERENCE_IMAGES = 4;
const DETAIL_LOCAL_EDIT_MIN_RECT_RATIO = 0.025;
const DETAIL_SCALE_TARGET_MIN_PERCENT = -80;
@ -579,33 +590,7 @@ function areAttachmentValuesEqual(
);
}
function mergePersistedHistoryMessages(
currentMessages: ChatMessage[],
persistedMessages: ChatMessage[]
) {
if (persistedMessages.length === 0) {
return currentMessages;
}
if (currentMessages.length === 0) {
return persistedMessages;
}
const persistedIds = new Set(persistedMessages.map((message) => message.id));
const extraLocalMessages = currentMessages.filter(
(message) => !persistedIds.has(message.id)
);
const mergedMessages = [...persistedMessages, ...extraLocalMessages];
if (
mergedMessages.length === currentMessages.length &&
mergedMessages.every((message, index) => currentMessages[index]?.id === message.id)
) {
return currentMessages;
}
return mergedMessages;
}
const mergePersistedHistoryMessages = mergeGenerationHistory;
function getAttachmentIdentityKey(
attachment:
@ -662,9 +647,10 @@ const AGENT_DEFERRED_SECTION_STYLE = {
containIntrinsicSize: "1px 760px",
} as CSSProperties;
const AGENT_DEFERRED_RESULTS_STYLE = {
contentVisibility: "auto",
containIntrinsicSize: "1px 900px",
// Dynamic result content must stay painted while scrolling. Deferring this subtree
// makes Chromium briefly discard image and animation layers at viewport boundaries.
const AGENT_STABLE_RESULTS_STYLE = {
contentVisibility: "visible",
} as CSSProperties;
const AGENT_DEFERRED_LIST_CARD_STYLE = {
@ -672,6 +658,10 @@ const AGENT_DEFERRED_LIST_CARD_STYLE = {
containIntrinsicSize: "1px 320px",
} as CSSProperties;
const AGENT_STABLE_RESULT_CARD_STYLE = {
contentVisibility: "visible",
} as CSSProperties;
function handlePreviewImageLoadError(
event: SyntheticEvent<HTMLImageElement, Event>,
originalUrl: string
@ -709,7 +699,7 @@ function normalizeAssistantText(value: string) {
.replace(/\n*<!-- GENERATING_IMAGE -->\n*/g, "")
.replace(/\n*<!-- IMAGE_GENERATION_START -->\n*/g, "")
.replace(/\n*<!-- IMAGE_GENERATION_END -->\n*/g, "")
.replace(/\n*<!-- AGENT_REPLACEMENT_CARD_ID:[^>]*-->\n*/g, "")
.replace(/\n*<!-- AGENT_REPLACEMENT_(?:CARD_ID|RUN_ID|STATUS):[^>]*-->\n*/g, "")
.replace(/!\[[^\]]*\]\((https?:\/\/[^\s)]+)\)/g, "")
.trim();
}
@ -832,7 +822,7 @@ async function uploadDetailLocalEditReferenceImage(
throw new Error("请上传图片文件");
}
if (file.size > DETAIL_LOCAL_EDIT_REFERENCE_MAX_UPLOAD_BYTES) {
throw new Error("参考图过大,请先压缩到 30MB 以内");
throw new Error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
}
const formData = new FormData();
@ -2574,10 +2564,24 @@ interface LocalGenerationPendingState {
modeType: AgentGenerationResultModeType;
expectedCount: number;
userTurnCountAtStart: number;
requestMessageId?: string | null;
backendTaskId?: string | null;
timestamp: number;
requestedCardIds?: string[] | null;
}
type DetailReplacementBatchTaskView = {
id: string;
status: "pending" | "running" | "succeeded" | "failed";
error: string | null;
attempts: number;
cards: DetailReplacementBatchCard[];
currentCardId: string | null;
requestedCardIds: string[];
createdAt: string;
queueStatus?: { queuePosition?: number } | null;
};
const AGENT_PENDING_GENERATION_STORAGE_KEY = "ai-chatbot-agent-pending-generations";
const AGENT_PENDING_GENERATION_EXPIRY_MS = 30 * 60 * 1000;
const AGENT_SCENE_PENDING_FAILURE_GRACE_MS = 3 * 60 * 1000;
@ -2628,6 +2632,8 @@ function getStoredAgentPendingGenerations(): LocalGenerationPendingState[] {
modeType: String(record.modeType || "").trim() as AgentGenerationResultModeType,
expectedCount: Number(record.expectedCount),
userTurnCountAtStart: Number(record.userTurnCountAtStart),
requestMessageId: typeof record.requestMessageId === "string" ? record.requestMessageId : null,
backendTaskId: typeof record.backendTaskId === "string" ? record.backendTaskId : null,
timestamp: Number(record.timestamp),
requestedCardIds: Array.isArray(record.requestedCardIds)
? record.requestedCardIds
@ -2660,16 +2666,6 @@ function saveStoredAgentPendingGenerations(items: LocalGenerationPendingState[])
}
}
const DETAIL_RESULT_PROCESSING_STAGES = [
"解码图像",
"定位选区",
"分离主体",
"重绘背景",
"融合边缘",
"检查细节",
"输出结果",
];
function DetailResultProcessingScanOverlay({
imageUrl,
label,
@ -2677,55 +2673,43 @@ function DetailResultProcessingScanOverlay({
imageUrl?: string;
label: string;
}) {
const [progress, setProgress] = useState(6);
useEffect(() => {
const startedAt = Date.now();
const durationMs = 240_000;
const timer = window.setInterval(() => {
const elapsed = Date.now() - startedAt;
const linear = Math.min(1, elapsed / durationMs);
const eased = 1 - Math.pow(1 - linear, 1.8);
const drift = Math.sin(elapsed / 3200) * 1.2;
const next = Math.max(6, Math.min(96, Math.round(eased * 96 + drift)));
setProgress((current) => Math.max(current, next));
}, 720);
return () => window.clearInterval(timer);
}, [imageUrl, label]);
const stageIndex = Math.min(
DETAIL_RESULT_PROCESSING_STAGES.length - 1,
Math.floor((progress / 100) * DETAIL_RESULT_PROCESSING_STAGES.length)
);
const stageName = DETAIL_RESULT_PROCESSING_STAGES[stageIndex] || "处理中";
return (
<div className="absolute inset-0 z-30 overflow-hidden rounded-[inherit] bg-[#07090f]">
<style>{`
@keyframes detailResultScanBreatheBlur {
0%, 100% { filter: blur(14px) brightness(0.8) saturate(0.9); transform: scale(1.08); }
50% { filter: blur(17px) brightness(0.9) saturate(0.98); transform: scale(1.1); }
@keyframes detailResultScanBreathe {
0%, 100% { opacity: 0.68; transform: translate3d(0, 0, 0) scale(1.06); }
50% { opacity: 0.8; transform: translate3d(0, 0, 0) scale(1.075); }
}
@keyframes detailResultScanSweep {
0% { top: -7%; opacity: 0; }
0% { opacity: 0; transform: translate3d(0, -100%, 0); }
8% { opacity: 1; }
92% { opacity: 1; }
100% { top: 107%; opacity: 0; }
100% { opacity: 0; transform: translate3d(0, 0, 0); }
}
@keyframes detailResultScanPulseRing {
0% { box-shadow: 0 0 0 0 rgba(125, 211, 252, 0.48); }
70% { box-shadow: 0 0 0 8px rgba(125, 211, 252, 0); }
100% { box-shadow: 0 0 0 0 rgba(125, 211, 252, 0); }
}
@media (prefers-reduced-motion: reduce) {
[data-detail-result-scan-animation] { animation: none !important; }
}
`}</style>
{isHttpImageUrl(imageUrl) ? (
<img
<AgentPreviewImage
alt=""
aria-hidden
data-detail-result-scan-animation
className="absolute inset-0 h-full w-full object-cover"
src={imageUrl}
style={{ animation: "detailResultScanBreatheBlur 6s ease-in-out infinite" }}
previewWidth={720}
previewQuality={62}
rootMargin="320px 0px"
style={{
animation: "detailResultScanBreathe 6s ease-in-out infinite",
filter: "blur(12px) brightness(0.84) saturate(0.92)",
willChange: "opacity, transform",
}}
/>
) : (
<div
@ -2747,8 +2731,12 @@ function DetailResultProcessingScanOverlay({
/>
<div
aria-hidden
className="absolute inset-x-0 top-0 z-40 h-px pointer-events-none"
style={{ animation: "detailResultScanSweep 2.6s cubic-bezier(0.45, 0.05, 0.55, 0.95) infinite" }}
data-detail-result-scan-animation
className="pointer-events-none absolute inset-0 z-40"
style={{
animation: "detailResultScanSweep 2.6s cubic-bezier(0.45, 0.05, 0.55, 0.95) infinite",
willChange: "opacity, transform",
}}
>
<div
className="absolute inset-x-0 bottom-0 h-[110px]"
@ -2759,7 +2747,7 @@ function DetailResultProcessingScanOverlay({
}}
/>
<div
className="absolute inset-x-0 top-0 h-[38px]"
className="absolute inset-x-0 -bottom-[38px] h-[38px]"
style={{
background:
"linear-gradient(0deg, transparent, rgba(255,255,255,0.1))",
@ -2769,15 +2757,23 @@ function DetailResultProcessingScanOverlay({
</div>
<div
aria-hidden
className="absolute inset-x-0 top-0 z-50 h-px pointer-events-none"
data-detail-result-scan-animation
className="pointer-events-none absolute inset-0 z-50"
style={{
animation: "detailResultScanSweep 2.6s cubic-bezier(0.45, 0.05, 0.55, 0.95) infinite",
background:
"linear-gradient(90deg, transparent, rgba(167,139,250,0.7) 18%, rgba(255,255,255,0.98) 50%, rgba(125,211,252,0.7) 82%, transparent)",
boxShadow:
"0 0 6px rgba(255,255,255,0.7), 0 0 18px rgba(125,211,252,0.55), 0 0 34px rgba(167,139,250,0.35)",
willChange: "opacity, transform",
}}
/>
>
<div
className="absolute inset-x-0 bottom-0 h-px"
style={{
background:
"linear-gradient(90deg, transparent, rgba(167,139,250,0.7) 18%, rgba(255,255,255,0.98) 50%, rgba(125,211,252,0.7) 82%, transparent)",
boxShadow:
"0 0 6px rgba(255,255,255,0.7), 0 0 18px rgba(125,211,252,0.55), 0 0 34px rgba(167,139,250,0.35)",
}}
/>
</div>
<div
className="absolute right-3 top-3 z-[70] isolate inline-flex items-center gap-1.5 rounded-full border border-white/14 bg-[#080b12]/95 px-2.5 py-1.5 text-[11px] font-semibold leading-none text-white shadow-[0_12px_28px_rgba(0,0,0,0.34)]"
style={{
@ -2790,26 +2786,6 @@ function DetailResultProcessingScanOverlay({
style={{ animation: "detailResultScanPulseRing 1.8s ease-out infinite" }}
/>
<span>{label}</span>
<span className="min-w-[34px] text-right tabular-nums text-slate-300">{progress}%</span>
</div>
<div
className="absolute bottom-3 left-3 z-[70] isolate inline-flex items-center gap-2 rounded-full border border-white/14 bg-[#080b12]/95 px-2.5 py-1.5 text-[11px] font-semibold leading-none text-slate-100 shadow-[0_12px_28px_rgba(0,0,0,0.32)]"
style={{
WebkitFontSmoothing: "antialiased",
textShadow: "0 1px 1px rgba(0,0,0,0.72)",
}}
>
<span className="tabular-nums text-sky-300">{String(stageIndex + 1).padStart(2, "0")}</span>
<span>{stageName}</span>
</div>
<div className="absolute inset-x-0 bottom-0 z-[60] h-0.5 bg-white/5">
<div
className="h-full rounded-r-full shadow-[0_0_8px_rgba(125,211,252,0.6)] transition-[width] duration-300 ease-linear"
style={{
background: "linear-gradient(90deg, #7dd3fc, #a78bfa)",
width: `${progress}%`,
}}
/>
</div>
</div>
);
@ -3152,6 +3128,8 @@ export function AgentGenerationWorkspaceDarkV2({
const lastSyncedDetailModelSlotKeyRef = useRef<string | null>(null);
const detailGenerationCancelRef = useRef(false);
const workspaceStateSignatureRef = useRef<string>("");
const detailBatchSubmittingRef = useRef(false);
const reportedDetailBatchFailureRef = useRef<string | null>(null);
const generationRequestSnapshotRef = useRef<{
hiddenPrompt: string | null;
replacementExecutionCardIds: string[] | null;
@ -3236,6 +3214,10 @@ export function AgentGenerationWorkspaceDarkV2({
const [psdTemplateDialogMode, setPsdTemplateDialogMode] = useState<AgentPsdTemplateMode>("main");
const [psdTemplateDialogResultModeType, setPsdTemplateDialogResultModeType] =
useState<AgentGenerationResultModeType>("scene_generation");
const [aiTemplateDialogSourceUrl, setAiTemplateDialogSourceUrl] =
useState<string | null>(null);
const [aiTemplateProcessingSourceUrl, setAiTemplateProcessingSourceUrl] =
useState<string | null>(null);
const [batchDownloadState, setBatchDownloadState] =
useState<null | BatchDownloadScope>(null);
const [batchDownloadDialog, setBatchDownloadDialog] =
@ -3388,7 +3370,7 @@ export function AgentGenerationWorkspaceDarkV2({
}
}, [closePsdTemplateDialog, isPsdTemplateDialogOpen, isPsdTemplateEnabled]);
const { data: historyData } = useSWR<{ messages: ChatMessage[] }>(
const { data: historyData, mutate: refreshHistory } = useSWR<{ messages: ChatMessage[] }>(
rawChatId ? `/api/messages?chatId=${rawChatId}` : null,
fetcher,
{
@ -3482,6 +3464,75 @@ export function AgentGenerationWorkspaceDarkV2({
const isDetailAiGenerationMethod = currentResultModeType === "detail_generation";
const isReplacementPreset =
currentResultModeType === "detail_replacement" || currentResultModeType === "sku_replacement";
const { data: detailBatchData, mutate: refreshDetailBatch } = useSWR<{
task: DetailReplacementBatchTaskView | null;
}>(
projectId && currentResultModeType === "detail_replacement"
? `/api/agents/detail-replacement/batches?projectId=${encodeURIComponent(projectId)}`
: null,
fetcher,
{ refreshInterval: 3000, revalidateOnFocus: true },
);
const detailBatchTask = detailBatchData?.task || null;
useEffect(() => {
if (!detailBatchData || currentResultModeType !== "detail_replacement") return;
const task = detailBatchData.task;
if (task && (task.status === "pending" || task.status === "running")) {
setPendingGeneration((current) => {
if (current?.backendTaskId === task.id) return current;
return {
chatId,
requestMessageId: task.id,
backendTaskId: task.id,
modeType: "detail_replacement",
expectedCount: task.requestedCardIds.length,
requestedCardIds: task.requestedCardIds,
userTurnCountAtStart: 0,
timestamp: Date.parse(task.createdAt) || Date.now(),
};
});
} else {
setPendingGeneration((current) =>
current?.modeType === "detail_replacement" &&
(!current.backendTaskId || current.backendTaskId === task?.id)
? null
: current,
);
}
}, [chatId, currentResultModeType, detailBatchData]);
useEffect(() => {
const cards = detailBatchTask?.cards || [];
if (cards.length === 0) return;
setReplacementResultState((current) => {
const next = { ...current.detail_replacement };
let changed = false;
for (const card of cards) {
const reference = detailReplacementReferenceCards.find((item) => item.id === card.cardId);
if (!reference || !card.imageUrl || next[card.cardId]?.url === card.imageUrl) continue;
next[card.cardId] = {
url: card.imageUrl,
text: card.text,
status: card.status,
updatedAt: Date.now(),
compareSourceUrl: reference.imageUrl,
referenceImageUrl: reference.imageUrl,
label: reference.label,
};
changed = true;
}
return changed ? { ...current, detail_replacement: next } : current;
});
}, [detailBatchTask?.cards, detailReplacementReferenceCards]);
useEffect(() => {
if (
detailBatchTask?.status !== "failed" ||
detailBatchTask.attempts < 3 ||
reportedDetailBatchFailureRef.current === detailBatchTask.id
) return;
reportedDetailBatchFailureRef.current = detailBatchTask.id;
toast.error(detailBatchTask.error || "详情替换批次中断,已保留生成成功的图片,可继续处理剩余图片");
}, [detailBatchTask]);
const isProductOnlyDetailReplacementMode =
currentResultModeType === "detail_replacement" &&
detailReplacementStrategy === "product_only";
@ -4129,6 +4180,16 @@ export function AgentGenerationWorkspaceDarkV2({
} = useChat<ChatMessage>({
id: chatId,
messages: historyMessages,
onData(part) {
if (part.type !== "data-appendMessage") return;
try {
const appended = parseAppendedGenerationMessage(part.data);
if (appended) setMessages((current) => appendGenerationMessage(current, appended as ChatMessage));
} catch {
void refreshHistory();
}
},
onFinish() { void refreshHistory(); },
experimental_throttle: 100,
generateId: generateUUID,
transport: new DefaultChatTransport({
@ -4262,6 +4323,20 @@ export function AgentGenerationWorkspaceDarkV2({
}),
});
useEffect(() => {
const reconcile = () => {
if (document.visibilityState === "visible") void refreshHistory();
};
document.addEventListener("visibilitychange", reconcile);
window.addEventListener("online", reconcile);
window.addEventListener("pageshow", reconcile);
return () => {
document.removeEventListener("visibilitychange", reconcile);
window.removeEventListener("online", reconcile);
window.removeEventListener("pageshow", reconcile);
};
}, [refreshHistory]);
useEffect(() => {
if (!historyMessages.length) {
return;
@ -4482,8 +4557,7 @@ export function AgentGenerationWorkspaceDarkV2({
if (!hasLikelyGenerationErrorText(text)) {
return false;
}
const imageUrls = extractImageUrlsFromText(text);
return imageUrls.length === 0;
return true;
}),
[assistantMessagesAfterLastUser]
);
@ -4791,6 +4865,12 @@ export function AgentGenerationWorkspaceDarkV2({
: currentResultModeType === "sku_replacement"
? skuReplacementResultUrls
: currentSceneGeneratedUrls;
const replacementRunRecovery = useMemo(() => {
if (!pendingGeneration || !["detail_replacement", "sku_replacement"].includes(pendingGeneration.modeType)) {
return { completedCardIds: [] as string[], failedCardIds: [] as string[], done: false };
}
return recoverReplacementRun({ messages, ...pendingGeneration });
}, [messages, pendingGeneration]);
const transportIsGenerating = status === "streaming" || status === "submitted";
const pendingGenerationProgressCount = useMemo(() => {
if (!pendingGeneration) {
@ -4812,13 +4892,14 @@ export function AgentGenerationWorkspaceDarkV2({
return detailGenerationRunResults.length;
}
return latestReplacementResults.length;
return replacementRunRecovery.completedCardIds.length;
}, [
currentSceneGeneratedUrls.length,
detailGenerationRunResults.length,
latestReplacementResults.length,
pendingGeneration,
userMessageCount,
replacementRunRecovery.completedCardIds.length,
]);
const isAnyGenerationRunning = transportIsGenerating || Boolean(pendingGeneration);
const isSceneResultModeGenerating =
@ -4948,21 +5029,33 @@ export function AgentGenerationWorkspaceDarkV2({
Array.isArray(pendingGeneration.requestedCardIds)
? pendingGeneration.requestedCardIds.length
: null,
status: getReplacementCardStatus({
status: detailBatchTask?.requestedCardIds.includes(card.id)
? resultMeta?.url
? resultMeta.status
: detailBatchTask.status === "running"
? detailBatchTask.currentCardId === card.id ? "generating" : "queued"
: detailBatchTask.status === "pending"
? "queued"
: detailBatchTask.status === "failed"
? "missing"
: "waiting"
: getReplacementCardStatus({
index,
resultUrl,
resultText,
generatedCount: pendingGeneration?.modeType === "detail_replacement"
? latestReplacementResults.length
: 0,
isGenerating: isDetailResultModeGenerating,
isGenerating: isDetailResultModeGenerating && !replacementRunRecovery.completedCardIds.includes(card.id),
activeRunIndex: activeRunIndex >= 0 ? activeRunIndex : null,
}),
};
}),
[
detailReplacementReferenceCards,
detailBatchTask,
isDetailResultModeGenerating,
replacementRunRecovery.completedCardIds,
latestReplacementResults,
normalizedDeletedResultCollections.detail_replacement,
pendingGeneration?.modeType,
@ -5014,13 +5107,14 @@ export function AgentGenerationWorkspaceDarkV2({
generatedCount: pendingGeneration?.modeType === "sku_replacement"
? latestReplacementResults.length
: 0,
isGenerating: isSkuResultModeGenerating,
isGenerating: isSkuResultModeGenerating && !replacementRunRecovery.completedCardIds.includes(card.id),
activeRunIndex: activeRunIndex >= 0 ? activeRunIndex : null,
}),
};
}),
[
isSkuResultModeGenerating,
replacementRunRecovery.completedCardIds,
latestReplacementResults,
normalizedDeletedResultCollections.sku_replacement,
pendingGeneration?.modeType,
@ -5114,7 +5208,7 @@ export function AgentGenerationWorkspaceDarkV2({
const detailRemainingCount = useMemo(
() =>
detailResultCards.filter(
(card) => !["completed", "passthrough", "fallback"].includes(card.status)
(card) => !["completed", "passthrough"].includes(card.status)
).length,
[detailResultCards]
);
@ -5389,7 +5483,15 @@ export function AgentGenerationWorkspaceDarkV2({
}, [currentResultModeType]);
useEffect(() => {
if (!pendingGeneration) {
if (!pendingGeneration || pendingGeneration.backendTaskId || !replacementRunRecovery.done) return;
// The backend has settled every requested card; an open SSE connection is no longer work.
stop();
transportGenerationStartedAtRef.current = null;
setPendingGeneration(null);
}, [pendingGeneration, replacementRunRecovery.done, stop]);
useEffect(() => {
if (!pendingGeneration || pendingGeneration.backendTaskId) {
return;
}
@ -5399,7 +5501,7 @@ export function AgentGenerationWorkspaceDarkV2({
}, [pendingGeneration, pendingGenerationProgressCount]);
useEffect(() => {
if (!pendingGeneration || !chatError) {
if (!pendingGeneration || pendingGeneration.backendTaskId || !chatError) {
return;
}
@ -5414,7 +5516,7 @@ export function AgentGenerationWorkspaceDarkV2({
}, [chatError, clearError, pendingGeneration]);
useEffect(() => {
if (!pendingGeneration) {
if (!pendingGeneration || pendingGeneration.backendTaskId) {
return;
}
@ -6140,7 +6242,6 @@ export function AgentGenerationWorkspaceDarkV2({
const processedStatuses = new Set<ReplacementCardStatus>([
"completed",
"passthrough",
"fallback",
]);
const remainingIds = resultCards
.filter((card) => !card.passthrough && !processedStatuses.has(card.status))
@ -6246,6 +6347,86 @@ export function AgentGenerationWorkspaceDarkV2({
toast.error("当前风格化绑定的模型不存在或已停用,请重新编辑风格化模型配置");
return;
}
if (replacementModeType === "detail_replacement") {
if (detailBatchSubmittingRef.current) return;
if (!isValidChatUuid(projectId)) {
toast.error("当前风格化项目尚未保存,请先保存项目后再执行详情替换");
return;
}
const productImageUrls = Array.from(new Set(
(detailReplacementProductSourceImages.length > 0
? detailReplacementProductSourceImages.map((item) => item.url)
: selectedProductAttachments.map((item) => item.url))
.filter((url) => isValidChatHttpUrl(url)),
));
if (productImageUrls.length === 0 && !replacementExecutionCards.every((card) => card.passthrough)) {
toast.error("请先在输入图区选择至少一张产品图");
return;
}
detailBatchSubmittingRef.current = true;
void (async () => {
try {
const saved = await onPersistProject?.();
if (saved === false) throw new Error("项目尚未保存,请重试");
const requestId = generateUUID();
const response = await fetchWithErrorHandlers("/api/agents/detail-replacement/batches", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
projectId,
requestId,
model: resolvedModelId,
imageSize: imageParams.imageSize,
replacementStrategy: detailReplacementStrategy,
prompt: detailReplacementStrategy === "product_only"
? "仅替换参考详情图中的产品主体"
: String(resolvedWorkflowContext?.promptText || "").trim(),
userPrompt: String(options?.promptText || input || "").trim(),
productImageUrls,
modelImageUrl: detailReplacementModelSourceImage?.url || selectedModelAttachment?.url || null,
references: replacementExecutionCards.map((card) => ({
id: card.id,
label: card.label,
imageUrl: card.imageUrl,
candidateId: card.candidateId,
passthrough: card.passthrough,
rerunReferenceImages:
options?.replacementRerunReferenceImages?.cardId === card.id
? options.replacementRerunReferenceImages.images
: [],
})),
}),
});
const payload = await response.json().catch(() => ({}));
if (!response.ok || !payload?.task?.id) {
throw new Error(payload?.error || "详情替换任务提交失败");
}
setPendingGeneration({
chatId,
requestMessageId: requestId,
backendTaskId: payload.task.id,
modeType: "detail_replacement",
expectedCount: replacementExecutionCards.length,
userTurnCountAtStart: userMessageCount,
timestamp: Date.now(),
requestedCardIds: replacementExecutionCards.map((card) => card.id),
});
setReplacementExecutionContext({
modeType: "detail_replacement",
cardIds: replacementExecutionCards.map((card) => card.id),
});
setActiveResultModeType("detail_replacement");
setSelectedResultUrl(null);
await refreshDetailBatch();
toast.success(`已提交 ${replacementExecutionCards.length} 张详情图,离开页面后仍会继续处理`);
} catch (error) {
toast.error(error instanceof Error ? error.message : "详情替换任务提交失败");
} finally {
detailBatchSubmittingRef.current = false;
}
})();
return;
}
const parts: Array<{ type: "text"; text: string } | { type: "file"; mediaType: string; name: string; url: string; assetId?: string }> = [];
const sanitizedAttachments = sceneMessageAttachments;
if (
@ -6344,8 +6525,10 @@ export function AgentGenerationWorkspaceDarkV2({
);
setActiveResultModeType(currentResultModeType);
setSelectedResultUrl(null);
const requestMessageId = generateUUID();
setPendingGeneration({
chatId,
requestMessageId,
modeType: currentResultModeType,
expectedCount:
currentResultModeType === "scene_generation"
@ -6366,7 +6549,7 @@ export function AgentGenerationWorkspaceDarkV2({
toast.success(`继续处理剩余 ${replacementExecutionCards.length} 张结果图`);
}
try {
void Promise.resolve(sendMessage({ role: "user", parts })).catch((error) => {
void Promise.resolve(sendMessage({ id: requestMessageId, role: "user", parts })).catch((error) => {
generationRequestSnapshotRef.current = {
hiddenPrompt: null,
replacementExecutionCardIds: null,
@ -6393,6 +6576,8 @@ export function AgentGenerationWorkspaceDarkV2({
detailResultCards,
detailReplacementReferenceCards.length,
detailReplacementStrategy,
detailReplacementProductSourceImages,
detailReplacementModelSourceImage,
handleRunAiDetailGeneration,
currentResultModeType,
getReplacementExecutionCards,
@ -6423,6 +6608,10 @@ export function AgentGenerationWorkspaceDarkV2({
scenePrimaryReferenceCount,
sceneSecondaryReferenceCount,
selectedModelAttachment,
selectedProductAttachments,
onPersistProject,
projectId,
refreshDetailBatch,
]);
const handleStopGeneration = useCallback(() => {
detailGenerationCancelRef.current = true;
@ -6818,6 +7007,23 @@ export function AgentGenerationWorkspaceDarkV2({
}, 2800);
}, []);
const handleAiTemplateResultCreated = useCallback(
(result: AgentAiTemplateResult) => {
const appended = appendResultUrlToCollection("scene_generation", result.imageUrl, {
compareSourceUrl: result.sourceImageUrl,
});
if (!appended) {
toast.error("当前项目未接入结果保存能力");
return;
}
setActiveResultModeType("scene_generation");
setSelectedResultUrl(result.imageUrl);
showProcessedResultHint(result.imageUrl, "AI套模版完成");
},
[appendResultUrlToCollection, showProcessedResultHint]
);
const updateDetailGenerationProcessedResult = useCallback(
(params: {
cardId?: string | null;
@ -9262,7 +9468,7 @@ export function AgentGenerationWorkspaceDarkV2({
</>
)}
</Button>
{isAnyGenerationRunning ? (
{isAnyGenerationRunning && !pendingGeneration?.backendTaskId ? (
<Button
type="button"
variant="outline"
@ -9640,7 +9846,7 @@ export function AgentGenerationWorkspaceDarkV2({
initial={{ opacity: 0, scale: 0.98 }}
animate={{ opacity: 1, scale: 1 }}
transition={{ delay: 0.12 }}
style={AGENT_DEFERRED_RESULTS_STYLE}
style={AGENT_STABLE_RESULTS_STYLE}
data-agent-style-section="results"
>
<div className="flex items-center justify-between border-b border-white/12 px-5 py-4">
@ -9826,7 +10032,7 @@ export function AgentGenerationWorkspaceDarkV2({
selectedProductAttachments.map((attachment, index) => (
<div
key={`${attachment?.url || "input"}-${index}`}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className="overflow-hidden rounded-[22px] border border-white/10 bg-black/20"
>
<div className="aspect-square overflow-hidden bg-black/30">
@ -9942,7 +10148,7 @@ export function AgentGenerationWorkspaceDarkV2({
return (
<div
key={card.id}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className={cn(
"w-full cursor-pointer overflow-hidden rounded-[22px] border bg-black/20 transition-colors",
isActive
@ -10107,7 +10313,7 @@ export function AgentGenerationWorkspaceDarkV2({
return (
<div
key={`${card.id}-result`}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className={cn(
"w-full cursor-pointer overflow-hidden rounded-[22px] border bg-black/20 transition-colors",
hasResult && isActive
@ -10378,7 +10584,7 @@ export function AgentGenerationWorkspaceDarkV2({
selectedProductAttachments.map((attachment, index) => (
<div
key={`${attachment?.url || "sku-input"}-${index}`}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className="overflow-hidden rounded-[22px] border border-white/10 bg-black/20"
>
<div className="aspect-square overflow-hidden bg-black/30">
@ -10550,7 +10756,7 @@ export function AgentGenerationWorkspaceDarkV2({
key={candidate.id}
type="button"
onClick={() => handleSelectSkuReplacementCandidate(candidate.id)}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className={cn(
"min-w-[280px] overflow-hidden rounded-[20px] border text-left transition-colors",
isSelected
@ -10617,7 +10823,7 @@ export function AgentGenerationWorkspaceDarkV2({
skuReplacementReferenceCards.map((card, index) => (
<div
key={card.id}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className={cn(
"w-full cursor-pointer overflow-hidden rounded-[22px] border bg-black/20 transition-colors",
activeSkuReferenceCardId === card.id
@ -10746,7 +10952,7 @@ export function AgentGenerationWorkspaceDarkV2({
return (
<div
key={`${card.id}-result`}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
className={cn(
"w-full cursor-pointer overflow-hidden rounded-[22px] border bg-black/20 transition-colors",
hasResult && activeSkuReferenceCardId === card.id
@ -10885,12 +11091,21 @@ export function AgentGenerationWorkspaceDarkV2({
className="flex flex-1 flex-col px-6 pb-6 pt-5 md:px-8"
>
<div className="mx-auto w-full max-w-[1200px]">
<div className="grid grid-cols-2 gap-4 md:grid-cols-4">
<div
className={cn(
"grid gap-4",
activeResultModeType === "scene_generation"
? "grid-cols-1 md:grid-cols-2"
: "grid-cols-2 md:grid-cols-4"
)}
>
{gridResultCards.map((card) => {
const isActive = card.url && displayedResultUrl === card.url;
const isSceneResultCard = activeResultModeType === "scene_generation";
const isDetailGenerationCard =
activeResultModeType === "detail_generation";
const isAiTemplateProcessing =
aiTemplateProcessingSourceUrl === card.url;
const isDetailEnhancing = detailEnhancingCardIds.has(card.id);
const isDetailLocalEditBusy =
detailLocalEditSubmittingId === card.id;
@ -10913,7 +11128,7 @@ export function AgentGenerationWorkspaceDarkV2({
key={card.id}
role={card.url ? "button" : undefined}
tabIndex={card.url ? 0 : -1}
style={AGENT_DEFERRED_LIST_CARD_STYLE}
style={AGENT_STABLE_RESULT_CARD_STYLE}
onClick={() => {
if (!card.url) return;
setSelectedResultUrl(card.url);
@ -10953,36 +11168,70 @@ export function AgentGenerationWorkspaceDarkV2({
/>
{isSceneResultCard ? (
<>
<button
type="button"
onClick={(event) => {
event.stopPropagation();
setSelectedResultUrl(card.url!);
setSceneResizeEditorTargetUrl(card.url!);
}}
className={cn("absolute left-3 top-3 flex h-9 w-9 items-center justify-center", IMAGE_OVERLAY_ACTIVE_BADGE_CLASS, "px-0 py-0")}
aria-label="调整尺寸"
>
<Maximize2 className="h-4 w-4" />
</button>
{isPsdTemplateEnabled ? (
<div className="absolute left-3 right-14 top-3 flex flex-wrap items-center gap-2">
<button
type="button"
onClick={(event) => {
event.stopPropagation();
setSelectedResultUrl(card.url!);
openPsdTemplateDialog({
sourceImageUrl: card.url!,
mode: getPsdTemplateModeByResultMode(activeResultModeType),
});
setSceneResizeEditorTargetUrl(card.url!);
}}
className={cn("absolute left-14 top-3 flex items-center gap-1", IMAGE_OVERLAY_ACTION_CLASS)}
aria-label="套用 PSD 模板"
className={cn(
"flex h-9 w-9 items-center justify-center px-0 py-0",
IMAGE_OVERLAY_ACTIVE_BADGE_CLASS
)}
aria-label="调整尺寸"
>
<ImagePlus className="h-3.5 w-3.5" />
<span>套PSD</span>
<Maximize2 className="h-4 w-4" />
</button>
) : null}
{isPsdTemplateEnabled ? (
<button
type="button"
onClick={(event) => {
event.stopPropagation();
setSelectedResultUrl(card.url!);
openPsdTemplateDialog({
sourceImageUrl: card.url!,
mode: getPsdTemplateModeByResultMode(
activeResultModeType
),
});
}}
className={cn(
"flex items-center gap-1",
IMAGE_OVERLAY_ACTION_CLASS
)}
aria-label="套用 PSD 模板"
>
<ImagePlus className="h-3.5 w-3.5" />
<span>套PSD</span>
</button>
) : null}
<button
type="button"
onClick={(event) => {
event.stopPropagation();
setSelectedResultUrl(card.url!);
setAiTemplateDialogSourceUrl(card.url!);
}}
disabled={Boolean(aiTemplateProcessingSourceUrl)}
className={cn(
"flex items-center gap-1 disabled:cursor-not-allowed disabled:opacity-55",
IMAGE_OVERLAY_ACTION_CLASS
)}
aria-label="AI套模版"
aria-busy={isAiTemplateProcessing}
>
{isAiTemplateProcessing ? (
<Loader2 className="h-3.5 w-3.5 animate-spin" />
) : (
<LayoutTemplate className="h-3.5 w-3.5" />
)}
<span>
{isAiTemplateProcessing ? "AI套版中" : "AI套模版"}
</span>
</button>
</div>
<div className="pointer-events-none absolute inset-x-3 bottom-3 rounded-2xl border border-white/25 bg-zinc-950/82 px-3 py-2 font-semibold text-[11px] text-white shadow-[0_12px_34px_rgba(0,0,0,0.48)] backdrop-blur-md">
双击查看大图
</div>
@ -11230,6 +11479,29 @@ export function AgentGenerationWorkspaceDarkV2({
</div>
</main>
<AgentAiTemplateDialog
imageSize={imageParams.imageSize}
model={selectedModelId !== "auto" ? selectedModelId : effectiveDefaultModelId}
projectId={projectId}
onOpenChange={(open) => {
if (!open) {
setAiTemplateDialogSourceUrl(null);
}
}}
onProcessingChange={(processing, sourceImageUrl) => {
setAiTemplateProcessingSourceUrl((current) =>
processing
? sourceImageUrl
: current === sourceImageUrl
? null
: current
);
}}
onResultCreated={handleAiTemplateResultCreated}
open={Boolean(aiTemplateDialogSourceUrl)}
sourceImageUrl={aiTemplateDialogSourceUrl}
/>
{sceneResizeEditorTargetUrl ? (
<ImageAnnotator
key={`agent-scene-resize-${sceneResizeEditorTargetUrl}`}

View File

@ -1,7 +1,11 @@
import { fetchUploadWithRetry } from "@/lib/client/upload-with-retry";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
import type { AgentDetailReplacementReferenceItem } from "@/types/agent";
const DETAIL_REFERENCE_UPLOAD_MAX_BYTES = 50 * 1024 * 1024;
const DETAIL_REFERENCE_UPLOAD_MAX_BYTES = IMAGE_UPLOAD_MAX_BYTES;
export interface UploadedDetailReferenceImage {
url: string;
@ -64,7 +68,7 @@ export async function uploadDetailReferenceImageFile(
throw new Error("请上传图片文件");
}
if (file.size > DETAIL_REFERENCE_UPLOAD_MAX_BYTES) {
throw new Error("目标详情图过大,请压缩到 50MB 以内");
throw new Error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
}
const dimensionsPromise = readLocalImageDimensions(file);

259
components/bloub-avatar.tsx Normal file
View File

@ -0,0 +1,259 @@
"use client";
import { useEffect, useId, useRef, useState } from "react";
import { defaultCycle } from "@/lib/bloub-official/cycles";
import { NOTIF_BLUE } from "@/lib/bloub-official/decor";
import { type BotFrame, BotEngine } from "@/lib/bloub-official/engine";
import {
DEFAULT_EXPRESSION,
EXPRESSION_BY_ID,
} from "@/lib/bloub-official/expressions";
import { DEMI_VIEWBOX, RAYON } from "@/lib/bloub-official/repere";
import { SHAPE_BY_ID } from "@/lib/bloub-official/skins";
import type { StateId } from "@/lib/bloub-official/states";
import { cn } from "@/lib/utils";
export type BloubAvatarProps = {
animated?: boolean;
className?: string;
label?: string;
playSequence?: boolean;
size?: number;
state?: StateId;
surface?: "chat" | "page";
};
export function BloubAvatar({
animated = true,
className,
label = "AI 助手",
playSequence = false,
size = 32,
state = "idle",
surface = "chat",
}: BloubAvatarProps) {
const rawId = useId();
const uid = rawId.replace(/[^a-zA-Z0-9_-]/g, "");
const maskId = `bloub-mask-${uid}`;
const engineRef = useRef<BotEngine | null>(null);
const clockRef = useRef(0);
if (!engineRef.current) {
engineRef.current = new BotEngine(
RAYON,
state,
SHAPE_BY_ID.get("squircle")?.radii ?? null,
EXPRESSION_BY_ID.get(DEFAULT_EXPRESSION) ?? null
);
}
const [frame, setFrame] = useState<BotFrame>(() =>
engineRef.current!.sample(0)
);
useEffect(() => {
if (playSequence) return;
const engine = engineRef.current;
if (!engine || engine.state === state) return;
engine.setState(state, clockRef.current);
setFrame(engine.sample(clockRef.current));
}, [playSequence, state]);
useEffect(() => {
const engine = engineRef.current;
if (!engine) return;
const cycle = defaultCycle().blocks;
const firstState = playSequence
? cycle[0]?.state ?? "idle"
: engine.state;
engine.reset(firstState, 0);
clockRef.current = 0;
setFrame(engine.sample(0));
const reducedMotion = window.matchMedia(
"(prefers-reduced-motion: reduce)"
).matches;
if (!animated || reducedMotion) return;
let animationFrame = 0;
let lastFrameAt = 0;
let blockIndex = 0;
let nextBlockAt = cycle[0]?.duration ?? Number.POSITIVE_INFINITY;
const tick = (timestamp: number) => {
const delta = lastFrameAt
? Math.min((timestamp - lastFrameAt) / 1000, 0.064)
: 0;
lastFrameAt = timestamp;
clockRef.current += delta;
if (playSequence && cycle.length > 0) {
while (clockRef.current >= nextBlockAt) {
blockIndex = (blockIndex + 1) % cycle.length;
const nextBlock = cycle[blockIndex]!;
engine.setState(nextBlock.state, nextBlockAt);
nextBlockAt += nextBlock.duration;
}
}
setFrame(engine.sample(clockRef.current));
animationFrame = window.requestAnimationFrame(tick);
};
animationFrame = window.requestAnimationFrame(tick);
return () => window.cancelAnimationFrame(animationFrame);
}, [animated, playSequence]);
const renderDot = (
dot: BotFrame["dots"][number],
index: number,
layer: "back" | "front"
) => {
const fill = dot.color ?? "currentColor";
const common = {
fill,
opacity:
dot.color || dot.depth === undefined
? dot.opacity
: dot.opacity * dot.depth,
};
const key = `${layer}-${index}`;
if (dot.d) {
return (
<path
{...common}
d={dot.d}
key={key}
transform={`translate(${dot.x} ${dot.y}) rotate(${dot.rot ?? 0}) scale(${RAYON})`}
/>
);
}
return (
<circle {...common} cx={dot.x} cy={dot.y} key={key} r={dot.r} />
);
};
return (
<svg
aria-label={label}
className={cn(
"block shrink-0 text-[#0a0a0c] dark:text-[#3ecf8e]",
surface === "page"
? "[--bloub-paper:#f4f4f5] dark:[--bloub-paper:#08090b]"
: "[--bloub-paper:#ffffff] dark:[--bloub-paper:#09090b]",
className
)}
height={size}
role="img"
viewBox={`${-DEMI_VIEWBOX} ${-DEMI_VIEWBOX} ${DEMI_VIEWBOX * 2} ${DEMI_VIEWBOX * 2}`}
width={size}
>
<defs>
<mask
height={DEMI_VIEWBOX * 2}
id={maskId}
maskUnits="userSpaceOnUse"
width={DEMI_VIEWBOX * 2}
x={-DEMI_VIEWBOX}
y={-DEMI_VIEWBOX}
>
<path d={frame.bodyPath} fill="#fff" />
{frame.eyes.map((eye, index) => (
<path
d={eye.d}
fill="#000"
key={`eye-${index}`}
opacity={eye.alpha}
transform={eye.matrix}
/>
))}
{frame.notch ? (
<circle
cx={frame.notch.x}
cy={frame.notch.y}
fill="#000"
r={frame.notch.r}
/>
) : null}
</mask>
{frame.arcs.map((arc) => (
<linearGradient
gradientUnits="userSpaceOnUse"
id={`${uid}-${arc.id}`}
key={`gradient-${arc.id}`}
x1={arc.grad.x1}
x2={arc.grad.x2}
y1={arc.grad.y1}
y2={arc.grad.y2}
>
{arc.grad.stops.map((color, index) => (
<stop
key={`${arc.id}-${color}-${index}`}
offset={index / (arc.grad.stops.length - 1)}
stopColor={color}
/>
))}
</linearGradient>
))}
</defs>
<g fill="none" strokeLinecap="round">
{frame.arcs.map((arc) => (
<path
d={arc.back}
key={`arc-back-${arc.id}`}
opacity={arc.opacity}
stroke={`url(#${uid}-${arc.id})`}
strokeWidth={arc.width}
/>
))}
</g>
{frame.dotsBehind ? (
<g>{frame.dots.map((dot, index) => renderDot(dot, index, "back"))}</g>
) : null}
<g opacity={frame.bodyAlpha}>
<path d={frame.bodyPath} fill="var(--bloub-paper)" />
<g mask={`url(#${maskId})`}>
<rect
fill="currentColor"
height={DEMI_VIEWBOX * 2}
width={DEMI_VIEWBOX * 2}
x={-DEMI_VIEWBOX}
y={-DEMI_VIEWBOX}
/>
</g>
</g>
{!frame.dotsBehind ? (
<g>{frame.dots.map((dot, index) => renderDot(dot, index, "front"))}</g>
) : null}
{frame.notif ? (
<circle
cx={frame.notif.x}
cy={frame.notif.y}
fill={NOTIF_BLUE}
r={frame.notif.r}
/>
) : null}
<g fill="none" strokeLinecap="round">
{frame.arcs.map((arc) => (
<path
d={arc.front}
key={`arc-front-${arc.id}`}
opacity={arc.opacity}
stroke={`url(#${uid}-${arc.id})`}
strokeWidth={arc.width}
/>
))}
</g>
</svg>
);
}

View File

@ -46,7 +46,10 @@ import {
type AiWashControlLevel,
type AiWashPreset,
} from "@/lib/ai-wash/config";
import { downloadCanvasImageAsPng } from "@/lib/client/canvas-image-download";
import {
downloadCanvasImageAsPng,
getCanvasImageDownloadErrorMessage,
} from "@/lib/client/canvas-image-download";
import { buildImageProxyUrl } from "@/lib/image-delivery";
import { cn } from "@/lib/utils";
import { imageGenerationService } from "@/services/canvas";
@ -1019,7 +1022,7 @@ export function AiWashWorkspace({
});
toast.success("图片已开始下载");
} catch (error) {
toast.error(error instanceof Error ? error.message : "下载失败");
toast.error(getCanvasImageDownloadErrorMessage(error));
} finally {
setDownloadingResultId((current) =>
current === downloadId ? null : current

View File

@ -0,0 +1,311 @@
"use client";
import { useEffect, useState } from "react";
import {
audioContract,
audioRequestSchema,
buildAudioRequest,
type AudioRequest,
} from "@/lib/video-canvas/audio-contract";
type Model = { id: string; modelId: string; name: string; kind: string };
export function AudioGenerationComposer({
initial,
connectedText = "",
busy,
onSubmit,
onUpload,
onDraft,
}: {
initial?: Record<string, unknown>;
connectedText?: string;
busy: boolean;
onSubmit: (params: AudioRequest, draft: AudioRequest) => Promise<unknown>;
onUpload: (file: File) => Promise<void>;
onDraft: (params: Record<string, unknown>) => void;
}) {
const [models, setModels] = useState<Model[]>([]),
[error, setError] = useState(""),
[loading, setLoading] = useState(true),
[sending, setSending] = useState(false);
const [draft, setDraft] = useState<Record<string, any>>(() => ({
mode: "tts",
text: "",
voice: "Cherry",
format: "wav",
language: "Chinese",
instrumental: true,
gender: "female",
rate: 1,
pitch: 1,
volume: 50,
...initial,
}));
const patch = (value: Record<string, unknown>) => {
const next = { ...draft, ...value };
setDraft(next);
onDraft(next);
};
useEffect(() => {
const controller = new AbortController();
fetch("/api/canvas/audio/models", { signal: controller.signal })
.then(async (r) => {
const data = await r.json();
if (!r.ok) throw Error(data.error || "音频模型读取失败");
setModels(data.models || []);
})
.catch((e) => {
if (!controller.signal.aborted) setError(e.message);
})
.finally(() => {
if (!controller.signal.aborted) setLoading(false);
});
return () => controller.abort();
}, []);
const options = models.filter((m) => m.kind === draft.mode),
model = options.find((m) => m.id === draft.modelRef) || options[0],
contract = model ? audioContract(model.modelId) : undefined;
const voice = contract?.voices.some((v) => v.id === draft.voice)
? draft.voice
: contract?.voices[0]?.id;
const format = contract?.formats.includes(draft.format) ? draft.format : contract?.formats[0];
const input = "mt-1 w-full rounded-lg border border-white/15 bg-[#303030] p-2 text-sm";
const run = async () => {
setError("");
setSending(true);
try {
if (!model) throw Error("当前模式没有已启用的官方模型");
const params = audioRequestSchema.parse({
...draft,
text: [draft.text, connectedText].filter(Boolean).join("\n\n"),
modelRef: model.id,
voice,
format,
...(!contract?.controls ? { rate: 1, pitch: 1, volume: 50 } : {}),
...(!contract?.instructions ? { instructions: "" } : {}),
});
buildAudioRequest(params, model.modelId);
const savedDraft = { ...params, text: String(draft.text || "") };
patch(savedDraft);
await onSubmit(params, savedDraft);
} catch (e) {
setError(e instanceof Error ? e.message : "提交失败");
} finally {
setSending(false);
}
};
return (
<section
className="fixed bottom-8 left-1/2 z-[100] max-h-[75vh] w-[min(620px,94vw)] -translate-x-1/2 overflow-y-auto rounded-2xl border border-white/10 bg-[#222] p-5 text-white shadow-xl"
data-canvas-overlay="true"
aria-label="阿里云音频生成"
>
<h3 className="font-semibold">语音与音乐 · 阿里云官方</h3>
<fieldset disabled={busy || sending} className="mt-3 space-y-3 disabled:opacity-60">
{connectedText && <p className="mb-3 whitespace-pre-wrap rounded bg-white/5 p-2 text-xs" aria-label="已连接文本">已连接文本:{connectedText}</p>}
<div className="grid grid-cols-2 gap-3">
<label>
生成类型
<select
aria-label="音频生成类型"
className={input}
value={draft.mode}
onChange={(e) => patch({ mode: e.target.value, modelRef: "" })}
>
<option value="tts">文字配音</option>
<option value="music">音乐 / 歌曲</option>
</select>
</label>
<label>
模型
<select
aria-label="音频模型"
className={input}
value={model?.id || ""}
onChange={(e) => patch({ modelRef: e.target.value })}
>
{!options.length ? (
<option value="">{loading ? "读取模型中…" : "暂无已开通模型"}</option>
) : (
options.map((m) => (
<option key={m.id} value={m.id}>
{m.name}
</option>
))
)}
</select>
</label>
</div>
<label className="block">
{draft.mode === "tts" ? "配音文本(最多 600 字)" : "音乐描述"}
<textarea
aria-label="音频内容"
className={input}
rows={4}
maxLength={draft.mode === "tts" ? 600 : 2000}
value={draft.text}
onChange={(e) => patch({ text: e.target.value })}
/>
</label>
{draft.mode === "tts" ? (
<>
<div className="grid grid-cols-2 gap-3">
<label>
音色
<select
aria-label="配音音色"
className={input}
value={voice || ""}
onChange={(e) => patch({ voice: e.target.value })}
>
{contract?.voices.map((v) => (
<option key={v.id} value={v.id}>
{v.name}
</option>
))}
</select>
</label>
<label>
语言
<select
aria-label="配音语言"
className={input}
value={draft.language}
onChange={(e) => patch({ language: e.target.value })}
>
<option value="Chinese">中文</option>
<option value="English">英文</option>
<option value="Auto">自动 / 中英混合</option>
</select>
</label>
</div>
{contract?.instructions ? (
<label className="block">
声音风格指令
<input
className={input}
aria-label="声音风格指令"
placeholder="例如:温暖自然,语速略慢"
maxLength={1200}
value={draft.instructions || ""}
onChange={(e) => patch({ instructions: e.target.value })}
/>
</label>
) : null}
{contract?.controls ? (
<div className="grid grid-cols-3 gap-2">
{(["rate", "pitch", "volume"] as const).map((key, i) => (
<label key={key}>
{["语速", "音调", "音量"][i]}
<input
aria-label={["语速", "音调", "音量"][i]}
className={input}
type="number"
min={key === "volume" ? 0 : 0.5}
max={key === "volume" ? 100 : 2}
step={key === "volume" ? 1 : 0.1}
value={draft[key]}
onChange={(e) => patch({ [key]: Number(e.target.value) })}
/>
</label>
))}
</div>
) : null}
</>
) : (
<>
<label className="flex gap-2">
<input
type="checkbox"
checked={draft.instrumental}
onChange={(e) => patch({ instrumental: e.target.checked })}
/>
纯音乐(无人声)
</label>
{!draft.instrumental ? (
<>
<label className="block">
自定义歌词(选填,填写后优先于描述)
<textarea
aria-label="音乐歌词"
className={input}
rows={3}
maxLength={2000}
value={draft.lyrics || ""}
onChange={(e) => patch({ lyrics: e.target.value })}
/>
</label>
<label>
演唱声音
<select
className={input}
aria-label="演唱声音"
value={draft.gender}
onChange={(e) => patch({ gender: e.target.value })}
>
<option value="female">女声</option>
<option value="male">男声</option>
</select>
</label>
</>
) : null}
{!options.length ? (
<p className="text-sm text-amber-200">
Fun-Music 需要阿里云邀测权限,开通并启用模型后即可生成。
</p>
) : null}
</>
)}
<div className="flex items-end gap-3">
<label>
格式
<select
aria-label="音频格式"
className={input}
value={format || "wav"}
onChange={(e) => patch({ format: e.target.value })}
>
{(contract?.formats || ["wav"]).map((f) => (
<option key={f} value={f}>
{f.toUpperCase()}
</option>
))}
</select>
</label>
<button
type="button"
disabled={!model || loading}
className="rounded-lg bg-white px-4 py-2 text-black disabled:opacity-40"
onClick={() => void run()}
>
{busy || sending ? "处理中…" : "生成音频"}
</button>
<label className="cursor-pointer rounded-lg border border-white/20 px-3 py-2">
上传音频
<input
type="file"
accept="audio/*"
className="hidden"
onChange={async (e) => {
const file = e.target.files?.[0];
e.target.value = "";
if (!file) return;
setSending(true);
try {
await onUpload(file);
} catch (e) {
setError(e instanceof Error ? e.message : "上传失败");
} finally {
setSending(false);
}
}}
/>
</label>
</div>
</fieldset>
{error ? (
<p role="alert" className="mt-3 text-sm text-red-300">
{error}
</p>
) : null}
</section>
);
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,125 @@
"use client";
import {
Download,
Play,
Ungroup,
Share2,
Grid3X3,
ScanLine,
} from "lucide-react";
export function CanvasGroupToolbar({
title,
color,
status,
busy,
storyboard,
canStoryboard,
onRename,
onColor,
onLayout,
onRun,
onUngroup,
onSave,
onDownload,
onStoryboard,
onCancel,
}: {
title: string;
color: string;
status: string;
busy: boolean;
storyboard: boolean;
canStoryboard: boolean;
onRename: (s: string) => void;
onColor: (s: string) => void;
onLayout: (s: "grid" | "horizontal" | "vertical") => void;
onRun: () => void;
onUngroup: () => void;
onSave: () => void;
onDownload: () => void;
onStoryboard: () => void;
onCancel: () => void;
}) {
return (
<div
data-canvas-overlay="true"
data-group-toolbar="true"
className="absolute left-1/2 top-[80px] z-[70] -translate-x-1/2 rounded-[22px] border border-white/10 bg-[#252525] p-4 text-white shadow-xl"
onPointerDown={(e) => e.stopPropagation()}
>
<div className="flex items-center gap-5 whitespace-nowrap text-[16px]">
<input
aria-label="编组颜色"
type="color"
value={color}
onChange={(e) => onColor(e.target.value)}
className="h-7 w-7 cursor-pointer rounded-full border-0 bg-transparent"
/>
<label title="编组布局" className="flex items-center gap-1">
<Grid3X3 size={20} />
<select
aria-label="编组布局"
defaultValue=""
onChange={(e) => {
onLayout(e.target.value as any);
e.target.value = "";
}}
className="w-5 bg-[#252525]"
>
<option value="" disabled>
⌄
</option>
<option value="grid">网格布局</option>
<option value="horizontal">横向布局</option>
<option value="vertical">纵向布局</option>
</select>
</label>
<span className="h-8 w-px bg-white/10" />
<button
type="button"
disabled={busy}
onClick={onRun}
className="flex items-center gap-2 disabled:opacity-40"
>
<Play size={22} />
整组执行
</button>
<button onClick={onSave} className="flex items-center gap-2">
<Share2 size={22} />
添加到工具箱
</button>
<button
type="button"
disabled={!canStoryboard}
title={
canStoryboard ? "" : "分镜组需要全部成员都是已就绪的图片或视频"
}
onClick={onStoryboard}
className="flex items-center gap-2 disabled:opacity-35"
>
<ScanLine size={22} />
{storyboard ? "预览分镜" : "转分镜组"}
</button>
<button onClick={onUngroup} className="flex items-center gap-2">
<Ungroup size={22} />
解组
</button>
<span className="h-8 w-px bg-white/10" />
<button title="下载编组模板" onClick={onDownload}>
<Download size={22} />
</button>
</div>
<div className="mt-3 flex items-center justify-between gap-4 text-xs text-white/55">
<input
aria-label="编组名称"
value={title}
onChange={(e) => onRename(e.target.value)}
maxLength={120}
className="w-48 border-b border-white/10 bg-transparent py-1"
/>
<span>{status || "Shift 点击多选 · ⌘/Ctrl+G 编组"}</span>
{busy && <button onClick={onCancel}>取消未开始任务</button>}
</div>
</div>
);
}

View File

@ -0,0 +1,953 @@
"use client";
import {
CircleAlert,
ImagePlus,
Images,
LayoutTemplate,
Loader2,
Plus,
RefreshCw,
Trash2,
Upload,
} from "lucide-react";
import type React from "react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { createPortal } from "react-dom";
import { toast } from "sonner";
import { Button } from "@/components/ui/button";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Progress } from "@/components/ui/progress";
import { Textarea } from "@/components/ui/textarea";
import { buildImageProxyUrl } from "@/lib/image-delivery";
import { cn } from "@/lib/utils";
import {
getAvailableCanvasLayoutReplaceSources,
getLayoutReplaceSourceIdentity,
getReadyCanvasLayoutReplaceSource,
type LayoutReplaceSource,
mergeUniqueLayoutReplaceSources,
} from "@/lib/canvas/layout-replace-source-selection";
import {
imageGenerationService,
type CanvasLayoutReplaceBatchItemSnapshot,
type CanvasLayoutReplaceBatchSnapshot,
} from "@/services/canvas";
import { useCanvasStore } from "@/store/canvas";
import type { CanvasNode, Size } from "@/types/canvas";
const MAX_BATCH_SOURCES = 5;
const ACTIVE_BATCH_STATUSES = new Set(["pending", "running"]);
type CanvasLayoutReplaceBatchDialogProps = {
costLabel?: string | null;
onOpenChange: (open: boolean) => void;
open: boolean;
primaryNode: CanvasNode;
};
function isHttpImageUrl(value: unknown) {
return /^https?:\/\//i.test(String(value || "").trim());
}
function buildInitialSources(
primaryNode: CanvasNode,
nodes: CanvasNode[],
selectedNodeIds: string[]
) {
const selectedById = new Map(nodes.map((node) => [node.id, node]));
const orderedIds = [primaryNode.id, ...selectedNodeIds.filter((id) => id !== primaryNode.id)];
const sources: LayoutReplaceSource[] = [];
const seen = new Set<string>();
for (const id of orderedIds) {
const node = id === primaryNode.id ? primaryNode : selectedById.get(id);
if (!node) continue;
const source = getReadyCanvasLayoutReplaceSource(node);
if (!source) continue;
const identity = getLayoutReplaceSourceIdentity(source);
if (seen.has(identity)) continue;
seen.add(identity);
sources.push(source);
if (sources.length >= MAX_BATCH_SOURCES) break;
}
return sources;
}
function sourcesFromBatch(batch: CanvasLayoutReplaceBatchSnapshot) {
return batch.items.map((item) => ({
id: item.sourceNodeId ? `canvas:${item.sourceNodeId}` : `upload:${item.id}`,
imageUrl: item.sourceImageUrl,
kind: item.sourceKind === "upload" ? ("upload" as const) : ("canvas" as const),
nodeId: item.sourceNodeId || undefined,
title: item.sourceTitle || `图片 ${item.position}`,
}));
}
function getBatchStatusLabel(status: string) {
if (status === "succeeded") return "全部完成";
if (status === "partial_failed") return "部分失败";
if (status === "failed") return "处理失败";
if (status === "running") return "处理中";
return "排队中";
}
function getItemStatusLabel(item: CanvasLayoutReplaceBatchItemSnapshot) {
if (item.status === "succeeded") return item.addedToCanvas ? "已加入画布" : "生成完成";
if (item.status === "failed") return "生成失败";
if (item.status === "running") return "正在生成";
return "排队中";
}
async function resolveImageSize(url: string, fallback: Size) {
if (typeof window === "undefined") return fallback;
return new Promise<Size>((resolve) => {
const image = new window.Image();
const timeout = window.setTimeout(() => resolve(fallback), 6000);
image.onload = () => {
window.clearTimeout(timeout);
const naturalWidth = image.naturalWidth || fallback.width;
const naturalHeight = image.naturalHeight || fallback.height;
const width = Math.max(180, Math.min(720, fallback.width || 360));
resolve({
width,
height: Math.max(120, Math.min(1000, Math.round((width * naturalHeight) / naturalWidth))),
});
};
image.onerror = () => {
window.clearTimeout(timeout);
resolve(fallback);
};
image.src = buildImageProxyUrl(url, { delivery: "proxy" });
});
}
export function CanvasLayoutReplaceBatchDialog({
costLabel,
onOpenChange,
open,
primaryNode,
}: CanvasLayoutReplaceBatchDialogProps) {
const allNodes = useCanvasStore((state) => state.nodes);
const selectedNodeIds = useCanvasStore((state) => state.selectedNodeIds);
const projectId = useCanvasStore((state) => state.projectId);
const addNode = useCanvasStore((state) => state.addNode);
const selectNodes = useCanvasStore((state) => state.selectNodes);
const canvasImageSelectionSession = useCanvasStore(
(state) => state.canvasImageSelectionSession
);
const startCanvasImageSelection = useCanvasStore(
(state) => state.startCanvasImageSelection
);
const endCanvasImageSelection = useCanvasStore(
(state) => state.endCanvasImageSelection
);
const sourceUploadInputRef = useRef<HTMLInputElement>(null);
const referenceUploadInputRef = useRef<HTMLInputElement>(null);
const seenSucceededItemIdsRef = useRef(new Set<string>());
const [sources, setSources] = useState<LayoutReplaceSource[]>(() =>
buildInitialSources(primaryNode, allNodes, selectedNodeIds)
);
const [referenceImageUrl, setReferenceImageUrl] = useState("");
const [supplementalPrompt, setSupplementalPrompt] = useState("");
const [batch, setBatch] = useState<CanvasLayoutReplaceBatchSnapshot | null>(null);
const [selectedResultIds, setSelectedResultIds] = useState<Set<string>>(new Set());
const [isLoadingBatch, setIsLoadingBatch] = useState(false);
const [isSubmitting, setIsSubmitting] = useState(false);
const [isUploadingSources, setIsUploadingSources] = useState(false);
const [isUploadingReference, setIsUploadingReference] = useState(false);
const [isAddingResults, setIsAddingResults] = useState(false);
const [retryingItemId, setRetryingItemId] = useState<string | null>(null);
const readyCanvasSources = useMemo(
() =>
allNodes
.map(getReadyCanvasLayoutReplaceSource)
.filter(Boolean) as LayoutReplaceSource[],
[allNodes]
);
const activeCanvasImageSelectionSession =
canvasImageSelectionSession?.ownerNodeId === primaryNode.id
? canvasImageSelectionSession
: null;
const successfulItems = useMemo(
() => batch?.items.filter((item) => item.status === "succeeded" && item.resultImageUrl) || [],
[batch]
);
const batchIsActive = Boolean(batch && ACTIVE_BATCH_STATUSES.has(batch.status));
const terminalItemCount = batch
? batch.items.filter((item) => item.status === "succeeded" || item.status === "failed").length
: 0;
const batchProgress = batch
? Math.round((terminalItemCount / Math.max(1, batch.totalCount)) * 100)
: 0;
const applyBatchSnapshot = useCallback((nextBatch: CanvasLayoutReplaceBatchSnapshot) => {
setBatch(nextBatch);
setSources(sourcesFromBatch(nextBatch));
setReferenceImageUrl(nextBatch.referenceImageUrl);
setSupplementalPrompt(nextBatch.supplementalPrompt || "");
const newlySucceeded = nextBatch.items
.filter(
(item) =>
item.status === "succeeded" &&
!item.addedToCanvas &&
!seenSucceededItemIdsRef.current.has(item.id)
)
.map((item) => item.id);
if (newlySucceeded.length > 0) {
setSelectedResultIds((current) => new Set([...current, ...newlySucceeded]));
}
nextBatch.items
.filter((item) => item.status === "succeeded")
.forEach((item) => seenSucceededItemIdsRef.current.add(item.id));
}, []);
const loadBatch = useCallback(
async (batchId: string) => {
const result = await imageGenerationService.getLayoutReplaceBatch(batchId);
if (result.success && result.data?.batch) {
applyBatchSnapshot(result.data.batch);
return result.data.batch;
}
return null;
},
[applyBatchSnapshot]
);
useEffect(() => {
if (!open) return;
const currentCanvas = useCanvasStore.getState();
const currentPrimaryNode =
currentCanvas.nodes.find((node) => node.id === primaryNode.id) || primaryNode;
const initialSources = buildInitialSources(
currentPrimaryNode,
currentCanvas.nodes,
currentCanvas.selectedNodeIds
);
setSources(initialSources);
setSelectedResultIds(new Set());
seenSucceededItemIdsRef.current = new Set();
if (!projectId) {
setBatch(null);
return;
}
let cancelled = false;
setIsLoadingBatch(true);
void imageGenerationService
.getLatestLayoutReplaceBatch({ projectId, sourceNodeId: primaryNode.id })
.then((result) => {
if (cancelled) return;
if (result.success && result.data?.batch) {
applyBatchSnapshot(result.data.batch);
} else {
setBatch(null);
}
})
.finally(() => {
if (!cancelled) setIsLoadingBatch(false);
});
return () => {
cancelled = true;
};
}, [applyBatchSnapshot, open, primaryNode.id, projectId]);
useEffect(
() => () => {
endCanvasImageSelection(primaryNode.id);
},
[endCanvasImageSelection, primaryNode.id]
);
useEffect(() => {
if (!activeCanvasImageSelectionSession) return;
const handleKeyDown = (event: KeyboardEvent) => {
if (event.key !== "Escape") return;
event.preventDefault();
endCanvasImageSelection(primaryNode.id);
};
window.addEventListener("keydown", handleKeyDown, true);
return () => window.removeEventListener("keydown", handleKeyDown, true);
}, [activeCanvasImageSelectionSession, endCanvasImageSelection, primaryNode.id]);
useEffect(() => {
if (!open || !batch?.id || !batchIsActive) return;
const interval = window.setInterval(() => {
void loadBatch(batch.id);
}, 2500);
return () => window.clearInterval(interval);
}, [batch?.id, batchIsActive, loadBatch, open]);
const startNewBatchDraft = useCallback(() => {
setBatch(null);
setSelectedResultIds(new Set());
seenSucceededItemIdsRef.current = new Set();
setSources(buildInitialSources(primaryNode, allNodes, selectedNodeIds));
}, [allNodes, primaryNode, selectedNodeIds]);
const removeSource = useCallback((sourceId: string) => {
setSources((current) => current.filter((source) => source.id !== sourceId));
}, []);
const startCanvasSourceSelection = useCallback(() => {
if (canvasImageSelectionSession) return;
const remainingCount = MAX_BATCH_SOURCES - sources.length;
if (remainingCount <= 0) {
toast.info(`最多添加 ${MAX_BATCH_SOURCES} 张图片`);
return;
}
const availableSources = getAvailableCanvasLayoutReplaceSources(
readyCanvasSources,
sources
);
if (availableSources.length === 0) {
toast.info("画布中暂无其他可添加的图片");
return;
}
startCanvasImageSelection({
ownerNodeId: primaryNode.id,
allowedNodeIds: availableSources.flatMap((source) =>
source.nodeId ? [source.nodeId] : []
),
maxCount: remainingCount,
});
}, [
canvasImageSelectionSession,
primaryNode.id,
readyCanvasSources,
sources,
startCanvasImageSelection,
]);
const cancelCanvasSourceSelection = useCallback(() => {
endCanvasImageSelection(primaryNode.id);
}, [endCanvasImageSelection, primaryNode.id]);
const confirmCanvasSourceSelection = useCallback(() => {
if (!activeCanvasImageSelectionSession) return;
const selectedIds = new Set(activeCanvasImageSelectionSession.selectedNodeIds);
const selectedSources = readyCanvasSources.filter(
(source) => source.nodeId && selectedIds.has(source.nodeId)
);
if (selectedSources.length === 0) return;
setSources((current) => {
return mergeUniqueLayoutReplaceSources(
current,
selectedSources,
MAX_BATCH_SOURCES
);
});
endCanvasImageSelection(primaryNode.id);
toast.success(`已从画布添加 ${selectedSources.length} 张图片`);
}, [
activeCanvasImageSelectionSession,
endCanvasImageSelection,
primaryNode.id,
readyCanvasSources,
]);
const handleSourceUpload = useCallback(
async (event: React.ChangeEvent<HTMLInputElement>) => {
const files = Array.from(event.currentTarget.files || []).filter((file) =>
file.type.startsWith("image/")
);
event.currentTarget.value = "";
const remaining = Math.max(0, MAX_BATCH_SOURCES - sources.length);
if (remaining === 0 || files.length === 0) return;
const selectedFiles = files.slice(0, remaining);
if (selectedFiles.length < files.length) {
toast.info(`最多添加 ${MAX_BATCH_SOURCES} 张图片`);
}
setIsUploadingSources(true);
try {
const uploaded: LayoutReplaceSource[] = [];
for (const file of selectedFiles) {
const result = await imageGenerationService.uploadImage(file);
if (!result.success || !result.data?.url) {
throw new Error(result.error || `${file.name} 上传失败`);
}
uploaded.push({
id: `upload:${crypto.randomUUID()}`,
imageUrl: result.data.url,
kind: "upload",
title: file.name.replace(/\.[^.]+$/, "") || "上传图片",
});
}
setSources((current) => {
return mergeUniqueLayoutReplaceSources(
current,
uploaded,
MAX_BATCH_SOURCES
);
});
toast.success(`已上传 ${uploaded.length} 张图片`);
} catch (error) {
toast.error(error instanceof Error ? error.message : "图片上传失败");
} finally {
setIsUploadingSources(false);
}
},
[sources.length]
);
const handleReferenceUpload = useCallback(async (event: React.ChangeEvent<HTMLInputElement>) => {
const file = event.currentTarget.files?.[0];
event.currentTarget.value = "";
if (!file || !file.type.startsWith("image/")) return;
setIsUploadingReference(true);
try {
const result = await imageGenerationService.uploadImage(file);
if (!result.success || !result.data?.url) {
throw new Error(result.error || "目标模版图上传失败");
}
setReferenceImageUrl(result.data.url);
toast.success("目标模版图已上传");
} catch (error) {
toast.error(error instanceof Error ? error.message : "目标模版图上传失败");
} finally {
setIsUploadingReference(false);
}
}, []);
const handleSubmit = useCallback(async () => {
if (!projectId) {
toast.error("当前画布尚未保存,无法创建批次");
return;
}
if (sources.length === 0) {
toast.error("请至少添加一张待处理图片");
return;
}
if (!isHttpImageUrl(referenceImageUrl)) {
toast.error("请上传目标模版图");
return;
}
if (isSubmitting || isUploadingSources || isUploadingReference) return;
setIsSubmitting(true);
try {
const result = await imageGenerationService.createLayoutReplaceBatch({
projectId,
sources: sources.map((source) => ({
sourceImageUrl: source.imageUrl,
sourceKind: source.kind,
sourceNodeId: source.nodeId,
sourceTitle: source.title,
})),
referenceImageUrl,
supplementalPrompt,
imageSize: "2K",
});
if (!result.success || !result.data?.batchId) {
throw new Error(result.error || "创建批次失败");
}
const nextBatch = await loadBatch(result.data.batchId);
if (!nextBatch) throw new Error("批次已创建,但状态读取失败");
toast.success(`已提交 ${sources.length} 张图片,正在顺序处理`);
} catch (error) {
toast.error(error instanceof Error ? error.message : "创建批次失败");
} finally {
setIsSubmitting(false);
}
}, [
isSubmitting,
isUploadingReference,
isUploadingSources,
loadBatch,
projectId,
referenceImageUrl,
sources,
supplementalPrompt,
]);
const addItemsToCanvas = useCallback(
async (items: CanvasLayoutReplaceBatchItemSnapshot[]) => {
if (!batch || items.length === 0 || isAddingResults) return;
setIsAddingResults(true);
try {
const created: Array<{ itemId: string; canvasNodeId: string }> = [];
for (const [index, item] of items.entries()) {
if (!item.resultImageUrl || item.addedToCanvas) continue;
const sourceNode = allNodes.find((candidate) => candidate.id === item.sourceNodeId);
const anchor = sourceNode || primaryNode;
const resultSize = await resolveImageSize(item.resultImageUrl, anchor.size);
const canvasNodeId = addNode({
type: "image",
position: {
x: anchor.position.x + anchor.size.width + 20,
y: anchor.position.y + index * 24,
},
size: resultSize,
data: {
src: item.resultImageUrl,
thumbnailUrl: item.resultThumbnailUrl || item.resultImageUrl,
status: "completed",
prompt: "批量排版替换结果",
title: `${item.sourceTitle || `图片 ${item.position}`}-排版替换`,
originalSrc: item.sourceImageUrl,
model: item.resultModelId || undefined,
metadata: {
editMode: "layout-replace",
batchId: batch.id,
batchItemId: item.id,
taskId: item.taskId,
sourceNodeId: item.sourceNodeId,
layoutReferenceUrl: batch.referenceImageUrl,
layoutReplaceCompletedAt: new Date().toISOString(),
},
},
connections: item.sourceNodeId ? [item.sourceNodeId] : [],
});
created.push({ itemId: item.id, canvasNodeId });
}
if (created.length === 0) return;
selectNodes(created.map((item) => item.canvasNodeId));
const marked = await imageGenerationService.markLayoutReplaceBatchItemsAdded({
batchId: batch.id,
items: created,
});
if (!marked.success) {
toast.warning("结果已加入画布,但同步批次状态失败");
}
await loadBatch(batch.id);
setSelectedResultIds((current) => {
const next = new Set(current);
created.forEach((item) => next.delete(item.itemId));
return next;
});
toast.success(`已将 ${created.length} 张结果加入画布`);
} catch (error) {
toast.error(error instanceof Error ? error.message : "添加到画布失败");
} finally {
setIsAddingResults(false);
}
},
[addNode, allNodes, batch, isAddingResults, loadBatch, primaryNode, selectNodes]
);
const selectedSuccessfulItems = successfulItems.filter(
(item) => selectedResultIds.has(item.id) && !item.addedToCanvas
);
const retryItem = useCallback(
async (item: CanvasLayoutReplaceBatchItemSnapshot) => {
if (!batch || batchIsActive || retryingItemId) return;
setRetryingItemId(item.id);
try {
const result = await imageGenerationService.retryLayoutReplaceBatchItem({
batchId: batch.id,
itemId: item.id,
});
if (!result.success) throw new Error(result.error || "重试失败");
await loadBatch(batch.id);
toast.success("已重新加入生成队列");
} catch (error) {
toast.error(error instanceof Error ? error.message : "重试失败");
} finally {
setRetryingItemId(null);
}
},
[batch, batchIsActive, loadBatch, retryingItemId]
);
return (
<>
<Dialog
onOpenChange={(nextOpen) => {
if (!activeCanvasImageSelectionSession) onOpenChange(nextOpen);
}}
open={open && !activeCanvasImageSelectionSession}
>
<DialogContent className="flex max-h-[calc(100dvh-1rem)] w-[calc(100vw-1rem)] max-w-[1180px] flex-col gap-0 overflow-hidden p-0 sm:max-h-[calc(100dvh-2rem)] sm:w-[calc(100vw-2rem)]">
<DialogHeader className="shrink-0 border-b px-4 py-3 pr-12 text-left sm:px-6 sm:py-4">
<div className="flex flex-wrap items-center justify-between gap-3">
<DialogTitle className="flex items-center gap-2 text-base">
<LayoutTemplate className="h-4 w-4" />
AI 模版替换
</DialogTitle>
{batch ? (
<div className="flex items-center gap-2 text-xs text-muted-foreground">
<span>{getBatchStatusLabel(batch.status)}</span>
<span>{terminalItemCount}/{batch.totalCount}</span>
</div>
) : null}
</div>
<DialogDescription className="sr-only">
最多选择五张图片,按照目标模版顺序生成并添加到画布
</DialogDescription>
</DialogHeader>
<div className="min-h-0 flex-1 overflow-y-auto overscroll-contain px-4 py-4 sm:px-6">
{isLoadingBatch ? (
<div className="flex min-h-[360px] items-center justify-center">
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
</div>
) : (
<div className="space-y-5">
<div className="grid gap-5 xl:grid-cols-[minmax(0,1fr)_360px]">
<section className="min-w-0 space-y-3">
<div className="flex flex-wrap items-center justify-between gap-2">
<Label>待处理图片 {sources.length}/{MAX_BATCH_SOURCES}</Label>
{!batch ? (
<div className="flex items-center gap-2">
<Button
disabled={
sources.length >= MAX_BATCH_SOURCES ||
Boolean(canvasImageSelectionSession)
}
onClick={startCanvasSourceSelection}
size="sm"
type="button"
variant="outline"
>
<Images className="mr-1.5 h-4 w-4" />
从画布中添加
</Button>
<input
accept="image/*"
className="hidden"
multiple
onChange={(event) => void handleSourceUpload(event)}
ref={sourceUploadInputRef}
type="file"
/>
<Button
disabled={isUploadingSources || sources.length >= MAX_BATCH_SOURCES}
onClick={() => sourceUploadInputRef.current?.click()}
size="sm"
type="button"
variant="outline"
>
{isUploadingSources ? (
<Loader2 className="mr-1.5 h-4 w-4 animate-spin" />
) : (
<Upload className="mr-1.5 h-4 w-4" />
)}
上传图片
</Button>
</div>
) : null}
</div>
<div className="grid grid-cols-2 gap-2 sm:grid-cols-3 lg:grid-cols-5">
{Array.from({ length: MAX_BATCH_SOURCES }, (_, index) => {
const source = sources[index];
return (
<div
className="relative aspect-square min-w-0 overflow-hidden rounded-md border bg-muted/25"
key={source?.id || `empty-${index}`}
>
<span className="absolute left-1.5 top-1.5 z-10 flex h-5 min-w-5 items-center justify-center rounded bg-black/65 px-1 text-[10px] text-white">
{index + 1}
</span>
{source ? (
<>
<img
alt={source.title}
className="h-full w-full object-cover"
src={buildImageProxyUrl(source.imageUrl, { width: 240 })}
/>
{!batch && index > 0 ? (
<button
aria-label={`移除第 ${index + 1} 张图片`}
className="absolute right-1.5 top-1.5 flex h-7 w-7 items-center justify-center rounded-md bg-black/65 text-white hover:bg-black/80"
onClick={() => removeSource(source.id)}
type="button"
>
<Trash2 className="h-3.5 w-3.5" />
</button>
) : null}
<div className="absolute inset-x-0 bottom-0 truncate bg-black/65 px-2 py-1.5 text-[11px] text-white">
{source.title}
</div>
</>
) : (
<button
aria-label={`添加第 ${index + 1} 张图片`}
className="flex h-full w-full items-center justify-center text-muted-foreground hover:bg-muted/50"
disabled={Boolean(batch)}
onClick={() => sourceUploadInputRef.current?.click()}
type="button"
>
<Plus className="h-5 w-5" />
</button>
)}
</div>
);
})}
</div>
<div className="space-y-2">
<Label htmlFor={`layout-batch-prompt-${primaryNode.id}`}>补充要求</Label>
<Textarea
className="min-h-[88px] max-h-[180px]"
disabled={Boolean(batch)}
id={`layout-batch-prompt-${primaryNode.id}`}
onChange={(event) => setSupplementalPrompt(event.currentTarget.value)}
placeholder="可选,只写产品替换细节"
value={supplementalPrompt}
/>
</div>
</section>
<section className="min-w-0 space-y-2">
<Label htmlFor={`layout-batch-reference-${primaryNode.id}`}>目标模版图</Label>
<Input
disabled={Boolean(batch)}
id={`layout-batch-reference-${primaryNode.id}`}
onChange={(event) => setReferenceImageUrl(event.currentTarget.value)}
placeholder="https://..."
value={referenceImageUrl}
/>
<input
accept="image/*"
className="hidden"
onChange={(event) => void handleReferenceUpload(event)}
ref={referenceUploadInputRef}
type="file"
/>
<button
className="flex h-[clamp(180px,34dvh,360px)] w-full items-center justify-center overflow-hidden rounded-md border border-dashed bg-muted/20"
disabled={Boolean(batch) || isUploadingReference}
onClick={() => referenceUploadInputRef.current?.click()}
type="button"
>
{isUploadingReference ? (
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
) : isHttpImageUrl(referenceImageUrl) ? (
<img
alt="目标模版图"
className="h-full w-full object-contain"
src={buildImageProxyUrl(referenceImageUrl, { width: 720 })}
/>
) : (
<div className="flex flex-col items-center gap-2 text-muted-foreground">
<ImagePlus className="h-6 w-6" />
<span className="text-xs">上传目标模版图</span>
</div>
)}
</button>
</section>
</div>
{batch ? (
<section className="space-y-3 border-t pt-5">
<div className="flex flex-wrap items-center justify-between gap-3">
<div className="space-y-1">
<Label>生成结果</Label>
<div className="text-xs text-muted-foreground">
{getBatchStatusLabel(batch.status)} · {terminalItemCount}/{batch.totalCount}
</div>
</div>
{!batchIsActive ? (
<Button onClick={startNewBatchDraft} size="sm" type="button" variant="outline">
<Plus className="mr-1.5 h-4 w-4" />
新建批次
</Button>
) : null}
</div>
<Progress className="h-1.5" value={batchProgress} />
<div className="grid gap-3 md:grid-cols-2 xl:grid-cols-3">
{batch.items.map((item) => {
const canAdd = item.status === "succeeded" && Boolean(item.resultImageUrl) && !item.addedToCanvas;
const checked = selectedResultIds.has(item.id);
return (
<div className="overflow-hidden rounded-md border" key={item.id}>
<div className="grid grid-cols-2 gap-px bg-border">
<div className="aspect-square bg-muted">
<img
alt={item.sourceTitle || `图片 ${item.position}`}
className="h-full w-full object-cover"
src={buildImageProxyUrl(item.sourceImageUrl, { width: 280 })}
/>
</div>
<div className="relative flex aspect-square items-center justify-center bg-muted/40">
{item.resultImageUrl ? (
<img
alt={`结果 ${item.position}`}
className="h-full w-full object-cover"
src={buildImageProxyUrl(item.resultThumbnailUrl || item.resultImageUrl, { width: 280 })}
/>
) : item.status === "failed" ? (
<CircleAlert className="h-6 w-6 text-destructive" />
) : item.status === "running" ? (
<Loader2 className="h-6 w-6 animate-spin text-primary" />
) : (
<Loader2 className="h-5 w-5 text-muted-foreground" />
)}
{canAdd ? (
<label className="absolute right-2 top-2 flex h-6 w-6 items-center justify-center rounded bg-background/90 shadow-sm">
<input
checked={checked}
className="h-4 w-4"
onChange={(event) => {
setSelectedResultIds((current) => {
const next = new Set(current);
if (event.currentTarget.checked) next.add(item.id);
else next.delete(item.id);
return next;
});
}}
type="checkbox"
/>
</label>
) : null}
</div>
</div>
<div className="flex min-h-12 items-center justify-between gap-2 px-3 py-2">
<div className="min-w-0">
<div className="truncate text-xs font-medium">
{item.sourceTitle || `图片 ${item.position}`}
</div>
<div
className={cn(
"text-[11px] text-muted-foreground",
item.status === "failed" && "text-destructive"
)}
>
{getItemStatusLabel(item)}
</div>
</div>
{canAdd ? (
<Button
disabled={isAddingResults}
onClick={() => void addItemsToCanvas([item])}
size="sm"
type="button"
variant="outline"
>
加入画布
</Button>
) : item.status === "failed" && !batchIsActive ? (
<Button
disabled={Boolean(retryingItemId)}
onClick={() => void retryItem(item)}
size="sm"
type="button"
variant="outline"
>
{retryingItemId === item.id ? (
<Loader2 className="mr-1.5 h-3.5 w-3.5 animate-spin" />
) : (
<RefreshCw className="mr-1.5 h-3.5 w-3.5" />
)}
重试
</Button>
) : null}
</div>
{item.error ? (
<div className="border-t px-3 py-2 text-xs text-destructive">
{item.error}
</div>
) : null}
</div>
);
})}
</div>
</section>
) : null}
</div>
)}
</div>
<DialogFooter className="shrink-0 gap-2 border-t bg-background px-4 py-3 sm:px-6">
<Button onClick={() => onOpenChange(false)} type="button" variant="outline">
关闭
</Button>
{batch ? (
<Button
disabled={selectedSuccessfulItems.length === 0 || isAddingResults}
onClick={() => void addItemsToCanvas(selectedSuccessfulItems)}
type="button"
>
{isAddingResults ? (
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
) : (
<Images className="mr-2 h-4 w-4" />
)}
添加所选到画布 ({selectedSuccessfulItems.length})
</Button>
) : (
<Button
disabled={
sources.length === 0 ||
!isHttpImageUrl(referenceImageUrl) ||
isSubmitting ||
isUploadingReference ||
isUploadingSources
}
onClick={() => void handleSubmit()}
type="button"
>
{isSubmitting ? (
<Loader2 className="mr-2 h-4 w-4 animate-spin" />
) : (
<LayoutTemplate className="mr-2 h-4 w-4" />
)}
批量处理 {sources.length} 张
{costLabel ? (
<span className="ml-1 rounded-full bg-white/18 px-1.5 py-0.5 text-[10px] text-white">
{sources.length} × {costLabel}
</span>
) : null}
</Button>
)}
</DialogFooter>
</DialogContent>
</Dialog>
{activeCanvasImageSelectionSession && typeof document !== "undefined"
? createPortal(
<div
className="pointer-events-none fixed inset-x-0 bottom-4 z-[10020] flex justify-center px-3 sm:bottom-6"
data-canvas-layout-replace-picker="true"
>
<div
className="pointer-events-auto flex w-full max-w-[720px] flex-wrap items-center gap-2 rounded-lg border border-border bg-background px-3 py-2.5 shadow-2xl sm:flex-nowrap sm:px-4"
data-canvas-overlay="true"
>
<div className="flex min-w-0 flex-1 items-center gap-2">
<span className="flex h-8 w-8 shrink-0 items-center justify-center rounded-md bg-[#168cff] text-white">
<Images className="h-4 w-4" />
</span>
<div className="min-w-0">
<div className="truncate text-sm font-medium">选择画布图片</div>
<div className="truncate text-xs text-muted-foreground">
已选 {activeCanvasImageSelectionSession.selectedNodeIds.length}/
{activeCanvasImageSelectionSession.maxCount}
</div>
</div>
</div>
<Button
onClick={cancelCanvasSourceSelection}
size="sm"
type="button"
variant="outline"
>
取消
</Button>
<Button
disabled={activeCanvasImageSelectionSession.selectedNodeIds.length === 0}
onClick={confirmCanvasSourceSelection}
size="sm"
type="button"
>
确认添加到图片卡槽
</Button>
</div>
</div>,
document.body
)
: null}
</>
);
}

View File

@ -25,6 +25,7 @@ import {
Heart,
HelpCircle,
ImagePlus,
LayoutTemplate,
LayoutGrid,
Lightbulb,
Minus,
@ -65,6 +66,10 @@ import {
type CanvasAgentPendingRequestWindow,
} from "@/lib/canvas/agent-pending-requests";
import { inferOriginalImageUrl } from "@/lib/image-url";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
type ImageNodeModule = typeof import("./nodes/ImageNode");
type Model3DNodeModule = typeof import("./nodes/Model3DNode");
@ -416,6 +421,9 @@ export function CanvasWorkspace({
const nodes = useCanvasStore(canvasSelectors.nodes);
const edges = useCanvasStore(canvasSelectors.edges);
const selectedNodeIds = useCanvasStore(canvasSelectors.selectedNodeIds);
const canvasImageSelectionSession = useCanvasStore(
(state) => state.canvasImageSelectionSession
);
const viewport = useCanvasStore(canvasSelectors.viewport);
const tool = useCanvasStore(canvasSelectors.tool);
const lastAddedNodeId = useCanvasStore((state) => state.lastAddedNodeId);
@ -715,12 +723,21 @@ export function CanvasWorkspace({
canvasToolbarConfig,
"imageLocalEdit"
);
const selectedImageLayoutReplaceItem = getCanvasToolbarItem(
canvasToolbarConfig,
"imageLayoutReplace"
);
// 局部编辑属于主无限画布图片的固定能力,旧配置缺少该 key 时也保持可见。
const selectedImageLocalEditEnabled = true;
// 排版替换同样是主无限画布图片的核心 AI 入口,旧配置缺少该 key 时也保持可见。
const selectedImageLayoutReplaceEnabled = true;
// Store actions
const selectNode = useCanvasStore((state) => state.selectNode);
const selectNodes = useCanvasStore((state) => state.selectNodes);
const toggleCanvasImageSelection = useCanvasStore(
(state) => state.toggleCanvasImageSelection
);
const clearSelection = useCanvasStore((state) => state.clearSelection);
const addNode = useCanvasStore((state) => state.addNode);
const addEdge = useCanvasStore((state) => state.addEdge);
@ -1238,6 +1255,7 @@ export function CanvasWorkspace({
// 处理画布点击(取消选择)
const handleCanvasClick = useCallback((e: React.MouseEvent) => {
if (canvasImageSelectionSession) return;
if (hasBlockingDialogOpen()) return;
setAgentSlotContextMenu(null);
if (!isEventFromCanvasContainer(e.target)) return;
@ -1256,10 +1274,11 @@ export function CanvasWorkspace({
fitAllImageNodesIntoViewport();
}
}
}, [clearLatestGeneratedImageHighlights, clearSelection, fitAllImageNodesIntoViewport, hasBlockingDialogOpen, isEventFromCanvasContainer, isMarqueeSelecting, isPointOnNodeElement]);
}, [canvasImageSelectionSession, clearLatestGeneratedImageHighlights, clearSelection, fitAllImageNodesIntoViewport, hasBlockingDialogOpen, isEventFromCanvasContainer, isMarqueeSelecting, isPointOnNodeElement]);
// 处理画布平移开始
const handleMouseDown = useCallback((e: React.MouseEvent) => {
if (canvasImageSelectionSession) return;
if (hasBlockingDialogOpen()) return;
if (!isEventFromCanvasContainer(e.target)) return;
if (connectionDraft) {
@ -1305,7 +1324,7 @@ export function CanvasWorkspace({
setMarqueeStart(start);
setMarqueeEnd(start);
}
}, [cancelPanPreview, clearBrowserTextSelection, clearLatestGeneratedImageHighlights, connectionDraft, hasBlockingDialogOpen, isEventFromCanvasContainer, isPointOnNodeElement, isWorkflowCanvas, toCanvasPosition, tool]);
}, [cancelPanPreview, canvasImageSelectionSession, clearBrowserTextSelection, clearLatestGeneratedImageHighlights, connectionDraft, hasBlockingDialogOpen, isEventFromCanvasContainer, isPointOnNodeElement, isWorkflowCanvas, toCanvasPosition, tool]);
// 处理画布平移
const handleMouseMove = useCallback((e: React.MouseEvent) => {
@ -1554,8 +1573,8 @@ export function CanvasWorkspace({
for (let index = 0; index < files.length; index += 1) {
const file = files[index];
if (!isImageFile(file)) continue;
if (file.size > 50 * 1024 * 1024) {
toast.error("图片大小不能超过 50MB");
if (file.size > IMAGE_UPLOAD_MAX_BYTES) {
toast.error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
continue;
}
@ -1824,7 +1843,14 @@ export function CanvasWorkspace({
// 渲染节点
const renderNode = (node: CanvasNode) => {
const isSelected = selectedNodeIds.includes(node.id);
const isCanvasImageSelectionCandidate = Boolean(
canvasImageSelectionSession?.allowedNodeIds.includes(node.id)
);
const isCanvasImageSelectionSelected = Boolean(
canvasImageSelectionSession?.selectedNodeIds.includes(node.id)
);
const isSelected =
!canvasImageSelectionSession && selectedNodeIds.includes(node.id);
const showWorkflowHandles = isWorkflowCanvas && (tool === "draw" || isSelected);
const isImageNode = node.type === "image";
const enableImageResizeHandles =
@ -2021,13 +2047,17 @@ export function CanvasWorkspace({
pushHistory();
}}
bounds="parent"
enableResizing={enableImageResizeHandles}
enableResizing={canvasImageSelectionSession ? false : enableImageResizeHandles}
resizeHandleStyles={imageResizeHandleStyles}
disableDragging={tool !== 'select'}
disableDragging={tool !== 'select' || Boolean(canvasImageSelectionSession)}
cancel={CANVAS_DRAG_CANCEL_SELECTOR}
className={cn(
"absolute",
tool === "select" && "cursor-pointer"
canvasImageSelectionSession
? isCanvasImageSelectionCandidate
? "cursor-crosshair"
: "cursor-default"
: tool === "select" && "cursor-pointer"
)}
data-canvas-node-id={node.id}
style={{ overflow: 'visible' }} // 允许工具栏显示在节点外部
@ -2038,6 +2068,12 @@ export function CanvasWorkspace({
}}
onClick={(e: React.MouseEvent) => {
e.stopPropagation();
if (canvasImageSelectionSession) {
if (isCanvasImageSelectionCandidate) {
toggleCanvasImageSelection(node.id);
}
return;
}
setAgentSlotContextMenu(null);
clearLatestGeneratedImageHighlights();
setSelectedEdgeId(null);
@ -2172,6 +2208,29 @@ export function CanvasWorkspace({
</div>
)}
</motion.div>
{isCanvasImageSelectionCandidate ? (
<div
className={cn(
"pointer-events-none absolute inset-0 z-[70] border-2",
isCanvasImageSelectionSelected
? "border-[#168cff] bg-[#168cff]/10"
: "border-dashed border-[#168cff]/70"
)}
>
<span
className={cn(
"absolute right-2 top-2 flex h-7 w-7 items-center justify-center rounded-full border-2 shadow-sm",
isCanvasImageSelectionSelected
? "border-[#168cff] bg-[#168cff] text-white"
: "border-white bg-white/95 text-[#168cff]"
)}
>
{isCanvasImageSelectionSelected ? (
<Check className="h-4 w-4" strokeWidth={3} />
) : null}
</span>
</div>
) : null}
</Rnd>
);
};
@ -2196,7 +2255,8 @@ export function CanvasWorkspace({
isWorkflowCanvas ? "bg-[#050607] text-white" : "bg-background",
tool === 'pan' && "cursor-grab",
tool === 'draw' && "cursor-crosshair",
isPanning && "cursor-grabbing"
isPanning && "cursor-grabbing",
canvasImageSelectionSession && "cursor-crosshair"
)}
style={
isWorkflowCanvas
@ -2494,7 +2554,9 @@ export function CanvasWorkspace({
)}
{/* 选中图片快捷操作 */}
{selectedPendingImageUploadCount > 0 && selectedImageNodes.length === 0 && (
{!canvasImageSelectionSession &&
selectedPendingImageUploadCount > 0 &&
selectedImageNodes.length === 0 && (
<div
className="absolute bottom-16 left-4 z-30 rounded-md border border-amber-200 bg-amber-50 px-3 py-2 text-xs text-amber-800 shadow-sm"
data-canvas-overlay
@ -2502,7 +2564,7 @@ export function CanvasWorkspace({
图片仍在本地,上传完成后可进行 AI 处理
</div>
)}
{selectedImageNodes.length > 0 && (
{!canvasImageSelectionSession && selectedImageNodes.length > 0 && (
<div
className="absolute bottom-16 left-4 z-30 flex items-center gap-2"
data-canvas-overlay
@ -2529,6 +2591,35 @@ export function CanvasWorkspace({
{selectedImageLocalEditItem.label}
</Button>
) : null}
{selectedImageLayoutReplaceEnabled &&
selectedImageNodes.length >= 1 ? (
<Button
variant={isWorkflowCanvas ? "outline" : "secondary"}
size="sm"
className={cn(
"h-8 gap-1.5 border border-border/60 px-3 text-xs shadow-sm",
isWorkflowCanvas &&
"rounded-full border-[#dbe2ee] bg-white text-[#3b4860] shadow-[0_0_0_1px_rgba(219,226,238,0.26)] hover:bg-[#f7f9ff]"
)}
onClick={() =>
requestImageNodeAction(selectedImageNodes[0].id, "layout-replace", {
args: {
sourceNodeIds: selectedImageNodes
.slice(0, 5)
.map((selectedNode) => selectedNode.id),
},
})
}
title={selectedImageLayoutReplaceItem.label}
>
<CanvasToolbarIcon
className="h-3.5 w-3.5"
fallback={LayoutTemplate}
item={selectedImageLayoutReplaceItem}
/>
{selectedImageLayoutReplaceItem.label}
</Button>
) : null}
<Button
variant={isWorkflowCanvas ? "outline" : "secondary"}
size="sm"

View File

@ -0,0 +1,122 @@
"use client";
import { useEffect, useState } from "react";
import { ArrowLeft, ArrowRight, X } from "lucide-react";
export type GroupShot = {
id: string;
title: string;
url: string;
type: "image" | "video";
};
export function GroupStoryboardPreview({
shots,
onOrder,
onClose,
}: {
shots: GroupShot[];
onOrder: (ids: string[]) => void;
onClose: () => void;
}) {
const [index, setIndex] = useState(0),
shot = shots[Math.min(index, shots.length - 1)];
useEffect(() => {
const key = (e: KeyboardEvent) => {
if (e.key === "Escape") onClose();
};
window.addEventListener("keydown", key);
return () => window.removeEventListener("keydown", key);
}, [onClose]);
if (!shot) return null;
const move = (delta: number) => {
const next = index + delta;
if (next < 0 || next >= shots.length) return;
const ids = shots.map((s) => s.id);
[ids[index], ids[next]] = [ids[next], ids[index]];
onOrder(ids);
setIndex(next);
};
return (
<div
data-canvas-overlay="true"
role="dialog"
aria-label="编组分镜预览"
className="fixed inset-0 z-[120] flex flex-col bg-black/95 p-8 text-white"
onPointerDown={(e) => e.stopPropagation()}
>
<header className="mb-4 flex items-center justify-between">
<h2>
分镜 {index + 1}/{shots.length} · {shot.title}
</h2>
<button aria-label="关闭分镜预览" onClick={onClose}>
<X />
</button>
</header>
<div className="flex min-h-0 flex-1 items-center justify-center">
{shot.type === "video" ? (
<video
key={shot.id}
src={shot.url}
controls
autoPlay
className="h-full max-w-full object-contain"
onEnded={() => setIndex((i) => Math.min(i + 1, shots.length - 1))}
/>
) : (
<img
src={shot.url}
alt={shot.title}
className="h-full max-w-full object-contain"
/>
)}
</div>
<div className="mt-4 flex items-center justify-center gap-6">
<button
type="button"
disabled={index === 0}
onClick={() => setIndex((i) => i - 1)}
className="disabled:opacity-30"
>
<ArrowLeft />
</button>
<span>
{index + 1} / {shots.length}
</span>
<button
type="button"
disabled={index === shots.length - 1}
onClick={() => setIndex((i) => i + 1)}
className="disabled:opacity-30"
>
<ArrowRight />
</button>
<button
type="button"
disabled={index === 0}
onClick={() => move(-1)}
className="disabled:opacity-30"
>
镜头前移
</button>
<button
type="button"
disabled={index === shots.length - 1}
onClick={() => move(1)}
className="disabled:opacity-30"
>
镜头后移
</button>
</div>
<div className="mt-5 flex justify-center gap-3 overflow-x-auto">
{shots.map((s, i) => (
<button
type="button"
key={s.id}
onClick={() => setIndex(i)}
className={`shrink-0 rounded-lg border px-4 py-2 text-sm ${i === index ? "border-white bg-white/20" : "border-white/20"}`}
>
{i + 1} · {s.title}
</button>
))}
</div>
</div>
);
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,78 @@
"use client";
import { useEffect, useRef, useState } from "react";
import { toast } from "sonner";
export function PromptTranslateButton({
prompt,
model,
onApply,
}: {
prompt: string;
model?: string;
onApply: (text: string) => void;
}) {
const [busy, setBusy] = useState(false),
[result, setResult] = useState("");
const controller = useRef<AbortController | null>(null);
useEffect(() => () => controller.current?.abort(), []);
return (
<>
<button
type="button"
disabled={busy || !prompt.trim()}
className="rounded px-2 py-1 text-sm disabled:opacity-40"
onClick={async () => {
const request = new AbortController();
controller.current = request;
setBusy(true);
try {
const response = await fetch("/api/canvas/prompt-tools", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mode: "translate", prompt, model }),
signal: AbortSignal.any([request.signal, AbortSignal.timeout(100000)]),
});
const body = await response.json();
if (!response.ok || !body.success || !body.data?.text)
throw Error(body.error || "翻译没有返回结果");
if (!request.signal.aborted) setResult(body.data.text);
} catch (error) {
if (!request.signal.aborted)
toast.error(error instanceof Error ? error.message : "翻译失败");
} finally {
setBusy(false);
}
}}
>
{busy ? "翻译中…" : "翻译"}
</button>
{result ? (
<div
role="dialog"
aria-label="翻译结果"
className="fixed inset-4 z-[700] flex flex-col gap-3 rounded-xl border border-white/20 bg-[#222] p-5 text-white"
>
<textarea
aria-label="翻译内容"
className="min-h-0 flex-1 resize-none bg-transparent"
value={result}
onChange={(event) => setResult(event.target.value)}
/>
<div className="flex justify-end gap-3">
<button type="button" onClick={() => setResult("")}>
放弃
</button>
<button
type="button"
onClick={() => {
onApply(result);
setResult("");
}}
>
应用翻译
</button>
</div>
</div>
) : null}
</>
);
}

View File

@ -0,0 +1,23 @@
"use client";
import { useRef, useState } from "react";
import { Download, Loader2 } from "lucide-react";
import { toast } from "sonner";
export function VideoArchiveButton({ url, automaticBusy = false, onSaved }: { url: string; automaticBusy?: boolean; onSaved: (url: string) => void }) {
const running = useRef(false);
const [busy, setBusy] = useState(false);
return <button type="button" disabled={busy || automaticBusy} title="当前视频使用临时链接,转存后可长期保留,无需重新生成" className="flex h-10 items-center gap-2 whitespace-nowrap rounded-lg px-3 text-[15px] text-amber-200 hover:bg-white/10 disabled:opacity-50" onClick={async () => {
if (running.current) return;
running.current = true;
setBusy(true);
try {
const response = await fetch("/api/canvas/video/archive", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ url }), signal: AbortSignal.timeout(290_000) });
const result = await response.json();
if (!response.ok || !result.success || !result.data?.url) throw new Error(result.error || "视频转存失败");
onSaved(result.data.url);
toast.success("视频已转存,画布正在保存新地址");
} catch (error) {
toast.error(error instanceof Error && error.name === "TimeoutError" ? "视频保存超时,原视频已保留,请稍后重试" : error instanceof Error ? error.message : "视频转存失败");
} finally { running.current = false; setBusy(false); }
}}>{busy || automaticBusy ? <Loader2 className="size-4 animate-spin" /> : <Download className="size-4" />}{busy || automaticBusy ? "正在转存" : "转存视频"}</button>;
}

View File

@ -0,0 +1,58 @@
"use client";
import { useState } from "react";
import type { CanvasConflictChoice } from "@/lib/video-canvas/merge-snapshot";
export function VideoCanvasConflictNotice({
fields,
onResolve,
onDownload,
}: {
fields: string[];
onResolve: (choice: CanvasConflictChoice) => Promise<void> | undefined;
onDownload: () => void;
}) {
const [busy, setBusy] = useState(false);
if (!fields.length) return null;
const resolve = async (choice: CanvasConflictChoice) => {
setBusy(true);
try {
await onResolve(choice);
} finally {
setBusy(false);
}
};
return (
<div
role="alert"
className="absolute left-1/2 top-20 z-[200] w-[min(680px,90vw)] -translate-x-1/2 rounded-xl border border-amber-400/40 bg-[#29251b] p-4 text-sm text-white shadow-xl"
>
<p className="font-semibold">其他页面修改了相同内容,当前草稿已保留</p>
<p className="mt-2 text-white/70">
共有 {fields.length} 处冲突。不同位置的修改会自动合并;以下选择仅处理冲突部分。
</p>
<ul className="my-2 max-h-32 overflow-auto text-xs text-white/60">
{fields.map((field, index) => (
<li key={`${field}-${index}`}>{field}</li>
))}
</ul>
<div className="flex flex-wrap gap-3">
<button
disabled={busy}
className="rounded bg-white/10 px-3 py-2 disabled:opacity-40"
onClick={() => void resolve("local")}
>
冲突处保留本页
</button>
<button
disabled={busy}
className="rounded bg-white/10 px-3 py-2 disabled:opacity-40"
onClick={() => void resolve("remote")}
>
冲突处采用服务器版
</button>
<button className="rounded bg-white/10 px-3 py-2" onClick={onDownload}>
下载当前草稿
</button>
</div>
</div>
);
}

View File

@ -0,0 +1,253 @@
"use client";
import { useRouter } from "next/navigation";
import { useEffect, useState } from "react";
import useSWR from "swr";
import {
ArrowLeft, Clapperboard, FileText, Film, FolderOpen, Image as ImageIcon,
LayoutGrid, MoreHorizontal, Music2, Plus, Scissors, Sparkles, Video,
WandSparkles, Zap,
} from "lucide-react";
import { toast } from "sonner";
import { filterCanvasModels, type CanvasModelDescriptor } from "@/lib/video-canvas/model-matching";
export type VideoCanvasHomeProject = {
id: string;
title: string;
thumbnail: string | null;
createdAt: string;
updatedAt: string;
};
type Section = "home" | "projects";
type Starter = "image" | "video" | "audio" | "script" | "smart-edit";
type BrandingSettings = {
siteName?: string | null;
siteLogo?: string | null;
siteLogoDark?: string | null;
};
const quickStarts: Array<{ label: string; icon: typeof Video; starter: Starter; mode?: string; model?: string }> = [
{ label: "文生视频", icon: WandSparkles, starter: "video", mode: "text-to-video" },
{ label: "首帧视频", icon: Video, starter: "video", mode: "first-frame" },
{ label: "多参考视频", icon: Film, starter: "video", mode: "reference-images" },
{ label: "视频生成", icon: Clapperboard, starter: "video" },
{ label: "图片生成", icon: ImageIcon, starter: "image" },
{ label: "音频生成", icon: Music2, starter: "audio" },
{ label: "剧本生成", icon: FileText, starter: "script" },
{ label: "智能剪辑", icon: Scissors, starter: "smart-edit" },
];
function projectHref(id: string, starter?: Starter, mode?: string, model?: string) {
const params = new URLSearchParams({ mode: "video", workspaceType: "video" });
if (starter) params.set("starter", starter);
if (mode) params.set("starterMode", mode);
if (model) params.set("starterModel", model);
return `/canvas/${encodeURIComponent(id)}?${params.toString()}`;
}
function projectDate(value: string) {
const parts = new Intl.DateTimeFormat("zh-CN", {
timeZone: "Asia/Shanghai", year: "numeric", month: "2-digit", day: "2-digit",
}).formatToParts(new Date(value));
const get = (name: string) => parts.find(part => part.type === name)?.value || "";
return `${get("year")}-${get("month")}-${get("day")}`;
}
function projectTitle(value: string) {
return !value || value === "未命名视频画布" ? "未命名" : value;
}
export function VideoCanvasHome({
initialHasMore, initialProjects, siteLogo, siteLogoDark, siteName, userName,
}: {
initialHasMore: boolean;
initialProjects: VideoCanvasHomeProject[];
siteLogo: string | null;
siteLogoDark: string | null;
siteName: string;
userName: string;
}) {
const router = useRouter();
const [section, setSection] = useState<Section>("home");
const [projects, setProjects] = useState(initialProjects);
const [hasMore, setHasMore] = useState(initialHasMore);
const [loadingMore, setLoadingMore] = useState(false);
const [creating, setCreating] = useState(false);
const [menuId, setMenuId] = useState<string | null>(null);
const [editingId, setEditingId] = useState<string | null>(null);
const [draftTitle, setDraftTitle] = useState("");
const { data: branding } = useSWR<BrandingSettings>(
"/api/site-settings?scope=branding",
async (url: string) => {
const response = await fetch(url, { cache: "no-store" });
if (!response.ok) throw new Error("品牌信息不可用");
return response.json();
},
{ refreshInterval: 30_000, revalidateOnFocus: true },
);
const brandName = String(branding?.siteName ?? siteName).trim() || "AI Chatbot";
const brandLogo = (branding ? branding.siteLogoDark : siteLogoDark)?.trim()
|| (branding ? branding.siteLogo : siteLogo)?.trim();
const { data: membership } = useSWR<{ summary?: { currentCredits?: number } }>(
"/api/user/membership-summary",
async (url: string) => {
const response = await fetch(url);
if (!response.ok) throw new Error("积分信息不可用");
return response.json();
},
);
const { data: modelData } = useSWR<{ models?: CanvasModelDescriptor[] }>("/api/models", async (url: string) => {
const response = await fetch(url);
if (!response.ok) throw new Error("模型目录不可用");
return response.json();
});
const availableVideoModels = filterCanvasModels(modelData?.models || [], "video", "text-to-video").slice(0, 3);
const shortcuts = quickStarts.map((shortcut, index) => {
const model = index < 3 ? availableVideoModels[index] : undefined;
return model ? {
...shortcut,
label: model.displayName || model.modelId || shortcut.label,
model: model.modelId || model.id,
mode: "text-to-video",
} : shortcut;
});
useEffect(() => {
let active = true;
async function refreshRecent() {
try {
const response = await fetch("/api/canvas?workspaceType=video&limit=24&page=1", { cache: "no-store" });
const result = await response.json();
if (!active || !response.ok || !result.success) return;
setProjects(current => result.data.map((item: VideoCanvasHomeProject) => ({
...item,
thumbnail: item.thumbnail || current.find(existing => existing.id === item.id)?.thumbnail || null,
})));
setHasMore(Boolean(result.pagination?.hasMore));
} catch {
// The server-rendered project list remains available if refreshing fails.
}
}
void refreshRecent();
return () => { active = false; };
}, []);
async function createProject(starter?: Starter, mode?: string, model?: string) {
if (creating) return;
setCreating(true);
try {
const response = await fetch("/api/canvas", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ workspaceType: "video", title: "未命名视频画布" }),
});
const result = await response.json();
if (!response.ok || !result.success || !result.data?.id) throw new Error(result.error || "创建项目失败");
setProjects(current => [{
id: result.data.id,
title: result.data.title || "未命名视频画布",
thumbnail: null,
createdAt: result.data.createdAt || new Date().toISOString(),
updatedAt: result.data.updatedAt || new Date().toISOString(),
}, ...current]);
router.push(projectHref(result.data.id, starter, mode, model));
} catch (error) {
toast.error(error instanceof Error ? error.message : "创建项目失败,请重试");
setCreating(false);
}
}
async function loadMore() {
if (loadingMore || !hasMore) return;
setLoadingMore(true);
try {
const page = Math.floor(projects.length / 24) + 1;
const response = await fetch(`/api/canvas?workspaceType=video&limit=24&page=${page}`, { cache: "no-store" });
const result = await response.json();
if (!response.ok || !result.success) throw new Error(result.error || "加载项目失败");
setProjects(current => [...current, ...result.data.filter((item: VideoCanvasHomeProject) => !current.some(existing => existing.id === item.id))]);
setHasMore(Boolean(result.pagination?.hasMore));
} catch (error) {
toast.error(error instanceof Error ? error.message : "加载项目失败");
} finally {
setLoadingMore(false);
}
}
async function saveTitle(id: string) {
const title = draftTitle.trim();
if (!title) { toast.error("项目名称不能为空"); return; }
try {
const response = await fetch(`/api/canvas/${encodeURIComponent(id)}`, {
method: "PUT", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ title, workspaceType: "video" }),
});
const result = await response.json();
if (!response.ok || !result.success) throw new Error(result.error || "重命名失败");
setProjects(current => current.map(item => item.id === id ? { ...item, title, updatedAt: result.data.updatedAt || item.updatedAt } : item));
setEditingId(null);
toast.success("项目已重命名");
} catch (error) {
toast.error(error instanceof Error ? error.message : "重命名失败");
}
}
const visibleProjects = section === "home" ? projects.slice(0, 4) : projects;
return (
<div className="flex h-svh min-w-0 overflow-hidden bg-[#191919] text-[#f7f7f7]">
<aside className="flex w-[210px] shrink-0 flex-col border-r border-white/[0.06] bg-[#1c1c1c] px-3 py-[18px] max-sm:w-[60px] max-sm:px-2">
<a aria-label={`返回${brandName}首页`} className="mb-4 flex h-7 min-w-0 items-center gap-2 px-2 text-[18px] font-semibold" href="/">
{brandLogo ? <img alt="" className="size-6 shrink-0 object-contain" src={brandLogo} /> : <span className="flex size-5 shrink-0 items-center justify-center rounded bg-white text-[#191919]"><Film className="size-3.5" /></span>}
<span className="truncate max-sm:hidden">{brandName}</span>
</a>
<button className="flex h-[33px] items-center gap-2 rounded-[6px] bg-[#15bde4] px-2.5 text-left text-[13px] font-medium text-black hover:bg-[#3ed0ed] disabled:opacity-55" disabled={creating} onClick={() => void createProject()} type="button">
<Plus className="size-4" /><span className="max-sm:hidden">{creating ? "正在创建…" : "新建项目"}</span>
</button>
<div className="mt-5 space-y-1 border-b border-white/[0.06] pb-4">
<button className="flex h-8 w-full items-center gap-2 rounded-md px-2.5 text-left text-[12px] text-white/75 hover:bg-white/[0.06]" onClick={() => void createProject("video")} type="button"><Sparkles className="size-4" /><span className="max-sm:hidden">视频创作</span></button>
<button className="flex h-8 w-full items-center gap-2 rounded-md px-2.5 text-left text-[12px] text-white/75 hover:bg-white/[0.06]" onClick={() => void createProject("smart-edit")} type="button"><Scissors className="size-4" /><span className="max-sm:hidden">智能剪辑</span></button>
</div>
<nav aria-label="视频画布导航" className="mt-3 space-y-1">
<button aria-current={section === "home" ? "page" : undefined} className={`flex h-8 w-full items-center gap-2 rounded-md px-2.5 text-left text-[12px] ${section === "home" ? "bg-[#282828] text-white" : "text-white/75 hover:bg-white/[0.06]"}`} onClick={() => setSection("home")} type="button"><LayoutGrid className="size-4" /><span className="max-sm:hidden">首页</span></button>
<button aria-current={section === "projects" ? "page" : undefined} className={`flex h-8 w-full items-center gap-2 rounded-md px-2.5 text-left text-[12px] ${section === "projects" ? "bg-[#282828] text-white" : "text-white/75 hover:bg-white/[0.06]"}`} onClick={() => setSection("projects")} type="button"><FolderOpen className="size-4" /><span className="max-sm:hidden">项目</span></button>
<a className="flex h-8 items-center gap-2 rounded-md px-2.5 text-[12px] text-white/75 hover:bg-white/[0.06]" href="/gallery"><ImageIcon className="size-4" /><span className="max-sm:hidden">资产</span></a>
</nav>
<a aria-label="返回生图画布首页" className="mt-3 flex h-8 items-center gap-2 rounded-md border border-white/10 px-2.5 text-[12px] text-white/75 hover:bg-white/[0.06] hover:text-white" href="/" title="返回生图画布首页"><ArrowLeft className="size-4 shrink-0" /><span className="truncate max-sm:hidden">返回生图画布首页</span></a>
<div className="mt-auto flex items-center gap-2 border-t border-white/[0.06] px-2 pt-4 text-[12px] text-white/70"><span className="flex size-7 items-center justify-center rounded-full bg-[#414a62] font-medium text-white">{(userName || "用").slice(0, 1)}</span><span className="truncate max-sm:hidden">{userName || "我的工作区"}</span></div>
</aside>
<main className="min-w-0 flex-1 overflow-y-auto">
<header className="flex h-[46px] items-center justify-end gap-2 border-b border-white/[0.04] px-5 md:px-9">
<span className="flex h-8 items-center gap-1 rounded-lg border border-white/10 px-3 text-[12px] text-white/75"><Zap className="size-3.5" />{membership?.summary?.currentCredits ?? "—"}</span>
<span aria-label={userName || "当前用户"} className="flex size-8 items-center justify-center rounded-full bg-[#414a62] text-[12px] font-medium">{(userName || "用").slice(0, 1)}</span>
</header>
<div className="mx-auto max-w-[1500px] px-5 pb-12 pt-[10px] md:px-9">
{section === "home" ? <>
<button aria-label="新建画布创作" className="flex h-[140px] w-full flex-col items-center justify-center gap-3 rounded-[12px] border border-white/[0.14] bg-[radial-gradient(circle_at_center,rgba(255,255,255,.045),transparent_65%),radial-gradient(#5c5c5c_0.8px,transparent_0.8px)] bg-[length:100%_100%,18px_18px] text-[14px] font-semibold hover:border-white/30" disabled={creating} onClick={() => void createProject()} type="button">
<span className="flex h-[48px] w-[80px] items-center justify-center rounded-[15px] bg-[linear-gradient(#fff,#e9e9e9)] text-black shadow-[0_8px_25px_rgba(255,255,255,.1)]"><Plus className="size-8 stroke-[1.8]" /></span>
新建画布创作
</button>
<div className="mt-7 grid grid-cols-4 gap-3 lg:grid-cols-8 lg:gap-5">
{shortcuts.map(({ label, icon: Icon, starter, mode, model }, index) => <button className="group min-w-0 text-center disabled:opacity-50" disabled={creating} key={`${index}-${label}`} onClick={() => void createProject(starter, mode, model)} title={label} type="button"><span className="mx-auto flex h-[50px] w-[86px] max-w-full items-center justify-center rounded-[16px] bg-[#262626] transition-colors group-hover:bg-[#343434]"><Icon className="size-5 text-white/90" /></span><span className="mt-2 block truncate text-[12px] font-medium">{label}</span></button>)}
</div>
</> : null}
<div className={section === "home" ? "mt-10" : "mt-1"}>
<div className="mb-4 flex items-center justify-between"><h1 className="text-[14px] font-semibold">{section === "home" ? "最近项目" : "全部项目"}</h1>{section === "home" && projects.length > 0 ? <button className="text-[12px] text-white/75 hover:text-white" onClick={() => setSection("projects")} type="button">查看全部 ›</button> : null}</div>
{visibleProjects.length ? <div className={`grid grid-cols-1 gap-3 ${section === "home" ? "sm:grid-cols-2 lg:grid-cols-4" : "sm:grid-cols-2"}`}>{visibleProjects.map(project => <div className="group relative flex h-[74px] min-w-0 items-center gap-3 rounded-[24px] border border-white/[0.09] bg-[#202020] pr-11 transition-colors hover:border-white/[0.19] hover:bg-[#292929]" key={project.id}>
<button aria-label={`打开项目 ${projectTitle(project.title)}`} className="absolute inset-0 rounded-[24px]" onClick={() => router.push(projectHref(project.id))} type="button" />
<span className="pointer-events-none z-[1] ml-1 flex h-[66px] w-[88px] shrink-0 items-center justify-center overflow-hidden rounded-[20px] bg-[#292929] text-white/30">{project.thumbnail ? <img alt="" className="size-full object-cover" src={project.thumbnail} /> : <Sparkles className="size-5" />}</span>
<span className="pointer-events-none z-[1] min-w-0 flex-1"><span className="block truncate text-[13px] font-semibold">{projectTitle(project.title)}</span><span className="mt-1 block text-[11px] text-white/50">{projectDate(project.updatedAt)}</span></span>
<button aria-label={`项目 ${projectTitle(project.title)} 的更多操作`} aria-expanded={menuId === project.id} className="absolute right-4 top-4 z-[2] flex size-7 items-center justify-center rounded-lg text-white/60 opacity-0 hover:bg-white/10 hover:text-white focus:opacity-100 group-hover:opacity-100" onClick={() => setMenuId(current => current === project.id ? null : project.id)} type="button"><MoreHorizontal className="size-4" /></button>
{menuId === project.id ? <div className="absolute right-3 top-11 z-[4] min-w-[110px] rounded-lg border border-white/10 bg-[#303030] p-1 shadow-xl"><button className="w-full rounded px-3 py-2 text-left text-[12px] hover:bg-white/10" onClick={() => { setEditingId(project.id); setDraftTitle(projectTitle(project.title)); setMenuId(null); }} type="button">重命名</button></div> : null}
{editingId === project.id ? <form className="absolute inset-1 z-[5] flex items-center gap-2 rounded-[20px] bg-[#292929] px-3" onSubmit={event => { event.preventDefault(); void saveTitle(project.id); }}><input aria-label="项目名称" autoFocus className="min-w-0 flex-1 rounded-md border border-white/20 bg-[#202020] px-3 py-2 text-[13px] outline-none focus:border-white/60" maxLength={256} onChange={event => setDraftTitle(event.target.value)} value={draftTitle} /><button className="text-[12px] text-cyan-300" type="submit">保存</button><button className="text-[12px] text-white/50" onClick={() => setEditingId(null)} type="button">取消</button></form> : null}
</div>)}</div> : <div className="rounded-2xl border border-dashed border-white/10 px-5 py-14 text-center text-[13px] text-white/55">还没有视频项目,点击上方“新建画布创作”开始。</div>}
{section === "projects" && hasMore ? <button className="mx-auto mt-6 block rounded-lg border border-white/15 px-5 py-2 text-[12px] text-white/70 hover:bg-white/10 disabled:opacity-50" disabled={loadingMore} onClick={() => void loadMore()} type="button">{loadingMore ? "加载中…" : "加载更多"}</button> : null}
</div>
</div>
</main>
</div>
);
}

View File

@ -0,0 +1,49 @@
"use client";
import { Dialog, DialogContent, DialogTitle } from "@/components/ui/dialog";
import type { PromptToolMode } from "@/lib/video-canvas/prompt-tools";
export function VideoPromptToolDialog({ mode, busy, error, output, source, currentPrompt, model, models, onModelChange, onOutputChange, onRun, onApply, onClose }: {
mode: PromptToolMode;
busy: boolean;
error: string;
output: string;
source: string;
currentPrompt: string;
model: string;
models: { id: string; label: string }[];
onModelChange: (value: string) => void;
onOutputChange: (value: string) => void;
onRun: () => void;
onApply: () => void;
onClose: () => void;
}) {
const title = mode === "optimize" ? "提示词优化" : "提示词翻译";
return (
<Dialog open onOpenChange={open => { if (!open) onClose(); }}>
<DialogContent hideCloseButton aria-describedby={undefined} onPointerDown={event => event.stopPropagation()} className="flex max-h-[90dvh] w-[620px] max-w-full flex-col gap-4 overflow-y-auto rounded-2xl border border-white/10 bg-[#262626] p-5 text-white shadow-2xl">
<div className="flex items-center justify-between">
<DialogTitle className="text-base font-medium">{title}</DialogTitle>
<button aria-label="关闭提示词工具" className="rounded px-2 py-1 text-white/60 hover:bg-white/10" onClick={onClose} type="button">✕</button>
</div>
<label className="flex items-center gap-3 text-sm text-white/65">
处理模型
<select aria-label="提示词处理模型" className="min-w-0 flex-1 rounded-lg border border-white/15 bg-[#202020] px-3 py-2" disabled={busy} onChange={event => onModelChange(event.target.value)} value={model}>
<option value="auto">使用管理员默认模型</option>
{model !== "auto" && !models.some(item => item.id === model) ? <option value={model}>当前配置模型(暂不可选)</option> : null}
{models.map(item => <option key={item.id} value={item.id}>{item.label}</option>)}
</select>
</label>
<details className="text-sm text-white/55"><summary className="cursor-pointer">查看原提示词</summary><p className="mt-2 whitespace-pre-wrap">{source}</p></details>
{error ? <p role="alert" className="rounded-lg bg-red-400/10 p-3 text-sm text-red-200">{error}</p> : null}
<textarea aria-label={`${title}结果`} className="min-h-[220px] w-full resize-y rounded-xl border border-white/10 bg-black/20 p-3 text-sm leading-6 outline-none focus:border-white/30" disabled={busy} onChange={event => onOutputChange(event.target.value)} placeholder={busy ? "正在处理,请稍候…" : "处理结果会显示在这里,确认后填入视频节点。"} value={output} />
{source !== currentPrompt ? <p className="text-xs text-amber-200">原文已修改,应用结果将替换当前提示词。</p> : null}
<div className="flex items-center justify-end gap-2">
<button className="rounded-lg px-3 py-2 text-sm text-white/65 hover:bg-white/10" onClick={onClose} type="button">{busy ? "取消处理" : "取消"}</button>
<button className="rounded-lg border border-white/15 px-3 py-2 text-sm disabled:opacity-40" disabled={busy} onClick={onRun} type="button">{busy ? "处理中…" : output ? "重新生成" : "重试"}</button>
<button className="rounded-lg bg-white px-4 py-2 text-sm text-black disabled:opacity-40" disabled={busy || !output.trim() || Boolean(error)} onClick={onApply} type="button">应用到提示词</button>
</div>
</DialogContent>
</Dialog>
);
}

View File

@ -1,5 +1,4 @@
import { Sparkles } from "lucide-react";
import { cn } from "@/lib/utils";
import { BloubAvatar } from "@/components/bloub-avatar";
type CanvasLogoLoadingViewProps = {
label?: string;
@ -10,46 +9,17 @@ type CanvasLogoLoadingViewProps = {
export function CanvasLogoLoadingView({
label = "正在加载画布...",
siteName = "AI Chatbot",
siteLogo,
siteLogoDark,
}: CanvasLogoLoadingViewProps) {
const resolvedSiteName = String(siteName || "AI Chatbot").trim();
const lightLogo = String(siteLogo || siteLogoDark || "").trim();
const darkLogo = String(siteLogoDark || siteLogo || "").trim();
const hasSeparateDarkLogo = Boolean(darkLogo && darkLogo !== lightLogo);
return (
<div className="flex h-dvh w-screen items-center justify-center bg-[#f4f4f5] text-[#20242a] dark:bg-[#08090b] dark:text-zinc-100">
<div className="flex flex-col items-center gap-3.5">
<div className="relative flex h-[78px] w-[78px] items-center justify-center">
<div className="absolute inset-0 animate-spin rounded-full bg-[conic-gradient(from_0deg,#ff7a00,#ffe04b,#48e6a5,#35a7ff,#b15cff,#ff4aa2,#ff7a00)] opacity-90 blur-[1px]" />
<div className="absolute inset-[6px] animate-ping rounded-full bg-[conic-gradient(from_180deg,#ff7a00,#35a7ff,#b15cff,#48e6a5,#ff7a00)] opacity-25" />
<div className="absolute inset-1 rounded-full bg-[#f4f4f5] dark:bg-[#08090b]" />
<div className="relative flex h-14 w-14 items-center justify-center overflow-hidden rounded-full border border-black/8 bg-white shadow-[0_13px_38px_rgba(15,23,42,0.16)] dark:border-white/10 dark:bg-zinc-950 dark:shadow-[0_17px_49px_rgba(0,0,0,0.45)]">
{lightLogo ? (
<>
<img
alt={resolvedSiteName}
className={cn(
"h-[34px] w-[34px] object-contain",
hasSeparateDarkLogo && "dark:hidden"
)}
src={lightLogo}
/>
{hasSeparateDarkLogo ? (
<img
alt={resolvedSiteName}
className="hidden h-[34px] w-[34px] object-contain dark:block"
src={darkLogo}
/>
) : null}
</>
) : (
<Sparkles className="h-[22px] w-[22px] text-[#20242a] dark:text-white" />
)}
</div>
</div>
<div className="flex flex-col items-center gap-2.5">
<BloubAvatar
label={label}
playSequence
size={104}
state="idle"
surface="page"
/>
<div className="text-[12px] font-medium tracking-[0.16em] text-[#6b7280] dark:text-zinc-400">
{label}
</div>

View File

@ -0,0 +1,598 @@
"use client";
/* eslint-disable @next/next/no-img-element */
import { OrbitControls, useGLTF } from "@react-three/drei";
import { Canvas, useFrame, useThree } from "@react-three/fiber";
import { AlertCircle } from "lucide-react";
import React, { Suspense, useEffect, useMemo, useRef } from "react";
import * as THREE from "three";
import type { Mesh, Object3D } from "three";
import type { OrbitControls as OrbitControlsImpl } from "three-stdlib";
import { LocalStudioEnvironment } from "@/components/canvas/model3d/LocalStudioEnvironment";
import { ModelLoadingFallback } from "@/components/canvas/model3d/ModelLoadingFallback";
import type {
OpenClawModelInteractionManifest,
OpenClawModelInteractionPart,
} from "@/lib/openclaw-chat/model-interaction";
import {
getOpenClawExplodedOffsetMm,
getOpenClawPartForSceneNode,
getOpenClawSourceNodeOffset,
mapOpenClawCadOffsetToGlb,
} from "@/lib/openclaw-chat/model-interaction";
export type OpenClawModelTransformRequest = {
actionId: string | null;
amount: number;
animate: boolean;
durationMs: number;
nonce: number;
};
export type OpenClawModelCameraState = {
position: [number, number, number];
target: [number, number, number];
};
type InteractiveEntry = {
node: Object3D;
part: OpenClawModelInteractionPart;
initialPosition: THREE.Vector3;
};
type AnimationState = {
startedAt: number;
durationMs: number;
starts: Map<string, THREE.Vector3>;
targets: Map<string, THREE.Vector3>;
easing: string;
};
type DragState = {
entry: InteractiveEntry;
plane: THREE.Plane;
startIntersection: THREE.Vector3;
startWorldPosition: THREE.Vector3;
startLocalPosition: THREE.Vector3;
pointerId: number;
};
class InteractiveViewerErrorBoundary extends React.Component<
{ children: React.ReactNode; fallback: React.ReactNode; resetKey: string },
{ hasError: boolean }
> {
state = { hasError: false };
static getDerivedStateFromError() {
return { hasError: true };
}
componentDidUpdate(previousProps: { resetKey: string }) {
if (
this.state.hasError &&
previousProps.resetKey !== this.props.resetKey
) {
this.setState({ hasError: false });
}
}
render() {
return this.state.hasError ? this.props.fallback : this.props.children;
}
}
function isMesh(node: Object3D): node is Mesh {
return (node as Mesh).isMesh === true;
}
function cloneScene(scene: Object3D) {
const clone = scene.clone(true);
clone.traverse((node) => {
if (!isMesh(node)) return;
node.castShadow = true;
node.receiveShadow = true;
const materials = Array.isArray(node.material)
? node.material
: [node.material];
const clones = materials.map((material) => material.clone());
node.material = Array.isArray(node.material)
? clones
: clones[0] || node.material;
});
return clone;
}
function getFitTransform(scene: Object3D) {
const bounds = new THREE.Box3().setFromObject(scene);
if (bounds.isEmpty()) {
return {
position: [0, 0, 0] as [number, number, number],
scale: 1,
};
}
const size = bounds.getSize(new THREE.Vector3());
const center = bounds.getCenter(new THREE.Vector3());
const largestDimension = Math.max(size.x, size.y, size.z);
if (!Number.isFinite(largestDimension) || largestDimension <= 0) {
return {
position: [0, 0, 0] as [number, number, number],
scale: 1,
};
}
const scale = 2 / largestDimension;
return {
position: [
-center.x * scale,
-center.y * scale,
-center.z * scale,
] as [number, number, number],
scale,
};
}
function getEasedProgress(progress: number, easing: string) {
if (easing.toLowerCase() === "linear") return progress;
return progress < 0.5
? 4 * progress * progress * progress
: 1 - Math.pow(-2 * progress + 2, 3) / 2;
}
function getPartTargetPosition(
entry: InteractiveEntry,
manifest: OpenClawModelInteractionManifest,
amount: number
) {
const offsetMm = getOpenClawExplodedOffsetMm(entry.part);
if (!offsetMm) return entry.initialPosition.clone();
const offset = mapOpenClawCadOffsetToGlb(offsetMm, manifest.scaleMmToGlb);
return entry.initialPosition
.clone()
.add(new THREE.Vector3(offset.x, offset.y, offset.z).multiplyScalar(amount));
}
function ModelInteractionScene({
allowDragging,
fitExplodedSource,
initialCameraState,
interactionParts,
manifest,
modelUrl,
onMatchChange,
onCameraStateChange,
onSelectedPartChange,
resetAllToken,
resetSelectedToken,
selectedNodeName,
sourceVariant,
transformRequest,
}: {
allowDragging: boolean;
fitExplodedSource: boolean;
initialCameraState?: OpenClawModelCameraState | null;
interactionParts: OpenClawModelInteractionPart[];
manifest: OpenClawModelInteractionManifest;
modelUrl: string;
onMatchChange: (matched: number, total: number) => void;
onCameraStateChange?: (state: OpenClawModelCameraState) => void;
onSelectedPartChange: (part: OpenClawModelInteractionPart | null) => void;
resetAllToken: number;
resetSelectedToken: number;
selectedNodeName: string | null;
sourceVariant: "assembled" | "exploded";
transformRequest: OpenClawModelTransformRequest;
}) {
const { camera, gl, scene: rootScene } = useThree();
const controlsRef = useRef<OrbitControlsImpl | null>(null);
const gltf = useGLTF(modelUrl, true);
const scene = useMemo(() => {
const cloned = cloneScene(gltf.scene);
if (sourceVariant !== "exploded") return cloned;
const partsByNodeName = new Map(
interactionParts.map((part) => [part.nodeName, part])
);
cloned.traverse((node) => {
const part = getOpenClawPartForSceneNode(node, partsByNodeName);
if (!part) return;
const offset = getOpenClawSourceNodeOffset(
part,
sourceVariant,
manifest.scaleMmToGlb
);
node.position.add(new THREE.Vector3(offset.x, offset.y, offset.z));
});
cloned.updateMatrixWorld(true);
return cloned;
}, [gltf.scene, interactionParts, manifest.scaleMmToGlb, sourceVariant]);
const fitTransform = useMemo(
() => getFitTransform(fitExplodedSource ? gltf.scene : scene),
[fitExplodedSource, gltf.scene, scene]
);
const entriesRef = useRef<Map<string, InteractiveEntry>>(new Map());
const objectEntriesRef = useRef<Map<Object3D, InteractiveEntry>>(new Map());
const animationRef = useRef<AnimationState | null>(null);
const dragRef = useRef<DragState | null>(null);
const currentAmountRef = useRef(0);
const selectionHelperRef = useRef<THREE.BoxHelper | null>(null);
useEffect(() => {
const partsByNodeName = new Map(
interactionParts.map((part) => [part.nodeName, part])
);
const entries = new Map<string, InteractiveEntry>();
const objectEntries = new Map<Object3D, InteractiveEntry>();
scene.traverse((node) => {
const part = getOpenClawPartForSceneNode(node, partsByNodeName);
if (!part || entries.has(part.nodeName)) return;
const entry = {
node,
part,
initialPosition: node.position.clone(),
};
entries.set(part.nodeName, entry);
objectEntries.set(node, entry);
});
entriesRef.current = entries;
objectEntriesRef.current = objectEntries;
onMatchChange(entries.size, interactionParts.length);
return () => {
entriesRef.current.clear();
objectEntriesRef.current.clear();
onMatchChange(0, interactionParts.length);
};
}, [interactionParts, onMatchChange, scene]);
useEffect(() => {
if (!initialCameraState || !controlsRef.current) return;
camera.position.set(...initialCameraState.position);
controlsRef.current.target.set(...initialCameraState.target);
camera.updateProjectionMatrix();
controlsRef.current.update();
}, [camera, initialCameraState]);
useEffect(() => {
const previous = selectionHelperRef.current;
if (previous) {
rootScene.remove(previous);
previous.geometry.dispose();
(previous.material as THREE.Material).dispose();
selectionHelperRef.current = null;
}
if (!selectedNodeName) return;
const entry = entriesRef.current.get(selectedNodeName);
if (!entry) return;
const helper = new THREE.BoxHelper(entry.node, 0x3ecf8e);
helper.raycast = () => undefined;
rootScene.add(helper);
selectionHelperRef.current = helper;
return () => {
rootScene.remove(helper);
helper.geometry.dispose();
(helper.material as THREE.Material).dispose();
if (selectionHelperRef.current === helper) {
selectionHelperRef.current = null;
}
};
}, [rootScene, selectedNodeName]);
useEffect(() => {
const amount = THREE.MathUtils.clamp(transformRequest.amount, 0, 1);
currentAmountRef.current = amount;
const starts = new Map<string, THREE.Vector3>();
const targets = new Map<string, THREE.Vector3>();
for (const [nodeName, entry] of entriesRef.current) {
starts.set(nodeName, entry.node.position.clone());
targets.set(nodeName, getPartTargetPosition(entry, manifest, amount));
}
if (!transformRequest.animate) {
animationRef.current = null;
for (const [nodeName, target] of targets) {
entriesRef.current.get(nodeName)?.node.position.copy(target);
}
return;
}
const targetState = amount >= 0.5 ? "exploded" : "assembled";
const action =
manifest.actions.find((item) => item.id === transformRequest.actionId) ||
manifest.actions.find((item) => item.toState === targetState);
animationRef.current = {
startedAt: performance.now(),
durationMs: action?.durationMs ?? transformRequest.durationMs,
starts,
targets,
easing: action?.easing || "easeInOutCubic",
};
}, [manifest, transformRequest]);
useEffect(() => {
if (resetAllToken <= 0) return;
animationRef.current = null;
for (const entry of entriesRef.current.values()) {
entry.node.position.copy(
getPartTargetPosition(entry, manifest, currentAmountRef.current)
);
}
}, [manifest, resetAllToken]);
useEffect(() => {
if (resetSelectedToken <= 0 || !selectedNodeName) return;
const entry = entriesRef.current.get(selectedNodeName);
if (!entry) return;
entry.node.position.copy(
getPartTargetPosition(entry, manifest, currentAmountRef.current)
);
}, [manifest, resetSelectedToken, selectedNodeName]);
useEffect(() => {
const element = gl.domElement;
const raycaster = new THREE.Raycaster();
const pointer = new THREE.Vector2();
const planeIntersection = new THREE.Vector3();
const worldPosition = new THREE.Vector3();
const updateRay = (event: PointerEvent) => {
const bounds = element.getBoundingClientRect();
if (bounds.width <= 0 || bounds.height <= 0) return false;
pointer.set(
((event.clientX - bounds.left) / bounds.width) * 2 - 1,
-((event.clientY - bounds.top) / bounds.height) * 2 + 1
);
raycaster.setFromCamera(pointer, camera);
return true;
};
const findEntry = (object: Object3D) => {
let current: Object3D | null = object;
while (current) {
const entry = objectEntriesRef.current.get(current);
if (entry) return entry;
current = current.parent;
}
return null;
};
const handlePointerDown = (event: PointerEvent) => {
if (event.button !== 0 || animationRef.current || !updateRay(event)) return;
const intersections = raycaster.intersectObjects(
[...entriesRef.current.values()].map((entry) => entry.node),
true
);
const entry = intersections
.map((intersection) => findEntry(intersection.object))
.find((candidate): candidate is InteractiveEntry => Boolean(candidate));
if (!entry) {
onSelectedPartChange(null);
return;
}
onSelectedPartChange(entry.part);
if (!allowDragging || !entry.part.draggable) return;
const normal = camera.getWorldDirection(new THREE.Vector3()).normalize();
const startWorldPosition = entry.node.getWorldPosition(new THREE.Vector3());
const plane = new THREE.Plane().setFromNormalAndCoplanarPoint(
normal,
startWorldPosition
);
if (!raycaster.ray.intersectPlane(plane, planeIntersection)) return;
dragRef.current = {
entry,
plane,
startIntersection: planeIntersection.clone(),
startWorldPosition,
startLocalPosition: entry.node.position.clone(),
pointerId: event.pointerId,
};
if (controlsRef.current) controlsRef.current.enabled = false;
element.setPointerCapture?.(event.pointerId);
event.preventDefault();
event.stopImmediatePropagation();
};
const handlePointerMove = (event: PointerEvent) => {
const drag = dragRef.current;
if (!drag || drag.pointerId !== event.pointerId || !updateRay(event)) return;
if (!raycaster.ray.intersectPlane(drag.plane, planeIntersection)) return;
const targetWorld = worldPosition
.copy(drag.startWorldPosition)
.add(planeIntersection.clone().sub(drag.startIntersection));
const targetLocal = drag.entry.node.parent
? drag.entry.node.parent.worldToLocal(targetWorld.clone())
: targetWorld;
const axes = manifest.drag.axes;
if (!axes.includes("x")) targetLocal.x = drag.startLocalPosition.x;
if (!axes.includes("z")) targetLocal.y = drag.startLocalPosition.y;
if (!axes.includes("y")) targetLocal.z = drag.startLocalPosition.z;
const maxDistance = manifest.drag.maxDistanceMm * manifest.scaleMmToGlb;
const fromInitial = targetLocal.clone().sub(drag.entry.initialPosition);
if (maxDistance > 0 && fromInitial.length() > maxDistance) {
fromInitial.setLength(maxDistance);
targetLocal.copy(drag.entry.initialPosition).add(fromInitial);
}
drag.entry.node.position.copy(targetLocal);
event.preventDefault();
event.stopImmediatePropagation();
};
const finishDrag = (event: PointerEvent) => {
const drag = dragRef.current;
if (!drag || drag.pointerId !== event.pointerId) return;
const snapDistance =
manifest.drag.snapDistanceMm * manifest.scaleMmToGlb;
if (snapDistance > 0) {
const delta = drag.entry.node.position
.clone()
.sub(drag.entry.initialPosition);
delta.set(
Math.round(delta.x / snapDistance) * snapDistance,
Math.round(delta.y / snapDistance) * snapDistance,
Math.round(delta.z / snapDistance) * snapDistance
);
drag.entry.node.position.copy(drag.entry.initialPosition).add(delta);
}
element.releasePointerCapture?.(event.pointerId);
dragRef.current = null;
if (controlsRef.current) controlsRef.current.enabled = true;
event.preventDefault();
event.stopImmediatePropagation();
};
element.addEventListener("pointerdown", handlePointerDown, true);
window.addEventListener("pointermove", handlePointerMove, true);
window.addEventListener("pointerup", finishDrag, true);
window.addEventListener("pointercancel", finishDrag, true);
return () => {
element.removeEventListener("pointerdown", handlePointerDown, true);
window.removeEventListener("pointermove", handlePointerMove, true);
window.removeEventListener("pointerup", finishDrag, true);
window.removeEventListener("pointercancel", finishDrag, true);
dragRef.current = null;
};
}, [allowDragging, camera, gl, manifest, onSelectedPartChange]);
useEffect(() => {
return () => {
scene.traverse((node) => {
if (!isMesh(node)) return;
const materials = Array.isArray(node.material)
? node.material
: [node.material];
for (const material of materials) material.dispose();
});
};
}, [scene]);
useFrame(() => {
selectionHelperRef.current?.update();
const animation = animationRef.current;
if (!animation) return;
const rawProgress =
animation.durationMs <= 0
? 1
: Math.min(1, (performance.now() - animation.startedAt) / animation.durationMs);
const progress = getEasedProgress(rawProgress, animation.easing);
for (const [nodeName, target] of animation.targets) {
const entry = entriesRef.current.get(nodeName);
const start = animation.starts.get(nodeName);
if (entry && start) entry.node.position.lerpVectors(start, target, progress);
}
if (rawProgress >= 1) animationRef.current = null;
});
return (
<>
<ambientLight intensity={0.75} />
<directionalLight castShadow intensity={1.6} position={[4, 7, 5]} />
<directionalLight intensity={0.45} position={[-4, 2, -3]} />
<primitive
object={scene}
position={fitTransform.position}
scale={fitTransform.scale}
/>
<OrbitControls
dampingFactor={0.08}
enableDamping
enablePan
enableRotate
enableZoom
makeDefault
maxPolarAngle={Math.PI / 1.65}
minDistance={1.2}
minPolarAngle={0}
onChange={() => {
const controls = controlsRef.current;
if (!controls || !onCameraStateChange) return;
onCameraStateChange({
position: camera.position.toArray() as [number, number, number],
target: controls.target.toArray() as [number, number, number],
});
}}
ref={controlsRef}
/>
</>
);
}
export function OpenClawInteractiveModelViewer({
allowDragging,
background,
fitExplodedSource = false,
initialCameraState,
interactionParts,
manifest,
modelUrl,
onMatchChange,
onCameraStateChange,
onSelectedPartChange,
previewImageUrl,
resetAllToken,
resetSelectedToken,
resetViewToken,
selectedNodeName,
sourceVariant,
transformRequest,
}: {
allowDragging: boolean;
background: string;
fitExplodedSource?: boolean;
initialCameraState?: OpenClawModelCameraState | null;
interactionParts?: OpenClawModelInteractionPart[];
manifest: OpenClawModelInteractionManifest;
modelUrl: string;
onMatchChange: (matched: number, total: number) => void;
onCameraStateChange?: (state: OpenClawModelCameraState) => void;
onSelectedPartChange: (part: OpenClawModelInteractionPart | null) => void;
previewImageUrl?: string | null;
resetAllToken: number;
resetSelectedToken: number;
resetViewToken: number;
selectedNodeName: string | null;
sourceVariant: "assembled" | "exploded";
transformRequest: OpenClawModelTransformRequest;
}) {
return (
<InteractiveViewerErrorBoundary
resetKey={`${modelUrl}:${resetViewToken}`}
fallback={
<div className="relative flex h-full w-full items-center justify-center overflow-hidden bg-zinc-950 text-zinc-200">
{previewImageUrl ? (
<img
alt="3D 模型静态预览"
className="absolute inset-0 h-full w-full object-contain opacity-80"
src={previewImageUrl}
/>
) : null}
<div className="relative flex max-w-xs flex-col items-center gap-2 rounded-md bg-black/70 px-4 py-3 text-center backdrop-blur-sm">
<AlertCircle className="h-6 w-6 text-amber-300" />
<p className="text-xs">{previewImageUrl ? "3D 交互预览加载失败,可查看静态预览或下载模型" : "3D 交互预览加载失败,请退出预览后重试或下载模型"}</p>
</div>
</div>
}
>
<Canvas
camera={{ fov: 42, position: [2.4, 1.8, 2.4] }}
dpr={[1, 1.75]}
gl={{ antialias: true, powerPreference: "high-performance" }}
key={`${modelUrl}:${resetViewToken}`}
shadows
style={{ background }}
>
<LocalStudioEnvironment />
<Suspense fallback={<ModelLoadingFallback />}>
<ModelInteractionScene
allowDragging={allowDragging}
fitExplodedSource={fitExplodedSource}
initialCameraState={initialCameraState}
interactionParts={interactionParts || manifest.parts}
manifest={manifest}
modelUrl={modelUrl}
onMatchChange={onMatchChange}
onCameraStateChange={onCameraStateChange}
onSelectedPartChange={onSelectedPartChange}
resetAllToken={resetAllToken}
resetSelectedToken={resetSelectedToken}
selectedNodeName={selectedNodeName}
sourceVariant={sourceVariant}
transformRequest={transformRequest}
/>
</Suspense>
</Canvas>
</InteractiveViewerErrorBoundary>
);
}

File diff suppressed because it is too large Load Diff

View File

@ -1,4 +1,5 @@
import type { Attachment, ChatMessage } from "@/lib/types";
import { isOpenClawChatFileAttachment } from "@/lib/openclaw-chat/media-attachments";
export const SESSION_TITLE_MAX_LEN = 18;
export const MAX_CHAT_TEXT_PART_LENGTH = 2000;
@ -8,6 +9,7 @@ export const MAX_CHAT_ATTACHMENT_NAME_LENGTH = 100;
const CHAT_UUID_REGEX =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const CANVAS_CHAT_QUERY_PARAM = "chat";
const ALLOWED_CHAT_MEDIA_TYPES = new Set([
"image/jpeg",
@ -43,6 +45,24 @@ export interface PickedObjectAttachment extends Attachment {
};
}
export interface CanvasChatSessionRecord {
id: string;
title: string;
mode: string;
createdAt: number;
updatedAt: number;
clawdSkillTagId?: string | null;
clawdSkillTagPromptApplied?: boolean;
}
export interface ClawdChatHistoryRecord {
id: string;
title: string;
createdAt: string | Date;
updatedAt: string | Date;
messageCount: number;
}
export function toRawCanvasChatId(value: string): string {
return value.startsWith("canvas-") ? value.replace("canvas-", "") : value;
}
@ -51,6 +71,42 @@ export function isValidChatUuid(value?: string | null): value is string {
return Boolean(value && CHAT_UUID_REGEX.test(String(value).trim()));
}
export function normalizeCanvasChatSessionId(value?: string | null) {
const raw = String(value || "").trim();
if (!raw) return null;
const rawUuid = raw.startsWith("canvas-") ? raw.slice("canvas-".length) : raw;
return isValidChatUuid(rawUuid) ? `canvas-${rawUuid}` : null;
}
export function replaceCanvasChatSessionHistory(params: {
currentUrl: string;
history: Pick<History, "replaceState">;
sessionId: string;
}) {
const normalized = normalizeCanvasChatSessionId(params.sessionId);
if (!normalized) return null;
const url = new URL(params.currentUrl);
url.searchParams.set(CANVAS_CHAT_QUERY_PARAM, normalized);
// Reusing Next.js' existing history state bypasses its URL synchronization.
params.history.replaceState(null, "", url);
return normalized;
}
export function resolveCanvasChatSessionNavigationTarget(params: {
activeSessionId: string;
browserSessionId: string | null;
rootSessionId: string;
routerSessionId: string | null;
}) {
if (params.browserSessionId !== params.routerSessionId) return null;
const targetSessionId = params.routerSessionId || params.rootSessionId;
return targetSessionId !== params.activeSessionId ? targetSessionId : null;
}
export function isValidChatHttpUrl(value?: string | null): value is string {
if (!value) return false;
try {
@ -80,6 +136,9 @@ export function normalizeChatMediaType(value?: string | null): string {
if (ALLOWED_CHAT_MEDIA_TYPES.has(normalized)) {
return normalized;
}
if (isOpenClawChatFileAttachment("", normalized)) {
return normalized;
}
return "image/jpeg";
}
@ -113,6 +172,76 @@ export function getSessionRootStorageKey(canvasId: string, scope: "canvas" | "ag
return `canvas-chat-root:${scope}:${canvasId}`;
}
function parseHistoryTimestamp(value: string | Date) {
const timestamp = value instanceof Date ? value.getTime() : Date.parse(value);
return Number.isFinite(timestamp) ? timestamp : 0;
}
export function mergeClawdHistoryIntoCanvasSessions<
T extends CanvasChatSessionRecord,
>(sessions: T[], history: ClawdChatHistoryRecord[]): T[] {
if (history.length === 0) return sessions;
const nextSessions = [...sessions];
const indexById = new Map(
nextSessions.map((session, index) => [session.id, index])
);
let changed = false;
for (const item of history) {
const rawId = toRawCanvasChatId(String(item.id || "").trim());
if (!isValidChatUuid(rawId)) continue;
const id = `canvas-${rawId}`;
const existingIndex = indexById.get(id);
const historyCreatedAt = parseHistoryTimestamp(item.createdAt);
const historyUpdatedAt =
parseHistoryTimestamp(item.updatedAt) || historyCreatedAt;
const historyTitle = String(item.title || "").trim();
if (existingIndex === undefined) {
const created = {
id,
title: historyTitle || "Claw 对话",
mode: "clawd",
createdAt: historyCreatedAt || historyUpdatedAt,
updatedAt: historyUpdatedAt || historyCreatedAt,
} as T;
indexById.set(id, nextSessions.length);
nextSessions.push(created);
changed = true;
continue;
}
const existing = nextSessions[existingIndex];
const nextTitle = historyTitle || existing.title || "Claw 对话";
const nextCreatedAt =
Number(existing.createdAt) || historyCreatedAt || historyUpdatedAt;
const nextUpdatedAt = Math.max(
Number(existing.updatedAt) || 0,
historyUpdatedAt
);
if (
existing.title === nextTitle &&
existing.mode === "clawd" &&
existing.createdAt === nextCreatedAt &&
existing.updatedAt === nextUpdatedAt
) {
continue;
}
nextSessions[existingIndex] = {
...existing,
title: nextTitle,
mode: "clawd",
createdAt: nextCreatedAt,
updatedAt: nextUpdatedAt,
};
changed = true;
}
if (!changed) return sessions;
return nextSessions.sort((a, b) => b.updatedAt - a.updatedAt);
}
export function extractUserTextPrompt(message: ChatMessage): string {
const parts = (message as any)?.parts || [];
const textFromParts = parts

View File

@ -1,6 +1,10 @@
import type { PickedObjectAttachment } from "@/components/canvas/chat-panel/canvas-chat-session";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
export const CANVAS_IMAGE_UPLOAD_MAX_BYTES = 50 * 1024 * 1024;
export const CANVAS_IMAGE_UPLOAD_MAX_BYTES = IMAGE_UPLOAD_MAX_BYTES;
export type CanvasUploadFileLike = {
name: string;
@ -18,7 +22,7 @@ export function validateCanvasImageUploadFile(file?: CanvasUploadFileLike | null
return { ok: false, message: "请上传图片文件" };
}
if (file.size > CANVAS_IMAGE_UPLOAD_MAX_BYTES) {
return { ok: false, message: "图片大小不能超过 50MB" };
return { ok: false, message: IMAGE_UPLOAD_TOO_LARGE_MESSAGE };
}
return { ok: true };
}

View File

@ -0,0 +1,470 @@
"use client";
import { transformDirectorMotion } from "@/lib/video-canvas/director-extensions";
import { useEffect, useRef, useState } from "react";
import { toast } from "sonner";
import {
presetDirectorMotion,
sampleDirectorMotion,
readDirectorMotion,
resizeDirectorMotion,
validDirectorPose,
type DirectorMotion,
type DirectorPose,
} from "@/lib/video-canvas/director-motion";
export function DirectorMotionPanel({
motion,
hasObjectMotion = false,
externalSeek,
compact = false,
pose,
disabled,
onChange,
onPose,
onPreview,
onExport,
onSend,
exportProgress,
onCancel,
onOverview,
}: {
compact?: boolean;
hasObjectMotion?: boolean;
externalSeek?: { time: number; editing?: boolean };
motion: DirectorMotion;
pose: DirectorPose;
disabled: boolean;
onChange: (motion: DirectorMotion) => void;
onPose: (pose: DirectorPose) => void;
onPreview: (time: number | null) => void;
onExport: () => void;
onSend: () => void;
exportProgress: number | null;
onCancel: () => void;
onOverview: () => void;
}) {
const [autoKeyframe, setAutoKeyframe] = useState(false);
const previousPose = useRef(JSON.stringify(pose));
const [time, setTime] = useState(0),
[playing, setPlaying] = useState(false),
[preview, setPreview] = useState(false);
const timeRef = useRef(time),
latest = useRef({ motion, onPreview });
latest.current = { motion, onPreview };
timeRef.current = time;
const input = useRef<HTMLInputElement>(null);
const button =
"rounded border border-white/15 px-2 py-1.5 text-xs hover:bg-white/10 disabled:opacity-40";
const seek = (t: number) => {
const next = Math.max(0, Math.min(motion.duration, t));
setTime(next);
timeRef.current = next;
setPreview(true);
onPreview(next);
};
const edit = () => {
if (preview) {
const sampled = sampleDirectorMotion(motion, timeRef.current);
if (sampled) onPose(sampled);
}
setPlaying(false);
setPreview(false);
onPreview(null);
};
useEffect(() => {
if (!playing) return;
let frame = 0,
previous = 0;
const tick = (now: number) => {
const m = latest.current.motion;
const next = timeRef.current + (previous ? (now - previous) / 1000 : 0);
previous = now;
const t = next >= m.duration ? (m.loop ? next % m.duration : m.duration) : next;
timeRef.current = t;
setTime(t);
latest.current.onPreview(t);
if (next >= m.duration && !m.loop) {
setPlaying(false);
return;
}
frame = requestAnimationFrame(tick);
};
const hide = () => {
if (document.hidden) {
setPlaying(false);
toast.info("页面已切换,运镜预览已暂停");
}
};
frame = requestAnimationFrame(tick);
document.addEventListener("visibilitychange", hide);
return () => {
cancelAnimationFrame(frame);
document.removeEventListener("visibilitychange", hide);
};
}, [playing]);
useEffect(() => {
if (disabled) setPlaying(false);
}, [disabled]);
useEffect(() => {
if (timeRef.current > motion.duration) {
setTime(motion.duration);
onPreview(preview ? motion.duration : null);
}
}, [motion.duration, onPreview, preview]);
const add = () => {
if (!validDirectorPose(pose)) {
toast.error("相机不能与注视点重合");
return;
}
const existing = motion.keyframes.find((k) => Math.abs(k.time - time) < 0.01);
if (!existing && motion.keyframes.length >= 100) {
toast.error("最多支持 100 个关键帧");
return;
}
const key = {
id: existing?.id || crypto.randomUUID(),
time: existing?.time ?? Number(time.toFixed(2)),
position: [...pose.position] as DirectorPose["position"],
target: [...pose.target] as DirectorPose["target"],
fov: pose.fov,
};
onChange({
...motion,
keyframes: [...motion.keyframes.filter((k) => k.id !== key.id), key].sort(
(a, b) => a.time - b.time,
),
});
};
useEffect(() => {
const signature = JSON.stringify(pose),
changed = signature !== previousPose.current;
previousPose.current = signature;
if (autoKeyframe && changed && !preview && !disabled) add();
}, [pose, autoKeyframe, preview, disabled]);
useEffect(() => {
if (!externalSeek) return;
setPlaying(false);
setTime(externalSeek.time);
timeRef.current = externalSeek.time;
setPreview(!externalSeek.editing);
}, [externalSeek]);
const downloadJson = () => {
const url = URL.createObjectURL(
new Blob([JSON.stringify({ version: 1, motion }, null, 2)], {
type: "application/json",
}),
);
const a = document.createElement("a");
a.href = url;
a.download = "director-motion.json";
a.click();
setTimeout(() => URL.revokeObjectURL(url), 1000);
};
return (
<section className="space-y-3 border-t border-white/15 pt-3" aria-label="镜头轨迹">
<div className="text-sm font-semibold">镜头轨迹</div>
<fieldset
disabled={disabled}
className={
compact
? "grid grid-cols-1 gap-3 sm:grid-cols-3 lg:grid-cols-4 disabled:opacity-50"
: "space-y-3 disabled:opacity-50"
}
>
<div className="grid grid-cols-2 gap-2">
<label className="text-xs">
时长(秒)
<input
aria-label="运镜时长"
type="number"
min={2}
max={15}
step={1}
value={motion.duration}
className="mt-1 w-full rounded bg-white/10 p-1"
onChange={(e) => {
const n = Number(e.target.value);
if (n >= 2 && n <= 15) {
edit();
onChange(resizeDirectorMotion(motion, n));
}
}}
/>
</label>
<label className="text-xs">
速度
<select
aria-label="运镜插值"
value={motion.easing}
className="mt-1 w-full rounded bg-[#303030] p-1"
onChange={(e) =>
onChange({
...motion,
easing: e.target.value as DirectorMotion["easing"],
})
}
>
<option value="smooth">缓入缓出</option>
<option value="linear">匀速</option>
</select>
</label>
</div>
<div className="flex flex-wrap gap-1">
{(["push", "pull", "orbit", "pan"] as const).map((kind, i) => (
<button
type="button"
className={button}
key={kind}
onClick={() => {
const base =
(preview ? sampleDirectorMotion(motion, timeRef.current) : pose) || pose;
if (!validDirectorPose(base)) {
toast.error("相机不能与注视点重合");
return;
}
edit();
setTime(0);
onChange(presetDirectorMotion(base, kind, motion.duration));
}}
>
{["推进轨迹", "拉远轨迹", "环绕轨迹", "横移轨迹"][i]}
</button>
))}
</div>
<button
type="button"
className={button}
disabled={motion.keyframes.length < 2 && !hasObjectMotion}
onClick={() => {
edit();
onOverview();
}}
>
查看完整轨迹
</button>
<p className="text-xs text-white/50">
预设会替换当前轨迹;黄色线条为相机路径。编辑机位后,在指定时间记录关键帧。
</p>
<label className="block text-xs">
时间 {time.toFixed(2)} / {motion.duration.toFixed(2)} 秒
<input
aria-label="运镜时间轴"
className="w-full"
type="range"
min={0}
max={motion.duration}
step={0.01}
value={time}
onChange={(e) => {
setPlaying(false);
seek(Number(e.target.value));
}}
/>
</label>
<div className="flex flex-wrap gap-1">
<button
type="button"
className={button}
disabled={motion.keyframes.length < 2 && !hasObjectMotion}
onClick={() => {
if (playing) {
setPlaying(false);
return;
}
seek(time >= motion.duration ? 0 : time);
setPlaying(true);
}}
>
{playing ? "暂停运镜" : "播放运镜"}
</button>
<button
type="button"
className={button}
onClick={() => {
setPlaying(false);
seek(0);
}}
>
回到起点
</button>
<button type="button" className={button} onClick={edit}>
编辑机位
</button>
<label className="flex items-center gap-1 text-xs">
<input
type="checkbox"
aria-label="循环运镜"
checked={motion.loop}
onChange={(e) => onChange({ ...motion, loop: e.target.checked })}
/>
循环
</label>
</div>
<label className="flex items-center gap-2 text-xs">
<input
type="checkbox"
aria-label="自动记录关键帧"
checked={autoKeyframe}
onChange={(e) => setAutoKeyframe(e.target.checked)}
/>
编辑机位时自动记录当前帧
</label>
<div className="flex flex-wrap gap-1">
<button
type="button"
className={button}
disabled={!motion.keyframes.length}
onClick={() => {
edit();
onChange(transformDirectorMotion(motion, [1, 0, 0], 0));
}}
>
整条轨迹右移 1 米
</button>
<button
type="button"
className={button}
disabled={!motion.keyframes.length}
onClick={() => {
edit();
onChange(transformDirectorMotion(motion, [0, 0, 0], 90));
}}
>
轨迹与朝向旋转 90°
</button>
</div>
<button type="button" className={button} disabled={preview} onClick={add}>
记录/更新当前机位
</button>
{preview ? (
<p className="text-xs text-amber-200">正在预览轨迹;点击“编辑机位”后可调整并记录机位。</p>
) : null}
<div className="max-h-48 space-y-1 overflow-y-auto">
{motion.keyframes.map((k, i) => (
<div key={k.id} className="flex items-center gap-1 rounded bg-white/5 p-1">
<button
type="button"
className={button}
aria-label={`编辑关键帧 ${i + 1}`}
onClick={() => {
edit();
setTime(k.time);
onPose(k);
}}
>
帧 {i + 1}
</button>
<input
aria-label={`关键帧 ${i + 1} 时间`}
type="number"
step={0.1}
min={0}
max={motion.duration}
value={Number(k.time.toFixed(3))}
className="min-w-0 flex-1 rounded bg-white/10 p-1 text-xs"
onChange={(e) => {
const n = Number(e.target.value);
if (n < 0 || n > motion.duration || !Number.isFinite(n)) return;
if (
motion.keyframes.some(
(other) => other.id !== k.id && Math.abs(other.time - n) < 0.01,
)
) {
toast.error("关键帧时间不能重复");
return;
}
onChange({
...motion,
keyframes: motion.keyframes
.map((other) => (other.id === k.id ? { ...k, time: n } : other))
.sort((a, b) => a.time - b.time),
});
}}
/>
<button
type="button"
className={button}
aria-label={`删除关键帧 ${i + 1}`}
onClick={() => {
edit();
onChange({
...motion,
keyframes: motion.keyframes.filter((other) => other.id !== k.id),
});
}}
>
删除
</button>
</div>
))}
</div>
<div className="flex flex-wrap gap-1">
<button type="button" className={button} onClick={downloadJson}>
保存轨迹文件
</button>
<button type="button" className={button} onClick={() => input.current?.click()}>
导入轨迹文件
</button>
</div>
<button
type="button"
className={`${button} w-full`}
disabled={motion.keyframes.length < 2 && !hasObjectMotion}
onClick={() => {
edit();
onExport();
}}
>
导出真实运镜视频
</button>
<button
type="button"
className={`${button} w-full`}
disabled={motion.keyframes.length < 2}
onClick={() => {
edit();
onSend();
}}
>
发送首帧与运镜到视频节点
</button>
<p className="text-xs leading-5 text-white/50">
真实导出严格使用本场景轨迹,不含音频。AI
生视频使用首帧与运镜描述作参考,不保证精确复现轨迹。
</p>
</fieldset>
{exportProgress !== null ? (
<div role="status" className="text-xs">
正在导出 {exportProgress.toFixed(1)} / {motion.duration} 秒{" "}
<button className={button} onClick={onCancel}>
取消导出
</button>
</div>
) : null}
<input
ref={input}
type="file"
accept=".json,application/json"
className="hidden"
onChange={async (e) => {
const file = e.target.files?.[0];
e.target.value = "";
if (!file) return;
try {
if (file.size > 256 * 1024) throw Error("轨迹文件不能超过 256 KiB");
const data = JSON.parse(await file.text());
if (data.version !== 1) throw Error("不支持的轨迹文件版本");
const m = readDirectorMotion(data.motion);
if (m.keyframes.length < 2 || m.keyframes.length !== data.motion?.keyframes?.length)
throw Error("轨迹数据无效或包含重复时间");
edit();
onChange(m);
setTime(0);
toast.success("轨迹已导入");
} catch (e) {
toast.error(e instanceof Error ? e.message : "轨迹导入失败");
}
}}
/>
</section>
);
}

View File

@ -0,0 +1,256 @@
"use client";
import { useState } from "react";
import { toast } from "sonner";
import type { DirectorScene } from "@/lib/video-canvas/director-scene";
import { recordObjectFrames } from "@/lib/video-canvas/director-object-tracks";
export function DirectorObjectTimeline({
scene,
time,
selected,
disabled,
preview,
auto,
onAuto,
onChange,
onSeek,
onEdit,
onSelect,
}: {
scene: DirectorScene;
time: number;
selected: string | null;
disabled: boolean;
preview: boolean;
auto: boolean;
onAuto: (v: boolean) => void;
onChange: (s: DirectorScene) => void;
onSeek: (t: number) => void;
onEdit: () => void;
onSelect: (id: string) => void;
}) {
const [unit, setUnit] = useState("seconds"),
[zoom, setZoom] = useState(1);
const factor = unit === "frames" ? 30 : 1,
track = selected ? scene.objectTracks?.[selected] : undefined;
const button = "rounded border border-white/20 px-2 py-1 text-xs disabled:opacity-40",
field = "rounded bg-[#303030] p-1 text-xs";
return (
<fieldset
disabled={disabled}
className="space-y-3 border-t border-white/15 p-3"
aria-label="对象多轨时间轴"
>
<legend className="px-2 text-sm">对象多轨 · 与相机同步</legend>
<div className="flex flex-wrap items-center gap-3">
<label>
单位{" "}
<select
aria-label="时间轴单位"
value={unit}
className={field}
onChange={(e) => setUnit(e.target.value)}
>
<option value="seconds">秒</option>
<option value="frames">帧(30 fps)</option>
</select>
</label>
<label>
缩放{" "}
<select
aria-label="对象时间轴缩放"
value={zoom}
className={field}
onChange={(e) => setZoom(Number(e.target.value))}
>
{[1, 2, 4, 8].map((n) => (
<option key={n} value={n}>
{n}×
</option>
))}
</select>
</label>
<span>
{(time * factor).toFixed(unit === "frames" ? 0 : 2)} /{" "}
{(scene.motion.duration * factor).toFixed(unit === "frames" ? 0 : 2)}{" "}
{unit === "frames" ? "帧" : "秒"}
</span>
<label>
<input type="checkbox" checked={auto} onChange={(e) => onAuto(e.target.checked)} />{" "}
自动记录对象帧
</label>
<button type="button" className={button} onClick={onEdit}>
编辑当前时间对象
</button>
<button
type="button"
className={button}
disabled={preview || !selected}
onClick={() => {
try {
onChange(recordObjectFrames(scene, [selected!], time, () => crypto.randomUUID()));
} catch (e) {
toast.error((e as Error).message);
}
}}
>
记录所选对象帧
</button>
</div>
<div className="overflow-x-auto">
<div style={{ minWidth: `${zoom * 100}%` }}>
<input
aria-label="对象播放头"
type="range"
min={0}
max={scene.motion.duration}
step={1 / 30}
value={time}
className="w-full"
onChange={(e) => onSeek(Number(e.target.value))}
/>
{scene.objects.map((o) => {
const t = scene.objectTracks?.[o.id];
return (
<div
key={o.id}
className="grid grid-cols-[150px_60px_1fr] items-center gap-2 border-t border-white/10 py-2 text-xs"
>
<button
type="button"
className={`truncate text-left ${selected === o.id ? "text-amber-300" : ""}`}
onClick={() => onSelect(o.id)}
>
{o.name}
</button>
<label>
<input
type="checkbox"
aria-label={`${o.name}启用轨道`}
checked={!t?.muted}
onChange={(e) =>
onChange({
...scene,
objectTracks: {
...scene.objectTracks,
[o.id]: {
keyframes: t?.keyframes || [],
easing: t?.easing || "linear",
muted: !e.target.checked,
},
},
})
}
/>
启用
</label>
<div className="relative h-6 rounded bg-white/5">
{t?.keyframes.map((k, i) => (
<button
key={k.id}
type="button"
aria-label={`${o.name}关键帧${i + 1}`}
className="absolute top-0 -translate-x-1/2 text-lg text-amber-300"
style={{ left: `${(k.time / scene.motion.duration) * 100}%` }}
onClick={() => {
onSelect(o.id);
onSeek(k.time);
}}
>
◆
</button>
))}
</div>
</div>
);
})}
</div>
</div>
{track && selected ? (
<div className="space-y-2">
<label>
对象插值{" "}
<select
aria-label="对象插值"
value={track.easing}
className={field}
onChange={(e) =>
onChange({
...scene,
objectTracks: {
...scene.objectTracks,
[selected]: { ...track, easing: e.target.value as "smooth" | "linear" },
},
})
}
>
<option value="linear">匀速</option>
<option value="smooth">缓入缓出</option>
</select>
</label>
<div className="flex flex-wrap gap-2">
{track.keyframes.map((k, i) => (
<label className="flex items-center gap-1" key={k.id}>
帧 {i + 1}
<input
className={`${field} w-20`}
aria-label={`对象关键帧${i + 1}时间`}
type="number"
min={0}
max={scene.motion.duration * factor}
step={unit === "frames" ? 1 : 0.01}
value={Number((k.time * factor).toFixed(2))}
onChange={(e) => {
const next = Number(e.target.value) / factor;
if (
!Number.isFinite(next) ||
next < 0 ||
next > scene.motion.duration ||
track.keyframes.some(
(other) => other.id !== k.id && Math.abs(other.time - next) < 0.001,
)
)
return;
onChange({
...scene,
objectTracks: {
...scene.objectTracks,
[selected]: {
...track,
keyframes: track.keyframes
.map((old) => (old.id === k.id ? { ...old, time: next } : old))
.sort((a, b) => a.time - b.time),
},
},
});
}}
/>
<button
type="button"
className={button}
aria-label={`删除对象关键帧${i + 1}`}
onClick={() =>
onChange({
...scene,
objectTracks: {
...scene.objectTracks,
[selected]: {
...track,
keyframes: track.keyframes.filter((old) => old.id !== k.id),
},
},
})
}
>
删除
</button>
</label>
))}
</div>
</div>
) : null}
<p className="text-xs text-white/50">
拖动播放头预览全部对象;点击“编辑当前时间对象”后调整场景对象,再记录关键帧。真实视频导出使用相同轨道。
</p>
</fieldset>
);
}

View File

@ -0,0 +1,118 @@
"use client";
import { useState } from "react";
import { toast } from "sonner";
import { readDirectorScene, type DirectorScene } from "@/lib/video-canvas/director-scene";
import { insertReconstructedScene } from "@/lib/video-canvas/director-reconstruction";
export type ReconstructionTask = {
status: string;
error?: string | null;
result?: Record<string, unknown> | null;
};
export function DirectorReconstructionPanel({
scene,
onChange,
onSubmit,
task,
disabled,
}: {
scene: DirectorScene;
onChange: (s: DirectorScene) => void;
onSubmit: (input: { imageUrl: string; prompt: string }) => Promise<unknown>;
task?: ReconstructionTask;
disabled: boolean;
}) {
const [file, setFile] = useState<File | null>(null),
[prompt, setPrompt] = useState(""),
[busy, setBusy] = useState(false);
const waiting = task && ["pending", "running"].includes(task.status);
const result = task?.status === "succeeded" ? task.result : undefined;
return (
<fieldset
disabled={disabled || busy}
className="space-y-2 rounded border border-white/15 p-3 text-xs"
>
<legend>AI 识图重建布景</legend>
<p className="text-white/60">
从参考图识别人物、物体与机位,生成可编辑的简化几何布景;不包含精细网格和纹理。
</p>
<input
aria-label="重建参考图"
type="file"
accept="image/png,image/jpeg,image/webp"
onChange={(e) => setFile(e.target.files?.[0] || null)}
/>
<textarea
aria-label="重建要求"
className="w-full rounded bg-[#303030] p-2"
value={prompt}
onChange={(e) => setPrompt(e.target.value)}
placeholder="需要保留的主体、构图与大致尺寸"
maxLength={2000}
/>
<button
type="button"
disabled={!file || !!waiting}
className="rounded bg-amber-400 px-3 py-2 text-black disabled:opacity-40"
onClick={async () => {
if (!file) return;
if (file.size > 20 * 1024 * 1024) {
toast.error("参考图需小于 20 MiB");
return;
}
setBusy(true);
try {
const form = new FormData();
form.append("file", file);
form.append("purpose", "director-studio");
const response = await fetch("/api/files/upload", {
method: "POST",
body: form,
signal: AbortSignal.timeout(120000),
});
const data = await response.json();
const url = data.url || data.data?.url;
if (!response.ok || !url) throw Error(data.error || "参考图上传失败");
await onSubmit({ imageUrl: url, prompt });
toast.success("重建任务已提交,结果将保存为独立导演台节点");
} catch (e) {
toast.error(e instanceof Error ? e.message : "重建失败");
} finally {
setBusy(false);
}
}}
>
{busy ? "正在提交…" : waiting ? "正在重建…" : "识图重建"}
</button>
{task?.error ? <p role="alert">{task.error}</p> : null}
{result?.directorScene ? (
<div>
<p className="my-2">{String(result.summary || "布景已生成")}</p>
{[false, true].map((replace) => (
<button
type="button"
key={String(replace)}
className="mr-2 rounded border border-white/20 px-2 py-1"
onClick={() => {
try {
onChange(
insertReconstructedScene(
scene,
readDirectorScene(result.directorScene),
replace,
() => crypto.randomUUID(),
),
);
toast.success(replace ? "已替换场景对象" : "已插入场景对象");
} catch (e) {
toast.error((e as Error).message);
}
}}
>
{replace ? "替换场景对象" : "插入当前场景"}
</button>
))}
</div>
) : null}
</fieldset>
);
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,764 @@
"use client";
import { sampleDirectorObjects } from "@/lib/video-canvas/director-object-tracks";
import { resolveDirectorCamera, directorBodies } from "@/lib/video-canvas/director-extensions";
import { directorPoses } from "@/lib/video-canvas/director-poses";
import { forwardRef, useEffect, useImperativeHandle, useRef, useState } from "react";
import * as THREE from "three";
import { buildThreeAssetUrl } from "@/lib/canvas/three-asset-url";
import { OrbitControls } from "three/examples/jsm/controls/OrbitControls.js";
import { TransformControls } from "three/examples/jsm/controls/TransformControls.js";
import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
import {
directorImageSize,
type DirectorScene,
type DirectorObject,
} from "@/lib/video-canvas/director-scene";
import { sampleDirectorMotion, type DirectorPose } from "@/lib/video-canvas/director-motion";
import { recordDirectorVideo } from "@/lib/video-canvas/director-recording";
export type DirectorViewportHandle = {
showPath: () => void;
record: (signal: AbortSignal, onProgress: (time: number) => void) => Promise<Blob>;
capture: (
pose?: DirectorPose,
time?: number,
) => Promise<{ blob: Blob; width: number; height: number }>;
};
type Props = {
scene: DirectorScene;
selected: string;
mode: "translate" | "rotate" | "scale";
cameraView: boolean;
previewTime: number | null;
locked: boolean;
onTransform: (id: string, patch: Partial<DirectorObject>) => void;
onCamera: (position: [number, number, number], target: [number, number, number]) => void;
};
function dispose(root: THREE.Object3D) {
root.traverse((o) => {
if (o instanceof THREE.Mesh || o instanceof THREE.Line) {
o.geometry.dispose();
for (const m of Array.isArray(o.material) ? o.material : [o.material]) {
for (const value of Object.values(m)) if (value instanceof THREE.Texture) value.dispose();
m.dispose();
}
}
});
}
export const DirectorViewport = forwardRef<DirectorViewportHandle, Props>(
function DirectorViewport(props, ref) {
const host = useRef<HTMLDivElement>(null);
const latest = useRef(props);
latest.current = props;
const runtime = useRef<{
renderer: THREE.WebGLRenderer;
world: THREE.Scene;
camera: THREE.PerspectiveCamera;
orbit: OrbitControls;
transform: TransformControls;
helper: THREE.Object3D;
floor: THREE.Mesh;
grid: THREE.GridHelper;
path: THREE.Group;
objects: THREE.Group;
pending: number;
resize: () => void;
} | null>(null);
const [ready, setReady] = useState(false);
const [rendererRevision, setRendererRevision] = useState(0);
const [resourceRevision, setResourceRevision] = useState(0);
const [error, setError] = useState("");
const [loading, setLoading] = useState(false);
const [backgroundPending, setBackgroundPending] = useState(false);
const [backgroundError, setBackgroundError] = useState("");
useEffect(() => {
const r = runtime.current;
if (!r) return;
r.grid.visible = props.scene.gridVisible !== false;
r.floor.visible = props.scene.groundVisible !== false;
}, [props.scene.gridVisible, props.scene.groundVisible, ready]);
const applyObjectTime = (time: number | null) => {
const objects = sampleDirectorObjects(latest.current.scene, time),
r = runtime.current;
if (r)
for (const item of objects) {
const mesh = r.objects.children.find((o) => o.name === item.id);
if (mesh) {
mesh.position.fromArray(item.position);
mesh.rotation.set(...item.rotation);
mesh.scale.fromArray(item.scale);
mesh.visible = item.visible !== false;
}
}
return { ...latest.current.scene, objects };
};
useEffect(() => {
if (!host.current) return;
setError("");
let renderer: THREE.WebGLRenderer;
try {
renderer = new THREE.WebGLRenderer({
antialias: true,
preserveDrawingBuffer: true,
});
} catch {
setError("无法初始化 3D 视图,请检查浏览器 WebGL 支持");
return;
}
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
renderer.outputColorSpace = THREE.SRGBColorSpace;
renderer.setClearColor("#080808");
host.current.appendChild(renderer.domElement);
renderer.domElement.setAttribute("aria-label", "3D 导演台场景");
Object.assign(renderer.domElement.style, {
position: "absolute",
left: "50%",
top: "50%",
transform: "translate(-50%, -50%)",
});
const world = new THREE.Scene();
world.background = new THREE.Color(props.scene.skyColor || "#080808");
world.add(new THREE.HemisphereLight(0xffffff, 0x647080, 2.4));
const light = new THREE.DirectionalLight(0xffffff, 3);
light.position.set(4, 8, 5);
world.add(light);
const floor = new THREE.Mesh(
new THREE.PlaneGeometry(100, 100),
new THREE.MeshBasicMaterial({ color: 0x0b0f15 }),
);
floor.rotation.x = -Math.PI / 2;
floor.position.y = -0.02;
world.add(floor);
const grid = new THREE.GridHelper(30, 30, 0x21445b, 0x183346);
world.add(grid);
const path = new THREE.Group();
world.add(path);
const camera = new THREE.PerspectiveCamera(50, 1, 0.05, 500);
camera.position.set(4, 3, 7);
const orbit = new OrbitControls(camera, renderer.domElement);
orbit.target.set(0, 1, 0);
orbit.minDistance = 0.3;
orbit.maxDistance = 80;
const transform = new TransformControls(camera, renderer.domElement);
const helper = transform.getHelper();
world.add(helper);
const objects = new THREE.Group();
world.add(objects);
transform.addEventListener("dragging-changed", (e) => {
orbit.enabled =
!e.value && !latest.current.cameraView && latest.current.previewTime === null;
});
transform.addEventListener("mouseUp", () => {
const o = transform.object;
if (o)
latest.current.onTransform(o.name, {
position: o.position.toArray(),
rotation: [o.rotation.x, o.rotation.y, o.rotation.z],
scale: o.scale.toArray(),
});
});
orbit.addEventListener("end", () => {
if (latest.current.previewTime !== null || latest.current.cameraView) return;
latest.current.onCamera(camera.position.toArray(), orbit.target.toArray());
});
const resize = () => {
if (!host.current) return;
const { width, height } = host.current.getBoundingClientRect();
const size = directorImageSize(latest.current.scene.ratio);
const aspect = size.width / size.height;
const fitWidth = Math.max(1, Math.min(width, height * aspect));
renderer.setSize(fitWidth, fitWidth / aspect);
camera.aspect = aspect;
camera.updateProjectionMatrix();
};
const state = {
renderer,
world,
camera,
orbit,
transform,
helper,
floor,
grid,
path,
objects,
pending: 0,
resize,
};
runtime.current = state;
const observer = new ResizeObserver(resize);
observer.observe(host.current);
resize();
renderer.setAnimationLoop(() => {
if (orbit.enabled) orbit.update();
renderer.render(world, camera);
});
const contextLost = (event: Event) => {
event.preventDefault();
renderer.setAnimationLoop(null);
setReady(false);
setError("3D 显示连接已中断,请点击重试恢复场景");
};
const contextRestored = () => setRendererRevision((value) => value + 1);
renderer.domElement.addEventListener("webglcontextlost", contextLost);
renderer.domElement.addEventListener("webglcontextrestored", contextRestored);
setReady(true);
return () => {
renderer.domElement.removeEventListener("webglcontextlost", contextLost);
renderer.domElement.removeEventListener("webglcontextrestored", contextRestored);
runtime.current = null;
observer.disconnect();
renderer.setAnimationLoop(null);
transform.dispose();
orbit.dispose();
dispose(world);
renderer.dispose();
renderer.domElement.remove();
};
}, [rendererRevision]);
useEffect(() => {
const r = runtime.current;
if (!r || !ready) return;
let active = true;
const timers: ReturnType<typeof setTimeout>[] = [];
r.transform.detach();
const desired = new Set(props.scene.objects.map((o) => o.id));
r.objects.children.slice().forEach((o) => {
if (!desired.has(o.name)) {
dispose(o);
r.objects.remove(o);
}
});
setError("");
r.pending = 0;
const attach = (object: THREE.Object3D, item: DirectorObject) => {
if (!active) {
dispose(object);
return;
}
object.name = item.id;
// A GLB may finish loading after playback has paused at a nonzero time.
const sampled =
latest.current.previewTime === null
? item
: sampleDirectorObjects(latest.current.scene, latest.current.previewTime).find(
(o) => o.id === item.id,
) || item;
object.visible = sampled.visible !== false;
object.userData.directorSignature = JSON.stringify([
item.kind,
item.url,
item.pose,
item.color,
item.body,
]);
object.position.fromArray(sampled.position);
object.rotation.set(...sampled.rotation);
object.scale.fromArray(sampled.scale);
r.objects.add(object);
if (
item.id === latest.current.selected &&
!item.locked &&
item.visible !== false &&
!latest.current.cameraView &&
latest.current.previewTime === null
)
r.transform.attach(object);
};
for (const item of props.scene.objects) {
const existing = r.objects.children.find((o) => o.name === item.id);
if (
existing &&
existing.userData.directorSignature ===
JSON.stringify([item.kind, item.url, item.pose, item.color, item.body])
) {
attach(existing, item);
continue;
}
if (existing) {
dispose(existing);
r.objects.remove(existing);
}
if (item.kind === "model" && item.url) {
r.pending++;
let settled = false;
const finish = (message?: string) => {
if (!active || settled) return false;
settled = true;
clearTimeout(timeout);
r.pending--;
setLoading(r.pending > 0);
if (message) setError(`${item.name}:${message}`);
return true;
};
const timeout = setTimeout(() => finish("模型加载超过 30 秒,请检查网络后重试"), 30000);
timers.push(timeout);
new GLTFLoader().load(
buildThreeAssetUrl(item.url),
(g) => {
if (!active || settled) {
dispose(g.scene);
return;
}
try {
const box = new THREE.Box3().setFromObject(g.scene);
const size = box.getSize(new THREE.Vector3());
if (
box.isEmpty() ||
!size.toArray().every(Number.isFinite) ||
size.length() < 0.000001
) {
throw new Error("模型没有可显示的有效几何内容");
}
const scale = 2 / Math.max(size.x, size.y, size.z, 0.001);
g.scene.scale.setScalar(scale);
const normalized = new THREE.Group();
normalized.add(g.scene);
const fitted = new THREE.Box3().setFromObject(normalized);
const center = fitted.getCenter(new THREE.Vector3());
g.scene.position.set(-center.x, -fitted.min.y, -center.z);
attach(normalized, item);
finish();
} catch (failure) {
dispose(g.scene);
finish(failure instanceof Error ? failure.message : "模型内容读取失败");
}
},
undefined,
() => finish("模型加载失败,请检查 GLB 文件及资源访问权限"),
);
continue;
}
const mat = new THREE.MeshStandardMaterial({
color: item.color || (item.kind === "actor" ? "#687bb0" : "#c3664c"),
roughness: 0.6,
});
let object: THREE.Object3D;
if (item.kind === "actor") {
const group = new THREE.Group();
const pose = directorPoses[item.pose || "站立"] || directorPoses["站立"];
const radians = THREE.MathUtils.degToRad;
const hips = new THREE.Group();
hips.position.y = 1 + ("height" in pose ? pose.height : 0);
group.add(hips);
const torso = new THREE.Group();
torso.rotation.x = radians(pose.lean);
hips.add(torso);
const body = new THREE.Mesh(new THREE.BoxGeometry(0.52, 0.65, 0.28), mat);
body.position.y = 0.38;
torso.add(body);
const head = new THREE.Mesh(new THREE.SphereGeometry(0.2, 20, 16), mat.clone());
head.position.y = 0.95;
torso.add(head);
const limb = (
parent: THREE.Object3D,
position: [number, number, number],
angle: number,
bend: number,
length: number,
radius: number,
spread = 0,
) => {
const root = new THREE.Group();
root.position.set(...position);
root.rotation.x = radians(angle);
root.rotation.z = radians(spread);
parent.add(root);
const upper = new THREE.Mesh(
new THREE.CapsuleGeometry(radius, length - radius * 2, 4, 12),
mat.clone(),
);
upper.position.y = -length / 2;
root.add(upper);
const joint = new THREE.Group();
joint.position.y = -length;
joint.rotation.x = radians(bend);
root.add(joint);
const lower = new THREE.Mesh(
new THREE.CapsuleGeometry(radius * 0.85, length - radius * 2, 4, 12),
mat.clone(),
);
lower.position.y = -length / 2;
joint.add(lower);
};
for (let i = 0; i < 2; i++) {
const side = i === 0 ? -1 : 1;
limb(
torso,
[side * 0.33, 0.65, 0],
pose.arms[i],
-pose.elbows[i],
0.34,
0.07,
("spread" in pose ? pose.spread : 5) * side,
);
limb(hips, [side * 0.15, 0, 0], pose.legs[i], pose.knees[i], 0.46, 0.1);
}
const [width, height, headScale] =
directorBodies[item.body || "标准"] || directorBodies["标准"];
group.scale.set(width, height, width);
head.scale.setScalar(headScale);
const root = new THREE.Group();
root.add(group);
object = root;
} else {
object = new THREE.Mesh(
item.kind === "sphere"
? new THREE.SphereGeometry(0.5, 32, 20)
: new THREE.BoxGeometry(1, 1, 1),
mat,
);
}
attach(object, item);
}
setLoading(r.pending > 0);
return () => {
active = false;
timers.forEach(clearTimeout);
};
}, [props.scene.objects, ready, resourceRevision]);
useEffect(() => {
const r = runtime.current;
if (!r) return;
let active = true;
let texture: THREE.Texture | undefined;
setBackgroundError("");
r.world.background = new THREE.Color(props.scene.skyColor || "#080808");
if (!props.scene.backgroundUrl) {
setBackgroundPending(false);
return;
}
setBackgroundPending(true);
const timeout = setTimeout(() => {
if (active) {
active = false;
setBackgroundPending(false);
setBackgroundError("全景背景加载超时,请重新选择背景");
}
}, 20000);
new THREE.TextureLoader().load(
buildThreeAssetUrl(props.scene.backgroundUrl),
(loaded) => {
clearTimeout(timeout);
if (!active) {
loaded.dispose();
return;
}
texture = loaded;
loaded.mapping = THREE.EquirectangularReflectionMapping;
loaded.colorSpace = THREE.SRGBColorSpace;
r.world.background = loaded;
setBackgroundPending(false);
},
undefined,
() => {
clearTimeout(timeout);
if (active) {
setBackgroundPending(false);
setBackgroundError("全景背景加载失败,请检查图片或切回默认背景");
}
},
);
return () => {
active = false;
clearTimeout(timeout);
if (r.world.background === texture) r.world.background = new THREE.Color(latest.current.scene.skyColor || "#080808");
texture?.dispose();
};
}, [props.scene.backgroundUrl, props.scene.skyColor, ready, resourceRevision]);
useEffect(() => {
const r = runtime.current;
if (!r) return;
if (props.locked) {
r.orbit.enabled = false;
r.transform.detach();
return;
}
const camera = resolveDirectorCamera(props.scene);
if (props.previewTime === null) {
r.camera.position.fromArray(camera.position);
r.camera.fov = camera.fov;
r.camera.updateProjectionMatrix();
r.orbit.target.fromArray(camera.target);
r.orbit.update();
}
r.orbit.enabled = !props.cameraView && props.previewTime === null;
r.transform.setMode(props.mode);
const selected = r.objects.getObjectByName(props.selected);
if (
selected &&
selected.visible &&
!props.scene.objects.find((o) => o.id === props.selected)?.locked &&
!props.cameraView &&
props.previewTime === null
)
r.transform.attach(selected);
else r.transform.detach();
}, [
props.scene.objects,
props.scene.cameras,
props.scene.activeCameraId,
props.selected,
props.mode,
props.cameraView,
props.previewTime === null,
props.locked,
ready,
]);
useEffect(() => {
const r = runtime.current;
if (!r) return;
dispose(r.path);
r.path.clear();
const motion = props.scene.motion;
const tracks = [
...(motion.keyframes.length >= 2
? [
{
color: 0xfbbf24,
keys: motion.keyframes,
points: Array.from(
{ length: 121 },
(_, i) =>
new THREE.Vector3(
...sampleDirectorMotion(motion, (motion.duration * i) / 120)!.position,
),
),
},
]
: []),
...Object.entries(props.scene.objectTracks || {})
.filter(([, t]) => !t.muted && t.keyframes.length >= 2)
.map(([id, t]) => ({
color: 0x67e8f9,
keys: t.keyframes,
points: Array.from(
{ length: 121 },
(_, i) =>
new THREE.Vector3(
...sampleDirectorObjects(props.scene, (motion.duration * i) / 120).find(
(o) => o.id === id,
)!.position,
),
),
})),
];
for (const track of tracks) {
r.path.add(
new THREE.Line(
new THREE.BufferGeometry().setFromPoints(track.points),
new THREE.LineBasicMaterial({ color: track.color }),
),
);
for (const k of track.keys) {
const dot = new THREE.Mesh(
new THREE.SphereGeometry(0.08, 10, 8),
new THREE.MeshBasicMaterial({ color: track.color }),
);
dot.position.fromArray(k.position);
r.path.add(dot);
}
}
}, [props.scene.motion, props.scene.objectTracks, props.scene.objects, ready]);
useEffect(() => {
const r = runtime.current;
if (!r) return;
r.path.visible = !props.locked && !props.cameraView && props.previewTime === null;
if (props.locked) return;
const sampledScene = applyObjectTime(props.previewTime);
if (props.previewTime === null) return;
const sampled = sampleDirectorMotion(props.scene.motion, props.previewTime);
const pose = resolveDirectorCamera(sampledScene, sampled || undefined);
r.camera.position.fromArray(pose.position);
r.orbit.target.fromArray(pose.target);
r.camera.fov = pose.fov;
r.camera.updateProjectionMatrix();
r.camera.lookAt(r.orbit.target);
r.transform.detach();
}, [
props.previewTime,
props.scene.motion,
props.scene.cameras,
props.scene.objects,
props.scene.objectTracks,
props.cameraView,
props.locked,
ready,
]);
useEffect(() => {
runtime.current?.resize();
}, [props.scene.ratio, ready]);
useImperativeHandle(
ref,
() => ({
showPath: () => {
const r = runtime.current;
if (!r) return;
const points = latest.current.scene.motion.keyframes.map(
(k) => new THREE.Vector3(...k.position),
);
for (const track of Object.values(latest.current.scene.objectTracks || {}))
if (!track.muted)
for (const key of track.keyframes) points.push(new THREE.Vector3(...key.position));
if (!points.length) return;
points.push(new THREE.Vector3(0, 1, 0));
const box = new THREE.Box3().setFromPoints(points),
center = box.getCenter(new THREE.Vector3());
const radius = Math.max(3, box.getSize(new THREE.Vector3()).length());
const position = center
.clone()
.add(new THREE.Vector3(1, 0.8, 1).normalize().multiplyScalar(radius * 1.4));
latest.current.onCamera(position.toArray(), center.toArray());
},
record: async (signal, onProgress) => {
if (backgroundPending || backgroundError)
throw new Error(backgroundError || "全景背景仍在加载,请稍后导出");
const r = runtime.current;
if (!r || !ready || r.pending || error) throw new Error(error || "3D 场景尚未准备好");
if (
latest.current.scene.motion.keyframes.length < 2 &&
!Object.values(latest.current.scene.objectTracks || {}).some(
(t) => !t.muted && t.keyframes.length >= 2,
)
)
throw new Error("相机或对象轨道至少需要两个关键帧");
const motion = latest.current.scene.motion;
const size = r.renderer.getSize(new THREE.Vector2()),
pixelRatio = r.renderer.getPixelRatio();
const position = r.camera.position.clone(),
target = r.orbit.target.clone(),
fov = r.camera.fov,
aspect = r.camera.aspect;
const visibility = [r.helper.visible, r.grid.visible, r.path.visible],
enabled = r.orbit.enabled;
const dimensions = directorImageSize(latest.current.scene.ratio);
r.helper.visible = false;
r.grid.visible = false;
r.path.visible = false;
r.orbit.enabled = false;
r.renderer.setPixelRatio(1);
r.renderer.setSize(dimensions.width, dimensions.height, false);
r.camera.aspect = dimensions.width / dimensions.height;
try {
return await recordDirectorVideo({
canvas: r.renderer.domElement,
duration: motion.duration,
signal,
onProgress,
render: (time) => {
if (r.renderer.getContext().isContextLost())
throw new Error("3D 显示连接中断,已取消视频导出");
const pose = resolveDirectorCamera(
applyObjectTime(time),
sampleDirectorMotion(motion, time) || undefined,
);
r.camera.position.fromArray(pose.position);
r.orbit.target.fromArray(pose.target);
r.camera.fov = pose.fov;
r.camera.updateProjectionMatrix();
r.camera.lookAt(r.orbit.target);
r.renderer.render(r.world, r.camera);
},
});
} finally {
if (runtime.current === r) {
applyObjectTime(latest.current.previewTime);
r.renderer.setPixelRatio(pixelRatio);
r.renderer.setSize(size.x, size.y);
r.camera.position.copy(position);
r.orbit.target.copy(target);
r.camera.fov = fov;
r.camera.aspect = aspect;
r.camera.updateProjectionMatrix();
r.orbit.enabled = enabled;
r.orbit.update();
[r.helper.visible, r.grid.visible, r.path.visible] = visibility;
}
}
},
capture: async (pose, time) => {
if (backgroundPending || backgroundError)
throw new Error(backgroundError || "全景背景仍在加载,请稍后截图");
const r = runtime.current;
if (!r || !ready || r.pending || error) throw new Error(error || "3D 场景尚未准备好");
const { width, height } = directorImageSize(latest.current.scene.ratio);
const previousSize = r.renderer.getSize(new THREE.Vector2());
const pixelRatio = r.renderer.getPixelRatio();
const aspect = r.camera.aspect;
const position = r.camera.position.clone(),
target = r.orbit.target.clone(),
fov = r.camera.fov;
const visibility = [r.helper.visible, r.grid.visible, r.path.visible];
const enabled = r.orbit.enabled;
r.orbit.enabled = false;
const capturedScene = applyObjectTime(time ?? latest.current.previewTime);
if (pose) {
const resolvedPose = resolveDirectorCamera(capturedScene, pose);
r.camera.position.fromArray(resolvedPose.position);
r.orbit.target.fromArray(resolvedPose.target);
r.camera.fov = resolvedPose.fov;
r.camera.lookAt(r.orbit.target);
}
r.path.visible = false;
r.helper.visible = false;
r.grid.visible = false;
r.renderer.setPixelRatio(1);
r.renderer.setSize(width, height, false);
r.camera.aspect = width / height;
r.camera.updateProjectionMatrix();
r.renderer.render(r.world, r.camera);
try {
const blob = await new Promise<Blob>((resolve, reject) =>
r.renderer.domElement.toBlob(
(b) => (b ? resolve(b) : reject(new Error("截图失败"))),
"image/png",
),
);
return { blob, width, height };
} finally {
applyObjectTime(latest.current.previewTime);
r.renderer.setPixelRatio(pixelRatio);
r.renderer.setSize(previousSize.x, previousSize.y);
r.camera.aspect = aspect;
r.camera.updateProjectionMatrix();
r.camera.position.copy(position);
r.orbit.target.copy(target);
r.camera.fov = fov;
r.camera.updateProjectionMatrix();
r.orbit.update();
[r.helper.visible, r.grid.visible, r.path.visible] = visibility;
r.orbit.enabled = enabled;
}
},
}),
[ready, error, backgroundPending, backgroundError],
);
return (
<div className="relative h-full w-full" data-testid="director-viewport">
<div ref={host} className="h-full w-full" />
{error || backgroundError ? (
<div
role="alert"
className="absolute inset-x-4 top-4 rounded bg-red-950/90 p-3 text-sm text-white"
>
{error || backgroundError}
<button
type="button"
className="ml-3 rounded border border-white/30 px-2 py-1"
onClick={() => {
if (ready) setResourceRevision((value) => value + 1);
else setRendererRevision((value) => value + 1);
}}
>
重试 3D 加载
</button>
</div>
) : loading || backgroundPending ? (
<div className="absolute left-4 top-4 rounded bg-black/60 p-2 text-white">
{backgroundPending ? "正在加载全景背景…" : "正在加载模型…"}
</div>
) : null}
</div>
);
},
);

View File

@ -0,0 +1,31 @@
"use client";
import { useThree } from "@react-three/fiber";
import { useLayoutEffect } from "react";
import { PMREMGenerator } from "three";
import { RoomEnvironment } from "three/addons/environments/RoomEnvironment.js";
// Generate studio reflections locally: an unavailable HDR CDN must not hide GLB models.
export function LocalStudioEnvironment() {
const { gl, scene } = useThree();
useLayoutEffect(() => {
const room = new RoomEnvironment();
const generator = new PMREMGenerator(gl);
const previous = scene.environment;
let target: ReturnType<PMREMGenerator["fromScene"]> | undefined;
try {
target = generator.fromScene(room, 0.04);
scene.environment = target.texture;
} catch {
// Direct lights still render the model on devices unable to build a PMREM.
} finally {
room.dispose();
generator.dispose();
}
return () => {
if (scene.environment === target?.texture) scene.environment = previous;
target?.dispose();
};
}, [gl, scene]);
return null;
}

View File

@ -16,6 +16,8 @@ import type { Mesh, Object3D } from "three";
import { KTX2Loader } from "three-stdlib";
import { Button } from "@/components/ui/button";
import { buildThreeAssetUrl } from "@/lib/canvas/three-asset-url";
import { LocalStudioEnvironment } from "./LocalStudioEnvironment";
import { ModelLoadingFallback } from "./ModelLoadingFallback";
export type Model3DLayer = {
id: string;
@ -53,6 +55,9 @@ export type Model3DViewerSettings = {
type Model3DViewerProps = {
modelUrl?: string | null;
modelFormat?: "glb" | "gltf" | null;
autoFitModel?: boolean;
localStudioEnvironment?: boolean;
previewImageUrl?: string | null;
emptyStateTitle?: string;
settings: Model3DViewerSettings;
@ -323,9 +328,14 @@ function cloneSceneWithIndependentMaterials(scene: Object3D) {
function resolveSceneGroundY(
scene: Object3D,
modelRotation?: [number, number, number]
modelRotation?: [number, number, number],
fitTransform?: { position: [number, number, number]; scale: number }
) {
const probeScene = scene.clone(true);
if (fitTransform) {
probeScene.position.set(...fitTransform.position);
probeScene.scale.setScalar(fitTransform.scale);
}
const probeGroup = new THREE.Group();
if (modelRotation) {
probeGroup.rotation.set(...modelRotation);
@ -338,6 +348,26 @@ function resolveSceneGroundY(
return bounds.min.y;
}
function getModelFitTransform(scene: Object3D) {
const bounds = new THREE.Box3().setFromObject(scene);
if (bounds.isEmpty()) return { position: [0, 0, 0] as [number, number, number], scale: 1 };
const size = bounds.getSize(new THREE.Vector3());
const center = bounds.getCenter(new THREE.Vector3());
const largestDimension = Math.max(size.x, size.y, size.z);
if (!Number.isFinite(largestDimension) || largestDimension <= 0) {
return { position: [0, 0, 0] as [number, number, number], scale: 1 };
}
const scale = 2 / largestDimension;
return {
position: [
-center.x * scale,
-center.y * scale,
-center.z * scale,
] as [number, number, number],
scale,
};
}
function ModelScene({
url,
settings,
@ -346,6 +376,7 @@ function ModelScene({
onLayersChange,
onStatsChange,
onGroundYChange,
autoFitModel,
}: {
url: string;
settings: Model3DViewerSettings;
@ -354,6 +385,7 @@ function ModelScene({
onLayersChange?: (layers: Model3DLayer[]) => void;
onStatsChange?: (stats: Model3DStats) => void;
onGroundYChange?: (groundY: number) => void;
autoFitModel?: boolean;
}) {
const { gl } = useThree();
const ktx2Loader = useMemo(() => {
@ -382,11 +414,18 @@ function ModelScene({
[model.scene]
);
const meta = useMemo(() => collectSceneMeta(scene), [scene]);
const fitTransform = useMemo(
() =>
autoFitModel
? getModelFitTransform(scene)
: { position: [0, 0, 0] as [number, number, number], scale: 1 },
[autoFitModel, scene]
);
const modelRotationKey = modelRotation?.join("|") || "";
const groundY = useMemo(
() => resolveSceneGroundY(scene, modelRotation),
() => resolveSceneGroundY(scene, modelRotation, fitTransform),
// eslint-disable-next-line react-hooks/exhaustive-deps
[scene, modelRotationKey]
[scene, modelRotationKey, fitTransform]
);
useEffect(() => {
@ -437,7 +476,11 @@ function ModelScene({
return (
<group rotation={modelRotation || [0, 0, 0]}>
<primitive object={scene} scale={1} />
<primitive
object={scene}
position={fitTransform.position}
scale={fitTransform.scale}
/>
</group>
);
}
@ -679,6 +722,9 @@ function EmptyState({
export function Model3DViewer({
modelUrl,
modelFormat,
autoFitModel = false,
localStudioEnvironment = false,
previewImageUrl,
emptyStateTitle = "准备生成 3D 资产",
settings,
@ -715,7 +761,7 @@ export function Model3DViewer({
);
}
if (!isGlbOrGltf(modelUrl)) {
if (!isGlbOrGltf(modelUrl) && modelFormat !== "glb" && modelFormat !== "gltf") {
return (
<div className="flex h-full w-full flex-col items-center justify-center gap-3 bg-zinc-950 p-4 text-zinc-400">
<AlertCircle className="h-8 w-8 text-yellow-500/50" />
@ -737,9 +783,17 @@ export function Model3DViewer({
return (
<ViewerErrorBoundary
fallback={
<div className="flex h-full w-full flex-col items-center justify-center gap-3 bg-zinc-950 p-4 text-zinc-400">
<AlertCircle className="h-8 w-8 text-red-500/50" />
<p className="text-sm">3D 预览加载失败,请尝试在新窗口打开</p>
<div className="relative flex h-full w-full flex-col items-center justify-center gap-3 overflow-hidden bg-zinc-950 p-4 text-zinc-300">
{previewImageUrl ? (
<img
alt="3D 模型静态预览"
className="absolute inset-0 h-full w-full object-contain opacity-80"
src={previewImageUrl}
/>
) : null}
<div className="relative flex max-w-xs flex-col items-center gap-3 rounded-lg bg-black/70 px-4 py-3 text-center backdrop-blur-sm">
<AlertCircle className="h-7 w-7 text-amber-300" />
<p className="text-sm">{previewImageUrl ? "3D 预览加载失败,可查看静态预览或下载模型" : "3D 预览加载失败,请重试或下载模型"}</p>
<Button
asChild
className="gap-2 border-white/10 text-zinc-300 hover:bg-white/5"
@ -750,6 +804,7 @@ export function Model3DViewer({
打开3D文件
</a>
</Button>
</div>
</div>
}
>
@ -800,8 +855,10 @@ export function Model3DViewer({
/>
) : null}
{localStudioEnvironment ? <LocalStudioEnvironment /> : null}
<Suspense
fallback={
localStudioEnvironment ? <ModelLoadingFallback /> :
<mesh>
<boxGeometry args={[0.5, 0.5, 0.5]} />
<meshStandardMaterial color="#27272a" wireframe />
@ -810,6 +867,7 @@ export function Model3DViewer({
>
<group position={[0, 0, 0]}>
<ModelScene
autoFitModel={autoFitModel}
layerVisibility={layerVisibility}
modelRotation={modelRotation}
onGroundYChange={handleGroundYChange}
@ -828,8 +886,12 @@ export function Model3DViewer({
scale={10}
/>
</group>
<Environment preset={settings.envPreset} />
</Suspense>
{!localStudioEnvironment ? (
<Suspense fallback={null}>
<Environment preset={settings.envPreset} />
</Suspense>
) : null}
<OrbitControls
autoRotate={settings.autoRotate}
dampingFactor={0.08}

View File

@ -0,0 +1,21 @@
"use client";
import { Html } from "@react-three/drei";
import { Loader2 } from "lucide-react";
import { useEffect, useState } from "react";
export function ModelLoadingFallback() {
const [slow, setSlow] = useState(false);
useEffect(() => {
const timer = setTimeout(() => setSlow(true), 15000);
return () => clearTimeout(timer);
}, []);
return (
<Html center>
<div role="status" className="flex w-60 flex-col items-center gap-2 rounded-lg bg-zinc-900/90 px-4 py-3 text-center text-xs text-white">
<Loader2 className="h-5 w-5 animate-spin" />
<p>{slow ? "模型加载较慢,可退出预览后重试,或下载模型查看" : "正在加载 3D 模型…"}</p>
</div>
</Html>
);
}

View File

@ -13,11 +13,13 @@ import {
Download,
Eraser,
Expand,
FileImage,
FlipHorizontal2,
FlipVertical2,
Globe2,
ImagePlus,
Layers,
LayoutTemplate,
Loader2,
Link2,
Maximize2,
@ -44,8 +46,13 @@ import { useParams, useRouter } from "next/navigation";
import { Fragment, useCallback, useEffect, useMemo, useRef, useState } from "react";
import { createPortal } from "react-dom";
import { toast } from "sonner";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
import useSWR from "swr";
import { AgentExecutionSnapshotDialog } from "@/components/agents/AgentExecutionSnapshotDialog";
import { CanvasLayoutReplaceBatchDialog } from "@/components/canvas/CanvasLayoutReplaceBatchDialog";
import { TextEditColorField } from "@/components/canvas/TextEditColorField";
import { DetailSurfaceOverlayRenderer } from "@/components/canvas/DetailSurfaceOverlayRenderer";
import { DetailUiDomRenderer } from "@/components/canvas/DetailUiDomRenderer";
@ -135,6 +142,7 @@ import { Button } from "@/components/ui/button";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
@ -166,6 +174,7 @@ import {
import { fetchUploadWithRetry } from "@/lib/client/upload-with-retry";
import {
downloadCanvasImageAsPng,
getCanvasImageDownloadErrorMessage,
type CanvasImageDownloadProgress,
} from "@/lib/client/canvas-image-download";
import type { DetailUiDslV2Document } from "@/lib/detail-ui-dsl-v2";
@ -648,7 +657,7 @@ const normalizeTextEditItems = (
.filter((item): item is ImageTextEditItem => !!item);
};
const CHAT_ATTACHMENT_MAX_UPLOAD_BYTES = 30 * 1024 * 1024;
const CHAT_ATTACHMENT_MAX_UPLOAD_BYTES = IMAGE_UPLOAD_MAX_BYTES;
const CHAT_ATTACHMENT_MAX_DATA_URL_LENGTH = 42 * 1024 * 1024;
const CANVAS_3D_POLL_INTERVAL_MS = 2000;
const CANVAS_3D_MAX_POLL_ATTEMPTS = 360;
@ -834,7 +843,7 @@ async function uploadChatAttachmentBlob(blob: Blob, displayName: string) {
}
if (uploadBlob.size > CHAT_ATTACHMENT_MAX_UPLOAD_BYTES) {
throw new Error("图片过大,请先压缩后再添加到对话");
throw new Error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
}
const formData = new FormData();
@ -1351,6 +1360,10 @@ export function ImageNode({
);
const imageBoxCutoutItem = getCanvasToolbarItem(canvasToolbarConfig, "imageBoxCutout");
const imageLocalEditItem = getCanvasToolbarItem(canvasToolbarConfig, "imageLocalEdit");
const imageLayoutReplaceItem = getCanvasToolbarItem(
canvasToolbarConfig,
"imageLayoutReplace"
);
const imageRemoveWatermarkItem = getCanvasToolbarItem(
canvasToolbarConfig,
"imageRemoveWatermark"
@ -1423,6 +1436,8 @@ export function ImageNode({
// 局部编辑是无限画布图片的核心入口:即使旧后台工具栏配置里还没有该 key,
// 也必须默认展示,避免历史配置把新功能吞掉。
const imageLocalEditEnabled = true;
// 排版替换是无限画布图片的核心 AI 入口;旧配置缺少该 key 时也保持可见。
const imageLayoutReplaceEnabled = true;
const imageLayerSplitEnabledByToolbar = isCanvasToolbarItemEnabled(
canvasToolbarConfig,
"imageLayerSplit"
@ -1488,6 +1503,7 @@ export function ImageNode({
{ key: "textEdit", enabled: imageTextEditEnabled },
{ key: "boxCutout", enabled: imageBoxCutoutEnabled },
{ key: "localEdit", enabled: imageLocalEditEnabled },
{ key: "layoutReplace", enabled: imageLayoutReplaceEnabled },
{ key: "removeWatermark", enabled: imageRemoveWatermarkEnabled },
{ key: "resize", enabled: imageResizeEnabled },
{ key: "multiAngle", enabled: imageMultiAngleEnabled },
@ -1523,6 +1539,7 @@ export function ImageNode({
imageRemoveBackgroundEnabled ||
imageRemoveWatermarkEnabled ||
imageEnhanceEnabled ||
imageLayoutReplaceEnabled ||
imageNoteEnabled ||
hasCanvasAgentAnnotations ||
(isLayerSplitEnabled && imageLayerSplitEnabledByToolbar) ||
@ -1681,6 +1698,7 @@ export function ImageNode({
DEFAULT_CANVAS_WORLD_MODEL || "world-standard"
);
const [regeneratePrompt, setRegeneratePrompt] = useState("");
const [layoutReplaceDialogOpen, setLayoutReplaceDialogOpen] = useState(false);
const [isUploading, setIsUploading] = useState(false);
const [annotatorOpen, setAnnotatorOpen] = useState(false);
const [fullscreenPreviewOpen, setFullscreenPreviewOpen] = useState(false);
@ -6053,6 +6071,21 @@ export function ImageNode({
await detectTextItems(true);
}, [data.src, data.textEditItems, data.textEditSource, detectTextItems]);
const handleOpenLayoutReplace = useCallback(() => {
setCropPanelOpen(false);
setCropInteraction(null);
setImageAdjustPanelOpen(false);
setFlipRotatePanelOpen(false);
setAnglePanelOpen(false);
setTextEditPanelOpen(false);
setNotePanelOpen(false);
setPromptDialogOpen(false);
setAiWashQuickPanelOpen(false);
resetBoxCutoutState();
resetLocalEditState();
setLayoutReplaceDialogOpen(true);
}, [resetBoxCutoutState, resetLocalEditState]);
useEffect(() => {
if (!imageNodeActionRequest || imageNodeActionRequest.nodeId !== node.id) {
return;
@ -6072,11 +6105,17 @@ export function ImageNode({
if (imageNodeActionRequest.action === "local-edit") {
handleEnterLocalEditMode();
return;
}
if (imageNodeActionRequest.action === "layout-replace") {
handleOpenLayoutReplace();
}
}, [
clearImageNodeActionRequest,
handleEnterBoxCutoutMode,
handleEnterLocalEditMode,
handleOpenLayoutReplace,
handleOpenTextEdit,
imageNodeActionRequest,
node.id,
@ -6287,9 +6326,8 @@ export function ImageNode({
return;
}
// 验证文件大小 (50MB)
if (file.size > 50 * 1024 * 1024) {
toast.error("图片大小不能超过 50MB");
if (file.size > IMAGE_UPLOAD_MAX_BYTES) {
toast.error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
return;
}
@ -6386,6 +6424,7 @@ export function ImageNode({
"prompt-regenerate": "重绘",
"box-cutout": "框选抠图",
"local-edit": "局部编辑",
"layout-replace": "排版替换",
};
return names[action] || action;
};
@ -6828,6 +6867,10 @@ export function ImageNode({
handleEnterLocalEditMode();
return;
}
if (action === "layout-replace") {
handleOpenLayoutReplace();
return;
}
if (action === "scale-adjust") {
handleEnterLocalEditMode(undefined, "scale_adjust");
return;
@ -8292,11 +8335,7 @@ export function ImageNode({
toast.message("已取消下载");
} else {
console.error("下载失败:", error);
toast.error(
error instanceof Error
? `${error.message},源图可能已失效,请稍后重试或重新生成图片`
: "图片源文件暂时不可访问,请稍后重试或重新生成图片"
);
toast.error(getCanvasImageDownloadErrorMessage(error));
}
} finally {
if (downloadAbortControllerRef.current === controller) {
@ -9593,6 +9632,19 @@ export function ImageNode({
{imageTextEditItem.label}
</DropdownMenuItem>
) : null}
{imageLayoutReplaceEnabled ? (
<DropdownMenuItem
onClick={() => handleAction("layout-replace")}
>
<CanvasToolbarIcon
className="mr-2 h-4 w-4"
fallback={LayoutTemplate}
item={imageLayoutReplaceItem}
/>
<span>{imageLayoutReplaceItem.label}</span>
<ImageActionCreditBadge label={localEditCostShortLabel} />
</DropdownMenuItem>
) : null}
{imageExtendEnabled ? (
<DropdownMenuItem onClick={() => handleAction("extend")}>
<CanvasToolbarIcon
@ -9744,6 +9796,15 @@ export function ImageNode({
prompt={regeneratePrompt}
/>
{layoutReplaceDialogOpen ? (
<CanvasLayoutReplaceBatchDialog
costLabel={localEditCostShortLabel}
onOpenChange={setLayoutReplaceDialogOpen}
open
primaryNode={node}
/>
) : null}
<ThreeDGenerationDialog
costLabel={
threedGenerationDialogMode === "world"
@ -9903,6 +9964,7 @@ export function ImageNode({
!isEditingTitle &&
!sceneFusionDialogOpen &&
!promptDialogOpen &&
!layoutReplaceDialogOpen &&
!threedGenerationDialogMode &&
!notePanelOpen &&
!boxCutoutMode &&
@ -10300,7 +10362,7 @@ export function ImageNode({
</Button>
) : null}
{isSelectedImageActionInline("localEdit") ? (
{isSelectedImageActionInline("localEdit") ? (
<Button
className="group relative h-10 rounded-xl px-3 font-medium text-[13px] text-slate-700 hover:bg-slate-100 hover:text-slate-900 dark:text-slate-100 dark:hover:bg-zinc-800"
disabled={isProcessing || isSubmittingLocalEdit}
@ -10321,6 +10383,27 @@ export function ImageNode({
</Button>
) : null}
{isSelectedImageActionInline("layoutReplace") ? (
<Button
className="group relative h-10 rounded-xl px-3 font-medium text-[13px] text-slate-700 hover:bg-slate-100 hover:text-slate-900 dark:text-slate-100 dark:hover:bg-zinc-800"
disabled={isProcessing}
onClick={() => handleAction("layout-replace")}
title={imageLayoutReplaceItem.label}
variant="ghost"
>
<CanvasToolbarIcon
className="mr-1.5 h-4 w-4"
fallback={LayoutTemplate}
item={imageLayoutReplaceItem}
/>
{imageLayoutReplaceItem.label}
<ImageActionCreditBadge
compact
label={localEditCostShortLabel}
/>
</Button>
) : null}
{isSelectedImageActionInline("removeWatermark") ? (
<Button
className="group relative h-10 rounded-xl px-3 font-medium text-[13px] text-slate-700 hover:bg-slate-100 hover:text-slate-900 dark:text-slate-100 dark:hover:bg-zinc-800"
@ -10593,6 +10676,19 @@ export function ImageNode({
/>
<span>{imageLocalEditItem.label}</span>
<ImageActionCreditBadge label={localEditCostShortLabel} />
</DropdownMenuItem>
) : null}
{isSelectedImageActionInMore("layoutReplace") ? (
<DropdownMenuItem
onClick={() => handleAction("layout-replace")}
>
<CanvasToolbarIcon
className="mr-2 h-4 w-4"
fallback={LayoutTemplate}
item={imageLayoutReplaceItem}
/>
<span>{imageLayoutReplaceItem.label}</span>
<ImageActionCreditBadge label={localEditCostShortLabel} />
</DropdownMenuItem>
) : null}
{isSelectedImageActionInMore("removeWatermark") ? (

View File

@ -11,6 +11,10 @@ import {
import type { CanvasNode, TextNodeData } from "@/types/canvas";
import { cn } from "@/lib/utils";
import { fetchUploadWithRetry } from "@/lib/client/upload-with-retry";
import {
IMAGE_UPLOAD_MAX_BYTES,
IMAGE_UPLOAD_TOO_LARGE_MESSAGE,
} from "@/lib/image-upload-limits";
import { toast } from "sonner";
interface TextNodeProps {
@ -20,7 +24,7 @@ interface TextNodeProps {
}
const DEFAULT_TEXT_NODE_COLOR = "#111827";
const PASTED_IMAGE_MAX_BYTES = 50 * 1024 * 1024;
const PASTED_IMAGE_MAX_BYTES = IMAGE_UPLOAD_MAX_BYTES;
const PASTED_IMAGE_MAX_SIZE = 360;
const PASTED_IMAGE_MIN_SIZE = 140;
@ -127,7 +131,7 @@ export function TextNode({
const file = files[index];
if (!file.type.startsWith("image/")) continue;
if (file.size > PASTED_IMAGE_MAX_BYTES) {
toast.error("图片大小不能超过 50MB");
toast.error(IMAGE_UPLOAD_TOO_LARGE_MESSAGE);
continue;
}

View File

@ -0,0 +1,226 @@
"use client";
import { useEffect, useLayoutEffect, useRef, useState } from "react";
import { Check, ChevronDown, Search } from "lucide-react";
import { captionFonts, captionFont } from "@/lib/video-canvas/smart-edit/fonts";
import type { CaptionStyle } from "@/lib/video-canvas/smart-edit/contract";
const loaded = new Map<string, Promise<FontFace>>();
export function loadCaptionFont(id: string) {
const font = captionFont(id);
if (!font) return Promise.resolve();
if (!loaded.has(id)) {
const face = new FontFace(
font.family,
`url(${font.url ? `/api/canvas/smart-edit/fonts/${font.id}` : `/fonts/smart-edit/${font.file}`})`,
{
weight: "400",
},
);
loaded.set(
id,
face
.load()
.then((loadedFace) => {
document.fonts.add(loadedFace);
return loadedFace;
})
.catch((error) => {
loaded.delete(id);
throw error;
}),
);
}
return loaded.get(id)!;
}
export function CaptionFontPicker({
value,
onChange,
onError,
}: {
value: CaptionStyle["fontFamily"];
onChange: (id: NonNullable<CaptionStyle["fontFamily"]>) => void;
onError: (message: string) => void;
}) {
const [open, setOpen] = useState(false),
[query, setQuery] = useState(""),
[category, setCategory] = useState("全部"),
[categoriesOpen, setCategoriesOpen] = useState(false),
[busy, setBusy] = useState(false),
[scrollTop, setScrollTop] = useState(0);
const root = useRef<HTMLDivElement>(null);
const [placement, setPlacement] = useState({ left: 0, top: 0, width: 300, height: 300 });
useLayoutEffect(() => {
if (!open) return;
const place = () => {
const rect = root.current?.getBoundingClientRect();
if (!rect) return;
const below = window.innerHeight - rect.bottom - 16;
const above = rect.top - 16;
const up = below < 394 && above > below;
const height = Math.max(40, Math.min(300, (up ? above : below) - 94));
const width = Math.min(window.innerWidth - 16, rect.width + 32);
setPlacement({
left: Math.max(8, Math.min(rect.left - 16, window.innerWidth - width - 8)),
top: up ? Math.max(8, rect.top - height - 94 - 8) : rect.bottom + 8,
width,
height,
});
};
place();
const scroll = (event: Event) => {
if (!root.current?.contains(event.target as Node)) place();
};
window.addEventListener("resize", place);
window.addEventListener("scroll", scroll, true);
return () => {
window.removeEventListener("resize", place);
window.removeEventListener("scroll", scroll, true);
};
}, [open]);
useEffect(() => {
const close = (e: globalThis.PointerEvent) => {
if (!root.current?.contains(e.target as Node)) setOpen(false);
};
document.addEventListener("pointerdown", close);
return () => document.removeEventListener("pointerdown", close);
}, []);
const options = captionFonts.filter(
(font) =>
font.name.toLowerCase().includes(query.toLowerCase()) &&
(category === "全部" ||
(category === "常用" && font.common) ||
(category === "推荐" && font.recommended) ||
(category === "Google" && font.google)),
);
const first = Math.max(0, Math.floor(scrollTop / 40) - 4);
useEffect(() => {
setScrollTop(0);
}, [query, category, open]);
useEffect(() => {
if (!open) return;
let stopped = false,
index = 0;
const visible = options.slice(first, first + 18);
const worker = async () => {
while (!stopped && index < visible.length) {
const font = visible[index++];
try {
await loadCaptionFont(font.id);
} catch {
/* Selection reports any download failure. */
}
}
};
void worker();
void worker();
return () => {
stopped = true;
};
}, [open, first, query, category]);
return (
<div className="se-font-picker" ref={root}>
<button
className="se-font"
aria-label="字幕字体"
aria-expanded={open}
onClick={() => setOpen(!open)}
>
{captionFont(value)?.name || "默认字体"}
<ChevronDown size={14} />
</button>
{open && (
<div
className="se-font-menu"
style={{
position: "fixed",
left: placement.left,
top: placement.top,
width: placement.width,
right: "auto",
}}
>
<div className="se-search">
<Search size={14} />
<input
aria-label="搜索字体"
placeholder="搜索字体"
value={query}
onChange={(e) => setQuery(e.target.value)}
/>
</div>
<div className="se-font-categories">
<button
type="button"
aria-haspopup="menu"
aria-expanded={categoriesOpen}
onClick={() => setCategoriesOpen(!categoriesOpen)}
>
{category === "Google" ? "Google 字体" : `${category}字体`}
<ChevronDown size={12} />
</button>
{categoriesOpen && (
<div role="menu" aria-label="字体分类">
{["全部", "常用", "推荐", "Google"].map((name) => (
<button
type="button"
role="menuitemradio"
aria-checked={category === name}
key={name}
onClick={() => {
setCategory(name);
setCategoriesOpen(false);
}}
>
{name === "Google" ? "Google 字体" : `${name}字体`}
{category === name && <Check size={14} />}
</button>
))}
</div>
)}
</div>
<div
key={`${query}:${category}`}
role="listbox"
style={{ height: placement.height }}
aria-label={`字体库,共 ${options.length} 项`}
onScroll={(e) => setScrollTop(e.currentTarget.scrollTop)}
>
<div style={{ height: options.length * 40, position: "relative" }}>
{options.slice(first, first + 18).map((font, index) => (
<button
style={{
position: "absolute",
top: (first + index) * 40,
fontFamily: `"${font.family}", "Source Han Sans SC", sans-serif`,
}}
aria-posinset={first + index + 1}
aria-setsize={options.length}
role="option"
aria-selected={value === font.id}
key={font.id}
disabled={busy}
onClick={async () => {
setBusy(true);
try {
await loadCaptionFont(font.id);
onChange(font.id);
setOpen(false);
} catch {
onError("字体加载失败,请检查网络后重试");
} finally {
setBusy(false);
}
}}
>
{font.name}
{value === font.id && <Check size={14} />}
</button>
))}
</div>
</div>
{!options.length && <div className="se-muted">没有匹配的字体</div>}
{busy && <div role="status">正在加载字体…</div>}
</div>
)}
</div>
);
}

View File

@ -0,0 +1,561 @@
"use client";
import { useEffect, useRef, useState } from "react";
import { captionFont } from "@/lib/video-canvas/smart-edit/fonts";
import { loadCaptionFont } from "./CaptionFonts";
import { usePreviewGain } from "./usePreviewGain";
import {
editableCaptions,
defaultCaptionStyle,
type CaptionStyle,
planTimeline,
SECOND,
type EditPlan,
defaultVideoTransform,
type VideoTransform,
type TimelineCaption,
} from "@/lib/video-canvas/smart-edit/contract";
export function EditPreview({
plan,
time,
playing,
onTime,
onPlaying,
onError,
fit = false,
cover = false,
externalClock = false,
selectedClipId,
selectedCaptionId,
onSelectClip,
onSelectCaption,
onTransform,
onCaptionPosition,
}: {
fit?: boolean;
cover?: boolean;
externalClock?: boolean;
selectedClipId?: string;
selectedCaptionId?: string;
onSelectClip?: (id: string) => void;
onSelectCaption?: (id: string) => void;
onTransform?: (id: string, value: VideoTransform) => void;
onCaptionPosition?: (id: string, value: { x: number; y: number }) => void;
plan: EditPlan;
time: number;
playing: boolean;
onTime: (t: number) => void;
onPlaying: (v: boolean) => void;
onError: (s: string) => void;
}) {
const videos = useRef(new Map<string, HTMLVideoElement>()),
music = useRef<HTMLAudioElement | null>(null),
audios = useRef(new Map<string, HTMLAudioElement>()),
latest = useRef({ time, plan, onTime, onPlaying });
latest.current = { time, plan, onTime, onPlaying };
const fontIds = [
...new Set(
editableCaptions(plan)
.map((c) => c.style?.fontFamily)
.filter(Boolean),
),
].join(",");
useEffect(() => {
let canceled = false;
void Promise.all(
[
...new Set([...fontIds.split(",").filter(Boolean), ...(fontIds ? ["source-han"] : [])]),
].map(loadCaptionFont),
).catch(() => {
if (!canceled) onError("字幕字体加载失败,请检查网络后重试");
});
return () => {
canceled = true;
};
}, [fontIds]);
const audioGain = usePreviewGain(plan, onError);
const frameRef = useRef<HTMLDivElement>(null);
const [frameSize, setFrameSize] = useState({ width: 0, height: 0 });
useEffect(() => {
const el = frameRef.current;
if (!el) return;
const observer = new ResizeObserver(([entry]) =>
setFrameSize({ width: entry.contentRect.width, height: entry.contentRect.height }),
);
observer.observe(el);
return () => observer.disconnect();
}, []);
const timeline = planTimeline(plan),
visibleTime = Math.min(time * SECOND, timeline.duration - 1),
current = timeline.clips.filter(
(c) => visibleTime >= c.renderStart && visibleTime < c.renderEnd,
),
subs = plan.subtitles
? editableCaptions(plan).filter((s) => time * SECOND >= s.start && time * SECOND < s.end)
: [];
useEffect(() => {
if (!playing || externalClock) return;
let frame = 0,
last = 0;
const tick = (now: number) => {
const state = latest.current;
if (last) {
const next = Math.min(
planTimeline(state.plan).duration / SECOND,
state.time + (now - last) / 1000,
);
state.onTime(next);
if (next >= planTimeline(state.plan).duration / SECOND) {
state.onPlaying(false);
return;
}
}
last = now;
frame = requestAnimationFrame(tick);
};
const hide = () => {
if (document.hidden) latest.current.onPlaying(false);
};
frame = requestAnimationFrame(tick);
document.addEventListener("visibilitychange", hide);
return () => {
cancelAnimationFrame(frame);
document.removeEventListener("visibilitychange", hide);
};
}, [playing, externalClock]);
useEffect(() => {
audioGain.retain([...videos.current.values(), ...audios.current.values()]);
for (const clip of timeline.clips) {
const video = videos.current.get(clip.id);
if (!video) continue;
const active = current.some((c) => c.id === clip.id);
if (!active) {
video.pause();
continue;
}
const target = Math.max(
clip.sourceIn / SECOND,
Math.min(
clip.sourceOut / SECOND - 1 / 30,
clip.sourceIn / SECOND + (time - clip.start / SECOND) * clip.speed,
),
);
const held = time * SECOND < clip.start || time * SECOND >= clip.end;
if (Number.isFinite(target) && Math.abs(video.currentTime - target) > 0.1)
video.currentTime = target;
video.playbackRate = clip.speed;
const index = timeline.clips.findIndex((c) => c.id === clip.id),
next = timeline.clips[index + 1];
let gain = 1;
if (clip.overlap && time * SECOND < clip.transitionStart + clip.overlap)
gain = (time * SECOND - clip.transitionStart) / clip.overlap;
if (next?.overlap && time * SECOND > next.transitionStart)
gain = Math.min(
gain,
(next.transitionStart + next.overlap - time * SECOND) / next.overlap,
);
const source = plan.assets.find((a) => a.id === clip.assetId)!.url;
if (!audioGain.ready(source)) {
video.pause();
if (playing) {
audioGain.retry();
onPlaying(false);
onError("正在准备音量预览素材,请稍后点击播放");
}
continue;
}
audioGain.volume(
video,
clip.audioDetached || clip.muted || held ? 0 : clip.gain * gain,
playing,
);
if (playing && !held)
void video.play().catch((error) => {
if (error?.name === "AbortError") return;
onPlaying(false);
onError("预览播放失败,请检查素材链接或点击播放重试");
});
else video.pause();
}
for (const clip of timeline.audioClips) {
const audio = audios.current.get(clip.id);
if (!audio) continue;
const relative = time - clip.start / SECOND;
if (relative < 0 || time * SECOND >= clip.end) {
audio.pause();
continue;
}
const target = clip.sourceIn / SECOND + relative * clip.speed;
if (Math.abs(audio.currentTime - target) > 0.1) audio.currentTime = target;
audio.playbackRate = clip.speed;
const length = clip.duration / SECOND,
fadeIn = Math.min(clip.fadeIn, length),
fadeOut = Math.min(clip.fadeOut, length);
const gain =
Math.min(
1,
fadeIn
? (clip.fadeInStart || 0) + ((1 - (clip.fadeInStart || 0)) * relative) / fadeIn
: 1,
) *
Math.min(
1,
fadeOut
? (clip.fadeOutEnd || 0) +
((1 - (clip.fadeOutEnd || 0)) * (length - relative)) / fadeOut
: 1,
);
const source = plan.assets.find((a) => a.id === clip.assetId)!.url;
if (!audioGain.ready(source)) {
audio.pause();
if (playing) {
audioGain.retry();
onPlaying(false);
onError("正在准备音量预览素材,请稍后点击播放");
}
continue;
}
audioGain.volume(audio, clip.muted ? 0 : clip.gain * gain, playing);
if (playing)
void audio.play().catch((error) => {
if (error?.name === "AbortError") return;
onPlaying(false);
onError("独立音轨无法播放");
});
else audio.pause();
}
const audio = music.current;
if (audio && plan.music) {
const target = time + plan.music.start / SECOND;
if (Number.isFinite(audio.duration) && target >= audio.duration) {
audio.pause();
return;
}
if (Math.abs(audio.currentTime - target) > 0.15) audio.currentTime = target;
audio.volume = Math.min(1, plan.music.gain);
if (playing)
void audio.play().catch((error) => {
if (error?.name === "AbortError") return;
onPlaying(false);
onError("背景音乐无法播放");
});
else audio.pause();
}
}, [time, playing, plan]);
const [ratioWidth, ratioHeight] = plan.ratio.split(":").map(Number);
const ratio = ratioWidth / ratioHeight;
const height = Math.min(frameSize.height || 720, (frameSize.width || 1280) / ratio);
const incomingClip = current.find(
(c) => c.overlap && visibleTime < c.transitionStart + c.overlap,
);
const progress = incomingClip
? (visibleTime - incomingClip.transitionStart) / incomingClip.overlap
: 1;
const [gesture, setGesture] = useState<{
clipId?: string;
captionId?: string;
transform?: VideoTransform;
x?: number;
y?: number;
}>();
const gestureCleanup = useRef<() => void>();
useEffect(() => () => gestureCleanup.current?.(), []);
const moveTransform = (e: React.PointerEvent, id: string, corner?: [number, number]) => {
e.preventDefault();
e.stopPropagation();
onPlaying(false);
onSelectClip?.(id);
const c = plan.clips.find((c) => c.id === id)!,
a = plan.assets.find((a) => a.id === c.assetId)!,
original = c.transform || defaultVideoTransform;
const frameWidth = height * ratio,
baseWidth = Math.min(frameWidth, (height * a.width) / a.height),
baseHeight = (baseWidth * a.height) / a.width,
startX = e.clientX,
startY = e.clientY;
let value = original;
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
const move = (ev: PointerEvent) => {
const dx = ev.clientX - startX,
dy = ev.clientY - startY;
if (corner) {
const nextScale = clamp(
original.scale + ((dx * corner[0]) / baseWidth + (dy * corner[1]) / baseHeight) / 2,
0.1,
3,
),
delta = nextScale - original.scale;
value = {
scale: nextScale,
x: clamp(original.x + (delta * baseWidth * corner[0]) / 2 / frameWidth, -1, 1),
y: clamp(original.y + (delta * baseHeight * corner[1]) / 2 / height, -1, 1),
};
} else
value = {
...original,
x: clamp(original.x + dx / frameWidth, -1, 1),
y: clamp(original.y + dy / height, -1, 1),
};
setGesture({ clipId: id, transform: value });
};
const cleanup = () => {
window.removeEventListener("pointermove", move);
window.removeEventListener("pointerup", end);
window.removeEventListener("pointercancel", cancel);
gestureCleanup.current = undefined;
};
const end = () => {
cleanup();
setGesture(undefined);
onTransform?.(id, value);
},
cancel = () => {
cleanup();
setGesture(undefined);
};
gestureCleanup.current?.();
gestureCleanup.current = cleanup;
window.addEventListener("pointermove", move);
window.addEventListener("pointerup", end);
window.addEventListener("pointercancel", cancel);
};
const moveCaption = (e: React.PointerEvent, id: string, style: CaptionStyle) => {
e.preventDefault();
e.stopPropagation();
onPlaying(false);
onSelectCaption?.(id);
const startX = e.clientX,
startY = e.clientY;
let value = { x: style.x ?? 0.5, y: style.y ?? 0.95 };
const move = (ev: PointerEvent) => {
value = {
x: Math.max(
0,
Math.min(1, (style.x ?? 0.5) + (ev.clientX - startX) / (height * ratio)),
),
y: Math.max(0, Math.min(1, (style.y ?? 0.95) + (ev.clientY - startY) / height)),
};
setGesture({ captionId: id, ...value });
};
const cleanup = () => {
window.removeEventListener("pointermove", move);
window.removeEventListener("pointerup", end);
window.removeEventListener("pointercancel", cancel);
gestureCleanup.current = undefined;
};
const end = () => {
cleanup();
setGesture(undefined);
onCaptionPosition?.(id, value);
},
cancel = () => {
cleanup();
setGesture(undefined);
};
gestureCleanup.current?.();
gestureCleanup.current = cleanup;
window.addEventListener("pointermove", move);
window.addEventListener("pointerup", end);
window.addEventListener("pointercancel", cancel);
};
const selectedClip = current.find((c) => c.id === selectedClipId);
const selectedAsset = selectedClip && plan.assets.find((a) => a.id === selectedClip.assetId);
const selectedTransform =
gesture && gesture.clipId === selectedClipId
? gesture.transform
: selectedClip?.transform || defaultVideoTransform;
return (
<div
ref={frameRef}
className={
fit
? "relative h-full w-full overflow-hidden"
: "relative mx-auto w-full max-w-4xl overflow-hidden bg-black"
}
style={fit ? undefined : { aspectRatio: plan.ratio.replace(":", "/") }}
aria-label="剪辑实时预览"
>
<div
style={
fit && !cover
? {
position: "absolute",
width: height * ratio,
height,
left: "50%",
top: "50%",
transform: "translate(-50%,-50%)",
overflow: "hidden",
background: "black",
}
: { position: "absolute", inset: 0 }
}
>
{timeline.clips.map((clip) => {
const asset = plan.assets.find((a) => a.id === clip.assetId)!;
const active = current.some((c) => c.id === clip.id),
incoming = clip.overlap && visibleTime < clip.transitionStart + clip.overlap;
return (
<video
key={`${clip.id}:${audioGain.source(asset.url)}`}
ref={(element) => {
if (element) videos.current.set(clip.id, element);
else videos.current.delete(clip.id);
}}
src={audioGain.source(asset.url)}
muted={clip.audioDetached || clip.muted || clip.gain === 0}
playsInline
preload="metadata"
className={`absolute inset-0 h-full w-full ${cover ? "object-cover" : "object-contain"}`}
style={{
visibility: active ? "visible" : "hidden",
transform: (() => {
const t =
gesture?.clipId === clip.id
? gesture.transform!
: clip.transform || defaultVideoTransform;
return `translate(${t.x * 100}%,${t.y * 100}%) scale(${t.scale})`;
})(),
opacity: incoming
? clip.transitionType === "fade"
? progress
: progress < 0.5
? 0
: 1
: 1,
}}
onLoadedMetadata={(e) => {
const target =
clip.sourceIn / SECOND +
(Math.max(0, visibleTime - clip.start) / SECOND) * clip.speed;
e.currentTarget.currentTime = Math.min(clip.sourceOut / SECOND - 0.001, target);
}}
onError={() => {
onPlaying(false);
onError(`${asset.name} 预览加载失败`);
}}
/>
);
})}
{timeline.audioClips.map((c) => (
<audio
key={`${c.id}:${audioGain.source(plan.assets.find((a) => a.id === c.assetId)!.url)}`}
ref={(el) => {
if (el) audios.current.set(c.id, el);
else audios.current.delete(c.id);
}}
src={audioGain.source(plan.assets.find((a) => a.id === c.assetId)!.url)}
preload="metadata"
/>
))}
{plan.music ? <audio ref={music} src={plan.music.url} preload="metadata" /> : null}
{incomingClip && incomingClip.transitionType !== "fade" && (
<div
style={{
position: "absolute",
inset: 0,
pointerEvents: "none",
background: incomingClip.transitionType === "fadeblack" ? "black" : "white",
opacity: 1 - Math.abs(2 * progress - 1),
}}
/>
)}
{selectedClip && selectedAsset && selectedTransform && onTransform && (
<div
aria-label="调整画面位置"
onClick={(e) => e.stopPropagation()}
onPointerDown={(e) => moveTransform(e, selectedClip.id)}
style={{
position: "absolute",
left: `${(selectedTransform.x + 0.5) * 100}%`,
top: `${(selectedTransform.y + 0.5) * 100}%`,
width:
Math.min(
height * ratio,
(height * selectedAsset.width) / selectedAsset.height,
) * selectedTransform.scale,
height:
Math.min(
height,
(height * ratio * selectedAsset.height) / selectedAsset.width,
) * selectedTransform.scale,
transform: "translate(-50%,-50%)",
border: "2px solid #87c9ff",
cursor: "move",
zIndex: 3,
}}
>
{(
[
[-1, -1],
[1, -1],
[1, 1],
[-1, 1],
] as [number, number][]
).map(([x, y]) => (
<button
key={`${x}:${y}`}
type="button"
aria-label="调整视频尺寸"
onPointerDown={(e) => moveTransform(e, selectedClip.id, [x, y])}
style={{
position: "absolute",
width: 12,
height: 12,
left: x < 0 ? 0 : "100%",
top: y < 0 ? 0 : "100%",
transform: "translate(-50%,-50%)",
border: "1px solid white",
background: "#87c9ff",
borderRadius: 2,
cursor: x === y ? "nwse-resize" : "nesw-resize",
}}
/>
))}
</div>
)}
<div className="pointer-events-none absolute inset-0 text-center" style={{ zIndex: 4 }}>
{subs.map((s, i) => {
const style = (s as { style?: CaptionStyle }).style || defaultCaptionStyle;
return (
<div
key={i}
onClick={(e) => e.stopPropagation()}
onPointerDown={(e) => {
if (onCaptionPosition && "id" in s) moveCaption(e, s.id as string, style);
}}
style={{
position: "absolute",
left: `${((gesture && gesture.captionId === (s as TimelineCaption).id ? gesture.x : style.x) ?? 0.5) * 100}%`,
bottom: `${(1 - ((gesture && gesture.captionId === (s as TimelineCaption).id ? gesture.y : style.y) ?? 0.95)) * 100}%`,
transform: "translateX(-50%)",
maxWidth: "95%",
pointerEvents: onCaptionPosition ? "auto" : "none",
cursor: onCaptionPosition ? "move" : undefined,
outline:
selectedCaptionId === (s as TimelineCaption).id
? "1px solid #87c9ff"
: undefined,
fontFamily: `"${captionFont(style.fontFamily)?.family || "Noto Sans CJK SC"}", "Source Han Sans SC", sans-serif`,
whiteSpace: "pre-wrap",
fontSize: (style.fontSize * height) / (style.referenceHeight || 720),
letterSpacing: (style.spacing * height) / (style.referenceHeight || 720),
color: style.color,
opacity: style.opacity,
WebkitTextStroke: `${((style.outlineWidth * height) / (style.referenceHeight || 720)) * 2}px ${style.outlineColor}${Math.round(
(style.outlineOpacity ?? 1) * 255,
)
.toString(16)
.padStart(2, "0")}`,
paintOrder: "stroke fill",
lineHeight: 1.2,
}}
>
{s.text}
</div>
);
})}
</div>
</div>
</div>
);
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,433 @@
.card {
position: relative;
width: 100%;
height: 100%;
padding: 8px;
border: 1px solid #454545;
border-radius: 18px;
background: #262626;
color: #eee;
font-family: Arial, "PingFang SC", sans-serif;
}
.title {
position: absolute;
top: -28px;
left: 0;
display: flex;
align-items: center;
gap: 6px;
font-size: 14px;
color: #929292;
white-space: nowrap;
}
.preview {
position: relative;
height: calc(100% - 80px);
overflow: hidden;
border-radius: 10px 10px 0 0;
background: #202020;
}
.emptyCard .preview {
height: 100%;
border-radius: 10px;
}
.emptyActions {
position: absolute;
bottom: 24px;
left: 28px;
display: flex;
flex-direction: column;
align-items: flex-start;
gap: 8px;
color: #a5a5a5;
font-size: 13px;
}
.emptyActions button {
padding: 0;
border: 0;
background: transparent;
color: #eee;
font-size: 14px;
line-height: 20px;
cursor: pointer;
}
.emptyActions button:hover {
color: white;
}
.preview > video {
width: 100%;
height: 100%;
object-fit: cover;
opacity: 0.35;
}
.open {
position: absolute;
inset: 0;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 16px;
}
.open button {
height: 32px;
padding: 0 12px;
border: 0;
border-radius: 8px;
background: white;
color: #181818;
font-size: 16px;
cursor: pointer;
}
.ruler {
height: 26px;
display: flex;
justify-content: space-around;
align-items: center;
border-bottom: 1px solid #393939;
font-size: 8px;
color: #929292;
}
.tracks {
padding: 8px;
}
.tracks > div {
display: flex;
gap: 3px;
height: 20px;
margin-bottom: 3px;
}
.tracks i {
flex: 1;
background: #333;
border-radius: 3px;
}
.composer {
position: absolute;
z-index: 30;
transform: translateX(-50%);
width: 660px;
max-width: calc(100vw - 32px);
padding: 8px;
border: 0.5px solid #333;
border-radius: 16px;
background: #262626;
color: #eee;
box-shadow: 0 4px 16px #0002;
font:
14px / 1.5 Arial,
"PingFang SC",
sans-serif;
display: flex;
flex-direction: column;
gap: 8px;
}
.composer button {
cursor: pointer;
border: 0;
color: inherit;
font: inherit;
background: none;
display: flex;
align-items: center;
justify-content: center;
gap: 4px;
}
.composer button:disabled {
opacity: 0.4;
cursor: default;
}
.referenceRow {
height: 30px;
display: flex;
justify-content: space-between;
padding: 4px 4px 0;
}
.referenceRow > button:first-child {
font-size: 12px;
border-radius: 16px;
background: #333;
color: #999;
padding: 0 6px;
height: 26px;
}
.referenceRow > button:last-child {
color: #999;
width: 24px;
height: 24px;
}
.sources {
display: flex;
gap: 8px;
min-height: 54.8px;
flex-wrap: wrap;
padding: 0 4px;
}
.source {
width: 48px;
height: 48px;
position: relative;
border-radius: 8px;
background: #191919;
}
.source video {
width: 100%;
height: 100%;
object-fit: cover;
border-radius: 8px;
}
.source b {
position: absolute;
top: 3px;
left: 3px;
font-size: 9px;
font-weight: 400;
background: #0009;
border-radius: 8px;
padding: 0 3px;
}
.source > svg {
position: absolute;
top: 16px;
left: 16px;
width: 16px;
height: 16px;
padding: 3px;
border-radius: 50%;
background: #0009;
}
.source > span {
position: absolute;
bottom: 2px;
left: 3px;
font-size: 8px;
background: #0009;
line-height: 11px;
}
.source > button {
display: none;
position: absolute;
top: -4px;
right: -4px;
width: 15px;
height: 15px;
border-radius: 50%;
background: #666;
}
.source:hover > button,
.source:focus-within > button {
display: flex;
}
.composer textarea {
display: block;
width: 100%;
resize: none;
border: 0;
outline: 0;
background: transparent;
color: #eee;
padding: 8px 4px;
font:
14px / 1.8 Arial,
"PingFang SC",
sans-serif;
}
.composer textarea::placeholder {
color: #777;
}
.controls {
height: 32px;
display: flex;
gap: 8px;
}
.controls .mode,
.controls .parameters {
font-size: 12px;
height: 32px;
padding: 0 8px;
border-radius: 8px;
background: #333;
}
.controls .mode {
background: none;
}
.controls .send {
margin-left: auto;
width: 32px;
height: 32px;
border-radius: 12px;
background: white;
color: #222;
}
.popup {
position: absolute;
z-index: 5;
bottom: 40px;
padding: 12px;
background: #262626;
border: 0.5px solid #393939;
border-radius: 16px;
box-shadow: 0 4px 16px #0006;
color: #929292;
}
.settings {
left: 112px;
width: 340px;
}
.musicDetails {
margin-top: 10px;
border-top: 1px solid #393939;
padding-top: 7px;
}
.musicDetails summary {
cursor: pointer;
font-size: 12px;
}
.settings label {
display: block;
font-size: 12px;
margin: 0 0 8px;
}
.ratios {
display: grid;
grid-template-columns: repeat(5, 1fr);
gap: 8px;
margin-bottom: 8px;
}
.ratios button {
height: 62px;
flex-direction: column;
gap: 8px;
font-size: 12px;
border: 1px solid #393939;
border-radius: 8px;
}
.ratios i {
display: block;
border: 1.5px solid currentColor;
border-radius: 1px;
}
.settings button[aria-pressed="true"] {
color: white;
background: #414141;
border-color: #999;
}
.duration {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 8px;
height: 22px;
}
.duration input[type="range"] {
min-width: 0;
flex: 1;
accent-color: white;
height: 4px;
}
.duration input[type="number"] {
width: 48px;
height: 22px;
appearance: textfield;
background: #333;
border: 0;
outline: 0;
border-radius: 6px;
color: #eee;
text-align: center;
font-size: 12px;
}
.duration input::-webkit-inner-spin-button {
appearance: none;
}
.duration > span {
font-size: 12px;
}
.resolutions {
display: flex;
gap: 8px;
}
.resolutions button {
flex: 1;
height: 34px;
border: 1px solid #393939;
border-radius: 10px;
font-size: 12px;
}
.modeMenu {
left: 8px;
width: 160px;
padding: 6px;
}
.modeMenu button {
width: 100%;
height: 36px;
padding: 0 8px;
justify-content: space-between;
color: #ddd;
font-size: 13px;
border-radius: 8px;
}
.modeMenu button:hover,
.sourceMenu button:hover {
background: #383838;
}
.sourceMenu {
left: 8px;
bottom: calc(100% - 40px);
width: 260px;
max-height: 320px;
overflow: auto;
}
.sourceMenu > div,
.sourceMenu > p {
font-size: 12px;
padding: 8px 4px;
}
.sourceMenu button {
width: 100%;
height: 36px;
justify-content: flex-start;
padding: 0 8px;
border-radius: 8px;
font-size: 13px;
color: #ddd;
}
.sourceMenu button span {
flex: 1;
text-overflow: ellipsis;
overflow: hidden;
white-space: nowrap;
text-align: left;
}
.error {
color: #fca5a5;
font-size: 12px;
padding: 4px;
}
@media (max-width: 500px) {
.settings {
left: 4px;
max-width: calc(100vw - 48px);
}
}
.tracks { max-height: 54px; overflow: hidden; }
.tracks > div { position: relative; height: 12px; }
.tracks i { position: absolute; top: 0; bottom: 0; border-right: 2px solid #262626; }
.settings { max-height: min(600px, calc(100vh - 180px)); overflow-y: auto; }
.settings select { width: 100%; height: 30px; padding: 0 8px; border-radius: 6px; background: #333; color: #eee; border: 0; }
.musicUpload { display: flex; align-items: center; gap: 4px; margin-top: 8px; font-size: 12px; }
.settings label.beatToggle { display: flex; gap: 8px; align-items: center; }
.emptyCard .open {
justify-content: flex-start;
padding-top: 94px;
gap: 16px;
color: #999;
font-size: 12px;
}
.emptyCard .open svg {
color: #aaa;
}

View File

@ -0,0 +1,511 @@
"use client";
import { useEffect, useRef, useState } from "react";
import {
ArrowUp,
Check,
ChevronDown,
Maximize2,
Play,
Plus,
Scissors,
Upload,
X,
} from "lucide-react";
import {
editRatios,
editResolutions,
planTimeline,
editableCaptions,
SECOND,
type EditPlan,
} from "@/lib/video-canvas/smart-edit/contract";
import { resolveSmartEditIntent, assertBeatMusic } from "@/lib/video-canvas/smart-edit/intent";
import styles from "./SmartEditNode.module.css";
export type SmartEditSource = { id: string; name: string; url: string; duration?: number };
export type SmartEditMode = "explain" | "ads" | "talk" | "montage";
const modes = [
{ id: "auto", name: "默认模式" },
{ id: "explain", name: "讲解视频" },
{ id: "ads", name: "批量广告" },
{ id: "talk", name: "口播视频" },
{ id: "montage", name: "素材混剪" },
];
export function SmartEditNodeCard({
title,
previewUrl,
plan,
onOpen,
onMode,
selected = false,
}: {
title: string;
selected?: boolean;
previewUrl?: string;
plan?: EditPlan;
onOpen: () => void;
onMode?: (mode: SmartEditMode) => void;
}) {
let timeline: ReturnType<typeof planTimeline> | undefined;
try {
if (plan) timeline = planTimeline(plan);
} catch {
/* Invalid saved drafts open in the editor for recovery. */
}
const length = timeline?.duration || 0,
tick = Math.max(1, Math.ceil(length / SECOND / 5));
const empty = !timeline && !previewUrl;
return (
<div className={`${styles.card} ${empty ? styles.emptyCard : ""}`} style={{ borderColor: selected ? "#888" : "#454545" }}>
<div className={styles.title}>
<Play size={12} fill="currentColor" />
{title}
</div>
<div className={styles.preview}>
{previewUrl && <video src={`${previewUrl}#t=0.1`} muted preload="metadata" />}
<div className={styles.open}>
<Scissors size={36} />
{empty ? (
<span>空空如也,请连接视频节点后操作</span>
) : (
<button onPointerDown={(e) => e.stopPropagation()} onClick={onOpen}>
打开智能剪辑
</button>
)}
</div>
</div>
{empty ? (
<div className={styles.emptyActions}>
<span>尝试:</span>
{([
["explain", "讲解视频"],
["ads", "批量广告"],
["talk", "口播视频"],
["montage", "素材混剪"],
] as const).map(([mode, label]) => (
<button key={mode} onPointerDown={(e) => e.stopPropagation()} onClick={() => onMode?.(mode)} type="button">{label}</button>
))}
</div>
) : null}
{!empty ? <>
<div className={styles.ruler} aria-label="剪辑时间刻度">
{timeline ? (
Array.from({ length: Math.floor(length / SECOND / tick) + 1 }, (_, i) => {
const seconds = i * tick;
return (
<span key={i}>
{Math.floor(seconds / 60)}:{String(seconds % 60).padStart(2, "0")}
</span>
);
})
) : (
<span>尚未生成时间轴</span>
)}
</div>
<div className={styles.tracks} aria-label="剪辑时间轴缩略图">
{timeline && (
<>
<div>
{timeline.clips.map((clip) => (
<i
key={clip.id}
title={clip.label}
style={{
left: `${(clip.start / length) * 100}%`,
width: `${(clip.duration / length) * 100}%`,
}}
/>
))}
</div>
{!!timeline.audioClips.length && (
<div>
{timeline.audioClips.map((clip) => (
<i
key={clip.id}
style={{
left: `${(clip.start / length) * 100}%`,
width: `${(clip.duration / length) * 100}%`,
background: "#346752",
}}
/>
))}
</div>
)}
{plan?.subtitles && (
<div>
{editableCaptions(plan).map((caption) => (
<i
key={caption.id}
title={caption.text}
style={{
left: `${(caption.start / length) * 100}%`,
width: `${((caption.end - caption.start) / length) * 100}%`,
background: "#75602e",
}}
/>
))}
</div>
)}
</>
)}
</div>
</> : null}
</div>
);
}
export function SmartEditNodeComposer({
anchor,
draft,
sources,
availableSources,
availableAudios = [],
onDraft,
onAddSource,
onRemoveSource,
onUpload,
onReference,
onSubmit,
busy,
}: {
anchor: { centerX: number; bottomY: number; zoom: number };
draft: Record<string, any>;
sources: SmartEditSource[];
availableSources: SmartEditSource[];
availableAudios?: { name: string; url: string }[];
onDraft: (patch: Record<string, unknown>) => void;
onAddSource: (id: string) => void;
onRemoveSource: (id: string) => void;
onUpload: (files: File[]) => void;
onReference?: () => void;
onSubmit: () => Promise<unknown>;
busy: boolean;
}) {
const [menu, setMenu] = useState<"mode" | "settings" | "sources" | null>(null),
[expanded, setExpanded] = useState(false),
[error, setError] = useState(""),
[submitting, setSubmitting] = useState(false);
useEffect(() => setError(""), [draft.music?.url, draft.beatSync]);
const root = useRef<HTMLDivElement>(null),
file = useRef<HTMLInputElement>(null),
musicFile = useRef<HTMLInputElement>(null);
useEffect(() => {
const close = (e: PointerEvent) => {
if (!root.current?.contains(e.target as Node)) setMenu(null);
};
document.addEventListener("pointerdown", close);
return () => document.removeEventListener("pointerdown", close);
}, []);
const ratio = draft.ratio || "16:9",
resolution = draft.resolution || "720p",
duration = draft.targetDuration ?? 30,
mode = modes.find((m) => m.id === draft.mode) || modes[0];
const toggle = (value: typeof menu) => setMenu(menu === value ? null : value);
return (
<div
ref={root}
className={styles.composer}
style={{ left: anchor.centerX, top: anchor.bottomY + 16 * anchor.zoom }}
onPointerDown={(e) => e.stopPropagation()}
onWheel={(e) => e.stopPropagation()}
onKeyDown={(e) => e.stopPropagation()}
>
<div className={styles.referenceRow}>
<button
onClick={() => (onReference ? onReference() : toggle("sources"))}
onContextMenu={(e) => {
e.preventDefault();
toggle("sources");
}}
>
<Plus size={14} />
参考
</button>
<button
aria-label={expanded ? "收起输入框" : "展开输入框"}
onClick={() => setExpanded(!expanded)}
>
<Maximize2 size={16} />
</button>
</div>
{!!sources.length && (
<div className={styles.sources}>
{sources.map((s, i) => (
<div className={styles.source} key={s.id} title={s.name}>
<video src={`${s.url}#t=0.1`} preload="metadata" muted />
<b>{i + 1}</b>
<Play size={12} fill="white" />
<span>{s.duration ? `${s.duration.toFixed(1)}s` : "视频"}</span>
<button aria-label={`移除 ${s.name}`} onClick={() => onRemoveSource(s.id)}>
<X size={10} />
</button>
</div>
))}
</div>
)}
<input
ref={musicFile}
hidden
type="file"
accept="audio/*"
onChange={async (e) => {
const file = e.target.files?.[0];
e.target.value = "";
if (!file) return;
if (file.size > 125 * 1024 * 1024) {
setError("单个素材不得超过 125 MiB");
return;
}
setSubmitting(true);
setError("");
try {
const body = new FormData();
body.append("file", file);
let response: Response;
try {
response = await fetch("/api/canvas/media/upload", {
method: "POST",
body,
signal: AbortSignal.timeout(600000),
});
} catch (e) {
throw new Error(
e instanceof DOMException && (e.name === "TimeoutError" || e.name === "AbortError")
? "上传超时,请检查网络后重试,或改用更小的素材"
: "上传失败,请检查网络后重试",
);
}
const data = await response.json(),
url = data.url || data.data?.url;
if (!response.ok || !url) throw new Error(data.error || "音乐上传失败");
onDraft({ music: { url, gain: 0.2, start: 0 } });
} catch (e) {
setError(e instanceof Error ? e.message : "音乐上传失败");
} finally {
setSubmitting(false);
}
}}
/>
<textarea
aria-label="智能剪辑要求"
placeholder="描述你想要的剪辑效果"
value={draft.prompt || ""}
onChange={(e) => onDraft({ prompt: e.target.value })}
style={{ height: expanded ? 192 : 96 }}
/>
<div className={styles.controls}>
<button className={styles.mode} onClick={() => toggle("mode")}>
{mode.name}
<ChevronDown size={12} />
</button>
<button className={styles.parameters} onClick={() => toggle("settings")}>
{ratio} · {String(resolution).toUpperCase()} · {duration ? `${duration}s` : "自动"}
<ChevronDown size={12} />
</button>
<button
className={styles.send}
disabled={busy || submitting || sources.length === 0}
aria-label="发送"
onClick={async () => {
setError("");
setSubmitting(true);
try {
assertBeatMusic(resolveSmartEditIntent(draft));
await onSubmit();
} catch (e) {
setError(e instanceof Error ? e.message : "提交失败");
} finally {
setSubmitting(false);
}
}}
>
<ArrowUp size={20} />
</button>
</div>
{error && (
<div role="alert" className={styles.error}>
{error}
</div>
)}
{menu === "mode" && (
<div className={`${styles.popup} ${styles.modeMenu}`}>
{modes.map((m) => (
<button
key={m.id}
onClick={() => {
onDraft({ mode: m.id });
setMenu(null);
}}
>
{m.name}
{m.id === mode.id && <Check size={14} />}
</button>
))}
</div>
)}
{menu === "settings" && (
<div className={`${styles.popup} ${styles.settings}`}>
<label>比例</label>
<div className={styles.ratios}>
{editRatios.map((r) => {
const [w, h] = r.split(":").map(Number);
return (
<button
key={r}
aria-pressed={r === ratio}
onClick={() => onDraft({ ratio: r })}
>
<i
style={{
width: Math.min(16, (16 * w) / h),
height: Math.min(16, (16 * h) / w),
}}
/>
{r}
</button>
);
})}
</div>
<label htmlFor="smart-edit-duration">时长</label>
<div className={styles.duration}>
<input
aria-label="目标时长"
type="range"
min={0}
max={300}
step={1}
value={duration}
onChange={(e) => onDraft({ targetDuration: Number(e.target.value) })}
/>
<input
id="smart-edit-duration"
type="number"
min={0}
max={300}
value={duration}
onChange={(e) =>
onDraft({ targetDuration: Math.max(0, Math.min(300, Number(e.target.value))) })
}
/>
<span>s</span>
</div>
{mode.id === "ads" && (
<>
<label>广告版本</label>
<div className={styles.resolutions}>
{[1, 2, 3].map((count) => (
<button
key={count}
aria-pressed={(draft.variants ?? 3) === count}
onClick={() => onDraft({ variants: count })}
>
{count} 版
</button>
))}
</div>
</>
)}
<details className={styles.musicDetails}>
<summary>背景音乐与卡点</summary>
<label htmlFor="smart-edit-music">背景音乐</label>
<select
id="smart-edit-music"
value={draft.music?.url || ""}
onChange={(e) =>
onDraft({
music: e.target.value
? { url: e.target.value, gain: 0.2, start: 0 }
: undefined,
})
}
>
<option value="">未选择</option>
{draft.music?.url && !availableAudios.some((a) => a.url === draft.music.url) && (
<option value={draft.music.url}>已上传音乐</option>
)}
{availableAudios.map((audio) => (
<option key={audio.url} value={audio.url}>
{audio.name}
</option>
))}
</select>
<button
className={styles.musicUpload}
disabled={submitting}
onClick={() => musicFile.current?.click()}
>
<Upload size={12} />
上传音乐
</button>
<label className={styles.beatToggle}>
<input
type="checkbox"
checked={resolveSmartEditIntent(draft).beatSync}
onChange={(e) => onDraft({ beatSync: e.target.checked })}
/>
音乐卡点
</label>
{resolveSmartEditIntent(draft).beatSync && (
<div className={styles.resolutions}>
{["2", "4"].map((beats) => (
<button
key={beats}
aria-pressed={(draft.beatsPerCut || "4") === beats}
onClick={() => onDraft({ beatsPerCut: beats })}
>
{beats} 拍切换
</button>
))}
</div>
)}
</details>
<label>清晰度</label>
<div className={styles.resolutions}>
{editResolutions.map((r) => (
<button
key={r}
aria-pressed={r === resolution}
onClick={() => onDraft({ resolution: r })}
>
{r.toUpperCase()}
</button>
))}
</div>
</div>
)}
{menu === "sources" && (
<div className={`${styles.popup} ${styles.sourceMenu}`}>
<button onClick={() => file.current?.click()} disabled={sources.length >= 12}>
<Upload size={14} />
上传视频
</button>
<div>画布素材 · {sources.length}/12</div>
{availableSources.map((s) => (
<button
key={s.id}
disabled={sources.length >= 12 && !sources.some((a) => a.id === s.id)}
onClick={() =>
sources.some((a) => a.id === s.id) ? onRemoveSource(s.id) : onAddSource(s.id)
}
>
<Play size={14} />
<span>{s.name}</span>
{sources.some((a) => a.id === s.id) && <Check size={14} />}
</button>
))}
{!availableSources.length && <p>上传视频或连接画布视频素材</p>}
</div>
)}
<input
ref={file}
type="file"
hidden
accept="video/*"
multiple
onChange={(e) => {
onUpload(Array.from(e.target.files || []).slice(0, 12 - sources.length));
e.target.value = "";
setMenu(null);
}}
/>
</div>
);
}

View File

@ -0,0 +1,75 @@
"use client";
import { useEffect, useMemo, useState } from "react";
import { type EditPlan, planTimeline, SECOND } from "@/lib/video-canvas/smart-edit/contract";
import { EditPreview } from "./EditPreview";
/** Preview the selected edit boundary with the same timing/compositing used by the editor. */
export function TransitionPreview({
plan,
index,
cover = false,
}: {
plan: EditPlan;
index: number;
cover?: boolean;
}) {
const preview = useMemo(
() => ({
...plan,
subtitles: false,
music: undefined,
audioClips: [],
clips: plan.clips
.slice(index - 1, index + 1)
.map((c) => ({ ...c, gain: 0, audioDetached: true })),
}),
[plan, index],
);
const timeline = useMemo(() => planTimeline(preview), [preview]);
const boundary = timeline.clips[1];
const start = Math.max(0, boundary.transitionStart / SECOND - 0.25);
const end = Math.min(
timeline.duration / SECOND,
(boundary.transitionStart + boundary.overlap) / SECOND + 0.25,
);
const [time, setTime] = useState(start),
[error, setError] = useState(""),
[visible, setVisible] = useState(true);
useEffect(() => {
const change = () => setVisible(!document.hidden);
document.addEventListener("visibilitychange", change);
return () => document.removeEventListener("visibilitychange", change);
}, []);
useEffect(() => {
let request = 0,
epoch = 0;
setError("");
setTime(start);
const tick = (now: number) => {
if (document.hidden) epoch = 0;
else {
if (!epoch) epoch = now;
setTime(start + (((now - epoch) / 1000) % (end - start)));
}
request = requestAnimationFrame(tick);
};
request = requestAnimationFrame(tick);
return () => cancelAnimationFrame(request);
}, [start, end, preview]);
return (
<div className="se-transition-preview" aria-label="当前片段转场动态预览">
<EditPreview
plan={preview}
time={time}
playing={visible}
externalClock
onTime={setTime}
onPlaying={() => {}}
onError={setError}
fit
cover={cover}
/>
{error && <div className="se-transition-preview-error">{error}</div>}
</div>
);
}

View File

@ -0,0 +1,111 @@
"use client";
import { useEffect, useRef, useState } from "react";
import type { EditPlan } from "@/lib/video-canvas/smart-edit/contract";
/** Only amplified cross-origin sources need a same-origin copy for Web Audio. */
export function usePreviewGain(plan: EditPlan, onError: (message: string) => void) {
const remote = [
...new Set(
[...plan.clips, ...(plan.audioClips || [])]
.filter((clip) => clip.gain > 1)
.map((clip) => plan.assets.find((asset) => asset.id === clip.assetId)!.url)
.filter(
(url) =>
typeof window !== "undefined" &&
new URL(url, window.location.href).origin !== window.location.origin,
),
),
].sort();
const key = JSON.stringify(remote);
const [copies, setCopies] = useState<{ key: string; urls: Record<string, string> }>({
key: "",
urls: {},
});
const [attempt, setAttempt] = useState(0),
[failed, setFailed] = useState(false);
const error = useRef(onError);
error.current = onError;
const graph = useRef<{
context: AudioContext;
nodes: Map<HTMLMediaElement, { source: MediaElementAudioSourceNode; gain: GainNode }>;
}>();
useEffect(() => {
setFailed(false);
const controller = new AbortController(),
urls: Record<string, string> = {};
void Promise.all(
(JSON.parse(key) as string[]).map(async (url) => {
const response = await fetch("/api/canvas/download-image", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url, name: "剪辑音量预览" }),
signal: controller.signal,
});
if (!response.ok) throw Error("无法读取音量预览素材,请检查链接后重试");
const blob = await response.blob();
if (!blob.size) throw Error("音量预览素材为空");
if (!controller.signal.aborted) urls[url] = URL.createObjectURL(blob);
}),
)
.then(() => {
if (!controller.signal.aborted) setCopies({ key, urls });
})
.catch((e) => {
if (!controller.signal.aborted) {
setFailed(true);
error.current(`${e.message},可点击播放重试`);
}
});
return () => {
controller.abort();
Object.values(urls).forEach(URL.revokeObjectURL);
};
}, [key, attempt]);
useEffect(
() => () => {
void graph.current?.context.close();
graph.current = undefined;
},
[],
);
return {
retry() {
if (failed) setAttempt((value) => value + 1);
},
source(url: string) {
return copies.key === key ? copies.urls[url] || url : url;
},
ready(url: string) {
return !remote.includes(url) || (copies.key === key && Boolean(copies.urls[url]));
},
retain(elements: HTMLMediaElement[]) {
for (const [element, node] of graph.current?.nodes || [])
if (!elements.includes(element)) {
element.pause();
node.source.disconnect();
node.gain.disconnect();
graph.current!.nodes.delete(element);
}
},
volume(element: HTMLMediaElement, gain: number, playing: boolean) {
let node = graph.current?.nodes.get(element);
if (gain > 1 && !node) {
graph.current ||= { context: new AudioContext(), nodes: new Map() };
const source = graph.current.context.createMediaElementSource(element),
amplifier = graph.current.context.createGain();
source.connect(amplifier);
amplifier.connect(graph.current.context.destination);
node = { source, gain: amplifier };
graph.current.nodes.set(element, node);
}
if (node) {
element.volume = 1;
node.gain.gain.value = Math.max(0, gain);
if (playing && graph.current?.context.state === "suspended")
void graph.current.context
.resume()
.catch(() => error.current("请再次点击播放以启用音频预览"));
} else element.volume = Math.min(1, Math.max(0, gain));
},
};
}

View File

@ -1917,9 +1917,6 @@ function WorldSplatCaptureBridge({
function WorldSplatViewer({
splatUrl,
splatUrls,
fallbackMeshUrl,
fallbackMeshSettings,
fallbackMeshLayerVisibility,
worldUrl,
previewImageUrl,
firstPerson,
@ -1927,8 +1924,6 @@ function WorldSplatViewer({
interactionMode,
onPick,
onHotspotSelect,
onFallbackMeshLayersChange,
onFallbackMeshStatsChange,
semanticsMetadata,
resetViewToken,
minimalFrame = false,
@ -1938,9 +1933,6 @@ function WorldSplatViewer({
}: {
splatUrl?: string | null;
splatUrls?: string[] | null;
fallbackMeshUrl?: string | null;
fallbackMeshSettings?: Model3DViewerSettings;
fallbackMeshLayerVisibility?: Record<string, boolean>;
worldUrl?: string | null;
previewImageUrl?: string | null;
minimalFrame?: boolean;
@ -1949,8 +1941,6 @@ function WorldSplatViewer({
interactionMode?: "navigate" | "hotspot";
onPick?: (pick: World3DPickResult) => void;
onHotspotSelect?: (hotspotId: string) => void;
onFallbackMeshLayersChange?: (layers: Model3DLayer[]) => void;
onFallbackMeshStatsChange?: (stats: Model3DStats) => void;
semanticsMetadata?: Record<string, unknown> | null;
resetViewToken?: number;
onCaptureReady?: (capture: (() => Promise<Blob | null>) | null) => void;
@ -1967,7 +1957,6 @@ function WorldSplatViewer({
const [prefetchedSplat, setPrefetchedSplat] = useState<PrefetchedSplatAsset | null>(
null
);
const [useMeshFallback, setUseMeshFallback] = useState(false);
const resolvedSplatUrls = useMemo(() => {
const rawUrls = [
splatUrl,
@ -1985,16 +1974,13 @@ function WorldSplatViewer({
}, [splatUrl, splatUrls]);
const resolvedSplatSignature = resolvedSplatUrls.join("|");
const resolvedSplatUrl = resolvedSplatUrls[activeSplatIndex] || "";
const resolvedFallbackMeshUrl = String(fallbackMeshUrl || "").trim() || null;
const hasMeshFallback = Boolean(resolvedFallbackMeshUrl);
const loadStartedAtRef = useRef(0);
const movementStateRef = useRef<MovementState>(createMovementState());
useEffect(() => {
setActiveSplatIndex(0);
setFailedSplatLoads([]);
setUseMeshFallback(false);
}, [resolvedFallbackMeshUrl, resolvedSplatSignature]);
}, [resolvedSplatSignature]);
useEffect(() => {
if (!resolvedSplatUrl) return;
@ -2047,10 +2033,6 @@ function WorldSplatViewer({
const failedUrl = resolvedSplatUrl;
const nextIndex = activeSplatIndex + 1;
const nextUrl = resolvedSplatUrls[nextIndex] || "";
const shouldUseMeshFallback =
hasMeshFallback &&
(message.includes("SPZ 加载超时") || !nextUrl);
if (failedUrl) {
setFailedSplatLoads((items) => [
...items,
@ -2058,20 +2040,6 @@ function WorldSplatViewer({
]);
}
if (shouldUseMeshFallback) {
console.warn("[WorldSplatViewer] Switching to mesh fallback", {
error: message,
failedUrl,
attemptedUrls: resolvedSplatUrls.slice(0, nextIndex),
fallbackMeshUrl: resolvedFallbackMeshUrl,
});
setPrefetchedSplat(null);
setLoadProgress(null);
setOverlayPhase("hidden");
setUseMeshFallback(true);
return;
}
if (nextUrl) {
console.warn("[WorldSplatViewer] Falling back to next SPZ asset", {
error: message,
@ -2096,15 +2064,13 @@ function WorldSplatViewer({
},
[
activeSplatIndex,
hasMeshFallback,
resolvedFallbackMeshUrl,
resolvedSplatUrl,
resolvedSplatUrls,
]
);
useEffect(() => {
if (!resolvedSplatUrl || useMeshFallback) return;
if (!resolvedSplatUrl) return;
const abortController = new AbortController();
let cancelled = false;
@ -2141,25 +2107,7 @@ function WorldSplatViewer({
globalThis.clearTimeout(timeoutId);
abortController.abort();
};
}, [handleSplatLoadStateChange, resolvedSplatUrl, useMeshFallback]);
if ((useMeshFallback || !resolvedSplatUrl) && resolvedFallbackMeshUrl) {
return (
<Model3DViewer
emptyStateTitle="准备加载 3D Mesh"
layerVisibility={fallbackMeshLayerVisibility}
modelRotation={WORLDLABS_ASSET_ROTATION}
modelUrl={resolvedFallbackMeshUrl}
onCaptureReady={onCaptureReady}
onLayersChange={onFallbackMeshLayersChange}
onStatsChange={onFallbackMeshStatsChange}
previewImageUrl={previewImageUrl}
resetViewToken={resetViewToken}
settings={fallbackMeshSettings || DEFAULT_MESH_SETTINGS}
showMovementControls={showMovementControls}
/>
);
}
}, [handleSplatLoadStateChange, resolvedSplatUrl]);
if (!resolvedSplatUrl) {
return (
@ -2375,13 +2323,8 @@ export function World3DViewer({
<WorldViewerInteractionSurface className={className}>
<WorldSplatViewer
firstPerson={firstPerson}
fallbackMeshLayerVisibility={meshLayerVisibility}
fallbackMeshSettings={meshSettings}
fallbackMeshUrl={meshUrl}
hotspots={hotspots}
interactionMode={interactionMode}
onFallbackMeshLayersChange={onMeshLayersChange}
onFallbackMeshStatsChange={onMeshStatsChange}
onCaptureReady={onCaptureReady}
onHotspotSelect={onHotspotSelect}
onPick={onPick}

View File

@ -282,6 +282,7 @@ export function WorldSharePage({
mode={activeView}
previewImageUrl={activeWorld?.previewImageUrl || null}
semanticsMetadata={activeWorld?.semanticsMetadata || null}
showInfoOverlay={false}
showMovementControls={activeView === "splat" || activeView === "mesh"}
splatUrl={activeWorld?.splatUrl || null}
worldUrl={activeWorld?.worldUrl || null}

Some files were not shown because too many files have changed in this diff Show More