fix: harden media upload and release safety

Protect upload cleanup and atomic completion paths, add durable media worker leases and brand asset validation, and introduce verified migration/bootstrap/deployment gates.
This commit is contained in:
chenpei 2026-08-12 18:04:08 +08:00
parent 17d45352d3
commit 7326e273e1
219 changed files with 32536 additions and 5834 deletions

View File

@ -54,6 +54,8 @@ prisma/engine-cache
prisma/engine-cache/** prisma/engine-cache/**
prisma/migrations prisma/migrations
prisma/migrations/** prisma/migrations/**
!prisma/migrations/
!prisma/migrations/**
# 备份文件 # 备份文件
backups backups
@ -121,6 +123,10 @@ scripts
scripts/** scripts/**
!scripts/ !scripts/
!scripts/public-traffic-rollup-from-nginx.py !scripts/public-traffic-rollup-from-nginx.py
!scripts/collapse-performance-survey-template-versions.mjs
!scripts/check-legacy-storage-boundaries.mjs
!scripts/safe-production-migrate.mjs
!scripts/create-verified-production-backup.mjs
__tests__ __tests__
*.test.ts *.test.ts
*.test.js *.test.js
@ -185,51 +191,7 @@ missing-*.json
missing-*.txt missing-*.txt
temp_*.sql temp_*.sql
# SQL 文件和数据文件 # Docker ignore 使用“最后匹配规则生效”。上面的全局 *.sql 规则不能再次
*.sql # 排除 Prisma 迁移,否则 builder/migrator 会在发布时缺少 migration.sql。
*.csv !prisma/migrations/
*.json !prisma/migrations/**
!package.json
!package-lock.json
!tsconfig.json
!next.config.js
!tailwind.config.ts
!postcss.config.js
!jest.config.js
!jest.setup.js
*.txt
!README.md
# 旧的 JS 文件(不在 app 或 components 中)
shadow_*.js
*_page*.js
delete-text-materials.js
find-text-materials.js
# 部署和配置文件(不需要在容器中)
docker-compose*.yml
deploy-*.ps1
check-*.ps1
sync-*.ps1
verify-*.ps1
install-*.ps1
quick-*.ps1
cleanup-*.ps1
restore-*.sh
run-*.ps1
test-*.ps1
# 报告和分析文件
*-report.csv
*-report.json
*-stats*.csv
*-stats*.txt
*-inventory*.csv
*-inventory*.json
material-inventory-*.csv
material-inventory-*.json
old-materials-*.csv
missing-*.csv
missing-*.json
missing-*.txt
temp_*.sql

View File

@ -1,5 +1,8 @@
# 正式系统功能分析与对比报告 # 正式系统功能分析与对比报告
> **历史快照(2025-12-20),不得作为当前生产配置或存储写入依据。**
> 当前主站与资源入口以 `docs/NETWORK_TOPOLOGY.md` 和 `docs/生产存储桶接入信息.md` 为准;文内旧域名只能用于理解历史状态。
## 📋 文档说明 ## 📋 文档说明
本文档通过对比正式系统(192.168.58.20:4320)和本地源码,系统性分析所有功能的实现状态。 本文档通过对比正式系统(192.168.58.20:4320)和本地源码,系统性分析所有功能的实现状态。
@ -483,14 +486,13 @@
#### 6.1 视频转码 #### 6.1 视频转码
- **自动转码** ✅ - **自动转码** ✅
- 工具: `lib/video-processor.ts` - 工具: `lib/video-processor.ts`
- H.265 (HEVC) 编码 - 默认 720P H.265 (HEVC) 主预览,另生成 10 秒片段和封面
- 高清晰度
- 状态:已实现 - 状态:已实现
- **转码队列** ✅ - **转码队列** ✅
- 工具: `lib/video-transcode-queue.ts` - 工具: `lib/video-transcode-queue.ts`
- Redis 队列管理 - **事故时旧实现**:Redis 队列管理;当前视频任务由 PostgreSQL 租约持久化
- 自动恢复 - 当前由启动扫描和每 60 秒恢复扫描接回 `pending/processing` 任务
- 状态:已实现 - 状态:已实现
- **转码重试** ✅ - **转码重试** ✅
@ -765,10 +767,9 @@
- 提升上传速度 - 提升上传速度
### 7. 视频转码 ### 7. 视频转码
- H.265 (HEVC) 编码 - 默认 720P H.265 (HEVC) 主预览 + 10 秒片段 + 封面
- 高清晰度 - **事故时旧实现**使用 Redis 队列;当前由 PostgreSQL 租约持久化,不依赖 Redis 视频队列
- Redis 队列管理 - 启动时恢复,之后每 60 秒重扫;状态为 `pending → processing → done/failed`
- 自动恢复
--- ---
@ -952,8 +953,8 @@ DATABASE_URL=postgresql://user:password@host:port/database
NEXTAUTH_URL=http://your-domain.com NEXTAUTH_URL=http://your-domain.com
NEXTAUTH_SECRET=your-secret-key NEXTAUTH_SECRET=your-secret-key
# S3 存储 # S3 存储(本段为 2025-12-20 历史示例,当前 endpoint/public domain 以生产存储文档为准)
S3_ENDPOINT=https://s3.yydd.ai S3_ENDPOINT=https://s3.yydd.ai # 历史值,不作为当前生产入口
S3_ACCESS_KEY_ID=your-access-key S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key S3_SECRET_ACCESS_KEY=your-secret-key
S3_BUCKET=gongxiangpan S3_BUCKET=gongxiangpan
@ -986,10 +987,10 @@ OPENAI_BASE_URL=https://api.openai.com/v1
- 地址:`192.168.58.20:5432` - 地址:`192.168.58.20:5432`
### 存储配置 ### 存储配置
- S3 Endpoint: `https://s3.yydd.ai` - S3 Endpoint(历史快照值): `https://s3.yydd.ai`;当前以 `docs/生产存储桶接入信息.md` 为准
- Bucket: `gongxiangpan` - Bucket: `gongxiangpan`
- 文件前缀: `由壹点AI/` - 文件前缀: `由壹点AI/`
- 本地存储: `./uploads` - 视频预览 / 片段 / 封面:S3 对象;canonical 源视频:生产机 uploads/NAS(容器内 `/app/uploads`)
--- ---
@ -1053,7 +1054,7 @@ OPENAI_BASE_URL=https://api.openai.com/v1
- 分块上传 + 断点续传 - 分块上传 + 断点续传
- 前端图片压缩 - 前端图片压缩
- 视频自动转码 - 视频自动转码
- Redis 队列管理 - **事故时旧实现** Redis 视频队列;当前使用 PostgreSQL 租约 + 启动/60 秒恢复
- 滚动位置保存 - 滚动位置保存
- 搜索防抖优化 - 搜索防抖优化

View File

@ -28,9 +28,9 @@ Next.js 14 + React 18 + TypeScript + TailwindCSS + Prisma + PostgreSQL + MinIO S
## 生产环境 ## 生产环境
- 数据库: `lobechat` @ `192.168.58.20:5432`(影子: `lobechat_stage`) - 数据库: `lobechat` @ `192.168.58.20:5432`(影子: `lobechat_stage`)
- S3: `https://s3.yydd.ai`,Bucket: `gongxiangpan`,前缀: `由壹点AI/` - S3: `https://s3.jiansun.vip`,Bucket: `gongxiangpan`,前缀: `由壹点AI/`
- 正式: 端口4320 | 影子: 端口5320 - 正式: 端口4320 | 影子: 端口5320
- 本地存储: `/mnt/PoolOne/apps/youyidian-3d/uploads` - 本地存储: `/mnt/PoolOne/youyidian-uploads`(当前正式独立 ZFS 数据集);历史路径 `/mnt/PoolOne/apps/youyidian-3d/uploads` 仅在人工显式选择兼容模式时使用,禁止据旧容器或旧文档自动推断。
--- ---
@ -70,10 +70,12 @@ ssh root@192.168.58.20 "docker exec -i gongxiangpan-postgres-1 psql -U gongxiang
| 项目 | 老素材 | 新素材 | | 项目 | 老素材 | 新素材 |
|------|--------|--------| |------|--------|--------|
| S3 | `s3.1080p.hk/jsxiangche` | `s3.yydd.ai/gongxiangpan/由壹点AI/` | | S3 | `s3.1080p.hk/jsxiangche`(仅历史读取) | `s3.jiansun.vip/gongxiangpan/由壹点AI/` |
| 识别 | `previewUrl`不含`由壹点AI/` | `previewUrl`含`由壹点AI/` | | 识别 | `previewUrl`不含`由壹点AI/` | `previewUrl`含`由壹点AI/` |
**原则**: 不改老素材 | 视频/音频存本地(MinIO机械盘慢)| 下载兼容新老路径 **原则**: 老素材地址只做读取兼容,不得用于默认值、新上传、重置或新数据库写入 | 视频/音频源文件存本地 | 下载兼容新老路径
站点 Header Logo、登录 Logo、favicon 属于当前系统配置,不属于“老素材不改”的范围;它们必须使用仓库内置品牌资源或当前 `s3.jiansun.vip/gongxiangpan/` 对象。
--- ---

View File

@ -420,11 +420,12 @@ lib/
- 使用 Sharp 库处理 - 使用 Sharp 库处理
**视频处理**(`lib/video-processor.ts`): **视频处理**(`lib/video-processor.ts`):
- 原视频 → 540p 预览 - canonical 源视频(生产机 `UPLOAD_DIR` / uploads/NAS)→ 默认 720P H.265(HEVC)主预览
- 生成片段(前30秒) - 生成 10 秒 H.265 预览片段
- 提取封面 - 提取封面
- 基于 FFMPEG 转码 - 基于 FFmpeg 转码;同一应用实例最多 1 条任务、FFmpeg 默认 1 线程
- 队列式处理 - 任务由 PostgreSQL 租约持久化,启动时恢复并每 60 秒重扫;Redis 只用于上传会话等业务,不承担视频队列
- 上传建档后异步经历 `pending → processing → done/failed`
**音频处理**(`lib/audio-processor.ts`): **音频处理**(`lib/audio-processor.ts`):
- 元数据提取(艺术家、专辑、时长) - 元数据提取(艺术家、专辑、时长)
@ -557,7 +558,7 @@ catch (error) {
2. **竞态条件** - 处理过程中避免重复更新 `previewUrl` 2. **竞态条件** - 处理过程中避免重复更新 `previewUrl`
3. **软删除** - 素材有 `deletedAt` 字段 - 始终过滤掉这些记录 3. **软删除** - 素材有 `deletedAt` 字段 - 始终过滤掉这些记录
4. **任务号格式** - 格式:`TASK` + 12-18位数字 + 可选的 `-XX` 后缀 4. **任务号格式** - 格式:`TASK` + 12-18位数字 + 可选的 `-XX` 后缀
5. **预览生成** - 标记上传完成前必须完成预览生成 5. **预览生成** - 上传先完成建档并进入 `pending`,预览由后台异步生成;三类产物完整且租约仍有效才是 `done`,失败写入 `failed`
6. **Docker 网络** - 容器必须在 `youyidian-3d_app-network` 上才能访问 Redis 6. **Docker 网络** - 容器必须在 `youyidian-3d_app-network` 上才能访问 Redis
7. **视频类型权限** - 视频类型改名后需同步更新部门权限表,或依赖代码兼容逻辑 7. **视频类型权限** - 视频类型改名后需同步更新部门权限表,或依赖代码兼容逻辑
8. **PDF 中文文件名** - 必须使用 `execFile` 而非 `exec` 调用 pdftoppm,避免路径注入 8. **PDF 中文文件名** - 必须使用 `execFile` 而非 `exec` 调用 pdftoppm,避免路径注入

View File

@ -1,6 +1,12 @@
# 基础镜像(使用 Alpine,手动处理 OpenSSL 兼容性) # 基础镜像(使用 Alpine,手动处理 OpenSSL 兼容性)
ARG BASE_IMAGE=node:18-alpine # 2026-08-12 通过 `docker buildx imagetools inspect node:18-alpine` 核验的
# Docker Hub 官方多架构清单 digest。生产默认值必须是不可变引用;本地测试
# 如需改镜像源,仍可显式传入 BASE_IMAGE,但发布脚本不得覆盖此默认值。
ARG BASE_IMAGE=node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
ARG ALPINE_MIRROR=https://mirrors.tuna.tsinghua.edu.cn/alpine ARG ALPINE_MIRROR=https://mirrors.tuna.tsinghua.edu.cn/alpine
# Local/CI builds remain possible without Git metadata. Production deployment
# supplies the exact SHA and rejects this sentinel before any container starts.
ARG VCS_REF=0000000000000000000000000000000000000000
FROM ${BASE_IMAGE} AS base FROM ${BASE_IMAGE} AS base
# 安装依赖阶段 # 安装依赖阶段
@ -10,23 +16,25 @@ RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk
apk add --no-cache libc6-compat python3 make g++ cairo-dev jpeg-dev pango-dev giflib-dev apk add --no-cache libc6-compat python3 make g++ cairo-dev jpeg-dev pango-dev giflib-dev
WORKDIR /app WORKDIR /app
# 复制 package.json(利用 Docker 层缓存) # package-lock.json 是生产构建的必需输入,缺失时 COPY 直接失败。
COPY package.json package-lock.json* ./ COPY package.json package-lock.json ./
# canvas 在项目里是可选依赖,且服务端代码已做降级处理。
# 构建机当前卡在 canvas 的 node-pre-gyp 安装阶段,因此这里只在镜像构建时跳过它。
RUN node -e "const fs=require('fs'); const pkg=require('./package.json'); if (pkg.optionalDependencies && pkg.optionalDependencies.canvas) { delete pkg.optionalDependencies.canvas; fs.writeFileSync('./package.json', JSON.stringify(pkg, null, 2) + '\\n'); }"
# 安装依赖(使用淘宝镜像 + BuildKit缓存) # 安装依赖(使用淘宝镜像 + BuildKit缓存)
RUN --mount=type=cache,target=/root/.npm \ RUN --mount=type=cache,target=/root/.npm \
manifest_sha="$(sha256sum package.json | awk '{print $1}')" && \
lock_sha="$(sha256sum package-lock.json | awk '{print $1}')" && \
echo "Installing dependencies (using cache if available)..." && \ echo "Installing dependencies (using cache if available)..." && \
npm config set registry https://registry.npmmirror.com && \ npm config set registry https://registry.npmmirror.com && \
npm install --prefer-offline --no-audit --no-fund --legacy-peer-deps && \ npm ci --prefer-offline --no-audit --no-fund --legacy-peer-deps && \
test "$manifest_sha" = "$(sha256sum package.json | awk '{print $1}')" && \
test "$lock_sha" = "$(sha256sum package-lock.json | awk '{print $1}')" && \
node -e "const sharp=require('sharp'); sharp(Buffer.alloc(4),{raw:{width:1,height:1,channels:4}}).png().toBuffer().then(()=>console.log('sharp dependency smoke passed')).catch((error)=>{console.error(error);process.exit(1)})" && \
echo "Dependencies installed ($(du -sh /app/node_modules | cut -f1))" echo "Dependencies installed ($(du -sh /app/node_modules | cut -f1))"
# 构建阶段 # 构建阶段
FROM base AS builder FROM base AS builder
ARG ALPINE_MIRROR ARG ALPINE_MIRROR
ARG VCS_REF
# 安装构建依赖 # 安装构建依赖
RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk/repositories && \ RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk/repositories && \
apk add --no-cache libc6-compat python3 make g++ cairo-dev jpeg-dev pango-dev giflib-dev ffmpeg openssl apk add --no-cache libc6-compat python3 make g++ cairo-dev jpeg-dev pango-dev giflib-dev ffmpeg openssl
@ -34,6 +42,26 @@ WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules COPY --from=deps /app/node_modules ./node_modules
COPY . . COPY . .
# The source revision is immutable image metadata, not a runtime environment
# assertion. Production builds must provide the exact 40-character Git SHA.
RUN case "$VCS_REF" in \
[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) ;; \
*) echo "VCS_REF must be the exact 40-character lowercase Git SHA" >&2; exit 1 ;; \
esac && \
printf '%s\n' "$VCS_REF" > /app/DEPLOYMENT_COMMIT && \
chmod 0444 /app/DEPLOYMENT_COMMIT
# 品牌资源是页面和 favicon 的离线兜底;缺失或空文件时禁止继续构建。
RUN test -s "/app/public/brand/header-logo.png" && \
test -s "/app/public/brand/login-logo.png" && \
test -s "/app/public/brand/favicon.png" && \
node -e "const sharp=require('sharp'); Promise.all(['header-logo.png','login-logo.png','favicon.png'].map((name)=>sharp('/app/public/brand/'+name,{failOn:'error',limitInputPixels:25000000,animated:false}).rotate().toBuffer())).then(()=>console.log('builder brand image decode smoke passed')).catch((error)=>{console.error(error);process.exit(1)})" && \
test -f "/app/prisma/migrations/20251118000000_add_upload_id_to_materials/migration.sql" && \
test -f "/app/prisma/migrations/20260811_add_video_transcode_lease/migration.sql" && \
test -f "/app/prisma/migrations/20260811_update_brand_logo_defaults/migration.sql" && \
test -f "/app/prisma/migrations/20260812_add_product_package_upload_job_lease/migration.sql" && \
test -f "/app/prisma/migrations/20260812_align_material_upload_idempotency_index/migration.sql"
# 这两个动态路由负责兜底访问本地上传文件;如果它们在构建上下文里丢失, # 这两个动态路由负责兜底访问本地上传文件;如果它们在构建上下文里丢失,
# 线上音频/视频就会直接 404。这里提前失败,避免再把缺路由的镜像发出去。 # 线上音频/视频就会直接 404。这里提前失败,避免再把缺路由的镜像发出去。
RUN test -f "/app/app/api/uploads/[...path]/route.ts" && \ RUN test -f "/app/app/api/uploads/[...path]/route.ts" && \
@ -68,16 +96,48 @@ RUN echo "清理 standalone 目录..." && \
rm -rf .next/standalone/scripts && \ rm -rf .next/standalone/scripts && \
echo "清理完成" echo "清理完成"
# Explicit, one-shot database migrator. It is never used by the application
# container or the normal deploy entry. The operator must supply production DB
# identity, exact source SHA and a hashed backup receipt before --apply works.
FROM base AS migrator
ARG ALPINE_MIRROR
ARG VCS_REF
WORKDIR /app
LABEL org.opencontainers.image.revision=$VCS_REF
ENV NODE_ENV=production
RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk/repositories && \
apk add --no-cache openssl postgresql-client && \
command -v openssl >/dev/null && \
command -v pg_dump >/dev/null && \
command -v pg_restore >/dev/null && \
command -v psql >/dev/null
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/scripts/safe-production-migrate.mjs ./scripts/safe-production-migrate.mjs
COPY --from=builder /app/scripts/create-verified-production-backup.mjs ./scripts/create-verified-production-backup.mjs
COPY --from=builder /app/DEPLOYMENT_COMMIT /app/DEPLOYMENT_COMMIT
RUN test "$(cat /app/DEPLOYMENT_COMMIT)" = "$VCS_REF" && \
chmod -R a-w /app/prisma /app/scripts /app/DEPLOYMENT_COMMIT
USER node
ENTRYPOINT ["node", "scripts/safe-production-migrate.mjs"]
# 生产运行阶段 # 生产运行阶段
FROM base AS runner FROM base AS runner
ARG VCS_REF
WORKDIR /app WORKDIR /app
LABEL org.opencontainers.image.revision=$VCS_REF
ENV NODE_ENV=production ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1 ENV NEXT_TELEMETRY_DISABLED=1
ARG ALPINE_MIRROR ARG ALPINE_MIRROR
RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk/repositories && \ RUN sed -i "s|https://dl-cdn.alpinelinux.org/alpine|${ALPINE_MIRROR}|g" /etc/apk/repositories && \
apk add --no-cache ffmpeg python3 cairo jpeg pango giflib openssl poppler-utils apk add --no-cache ffmpeg python3 cairo jpeg pango giflib openssl poppler-utils flock && \
command -v flock >/dev/null && \
flock --exclusive --timeout 1 --conflict-exit-code 75 --no-fork \
/tmp/material-upload-flock-probe.lock sh -c 'true' && \
rm -f /tmp/material-upload-flock-probe.lock
RUN addgroup --system --gid 1001 nodejs && \ RUN addgroup --system --gid 1001 nodejs && \
adduser --system --uid 1001 nextjs adduser --system --uid 1001 nextjs
@ -88,6 +148,8 @@ RUN mkdir -p /app/scripts
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/app ./app COPY --from=builder --chown=nextjs:nodejs /app/app ./app
COPY --from=builder --chown=root:root /app/public ./public
COPY --from=builder --chown=root:root /app/DEPLOYMENT_COMMIT /app/DEPLOYMENT_COMMIT
# 只复制运行时必需的 node_modules # 只复制运行时必需的 node_modules
COPY --from=builder --chown=nextjs:nodejs /app/node_modules/.prisma ./node_modules/.prisma COPY --from=builder --chown=nextjs:nodejs /app/node_modules/.prisma ./node_modules/.prisma
@ -97,8 +159,13 @@ COPY --from=builder --chown=nextjs:nodejs /app/node_modules/@ioredis ./node_modu
COPY --from=builder --chown=nextjs:nodejs /app/node_modules/@aws-sdk ./node_modules/@aws-sdk COPY --from=builder --chown=nextjs:nodejs /app/node_modules/@aws-sdk ./node_modules/@aws-sdk
COPY --from=builder --chown=nextjs:nodejs /app/node_modules/@smithy ./node_modules/@smithy COPY --from=builder --chown=nextjs:nodejs /app/node_modules/@smithy ./node_modules/@smithy
# 只复制 schema.prisma,不复制 migrations 和 engine-cache(减少 35MB) # 复制运行时 schema,以及切流前只读校验所需的精确 migration;仍不复制 engine-cache。
COPY --from=builder --chown=nextjs:nodejs /app/prisma/schema.prisma ./prisma/schema.prisma COPY --from=builder --chown=nextjs:nodejs /app/prisma/schema.prisma ./prisma/schema.prisma
COPY --from=builder --chown=root:root /app/prisma/migrations/20251118000000_add_upload_id_to_materials ./prisma/migrations/20251118000000_add_upload_id_to_materials
COPY --from=builder --chown=root:root /app/prisma/migrations/20260811_add_video_transcode_lease ./prisma/migrations/20260811_add_video_transcode_lease
COPY --from=builder --chown=root:root /app/prisma/migrations/20260811_update_brand_logo_defaults ./prisma/migrations/20260811_update_brand_logo_defaults
COPY --from=builder --chown=root:root /app/prisma/migrations/20260812_add_product_package_upload_job_lease ./prisma/migrations/20260812_add_product_package_upload_job_lease
COPY --from=builder --chown=root:root /app/prisma/migrations/20260812_align_material_upload_idempotency_index ./prisma/migrations/20260812_align_material_upload_idempotency_index
COPY --from=builder --chown=nextjs:nodejs /app/lib ./lib COPY --from=builder --chown=nextjs:nodejs /app/lib ./lib
COPY --from=builder --chown=nextjs:nodejs /app/scripts/public-traffic-rollup-from-nginx.py ./scripts/public-traffic-rollup-from-nginx.py COPY --from=builder --chown=nextjs:nodejs /app/scripts/public-traffic-rollup-from-nginx.py ./scripts/public-traffic-rollup-from-nginx.py
COPY --from=builder --chown=nextjs:nodejs /app/scripts/collapse-performance-survey-template-versions.mjs ./scripts/collapse-performance-survey-template-versions.mjs COPY --from=builder --chown=nextjs:nodejs /app/scripts/collapse-performance-survey-template-versions.mjs ./scripts/collapse-performance-survey-template-versions.mjs
@ -115,7 +182,13 @@ RUN mkdir -p /app/public /app/uploads && \
mkdir -p /app/uploads/cleanup && \ mkdir -p /app/uploads/cleanup && \
chown -R root:root /app && \ chown -R root:root /app && \
chown -R nextjs:nodejs /app/.next/cache /app/uploads && \ chown -R nextjs:nodejs /app/.next/cache /app/uploads && \
chmod 0750 /app/uploads /app/uploads/temp /app/uploads/assembled /app/uploads/materials /app/uploads/cleanup chmod 0750 /app/uploads /app/uploads/temp /app/uploads/assembled /app/uploads/materials /app/uploads/cleanup && \
test "$(cat /app/DEPLOYMENT_COMMIT)" = "$VCS_REF" && \
chmod 0444 /app/DEPLOYMENT_COMMIT && \
test -s /app/public/brand/header-logo.png && \
test -s /app/public/brand/login-logo.png && \
test -s /app/public/brand/favicon.png && \
node -e "const sharp=require('sharp'); Promise.all(['header-logo.png','login-logo.png','favicon.png'].map((name)=>sharp('/app/public/brand/'+name,{failOn:'error',limitInputPixels:25000000,animated:false}).rotate().toBuffer())).then(()=>console.log('runner brand image decode smoke passed')).catch((error)=>{console.error(error);process.exit(1)})"
USER nextjs USER nextjs

View File

@ -42,7 +42,8 @@
- SKU 维度文件管理、排序、批量下载 - SKU 维度文件管理、排序、批量下载
- 详情图 / 素材资产管理 - 详情图 / 素材资产管理
- 商品套包未完成上传的中止与回滚 - 商品套包未完成上传的中止与回滚
- 视频转码、封面 / 预览片段生成、转码队列与重试 - 视频转码、封面 / 预览片段生成、PostgreSQL 租约任务恢复与重试
- 默认生成 720P H.265 主预览和 10 秒片段;素材建档后异步经历 `pending → processing → done/failed`
### 4. 后台与权限 ### 4. 后台与权限
@ -139,7 +140,7 @@ NEXTAUTH_SECRET=replace-this-with-a-strong-secret
NEXTAUTH_URL=http://localhost:3000 NEXTAUTH_URL=http://localhost:3000
APP_PUBLIC_URL=http://localhost:3000 APP_PUBLIC_URL=http://localhost:3000
# 本地文件目录 # 仅限直接运行 `npm run dev` 的非容器本地开发目录
UPLOAD_DIR=./uploads UPLOAD_DIR=./uploads
# S3 / MinIO # S3 / MinIO
@ -179,7 +180,8 @@ SERVER_IP=127.0.0.1
MINIO_API_PORT=9000 MINIO_API_PORT=9000
# 视频转码(按需) # 视频转码(按需)
VIDEO_TRANSCODE_MAX_CONCURRENT=2 VIDEO_TRANSCODE_MAX_CONCURRENT=1
VIDEO_TRANSCODE_THREADS=1
VIDEO_TRANSCODE_TIMEOUT=3600000 VIDEO_TRANSCODE_TIMEOUT=3600000
VIDEO_PREVIEW_RESOLUTION=720 VIDEO_PREVIEW_RESOLUTION=720
VIDEO_CLIP_DURATION=10 VIDEO_CLIP_DURATION=10
@ -216,6 +218,23 @@ WHERE email = 'your@email.com';
npm run dev npm run dev
``` ```
直接运行 `npm run dev` 时,`UPLOAD_DIR=./uploads` 只保存本机开发数据。容器化的本地集成测试请使用 `docker-compose.local-test.yml` 中的独立命名卷,不要把相对目录配置复制到默认 Compose 或生产环境。
### 6. 默认 Compose 的 uploads 挂载门禁
根目录 `docker-compose.yml` 是生产形态的单机参考配置(正式发布仍以 `scripts/git-incremental-deploy.ps1` 为准)。它不会再自动创建或挂载仓库内的 `./uploads`,启动前必须显式提供已经核验的宿主机 uploads 挂载:
```bash
export GONGXIANGP2_UPLOADS_MOUNT=/mnt/PoolOne/youyidian-uploads
docker compose config
docker compose up -d
```
- 当前生产宿主机的真实独立 ZFS 数据集路径是 `/mnt/PoolOne/youyidian-uploads`;先在宿主机用 `findmnt` 核验它确实是预期挂载,不能只确认目录存在。
- 容器内统一使用 `UPLOAD_DIR=/app/uploads`,不要把宿主机路径写进容器环境变量。
- `create_host_path: false` 会在路径缺失时直接失败,避免 Docker 静默新建空目录后让素材写入错误磁盘。
- 相对 `./uploads` 只允许用于明确的非容器开发;影子环境使用其独立的 `uploads-shadow`,本地集成测试使用命名卷,二者都不能当作生产数据目录。
默认访问地址: 默认访问地址:
```text ```text
@ -279,7 +298,7 @@ npm run start
- 登录是否正常 - 登录是否正常
- 首页筛选、上传、下载、分享是否正常 - 首页筛选、上传、下载、分享是否正常
- 管理后台核心页签是否可用 - 管理后台核心页签是否可用
- 视频转码队列是否异常堆积 - 视频转码数据库任务 / 租约是否异常堆积(Redis 仅检查上传会话,不作为视频队列)
- `/api/health` 是否正常 - `/api/health` 是否正常
- 如果改了网络或下载链路,再验证一次内外网访问 - 如果改了网络或下载链路,再验证一次内外网访问

View File

@ -2,6 +2,8 @@ import { fireEvent, render, screen, waitFor } from '@testing-library/react'
import { useSession } from 'next-auth/react' import { useSession } from 'next-auth/react'
import SystemSettings from '@/app/admin/settings/page' import SystemSettings from '@/app/admin/settings/page'
const showToastMock = jest.fn()
jest.mock('next-auth/react', () => ({ jest.mock('next-auth/react', () => ({
useSession: jest.fn(), useSession: jest.fn(),
})) }))
@ -21,7 +23,7 @@ jest.mock('@/components/Header', () => ({
jest.mock('@/hooks/useToast', () => ({ jest.mock('@/hooks/useToast', () => ({
useToast: () => ({ useToast: () => ({
toast: null, toast: null,
showToast: jest.fn(), showToast: showToastMock,
}), }),
})) }))
@ -44,8 +46,10 @@ describe('系统设置页凭据脱敏', () => {
json: async () => ({ json: async () => ({
headerBgColor: '#1f2937', headerBgColor: '#1f2937',
siteName: '由壹点3D素材', siteName: '由壹点3D素材',
headerLogoUrl: '', headerLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/header.png',
loginLogoUrl: '', headerLogoPreviewUrl: '/api/proxy-media?url=header-preview',
loginLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/login.png',
loginLogoPreviewUrl: '/api/proxy-media?url=login-preview',
s3Endpoint: 'https://storage.example.com', s3Endpoint: 'https://storage.example.com',
s3Bucket: 'materials', s3Bucket: 'materials',
s3Region: 'us-east-1', s3Region: 'us-east-1',
@ -94,5 +98,67 @@ describe('系统设置页凭据脱敏', () => {
const payload = JSON.parse(requestInit.body) const payload = JSON.parse(requestInit.body)
expect(payload).not.toHaveProperty('s3AccessKey') expect(payload).not.toHaveProperty('s3AccessKey')
expect(payload).not.toHaveProperty('s3SecretKey') expect(payload).not.toHaveProperty('s3SecretKey')
expect(payload.headerLogoUrl).toBe('https://s3.jiansun.vip/gongxiangpan/brand/header.png')
expect(payload.loginLogoUrl).toBe('https://s3.jiansun.vip/gongxiangpan/brand/login.png')
expect(JSON.stringify(payload)).not.toContain('/api/proxy-media')
})
test('恢复默认使用镜像内置品牌图,不再写入旧存储域名', async () => {
render(<SystemSettings />)
await waitFor(() => {
expect(screen.getByText('已配置')).toBeInTheDocument()
})
const resetButtons = screen.getAllByRole('button', { name: '恢复默认' })
fireEvent.click(resetButtons[0])
fireEvent.click(resetButtons[1])
fireEvent.click(screen.getByRole('button', { name: '保存设置' }))
await waitFor(() => {
expect(global.fetch).toHaveBeenCalledTimes(2)
})
const [, requestInit] = (global.fetch as jest.Mock).mock.calls[1]
const payload = JSON.parse(requestInit.body)
expect(payload.headerLogoUrl).toBe('/brand/header-logo.png')
expect(payload.loginLogoUrl).toBe('/brand/login-logo.png')
expect(JSON.stringify(payload)).not.toContain('s3.1080p.hk')
})
test('文件选择器和前端校验只接受服务端支持的三种 Logo 格式', async () => {
const { container } = render(<SystemSettings />)
await waitFor(() => {
expect(screen.getByText('已配置')).toBeInTheDocument()
})
const inputs = Array.from(container.querySelectorAll<HTMLInputElement>('input[type="file"]'))
expect(inputs).toHaveLength(2)
expect(inputs.every((input) => input.accept === 'image/png,image/jpeg,image/webp')).toBe(true)
fireEvent.change(inputs[0], {
target: { files: [new File(['<svg />'], 'logo.svg', { type: 'image/svg+xml' })] },
})
expect(showToastMock).toHaveBeenCalledWith(
'LOGO仅支持 PNG、JPEG 或 WebP 格式',
{ variant: 'error' },
)
expect(global.fetch).toHaveBeenCalledTimes(1)
})
test.each([
{ user: { id: 'system-admin-1', role: 'SYSTEM_ADMIN' }, label: 'SYSTEM_ADMIN' },
{ user: { id: 'super-admin-1', role: 'USER', isSuperAdmin: true }, label: '超级管理员' },
])('$label 权限账号可以正常看到设置页面', async ({ user }) => {
;(useSession as jest.Mock).mockReturnValue({
data: { user },
status: 'authenticated',
})
render(<SystemSettings />)
expect(await screen.findByRole('heading', { name: '系统设置' })).toBeInTheDocument()
}) })
}) })

View File

@ -4,9 +4,14 @@ import { useEffect, useState } from 'react'
import { useSession } from 'next-auth/react' import { useSession } from 'next-auth/react'
import { useRouter } from 'next/navigation' import { useRouter } from 'next/navigation'
import Header from '@/components/Header' import Header from '@/components/Header'
import BrandLogo from '@/components/BrandLogo'
import { useToast } from '@/hooks/useToast' import { useToast } from '@/hooks/useToast'
import ToastBubble from '@/components/ToastBubble' import ToastBubble from '@/components/ToastBubble'
import { parseFetchError, handleError } from '@/lib/error-handler' import { parseFetchError, handleError } from '@/lib/error-handler'
import { DEFAULT_HEADER_LOGO_URL, DEFAULT_LOGIN_LOGO_URL } from '@/lib/brand-assets'
const BRAND_LOGO_ACCEPT = 'image/png,image/jpeg,image/webp'
const SUPPORTED_BRAND_LOGO_MIME_TYPES = new Set(BRAND_LOGO_ACCEPT.split(','))
type SettingsSessionUser = { type SettingsSessionUser = {
role?: string | null role?: string | null
@ -83,10 +88,10 @@ export default function SystemSettings() {
const data = await response.json() const data = await response.json()
setHeaderBgColor(data.headerBgColor || '#1f2937') setHeaderBgColor(data.headerBgColor || '#1f2937')
setSiteName(data.siteName || '由壹点3D素材') setSiteName(data.siteName || '由壹点3D素材')
setHeaderLogoUrl(data.headerLogoUrl || '') setHeaderLogoUrl(data.headerLogoUrl || DEFAULT_HEADER_LOGO_URL)
setHeaderLogoPreview(data.headerLogoUrl || '') setHeaderLogoPreview(data.headerLogoPreviewUrl || data.headerLogoUrl || DEFAULT_HEADER_LOGO_URL)
setLoginLogoUrl(data.loginLogoUrl || '') setLoginLogoUrl(data.loginLogoUrl || DEFAULT_LOGIN_LOGO_URL)
setLoginLogoPreview(data.loginLogoUrl || '') setLoginLogoPreview(data.loginLogoPreviewUrl || data.loginLogoUrl || DEFAULT_LOGIN_LOGO_URL)
setS3Endpoint(data.s3Endpoint || '') setS3Endpoint(data.s3Endpoint || '')
setNewS3AccessKey('') setNewS3AccessKey('')
setNewS3SecretKey('') setNewS3SecretKey('')
@ -174,8 +179,8 @@ export default function SystemSettings() {
if (!file) return if (!file) return
// 验证文件类型 // 验证文件类型
if (!file.type.startsWith('image/')) { if (!SUPPORTED_BRAND_LOGO_MIME_TYPES.has(file.type.toLowerCase())) {
showToast('请选择图片文件', { variant: 'error' }) showToast('LOGO仅支持 PNG、JPEG 或 WebP 格式', { variant: 'error' })
return return
} }
@ -202,7 +207,6 @@ export default function SystemSettings() {
setHeaderLogoPreview(data.logoUrl) setHeaderLogoPreview(data.logoUrl)
showToast('导航栏LOGO上传成功!', { variant: 'success' }) showToast('导航栏LOGO上传成功!', { variant: 'success' })
} else { } else {
const error = await response.json()
const parsedError = await parseFetchError(response) const parsedError = await parseFetchError(response)
showToast(parsedError.message, { variant: 'error' }) showToast(parsedError.message, { variant: 'error' })
} }
@ -220,8 +224,8 @@ export default function SystemSettings() {
if (!file) return if (!file) return
// 验证文件类型 // 验证文件类型
if (!file.type.startsWith('image/')) { if (!SUPPORTED_BRAND_LOGO_MIME_TYPES.has(file.type.toLowerCase())) {
showToast('请选择图片文件', { variant: 'error' }) showToast('LOGO仅支持 PNG、JPEG 或 WebP 格式', { variant: 'error' })
return return
} }
@ -248,7 +252,6 @@ export default function SystemSettings() {
setLoginLogoPreview(data.logoUrl) setLoginLogoPreview(data.logoUrl)
showToast('登录页LOGO上传成功!', { variant: 'success' }) showToast('登录页LOGO上传成功!', { variant: 'success' })
} else { } else {
const error = await response.json()
const parsedError = await parseFetchError(response) const parsedError = await parseFetchError(response)
showToast(parsedError.message, { variant: 'error' }) showToast(parsedError.message, { variant: 'error' })
} }
@ -266,13 +269,13 @@ export default function SystemSettings() {
} }
const handleHeaderLogoReset = () => { const handleHeaderLogoReset = () => {
setHeaderLogoPreview('https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png') setHeaderLogoPreview(DEFAULT_HEADER_LOGO_URL)
setHeaderLogoUrl('https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png') setHeaderLogoUrl(DEFAULT_HEADER_LOGO_URL)
} }
const handleLoginLogoReset = () => { const handleLoginLogoReset = () => {
setLoginLogoPreview('https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png') setLoginLogoPreview(DEFAULT_LOGIN_LOGO_URL)
setLoginLogoUrl('https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png') setLoginLogoUrl(DEFAULT_LOGIN_LOGO_URL)
} }
if (status === 'loading' || loading) { if (status === 'loading' || loading) {
@ -283,7 +286,10 @@ export default function SystemSettings() {
) )
} }
if (!session || (session.user as any)?.role !== 'ADMIN') { if (
!session ||
!canManageSystemSettings(session.user as SettingsSessionUser)
) {
return null return null
} }
@ -340,7 +346,15 @@ export default function SystemSettings() {
<div className="flex items-center gap-4"> <div className="flex items-center gap-4">
<div className="w-32 h-32 rounded-lg border-2 border-gray-300 dark:border-gray-600 flex items-center justify-center overflow-hidden" style={{ backgroundColor: headerBgColor }}> <div className="w-32 h-32 rounded-lg border-2 border-gray-300 dark:border-gray-600 flex items-center justify-center overflow-hidden" style={{ backgroundColor: headerBgColor }}>
{headerLogoPreview ? ( {headerLogoPreview ? (
<img src={headerLogoPreview} alt="导航栏LOGO" className="max-w-full max-h-full object-contain" /> <BrandLogo
src={headerLogoPreview}
fallbackSrc={DEFAULT_HEADER_LOGO_URL}
alt="导航栏LOGO"
width={160}
height={64}
unoptimized
className="max-h-full max-w-full object-contain"
/>
) : ( ) : (
<svg className="w-16 h-16 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"> <svg className="w-16 h-16 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16l4.586-4.586a2 2 0 012.828 0L16 16m-2-2l1.586-1.586a2 2 0 012.828 0L20 14m-6-6h.01M6 20h12a2 2 0 002-2V6a2 2 0 00-2-2H6a2 2 0 00-2 2v12a2 2 0 002 2z" /> <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16l4.586-4.586a2 2 0 012.828 0L16 16m-2-2l1.586-1.586a2 2 0 012.828 0L20 14m-6-6h.01M6 20h12a2 2 0 002-2V6a2 2 0 00-2-2H6a2 2 0 00-2 2v12a2 2 0 002 2z" />
@ -352,7 +366,7 @@ export default function SystemSettings() {
用于深色导航栏,建议使用浅色LOGO 用于深色导航栏,建议使用浅色LOGO
</p> </p>
<p className="text-sm text-gray-600 dark:text-gray-400 mb-2"> <p className="text-sm text-gray-600 dark:text-gray-400 mb-2">
建议尺寸:200x200px,支持PNG、JPG、SVG格式 建议尺寸:200x200px,支持 PNG、JPEG、WebP 格式
</p> </p>
<p className="text-sm text-gray-600 dark:text-gray-400"> <p className="text-sm text-gray-600 dark:text-gray-400">
文件大小不超过5MB 文件大小不超过5MB
@ -381,7 +395,7 @@ export default function SystemSettings() {
)} )}
<input <input
type="file" type="file"
accept="image/*" accept={BRAND_LOGO_ACCEPT}
onChange={handleHeaderLogoUpload} onChange={handleHeaderLogoUpload}
disabled={uploadingHeader} disabled={uploadingHeader}
className="hidden" className="hidden"
@ -415,7 +429,15 @@ export default function SystemSettings() {
<div className="flex items-center gap-4"> <div className="flex items-center gap-4">
<div className="w-32 h-32 rounded-lg border-2 border-gray-300 dark:border-gray-600 flex items-center justify-center overflow-hidden bg-white"> <div className="w-32 h-32 rounded-lg border-2 border-gray-300 dark:border-gray-600 flex items-center justify-center overflow-hidden bg-white">
{loginLogoPreview ? ( {loginLogoPreview ? (
<img src={loginLogoPreview} alt="登录页LOGO" className="max-w-full max-h-full object-contain" /> <BrandLogo
src={loginLogoPreview}
fallbackSrc={DEFAULT_LOGIN_LOGO_URL}
alt="登录页LOGO"
width={160}
height={96}
unoptimized
className="max-h-full max-w-full object-contain"
/>
) : ( ) : (
<svg className="w-16 h-16 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"> <svg className="w-16 h-16 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16l4.586-4.586a2 2 0 012.828 0L16 16m-2-2l1.586-1.586a2 2 0 012.828 0L20 14m-6-6h.01M6 20h12a2 2 0 002-2V6a2 2 0 00-2-2H6a2 2 0 00-2 2v12a2 2 0 002 2z" /> <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16l4.586-4.586a2 2 0 012.828 0L16 16m-2-2l1.586-1.586a2 2 0 012.828 0L20 14m-6-6h.01M6 20h12a2 2 0 002-2V6a2 2 0 00-2-2H6a2 2 0 00-2 2v12a2 2 0 002 2z" />
@ -427,7 +449,7 @@ export default function SystemSettings() {
用于浅色登录页面,建议使用深色LOGO 用于浅色登录页面,建议使用深色LOGO
</p> </p>
<p className="text-sm text-gray-600 dark:text-gray-400 mb-2"> <p className="text-sm text-gray-600 dark:text-gray-400 mb-2">
建议尺寸:200x200px,支持PNG、JPG、SVG格式 建议尺寸:200x200px,支持 PNG、JPEG、WebP 格式
</p> </p>
<p className="text-sm text-gray-600 dark:text-gray-400"> <p className="text-sm text-gray-600 dark:text-gray-400">
文件大小不超过5MB 文件大小不超过5MB
@ -456,7 +478,7 @@ export default function SystemSettings() {
)} )}
<input <input
type="file" type="file"
accept="image/*" accept={BRAND_LOGO_ACCEPT}
onChange={handleLoginLogoUpload} onChange={handleLoginLogoUpload}
disabled={uploadingLogin} disabled={uploadingLogin}
className="hidden" className="hidden"
@ -647,7 +669,7 @@ export default function SystemSettings() {
type="text" type="text"
value={s3Endpoint} value={s3Endpoint}
onChange={(e) => setS3Endpoint(e.target.value)} onChange={(e) => setS3Endpoint(e.target.value)}
placeholder="例如: s3.1080p.hk" placeholder="例如: https://storage.example.com"
className="w-full px-4 py-2 rounded-lg border border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:text-white focus:border-blue-500 dark:focus:border-blue-400 focus:ring-2 focus:ring-blue-200 dark:focus:ring-blue-600 outline-none transition-all" className="w-full px-4 py-2 rounded-lg border border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:text-white focus:border-blue-500 dark:focus:border-blue-400 focus:ring-2 focus:ring-blue-200 dark:focus:ring-blue-600 outline-none transition-all"
/> />
</div> </div>

View File

@ -3,10 +3,7 @@ import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
import { isSystemAdmin } from '@/lib/department-permission' import { isSystemAdmin } from '@/lib/department-permission'
import { deleteFiles } from '@/lib/file-cleanup'
import { collectMaterialFiles, deleteMaterialRelatedRecords } from '@/lib/material-cleanup'
import { createMaterialLog } from '@/lib/material-log' import { createMaterialLog } from '@/lib/material-log'
import { ensureMaterialDownloadLogsTable } from '@/lib/material-download-log'
// UUID 格式校验正则 // UUID 格式校验正则
const UUID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i const UUID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
@ -50,116 +47,10 @@ export async function POST(request: NextRequest) {
} }
if (permanent) { if (permanent) {
// ========== 永久删除:物理删除文件 + 删除数据库记录 ==========
console.log(`[BatchDelete] 开始永久删除 ${ids.length} 条素材...`)
// 确保下载日志表已应用“删除素材后保留日志”策略
await ensureMaterialDownloadLogsTable()
// 1. 先查询所有素材(获取文件路径和素材类型)
const materials = await prisma.material.findMany({
where: { id: { in: ids } },
})
if (materials.length === 0) {
return NextResponse.json({ error: '未找到要删除的素材' }, { status: 404 })
}
// 2. 收集所有文件并删除
let totalFilesDeleted = 0
let totalFilesFailed = 0
const allErrors: Array<{ materialId: string; fileUrl: string; error: string }> = []
for (const material of materials) {
const materialType = (material as any).materialType || ''
// 记录永久删除日志(先记录,避免数据库删除后日志级联删除前丢失上下文)
await createMaterialLog({
materialId: material.id,
userId: operatorId,
action: 'DELETE',
changes: {
materialSnapshot: {
oldValue: null,
newValue: {
id: material.id,
productModel: material.productModel,
materialType: materialType || null,
},
},
deletedAt: {
oldValue: material.deletedAt ? material.deletedAt.toISOString() : null,
newValue: 'PERMANENTLY_DELETED',
},
deleteMode: {
oldValue: material.deletedAt ? 'soft' : 'active',
newValue: 'permanent',
},
materialType: {
oldValue: materialType || null,
newValue: materialType || null,
},
},
})
// 收集该素材的所有文件
const fileInfo = await collectMaterialFiles(material.id, material)
const filesToDelete = [...fileInfo.fileUrls, ...fileInfo.directories]
console.log(`[BatchDelete] Material ${material.id} (${materialType}):`, {
fileUrls: fileInfo.fileUrls.length,
directories: fileInfo.directories.length,
relatedRecords: fileInfo.relatedRecords,
})
// 删除文件
if (filesToDelete.length > 0) {
const result = await deleteFiles(filesToDelete)
totalFilesDeleted += result.deletedCount
totalFilesFailed += result.failedCount
if (result.errors.length > 0) {
allErrors.push(...result.errors.map(e => ({
materialId: material.id,
fileUrl: e.fileUrl,
error: e.error,
})))
}
}
// 删除关联记录(MaterialAsset, MaterialSku)
if (materialType) {
await deleteMaterialRelatedRecords(material.id, materialType)
}
}
// 3. 删除数据库记录
const deleteResult = await prisma.material.deleteMany({
where: { id: { in: ids } }
})
console.log(`[BatchDelete] 永久删除完成:`, {
requestedCount: ids.length,
deletedRecords: deleteResult.count,
filesDeleted: totalFilesDeleted,
filesFailed: totalFilesFailed,
errorCount: allErrors.length,
})
// 如果有文件删除失败,记录警告但不影响整体结果
if (allErrors.length > 0) {
console.warn(`[BatchDelete] 部分文件删除失败:`, allErrors.slice(0, 5))
}
return NextResponse.json({ return NextResponse.json({
success: true, error: '批量永久删除已暂停:请使用可恢复删除;待持久化清理任务上线后再开放物理删除',
message: `批量永久删除成功`, code: 'PERMANENT_DELETE_DISABLED',
details: { }, { status: 409 })
deletedRecords: deleteResult.count,
filesDeleted: totalFilesDeleted,
filesFailed: totalFilesFailed,
}
})
} else { } else {
// ========== 软删除:仅标记 deletedAt ========== // ========== 软删除:仅标记 deletedAt ==========

View File

@ -1,88 +1,312 @@
import { createHash, timingSafeEqual } from 'crypto'
import { lstat, realpath } from 'fs/promises'
import path from 'path'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import { prisma } from '@/lib/prisma' import { hasSystemAdminPermission, isSuperAdmin } from '@/lib/department-permission'
import { withImageProcessingSlot } from '@/lib/image-processing-limiter'
import { processPdfUpload } from '@/lib/pdf-upload-handler' import { processPdfUpload } from '@/lib/pdf-upload-handler'
import { readFile } from 'fs/promises' import { prisma } from '@/lib/prisma'
import { getUploadRootDir, resolveStoredUploadPath } from '@/lib/upload-paths'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
const MAX_REQUEST_BODY_BYTES = 2 * 1024
const MAX_API_KEY_BYTES = 1024
const MAX_PDF_SOURCE_BYTES = 128 * 1024 * 1024
const MATERIAL_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/
class RegeneratePdfPreviewInputError extends Error {
constructor(
message: string,
readonly status: number = 400,
) {
super(message)
this.name = 'RegeneratePdfPreviewInputError'
}
}
type FileSnapshot = {
size: number | bigint
dev: number | bigint
ino: number | bigint
mtimeMs: number | bigint
}
function apiKeyMatchesConfiguredSecret(providedApiKey: string | null): boolean {
const configuredApiKey = process.env.ADMIN_API_KEY
if (!configuredApiKey || configuredApiKey.trim().length === 0 || !providedApiKey) {
return false
}
if (Buffer.byteLength(providedApiKey, 'utf8') > MAX_API_KEY_BYTES) {
return false
}
// 固定长度摘要后再恒时比较,避免直接字符串比较泄露已配置密钥的长度或前缀。
const configuredDigest = createHash('sha256').update(configuredApiKey, 'utf8').digest()
const providedDigest = createHash('sha256').update(providedApiKey, 'utf8').digest()
return timingSafeEqual(configuredDigest, providedDigest)
}
async function authorizeRequest(request: NextRequest): Promise<NextResponse | null> {
if (apiKeyMatchesConfiguredSecret(request.headers.get('x-api-key'))) {
return null
}
const session = await getServerSession(authOptions)
const userId = String((session?.user as { id?: unknown } | undefined)?.id || '').trim()
if (!userId) {
return NextResponse.json({ error: '未登录' }, { status: 401 })
}
const role = (session?.user as { role?: string | null } | undefined)?.role
if (!hasSystemAdminPermission(role, await isSuperAdmin(userId))) {
return NextResponse.json({ error: '无权限操作' }, { status: 403 })
}
return null
}
async function readBoundedJsonBody(request: NextRequest): Promise<Record<string, unknown>> {
const contentType = String(request.headers.get('content-type') || '').toLowerCase()
if (!/^application\/json(?:\s*;|$)/.test(contentType)) {
throw new RegeneratePdfPreviewInputError('请求必须使用 JSON 格式', 415)
}
const contentEncoding = String(request.headers.get('content-encoding') || 'identity').toLowerCase()
if (contentEncoding !== 'identity') {
throw new RegeneratePdfPreviewInputError('不支持压缩请求体', 415)
}
const contentLengthHeader = request.headers.get('content-length')
if (contentLengthHeader !== null) {
if (!/^\d+$/.test(contentLengthHeader)) {
throw new RegeneratePdfPreviewInputError('请求体长度无效')
}
const contentLength = Number(contentLengthHeader)
if (!Number.isSafeInteger(contentLength) || contentLength > MAX_REQUEST_BODY_BYTES) {
throw new RegeneratePdfPreviewInputError('请求体过大', 413)
}
}
if (!request.body) {
throw new RegeneratePdfPreviewInputError('请求体不能为空')
}
const reader = request.body.getReader()
const chunks: Uint8Array[] = []
let totalBytes = 0
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value || value.byteLength === 0) continue
totalBytes += value.byteLength
if (totalBytes > MAX_REQUEST_BODY_BYTES) {
await reader.cancel().catch(() => undefined)
throw new RegeneratePdfPreviewInputError('请求体过大', 413)
}
chunks.push(value)
}
} finally {
reader.releaseLock()
}
if (totalBytes === 0) {
throw new RegeneratePdfPreviewInputError('请求体不能为空')
}
let parsed: unknown
try {
parsed = JSON.parse(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)), totalBytes).toString('utf8'))
} catch {
throw new RegeneratePdfPreviewInputError('JSON 格式无效')
}
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new RegeneratePdfPreviewInputError('请求体格式无效')
}
return parsed as Record<string, unknown>
}
function isStrictChildPath(rootPath: string, candidatePath: string): boolean {
const relativePath = path.relative(rootPath, candidatePath)
return relativePath.length > 0
&& relativePath !== '..'
&& !relativePath.startsWith(`..${path.sep}`)
&& !path.isAbsolute(relativePath)
}
function toFileSnapshot(stats: Awaited<ReturnType<typeof lstat>>): FileSnapshot {
return {
size: stats.size,
dev: stats.dev,
ino: stats.ino,
mtimeMs: stats.mtimeMs,
}
}
function matchesFileSnapshot(
stats: Awaited<ReturnType<typeof lstat>>,
snapshot: FileSnapshot,
): boolean {
return stats.size === snapshot.size
&& stats.dev === snapshot.dev
&& stats.ino === snapshot.ino
&& stats.mtimeMs === snapshot.mtimeMs
}
function assertRegularPdfStats(stats: Awaited<ReturnType<typeof lstat>>): void {
if (stats.isSymbolicLink() || !stats.isFile()) {
throw new RegeneratePdfPreviewInputError('PDF 源文件不是普通文件')
}
if (stats.size <= 0) {
throw new RegeneratePdfPreviewInputError('PDF 源文件为空')
}
if (stats.size > MAX_PDF_SOURCE_BYTES) {
throw new RegeneratePdfPreviewInputError('PDF 源文件过大', 413)
}
}
async function inspectTrustedPdfSource(storedPath: string): Promise<{
realPath: string
snapshot: FileSnapshot
}> {
const uploadRootPath = path.resolve(getUploadRootDir())
const sourcePath = path.resolve(resolveStoredUploadPath(storedPath))
if (path.extname(sourcePath).toLowerCase() !== '.pdf') {
throw new RegeneratePdfPreviewInputError('源文件必须是 PDF')
}
if (!isStrictChildPath(uploadRootPath, sourcePath)) {
throw new RegeneratePdfPreviewInputError('PDF 源文件路径越界')
}
const uploadRootStats = await lstat(uploadRootPath)
if (uploadRootStats.isSymbolicLink() || !uploadRootStats.isDirectory()) {
throw new Error('UPLOAD_DIR 不是可信目录')
}
let sourceStats: Awaited<ReturnType<typeof lstat>>
try {
sourceStats = await lstat(sourcePath)
} catch (error) {
if ((error as NodeJS.ErrnoException | undefined)?.code === 'ENOENT') {
throw new RegeneratePdfPreviewInputError('PDF 源文件不存在', 404)
}
throw error
}
assertRegularPdfStats(sourceStats)
let uploadRootRealPath: string
let sourceRealPath: string
try {
;[uploadRootRealPath, sourceRealPath] = await Promise.all([
realpath(uploadRootPath),
realpath(sourcePath),
])
} catch (error) {
if ((error as NodeJS.ErrnoException | undefined)?.code === 'ENOENT') {
throw new RegeneratePdfPreviewInputError('PDF 源文件不存在', 404)
}
throw error
}
if (!isStrictChildPath(uploadRootRealPath, sourceRealPath)) {
throw new RegeneratePdfPreviewInputError('PDF 源文件路径越界')
}
if (path.extname(sourceRealPath).toLowerCase() !== '.pdf') {
throw new RegeneratePdfPreviewInputError('源文件必须是 PDF')
}
const realSourceStats = await lstat(sourceRealPath)
assertRegularPdfStats(realSourceStats)
if (!matchesFileSnapshot(realSourceStats, toFileSnapshot(sourceStats))) {
throw new RegeneratePdfPreviewInputError('PDF 源文件在检查期间发生变化', 409)
}
return {
realPath: sourceRealPath,
snapshot: toFileSnapshot(realSourceStats),
}
}
async function processTrustedPdfSource(
realPath: string,
snapshot: FileSnapshot,
) {
return withImageProcessingSlot(async () => {
// processPdfUpload 会读取 PDF 文件头和生成内容;这些读取必须全程处在全局锁内。
const currentRealPath = await realpath(realPath)
if (path.resolve(currentRealPath) !== path.resolve(realPath)) {
throw new RegeneratePdfPreviewInputError('PDF 源文件在处理前发生变化', 409)
}
const currentStats = await lstat(currentRealPath)
assertRegularPdfStats(currentStats)
if (!matchesFileSnapshot(currentStats, snapshot)) {
throw new RegeneratePdfPreviewInputError('PDF 源文件在处理前发生变化', 409)
}
return processPdfUpload(currentRealPath, '')
})
}
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
try { try {
// 验证管理员权限(支持API密钥或会话) const authorizationError = await authorizeRequest(request)
const apiKey = request.headers.get('x-api-key') if (authorizationError) return authorizationError
const session = await getServerSession(authOptions)
// API密钥或会话任一有效即可 const body = await readBoundedJsonBody(request)
const isValidApiKey = apiKey === process.env.ADMIN_API_KEY if (Object.keys(body).some((key) => key !== 'materialId')) {
const isValidSession = session && (session.user as any).id throw new RegeneratePdfPreviewInputError('请求体包含不支持的字段')
if (!isValidApiKey && !isValidSession) {
return NextResponse.json({ error: '未授权' }, { status: 401 })
} }
const body = await request.json() const materialId = typeof body.materialId === 'string' ? body.materialId.trim() : ''
const { materialId } = body if (!MATERIAL_ID_PATTERN.test(materialId)) {
throw new RegeneratePdfPreviewInputError('素材编号无效')
if (!materialId) {
return NextResponse.json({ error: '缺少 materialId' }, { status: 400 })
} }
console.log('[Regenerate PDF Preview] Material ID:', materialId)
// 获取素材详情
const material = await prisma.material.findUnique({ const material = await prisma.material.findUnique({
where: { id: materialId }, where: { id: materialId },
}) select: {
id: true,
if (!material) { materialType: true,
return NextResponse.json({ error: '素材不存在' }, { status: 404 }) zipFilePath: true,
}
if (material.materialType !== '品牌VI灵感') {
return NextResponse.json({ error: '此API仅适用于品牌VI灵感类型素材' }, { status: 400 })
}
// 检查本地PDF文件是否存在
if (!material.zipFilePath) {
return NextResponse.json({ error: '本地PDF文件路径不存在' }, { status: 400 })
}
try {
// 检查文件是否存在
await readFile(material.zipFilePath)
} catch {
return NextResponse.json({ error: `PDF文件不存在: ${material.zipFilePath}` }, { status: 404 })
}
console.log('[Regenerate PDF Preview] PDF file path:', material.zipFilePath)
// 提取PDF预览图
const pdfResult = await processPdfUpload(material.zipFilePath, '')
console.log('[Regenerate PDF Preview] PDF预览图提取结果:', {
success: pdfResult.success,
hasPreviewUrl: !!pdfResult.previewUrl,
previewUrl: pdfResult.previewUrl,
error: pdfResult.error
})
if (!pdfResult.success || !pdfResult.previewUrl) {
const errorMsg = pdfResult.error || 'PDF预览图提取失败'
console.error('[Regenerate PDF Preview] ❌ PDF预览图提取失败:', errorMsg)
return NextResponse.json({ error: errorMsg }, { status: 500 })
}
// 更新素材的 previewUrl
await prisma.material.update({
where: { id: materialId },
data: {
previewUrl: pdfResult.previewUrl,
}, },
}) })
console.log('[Regenerate PDF Preview] ✅ Updated material.previewUrl:', pdfResult.previewUrl) if (!material) {
throw new RegeneratePdfPreviewInputError('素材不存在', 404)
}
if (material.materialType !== '品牌VI灵感') {
throw new RegeneratePdfPreviewInputError('此接口仅适用于品牌VI灵感类型素材')
}
const storedPath = String(material.zipFilePath || '').trim()
if (!storedPath) {
throw new RegeneratePdfPreviewInputError('PDF 源文件路径不存在')
}
const source = await inspectTrustedPdfSource(storedPath)
const pdfResult = await processTrustedPdfSource(source.realPath, source.snapshot)
if (!pdfResult.success || !pdfResult.previewUrl) {
console.error('[Regenerate PDF Preview] PDF 预览生成失败:', pdfResult.error)
throw new Error('PDF 预览生成失败')
}
await prisma.material.update({
where: { id: materialId },
data: { previewUrl: pdfResult.previewUrl },
})
return NextResponse.json({ return NextResponse.json({
success: true, success: true,
@ -90,9 +314,14 @@ export async function POST(request: NextRequest) {
}) })
} catch (error) { } catch (error) {
console.error('[Regenerate PDF Preview] Error:', error) console.error('[Regenerate PDF Preview] Error:', error)
const status = error instanceof RegeneratePdfPreviewInputError ? error.status : 500
return NextResponse.json( return NextResponse.json(
{ error: error instanceof Error ? error.message : '服务器错误' }, {
{ status: 500 } error: error instanceof RegeneratePdfPreviewInputError
? error.message
: 'PDF 预览重建失败,请稍后重试',
},
{ status },
) )
} }
} }

View File

@ -1,179 +1,430 @@
import { NextRequest, NextResponse } from 'next/server' import { randomUUID } from 'crypto'
import { mkdtemp, rm } from 'fs/promises'
import path from 'path'
import AdmZip from 'adm-zip'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { NextRequest, NextResponse } from 'next/server'
import { prisma } from '@/lib/prisma'
import { readFile, mkdir, rm, readdir } from 'fs/promises'
import { join } from 'path'
import sharp from 'sharp' import sharp from 'sharp'
import { sliceImage } from '@/lib/image-processor'
import { authOptions } from '@/lib/auth'
import { readBoundedRegularFile } from '@/lib/bounded-file-reader'
import { hasSystemAdminPermission, isSuperAdmin } from '@/lib/department-permission' import { hasSystemAdminPermission, isSuperAdmin } from '@/lib/department-permission'
import { isPathInAllowedDirs } from '@/lib/path-security' import { assertProcessableMetadata, sliceImage } from '@/lib/image-processor'
import { getUploadRootDir, resolveStoredUploadPath } from '@/lib/upload-paths' import { withImageProcessingSlot } from '@/lib/image-processing-limiter'
import { acquireTaskMaterialCompletionLock } from '@/lib/material-upload-cross-process-lock'
import {
ensureSafeDirectoryWithinRoot,
isPathFreeOfSymbolicLinks,
isPathInAllowedDirs,
} from '@/lib/path-security'
import { prisma } from '@/lib/prisma'
import { import {
buildTaskDetailPublicDisplayPath, buildTaskDetailPublicDisplayPath,
uploadPublicDisplayToS3, uploadPublicDisplayToS3,
} from '@/lib/public-display' } from '@/lib/public-display'
import AdmZip from 'adm-zip' import { deleteFromS3, verifyUploadedS3Object } from '@/lib/s3'
import { getUploadRootDir, resolveStoredUploadPath } from '@/lib/upload-paths'
const MATERIAL_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/
const MAX_REQUEST_BODY_BYTES = 2 * 1024
const MAX_SOURCE_ZIP_BYTES = 64 * 1024 * 1024
const MAX_ZIP_ENTRIES = 2_000
const MAX_ZIP_ENTRY_BYTES = 48 * 1024 * 1024
const MAX_ZIP_TOTAL_UNCOMPRESSED_BYTES = 256 * 1024 * 1024
const MAX_ZIP_COMPRESSION_RATIO = 200
const MAX_IMAGE_PIXELS = 60_000_000
const MAX_IMAGE_DIMENSION = 20_000
const ALLOWED_IMAGE_ENTRY_PATTERN = /\.(?:jpe?g|png|webp)$/i
class RegeneratePreviewInputError extends Error {
constructor(
message: string,
readonly status: number = 400,
) {
super(message)
this.name = 'RegeneratePreviewInputError'
}
}
async function readBoundedRequestBody(request: NextRequest): Promise<Record<string, unknown>> {
const contentType = String(request.headers.get('content-type') || '').toLowerCase()
if (!/^application\/json(?:\s*;|$)/.test(contentType)) {
throw new RegeneratePreviewInputError('请求必须使用 JSON 格式', 415)
}
const contentEncoding = String(request.headers.get('content-encoding') || 'identity').toLowerCase()
if (contentEncoding !== 'identity') {
throw new RegeneratePreviewInputError('不支持压缩请求体', 415)
}
const contentLengthHeader = request.headers.get('content-length')
if (contentLengthHeader === null) {
throw new RegeneratePreviewInputError('请求体长度缺失', 411)
}
if (!/^\d+$/.test(contentLengthHeader)) {
throw new RegeneratePreviewInputError('请求体长度无效')
}
const declaredBytes = Number(contentLengthHeader)
if (!Number.isSafeInteger(declaredBytes) || declaredBytes <= 0) {
throw new RegeneratePreviewInputError('请求体长度无效')
}
if (declaredBytes > MAX_REQUEST_BODY_BYTES) {
throw new RegeneratePreviewInputError('请求体过大', 413)
}
if (!request.body) throw new RegeneratePreviewInputError('请求体不能为空')
const reader = request.body.getReader()
const chunks: Uint8Array[] = []
let totalBytes = 0
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value || value.byteLength === 0) continue
totalBytes += value.byteLength
if (totalBytes > MAX_REQUEST_BODY_BYTES || totalBytes > declaredBytes) {
await reader.cancel().catch(() => undefined)
throw new RegeneratePreviewInputError('请求体实际大小与声明不符', 413)
}
chunks.push(value)
}
} finally {
reader.releaseLock()
}
if (totalBytes === 0 || totalBytes !== declaredBytes) {
throw new RegeneratePreviewInputError('请求体实际大小与声明不符')
}
let parsed: unknown
try {
parsed = JSON.parse(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)), totalBytes).toString('utf8'))
} catch {
throw new RegeneratePreviewInputError('JSON 格式无效')
}
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new RegeneratePreviewInputError('请求体格式无效')
}
const keys = Object.keys(parsed)
if (keys.length !== 1 || keys[0] !== 'materialId') {
throw new RegeneratePreviewInputError('请求体只能包含素材编号')
}
return parsed as Record<string, unknown>
}
class RegeneratePreviewTemporaryError extends Error {
readonly status = 503
constructor(message: string) {
super(message)
this.name = 'RegeneratePreviewTemporaryError'
}
}
function validateZipEntryName(entryName: string): string {
const normalized = String(entryName || '').normalize('NFC').replace(/\\/g, '/')
const segments = normalized.split('/').filter(Boolean)
if (
!normalized
|| normalized.startsWith('/')
|| path.posix.isAbsolute(normalized)
|| path.win32.isAbsolute(normalized)
|| /^[A-Za-z]:/.test(normalized)
|| /[\x00-\x1f\x7f]/.test(normalized)
|| segments.length === 0
|| segments.some((segment) => segment === '.' || segment === '..')
) {
throw new RegeneratePreviewInputError('压缩包包含不安全的文件路径')
}
return normalized
}
function selectBoundedImageEntry(zip: AdmZip): AdmZip.IZipEntry {
const entries = zip.getEntries()
if (entries.length === 0 || entries.length > MAX_ZIP_ENTRIES) {
throw new RegeneratePreviewInputError('压缩包文件数量超出安全范围')
}
let totalUncompressedBytes = 0
let selected: AdmZip.IZipEntry | null = null
for (const entry of entries) {
// AdmZip 会把部分 ../ 片段静默净化后才暴露 entryName;必须同时检查 ZIP
// 中原始文件名,否则不能把库的展示层净化误当成档案本身安全。
validateZipEntryName(entry.rawEntryName.toString('utf8'))
const entryName = validateZipEntryName(entry.entryName)
if (entry.isDirectory) continue
if (entry.header.encripted) {
throw new RegeneratePreviewInputError('不支持加密压缩包')
}
const size = Number(entry.header.size)
const compressedSize = Number(entry.header.compressedSize)
if (
!Number.isSafeInteger(size)
|| size <= 0
|| !Number.isSafeInteger(compressedSize)
|| compressedSize <= 0
) {
throw new RegeneratePreviewInputError('压缩包文件大小信息无效')
}
if (size > MAX_ZIP_ENTRY_BYTES) {
throw new RegeneratePreviewInputError('压缩包内单个文件超过安全上限')
}
if (size / compressedSize > MAX_ZIP_COMPRESSION_RATIO) {
throw new RegeneratePreviewInputError('压缩包压缩比异常,拒绝处理')
}
totalUncompressedBytes += size
if (!Number.isSafeInteger(totalUncompressedBytes)
|| totalUncompressedBytes > MAX_ZIP_TOTAL_UNCOMPRESSED_BYTES) {
throw new RegeneratePreviewInputError('压缩包解压后总大小超过安全上限')
}
if (!selected && ALLOWED_IMAGE_ENTRY_PATTERN.test(entryName)) selected = entry
}
if (!selected) throw new RegeneratePreviewInputError('压缩包内未找到支持的图片文件')
return selected
}
async function readZipEntryBounded(entry: AdmZip.IZipEntry): Promise<Buffer> {
const expectedSize = Number(entry.header.size)
const buffer = await new Promise<Buffer>((resolve, reject) => {
entry.getDataAsync((data, error) => {
if (error) {
reject(new RegeneratePreviewInputError('压缩包内图片读取失败'))
return
}
resolve(data)
})
})
if (buffer.length !== expectedSize || buffer.length > MAX_ZIP_ENTRY_BYTES) {
throw new RegeneratePreviewInputError('压缩包内图片实际大小与记录不一致')
}
return buffer
}
async function inspectPreviewCommit(materialId: string, newUrls: string[]) {
try {
const [material, assets] = await Promise.all([
prisma.material.findUnique({
where: { id: materialId },
select: { previewUrl: true },
}),
prisma.materialAsset.findMany({
where: { materialId, assetType: 'detail-preview' },
select: { fileUrl: true, sortOrder: true },
orderBy: { sortOrder: 'asc' },
}),
])
const actualUrls = assets.map((asset) => asset.fileUrl)
if (
material?.previewUrl === newUrls[0]
&& actualUrls.length === newUrls.length
&& actualUrls.every((url, index) => url === newUrls[index])
) {
return 'committed' as const
}
if (
material?.previewUrl !== newUrls[0]
&& actualUrls.every((url) => !newUrls.includes(url))
) {
return 'not-committed' as const
}
return 'ambiguous' as const
} catch {
return 'ambiguous' as const
}
}
async function cleanupDerivedObjects(urls: string[]): Promise<void> {
await Promise.allSettled(urls.map((url) => deleteFromS3(url)))
}
async function regeneratePreview(materialId: string) {
let releaseMaterialLock: Awaited<ReturnType<typeof acquireTaskMaterialCompletionLock>> | null = null
let tempDir: string | null = null
const uploadedUrls: string[] = []
let allowDerivedCleanup = true
try {
releaseMaterialLock = await acquireTaskMaterialCompletionLock(materialId)
releaseMaterialLock.assertHeld()
const material = await prisma.material.findUnique({ where: { id: materialId } })
if (!material) throw new RegeneratePreviewInputError('素材不存在')
const detailAssets = await prisma.materialAsset.findMany({
where: { materialId, assetType: 'detail' },
orderBy: { sortOrder: 'asc' },
})
if (detailAssets.length === 0) {
throw new RegeneratePreviewInputError('没有找到详情图源文件')
}
const detailAsset = detailAssets[0]
if (!detailAsset.fileUrl.startsWith('uploads/')) {
throw new RegeneratePreviewInputError('该详情图源文件不支持安全重建')
}
const uploadRootDir = getUploadRootDir()
const localPath = resolveStoredUploadPath(detailAsset.fileUrl)
if (
!isPathInAllowedDirs(localPath, [uploadRootDir])
|| !(await isPathFreeOfSymbolicLinks(uploadRootDir, localPath))
|| path.extname(localPath).toLowerCase() !== '.zip'
) {
throw new RegeneratePreviewInputError('详情图源文件路径无效')
}
const tempRoot = await ensureSafeDirectoryWithinRoot(uploadRootDir, 'temp')
if (!tempRoot) throw new Error('无法创建安全的预览重建工作目录')
tempDir = await mkdtemp(path.join(tempRoot, `regenerate-${materialId}-`))
const sliceBuffers = await withImageProcessingSlot(async () => {
let zipBuffer: Buffer
try {
zipBuffer = await readBoundedRegularFile(localPath, {
maxBytes: MAX_SOURCE_ZIP_BYTES,
label: '详情图压缩包',
})
} catch (error) {
throw new RegeneratePreviewInputError(
error instanceof Error ? error.message : '详情图压缩包读取失败',
)
}
// AdmZip 只打开已经过磁盘大小上限检查的 ZIP;不再 extractAllTo,避免
// Zip Slip/Zip Bomb。ZIP 读取、解码、像素检查和切片全程共用同一把全局锁。
let zip: AdmZip
try {
zip = new AdmZip(zipBuffer)
} catch {
throw new RegeneratePreviewInputError('详情图压缩包格式无效')
}
const imageEntry = selectBoundedImageEntry(zip)
const buffer = await readZipEntryBounded(imageEntry)
const metadata = await sharp(buffer, {
limitInputPixels: MAX_IMAGE_PIXELS,
sequentialRead: true,
failOn: 'error',
}).metadata()
assertProcessableMetadata(metadata, {
maxInputWidth: MAX_IMAGE_DIMENSION,
maxInputHeight: MAX_IMAGE_DIMENSION,
maxInputPixels: MAX_IMAGE_PIXELS,
})
releaseMaterialLock!.assertHeld()
// sliceImage 内部也使用 withImageProcessingSlot;AsyncLocalStorage 会安全
// 复用当前锁,避免释放大 Buffer 后重新排队或发生同锁自锁。
return sliceImage(buffer, 1500, 200)
})
if (sliceBuffers.length === 0) throw new Error('未生成任何预览切片')
releaseMaterialLock.assertHeld()
const operationId = randomUUID()
for (let index = 0; index < sliceBuffers.length; index += 1) {
const slice = sliceBuffers[index]
const fileName = `detail-preview-${operationId}-${index}.webp`
const url = await uploadPublicDisplayToS3(
slice,
buildTaskDetailPublicDisplayPath(materialId, 'detail', fileName),
'image/webp',
)
uploadedUrls.push(url)
await verifyUploadedS3Object(url, slice.length, 'image/webp')
releaseMaterialLock.assertHeld()
}
const assetRecords = uploadedUrls.map((url, index) => ({
materialId,
assetType: 'detail-preview',
fileUrl: url,
fileName: `detail-preview-${index}.webp`,
fileSize: BigInt(sliceBuffers[index].length),
sortOrder: index,
}))
try {
releaseMaterialLock.assertHeld()
await prisma.$transaction(async (tx) => {
releaseMaterialLock!.assertHeld()
await tx.materialAsset.deleteMany({
where: { materialId, assetType: 'detail-preview' },
})
releaseMaterialLock!.assertHeld()
await tx.materialAsset.createMany({ data: assetRecords })
releaseMaterialLock!.assertHeld()
await tx.material.update({
where: { id: materialId },
data: { previewUrl: uploadedUrls[0] },
})
})
releaseMaterialLock.assertHeld()
allowDerivedCleanup = false
} catch (transactionError) {
const commitState = await inspectPreviewCommit(materialId, uploadedUrls)
if (commitState === 'committed') {
allowDerivedCleanup = false
} else if (commitState === 'ambiguous') {
// 数据库提交结果不明时绝不删除可能已被引用的新对象。
allowDerivedCleanup = false
throw new RegeneratePreviewTemporaryError(
'预览记录提交结果不明确,已保留全部文件,请管理员核查',
)
} else {
throw transactionError
}
}
return { previewCount: uploadedUrls.length, previewUrls: uploadedUrls }
} catch (error) {
if (allowDerivedCleanup && uploadedUrls.length > 0) {
await cleanupDerivedObjects(uploadedUrls)
}
throw error
} finally {
if (tempDir) await rm(tempDir, { recursive: true, force: true }).catch(() => undefined)
if (releaseMaterialLock) await releaseMaterialLock().catch(() => undefined)
}
}
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
try { try {
// 验证管理员权限
const session = await getServerSession(authOptions) const session = await getServerSession(authOptions)
if (!session || !(session.user as any).id) { if (!session || !(session.user as any).id) {
return NextResponse.json({ error: '未登录' }, { status: 401 }) return NextResponse.json({ error: '未登录' }, { status: 401 })
} }
const userId = (session.user as any).id const userId = String((session.user as any).id)
const userRole = (session.user as any).role const userRole = (session.user as any).role
const isSuperAdminUser = await isSuperAdmin(userId) if (!hasSystemAdminPermission(userRole, await isSuperAdmin(userId))) {
if (!hasSystemAdminPermission(userRole, isSuperAdminUser)) {
return NextResponse.json({ error: '无权限操作' }, { status: 403 }) return NextResponse.json({ error: '无权限操作' }, { status: 403 })
} }
const body = await request.json() const materialId = String((await readBoundedRequestBody(request)).materialId || '').trim()
const { materialId } = body if (!MATERIAL_ID_PATTERN.test(materialId)) {
return NextResponse.json({ error: '素材编号无效' }, { status: 400 })
if (!materialId) {
return NextResponse.json({ error: '缺少 materialId' }, { status: 400 })
} }
console.log('[Regenerate Preview] Material ID:', materialId) const result = await regeneratePreview(materialId)
return NextResponse.json({ success: true, ...result })
// 获取素材详情
const material = await prisma.material.findUnique({
where: { id: materialId },
})
if (!material) {
return NextResponse.json({ error: '素材不存在' }, { status: 404 })
}
// 查找详情图资产
const detailAssets = await prisma.materialAsset.findMany({
where: {
materialId,
assetType: 'detail',
},
orderBy: { sortOrder: 'asc' },
})
if (detailAssets.length === 0) {
return NextResponse.json({ error: '没有找到详情图' }, { status: 404 })
}
// 获取第一个详情图
const detailAsset = detailAssets[0]
console.log('[Regenerate Preview] Detail asset:', detailAsset.fileUrl)
// 如果是本地文件,解压并处理
if (detailAsset.fileUrl.startsWith('uploads/')) {
const uploadRootDir = getUploadRootDir()
const localPath = resolveStoredUploadPath(detailAsset.fileUrl)
if (!isPathInAllowedDirs(localPath, [uploadRootDir])) {
return NextResponse.json({ error: '详情图文件路径无效' }, { status: 400 })
}
// 创建临时目录
const tempDir = join(uploadRootDir, 'temp', `regenerate-${materialId}`)
await mkdir(tempDir, { recursive: true })
// ⚠️ 安全修复:使用 AdmZip 替代 unzip 命令,防止命令注入(R40)
try {
const zip = new AdmZip(localPath)
zip.extractAllTo(tempDir, true)
console.log('[Regenerate Preview] Unzip complete')
} catch (unzipError) {
console.error('[Regenerate Preview] Unzip failed:', unzipError)
return NextResponse.json({ error: '解压文件失败' }, { status: 400 })
}
// ⚠️ 安全修复:使用 fs.readdir 替代 ls 命令,防止命令注入(R40)
const files = await readdir(tempDir)
const imageFile = files.find(f => /\.(jpg|jpeg|png|webp)$/i.test(f))
if (!imageFile) {
return NextResponse.json({ error: '未找到图片文件' }, { status: 400 })
}
const imagePath = join(tempDir, imageFile)
console.log('[Regenerate Preview] Image path:', imagePath)
// 读取图片
const buffer = await readFile(imagePath)
const metadata = await sharp(buffer, {
limitInputPixels: 60_000_000,
sequentialRead: true,
failOn: 'error',
}).metadata()
console.log('[Regenerate Preview] Image size:', metadata.width, 'x', metadata.height)
// 切片处理
const sliceBuffers = await sliceImage(buffer, 1500, 200)
console.log('[Regenerate Preview] Slices:', sliceBuffers.length)
// 上传新预览图
const timestamp = Date.now()
const newPreviewUrls: string[] = []
for (let i = 0; i < sliceBuffers.length; i++) {
const fileName = `detail-preview-${timestamp}-${i}-slice${i}.webp`
const url = await uploadPublicDisplayToS3(
sliceBuffers[i],
buildTaskDetailPublicDisplayPath(materialId, 'detail', fileName),
'image/webp',
)
newPreviewUrls.push(url)
console.log('[Regenerate Preview] Uploaded slice', i + 1, '/', sliceBuffers.length, ':', url)
}
// 删除旧的预览图记录
await prisma.materialAsset.deleteMany({
where: {
materialId,
assetType: 'detail-preview',
},
})
// 创建新的预览图记录
const assetRecords = newPreviewUrls.map((url, index) => ({
materialId,
assetType: 'detail-preview',
fileUrl: url,
fileName: `detail-preview-${index}.webp`,
sortOrder: index,
}))
await prisma.materialAsset.createMany({
data: assetRecords,
})
// 更新素材的 previewUrl(用于首页卡片显示)
await prisma.material.update({
where: { id: materialId },
data: {
previewUrl: newPreviewUrls[0], // 使用第一张切片作为卡片预览图
},
})
console.log('[Regenerate Preview] Updated material.previewUrl:', newPreviewUrls[0])
// ⚠️ 安全修复:使用 fs.rm 替代 rm 命令,防止命令注入(R40)
await rm(tempDir, { recursive: true, force: true })
console.log('[Regenerate Preview] Complete! Generated', newPreviewUrls.length, 'previews')
return NextResponse.json({
success: true,
previewCount: newPreviewUrls.length,
previewUrls: newPreviewUrls,
})
}
return NextResponse.json({ error: '不支持的文件类型' }, { status: 400 })
} catch (error) { } catch (error) {
console.error('[Regenerate Preview] Error:', error) console.error('[Regenerate Preview] Error:', error)
const status = error instanceof RegeneratePreviewInputError
? (error.message === '素材不存在' || error.message === '没有找到详情图源文件'
? 404
: error.status)
: Number((error as { status?: number; statusCode?: number } | undefined)?.status
|| (error as { statusCode?: number } | undefined)?.statusCode
|| 500)
const exposeMessage = error instanceof RegeneratePreviewInputError
|| error instanceof RegeneratePreviewTemporaryError
|| status === 503
return NextResponse.json( return NextResponse.json(
{ error: error instanceof Error ? error.message : '服务器错误' }, {
{ status: 500 } error: exposeMessage && error instanceof Error
? error.message
: '预览重建失败,请稍后重试',
},
{ status: Number.isInteger(status) && status >= 400 && status <= 599 ? status : 500 },
) )
} }
} }

View File

@ -0,0 +1,90 @@
/** @jest-environment node */
import { NextRequest, NextResponse } from 'next/server'
import { POST } from '@/app/api/dingtalk/h5-login/route'
import { loginDingtalkUserByH5AuthCode } from '@/lib/dingtalk-login'
jest.mock('@/lib/dingtalk-login', () => ({
authorizeDingtalkH5UserForNextAuth: jest.fn(),
loginDingtalkUserByH5AuthCode: jest.fn(),
}))
jest.mock('@/lib/dingtalk-performance-session', () => ({
createDingtalkPerformanceSessionToken: jest.fn(),
}))
jest.mock('@/lib/public-app-url', () => ({
getPublicAppOrigin: jest.fn(() => 'https://example.test'),
}))
jest.mock('@/lib/safe-redirect', () => ({
normalizeInternalRedirectPath: jest.fn((_value: unknown, fallback: string) => fallback),
}))
describe('DingTalk H5 public login body limit', () => {
beforeEach(() => {
jest.clearAllMocks()
jest.spyOn(console, 'error').mockImplementation(() => undefined)
})
afterEach(() => {
jest.restoreAllMocks()
})
test('超限 multipart 在 formData 解析前返回 413', async () => {
const formData = jest.fn()
const request = {
headers: new Headers({
'content-type': 'multipart/form-data; boundary=test',
'content-length': String(64 * 1024 + 1),
}),
formData,
} as unknown as NextRequest
const response = await POST(request)
const body = await response.json()
expect(response.status).toBe(413)
expect(body.code).toBe('REQUEST_TOO_LARGE')
expect(formData).not.toHaveBeenCalled()
expect(loginDingtalkUserByH5AuthCode).not.toHaveBeenCalled()
})
test('缺少 Content-Length 时不读取匿名请求体', async () => {
const text = jest.fn()
const request = {
headers: new Headers({ 'content-type': 'text/plain' }),
text,
} as unknown as NextRequest
const response = await POST(request)
expect(response.status).toBe(411)
expect(text).not.toHaveBeenCalled()
})
test('小型 JSON 登录请求保持原有成功链路', async () => {
const payload = JSON.stringify({ code: 'short-code' })
;(loginDingtalkUserByH5AuthCode as jest.Mock).mockResolvedValue(
NextResponse.json({ success: true }),
)
const request = new NextRequest('http://localhost/api/dingtalk/h5-login', {
method: 'POST',
headers: {
'content-type': 'application/json',
'content-length': String(Buffer.byteLength(payload)),
},
body: payload,
})
const response = await POST(request)
expect(response.status).toBe(200)
expect(loginDingtalkUserByH5AuthCode).toHaveBeenCalledWith(
request,
'short-code',
{ appSource: 'workNotice' },
)
})
})

View File

@ -10,6 +10,18 @@ import { normalizeInternalRedirectPath } from '@/lib/safe-redirect'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
const H5_LOGIN_MAX_BODY_BYTES = 64 * 1024
class H5LoginRequestError extends Error {
constructor(
message: string,
readonly status: number,
readonly code: string,
) {
super(message)
}
}
const SURVEY_OPENING_PAGE_STYLE = ` const SURVEY_OPENING_PAGE_STYLE = `
html,body{margin:0;min-height:100%;background:#f6f8fb;color:#111827;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif} html,body{margin:0;min-height:100%;background:#f6f8fb;color:#111827;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}
body{min-height:100vh;min-height:100dvh;overflow:hidden} body{min-height:100vh;min-height:100dvh;overflow:hidden}
@ -323,6 +335,7 @@ function createH5LoginErrorPage(message: string) {
} }
async function readH5LoginCode(request: NextRequest) { async function readH5LoginCode(request: NextRequest) {
assertBoundedH5LoginBody(request)
const contentType = request.headers.get('content-type') || '' const contentType = request.headers.get('content-type') || ''
if (contentType.includes('application/json')) { if (contentType.includes('application/json')) {
const body = await request.json().catch(() => ({})) const body = await request.json().catch(() => ({}))
@ -341,6 +354,33 @@ async function readH5LoginCode(request: NextRequest) {
return new URLSearchParams(body).get('code')?.trim() || '' return new URLSearchParams(body).get('code')?.trim() || ''
} }
function assertBoundedH5LoginBody(request: NextRequest): void {
const rawContentLength = request.headers.get('content-length')
if (!rawContentLength || !/^\d+$/.test(rawContentLength.trim())) {
throw new H5LoginRequestError(
'登录请求缺少有效的 Content-Length',
411,
'CONTENT_LENGTH_REQUIRED',
)
}
const contentLength = Number(rawContentLength)
if (!Number.isSafeInteger(contentLength) || contentLength < 0) {
throw new H5LoginRequestError(
'登录请求的 Content-Length 无效',
411,
'CONTENT_LENGTH_REQUIRED',
)
}
if (contentLength > H5_LOGIN_MAX_BODY_BYTES) {
throw new H5LoginRequestError(
'登录请求内容过大',
413,
'REQUEST_TOO_LARGE',
)
}
}
function isHtmlNavigationRequest(request: NextRequest) { function isHtmlNavigationRequest(request: NextRequest) {
const accept = request.headers.get('accept') || '' const accept = request.headers.get('accept') || ''
const fetchMode = request.headers.get('sec-fetch-mode') || '' const fetchMode = request.headers.get('sec-fetch-mode') || ''
@ -442,6 +482,12 @@ export async function POST(request: NextRequest) {
return response return response
} catch (error: any) { } catch (error: any) {
console.error('钉钉H5免登失败:', error) console.error('钉钉H5免登失败:', error)
if (error instanceof H5LoginRequestError) {
return NextResponse.json(
{ success: false, message: error.message, code: error.code },
{ status: error.status },
)
}
return NextResponse.json( return NextResponse.json(
{ {
success: false, success: false,

View File

@ -800,7 +800,8 @@ async function downloadNonProductPackageMaterial(material: any, downloadType: st
if (!material.zipFilePath || material.zipFilePath.trim() === '') { if (!material.zipFilePath || material.zipFilePath.trim() === '') {
const zipUrl = (material.zipFileName || '').trim(); const zipUrl = (material.zipFileName || '').trim();
// 场景:音频/视频源文件直接存放在S3,zipFileName 是 S3 URL // 仅兼容历史素材:部分音频/视频源文件曾把 S3 URL 写入 zipFileName。
// 新视频源以 videoSourceUrl 对应的生产 uploads/NAS 文件为准,预览产物才写入 S3。
if (zipUrl && isHttpUrl(zipUrl)) { if (zipUrl && isHttpUrl(zipUrl)) {
console.log('[external/download] zipFilePath empty, proxy S3 url from zipFileName:', zipUrl); console.log('[external/download] zipFilePath empty, proxy S3 url from zipFileName:', zipUrl);

View File

@ -26,6 +26,10 @@ import {
STANDARD_MODEL_WHITE_BG_UPLOAD_LIMITS, STANDARD_MODEL_WHITE_BG_UPLOAD_LIMITS,
WhiteBackgroundUploadValidationError, WhiteBackgroundUploadValidationError,
} from '@/lib/standard-model-white-background-upload-guard' } from '@/lib/standard-model-white-background-upload-guard'
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
} from '@/lib/multipart-route-limits'
export const runtime = 'nodejs' export const runtime = 'nodejs'
@ -182,6 +186,7 @@ export async function POST(request: NextRequest) {
trace.apiKeyName = keyRecord.name trace.apiKeyName = keyRecord.name
let idempotencyRecordId: string | null = null let idempotencyRecordId: string | null = null
let releaseRequestSlot: (() => void) | null = null
try { try {
const contentLength = Number(request.headers.get('content-length')) const contentLength = Number(request.headers.get('content-length'))
@ -200,6 +205,7 @@ export async function POST(request: NextRequest) {
) )
} }
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire()
const formData = await request.formData() const formData = await request.formData()
const clientRequestId = String(formData.get('clientRequestId') || '').trim() const clientRequestId = String(formData.get('clientRequestId') || '').trim()
const barcode = String(formData.get('barcode') || '').trim() const barcode = String(formData.get('barcode') || '').trim()
@ -313,6 +319,20 @@ export async function POST(request: NextRequest) {
await failExternalApiIdempotencyRecord(idempotencyRecordId) await failExternalApiIdempotencyRecord(idempotencyRecordId)
} }
const boundedFailure = getBoundedMultipartFailure(error)
if (boundedFailure) {
await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: boundedFailure.status })
const response = buildErrorResponse(
boundedFailure.status,
boundedFailure.code,
boundedFailure.message,
)
if (boundedFailure.retryAfterSeconds) {
response.headers.set('Retry-After', String(boundedFailure.retryAfterSeconds))
}
return finalizeResponse(response, trace)
}
if (error instanceof ExternalWhiteBackgroundBarcodeError) { if (error instanceof ExternalWhiteBackgroundBarcodeError) {
await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: error.status }) await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: error.status })
return finalizeResponse(buildErrorResponse(error.status, error.code, error.message), trace) return finalizeResponse(buildErrorResponse(error.status, error.code, error.message), trace)
@ -330,5 +350,7 @@ export async function POST(request: NextRequest) {
console.error('[external/standard-model-white-background/by-barcode] POST error:', error) console.error('[external/standard-model-white-background/by-barcode] POST error:', error)
await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: 500 }) await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: 500 })
return finalizeResponse(buildErrorResponse(500, 'UPLOAD_FAILED', message || '上传白底图失败'), trace) return finalizeResponse(buildErrorResponse(500, 'UPLOAD_FAILED', message || '上传白底图失败'), trace)
} finally {
releaseRequestSlot?.()
} }
} }

View File

@ -28,6 +28,10 @@ import {
STANDARD_MODEL_WHITE_BG_UPLOAD_LIMITS, STANDARD_MODEL_WHITE_BG_UPLOAD_LIMITS,
WhiteBackgroundUploadValidationError, WhiteBackgroundUploadValidationError,
} from '@/lib/standard-model-white-background-upload-guard' } from '@/lib/standard-model-white-background-upload-guard'
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
} from '@/lib/multipart-route-limits'
export const runtime = 'nodejs' export const runtime = 'nodejs'
@ -576,6 +580,7 @@ export async function POST(request: NextRequest) {
trace.apiKeyName = keyRecord.name trace.apiKeyName = keyRecord.name
let idempotencyRecordId: string | null = null let idempotencyRecordId: string | null = null
let releaseRequestSlot: (() => void) | null = null
try { try {
const contentLength = Number(request.headers.get('content-length')) const contentLength = Number(request.headers.get('content-length'))
@ -594,6 +599,7 @@ export async function POST(request: NextRequest) {
) )
} }
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire()
const formData = await request.formData() const formData = await request.formData()
const clientRequestId = getFormText(formData, 'clientRequestId') const clientRequestId = getFormText(formData, 'clientRequestId')
const materialType = getFormText(formData, 'materialType') const materialType = getFormText(formData, 'materialType')
@ -758,6 +764,20 @@ export async function POST(request: NextRequest) {
await failExternalApiIdempotencyRecord(idempotencyRecordId) await failExternalApiIdempotencyRecord(idempotencyRecordId)
} }
const boundedFailure = getBoundedMultipartFailure(error)
if (boundedFailure) {
await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: boundedFailure.status })
const response = buildErrorResponse(
boundedFailure.status,
boundedFailure.code,
boundedFailure.message,
)
if (boundedFailure.retryAfterSeconds) {
response.headers.set('Retry-After', String(boundedFailure.retryAfterSeconds))
}
return finalizeResponse(response, trace)
}
if ( if (
error instanceof ExternalWhiteBackgroundBySpuError || error instanceof ExternalWhiteBackgroundBySpuError ||
error instanceof ExternalWhiteBackgroundBarcodeError error instanceof ExternalWhiteBackgroundBarcodeError
@ -778,5 +798,7 @@ export async function POST(request: NextRequest) {
console.error('[external/standard-model-white-background/by-spu] POST error:', error) console.error('[external/standard-model-white-background/by-spu] POST error:', error)
await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: 500 }) await writeApiLog({ apiKeyId: keyRecord.id, trace, statusCode: 500 })
return finalizeResponse(buildErrorResponse(500, 'UPLOAD_FAILED', message || '上传白底图失败'), trace) return finalizeResponse(buildErrorResponse(500, 'UPLOAD_FAILED', message || '上传白底图失败'), trace)
} finally {
releaseRequestSlot?.()
} }
} }

View File

@ -0,0 +1,238 @@
/** @jest-environment node */
import { readFile } from 'node:fs/promises'
import { join } from 'node:path'
import { NextRequest } from 'next/server'
import { GET } from '@/app/api/favicon/route'
import { prisma } from '@/lib/prisma'
jest.mock('@/lib/prisma', () => ({
prisma: {
systemSettings: {
findFirst: jest.fn(),
},
},
}))
jest.mock('@/lib/url-rewrite', () => ({
rewriteStorageUrlForClient: jest.fn((url: string) => url),
}))
const request = new NextRequest('https://sc.jiansun.vip/api/favicon')
let remoteLogoSequence = 0
function mockFetchResponse({
ok,
contentType,
body = Buffer.alloc(0),
chunks = [body],
contentLength = String(body.length),
cancel = jest.fn(),
}: {
ok: boolean
contentType: string
body?: Buffer
chunks?: Buffer[]
contentLength?: string | null
cancel?: jest.Mock
}) {
let index = 0
const stream = new ReadableStream<Uint8Array>({
pull(controller) {
if (index >= chunks.length) {
controller.close()
return
}
controller.enqueue(chunks[index])
index += 1
},
cancel,
})
const arrayBuffer = jest.fn().mockRejectedValue(
new Error('favicon route must not buffer the unbounded response'),
)
;(global.fetch as jest.Mock).mockResolvedValue({
ok,
headers: {
get: jest.fn((name: string) => {
if (name.toLowerCase() === 'content-type') return contentType
if (name.toLowerCase() === 'content-length') return contentLength
return null
}),
},
body: stream,
arrayBuffer,
})
return { arrayBuffer, cancel }
}
describe('/api/favicon', () => {
let localFavicon: Buffer
beforeAll(async () => {
localFavicon = await readFile(join(process.cwd(), 'public', 'brand', 'favicon.png'))
})
beforeEach(() => {
jest.clearAllMocks()
global.fetch = jest.fn()
remoteLogoSequence += 1
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue({
headerLogoUrl: `https://s3.jiansun.vip/gongxiangpan/brand/header-${remoteLogoSequence}.png`,
})
})
test.each([
{ name: '404', ok: false, contentType: 'text/html' },
{ name: '返回 HTML', ok: true, contentType: 'text/html; charset=utf-8' },
])('上游 $name 时返回真实本地图标', async ({ ok, contentType }) => {
const upstream = mockFetchResponse({
ok,
contentType,
body: Buffer.from('<html>bad gateway</html>'),
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(upstream.cancel).toHaveBeenCalledTimes(1)
expect(response.status).toBe(200)
expect(response.headers.get('content-type')).toBe('image/png')
expect(bytes.equals(localFavicon)).toBe(true)
})
test('Content-Length 声明超过 favicon 的 1MB 上限时不读取正文并立即取消', async () => {
const upstream = mockFetchResponse({
ok: true,
contentType: 'image/png',
body: localFavicon,
contentLength: String(1024 * 1024 + 1),
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(upstream.arrayBuffer).not.toHaveBeenCalled()
expect(upstream.cancel).toHaveBeenCalledTimes(1)
expect(bytes.equals(localFavicon)).toBe(true)
})
test('无 Content-Length 的流超过 favicon 的 1MB 上限时停止读取、取消上游并回退', async () => {
const upstream = mockFetchResponse({
ok: true,
contentType: 'image/png',
// 第三个分片让流保持打开,可确认超限时确实主动 cancel,而不是自然结束。
chunks: [
Buffer.alloc(768 * 1024),
Buffer.alloc(512 * 1024),
Buffer.alloc(1),
],
contentLength: null,
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(upstream.arrayBuffer).not.toHaveBeenCalled()
expect(upstream.cancel).toHaveBeenCalledTimes(1)
expect(bytes.equals(localFavicon)).toBe(true)
})
test('Content-Length 撒谎时仍由流式硬上限拦截', async () => {
const upstream = mockFetchResponse({
ok: true,
contentType: 'image/png',
chunks: [
Buffer.alloc(768 * 1024),
Buffer.alloc(512 * 1024),
Buffer.alloc(1),
],
contentLength: '1024',
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(upstream.cancel).toHaveBeenCalledTimes(1)
expect(bytes.equals(localFavicon)).toBe(true)
})
test('无 Content-Length 的正常分片图片可以完成解码并返回', async () => {
const splitAt = Math.floor(localFavicon.length / 2)
const upstream = mockFetchResponse({
ok: true,
contentType: 'image/png',
chunks: [localFavicon.subarray(0, splitAt), localFavicon.subarray(splitAt)],
contentLength: null,
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(upstream.arrayBuffer).not.toHaveBeenCalled()
expect(upstream.cancel).not.toHaveBeenCalled()
expect(response.headers.get('content-type')).toBe('image/png')
expect(bytes.equals(localFavicon)).toBe(true)
})
test('上游请求超时或网络失败时返回真实本地图标', async () => {
;(global.fetch as jest.Mock).mockRejectedValue(new Error('request timeout'))
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(response.status).toBe(200)
expect(bytes.equals(localFavicon)).toBe(true)
})
test('数据库是本地品牌地址时不发起网络请求', async () => {
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue({
headerLogoUrl: '/brand/header-logo.png',
})
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(global.fetch).not.toHaveBeenCalled()
expect(bytes.equals(localFavicon)).toBe(true)
})
test('只透传真正能够完整解码的受控上游图片', async () => {
mockFetchResponse({ ok: true, contentType: 'image/png', body: localFavicon })
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(response.headers.get('content-type')).toBe('image/png')
expect(bytes.equals(localFavicon)).toBe(true)
})
test('同一 Logo 的并发和后续请求只下载并解码一次', async () => {
mockFetchResponse({ ok: true, contentType: 'image/png', body: localFavicon })
const [first, second] = await Promise.all([GET(request), GET(request)])
const third = await GET(request)
const bodies = await Promise.all(
[first, second, third].map(async response => Buffer.from(await response.arrayBuffer())),
)
expect(global.fetch).toHaveBeenCalledTimes(1)
expect(bodies.every(bytes => bytes.equals(localFavicon))).toBe(true)
})
test('只有图片文件头的截断响应会回退本地图标', async () => {
const truncatedPng = Buffer.from([
0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3,
])
mockFetchResponse({ ok: true, contentType: 'image/png', body: truncatedPng })
const response = await GET(request)
const bytes = Buffer.from(await response.arrayBuffer())
expect(response.headers.get('content-type')).toBe('image/png')
expect(bytes.equals(localFavicon)).toBe(true)
expect(bytes.equals(truncatedPng)).toBe(false)
})
})

View File

@ -1,67 +1,215 @@
// Favicon API - 返回系统设置的 logo 作为 favicon // Favicon API - 优先返回受控远端品牌图,任何异常都使用镜像内置真实图标。
import { readFile } from 'node:fs/promises'
import { join } from 'node:path'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
import { rewriteStorageUrlForClient } from '@/lib/url-rewrite' import { rewriteStorageUrlForClient } from '@/lib/url-rewrite'
import { normalizeBrandLogoUrl } from '@/lib/brand-assets'
import { MAX_BRAND_IMAGE_BYTES, validateBrandImage } from '@/lib/brand-image-validation'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
export const runtime = 'nodejs' export const runtime = 'nodejs'
const FAVICON_PATH = join(process.cwd(), 'public', 'brand', 'favicon.png')
const FETCH_TIMEOUT_MS = 5_000
const INITIAL_REMOTE_BUFFER_BYTES = 64 * 1024
const MAX_REMOTE_FAVICON_BYTES = Math.min(MAX_BRAND_IMAGE_BYTES, 1024 * 1024)
const SUCCESS_CACHE_TTL_MS = 10 * 60 * 1000
const FAILURE_CACHE_TTL_MS = 30 * 1000
const FAVICON_HEADERS = {
'Content-Type': 'image/png',
'Cache-Control': 'public, max-age=3600',
}
type RemoteFaviconPayload = {
bytes: Buffer
contentType: string
}
let successCache: (RemoteFaviconPayload & { url: string; expiresAt: number }) | null = null
let failureCache: { url: string; expiresAt: number } | null = null
let latestRequestedUrl = ''
const inFlightRequests = new Map<string, Promise<RemoteFaviconPayload | null>>()
async function localFaviconResponse() {
const favicon = await readFile(FAVICON_PATH)
return new NextResponse(Uint8Array.from(favicon).buffer, { headers: FAVICON_HEADERS })
}
function declaresOversizedBody(contentLength: string | null) {
const normalized = contentLength?.trim()
if (!normalized || !/^\d+$/.test(normalized)) return false
const parsed = Number(normalized)
return !Number.isSafeInteger(parsed) || parsed > MAX_REMOTE_FAVICON_BYTES
}
async function cancelBody(body: ReadableStream<Uint8Array> | null) {
if (!body) return
try {
await body.cancel('favicon response exceeds size limit')
} catch {
// 连接可能已经由 fetch/AbortController 关闭,无需掩盖本地回退。
}
}
async function readRemoteBodyWithLimit(response: Response) {
if (declaresOversizedBody(response.headers.get('content-length'))) {
await cancelBody(response.body)
return null
}
const reader = response.body?.getReader()
if (!reader) return null
let bytesRead = 0
let output = Buffer.allocUnsafe(
Math.min(INITIAL_REMOTE_BUFFER_BYTES, MAX_REMOTE_FAVICON_BYTES),
)
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value?.byteLength) continue
const nextBytesRead = bytesRead + value.byteLength
if (nextBytesRead > MAX_REMOTE_FAVICON_BYTES) {
try {
await reader.cancel('favicon response exceeds size limit')
} catch {
// cancel 失败也不得继续读取或保留超限内容。
}
return null
}
if (nextBytesRead > output.length) {
const nextCapacity = Math.min(
MAX_REMOTE_FAVICON_BYTES,
Math.max(nextBytesRead, output.length * 2),
)
const expanded = Buffer.allocUnsafe(nextCapacity)
output.copy(expanded, 0, 0, bytesRead)
output = expanded
}
Buffer.from(value.buffer, value.byteOffset, value.byteLength).copy(
output,
bytesRead,
)
bytesRead = nextBytesRead
}
} finally {
reader.releaseLock()
}
if (bytesRead === 0) return null
return Buffer.from(output.subarray(0, bytesRead))
}
async function fetchRemoteFavicon(url: string): Promise<RemoteFaviconPayload | null> {
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS)
try {
const response = await fetch(url, {
headers: {
'User-Agent': 'Mozilla/5.0 (compatible; FaviconFetcher/2.0)',
},
cache: 'no-store',
signal: controller.signal,
})
const contentType = response.headers.get('content-type')?.split(';')[0].trim().toLowerCase()
if (!response.ok || !contentType?.startsWith('image/')) {
await cancelBody(response.body)
return null
}
const bytes = await readRemoteBodyWithLimit(response)
if (!bytes) return null
let validated
try {
validated = await validateBrandImage(bytes, contentType)
} catch {
return null
}
return { bytes, contentType: validated.contentType }
} finally {
clearTimeout(timeout)
}
}
async function getRemoteFavicon(url: string): Promise<RemoteFaviconPayload | null> {
const now = Date.now()
latestRequestedUrl = url
if (successCache?.url === url && successCache.expiresAt > now) {
return successCache
}
if (failureCache?.url === url && failureCache.expiresAt > now) {
return null
}
const existingRequest = inFlightRequests.get(url)
if (existingRequest) return existingRequest
const request = fetchRemoteFavicon(url)
.then((payload) => {
if (latestRequestedUrl !== url) return payload
if (payload) {
successCache = { ...payload, url, expiresAt: Date.now() + SUCCESS_CACHE_TTL_MS }
failureCache = null
} else {
failureCache = { url, expiresAt: Date.now() + FAILURE_CACHE_TTL_MS }
}
return payload
})
.catch((error) => {
if (latestRequestedUrl === url) {
failureCache = { url, expiresAt: Date.now() + FAILURE_CACHE_TTL_MS }
}
throw error
})
.finally(() => {
inFlightRequests.delete(url)
})
inFlightRequests.set(url, request)
return request
}
function remoteFaviconResponse(payload: RemoteFaviconPayload) {
return new NextResponse(Uint8Array.from(payload.bytes).buffer, {
headers: {
'Content-Type': payload.contentType,
'Cache-Control': 'public, max-age=3600',
},
})
}
export async function GET(request: NextRequest) { export async function GET(request: NextRequest) {
try { try {
// 获取系统设置中的 headerLogoUrl
const settings = await prisma.systemSettings.findFirst({ const settings = await prisma.systemSettings.findFirst({
select: { select: { headerLogoUrl: true },
headerLogoUrl: true, })
const canonicalLogoUrl = normalizeBrandLogoUrl(settings?.headerLogoUrl)
if (canonicalLogoUrl?.startsWith('https://')) {
const remoteUrl = rewriteStorageUrlForClient(canonicalLogoUrl, request) || canonicalLogoUrl
try {
const remotePayload = await getRemoteFavicon(remoteUrl)
if (remotePayload) return remoteFaviconResponse(remotePayload)
} catch (error) {
console.warn('[Favicon API] 远端品牌图不可用,改用内置图标:', error)
} }
})
const logoUrl = settings?.headerLogoUrl || 'https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png'
// 根据用户网络位置重写 URL(内网用户使用内网地址)
const finalUrl = rewriteStorageUrlForClient(logoUrl, request) || logoUrl
// 获取 logo 图片
const response = await fetch(finalUrl, {
headers: {
'User-Agent': 'Mozilla/5.0 (compatible; FaviconFetcher/1.0)',
},
})
if (!response.ok) {
// 如果获取失败,返回默认的透明 PNG
const FAVICON_PNG = Buffer.from(
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==',
'base64'
)
return new NextResponse(FAVICON_PNG, {
headers: {
'Content-Type': 'image/png',
'Cache-Control': 'public, max-age=3600',
},
})
} }
const arrayBuffer = await response.arrayBuffer() return await localFaviconResponse()
const contentType = response.headers.get('content-type') || 'image/png'
return new NextResponse(arrayBuffer, {
headers: {
'Content-Type': contentType,
'Cache-Control': 'public, max-age=3600', // 缓存1小时
},
})
} catch (error) { } catch (error) {
console.error('[Favicon API] Error:', error) console.error('[Favicon API] Error:', error)
// 返回默认的透明 PNG return localFaviconResponse()
const FAVICON_PNG = Buffer.from(
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==',
'base64'
)
return new NextResponse(FAVICON_PNG, {
headers: {
'Content-Type': 'image/png',
'Cache-Control': 'public, max-age=3600',
},
})
} }
} }

View File

@ -0,0 +1,85 @@
/** @jest-environment node */
import { NextRequest } from 'next/server'
import { GET } from '@/app/api/health/route'
import { prisma } from '@/lib/prisma'
import { redis } from '@/lib/upload-sessions-redis'
import { readFileSync } from 'fs'
jest.mock('fs', () => ({
...jest.requireActual('fs'),
readFileSync: jest.fn(),
}))
jest.mock('@/lib/prisma', () => ({
prisma: {
$queryRaw: jest.fn(),
},
}))
jest.mock('@/lib/upload-sessions-redis', () => ({
redis: {
ping: jest.fn(),
},
}))
describe('/api/health deployment identity', () => {
const originalUseRedis = process.env.USE_REDIS
const originalRedisUrl = process.env.REDIS_URL
const originalCommit = process.env.DEPLOYMENT_COMMIT
const bakedCommit = '0123456789abcdef0123456789abcdef01234567'
beforeEach(() => {
jest.clearAllMocks()
process.env.USE_REDIS = 'true'
process.env.REDIS_URL = 'redis://unit-test.invalid:6379'
;(readFileSync as jest.Mock).mockReturnValue(`${bakedCommit}\n`)
;(prisma.$queryRaw as jest.Mock).mockResolvedValue([{ ok: 1 }])
;(redis!.ping as jest.Mock).mockResolvedValue('PONG')
})
afterAll(() => {
if (originalUseRedis === undefined) delete process.env.USE_REDIS
else process.env.USE_REDIS = originalUseRedis
if (originalRedisUrl === undefined) delete process.env.REDIS_URL
else process.env.REDIS_URL = originalRedisUrl
if (originalCommit === undefined) delete process.env.DEPLOYMENT_COMMIT
else process.env.DEPLOYMENT_COMMIT = originalCommit
})
test('returns exact deployment commit with healthy database and Redis checks', async () => {
const response = await GET(new NextRequest('https://example.invalid/api/health'))
const body = await response.json()
expect(response.status).toBe(200)
expect(body).toMatchObject({
status: 'healthy',
deploymentCommit: bakedCommit,
checks: {
database: { status: 'ok' },
redis: { status: 'ok' },
},
})
})
test('keeps Redis failure visible as degraded so deploy can reject it', async () => {
;(redis!.ping as jest.Mock).mockRejectedValue(new Error('redis unavailable'))
const response = await GET(new NextRequest('https://example.invalid/api/health'))
const body = await response.json()
expect(response.status).toBe(200)
expect(body.status).toBe('degraded')
expect(body.checks.redis.status).toBe('error')
})
test('never trusts a mutable runtime environment variable as image identity', async () => {
process.env.DEPLOYMENT_COMMIT = 'f'.repeat(40)
;(readFileSync as jest.Mock).mockReturnValue('not-an-immutable-git-sha')
const response = await GET(new NextRequest('https://example.invalid/api/health'))
const body = await response.json()
expect(body.deploymentCommit).toBeNull()
})
})

View File

@ -14,14 +14,27 @@
*/ */
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { readFileSync } from 'fs'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
const DEPLOYMENT_COMMIT_FILE = '/app/DEPLOYMENT_COMMIT'
function readBakedDeploymentCommit(): string | null {
try {
const value = readFileSync(DEPLOYMENT_COMMIT_FILE, 'utf8').trim()
return /^[0-9a-f]{40}$/.test(value) ? value : null
} catch {
return null
}
}
// 健康检查结果类型 // 健康检查结果类型
interface HealthStatus { interface HealthStatus {
status: 'healthy' | 'degraded' | 'unhealthy' status: 'healthy' | 'degraded' | 'unhealthy'
timestamp: string timestamp: string
uptime: number uptime: number
version: string version: string
deploymentCommit: string | null
checks: { checks: {
database: CheckResult database: CheckResult
redis?: CheckResult redis?: CheckResult
@ -60,6 +73,7 @@ export async function GET(req: NextRequest) {
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
uptime: process.uptime(), uptime: process.uptime(),
version: process.env.npm_package_version || '1.0.0', version: process.env.npm_package_version || '1.0.0',
deploymentCommit: readBakedDeploymentCommit(),
checks: { checks: {
database: { status: 'ok' }, database: { status: 'ok' },
}, },

View File

@ -16,6 +16,12 @@ import { detectFileKindFromName, getCertificateReportFileKindByAssetType } from
import { DownloadErrorCode } from '@/lib/download-errors'; import { DownloadErrorCode } from '@/lib/download-errors';
import { getVideoPreviewPolicyFields } from '@/lib/video-download-policy'; import { getVideoPreviewPolicyFields } from '@/lib/video-download-policy';
import { is3DOriginalType, isProductPackageType } from '@/lib/material-type-mapper'; import { is3DOriginalType, isProductPackageType } from '@/lib/material-type-mapper';
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
// 解析 displayImages 字段 // 解析 displayImages 字段
function parseDisplayImages(displayImages: any): string[] { function parseDisplayImages(displayImages: any): string[] {
@ -535,6 +541,7 @@ export async function PUT(
request: NextRequest, request: NextRequest,
{ params }: { params: { id: string } } { params }: { params: { id: string } }
) { ) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session?.user) { if (!session?.user) {
@ -546,7 +553,20 @@ export async function PUT(
let previewImage: File | null = null; let previewImage: File | null = null;
if (contentType.includes('multipart/form-data')) { if (contentType.includes('multipart/form-data')) {
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.materialUpdateRequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
previewImage: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.materialUpdatePreviewBytes,
allowedMimeTypes: ['image/jpeg', 'image/png', 'image/webp', 'image/avif'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.materialUpdatePreviewBytes,
});
input = { input = {
productModel: String(formData.get('productModel') || ''), productModel: String(formData.get('productModel') || ''),
...(formData.has('reportTitle') ...(formData.has('reportTitle')
@ -623,6 +643,20 @@ export async function PUT(
}); });
} catch (error) { } catch (error) {
console.error('更新素材失败:', error); console.error('更新素材失败:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json({ error: '更新素材失败' }, { status: 500 }); return NextResponse.json({ error: '更新素材失败' }, { status: 500 });
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -7,6 +7,12 @@ import path from 'path'
import { existsSync } from 'fs' import { existsSync } from 'fs'
import { sanitizeFileName } from '@/lib/path-security' import { sanitizeFileName } from '@/lib/path-security'
import { canManageSkuMaterial } from '../authorization' import { canManageSkuMaterial } from '../authorization'
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard'
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
@ -17,6 +23,7 @@ export async function POST(
request: NextRequest, request: NextRequest,
{ params }: { params: { id: string } } { params }: { params: { id: string } }
) { ) {
let releaseRequestSlot: (() => void) | null = null
try { try {
const session = await getServerSession(authOptions) const session = await getServerSession(authOptions)
if (!session?.user || !(session.user as any).id) { if (!session?.user || !(session.user as any).id) {
@ -43,7 +50,19 @@ export async function POST(
return NextResponse.json({ error: '无权限操作此素材' }, { status: 403 }) return NextResponse.json({ error: '无权限操作此素材' }, { status: 403 })
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.skuArchiveRequestBytes)
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire()
const formData = await request.formData() const formData = await request.formData()
assertMultipartFiles(formData, {
fields: {
file: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.skuArchiveFileBytes,
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.skuArchiveFileBytes,
})
const skuName = formData.get('skuName') as string const skuName = formData.get('skuName') as string
const file = formData.get('file') as File const file = formData.get('file') as File
@ -51,6 +70,11 @@ export async function POST(
return NextResponse.json({ error: '缺少必填字段' }, { status: 400 }) return NextResponse.json({ error: '缺少必填字段' }, { status: 400 })
} }
const archiveExtension = path.extname(path.basename(file.name)).toLowerCase()
if (!['.zip', '.rar', '.7z'].includes(archiveExtension)) {
return NextResponse.json({ error: '模型压缩包只支持 ZIP、RAR 或 7Z 格式' }, { status: 415 })
}
// 创建上传目录 // 创建上传目录
const uploadDir = path.join(process.cwd(), 'uploads', 'skus', materialId) const uploadDir = path.join(process.cwd(), 'uploads', 'skus', materialId)
if (!existsSync(uploadDir)) { if (!existsSync(uploadDir)) {
@ -79,6 +103,20 @@ export async function POST(
}, { status: 200 }) }, { status: 200 })
} catch (error) { } catch (error) {
console.error('上传 SKU 失败:', error) console.error('上传 SKU 失败:', error)
const boundedFailure = getBoundedMultipartFailure(error)
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
)
}
return NextResponse.json({ error: '上传 SKU 失败' }, { status: 500 }) return NextResponse.json({ error: '上传 SKU 失败' }, { status: 500 })
} finally {
releaseRequestSlot?.()
} }
} }

View File

@ -18,6 +18,10 @@ import {
canManageMaterial, canManageMaterial,
type MaterialAuthorizationUser, type MaterialAuthorizationUser,
} from '@/lib/material-authorization' } from '@/lib/material-authorization'
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
} from '@/lib/multipart-route-limits'
export const runtime = 'nodejs' export const runtime = 'nodejs'
const MAX_WHITE_BACKGROUND_REQUEST_BYTES = const MAX_WHITE_BACKGROUND_REQUEST_BYTES =
@ -27,6 +31,7 @@ export async function POST(
request: NextRequest, request: NextRequest,
{ params }: { params: { id: string } }, { params }: { params: { id: string } },
) { ) {
let releaseRequestSlot: (() => void) | null = null
try { try {
const session = await getServerSession(authOptions) const session = await getServerSession(authOptions)
if (!session?.user || !(session.user as any).id) { if (!session?.user || !(session.user as any).id) {
@ -67,6 +72,7 @@ export async function POST(
return NextResponse.json({ error: '白底图上传请求过大' }, { status: 413 }) return NextResponse.json({ error: '白底图上传请求过大' }, { status: 413 })
} }
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire()
const formData = await request.formData() const formData = await request.formData()
const skuId = ((formData.get('skuId') as string) || '').trim() const skuId = ((formData.get('skuId') as string) || '').trim()
const skuName = ((formData.get('skuName') as string) || '').trim() const skuName = ((formData.get('skuName') as string) || '').trim()
@ -151,7 +157,21 @@ export async function POST(
...result, ...result,
}) })
} catch (error) { } catch (error) {
const boundedFailure = getBoundedMultipartFailure(error)
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
)
}
console.error('上传白底图失败:', error) console.error('上传白底图失败:', error)
return NextResponse.json({ error: '上传白底图失败' }, { status: 500 }) return NextResponse.json({ error: '上传白底图失败' }, { status: 500 })
} finally {
releaseRequestSlot?.()
} }
} }

View File

@ -3,11 +3,7 @@ import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { isSystemAdmin } from '@/lib/department-permission'; import { isSystemAdmin } from '@/lib/department-permission';
import { deleteFromS3, isS3Url } from '@/lib/s3';
import { deleteFiles } from '@/lib/file-cleanup';
import { collectMaterialFiles, deleteMaterialRelatedRecords } from '@/lib/material-cleanup';
import { createMaterialLog } from '@/lib/material-log'; import { createMaterialLog } from '@/lib/material-log';
import { ensureMaterialDownloadLogsTable } from '@/lib/material-download-log';
import { normalizeUrl, normalizeMaterial } from '@/lib/utils'; import { normalizeUrl, normalizeMaterial } from '@/lib/utils';
import { rewriteMediaUrlForClient } from '@/lib/url-rewrite'; import { rewriteMediaUrlForClient } from '@/lib/url-rewrite';
import { detectIsIntranet } from '@/lib/network-detection'; import { detectIsIntranet } from '@/lib/network-detection';
@ -975,92 +971,19 @@ export async function DELETE(request: NextRequest) {
// 永久删除 // 永久删除
{ {
// 确保下载日志表已应用“删除素材后保留日志”策略
await ensureMaterialDownloadLogsTable();
// 记录日志
await createMaterialLog({
materialId: id,
userId,
action: 'DELETE',
changes: {
materialSnapshot: {
oldValue: null,
newValue: {
id: material.id,
productModel: material.productModel,
materialType: material.materialType,
},
},
deletedAt: {
oldValue: material.deletedAt ? material.deletedAt.toISOString() : null,
newValue: 'PERMANENTLY_DELETED',
},
deleteMode: {
oldValue: material.deletedAt ? 'soft' : 'active',
newValue: 'permanent',
},
},
});
// 使用统一的素材文件收集函数(根据素材类型自动收集所有文件)
const fileInfo = await collectMaterialFiles(id, material);
const filesToDelete = [...fileInfo.fileUrls, ...fileInfo.directories];
console.log(`[Delete] Material ${id} (${(material as any).materialType}) file collection:`, {
fileUrls: fileInfo.fileUrls.length,
directories: fileInfo.directories.length,
relatedRecords: fileInfo.relatedRecords,
});
// 使用统一的文件删除工具批量删除
const result = await deleteFiles(filesToDelete);
const deletedCount = result.deletedCount;
const failedCount = result.failedCount;
if (result.errors.length > 0) {
console.error(`[Delete] Material ${id} files cleanup errors:`, result.errors);
}
console.log(`[Delete] Material ${id} files cleanup: ${deletedCount} deleted, ${failedCount} failed`);
// 删除关联记录(MaterialAsset, MaterialSku 等)
const relatedResult = await deleteMaterialRelatedRecords(id, (material as any).materialType);
if (relatedResult.deletedAssets !== undefined || relatedResult.deletedSkus !== undefined) {
console.log(`[Delete] Material ${id} related records deleted:`, relatedResult);
}
// 从数据库删除
await prisma.material.delete({
where: { id },
});
return NextResponse.json({ return NextResponse.json({
message: '永久删除成功', error: '永久删除已暂停:请先使用可恢复删除;待持久化清理任务上线后再开放物理删除',
deletedFiles: deletedCount, code: 'PERMANENT_DELETE_DISABLED',
failedFiles: failedCount, }, { status: 409 });
});
} }
} catch (error: any) { } catch (error: any) {
// 兼容旧版本数据库(没有deletedAt字段) // 缺少软删除列时必须失败关闭;绝不能悄悄退化成不可恢复的物理删除。
if (error?.meta?.column?.includes('deletedAt')) { if (error?.meta?.column?.includes('deletedAt')) {
console.warn('[Materials API] deletedAt field not found, using hard delete'); console.error('[Materials API] deletedAt field is unavailable; refusing destructive fallback');
const { searchParams } = new URL(request.url); return NextResponse.json({
const id = searchParams.get('id'); error: '当前数据库未完成软删除结构升级,已拒绝不可恢复删除',
code: 'SOFT_DELETE_SCHEMA_REQUIRED',
if (id) { }, { status: 503 });
try {
await prisma.material.delete({
where: { id },
});
return NextResponse.json({ message: '删除成功' });
} catch (e) {
console.error('Error deleting material:', e);
return NextResponse.json(
{ error: '删除失败' },
{ status: 500 }
);
}
}
} }
console.error('Error deleting material:', error); console.error('Error deleting material:', error);

View File

@ -2,7 +2,7 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { transcodeQueue, cleanupTranscodeResidues, VIDEO_MATERIAL_TYPES } from '@/lib/video-transcode-queue'; import { transcodeQueue, VIDEO_MATERIAL_TYPES } from '@/lib/video-transcode-queue';
import { executeTranscodeTask } from '@/lib/video-transcode-task'; import { executeTranscodeTask } from '@/lib/video-transcode-task';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { canUserViewMaterial, hasSystemAdminPermission } from '@/lib/department-permission'; import { canUserViewMaterial, hasSystemAdminPermission } from '@/lib/department-permission';
@ -10,8 +10,6 @@ import { canUserViewMaterial, hasSystemAdminPermission } from '@/lib/department-
export const dynamic = 'force-dynamic'; export const dynamic = 'force-dynamic';
export const maxDuration = 3600; // 1小时 export const maxDuration = 3600; // 1小时
// 恢复标志:确保只执行一次
let recoveryChecked = false;
const MATERIAL_ID_PATTERN = /^[A-Za-z0-9_-]{1,100}$/; const MATERIAL_ID_PATTERN = /^[A-Za-z0-9_-]{1,100}$/;
const RATE_LIMIT_WINDOW_MS = 60 * 1000; const RATE_LIMIT_WINDOW_MS = 60 * 1000;
const STATUS_RATE_LIMIT = 120; const STATUS_RATE_LIMIT = 120;
@ -39,70 +37,6 @@ function rateLimitResponse(retryAfter: number) {
); );
} }
async function countInterruptedVideoTasks(): Promise<number> {
try {
return await prisma.material.count({
where: {
materialType: { in: VIDEO_MATERIAL_TYPES },
videoProcessingStatus: 'processing',
videoSourceUrl: { not: null },
deletedAt: null,
},
});
} catch (error) {
console.error('[Transcode] Failed to count interrupted video tasks:', error);
// 返回 -1 代表统计失败,后续逻辑会谨慎地触发兜底恢复
return -1;
}
}
/**
* R1修复:懒加载恢复中断的转码任务
* - 优先从新的 VideoTranscodeProgress 表恢复(持久化队列状态)
* - 降级到旧的 Material.videoProcessingStatus 恢复(向后兼容)
*/
async function ensureRecoveryCompleted(): Promise<void> {
if (recoveryChecked) {
return;
}
recoveryChecked = true;
try {
console.log('[Transcode] Starting recovery check...');
// R1修复:优先使用新的持久化队列恢复
let recovered = await transcodeQueue.recoverOnStartup(executeTranscodeTask);
// 如果新表没有数据,并且存在真正被中断的任务,再降级到旧方法
if (recovered === 0) {
const interruptedCount = await countInterruptedVideoTasks();
if (interruptedCount > 0 || interruptedCount === -1) {
if (interruptedCount > 0) {
console.log(`[Transcode] Found ${interruptedCount} interrupted processing tasks, running legacy recovery...`);
} else {
console.warn('[Transcode] Unable to verify interrupted tasks, running legacy recovery as a safeguard...');
}
const { recovered: legacyRecovered, requeued } = await transcodeQueue.recoverInterruptedTasks(
executeTranscodeTask,
cleanupTranscodeResidues,
{ videoTypes: VIDEO_MATERIAL_TYPES }
);
recovered += legacyRecovered + requeued;
} else {
console.log('[Transcode] No interrupted processing tasks detected, skipping legacy fallback.');
}
}
console.log(`[Transcode] Recovery completed: ${recovered} tasks recovered`);
} catch (error) {
console.error('[Transcode] Recovery failed:', error);
// 恢复失败不影响正常流程,重置标志以便下次重试
recoveryChecked = false;
}
}
// GET - 查询转码状态 // GET - 查询转码状态
export async function GET(request: NextRequest) { export async function GET(request: NextRequest) {
try { try {
@ -121,10 +55,6 @@ export async function GET(request: NextRequest) {
return rateLimitResponse(retryAfter); return rateLimitResponse(retryAfter);
} }
if (isSystemAdministrator) {
await ensureRecoveryCompleted();
}
const { searchParams } = new URL(request.url); const { searchParams } = new URL(request.url);
const materialId = String(searchParams.get('materialId') || '').trim(); const materialId = String(searchParams.get('materialId') || '').trim();
@ -228,10 +158,6 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ error: '请求参数过大' }, { status: 413 }); return NextResponse.json({ error: '请求参数过大' }, { status: 413 });
} }
if (isSystemAdministrator) {
await ensureRecoveryCompleted();
}
const { materialId, forceRetry } = await request.json(); const { materialId, forceRetry } = await request.json();
if (!MATERIAL_ID_PATTERN.test(String(materialId || '').trim())) { if (!MATERIAL_ID_PATTERN.test(String(materialId || '').trim())) {
@ -254,6 +180,8 @@ export async function POST(request: NextRequest) {
videoCoverUrl: true, videoCoverUrl: true,
videoProcessingStatus: true, videoProcessingStatus: true,
materialType: true, materialType: true,
transcodeLeaseToken: true,
transcodeLeaseExpiresAt: true,
} as any, } as any,
}); });
@ -285,11 +213,21 @@ export async function POST(request: NextRequest) {
} }
// 检查转码状态 // 检查转码状态
if ((material as any).videoProcessingStatus === 'processing' && !forceRetry) { if ((material as any).videoProcessingStatus === 'processing') {
const hasActiveLease = Boolean(
(material as any).transcodeLeaseToken
&& (material as any).transcodeLeaseExpiresAt
&& new Date((material as any).transcodeLeaseExpiresAt).getTime() > Date.now(),
)
return NextResponse.json( return NextResponse.json(
{ error: '转码任务正在进行中', materialId: normalizedMaterialId }, {
{ status: 409 } error: hasActiveLease
); ? '转码任务正在由后台安全处理,请稍后再试'
: '转码任务仍处于处理中,后台会在确认任务失效后自动恢复',
materialId: normalizedMaterialId,
},
{ status: 409 },
)
} }
if ((material as any).videoProcessingStatus === 'done' && !forceRetry) { if ((material as any).videoProcessingStatus === 'done' && !forceRetry) {
@ -314,9 +252,9 @@ export async function POST(request: NextRequest) {
const queueStatus = transcodeQueue.getStatus(); const queueStatus = transcodeQueue.getStatus();
return NextResponse.json({ return NextResponse.json({
message: '转码任务已启动', message: '转码任务已进入后台队列',
materialId: normalizedMaterialId, materialId: normalizedMaterialId,
status: 'processing', status: queueStatus.processingIds.includes(normalizedMaterialId) ? 'processing' : 'queued',
queueStatus: isSystemAdministrator queueStatus: isSystemAdministrator
? queueStatus ? queueStatus
: { : {

View File

@ -3,9 +3,16 @@ import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { updateMaterialWithSharedRules, type MaterialUpdateInput } from '@/lib/material-update'; import { updateMaterialWithSharedRules, type MaterialUpdateInput } from '@/lib/material-update';
import { isWebFileLike } from '@/lib/web-file'; import { isWebFileLike } from '@/lib/web-file';
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
// PUT /api/materials/update - 更新素材(包括预览图) // PUT /api/materials/update - 更新素材(包括预览图)
export async function PUT(request: NextRequest) { export async function PUT(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session?.user) { if (!session?.user) {
@ -18,7 +25,20 @@ export async function PUT(request: NextRequest) {
let previewImage: File | null = null; let previewImage: File | null = null;
if (contentType.includes('multipart/form-data')) { if (contentType.includes('multipart/form-data')) {
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.materialUpdateRequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
previewImage: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.materialUpdatePreviewBytes,
allowedMimeTypes: ['image/jpeg', 'image/png', 'image/webp', 'image/avif'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.materialUpdatePreviewBytes,
});
id = String(formData.get('id') || ''); id = String(formData.get('id') || '');
input = { input = {
productModel: String(formData.get('productModel') || ''), productModel: String(formData.get('productModel') || ''),
@ -91,6 +111,20 @@ export async function PUT(request: NextRequest) {
); );
} catch (error) { } catch (error) {
console.error('Update error:', error); console.error('Update error:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json({ error: '更新失败,请稍后再试' }, { status: 500 }); return NextResponse.json({ error: '更新失败,请稍后再试' }, { status: 500 });
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -1,29 +1,14 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { completeTaskImageMaterial } from '@/lib/task-image-material-completion';
import { assignTaskNoOnComplete } from '@/lib/product-package';
import {
createZipFromFiles,
generatePreviewImages,
cleanupTempFiles,
cleanupOnError,
saveAssets,
} from '@/lib/task-material-processor';
import { createMaterialLog } from '@/lib/material-log';
export const runtime = 'nodejs' export const runtime = 'nodejs';
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic';
export const maxDuration = 10800 export const maxDuration = 10800;
// POST - 完成3D原图上传
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let materialId: string | null = null;
const tempFiles: string[] = []; // 需要清理的临时文件路径
let zipFilePath: string | null = null;
const uploadedPreviewUrls: string[] = []; // 新生成的S3预览图URLs
const uploadIds: string[] = []; // 需要清理的uploadId列表
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -31,138 +16,26 @@ export async function POST(request: NextRequest) {
} }
const body = await request.json(); const body = await request.json();
const { materialId: matId, assets, taskNo, dingtalkApprovalNo } = body; const result = await completeTaskImageMaterial({
materialId: body?.materialId,
materialId = matId;
if (!materialId || !assets) {
return NextResponse.json(
{ error: '缺少必填字段' },
{ status: 400 }
);
}
if (!assets.main || assets.main.length === 0) {
return NextResponse.json(
{ error: '至少需要上传一张原图' },
{ status: 400 }
);
}
console.log('[3D Original Complete] Material ID:', materialId);
console.log('[3D Original Complete] Original images count:', assets.main.length);
// 验证素材是否存在且属于当前用户
const material: any = await prisma.material.findUnique({
where: { id: materialId },
});
if (!material) {
return NextResponse.json(
{ success: false, error: '素材不存在' },
{ status: 404 }
);
}
if (material.userId !== (session.user as any).id) {
return NextResponse.json(
{ success: false, error: '无权限操作此素材' },
{ status: 403 }
);
}
// 步骤1: 从chunk临时目录读取原图文件
console.log(`[3D Original] Processing ${assets.main.length} original images...`);
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions');
const originalFiles: Array<{ path: string; name: string; size: number }> = [];
for (const asset of assets.main) {
try {
const filePath = await getOwnedUploadedFile(asset.uploadId, (session.user as any).id);
if (filePath) {
originalFiles.push({
path: filePath,
name: asset.fileName,
size: asset.fileSize,
});
tempFiles.push(filePath);
uploadIds.push(asset.uploadId);
} else {
throw new Error(`找不到uploadId对应的文件: ${asset.uploadId}`);
}
} catch (error) {
console.error(`[3D Original] 获取原图文件失败 ${asset.uploadId}:`, error);
throw new Error(`获取原图文件失败: ${asset.fileName}`);
}
}
// 步骤2: 打包原图成zip(保存到本地文件系统)
console.log(`[3D Original] Creating zip from ${originalFiles.length} original files...`);
const zipInfo = await createZipFromFiles(
originalFiles,
materialId!,
material.productModel || '3d-original',
'任务-3D原图'
);
console.log(`[3D Original] Zip created: ${zipInfo.fileName} (${zipInfo.fileSize} bytes)`);
zipFilePath = zipInfo.filePath;
// 步骤3: 生成预览图并上传到S3
const previewUrls = await generatePreviewImages(
assets.main.map((a: any) => ({ uploadId: a.uploadId, fileName: a.fileName })),
'任务-3D原图'
);
uploadedPreviewUrls.push(...previewUrls);
if (uploadedPreviewUrls.length === 0) {
throw new Error('生成预览图失败,至少需要一张预览图');
}
// 步骤4: 保存资产记录到数据库
await saveAssets(materialId!, uploadedPreviewUrls, zipInfo);
// 步骤4.5: 仅在素材成功完成后写入任务号
await assignTaskNoOnComplete(materialId!, taskNo, dingtalkApprovalNo);
// 步骤5: 清理临时文件
await cleanupTempFiles(tempFiles, uploadIds);
// 步骤6: 记录上传日志
await createMaterialLog({
materialId: materialId!,
userId: (session.user as any).id, userId: (session.user as any).id,
action: 'UPLOAD', materialType: '任务-3D原图',
changes: { fallbackProductModel: '3d-original',
uploadType: { assets: Array.isArray(body?.assets?.main) ? body.assets.main : [],
oldValue: null, taskNo: body?.taskNo,
newValue: '任务-3D原图', dingtalkApprovalNo: body?.dingtalkApprovalNo,
},
previewCount: {
oldValue: 0,
newValue: uploadedPreviewUrls.length,
},
zipFileName: {
oldValue: null,
newValue: zipInfo.fileName,
},
},
});
console.log('[3D Original] Complete success!');
return NextResponse.json({
success: true,
materialId,
}); });
return NextResponse.json({ success: true, ...result });
} catch (error: any) { } catch (error: any) {
console.error('[3D Original Complete Error]', error); console.error('[3D Original Complete Error]', error);
const status = Number.isInteger(error?.status) ? error.status : 500;
// 清理失败时的资源 const response = NextResponse.json(
await cleanupOnError(uploadedPreviewUrls, zipFilePath, tempFiles, materialId); { error: error?.message || '完成上传失败' },
{ status },
return NextResponse.json(
{ error: error.message || '完成上传失败' },
{ status: 500 }
); );
if (Number.isFinite(error?.retryAfterSeconds)) {
response.headers.set('Retry-After', String(error.retryAfterSeconds));
}
return response;
} }
} }

View File

@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { purgeDeletedMaterialsByTaskNo, validateTaskNo } from '@/lib/product-package'; import { validateTaskNo } from '@/lib/product-package';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
// POST - 初始化3D原图素材记录 // POST - 初始化3D原图素材记录
@ -45,14 +45,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(`[3D Original Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条`);
}
} catch (cleanupError) {
console.error('[3D Original Init] 清理旧软删素材失败:', cleanupError);
}
} }
const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo); const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo);
@ -87,6 +79,7 @@ export async function POST(request: NextRequest) {
previewUrl: '', // 临时空值,完成上传时更新 previewUrl: '', // 临时空值,完成上传时更新
zipFilePath: '', // 临时空值,完成上传时更新 zipFilePath: '', // 临时空值,完成上传时更新
zipFileName: '', // 临时空值,完成上传时更新 zipFileName: '', // 临时空值,完成上传时更新
taskStatus: 'uploading',
renderer: '', // 3D原图不需要渲染器 renderer: '', // 3D原图不需要渲染器
}, },
}); });

View File

@ -1,29 +1,14 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { completeTaskImageMaterial } from '@/lib/task-image-material-completion';
import { assignTaskNoOnComplete } from '@/lib/product-package';
import {
createZipFromFiles,
generatePreviewImages,
cleanupTempFiles,
cleanupOnError,
saveAssets,
} from '@/lib/task-material-processor';
import { createMaterialLog } from '@/lib/material-log';
export const runtime = 'nodejs' export const runtime = 'nodejs';
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic';
export const maxDuration = 10800 export const maxDuration = 10800;
// POST - 完成AI原图上传
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let materialId: string | null = null;
const tempFiles: string[] = []; // 需要清理的临时文件路径
let zipFilePath: string | null = null;
const uploadedPreviewUrls: string[] = []; // 新生成的S3预览图URLs
const uploadIds: string[] = []; // 需要清理的uploadId列表
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -31,138 +16,26 @@ export async function POST(request: NextRequest) {
} }
const body = await request.json(); const body = await request.json();
const { materialId: matId, assets, taskNo, dingtalkApprovalNo } = body; const result = await completeTaskImageMaterial({
materialId: body?.materialId,
materialId = matId;
if (!materialId || !assets) {
return NextResponse.json(
{ error: '缺少必填字段' },
{ status: 400 }
);
}
if (!assets.main || assets.main.length === 0) {
return NextResponse.json(
{ error: '至少需要上传一张原图' },
{ status: 400 }
);
}
console.log('[AI Original Complete] Starting, materialId:', materialId);
console.log('[AI Original Complete] Original images count:', assets.main.length);
// 验证素材是否存在且属于当前用户
const material: any = await prisma.material.findUnique({
where: { id: materialId },
});
if (!material) {
return NextResponse.json(
{ success: false, error: '素材不存在' },
{ status: 404 }
);
}
if (material.userId !== (session.user as any).id) {
return NextResponse.json(
{ success: false, error: '无权限操作此素材' },
{ status: 403 }
);
}
// 步骤1: 从chunk临时目录读取原图文件
console.log(`[AI Original] Processing ${assets.main.length} original images...`);
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions');
const originalFiles: Array<{ path: string; name: string; size: number }> = [];
for (const asset of assets.main) {
try {
const filePath = await getOwnedUploadedFile(asset.uploadId, (session.user as any).id);
if (filePath) {
originalFiles.push({
path: filePath,
name: asset.fileName,
size: asset.fileSize,
});
tempFiles.push(filePath);
uploadIds.push(asset.uploadId);
} else {
throw new Error(`找不到uploadId对应的文件: ${asset.uploadId}`);
}
} catch (error) {
console.error(`[AI Original] Failed to get file for uploadId ${asset.uploadId}:`, error);
throw new Error(`获取原图文件失败: ${asset.fileName}`);
}
}
// 步骤2: 打包原图成zip(保存到本地文件系统)
console.log(`[AI Original] Creating zip from ${originalFiles.length} original files...`);
const zipInfo = await createZipFromFiles(
originalFiles,
materialId!,
material.productModel || 'ai-original',
'任务-AI原图'
);
console.log(`[AI Original] Zip created: ${zipInfo.fileName} (${zipInfo.fileSize} bytes)`);
zipFilePath = zipInfo.filePath;
// 步骤3: 生成预览图并上传到S3
const previewUrls = await generatePreviewImages(
assets.main.map((a: any) => ({ uploadId: a.uploadId, fileName: a.fileName })),
'任务-AI原图'
);
uploadedPreviewUrls.push(...previewUrls);
if (uploadedPreviewUrls.length === 0) {
throw new Error('生成预览图失败,至少需要一张预览图');
}
// 步骤4: 保存资产记录到数据库
await saveAssets(materialId!, uploadedPreviewUrls, zipInfo);
// 步骤4.5: 仅在素材成功完成后写入任务号
await assignTaskNoOnComplete(materialId!, taskNo, dingtalkApprovalNo);
// 步骤5: 清理临时文件
await cleanupTempFiles(tempFiles, uploadIds);
// 步骤6: 记录上传日志
await createMaterialLog({
materialId: materialId!,
userId: (session.user as any).id, userId: (session.user as any).id,
action: 'UPLOAD', materialType: '任务-AI原图',
changes: { fallbackProductModel: 'ai-original',
uploadType: { assets: Array.isArray(body?.assets?.main) ? body.assets.main : [],
oldValue: null, taskNo: body?.taskNo,
newValue: '任务-AI原图', dingtalkApprovalNo: body?.dingtalkApprovalNo,
},
previewCount: {
oldValue: 0,
newValue: uploadedPreviewUrls.length,
},
zipFileName: {
oldValue: null,
newValue: zipInfo.fileName,
},
},
});
console.log('[AI Original] Complete success!');
return NextResponse.json({
success: true,
materialId,
}); });
return NextResponse.json({ success: true, ...result });
} catch (error: any) { } catch (error: any) {
console.error('[AI Original Complete Error]', error); console.error('[AI Original Complete Error]', error);
const status = Number.isInteger(error?.status) ? error.status : 500;
// 清理失败时的资源 const response = NextResponse.json(
await cleanupOnError(uploadedPreviewUrls, zipFilePath, tempFiles, materialId); { error: error?.message || '完成上传失败' },
{ status },
return NextResponse.json(
{ error: error.message || '完成上传失败' },
{ status: 500 }
); );
if (Number.isFinite(error?.retryAfterSeconds)) {
response.headers.set('Retry-After', String(error.retryAfterSeconds));
}
return response;
} }
} }

View File

@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { purgeDeletedMaterialsByTaskNo, validateTaskNo } from '@/lib/product-package'; import { validateTaskNo } from '@/lib/product-package';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
// POST - 初始化AI原图素材记录 // POST - 初始化AI原图素材记录
@ -49,14 +49,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(`[AI Original Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条`);
}
} catch (cleanupError) {
console.error('[AI Original Init] 清理旧软删素材失败:', cleanupError);
}
} }
const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo); const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo);
@ -91,6 +83,7 @@ export async function POST(request: NextRequest) {
previewUrl: '', // 临时空值,完成上传时更新 previewUrl: '', // 临时空值,完成上传时更新
zipFilePath: '', // 临时空值,完成上传时更新 zipFilePath: '', // 临时空值,完成上传时更新
zipFileName: '', // 临时空值,完成上传时更新 zipFileName: '', // 临时空值,完成上传时更新
taskStatus: 'uploading',
renderer: '', // AI原图不需要渲染器 renderer: '', // AI原图不需要渲染器
}, },
}); });

View File

@ -0,0 +1,354 @@
/** @jest-environment node */
import { NextRequest } from 'next/server'
import { getServerSession } from 'next-auth'
import { POST } from '@/app/api/materials/upload/certificate-report/complete/route'
import { prisma } from '@/lib/prisma'
import { getOwnedUploadedFile } from '@/lib/upload-sessions'
import {
uploadPublicDisplayFileToS3,
uploadPublicDisplayToS3,
} from '@/lib/public-display'
import { cleanupTempFiles } from '@/lib/task-material-processor'
import { createMaterialLog } from '@/lib/material-log'
import { deleteFromS3 } from '@/lib/s3'
import { inspectBoundedRegularFile } from '@/lib/bounded-file-reader'
import {
acquireTaskMaterialCompletionLock,
acquireTaskMaterialUploadSessionLocks,
} from '@/lib/material-upload-cross-process-lock'
import { MaterialUploadRequestError } from '@/lib/material-upload-request-guard'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('@/lib/prisma', () => ({
prisma: {
material: {
findUnique: jest.fn(),
},
$transaction: jest.fn(),
},
}))
jest.mock('@/lib/upload-sessions', () => ({
getOwnedUploadedFile: jest.fn(),
}))
jest.mock('@/lib/public-display', () => ({
buildCertificateReportPublicDisplayPath: jest.fn(
(materialId: string, kind: string, fileName: string) => `${materialId}/${kind}/${fileName}`,
),
uploadPublicDisplayFileToS3: jest.fn(),
uploadPublicDisplayToS3: jest.fn(),
}))
jest.mock('@/lib/task-material-processor', () => ({
cleanupTempFiles: jest.fn(),
}))
jest.mock('@/lib/material-log', () => ({
createMaterialLog: jest.fn(),
}))
jest.mock('@/lib/s3', () => ({
deleteFromS3: jest.fn(),
}))
jest.mock('@/lib/pdf-processor', () => ({
generatePdfPreview: jest.fn(),
}))
jest.mock('@/lib/bounded-file-reader', () => ({
inspectBoundedRegularFile: jest.fn(),
}))
jest.mock('@/lib/material-upload-cross-process-lock', () => ({
acquireTaskMaterialCompletionLock: jest.fn(),
acquireTaskMaterialUploadSessionLocks: jest.fn(),
}))
const materialId = 'material-report-1'
const userId = 'user-1'
const uploadId = 'upload-report-1'
const uploadedUrl = 'https://s3.jiansun.vip/gongxiangpan/report.jpg'
function completeRequest(files: unknown[] = [{
uploadId,
fileName: 'report.jpg',
fileSize: 4,
sortOrder: 0,
}]) {
return new NextRequest('http://localhost/api/materials/upload/certificate-report/complete', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ materialId, files }),
})
}
function reportAsset(overrides: Record<string, unknown> = {}) {
return {
assetType: 'report-image',
fileUrl: uploadedUrl,
fileName: 'report.jpg',
fileSize: BigInt(4),
...overrides,
}
}
function materialWithStatus(
taskStatus: string | null,
overrides: Record<string, unknown> = {},
) {
const completed = taskStatus === 'completed'
return {
id: materialId,
userId,
materialType: '证书-检测报告',
taskStatus,
previewUrl: completed ? uploadedUrl : '',
zipFileName: completed ? 'report.jpg' : '',
zipFilePath: '',
zipFileSize: completed ? BigInt(4) : BigInt(0),
deletedAt: null,
assets: completed ? [reportAsset()] : [],
...overrides,
}
}
function createReleaseLock() {
return Object.assign(jest.fn().mockResolvedValue(undefined), {
assertHeld: jest.fn(),
})
}
describe('certificate report complete data safety', () => {
let transactionMaterialUpdateMany: jest.Mock
let transactionAssetCreateMany: jest.Mock
let releaseCompletionLock: ReturnType<typeof createReleaseLock>
let releaseUploadSessionLocks: ReturnType<typeof createReleaseLock>
let consoleError: jest.SpyInstance
let consoleWarn: jest.SpyInstance
beforeEach(() => {
jest.clearAllMocks()
consoleError = jest.spyOn(console, 'error').mockImplementation(() => undefined)
consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => undefined)
releaseCompletionLock = createReleaseLock()
releaseUploadSessionLocks = createReleaseLock()
transactionMaterialUpdateMany = jest.fn().mockResolvedValue({ count: 1 })
transactionAssetCreateMany = jest.fn().mockResolvedValue({ count: 1 })
;(getServerSession as jest.Mock).mockResolvedValue({ user: { id: userId } })
;(prisma.material.findUnique as jest.Mock).mockResolvedValue(materialWithStatus('uploading'))
;(prisma.$transaction as jest.Mock).mockImplementation(async (callback: Function) => (
callback({
material: { updateMany: transactionMaterialUpdateMany },
materialAsset: { createMany: transactionAssetCreateMany },
})
))
;(getOwnedUploadedFile as jest.Mock).mockResolvedValue('/uploads/assembled/report.jpg')
;(inspectBoundedRegularFile as jest.Mock).mockResolvedValue({ size: 4 })
;(uploadPublicDisplayFileToS3 as jest.Mock).mockResolvedValue(uploadedUrl)
;(uploadPublicDisplayToS3 as jest.Mock).mockResolvedValue(uploadedUrl)
;(cleanupTempFiles as jest.Mock).mockResolvedValue(undefined)
;(createMaterialLog as jest.Mock).mockResolvedValue(undefined)
;(deleteFromS3 as jest.Mock).mockResolvedValue(undefined)
;(acquireTaskMaterialCompletionLock as jest.Mock).mockResolvedValue(releaseCompletionLock)
;(acquireTaskMaterialUploadSessionLocks as jest.Mock).mockResolvedValue(releaseUploadSessionLocks)
})
afterEach(() => {
consoleError.mockRestore()
consoleWarn.mockRestore()
})
test('处理期间数据库保持 uploading,最终资产和 completed 在同一事务提交', async () => {
;(uploadPublicDisplayFileToS3 as jest.Mock).mockImplementation(async () => {
expect(prisma.$transaction).not.toHaveBeenCalled()
return uploadedUrl
})
const response = await POST(completeRequest())
expect(response.status).toBe(200)
expect(acquireTaskMaterialCompletionLock).toHaveBeenCalledWith(materialId)
expect(acquireTaskMaterialUploadSessionLocks).toHaveBeenCalledWith([uploadId])
expect(transactionAssetCreateMany).toHaveBeenCalledWith({
data: [expect.objectContaining({
materialId,
assetType: 'report-image',
fileUrl: uploadedUrl,
fileSize: BigInt(4),
})],
})
expect(transactionMaterialUpdateMany).toHaveBeenCalledWith(expect.objectContaining({
where: expect.objectContaining({
id: materialId,
OR: [
{ taskStatus: 'uploading' },
{ taskStatus: null, previewUrl: '', zipFilePath: '' },
],
}),
data: expect.objectContaining({
previewUrl: uploadedUrl,
zipFileName: 'report.jpg',
zipFileSize: BigInt(4),
taskStatus: 'completed',
}),
}))
expect(cleanupTempFiles).toHaveBeenCalledWith(
['/uploads/assembled/report.jpg'],
[uploadId],
)
expect(releaseUploadSessionLocks).toHaveBeenCalledTimes(1)
expect(releaseCompletionLock).toHaveBeenCalledTimes(1)
})
test('同一素材锁繁忙时拒绝重复完成,不上传也不清理文件', async () => {
;(acquireTaskMaterialCompletionLock as jest.Mock).mockRejectedValueOnce(
new MaterialUploadRequestError(
'该素材正在处理,请勿重复提交',
409,
'TASK_MATERIAL_COMPLETION_BUSY',
3,
),
)
const response = await POST(completeRequest())
const body = await response.json()
expect(response.status).toBe(409)
expect(response.headers.get('retry-after')).toBe('3')
expect(body.code).toBe('TASK_MATERIAL_COMPLETION_BUSY')
expect(uploadPublicDisplayFileToS3).not.toHaveBeenCalled()
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).not.toHaveBeenCalled()
})
test('已经 completed 且关键资产完整时幂等成功,并补做源会话清理', async () => {
;(prisma.material.findUnique as jest.Mock).mockResolvedValue(materialWithStatus('completed'))
const response = await POST(completeRequest())
const body = await response.json()
expect(response.status).toBe(200)
expect(body).toMatchObject({ success: true, materialId, alreadyCompleted: true })
expect(prisma.$transaction).not.toHaveBeenCalled()
expect(uploadPublicDisplayFileToS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).toHaveBeenCalledWith([], [uploadId])
})
test('completed 但资产缺失或汇总大小不一致时拒绝假成功并保留文件', async () => {
;(prisma.material.findUnique as jest.Mock).mockResolvedValue(materialWithStatus('completed', {
zipFileSize: BigInt(8),
}))
const response = await POST(completeRequest())
expect(response.status).toBe(409)
expect(uploadPublicDisplayFileToS3).not.toHaveBeenCalled()
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).not.toHaveBeenCalled()
})
test('事务响应异常但数据库实际完整 completed 时保留派生文件并返回成功警告', async () => {
;(prisma.material.findUnique as jest.Mock)
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('completed'))
;(prisma.$transaction as jest.Mock).mockRejectedValueOnce(new Error('connection reset'))
const response = await POST(completeRequest())
const body = await response.json()
expect(response.status).toBe(200)
expect(body).toMatchObject({ success: true, materialId })
expect(body.warning).toContain('响应确认')
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).toHaveBeenCalledWith(
['/uploads/assembled/report.jpg'],
[uploadId],
)
})
test('事务响应异常且 completed 证据不完整时拒绝假成功并保留全部文件', async () => {
;(prisma.material.findUnique as jest.Mock)
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('completed', { assets: [] }))
;(prisma.$transaction as jest.Mock).mockRejectedValueOnce(new Error('connection reset'))
const response = await POST(completeRequest())
expect(response.status).toBe(409)
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).not.toHaveBeenCalled()
expect(releaseUploadSessionLocks).toHaveBeenCalledTimes(1)
expect(releaseCompletionLock).toHaveBeenCalledTimes(1)
})
test('无法确认数据库最终状态时 fail-closed 保留源文件和本次派生文件', async () => {
;(prisma.material.findUnique as jest.Mock)
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockRejectedValueOnce(new Error('database unavailable'))
;(prisma.$transaction as jest.Mock).mockRejectedValueOnce(new Error('connection reset'))
const response = await POST(completeRequest())
expect(response.status).toBe(500)
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).not.toHaveBeenCalled()
expect(releaseUploadSessionLocks).toHaveBeenCalledTimes(1)
expect(releaseCompletionLock).toHaveBeenCalledTimes(1)
})
test('事务明确失败且数据库仍 uploading 时只删除本次派生 S3,不删除源会话和素材', async () => {
;(prisma.material.findUnique as jest.Mock)
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('uploading'))
.mockResolvedValueOnce(materialWithStatus('uploading'))
;(prisma.$transaction as jest.Mock).mockRejectedValueOnce(new Error('write failed'))
const response = await POST(completeRequest())
expect(response.status).toBe(500)
expect(deleteFromS3).toHaveBeenCalledWith(uploadedUrl)
expect(cleanupTempFiles).not.toHaveBeenCalled()
expect(releaseUploadSessionLocks).toHaveBeenCalledTimes(1)
expect(releaseCompletionLock).toHaveBeenCalledTimes(1)
})
test('一次处理异常释放锁后,同一所有者可直接重试而无需重传源文件', async () => {
;(uploadPublicDisplayFileToS3 as jest.Mock).mockRejectedValueOnce(new Error('temporary s3 failure'))
const failedResponse = await POST(completeRequest())
const retryResponse = await POST(completeRequest())
expect(failedResponse.status).toBe(500)
expect(retryResponse.status).toBe(200)
expect(acquireTaskMaterialCompletionLock).toHaveBeenCalledTimes(2)
expect(acquireTaskMaterialUploadSessionLocks).toHaveBeenCalledTimes(2)
expect(getOwnedUploadedFile).toHaveBeenCalledTimes(2)
expect(releaseUploadSessionLocks).toHaveBeenCalledTimes(2)
expect(releaseCompletionLock).toHaveBeenCalledTimes(2)
})
test('重复 uploadId 在加锁和读取源文件前直接拒绝', async () => {
const response = await POST(completeRequest([
{ uploadId, fileName: 'report-a.jpg', fileSize: 4, sortOrder: 0 },
{ uploadId, fileName: 'report-b.jpg', fileSize: 4, sortOrder: 1 },
]))
expect(response.status).toBe(400)
expect(acquireTaskMaterialCompletionLock).not.toHaveBeenCalled()
expect(getOwnedUploadedFile).not.toHaveBeenCalled()
expect(deleteFromS3).not.toHaveBeenCalled()
expect(cleanupTempFiles).not.toHaveBeenCalled()
})
})

View File

@ -3,21 +3,33 @@ import { getServerSession } from 'next-auth'
import { lookup as mimeTypeLookup } from 'mime-types' import { lookup as mimeTypeLookup } from 'mime-types'
import path from 'path' import path from 'path'
import { readFile, unlink } from 'fs/promises' import { readFile, unlink } from 'fs/promises'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
import { import {
CERTIFICATE_REPORT_ASSET_TYPE, CERTIFICATE_REPORT_ASSET_TYPE,
CERTIFICATE_REPORT_MATERIAL_TYPE, CERTIFICATE_REPORT_MATERIAL_TYPE,
CERTIFICATE_REPORT_UPLOAD_LIMITS,
detectFileKindFromName, detectFileKindFromName,
isSupportedCertificateReportFileName,
} from '@/lib/certificate-report' } from '@/lib/certificate-report'
import { import {
buildCertificateReportPublicDisplayPath, buildCertificateReportPublicDisplayPath,
uploadPublicDisplayFileToS3,
uploadPublicDisplayToS3, uploadPublicDisplayToS3,
} from '@/lib/public-display' } from '@/lib/public-display'
import { cleanupTempFiles } from '@/lib/task-material-processor' import { cleanupTempFiles } from '@/lib/task-material-processor'
import { createMaterialLog } from '@/lib/material-log' import { createMaterialLog } from '@/lib/material-log'
import { deleteFromS3 } from '@/lib/s3' import { deleteFromS3 } from '@/lib/s3'
import { generatePdfPreview } from '@/lib/pdf-processor' import { generatePdfPreview } from '@/lib/pdf-processor'
import { inspectBoundedRegularFile } from '@/lib/bounded-file-reader'
import { getOwnedUploadedFile } from '@/lib/upload-sessions'
import {
acquireTaskMaterialCompletionLock,
acquireTaskMaterialUploadSessionLocks,
type MaterialUploadStorageLockRelease,
} from '@/lib/material-upload-cross-process-lock'
import { MaterialUploadRequestError } from '@/lib/material-upload-request-guard'
interface UploadedReportFile { interface UploadedReportFile {
uploadId: string uploadId: string
@ -27,37 +39,192 @@ interface UploadedReportFile {
fileKind: 'pdf' | 'image' fileKind: 'pdf' | 'image'
} }
interface CertificateReportAssetSnapshot {
assetType: string
fileUrl: string
fileName: string | null
fileSize: bigint | number | string
}
interface CertificateReportCompletionSnapshot {
id: string
userId: string
materialType: string
taskStatus: string | null
previewUrl: string
zipFileName: string
zipFilePath: string
zipFileSize: bigint | number | string
deletedAt: Date | null
assets: CertificateReportAssetSnapshot[]
}
type CertificateReportCompletionInspection =
| { state: 'completed'; snapshot: CertificateReportCompletionSnapshot }
| { state: 'completed_incomplete'; snapshot: CertificateReportCompletionSnapshot }
| { state: 'other'; snapshot: CertificateReportCompletionSnapshot | null }
| { state: 'unknown'; error: unknown }
class CertificateReportRequestError extends Error {
constructor(message: string, readonly status = 400) {
super(message)
this.name = 'CertificateReportRequestError'
}
}
function sanitizeFileNameForStorage(fileName: string): string { function sanitizeFileNameForStorage(fileName: string): string {
return String(fileName || '') return String(fileName || '')
.trim() .trim()
.replace(/[\/\\:*?"<>|]/g, '_') .replace(/[\/\\:*?"<>|]/g, '_')
} }
function toPositiveBigInt(value: unknown): bigint | null {
try {
const normalized = BigInt(value as bigint | number | string)
return normalized > BigInt(0) ? normalized : null
} catch {
return null
}
}
function hasCompleteCertificateReportEvidence(
material: CertificateReportCompletionSnapshot,
): boolean {
if (material.taskStatus !== 'completed') {
return false
}
const previewUrl = String(material.previewUrl || '').trim()
const summaryFileName = String(material.zipFileName || '').trim()
const summarySize = toPositiveBigInt(material.zipFileSize)
const reportAssets = Array.isArray(material.assets) ? material.assets : []
if (!previewUrl || !summaryFileName || !summarySize || reportAssets.length === 0) {
return false
}
let assetSizeTotal = BigInt(0)
for (const asset of reportAssets) {
if (
asset.assetType !== CERTIFICATE_REPORT_ASSET_TYPE.PDF
&& asset.assetType !== CERTIFICATE_REPORT_ASSET_TYPE.IMAGE
) {
return false
}
if (!String(asset.fileUrl || '').trim() || !String(asset.fileName || '').trim()) {
return false
}
const assetSize = toPositiveBigInt(asset.fileSize)
if (!assetSize) {
return false
}
assetSizeTotal += assetSize
}
return assetSizeTotal === summarySize
}
async function readCertificateReportCompletionSnapshot(
materialId: string,
): Promise<CertificateReportCompletionSnapshot | null> {
return prisma.material.findUnique({
where: { id: materialId },
select: {
id: true,
userId: true,
materialType: true,
taskStatus: true,
previewUrl: true,
zipFileName: true,
zipFilePath: true,
zipFileSize: true,
deletedAt: true,
assets: {
where: {
assetType: {
in: [CERTIFICATE_REPORT_ASSET_TYPE.PDF, CERTIFICATE_REPORT_ASSET_TYPE.IMAGE],
},
},
select: {
assetType: true,
fileUrl: true,
fileName: true,
fileSize: true,
},
},
},
}) as Promise<CertificateReportCompletionSnapshot | null>
}
async function inspectCertificateReportCompletionAfterError(
materialId: string,
): Promise<CertificateReportCompletionInspection> {
try {
const snapshot = await readCertificateReportCompletionSnapshot(materialId)
if (snapshot?.taskStatus === 'completed') {
return hasCompleteCertificateReportEvidence(snapshot)
? { state: 'completed', snapshot }
: { state: 'completed_incomplete', snapshot }
}
return { state: 'other', snapshot }
} catch (error) {
return { state: 'unknown', error }
}
}
function buildErrorResponse(error: unknown): NextResponse {
const message = error instanceof Error ? error.message : '完成上传失败'
if (error instanceof MaterialUploadRequestError) {
const headers = error.retryAfterSeconds
? { 'Retry-After': String(error.retryAfterSeconds) }
: undefined
return NextResponse.json(
{ error: message, code: error.code },
{ status: error.status, headers },
)
}
return NextResponse.json(
{ error: message },
{ status: error instanceof CertificateReportRequestError ? error.status : 500 },
)
}
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
const tempFiles: string[] = [] const tempFiles: string[] = []
const uploadIds: string[] = [] const uploadIds: string[] = []
const uploadedAssetUrls: string[] = [] const uploadedAssetUrls: string[] = []
const localPreviewPaths = new Set<string>()
let previewUrlToCleanup = '' let previewUrlToCleanup = ''
let materialCommitted = false let materialCommitted = false
let committedMaterialId = '' let committedMaterialId = ''
let authenticatedUserId = ''
let releaseCompletionLock: MaterialUploadStorageLockRelease | null = null
let releaseUploadSessionLocks: MaterialUploadStorageLockRelease | null = null
try { try {
const session = await getServerSession(authOptions) const session = await getServerSession(authOptions)
if (!session?.user) { if (!session?.user) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }) return NextResponse.json({ error: '请先登录' }, { status: 401 })
} }
const userId = String((session.user as any).id || '').trim()
authenticatedUserId = userId
const body = await request.json() const body = await request.json()
const materialId = String(body.materialId || '').trim() const materialId = String(body.materialId || '').trim()
committedMaterialId = materialId committedMaterialId = materialId
const rawFiles = Array.isArray(body.files) ? body.files : [] const rawFiles = Array.isArray(body.files) ? body.files : []
const files = rawFiles if (rawFiles.length > CERTIFICATE_REPORT_UPLOAD_LIMITS.maxFiles) {
.map((item: any) => ({ throw new CertificateReportRequestError(
`检测报告最多上传 ${CERTIFICATE_REPORT_UPLOAD_LIMITS.maxFiles} 个文件,请分批提交`,
413,
)
}
const files: UploadedReportFile[] = rawFiles
.map((item: any, index: number): UploadedReportFile => ({
uploadId: String(item.uploadId || '').trim(), uploadId: String(item.uploadId || '').trim(),
fileName: String(item.fileName || '').trim(), fileName: String(item.fileName || '').trim(),
fileSize: Number(item.fileSize || 0), fileSize: Number(item.fileSize || 0),
sortOrder: Number.isFinite(Number(item.sortOrder)) ? Number(item.sortOrder) : 0, sortOrder: Number.isFinite(Number(item.sortOrder)) ? Number(item.sortOrder) : index,
fileKind: item.fileKind === 'pdf' ? 'pdf' : 'image', fileKind: detectFileKindFromName(item.fileName),
})) }))
.filter((item: UploadedReportFile) => item.uploadId && item.fileName) .filter((item: UploadedReportFile) => item.uploadId && item.fileName)
.sort((left: UploadedReportFile, right: UploadedReportFile) => left.sortOrder - right.sortOrder) .sort((left: UploadedReportFile, right: UploadedReportFile) => left.sortOrder - right.sortOrder)
@ -65,33 +232,86 @@ export async function POST(request: NextRequest) {
if (!materialId || files.length === 0) { if (!materialId || files.length === 0) {
return NextResponse.json({ error: '缺少必填字段' }, { status: 400 }) return NextResponse.json({ error: '缺少必填字段' }, { status: 400 })
} }
if (files.length !== rawFiles.length) {
throw new CertificateReportRequestError('检测报告文件参数不完整,请重新上传')
}
if (files.some((item) => !Number.isSafeInteger(item.sortOrder) || item.sortOrder < 0)) {
throw new CertificateReportRequestError('检测报告文件顺序无效,请重新上传')
}
if (new Set(files.map((item) => item.uploadId)).size !== files.length) {
throw new CertificateReportRequestError('检测报告上传会话编号重复,请重新选择文件')
}
const unsupportedFile = files.find(
(item: UploadedReportFile) => !isSupportedCertificateReportFileName(item.fileName),
)
if (unsupportedFile) {
throw new CertificateReportRequestError(
`检测报告仅支持 PDF、JPEG、PNG、WebP:${unsupportedFile.fileName}`,
)
}
const material = await prisma.material.findUnique({ uploadIds.push(...files.map((item) => item.uploadId))
// 加锁前只做鉴权和类型检查。真正决定状态必须在跨容器锁内重新读取。
const initialMaterial = await prisma.material.findUnique({
where: { id: materialId }, where: { id: materialId },
select: { select: {
id: true, id: true,
userId: true, userId: true,
materialType: true, materialType: true,
productModel: true, deletedAt: true,
reportTitle: true, },
} as any,
}) })
if (!initialMaterial || initialMaterial.deletedAt) {
if (!material) {
return NextResponse.json({ error: '素材不存在' }, { status: 404 }) return NextResponse.json({ error: '素材不存在' }, { status: 404 })
} }
if (initialMaterial.userId !== userId) {
const materialRecord = material as any
if (materialRecord.userId !== (session.user as any).id) {
return NextResponse.json({ error: '无权限操作此素材' }, { status: 403 }) return NextResponse.json({ error: '无权限操作此素材' }, { status: 403 })
} }
if (initialMaterial.materialType !== CERTIFICATE_REPORT_MATERIAL_TYPE) {
if (materialRecord.materialType !== CERTIFICATE_REPORT_MATERIAL_TYPE) {
return NextResponse.json({ error: '素材类型不匹配' }, { status: 400 }) return NextResponse.json({ error: '素材类型不匹配' }, { status: 400 })
} }
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions') releaseCompletionLock = await acquireTaskMaterialCompletionLock(materialId)
releaseUploadSessionLocks = await acquireTaskMaterialUploadSessionLocks(uploadIds)
const material = await readCertificateReportCompletionSnapshot(materialId)
if (!material || material.deletedAt) {
throw new CertificateReportRequestError('素材不存在', 404)
}
if (material.userId !== userId || material.materialType !== CERTIFICATE_REPORT_MATERIAL_TYPE) {
throw new CertificateReportRequestError('素材归属或类型已变化,请刷新后重试', 409)
}
if (material.taskStatus === 'completed') {
if (!hasCompleteCertificateReportEvidence(material)) {
throw new CertificateReportRequestError(
'检测报告完成记录不完整,服务器已保留文件,请联系管理员核查',
409,
)
}
releaseUploadSessionLocks.assertHeld()
await cleanupTempFiles([], uploadIds).catch((cleanupError) => {
console.warn('[Certificate Report Complete] 幂等请求确认已完成,但源会话清理失败:', cleanupError)
})
return NextResponse.json({ success: true, materialId, alreadyCompleted: true })
}
const isLegacyEmptyUpload = (
material.taskStatus == null
&& !material.previewUrl
&& !material.zipFilePath
&& material.assets.length === 0
)
if (material.taskStatus !== 'uploading' && !isLegacyEmptyUpload) {
throw new CertificateReportRequestError('该检测报告正在处理或状态异常,请稍后刷新', 409)
}
if (material.assets.length > 0) {
throw new CertificateReportRequestError(
'检测报告上传记录已有未确认资产,服务器已保留文件,请联系管理员核查',
409,
)
}
const assetRecords: Array<{ const assetRecords: Array<{
materialId: string materialId: string
assetType: string assetType: string
@ -102,46 +322,82 @@ export async function POST(request: NextRequest) {
}> = [] }> = []
let nextPreviewUrl = '' let nextPreviewUrl = ''
let totalFileSize = BigInt(0) let totalFileSize = 0
let firstFileName = '' let firstFileName = ''
const resolvedFiles: Array<UploadedReportFile & { filePath: string; actualSize: number }> = []
for (let index = 0; index < files.length; index += 1) { for (const item of files) {
const item = files[index] const filePath = await getOwnedUploadedFile(item.uploadId, userId)
const filePath = await getOwnedUploadedFile(item.uploadId, (session.user as any).id)
if (!filePath) { if (!filePath) {
throw new Error(`找不到上传文件:${item.fileName}`) throw new CertificateReportRequestError(`找不到上传文件:${item.fileName}`)
}
const maxBytes = item.fileKind === 'pdf'
? CERTIFICATE_REPORT_UPLOAD_LIMITS.maxPdfBytes
: CERTIFICATE_REPORT_UPLOAD_LIMITS.maxImageBytes
let actualSize: number
try {
actualSize = (await inspectBoundedRegularFile(filePath, {
maxBytes,
expectedBytes: item.fileSize,
label: `检测报告文件 ${item.fileName}`,
})).size
} catch (error) {
throw new CertificateReportRequestError(
error instanceof Error ? error.message : `检测报告文件校验失败:${item.fileName}`,
413,
)
}
totalFileSize += actualSize
if (
!Number.isSafeInteger(totalFileSize)
|| totalFileSize > CERTIFICATE_REPORT_UPLOAD_LIMITS.maxTotalBytes
) {
throw new CertificateReportRequestError('检测报告文件总大小不能超过 1GB,请分批提交', 413)
} }
tempFiles.push(filePath) tempFiles.push(filePath)
uploadIds.push(item.uploadId) resolvedFiles.push({ ...item, filePath, actualSize })
totalFileSize += BigInt(item.fileSize || 0)
if (!firstFileName) { if (!firstFileName) {
firstFileName = item.fileName firstFileName = item.fileName
} }
}
const buffer = await readFile(filePath) const targetRelativePaths = new Set<string>()
for (const item of resolvedFiles) {
const safeFileName = sanitizeFileNameForStorage(item.fileName) const safeFileName = sanitizeFileNameForStorage(item.fileName)
const fileKind = item.fileKind || detectFileKindFromName(item.fileName) const fileKind = item.fileKind
const contentType = const contentType =
mimeTypeLookup(path.extname(item.fileName))?.toString() || mimeTypeLookup(path.extname(item.fileName))?.toString()
(fileKind === 'pdf' ? 'application/pdf' : 'image/jpeg') || (fileKind === 'pdf' ? 'application/pdf' : 'image/jpeg')
const targetRelativePath = buildCertificateReportPublicDisplayPath( const targetRelativePath = buildCertificateReportPublicDisplayPath(
materialId, materialId,
fileKind === 'pdf' ? 'pdf' : 'image', fileKind === 'pdf' ? 'pdf' : 'image',
`${String(item.sortOrder).padStart(3, '0')}-${safeFileName}`, `${String(item.sortOrder).padStart(3, '0')}-${safeFileName}`,
) )
if (targetRelativePaths.has(targetRelativePath)) {
throw new CertificateReportRequestError(
`检测报告文件名与顺序重复:${item.fileName}`,
)
}
targetRelativePaths.add(targetRelativePath)
const uploadedUrl = await uploadPublicDisplayToS3(buffer, targetRelativePath, contentType) const uploadedUrl = await uploadPublicDisplayFileToS3(
item.filePath,
targetRelativePath,
contentType,
item.actualSize,
)
uploadedAssetUrls.push(uploadedUrl) uploadedAssetUrls.push(uploadedUrl)
assetRecords.push({ assetRecords.push({
materialId, materialId,
assetType: fileKind === 'pdf' ? CERTIFICATE_REPORT_ASSET_TYPE.PDF : CERTIFICATE_REPORT_ASSET_TYPE.IMAGE, assetType: fileKind === 'pdf'
? CERTIFICATE_REPORT_ASSET_TYPE.PDF
: CERTIFICATE_REPORT_ASSET_TYPE.IMAGE,
fileUrl: uploadedUrl, fileUrl: uploadedUrl,
fileName: item.fileName, fileName: item.fileName,
fileSize: BigInt(item.fileSize || 0), fileSize: BigInt(item.actualSize),
sortOrder: item.sortOrder, sortOrder: item.sortOrder,
}) })
@ -149,8 +405,9 @@ export async function POST(request: NextRequest) {
if (fileKind === 'image') { if (fileKind === 'image') {
nextPreviewUrl = uploadedUrl nextPreviewUrl = uploadedUrl
} else { } else {
const previewResult = await generatePdfPreview(filePath) const previewResult = await generatePdfPreview(item.filePath)
if (previewResult.success && previewResult.previewPath) { if (previewResult.success && previewResult.previewPath) {
localPreviewPaths.add(previewResult.previewPath)
const previewBuffer = await readFile(previewResult.previewPath) const previewBuffer = await readFile(previewResult.previewPath)
const previewPath = buildCertificateReportPublicDisplayPath( const previewPath = buildCertificateReportPublicDisplayPath(
materialId, materialId,
@ -159,44 +416,52 @@ export async function POST(request: NextRequest) {
) )
nextPreviewUrl = await uploadPublicDisplayToS3(previewBuffer, previewPath, 'image/jpeg') nextPreviewUrl = await uploadPublicDisplayToS3(previewBuffer, previewPath, 'image/jpeg')
previewUrlToCleanup = nextPreviewUrl previewUrlToCleanup = nextPreviewUrl
await unlink(previewResult.previewPath).catch(() => {}) await unlink(previewResult.previewPath).catch(() => undefined)
} }
} }
} }
} }
if (!nextPreviewUrl) { if (!nextPreviewUrl || assetRecords.length !== resolvedFiles.length) {
const firstImageAsset = assetRecords.find((item) => item.assetType === CERTIFICATE_REPORT_ASSET_TYPE.IMAGE) throw new Error('检测报告预览或文件资产生成不完整,本次素材未保存')
if (firstImageAsset) {
nextPreviewUrl = firstImageAsset.fileUrl
}
} }
await prisma.$transaction([ releaseCompletionLock.assertHeld()
prisma.materialAsset.createMany({ releaseUploadSessionLocks.assertHeld()
data: assetRecords, await prisma.$transaction(async (tx) => {
}), await tx.materialAsset.createMany({ data: assetRecords })
prisma.material.update({ const finalizeResult = await tx.material.updateMany({
where: { id: materialId }, where: {
id: materialId,
userId,
materialType: CERTIFICATE_REPORT_MATERIAL_TYPE,
deletedAt: null,
OR: [
{ taskStatus: 'uploading' },
{ taskStatus: null, previewUrl: '', zipFilePath: '' },
],
},
data: { data: {
previewUrl: nextPreviewUrl, previewUrl: nextPreviewUrl,
zipFileName: firstFileName, zipFileName: firstFileName,
zipFileSize: totalFileSize, zipFileSize: BigInt(totalFileSize),
taskStatus: 'completed',
} as any, } as any,
}), })
]) if (finalizeResult.count !== 1) {
throw new Error('检测报告完成状态已变化,已停止写入')
}
})
materialCommitted = true materialCommitted = true
try { // 只有 completed 事务确认后才消费源上传会话。清理失败不能反向删除正式资产。
await cleanupTempFiles(tempFiles, uploadIds) await cleanupTempFiles(tempFiles, uploadIds).catch((cleanupError) => {
} catch (cleanupError) { console.warn('[Certificate Report Complete] 清理源上传会话失败,但素材已保存:', cleanupError)
console.warn('[Certificate Report Complete] 清理临时文件失败,但素材已保存:', cleanupError) })
}
try {
await createMaterialLog({ await createMaterialLog({
materialId, materialId,
userId: (session.user as any).id, userId,
action: 'UPLOAD', action: 'UPLOAD',
changes: { changes: {
uploadType: { uploadType: {
@ -208,16 +473,12 @@ export async function POST(request: NextRequest) {
newValue: assetRecords.length, newValue: assetRecords.length,
}, },
}, },
}) }).catch((logError) => {
} catch (logError) {
console.warn('[Certificate Report Complete] 上传日志记录失败,但素材已保存:', logError) console.warn('[Certificate Report Complete] 上传日志记录失败,但素材已保存:', logError)
}
return NextResponse.json({
success: true,
materialId,
}) })
} catch (error: any) {
return NextResponse.json({ success: true, materialId })
} catch (error) {
console.error('[Certificate Report Complete Error]', error) console.error('[Certificate Report Complete Error]', error)
if (materialCommitted) { if (materialCommitted) {
@ -228,22 +489,86 @@ export async function POST(request: NextRequest) {
}) })
} }
if (releaseCompletionLock && committedMaterialId) {
const completionState = await inspectCertificateReportCompletionAfterError(committedMaterialId)
if (completionState.state === 'completed') {
if (releaseUploadSessionLocks) {
try {
releaseUploadSessionLocks.assertHeld()
await cleanupTempFiles(tempFiles, uploadIds)
} catch (cleanupError) {
console.warn('[Certificate Report Complete] 数据库已完成,但源会话清理失败或锁已失效:', cleanupError)
}
}
return NextResponse.json({
success: true,
materialId: committedMaterialId,
warning: '素材已保存,但响应确认阶段出现异常',
})
}
if (completionState.state === 'completed_incomplete') {
console.error('[Certificate Report Complete] 数据库已标记完成,但关键资产证据不完整:', completionState)
return NextResponse.json(
{ error: '检测报告完成记录不完整,服务器已保留文件,请联系管理员核查' },
{ status: 409 },
)
}
if (completionState.state === 'unknown') {
console.error('[Certificate Report Complete] 无法确认数据库最终状态,保留全部文件:', completionState.error)
return NextResponse.json(
{ error: '完成状态暂时无法确认,服务器已保留上传文件,请勿重复提交' },
{ status: 500 },
)
}
const snapshot = completionState.snapshot
const isRetryableUploadState = Boolean(
snapshot
&& !snapshot.deletedAt
&& snapshot.userId === authenticatedUserId
&& snapshot.materialType === CERTIFICATE_REPORT_MATERIAL_TYPE
&& (snapshot.taskStatus === 'uploading' || snapshot.taskStatus == null),
)
if (isRetryableUploadState) {
const cleanupTargets = previewUrlToCleanup const cleanupTargets = previewUrlToCleanup
? Array.from(new Set([...uploadedAssetUrls, previewUrlToCleanup])) ? Array.from(new Set([...uploadedAssetUrls, previewUrlToCleanup]))
: uploadedAssetUrls : Array.from(new Set(uploadedAssetUrls))
let cleanupLocksHeld = true
for (const fileUrl of cleanupTargets) {
try { try {
await deleteFromS3(fileUrl) releaseCompletionLock.assertHeld()
} catch (cleanupError) { if (cleanupTargets.length > 0) {
console.error('[Certificate Report Cleanup] 删除S3文件失败:', cleanupError) if (!releaseUploadSessionLocks) {
cleanupLocksHeld = false
} else {
releaseUploadSessionLocks.assertHeld()
} }
} }
} catch (lockError) {
await cleanupTempFiles(tempFiles, uploadIds).catch((cleanupError) => { cleanupLocksHeld = false
console.error('[Certificate Report Cleanup] 清理临时文件失败:', cleanupError) console.error('[Certificate Report Cleanup] 完成锁已失效,保留派生文件:', lockError)
}
if (cleanupLocksHeld) {
for (const fileUrl of cleanupTargets) {
await deleteFromS3(fileUrl).catch((cleanupError) => {
console.error('[Certificate Report Cleanup] 删除本次派生 S3 文件失败:', cleanupError)
}) })
}
}
} else {
console.warn('[Certificate Report Complete] 素材状态已由其它流程接管,保留全部文件:', completionState)
}
}
return NextResponse.json({ error: error.message || '完成上传失败' }, { status: 500 }) return buildErrorResponse(error)
} finally {
for (const previewPath of Array.from(localPreviewPaths)) {
await unlink(previewPath).catch(() => undefined)
}
await releaseUploadSessionLocks?.().catch((releaseError) => {
console.error('[Certificate Report Complete] 释放上传会话锁失败:', releaseError)
})
await releaseCompletionLock?.().catch((releaseError) => {
console.error('[Certificate Report Complete] 释放素材完成锁失败:', releaseError)
})
} }
} }

View File

@ -49,6 +49,7 @@ export async function POST(request: NextRequest) {
previewUrl: '', previewUrl: '',
zipFilePath: '', zipFilePath: '',
zipFileName: '', zipFileName: '',
taskStatus: 'uploading',
} as any, } as any,
}) })

View File

@ -1,15 +1,44 @@
import { once } from 'events' import { randomUUID } from 'crypto'
import { createWriteStream } from 'fs' import { mkdir, open, rename, rm } from 'fs/promises'
import { mkdir, writeFile } from 'fs/promises'
import path from 'path' import path from 'path'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { getAuthenticatedUploadUserId } from '@/lib/upload-auth' import { getAuthenticatedUploadUserId } from '@/lib/upload-auth'
import { getUploadSession, updateChunkWithStats } from '@/lib/upload-sessions' import { getUploadSession, updateChunkWithStats } from '@/lib/upload-sessions'
import {
getExpectedUploadChunkBytes,
MAX_UPLOAD_CHUNK_BYTES,
resolveUploadChunkSize,
} from '@/lib/material-upload-chunk-policy'
import {
MaterialUploadRequestError,
materialUploadChunkRequestLimiter,
} from '@/lib/material-upload-request-guard'
import {
acquireMaterialUploadStorageLock,
type MaterialUploadStorageLockRelease,
} from '@/lib/material-upload-cross-process-lock'
import { materialUploadStorageReservations } from '@/lib/material-upload-storage-reservation'
import { assertUploadSessionAcceptingChunks } from '@/lib/material-upload-session-seal'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
export const maxDuration = 10800 export const maxDuration = 10800
const MEBIBYTE = 1024 * 1024
const MAX_CHUNK_BYTES = MAX_UPLOAD_CHUNK_BYTES
const MAX_MULTIPART_CHUNK_REQUEST_BYTES = 21 * MEBIBYTE
class ChunkUploadRequestError extends Error {
constructor(
message: string,
public readonly status: number,
public readonly code: string,
) {
super(message)
this.name = 'ChunkUploadRequestError'
}
}
interface ChunkRequestPayload { interface ChunkRequestPayload {
uploadId: string uploadId: string
chunkIndex: number chunkIndex: number
@ -21,6 +50,10 @@ interface ChunkRequestPayload {
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
const requestStartTime = Date.now() const requestStartTime = Date.now()
let releaseChunkSlot: (() => void) | null = null
let releaseCrossProcessLock: MaterialUploadStorageLockRelease | null = null
let releaseStorageReservation: (() => void) | null = null
let pendingChunkPath: string | null = null
console.log(`[UploadChunk] 📥 收到请求 at ${new Date().toISOString()}`) console.log(`[UploadChunk] 📥 收到请求 at ${new Date().toISOString()}`)
try { try {
@ -33,6 +66,10 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }) return NextResponse.json({ error: '请先登录' }, { status: 401 })
} }
validateChunkRequestHeaders(req)
// multipart 的 formData() 会完整解析请求体,必须先进入并发槽,不能让
// 任意数量的 21MiB 请求在限流之外同时占用 Node 内存。
releaseChunkSlot = await materialUploadChunkRequestLimiter.acquire()
const payloadStartedAt = Date.now() const payloadStartedAt = Date.now()
const payload = await resolveChunkPayload(req) const payload = await resolveChunkPayload(req)
const payloadResolveTime = Date.now() - payloadStartedAt const payloadResolveTime = Date.now() - payloadStartedAt
@ -43,6 +80,15 @@ export async function POST(req: NextRequest) {
if (!payload.uploadId || Number.isNaN(payload.chunkIndex)) { if (!payload.uploadId || Number.isNaN(payload.chunkIndex)) {
return NextResponse.json({ error: '缺少必要参数' }, { status: 400 }) return NextResponse.json({ error: '缺少必要参数' }, { status: 400 })
} }
if (!Number.isSafeInteger(payload.chunkSizeHint) || payload.chunkSizeHint <= 0) {
return NextResponse.json({ error: '上传分片大小无效' }, { status: 400 })
}
if (payload.chunkSizeHint > MAX_CHUNK_BYTES) {
return NextResponse.json(
{ error: '单个上传分片不能超过 20MB', code: 'CHUNK_TOO_LARGE' },
{ status: 413 },
)
}
const sessionStartedAt = Date.now() const sessionStartedAt = Date.now()
const sessionData = await getUploadSession(payload.uploadId) const sessionData = await getUploadSession(payload.uploadId)
@ -58,6 +104,10 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: '无权访问此上传会话' }, { status: 403 }) return NextResponse.json({ error: '无权访问此上传会话' }, { status: 403 })
} }
// 已经完成/正在完成的会话不能重新接收分片。这里先快速拒绝;慢网络写完后
// 还会在跨容器共享锁内复查一次,封住 complete 与迟到请求的竞态窗口。
await assertUploadSessionAcceptingChunks(payload.uploadId)
if (payload.chunkIndex < 0 || payload.chunkIndex >= sessionData.totalChunks) { if (payload.chunkIndex < 0 || payload.chunkIndex >= sessionData.totalChunks) {
console.error( console.error(
`[UploadChunk] ❌ chunkIndex超出范围: ${payload.chunkIndex}, 总数: ${sessionData.totalChunks}`, `[UploadChunk] ❌ chunkIndex超出范围: ${payload.chunkIndex}, 总数: ${sessionData.totalChunks}`,
@ -68,6 +118,32 @@ export async function POST(req: NextRequest) {
) )
} }
const sessionChunkSize = resolveUploadChunkSize(
sessionData.fileSize,
sessionData.totalChunks,
sessionData.chunkSize,
)
const expectedChunkBytes = sessionChunkSize
? getExpectedUploadChunkBytes({
fileSize: sessionData.fileSize,
totalChunks: sessionData.totalChunks,
chunkSize: sessionChunkSize,
chunkIndex: payload.chunkIndex,
})
: null
if (!expectedChunkBytes) {
return NextResponse.json(
{ error: '上传会话的分片布局无效,请重新开始上传', code: 'CHUNK_LAYOUT_INVALID' },
{ status: 409 },
)
}
if (payload.chunkSizeHint !== expectedChunkBytes) {
return NextResponse.json(
{ error: '上传分片大小与会话声明不一致', code: 'CHUNK_SIZE_MISMATCH' },
{ status: 400 },
)
}
const wasChunkAlreadyRecorded = sessionData.chunks.has(payload.chunkIndex) const wasChunkAlreadyRecorded = sessionData.chunks.has(payload.chunkIndex)
const nextUploadedChunkCount = wasChunkAlreadyRecorded const nextUploadedChunkCount = wasChunkAlreadyRecorded
? sessionData.chunks.size ? sessionData.chunks.size
@ -78,12 +154,36 @@ export async function POST(req: NextRequest) {
await mkdir(tempDir, { recursive: true }) await mkdir(tempDir, { recursive: true })
const mkdirTime = Date.now() - mkdirStartedAt const mkdirTime = Date.now() - mkdirStartedAt
releaseStorageReservation = await materialUploadStorageReservations.acquire(
tempDir,
expectedChunkBytes,
)
const chunkFilePath = path.join(tempDir, `chunk-${payload.chunkIndex}`) const chunkFilePath = path.join(tempDir, `chunk-${payload.chunkIndex}`)
pendingChunkPath = `${chunkFilePath}.partial-${randomUUID()}`
const writeStartedAt = Date.now() const writeStartedAt = Date.now()
const writtenBytes = const writtenBytes =
payload.mode === 'raw' payload.mode === 'raw'
? await writeRawRequestBodyToFile(req, chunkFilePath) ? await writeRawRequestBodyToFile(req, pendingChunkPath, MAX_CHUNK_BYTES)
: await writeMultipartChunkToFile(payload.chunkFile, chunkFilePath) : await writeMultipartChunkToFile(payload.chunkFile, pendingChunkPath, MAX_CHUNK_BYTES)
if (writtenBytes !== payload.chunkSizeHint) {
throw new ChunkUploadRequestError(
'上传分片实际大小与声明不一致',
400,
'CHUNK_SIZE_MISMATCH',
)
}
// 慢公网接收期间只写唯一 partial,不持有全局共享锁,避免一个 10/20MiB
// 弱网分片让所有用户的 complete 长时间饥饿。写盘完成后才用短临界区
// 发布 chunk-N;跨容器在途分片的最大体量由 20MiB*并发上限约束,且
// statfs 始终额外保留 2GiB,不依赖这把锁承担网络阶段的空间预留。
releaseCrossProcessLock = await acquireMaterialUploadStorageLock('shared')
releaseCrossProcessLock.assertHeld()
await assertUploadSessionAcceptingChunks(payload.uploadId)
await rename(pendingChunkPath, chunkFilePath)
pendingChunkPath = null
const writeTime = Date.now() - writeStartedAt const writeTime = Date.now() - writeStartedAt
const statsStartedAt = Date.now() const statsStartedAt = Date.now()
@ -117,8 +217,85 @@ export async function POST(req: NextRequest) {
mode: payload.mode, mode: payload.mode,
}) })
} catch (error) { } catch (error) {
if (pendingChunkPath) {
await rm(pendingChunkPath, { force: true }).catch(() => undefined)
pendingChunkPath = null
}
if (error instanceof MaterialUploadRequestError) {
return NextResponse.json(
{ error: error.message, code: error.code },
{
status: error.status,
headers: error.retryAfterSeconds
? { 'Retry-After': String(error.retryAfterSeconds) }
: undefined,
},
)
}
if (error instanceof ChunkUploadRequestError) {
return NextResponse.json(
{ error: error.message, code: error.code },
{ status: error.status },
)
}
console.error('[UploadChunk] Error:', error) console.error('[UploadChunk] Error:', error)
return NextResponse.json({ error: '上传块失败' }, { status: 500 }) return NextResponse.json({ error: '上传块失败' }, { status: 500 })
} finally {
releaseStorageReservation?.()
await releaseCrossProcessLock?.()
releaseChunkSlot?.()
}
}
function validateChunkRequestHeaders(req: NextRequest): void {
const contentType = (req.headers.get('content-type') || '').toLowerCase()
const isMultipart = contentType.includes('multipart/form-data')
const rawContentLength = req.headers.get('content-length')
if (!rawContentLength || !/^\d+$/.test(rawContentLength.trim())) {
throw new ChunkUploadRequestError(
'上传分片缺少有效的 Content-Length',
411,
'CONTENT_LENGTH_REQUIRED',
)
}
const contentLength = Number(rawContentLength)
const requestLimit = isMultipart ? MAX_MULTIPART_CHUNK_REQUEST_BYTES : MAX_CHUNK_BYTES
if (!Number.isSafeInteger(contentLength) || contentLength <= 0) {
throw new ChunkUploadRequestError(
'上传分片的 Content-Length 无效',
411,
'CONTENT_LENGTH_REQUIRED',
)
}
if (contentLength > requestLimit) {
throw new ChunkUploadRequestError(
'单个上传分片不能超过 20MB',
413,
'CHUNK_TOO_LARGE',
)
}
const rawHint = req.headers.get('x-chunk-size')
if (rawHint) {
if (!/^\d+$/.test(rawHint.trim())) {
throw new ChunkUploadRequestError('X-Chunk-Size 无效', 400, 'CHUNK_SIZE_INVALID')
}
const hint = Number(rawHint)
if (!Number.isSafeInteger(hint) || hint <= 0 || hint > MAX_CHUNK_BYTES) {
throw new ChunkUploadRequestError(
'单个上传分片不能超过 20MB',
413,
'CHUNK_TOO_LARGE',
)
}
if (!isMultipart && hint !== contentLength) {
throw new ChunkUploadRequestError(
'Content-Length 与 X-Chunk-Size 不一致',
400,
'CHUNK_SIZE_MISMATCH',
)
}
} }
} }
@ -161,51 +338,98 @@ async function resolveChunkPayload(req: NextRequest): Promise<ChunkRequestPayloa
} }
} }
async function writeRawRequestBodyToFile(req: NextRequest, chunkFilePath: string): Promise<number> { async function writeRawRequestBodyToFile(
req: NextRequest,
chunkFilePath: string,
maxBytes: number,
): Promise<number> {
if (!req.body) { if (!req.body) {
throw new Error('请求体为空') throw new Error('请求体为空')
} }
return writeReadableStreamToFile(req.body, chunkFilePath, maxBytes)
}
const reader = req.body.getReader() async function writeMultipartChunkToFile(
const fileStream = createWriteStream(chunkFilePath) chunk: File | null | undefined,
chunkFilePath: string,
maxBytes: number,
): Promise<number> {
if (!chunk) {
throw new Error('缺少 chunk 文件')
}
if (!Number.isSafeInteger(chunk.size) || chunk.size <= 0 || chunk.size > maxBytes) {
throw new ChunkUploadRequestError(
'单个上传分片不能超过 20MB',
413,
'CHUNK_TOO_LARGE',
)
}
return writeReadableStreamToFile(chunk.stream(), chunkFilePath, maxBytes)
}
async function writeReadableStreamToFile(
stream: ReadableStream<Uint8Array>,
targetPath: string,
maxBytes: number,
): Promise<number> {
const reader = stream.getReader()
let fileHandle: Awaited<ReturnType<typeof open>> | null = null
let writtenBytes = 0 let writtenBytes = 0
try { try {
// FileHandle 的 open/write/sync 都是可 await 的 Promise;磁盘满、权限和
// EIO 会回到当前 try/catch,不会像无 error listener 的 WriteStream 那样
// 触发未捕获事件并杀掉整个 Node 进程。
fileHandle = await open(targetPath, 'wx')
while (true) { while (true) {
const { done, value } = await reader.read() const { done, value } = await reader.read()
if (done) { if (done) {
break break
} }
if (!value || value.length === 0) { if (!value || value.length === 0) {
continue continue
} }
writtenBytes += value.length writtenBytes += value.length
if (!fileStream.write(Buffer.from(value))) { if (writtenBytes > maxBytes) {
await once(fileStream, 'drain') await reader.cancel('chunk too large').catch(() => undefined)
throw new ChunkUploadRequestError(
'单个上传分片不能超过 20MB',
413,
'CHUNK_TOO_LARGE',
)
}
let offset = 0
while (offset < value.length) {
const { bytesWritten } = await fileHandle.write(
value,
offset,
value.length - offset,
null,
)
if (bytesWritten <= 0) {
throw new Error('上传分片写入失败')
}
offset += bytesWritten
} }
} }
await fileHandle.sync()
await fileHandle.close()
fileHandle = null
return writtenBytes
} catch (error) { } catch (error) {
fileStream.destroy() if (fileHandle) {
await fileHandle.close().catch(() => undefined)
}
await rm(targetPath, { force: true }).catch(() => undefined)
throw error throw error
} finally { } finally {
reader.releaseLock() reader.releaseLock()
} }
fileStream.end()
await once(fileStream, 'close')
return writtenBytes
}
async function writeMultipartChunkToFile(chunk: File | null | undefined, chunkFilePath: string): Promise<number> {
if (!chunk) {
throw new Error('缺少 chunk 文件')
}
const buffer = Buffer.from(await chunk.arrayBuffer())
await writeFile(chunkFilePath, buffer)
return buffer.length
} }
function safeDecodeURIComponent(value: string): string { function safeDecodeURIComponent(value: string): string {

View File

@ -0,0 +1,116 @@
/** @jest-environment node */
import { mkdir, mkdtemp, rm } from 'fs/promises'
import os from 'os'
import path from 'path'
import { NextRequest } from 'next/server'
import { getServerSession } from 'next-auth'
import { GET, POST } from './route'
import { hasSystemAdminPermission } from '@/lib/department-permission'
import { safelyCleanupMaterialUploadSession } from '@/lib/temp-cleanup'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('@/lib/department-permission', () => ({
hasSystemAdminPermission: jest.fn(),
}))
jest.mock('@/lib/temp-cleanup', () => ({
safelyCleanupMaterialUploadSession: jest.fn(),
}))
jest.mock('@/lib/upload-sessions', () => ({
isSafeUploadId: (value: string) => /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/.test(value),
}))
describe('material upload cleanup route helper coverage', () => {
let uploadRoot: string
beforeEach(async () => {
jest.clearAllMocks()
uploadRoot = await mkdtemp(path.join(os.tmpdir(), 'cleanup-route-'))
process.env.UPLOAD_DIR = uploadRoot
;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'user-1', role: 'SYSTEM_ADMIN', isSuperAdmin: true },
})
;(hasSystemAdminPermission as jest.Mock).mockReturnValue(true)
;(safelyCleanupMaterialUploadSession as jest.Mock).mockResolvedValue({
status: 'deleted',
success: true,
deletedFiles: 2,
freedSpace: 4,
})
})
afterEach(async () => {
delete process.env.UPLOAD_DIR
await rm(uploadRoot, { recursive: true, force: true })
})
test('single-id GET delegates cancellation to the exclusive-lock helper', async () => {
const response = await GET(new NextRequest(
'http://localhost/api/materials/upload/cleanup?uploadId=single-upload',
))
expect(response.status).toBe(200)
expect(safelyCleanupMaterialUploadSession).toHaveBeenCalledWith({
uploadId: 'single-upload',
intent: 'cancel',
})
})
test('batch GET delegates every safe candidate to sweep helper', async () => {
await mkdir(path.join(uploadRoot, 'temp', 'batch-upload'), { recursive: true })
const response = await GET(new NextRequest(
'http://localhost/api/materials/upload/cleanup?maxAge=12&expiredOnly=false',
))
expect(response.status).toBe(200)
expect(safelyCleanupMaterialUploadSession).toHaveBeenCalledWith({
uploadId: 'batch-upload',
intent: 'sweep',
minAgeMs: 12 * 60 * 60 * 1000,
allowYoungOrphaned: true,
})
})
test('POST delegates owner recheck to the exclusive-lock helper', async () => {
const request = new NextRequest('http://localhost/api/materials/upload/cleanup', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ uploadId: 'owned-upload' }),
})
const response = await POST(request)
expect(response.status).toBe(200)
expect(safelyCleanupMaterialUploadSession).toHaveBeenCalledWith({
uploadId: 'owned-upload',
intent: 'cancel',
expectedOwnerId: 'user-1',
})
})
test('helper uncertainty is surfaced as 503 without route-side deletion', async () => {
;(safelyCleanupMaterialUploadSession as jest.Mock).mockResolvedValue({
status: 'unavailable',
success: false,
deletedFiles: 0,
freedSpace: 0,
error: 'Redis 不可用,已保留数据',
code: 'UPLOAD_CLEANUP_REDIS_UNAVAILABLE',
httpStatus: 503,
})
const response = await GET(new NextRequest(
'http://localhost/api/materials/upload/cleanup?uploadId=uncertain-upload',
))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.code).toBe('UPLOAD_CLEANUP_REDIS_UNAVAILABLE')
})
})

View File

@ -1,22 +1,25 @@
// 清理过期或失败的临时上传文件 // Material upload 取消/过期清理公开接口。
// 用于清理:1. 过期的上传会话对应的临时文件 2. 失败的上传临时文件 3. 用户取消上传的临时文件 // 所有入口只做认证和锁外候选筛选;真实删除统一交给独占锁安全 helper。
import { lstat, readdir } from 'fs/promises'
import path from 'path'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import {
deleteOwnedUploadSession,
getOwnedUploadSession,
getUploadSession,
isSafeUploadId,
resolveOwnedMergedUpload,
} from '@/lib/upload-sessions'
import { hasSystemAdminPermission } from '@/lib/department-permission' import { hasSystemAdminPermission } from '@/lib/department-permission'
import { promises as fs } from 'fs' import { resolveStrictChildPath } from '@/lib/path-security'
import path from 'path' import {
safelyCleanupMaterialUploadSession,
type SafeMaterialUploadCleanupResult,
} from '@/lib/temp-cleanup'
import {
isSafeUploadId,
} from '@/lib/upload-sessions'
import { getUploadRootDir } from '@/lib/upload-paths'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
export const maxDuration = 300 // 5分钟 export const maxDuration = 300
const RATE_LIMIT_WINDOW_MS = 60 * 1000 const RATE_LIMIT_WINDOW_MS = 60 * 1000
const requestCounts = new Map<string, { count: number; resetAt: number }>() const requestCounts = new Map<string, { count: number; resetAt: number }>()
@ -43,83 +46,7 @@ function rateLimitResponse(retryAfter: number) {
} }
/** /**
* 清理单个上传会话的临时文件 * GET - 系统管理员清理单个会话,或扫描过期/孤儿目录。
*/
async function cleanupUploadSession(uploadId: string, uploadDir: string): Promise<{
success: boolean
deletedFiles: number
freedSpace: number
error?: string
}> {
const tempDir = path.join(uploadDir, 'temp', uploadId)
let deletedFiles = 0
let freedSpace = 0
try {
// 检查目录是否存在
try {
await fs.access(tempDir)
} catch {
// 目录不存在,说明已经清理过了
return { success: true, deletedFiles: 0, freedSpace: 0 }
}
// 先计算目录大小(用于统计释放空间)
try {
const files = await fs.readdir(tempDir)
for (const file of files) {
const filePath = path.join(tempDir, file)
try {
const stat = await fs.stat(filePath)
if (stat.isFile()) {
freedSpace += stat.size
deletedFiles++ // 统计文件数量
}
} catch {
// 忽略单个文件的错误
}
}
} catch {
// 忽略读取目录的错误
}
// 使用统一的文件删除工具递归删除整个目录
const { deleteLocalFile } = await import('@/lib/file-cleanup')
const result = await deleteLocalFile(tempDir, {
allowDirectoryDelete: true,
})
if (result.success) {
// 如果之前没有统计到文件,至少算删除了目录本身
if (deletedFiles === 0) {
deletedFiles = 1
}
} else {
console.warn(`[Cleanup] 删除目录失败: ${tempDir}`, result.error)
}
return {
success: result.success,
deletedFiles,
freedSpace,
error: result.success ? undefined : result.error || '删除临时目录失败',
}
} catch (error: any) {
return {
success: false,
deletedFiles,
freedSpace,
error: error.message || '清理失败'
}
}
}
/**
* GET - 清理过期的临时上传文件
* 参数:
* - uploadId: 可选,指定要清理的上传ID
* - expiredOnly: 可选,是否只清理过期的(默认true)
* - maxAge: 可选,最大年龄(小时),默认8小时
*/ */
export async function GET(req: NextRequest) { export async function GET(req: NextRequest) {
try { try {
@ -138,7 +65,7 @@ export async function GET(req: NextRequest) {
} }
const { searchParams } = new URL(req.url) const { searchParams } = new URL(req.url)
const uploadId = searchParams.get('uploadId') const uploadId = String(searchParams.get('uploadId') || '').trim()
const expiredOnly = searchParams.get('expiredOnly') !== 'false' const expiredOnly = searchParams.get('expiredOnly') !== 'false'
const maxAgeHours = Number(searchParams.get('maxAge') || '8') const maxAgeHours = Number(searchParams.get('maxAge') || '8')
const scanLimit = Number(searchParams.get('limit') || '100') const scanLimit = Number(searchParams.get('limit') || '100')
@ -151,99 +78,68 @@ export async function GET(req: NextRequest) {
if (uploadId && !isSafeUploadId(uploadId)) { if (uploadId && !isSafeUploadId(uploadId)) {
return NextResponse.json({ error: 'uploadId 格式无效' }, { status: 400 }) return NextResponse.json({ error: 'uploadId 格式无效' }, { status: 400 })
} }
const maxAge = maxAgeHours * 60 * 60 * 1000 // 转换为毫秒
const uploadDir = process.env.UPLOAD_DIR || './uploads' if (uploadId) {
const tempBaseDir = path.join(uploadDir, 'temp') const cleanup = await safelyCleanupMaterialUploadSession({
uploadId,
intent: 'cancel',
})
return singleCleanupResponse(cleanup)
}
// 确保临时目录存在 const tempBaseDir = path.join(getUploadRootDir(), 'temp')
try { try {
await fs.access(tempBaseDir) const stats = await lstat(tempBaseDir)
} catch { if (stats.isSymbolicLink() || !stats.isDirectory()) {
return NextResponse.json(
{ error: '临时目录状态异常,已停止清理' },
{ status: 503 },
)
}
} catch (error: any) {
if (error?.code === 'ENOENT') {
return NextResponse.json({ return NextResponse.json({
message: '临时目录不存在,无需清理', message: '临时目录不存在,无需清理',
cleaned: 0, cleaned: 0,
freedSpace: 0 freedSpace: 0,
skipped: 0,
}) })
} }
throw error
}
const candidates = (await readdir(tempBaseDir)).slice(0, scanLimit)
let cleaned = 0 let cleaned = 0
let totalFreedSpace = 0 let totalFreedSpace = 0
let skipped = 0
const errors: string[] = [] const errors: string[] = []
const maxAgeMs = maxAgeHours * 60 * 60 * 1000
if (uploadId) { for (const candidate of candidates) {
// 清理指定的上传ID if (!isSafeUploadId(candidate)) {
const result = await cleanupUploadSession(uploadId, uploadDir)
if (result.success) {
cleaned = result.deletedFiles
totalFreedSpace = result.freedSpace
} else {
errors.push(result.error || '清理失败')
}
} else {
// 清理所有过期或无效的临时文件
const now = Date.now()
const tempDirs = (await fs.readdir(tempBaseDir)).slice(0, scanLimit)
for (const dir of tempDirs) {
const uploadIdFromDir = dir
if (!isSafeUploadId(uploadIdFromDir)) {
continue continue
} }
const tempDir = path.join(tempBaseDir, dir) // 路径检查只负责候选边界;helper 会在独占锁内重新 lstat/读 meta。
if (!resolveStrictChildPath(tempBaseDir, candidate)) {
try { skipped += 1
// 检查目录
const stat = await fs.stat(tempDir)
if (!stat.isDirectory()) {
continue continue
} }
// 读取元数据文件(如果存在) const cleanup = await safelyCleanupMaterialUploadSession({
const metaPath = path.join(tempDir, 'meta.json') uploadId: candidate,
let createdAt = stat.mtimeMs // 使用目录修改时间作为fallback intent: 'sweep',
minAgeMs: maxAgeMs,
try { allowYoungOrphaned: !expiredOnly,
const metaContent = await fs.readFile(metaPath, 'utf-8') })
const meta = JSON.parse(metaContent) if (cleanup.success && cleanup.status === 'deleted') {
if (meta.createdAt) { cleaned += cleanup.deletedFiles
createdAt = meta.createdAt totalFreedSpace += cleanup.freedSpace
} continue
} catch {
// 元数据文件不存在,使用目录修改时间
} }
// 检查是否过期 skipped += 1
const age = now - createdAt if (cleanup.status === 'unavailable') {
const isExpired = age > maxAge errors.push(`${candidate}: ${cleanup.error || '状态无法确认,已保留数据'}`)
// 检查会话是否存在
const session = await getUploadSession(uploadIdFromDir)
const sessionExists = !!session
// 决定是否清理
let shouldCleanup = false
if (expiredOnly) {
// 只清理过期的
shouldCleanup = isExpired
} else {
// 清理所有:过期的 或 会话不存在的
shouldCleanup = isExpired || !sessionExists
}
if (shouldCleanup) {
const result = await cleanupUploadSession(uploadIdFromDir, uploadDir)
if (result.success) {
cleaned += result.deletedFiles
totalFreedSpace += result.freedSpace
console.log(`[Cleanup] 清理上传会话: ${uploadIdFromDir}, 文件数: ${result.deletedFiles}, 释放空间: ${formatBytes(result.freedSpace)}`)
} else {
errors.push(`${uploadIdFromDir}: ${result.error}`)
}
}
} catch (err: any) {
errors.push(`${dir}: ${err.message || '读取失败'}`)
}
} }
} }
@ -252,19 +148,21 @@ export async function GET(req: NextRequest) {
cleaned, cleaned,
freedSpace: totalFreedSpace, freedSpace: totalFreedSpace,
freedSpaceFormatted: formatBytes(totalFreedSpace), freedSpaceFormatted: formatBytes(totalFreedSpace),
errors: errors.length > 0 ? errors : undefined skipped,
errors: errors.length > 0 ? errors : undefined,
}) })
} catch (error: any) { } catch (error: any) {
console.error('[Cleanup] Error:', error) console.error('[Cleanup] GET Error:', error)
return NextResponse.json( return NextResponse.json(
{ error: '清理失败: ' + (error.message || '未知错误') }, { error: '清理失败: ' + (error.message || '未知错误') },
{ status: 500 } { status: 503 },
) )
} }
} }
/** /**
* POST - 清理指定的上传会话(用于用户取消上传时调用) * POST - 当前用户取消自己的上传。候选 owner 可在锁外读取,但 helper 会在
* 独占锁内从 Redis/内存/meta.json 重新核对,且 completed 永远不能公开删除。
*/ */
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
try { try {
@ -273,7 +171,11 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }) return NextResponse.json({ error: '请先登录' }, { status: 401 })
} }
const retryAfter = consumeRateLimit(`own:${String((session.user as any)?.id || 'unknown')}`, 30) const currentUserId = String((session.user as any)?.id || '').trim()
if (!currentUserId) {
return NextResponse.json({ error: '登录会话缺少用户标识' }, { status: 401 })
}
const retryAfter = consumeRateLimit(`own:${currentUserId}`, 30)
if (retryAfter) { if (retryAfter) {
return rateLimitResponse(retryAfter) return rateLimitResponse(retryAfter)
} }
@ -282,70 +184,74 @@ export async function POST(req: NextRequest) {
const uploadId = typeof body?.uploadId === 'string' ? body.uploadId.trim() : '' const uploadId = typeof body?.uploadId === 'string' ? body.uploadId.trim() : ''
const existingFilePath = const existingFilePath =
typeof body?.existingFilePath === 'string' ? body.existingFilePath.trim() : '' typeof body?.existingFilePath === 'string' ? body.existingFilePath.trim() : ''
if (!uploadId && !existingFilePath) { if (!uploadId && !existingFilePath) {
return NextResponse.json({ error: '缺少 uploadId 或 existingFilePath' }, { status: 400 }) return NextResponse.json({ error: '缺少 uploadId 或 existingFilePath' }, { status: 400 })
} }
if (uploadId && !isSafeUploadId(uploadId)) { if (uploadId && !isSafeUploadId(uploadId)) {
return NextResponse.json({ error: 'uploadId 格式无效' }, { status: 400 }) return NextResponse.json({ error: 'uploadId 格式无效' }, { status: 400 })
} }
const uploadDir = process.env.UPLOAD_DIR || './uploads'
let deletedFiles = 0
let freedSpace = 0
const currentUserId = String((session.user as any).id || '')
let effectiveUploadId = uploadId let effectiveUploadId = uploadId
if (existingFilePath) { if (existingFilePath) {
const ownedMergedUpload = await resolveOwnedMergedUpload({ const derivedUploadId = deriveUploadIdCandidate(existingFilePath, uploadId || null)
existingFilePath, if (!derivedUploadId) {
uploadId: uploadId || null, return NextResponse.json({ error: '合并文件路径无效' }, { status: 400 })
userId: currentUserId, }
}) effectiveUploadId = derivedUploadId
if (!ownedMergedUpload) { }
return NextResponse.json({ error: '上传会话不存在或不属于当前用户' }, { status: 404 })
} const cleanup = await safelyCleanupMaterialUploadSession({
effectiveUploadId = ownedMergedUpload.uploadId uploadId: effectiveUploadId,
} else { intent: 'cancel',
const ownedSession = await getOwnedUploadSession(uploadId, currentUserId) expectedOwnerId: currentUserId,
if (!ownedSession) {
return NextResponse.json({ error: '上传会话不存在或不属于当前用户' }, { status: 404 })
}
}
const deletedSession = await deleteOwnedUploadSession(effectiveUploadId, currentUserId)
if (!deletedSession) {
return NextResponse.json({ error: '上传会话不存在或不属于当前用户' }, { status: 404 })
}
if (effectiveUploadId) {
const result = await cleanupUploadSession(effectiveUploadId, uploadDir)
if (!result.success) {
return NextResponse.json(
{ error: result.error || '清理失败' },
{ status: 500 }
)
}
deletedFiles += result.deletedFiles
freedSpace += result.freedSpace
}
return NextResponse.json({
message: '清理成功',
deletedFiles,
freedSpace,
freedSpaceFormatted: formatBytes(freedSpace)
}) })
return singleCleanupResponse(cleanup)
} catch (error: any) { } catch (error: any) {
console.error('[Cleanup] Error:', error) console.error('[Cleanup] POST Error:', error)
return NextResponse.json( return NextResponse.json(
{ error: '清理失败: ' + (error.message || '未知错误') }, { error: '清理失败: ' + (error.message || '未知错误') },
{ status: 500 } { status: 503 },
) )
} }
} }
function deriveUploadIdCandidate(
existingFilePath: string,
suppliedUploadId: string | null,
): string | null {
const tempRoot = path.resolve(getUploadRootDir(), 'temp')
const targetPath = path.resolve(existingFilePath)
const sessionDir = path.dirname(targetPath)
const derivedUploadId = path.basename(sessionDir)
if (
path.dirname(sessionDir) !== tempRoot
|| !path.basename(targetPath).startsWith('merged-')
|| !isSafeUploadId(derivedUploadId)
|| (suppliedUploadId && suppliedUploadId !== derivedUploadId)
) {
return null
}
return derivedUploadId
}
function singleCleanupResponse(cleanup: SafeMaterialUploadCleanupResult) {
if (!cleanup.success) {
return NextResponse.json(
{
error: cleanup.error || '该上传会话当前不能清理',
code: cleanup.code,
},
{ status: cleanup.httpStatus || (cleanup.status === 'unavailable' ? 503 : 409) },
)
}
return NextResponse.json({
message: cleanup.status === 'absent' ? '上传会话已清理' : '清理成功',
deletedFiles: cleanup.deletedFiles,
freedSpace: cleanup.freedSpace,
freedSpaceFormatted: formatBytes(cleanup.freedSpace),
})
}
function formatBytes(bytes: number): string { function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B' if (bytes === 0) return '0 B'
const k = 1024 const k = 1024

View File

@ -0,0 +1,152 @@
/** @jest-environment node */
import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'fs/promises'
import os from 'os'
import path from 'path'
import { NextRequest } from 'next/server'
import { getServerSession } from 'next-auth'
import { POST } from '@/app/api/materials/upload/complete/route'
import {
mergeUploadChunksAtomically,
} from '@/lib/material-upload-merge'
import { MaterialUploadStorageReservationManager } from '@/lib/material-upload-storage-reservation'
import { cancelUploadSessionSeal } from '@/lib/material-upload-session-seal'
import {
deleteUploadSession,
getOwnedUploadSession,
} from '@/lib/upload-sessions'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('@/lib/upload-sessions', () => ({
deleteUploadSession: jest.fn(),
getOwnedUploadSession: jest.fn(),
isSafeUploadId: jest.fn((uploadId: string) => /^upload-[A-Za-z0-9-]+$/.test(uploadId)),
}))
function completeRequest(uploadId = 'upload-test-1') {
return new NextRequest('http://localhost/api/materials/upload/complete', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ uploadId, originalFileName: 'video.mp4' }),
})
}
describe('material upload complete safety', () => {
let uploadRoot: string
const uploadId = 'upload-test-1'
const fileName = 'video.mp4'
const fileBytes = Buffer.from('abcdefgh')
beforeEach(async () => {
jest.clearAllMocks()
uploadRoot = await mkdtemp(path.join(os.tmpdir(), 'material-complete-'))
process.env.UPLOAD_DIR = uploadRoot
;(getServerSession as jest.Mock).mockResolvedValue({ user: { id: 'user-1' } })
;(getOwnedUploadSession as jest.Mock).mockResolvedValue({
fileName,
fileSize: fileBytes.length,
totalChunks: 2,
chunkSize: 4,
chunks: new Map(),
createdAt: Date.now(),
userId: 'user-1',
})
;(deleteUploadSession as jest.Mock).mockResolvedValue(undefined)
const tempDir = path.join(uploadRoot, 'temp', uploadId)
await mkdir(tempDir, { recursive: true })
await writeFile(path.join(tempDir, 'meta.json'), '{}')
await writeFile(path.join(tempDir, 'chunk-0'), fileBytes.subarray(0, 4))
await writeFile(path.join(tempDir, 'chunk-1'), fileBytes.subarray(4))
})
afterEach(async () => {
delete process.env.UPLOAD_DIR
await rm(uploadRoot, { recursive: true, force: true })
})
test('首次成功响应丢失后,第二次 complete 幂等返回同一个完整文件', async () => {
const firstResponse = await POST(completeRequest())
const firstBody = await firstResponse.json()
expect(firstResponse.status).toBe(200)
expect(await readFile(firstBody.existingFilePath)).toEqual(fileBytes)
await expect(readFile(path.join(uploadRoot, 'temp', uploadId, 'chunk-0')))
.rejects.toMatchObject({ code: 'ENOENT' })
const secondResponse = await POST(completeRequest())
const secondBody = await secondResponse.json()
expect(secondResponse.status).toBe(200)
expect(secondBody.existingFilePath).toBe(firstBody.existingFilePath)
expect(secondBody.fileSize).toBe(fileBytes.length)
expect(await readFile(secondBody.existingFilePath)).toEqual(fileBytes)
})
test('cancelled 终态让 complete 结构化 fail-closed', async () => {
await cancelUploadSessionSeal(uploadId)
const response = await POST(completeRequest())
const body = await response.json()
expect(response.status).toBe(409)
expect(body.code).toBe('UPLOAD_SESSION_CANCELLED')
await expect(readFile(path.join(uploadRoot, 'temp', uploadId, 'chunk-0')))
.resolves.toEqual(fileBytes.subarray(0, 4))
expect(deleteUploadSession).not.toHaveBeenCalled()
})
test('合并前磁盘不足时 fail-closed 并返回可识别的 507 错误', async () => {
const reservations = new MaterialUploadStorageReservationManager(
100,
async () => BigInt(1_000),
)
await expect(
reservations.acquire(uploadRoot, 901),
).rejects.toMatchObject({
status: 507,
code: 'UPLOAD_STORAGE_LOW',
})
})
test('合并预留会持续扣住额度,后续分片不能吃掉尚未写入的空间', async () => {
const reservations = new MaterialUploadStorageReservationManager(
100,
async () => BigInt(1_000),
)
const releaseMerge = await reservations.acquire(uploadRoot, 700)
expect(reservations.snapshot()).toEqual({ reservedBytes: BigInt(700) })
await expect(reservations.acquire(uploadRoot, 201)).rejects.toMatchObject({
status: 507,
code: 'UPLOAD_STORAGE_LOW',
})
releaseMerge()
const releaseChunk = await reservations.acquire(uploadRoot, 201)
expect(reservations.snapshot()).toEqual({ reservedBytes: BigInt(201) })
releaseChunk()
expect(reservations.snapshot()).toEqual({ reservedBytes: BigInt(0) })
})
test('原子改名失败时删除本次 partial,不留下占盘半成品', async () => {
const chunkPath = path.join(uploadRoot, 'chunk.bin')
const occupiedOutputPath = path.join(uploadRoot, 'occupied-output')
await writeFile(chunkPath, Buffer.from('safe'))
await mkdir(occupiedOutputPath)
await expect(mergeUploadChunksAtomically({
chunks: [{ path: chunkPath, expectedBytes: 4 }],
expectedTotalBytes: 4,
outputPath: occupiedOutputPath,
})).rejects.toBeTruthy()
expect(
(await readdir(uploadRoot)).filter((name) => name.startsWith('occupied-output.partial-')),
).toEqual([])
})
})

View File

@ -1,123 +1,277 @@
// 分片上传完成合并接口 // 分片上传完成合并接口
// 负责将分片文件按顺序合并为完整文件,校验大小,并返回服务器侧可复用的文件路径 // 负责将分片文件按顺序合并为完整文件,校验大小,并返回服务器侧可复用的文件路径
// 供前端在第二步调用正式上传接口时复用,避免重复上传大文件 // 供前端在第二步调用正式上传接口时复用,避免重复上传大文件
import { lstat, rm } from 'fs/promises'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import path from 'path' import {
import fs from 'fs/promises' acquireUploadMergeDiskReservation,
import { createWriteStream, createReadStream } from 'fs' mergeUploadChunksAtomically,
import { pipeline } from 'stream/promises' type UploadChunkToMerge,
import { getUploadSession, deleteUploadSession } from '@/lib/upload-sessions' } from '@/lib/material-upload-merge'
import { sanitizeFileName } from '@/lib/image-processor' import {
getExpectedUploadChunkBytes,
resolveUploadChunkSize,
} from '@/lib/material-upload-chunk-policy'
import {
MaterialUploadRequestError,
materialUploadMergeRequestLimiter,
} from '@/lib/material-upload-request-guard'
import {
deleteUploadSession,
getOwnedUploadSession,
isSafeUploadId,
} from '@/lib/upload-sessions'
import { getUploadRootDir } from '@/lib/upload-paths'
import { normalizeSafeFileName, resolveStrictChildPath } from '@/lib/path-security'
import {
acquireMaterialUploadStorageLock,
type MaterialUploadStorageLockRelease,
} from '@/lib/material-upload-cross-process-lock'
import {
abortUploadCompletionSeal,
beginUploadCompletionSeal,
completeUploadCompletionSeal,
type UploadCompletionSealHandle,
} from '@/lib/material-upload-session-seal'
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
export const maxDuration = 10800 // 最长3小时 export const maxDuration = 10800 // 最长3小时
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
let releaseMergeSlot: (() => void) | null = null
let releaseCrossProcessLock: MaterialUploadStorageLockRelease | null = null
let releaseStorageReservation: (() => void) | null = null
let completionSealHandle: UploadCompletionSealHandle | null = null
let completionSealFinalized = false
try { try {
const session = await getServerSession(authOptions) const session = await getServerSession(authOptions)
if (!session) { const userId = String((session?.user as { id?: unknown } | undefined)?.id || '').trim()
if (!userId) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }) return NextResponse.json({ error: '请先登录' }, { status: 401 })
} }
const { uploadId, originalFileName } = await req.json() const body = await req.json()
const uploadId = String(body?.uploadId || '').trim()
if (!uploadId) { if (!isSafeUploadId(uploadId)) {
return NextResponse.json({ error: '缺少 uploadId' }, { status: 400 }) return NextResponse.json({ error: 'uploadId 无效' }, { status: 400 })
} }
const uploadDir = process.env.UPLOAD_DIR || './uploads' // Redis 会话可能在第一次成功响应前后被清理;归属明确的 meta.json 是
const tempDir = path.join(uploadDir, 'temp', uploadId) // 幂等重试依据,保证公网/FRP 丢响应后无需重新上传整个大文件。
const uploadSession = await getOwnedUploadSession(uploadId, userId)
// 读取会话
const uploadSession = await getUploadSession(uploadId)
if (!uploadSession) { if (!uploadSession) {
return NextResponse.json({ error: '上传会话不存在或已过期' }, { status: 404 }) return NextResponse.json({ error: '上传会话不存在、已过期或无权访问' }, { status: 404 })
} }
// 会话归属校验 const mergedFileName = normalizeSafeFileName(uploadSession.fileName)
if (uploadSession.userId !== (session.user as any).id) { const uploadRoot = getUploadRootDir()
return NextResponse.json({ error: '无权访问此上传会话' }, { status: 403 }) const tempDir = resolveStrictChildPath(uploadRoot, 'temp', uploadId)
const mergedPath = mergedFileName
? resolveStrictChildPath(tempDir || '', `merged-${mergedFileName}`)
: null
if (!tempDir || !mergedFileName || !mergedPath) {
return NextResponse.json({ error: '上传会话路径无效' }, { status: 400 })
} }
const totalChunks = uploadSession.totalChunks const chunkSize = resolveUploadChunkSize(
const fileSize = uploadSession.fileSize uploadSession.fileSize,
uploadSession.totalChunks,
// 收集分片文件 uploadSession.chunkSize,
const chunkFiles = [] )
for (let i = 0; i < totalChunks; i++) { if (!chunkSize) {
const chunkPath = path.join(tempDir, `chunk-${i}`)
chunkFiles.push(chunkPath)
}
// 校验分片是否齐全
for (const chunkPath of chunkFiles) {
try {
await fs.access(chunkPath)
} catch {
return NextResponse.json({ error: '分片不完整,请重新上传' }, { status: 400 })
}
}
// 合并文件
await fs.mkdir(tempDir, { recursive: true })
// 处理文件名:提取纯文件名(去除路径),并清理特殊字符,避免中文路径或特殊字符导致的问题
const rawFileName = originalFileName || uploadSession.fileName || `upload-${uploadId}`
const baseFileName = path.basename(rawFileName) // 提取文件名部分,去除路径
const mergedFileName = sanitizeFileName(baseFileName) || `upload-${uploadId}` // 清理特殊字符
const mergedPath = path.join(tempDir, `merged-${mergedFileName}`)
const mergedStream = createWriteStream(mergedPath)
for (const chunkPath of chunkFiles) {
const chunkStream = createReadStream(chunkPath)
await pipeline(chunkStream, mergedStream, { end: false })
}
// 手动结束写入并等待关闭
mergedStream.end()
await new Promise<void>((resolve, reject) => {
mergedStream.on('close', () => resolve())
mergedStream.on('error', (err) => reject(err))
})
// 校验大小
const stat = await fs.stat(mergedPath)
if (stat.size !== fileSize) {
return NextResponse.json( return NextResponse.json(
{ error: `合并后的文件大小不匹配,期望 ${fileSize} 实际 ${stat.size}` }, { error: '上传会话的分片布局无效,请重新开始上传', code: 'CHUNK_LAYOUT_INVALID' },
{ status: 400 } { status: 409 },
) )
} }
// 清理分片,保留合并后的文件供后续正式上传使用 const chunks: UploadChunkToMerge[] = []
// 使用统一的文件删除工具 for (let index = 0; index < uploadSession.totalChunks; index += 1) {
const { deleteFiles } = await import('@/lib/file-cleanup') const expectedBytes = getExpectedUploadChunkBytes({
const result = await deleteFiles(chunkFiles) fileSize: uploadSession.fileSize,
if (result.failedCount > 0) { totalChunks: uploadSession.totalChunks,
console.warn('[UploadComplete] 清理分片失败:', result.errors) chunkSize,
chunkIndex: index,
})
const chunkPath = resolveStrictChildPath(tempDir, `chunk-${index}`)
if (!expectedBytes || !chunkPath) {
return NextResponse.json(
{ error: '上传会话的分片布局无效,请重新开始上传', code: 'CHUNK_LAYOUT_INVALID' },
{ status: 409 },
)
}
chunks.push({ path: chunkPath, expectedBytes })
} }
// 上传完成后,立即删除会话(不再占用Redis内存) releaseMergeSlot = await materialUploadMergeRequestLimiter.acquire()
try { releaseCrossProcessLock = await acquireMaterialUploadStorageLock('exclusive')
const { deleteUploadSession } = await import('@/lib/upload-sessions-redis') const completionSeal = await beginUploadCompletionSeal(uploadId)
await deleteUploadSession(uploadId) if (completionSeal.state === 'sealing') {
console.log('[UploadComplete] 已删除会话:', uploadId) completionSealHandle = completionSeal.handle
} catch (deleteError) {
console.warn('[UploadComplete] 删除会话失败(忽略):', deleteError)
} }
// 返回合并结果 // 第一次请求可能已经成功,只是响应在公网链路中丢失。完整且大小一致的
return NextResponse.json({ // merged 文件是同一会话的确定结果,重复调用直接返回同一路径。
message: '分片合并完成', const existingMerged = await inspectExistingMergedFile(
existingFilePath: mergedPath, mergedPath,
fileName: mergedFileName, uploadSession.fileSize,
fileSize: stat.size, )
if (existingMerged === 'valid') {
if (completionSealHandle) {
await completeUploadCompletionSeal(completionSealHandle)
completionSealFinalized = true
}
await cleanupCompletedUpload(uploadId, chunks.map((chunk) => chunk.path))
return createCompletedResponse({
uploadId, uploadId,
mergedPath,
mergedFileName,
fileSize: uploadSession.fileSize,
})
}
if (completionSeal.state === 'completed') {
throw new MaterialUploadRequestError(
'上传会话已完成,但合并文件状态异常;服务器已停止覆盖,请联系管理员核查',
409,
'UPLOAD_COMPLETED_FILE_INVALID',
)
}
if (existingMerged === 'invalid') {
await rm(mergedPath, { force: true })
}
for (let index = 0; index < chunks.length; index += 1) {
const chunk = chunks[index]
try {
const stats = await lstat(chunk.path)
if (
stats.isSymbolicLink()
|| !stats.isFile()
|| stats.size !== chunk.expectedBytes
) {
return NextResponse.json(
{ error: `分片 ${index} 大小不匹配,请重新上传该文件`, code: 'CHUNK_SIZE_MISMATCH' },
{ status: 400 },
)
}
} catch (error: any) {
if (error?.code === 'ENOENT') {
return NextResponse.json(
{ error: '分片不完整,请继续上传后重试', code: 'CHUNK_INCOMPLETE' },
{ status: 400 },
)
}
throw error
}
}
// init 的空间判断只是上传开始时的快照。这里在唯一合并槽内重新确认
// “第二份完整文件 + 2GiB 安全余量”,否则必须 fail-closed。
releaseStorageReservation = await acquireUploadMergeDiskReservation(
tempDir,
uploadSession.fileSize,
)
await mergeUploadChunksAtomically({
chunks,
expectedTotalBytes: uploadSession.fileSize,
outputPath: mergedPath,
assertLockHeld: () => releaseCrossProcessLock?.assertHeld(),
})
if (!completionSealHandle) {
throw new Error('上传完成标记句柄缺失')
}
await completeUploadCompletionSeal(completionSealHandle)
completionSealFinalized = true
await cleanupCompletedUpload(uploadId, chunks.map((chunk) => chunk.path))
return createCompletedResponse({
uploadId,
mergedPath,
mergedFileName,
fileSize: uploadSession.fileSize,
}) })
} catch (error) { } catch (error) {
if (error instanceof MaterialUploadRequestError) {
return NextResponse.json(
{ error: error.message, code: error.code },
{
status: error.status,
headers: error.retryAfterSeconds
? { 'Retry-After': String(error.retryAfterSeconds) }
: undefined,
},
)
}
console.error('[UploadComplete] Error:', error) console.error('[UploadComplete] Error:', error)
return NextResponse.json({ error: '合并分片失败' }, { status: 500 }) return NextResponse.json({ error: '合并分片失败' }, { status: 500 })
} finally {
releaseStorageReservation?.()
if (completionSealHandle && !completionSealFinalized) {
await abortUploadCompletionSeal(completionSealHandle).catch((error) => {
console.error('[UploadComplete] 回收未完成封存标记失败:', error)
})
}
await releaseCrossProcessLock?.()
releaseMergeSlot?.()
} }
} }
async function inspectExistingMergedFile(
mergedPath: string,
expectedBytes: number,
): Promise<'missing' | 'valid' | 'invalid'> {
try {
const stats = await lstat(mergedPath)
return !stats.isSymbolicLink()
&& stats.isFile()
&& stats.size === expectedBytes
? 'valid'
: 'invalid'
} catch (error: any) {
if (error?.code === 'ENOENT') {
return 'missing'
}
throw error
}
}
async function cleanupCompletedUpload(uploadId: string, chunkFiles: string[]): Promise<void> {
for (const chunkFile of chunkFiles) {
try {
await rm(chunkFile, { force: true })
} catch (error) {
console.warn('[UploadComplete] 清理分片异常(保留已验证的合并文件):', {
chunkFile,
error,
})
}
}
try {
await deleteUploadSession(uploadId)
console.log('[UploadComplete] 已删除会话:', uploadId)
} catch (error) {
console.warn('[UploadComplete] 删除会话失败(保留已验证的合并文件):', error)
}
}
function createCompletedResponse(input: {
uploadId: string
mergedPath: string
mergedFileName: string
fileSize: number
}) {
return NextResponse.json({
message: '分片合并完成',
existingFilePath: input.mergedPath,
fileName: input.mergedFileName,
fileSize: input.fileSize,
uploadId: input.uploadId,
})
}

View File

@ -4,7 +4,7 @@ import { POST } from '@/app/api/materials/upload/init/route'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { saveUploadSession } from '@/lib/upload-sessions' import { saveUploadSession } from '@/lib/upload-sessions'
import { ensureSafeDirectoryWithinRoot } from '@/lib/path-security' import { ensureSafeDirectoryWithinRoot } from '@/lib/path-security'
import { writeFile } from 'fs/promises' import { rm, statfs, writeFile } from 'fs/promises'
jest.mock('next-auth', () => ({ jest.mock('next-auth', () => ({
getServerSession: jest.fn(), getServerSession: jest.fn(),
@ -27,6 +27,11 @@ jest.mock('@/lib/path-security', () => {
}) })
jest.mock('fs/promises', () => ({ jest.mock('fs/promises', () => ({
rm: jest.fn().mockResolvedValue(undefined),
statfs: jest.fn().mockResolvedValue({
bavail: BigInt(1024) * BigInt(1024),
bsize: BigInt(1024) * BigInt(1024),
}),
writeFile: jest.fn().mockResolvedValue(undefined), writeFile: jest.fn().mockResolvedValue(undefined),
})) }))
@ -92,11 +97,45 @@ describe('/api/materials/upload/init', () => {
expect(body).toEqual({ error: '上传分片参数无效' }) expect(body).toEqual({ error: '上传分片参数无效' })
}) })
test('超过10GB的逻辑源文件在创建会话前拒绝', async () => {
const response = await POST(createRequest({
fileName: 'huge.psd',
fileSize: 10 * 1024 * 1024 * 1024 + 1,
totalChunks: 1024,
}))
const body = await response.json()
expect(response.status).toBe(413)
expect(body.code).toBe('SOURCE_FILE_TOO_LARGE')
expect(ensureSafeDirectoryWithinRoot).not.toHaveBeenCalled()
expect(saveUploadSession).not.toHaveBeenCalled()
})
test('上传盘无法容纳分片和合并文件时清理空会话目录并拒绝', async () => {
;(statfs as jest.Mock).mockResolvedValueOnce({ bavail: BigInt(1), bsize: BigInt(1) })
const response = await POST(createRequest({
fileName: 'source.psd',
fileSize: 20 * 1024 * 1024,
totalChunks: 1,
}))
const body = await response.json()
expect(response.status).toBe(507)
expect(body.code).toBe('UPLOAD_STORAGE_LOW')
expect(rm).toHaveBeenCalledWith(
'D:\\temp\\uploads\\temp\\upload-fixed',
{ recursive: true, force: true },
)
expect(saveUploadSession).not.toHaveBeenCalled()
})
test('有效请求会保存标准化后的上传会话', async () => { test('有效请求会保存标准化后的上传会话', async () => {
const response = await POST(createRequest({ const response = await POST(createRequest({
fileName: ' source.psb ', fileName: ' source.psb ',
fileSize: '2048', fileSize: '2048',
totalChunks: '2', totalChunks: '2',
chunkSize: '1024',
})) }))
const body = await response.json() const body = await response.json()
@ -114,11 +153,26 @@ describe('/api/materials/upload/init', () => {
fileName: 'source.psb', fileName: 'source.psb',
fileSize: 2048, fileSize: 2048,
totalChunks: 2, totalChunks: 2,
chunkSize: 1024,
userId: 'user-1', userId: 'user-1',
}), }),
) )
}) })
test('拒绝总大小与分片数量不可能对应的伪造会话', async () => {
const response = await POST(createRequest({
fileName: 'forged.psd',
fileSize: 1,
totalChunks: 1024,
}))
const body = await response.json()
expect(response.status).toBe(400)
expect(body.code).toBe('CHUNK_LAYOUT_INVALID')
expect(ensureSafeDirectoryWithinRoot).not.toHaveBeenCalled()
expect(saveUploadSession).not.toHaveBeenCalled()
})
test.each([ test.each([
'../outside.psd', '../outside.psd',
'..\\outside.psd', '..\\outside.psd',

View File

@ -7,8 +7,17 @@ import {
normalizeSafeFileName, normalizeSafeFileName,
} from '@/lib/path-security' } from '@/lib/path-security'
import { randomBytes } from 'crypto' import { randomBytes } from 'crypto'
import { writeFile } from 'fs/promises' import { rm, statfs, writeFile } from 'fs/promises'
import path from 'path' import path from 'path'
import {
MAX_LOGICAL_UPLOAD_BYTES,
MAX_UPLOAD_CHUNKS,
resolveUploadChunkSize,
} from '@/lib/material-upload-chunk-policy'
const MEBIBYTE = 1024 * 1024
const GIBIBYTE = 1024 * MEBIBYTE
const MIN_FREE_SPACE_RESERVE_BYTES = 2 * GIBIBYTE
export async function POST(req: NextRequest) { export async function POST(req: NextRequest) {
try { try {
@ -40,14 +49,36 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ error: '文件名包含不安全的路径或字符' }, { status: 400 }) return NextResponse.json({ error: '文件名包含不安全的路径或字符' }, { status: 400 })
} }
if (!Number.isFinite(normalizedFileSize) || normalizedFileSize <= 0) { if (!Number.isSafeInteger(normalizedFileSize) || normalizedFileSize <= 0) {
return NextResponse.json({ error: '上传文件不能为空' }, { status: 400 }) return NextResponse.json({ error: '上传文件不能为空' }, { status: 400 })
} }
if (normalizedFileSize > MAX_LOGICAL_UPLOAD_BYTES) {
return NextResponse.json(
{ error: '单个源文件不能超过 10GB', code: 'SOURCE_FILE_TOO_LARGE' },
{ status: 413 },
)
}
if (!Number.isInteger(normalizedTotalChunks) || normalizedTotalChunks <= 0) { if (
!Number.isInteger(normalizedTotalChunks)
|| normalizedTotalChunks <= 0
|| normalizedTotalChunks > MAX_UPLOAD_CHUNKS
) {
return NextResponse.json({ error: '上传分片参数无效' }, { status: 400 }) return NextResponse.json({ error: '上传分片参数无效' }, { status: 400 })
} }
const normalizedChunkSize = resolveUploadChunkSize(
normalizedFileSize,
normalizedTotalChunks,
body?.chunkSize,
)
if (!normalizedChunkSize) {
return NextResponse.json(
{ error: '上传分片大小与文件总大小不匹配', code: 'CHUNK_LAYOUT_INVALID' },
{ status: 400 },
)
}
// 生成上传ID // 生成上传ID
const uploadId = `upload-${Date.now()}-${randomBytes(16).toString('hex')}` const uploadId = `upload-${Date.now()}-${randomBytes(16).toString('hex')}`
@ -57,12 +88,35 @@ export async function POST(req: NextRequest) {
if (!tempDir) { if (!tempDir) {
throw new Error('上传临时目录不安全') throw new Error('上传临时目录不安全')
} }
// 分片合并期间会同时保留分片与完整输出,必须为两份源文件再加固定
// 安全余量预留空间。无法确认剩余空间时保守拒绝,避免写满 uploads/NAS。
try {
const disk = await statfs(tempDir, { bigint: true })
const availableBytes = disk.bavail * disk.bsize
const requiredBytes = BigInt(normalizedFileSize) * BigInt(2) + BigInt(MIN_FREE_SPACE_RESERVE_BYTES)
if (availableBytes < requiredBytes) {
await rm(tempDir, { recursive: true, force: true })
return NextResponse.json(
{ error: '上传存储空间不足,请联系管理员清理后重试', code: 'UPLOAD_STORAGE_LOW' },
{ status: 507 },
)
}
} catch (diskError) {
await rm(tempDir, { recursive: true, force: true }).catch(() => undefined)
console.error('[UploadInit] 无法确认上传盘剩余空间:', diskError)
return NextResponse.json(
{ error: '暂时无法确认上传存储空间,请稍后重试', code: 'UPLOAD_STORAGE_UNKNOWN' },
{ status: 503 },
)
}
// 写入元数据,方便在Redis丢失时回溯 // 写入元数据,方便在Redis丢失时回溯
const metaPath = path.join(tempDir, 'meta.json') const metaPath = path.join(tempDir, 'meta.json')
const metaPayload = { const metaPayload = {
fileName: normalizedFileName, fileName: normalizedFileName,
fileSize: normalizedFileSize, fileSize: normalizedFileSize,
totalChunks: normalizedTotalChunks, totalChunks: normalizedTotalChunks,
chunkSize: normalizedChunkSize,
userId, userId,
createdAt: Date.now(), createdAt: Date.now(),
} }
@ -74,6 +128,7 @@ export async function POST(req: NextRequest) {
fileName: normalizedFileName, fileName: normalizedFileName,
fileSize: normalizedFileSize, fileSize: normalizedFileSize,
totalChunks: normalizedTotalChunks, totalChunks: normalizedTotalChunks,
chunkSize: normalizedChunkSize,
chunks: new Map(), chunks: new Map(),
createdAt: now, createdAt: now,
lastActivityAt: now, lastActivityAt: now,

View File

@ -3,18 +3,12 @@ import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
import { import {
deleteOwnedUploadSession,
getOwnedUploadSession, getOwnedUploadSession,
isSafeUploadId, isSafeUploadId,
} from '@/lib/upload-sessions' } from '@/lib/upload-sessions'
import { getUploadRootDir } from '@/lib/upload-paths'
import {
isPathFreeOfSymbolicLinks,
resolveStrictChildPath,
} from '@/lib/path-security'
import { acquireTaskDetailUploadLock } from '@/lib/task-detail-upload-jobs' import { acquireTaskDetailUploadLock } from '@/lib/task-detail-upload-jobs'
import { TASK_DETAIL_UPLOAD_LIMITS } from '@/lib/task-detail-upload-policy' import { TASK_DETAIL_UPLOAD_LIMITS } from '@/lib/task-detail-upload-policy'
import { rm, unlink } from 'fs/promises' import { safelyCleanupMaterialUploadSession } from '@/lib/temp-cleanup'
interface AbortRequestBody { interface AbortRequestBody {
materialId?: string materialId?: string
@ -23,8 +17,6 @@ interface AbortRequestBody {
interface ValidatedAbortUpload { interface ValidatedAbortUpload {
uploadId: string uploadId: string
assembledPath: string
tempDir: string
} }
const ABORT_VALIDATION_CONCURRENCY = 12 const ABORT_VALIDATION_CONCURRENCY = 12
@ -54,47 +46,12 @@ async function validateAbortUploads(
uploadIds: string[], uploadIds: string[],
userId: string, userId: string,
): Promise<ValidatedAbortUpload[]> { ): Promise<ValidatedAbortUpload[]> {
const uploadRoot = getUploadRootDir()
const assembledRoot = resolveStrictChildPath(uploadRoot, 'assembled')
const tempRoot = resolveStrictChildPath(uploadRoot, 'temp')
if (!assembledRoot || !tempRoot) {
throw new Error('上传目录配置不安全')
}
return mapWithConcurrency(uploadIds, ABORT_VALIDATION_CONCURRENCY, async (uploadId) => { return mapWithConcurrency(uploadIds, ABORT_VALIDATION_CONCURRENCY, async (uploadId) => {
const uploadSession = await getOwnedUploadSession(uploadId, userId) const uploadSession = await getOwnedUploadSession(uploadId, userId)
if (!uploadSession) { if (!uploadSession) {
throw new Error(`上传会话不存在或不属于当前用户:${uploadId}`) throw new Error(`上传会话不存在或不属于当前用户:${uploadId}`)
} }
return { uploadId }
const assembledPath = resolveStrictChildPath(
assembledRoot,
`${uploadId}-${uploadSession.fileName}`,
)
const tempDir = resolveStrictChildPath(tempRoot, uploadId)
const [assembledPathIsSafe, tempDirIsSafe] = assembledPath && tempDir
? await Promise.all([
isPathFreeOfSymbolicLinks(uploadRoot, assembledPath, {
allowMissingDescendants: true,
}),
isPathFreeOfSymbolicLinks(uploadRoot, tempDir, {
allowMissingDescendants: true,
}),
])
: [false, false]
if (
!assembledPath ||
!tempDir ||
!assembledPathIsSafe ||
!tempDirIsSafe
) {
throw new Error(`上传会话路径不安全:${uploadId}`)
}
return {
uploadId,
assembledPath,
tempDir,
}
}) })
} }
@ -200,17 +157,34 @@ export async function POST(req: NextRequest) {
return NextResponse.json({ success: true, deleted: false, reason: deletion.reason }) return NextResponse.json({ success: true, deleted: false, reason: deletion.reason })
} }
const cleanupResults = []
for (const upload of validatedUploads) { for (const upload of validatedUploads) {
await unlink(upload.assembledPath).catch((error: any) => { const cleanup = await safelyCleanupMaterialUploadSession({
if (error?.code !== 'ENOENT') { uploadId: upload.uploadId,
throw error intent: 'cancel',
} expectedOwnerId: userId,
}) })
await rm(upload.tempDir, { recursive: true, force: true }) cleanupResults.push({ uploadId: upload.uploadId, cleanup })
} }
const failedCleanups = cleanupResults.filter(({ cleanup }) => !cleanup.success)
for (const upload of validatedUploads) { if (failedCleanups.length > 0) {
await deleteOwnedUploadSession(upload.uploadId, userId) console.error('[product-package/abort] 素材占位已删除,但部分上传会话安全清理未完成:', {
materialId,
failures: failedCleanups.map(({ uploadId, cleanup }) => ({
uploadId,
status: cleanup.status,
code: cleanup.code,
})),
})
return NextResponse.json({
success: false,
deleted: true,
materialId,
cleanedUploadIds: cleanupResults.length - failedCleanups.length,
pendingUploadIds: failedCleanups.map(({ uploadId }) => uploadId),
code: 'UPLOAD_CANCEL_CLEANUP_INCOMPLETE',
error: '未完成素材已删除,但部分上传会话暂未安全清理,请稍后重试',
}, { status: 503 })
} }
return NextResponse.json({ return NextResponse.json({
success: true, success: true,

View File

@ -7,7 +7,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { validateTaskNo, isTaskNoExists, purgeDeletedMaterialsByTaskNo } from '@/lib/product-package'; import { validateTaskNo, isTaskNoExists } from '@/lib/product-package';
import { normalizeTaskDetailSourceType } from '@/lib/task-detail-source-type'; import { normalizeTaskDetailSourceType } from '@/lib/task-detail-source-type';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
@ -112,17 +112,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(
`[Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条,删除文件 ${cleanupResult.deletedFiles} 个,失败 ${cleanupResult.failedFiles} 个`
);
}
} catch (cleanupError) {
console.error(`[Init] 清理旧素材失败:`, cleanupError);
// 保持旧行为:清理失败时仅记录,不阻断新建流程
}
} }
// 创建素材记录(不写入taskNo,上传完成后再写入) // 创建素材记录(不写入taskNo,上传完成后再写入)

View File

@ -1,29 +1,14 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { completeTaskImageMaterial } from '@/lib/task-image-material-completion';
import { assignTaskNoOnComplete } from '@/lib/product-package';
import {
createZipFromFiles,
generatePreviewImages,
cleanupTempFiles,
cleanupOnError,
saveAssets,
} from '@/lib/task-material-processor';
import { createMaterialLog } from '@/lib/material-log';
export const runtime = 'nodejs' export const runtime = 'nodejs';
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic';
export const maxDuration = 10800 export const maxDuration = 10800;
// POST - 完成实拍原图上传
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let materialId: string | null = null;
const tempFiles: string[] = []; // 需要清理的临时文件路径
let zipFilePath: string | null = null;
const uploadedPreviewUrls: string[] = []; // 新生成的S3预览图URLs
const uploadIds: string[] = []; // 需要清理的uploadId列表
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -31,138 +16,26 @@ export async function POST(request: NextRequest) {
} }
const body = await request.json(); const body = await request.json();
const { materialId: matId, assets, taskNo, dingtalkApprovalNo } = body; const result = await completeTaskImageMaterial({
materialId: body?.materialId,
materialId = matId;
if (!materialId || !assets) {
return NextResponse.json(
{ error: '缺少必填字段' },
{ status: 400 }
);
}
if (!assets.main || assets.main.length === 0) {
return NextResponse.json(
{ error: '至少需要上传一张原图' },
{ status: 400 }
);
}
console.log('[Real Photo Complete] Starting, materialId:', materialId);
console.log('[Real Photo Complete] Original images count:', assets.main.length);
// 验证素材是否存在且属于当前用户
const material: any = await prisma.material.findUnique({
where: { id: materialId },
});
if (!material) {
return NextResponse.json(
{ success: false, error: '素材不存在' },
{ status: 404 }
);
}
if (material.userId !== (session.user as any).id) {
return NextResponse.json(
{ success: false, error: '无权限操作此素材' },
{ status: 403 }
);
}
// 步骤1: 从chunk临时目录读取原图文件
console.log(`[Real Photo] Processing ${assets.main.length} original images...`);
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions');
const originalFiles: Array<{ path: string; name: string; size: number }> = [];
for (const asset of assets.main) {
try {
const filePath = await getOwnedUploadedFile(asset.uploadId, (session.user as any).id);
if (filePath) {
originalFiles.push({
path: filePath,
name: asset.fileName,
size: asset.fileSize,
});
tempFiles.push(filePath);
uploadIds.push(asset.uploadId);
} else {
throw new Error(`找不到uploadId对应的文件: ${asset.uploadId}`);
}
} catch (error) {
console.error(`[Real Photo] Failed to get file for uploadId ${asset.uploadId}:`, error);
throw new Error(`获取原图文件失败: ${asset.fileName}`);
}
}
// 步骤2: 打包原图成zip(保存到本地文件系统)
console.log(`[Real Photo] Creating zip from ${originalFiles.length} original files...`);
const zipInfo = await createZipFromFiles(
originalFiles,
materialId!,
material.productModel || 'real-photo',
'任务-实拍原图'
);
console.log(`[Real Photo] Zip created: ${zipInfo.fileName} (${zipInfo.fileSize} bytes)`);
zipFilePath = zipInfo.filePath;
// 步骤3: 生成预览图并上传到S3
const previewUrls = await generatePreviewImages(
assets.main.map((a: any) => ({ uploadId: a.uploadId, fileName: a.fileName })),
'任务-实拍原图'
);
uploadedPreviewUrls.push(...previewUrls);
if (uploadedPreviewUrls.length === 0) {
throw new Error('生成预览图失败,至少需要一张预览图');
}
// 步骤4: 保存资产记录到数据库
await saveAssets(materialId!, uploadedPreviewUrls, zipInfo);
// 步骤4.5: 仅在素材成功完成后写入任务号
await assignTaskNoOnComplete(materialId!, taskNo, dingtalkApprovalNo);
// 步骤5: 清理临时文件
await cleanupTempFiles(tempFiles, uploadIds);
// 步骤6: 记录上传日志
await createMaterialLog({
materialId: materialId!,
userId: (session.user as any).id, userId: (session.user as any).id,
action: 'UPLOAD', materialType: '任务-实拍原图',
changes: { fallbackProductModel: 'real-photo',
uploadType: { assets: Array.isArray(body?.assets?.main) ? body.assets.main : [],
oldValue: null, taskNo: body?.taskNo,
newValue: '任务-实拍原图', dingtalkApprovalNo: body?.dingtalkApprovalNo,
},
previewCount: {
oldValue: 0,
newValue: uploadedPreviewUrls.length,
},
zipFileName: {
oldValue: null,
newValue: zipInfo.fileName,
},
},
});
console.log('[Real Photo] Complete success!');
return NextResponse.json({
success: true,
materialId,
}); });
return NextResponse.json({ success: true, ...result });
} catch (error: any) { } catch (error: any) {
console.error('[Real Photo Complete Error]', error); console.error('[Real Photo Complete Error]', error);
const status = Number.isInteger(error?.status) ? error.status : 500;
// 清理失败时的资源 const response = NextResponse.json(
await cleanupOnError(uploadedPreviewUrls, zipFilePath, tempFiles, materialId); { error: error?.message || '完成上传失败' },
{ status },
return NextResponse.json(
{ error: error.message || '完成上传失败' },
{ status: 500 }
); );
if (Number.isFinite(error?.retryAfterSeconds)) {
response.headers.set('Retry-After', String(error.retryAfterSeconds));
}
return response;
} }
} }

View File

@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { purgeDeletedMaterialsByTaskNo, validateTaskNo } from '@/lib/product-package'; import { validateTaskNo } from '@/lib/product-package';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
// POST - 初始化实拍原图素材记录 // POST - 初始化实拍原图素材记录
@ -49,14 +49,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(`[Real Photo Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条`);
}
} catch (cleanupError) {
console.error('[Real Photo Init] 清理旧软删素材失败:', cleanupError);
}
} }
const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo); const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo);
@ -91,6 +83,7 @@ export async function POST(request: NextRequest) {
previewUrl: '', // 临时空值,完成上传时更新 previewUrl: '', // 临时空值,完成上传时更新
zipFilePath: '', // 临时空值,完成上传时更新 zipFilePath: '', // 临时空值,完成上传时更新
zipFileName: '', // 临时空值,完成上传时更新 zipFileName: '', // 临时空值,完成上传时更新
taskStatus: 'uploading',
renderer: '', // 实拍原图不需要渲染器 renderer: '', // 实拍原图不需要渲染器
}, },
}); });

View File

@ -0,0 +1,551 @@
/** @jest-environment node */
import { NextRequest } from 'next/server'
import { getServerSession } from 'next-auth'
import { promises as fs } from 'fs'
import { GET, POST } from './route'
import { deleteFiles } from '@/lib/file-cleanup'
import { processPdfUpload } from '@/lib/pdf-upload-handler'
import { prisma } from '@/lib/prisma'
import {
deleteOwnedUploadSession,
getOwnedUploadSession,
resolveOwnedMergedUpload,
} from '@/lib/upload-sessions'
import {
recoverMaterialUploadConsumptionJournal,
removeMaterialUploadConsumptionJournal,
syncMaterialUploadRootDirectory,
writeMaterialUploadConsumptionJournal,
} from '@/lib/material-upload-consumption-journal'
import { acquireTaskMaterialUploadSessionLocks } from '@/lib/material-upload-cross-process-lock'
import {
CanonicalMaterialSourceValidationError,
validateCanonicalMaterialSource,
} from '@/lib/material-canonical-source'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('fs', () => ({
promises: {
mkdir: jest.fn().mockResolvedValue(undefined),
rename: jest.fn().mockResolvedValue(undefined),
stat: jest.fn().mockResolvedValue({ size: 5 }),
},
}))
const mockFsPromises = fs as any
jest.mock('@/lib/prisma', () => ({
prisma: {
user: { findUnique: jest.fn() },
material: {
findFirst: jest.fn(),
findUnique: jest.fn(),
create: jest.fn(),
update: jest.fn(),
delete: jest.fn(),
},
materialAsset: { createMany: jest.fn() },
},
}))
jest.mock('@/lib/upload-sessions', () => ({
deleteOwnedUploadSession: jest.fn(),
getOwnedUploadSession: jest.fn(),
isSafeUploadId: jest.fn((value: string) => /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/.test(value)),
resolveOwnedMergedUpload: jest.fn(),
}))
jest.mock('@/lib/material-upload-consumption-journal', () => ({
recoverMaterialUploadConsumptionJournal: jest.fn(),
removeMaterialUploadConsumptionJournal: jest.fn(),
syncMaterialUploadRootDirectory: jest.fn(),
writeMaterialUploadConsumptionJournal: jest.fn(),
}))
jest.mock('@/lib/material-upload-cross-process-lock', () => ({
acquireTaskMaterialUploadSessionLocks: jest.fn(),
}))
jest.mock('@/lib/material-canonical-source', () => {
const actual = jest.requireActual('@/lib/material-canonical-source')
return {
...actual,
validateCanonicalMaterialSource: jest.fn(),
}
})
jest.mock('@/lib/temp-cleanup', () => ({
cleanupUploadSession: jest.fn().mockResolvedValue(true),
}))
jest.mock('@/lib/s3', () => ({
uploadToS3: jest.fn(),
}))
jest.mock('@/lib/audio-processor', () => ({
parseAudioMetadata: jest.fn(),
isAudioFile: jest.fn(() => false),
}))
jest.mock('@/lib/video-transcode-task', () => ({
enqueueVideoTranscodeTask: jest.fn().mockResolvedValue(undefined),
}))
jest.mock('@/lib/image-processor', () => ({
compressImage: jest.fn(),
sanitizeFileName: jest.fn((value: string) => value),
}))
jest.mock('@/lib/pdf-upload-handler', () => ({
processPdfUpload: jest.fn(),
}))
jest.mock('@/lib/file-cleanup', () => ({
deleteFiles: jest.fn().mockResolvedValue({ deletedCount: 0, failedCount: 0 }),
}))
jest.mock('@/lib/material-cleanup', () => ({
deleteMaterialRelatedRecords: jest.fn(),
}))
jest.mock('@/lib/material-log', () => ({
createMaterialLog: jest.fn().mockResolvedValue(undefined),
}))
jest.mock('@/lib/public-display', () => ({
buildMaterialPublicDisplayPath: jest.fn((fileName: string) => `materials/${fileName}`),
uploadPublicDisplayToS3: jest.fn(),
}))
type MaterialFixture = ReturnType<typeof createMaterial>
function createMaterial(overrides: Record<string, unknown> = {}) {
return {
id: 'material-created',
uploadId: 'upload-owned',
previewUrl: 'data:image/svg+xml;base64,preview',
productModel: 'SPU-001',
categoryName: '通用',
sceneName: '通用',
platformName: '',
productCategoryName: '',
materialType: '音乐',
renderer: '',
authorName: '安乔',
zipFileName: 'audio-source.mp3',
zipFilePath: 'uploads/audio-source.mp3',
zipFileSize: BigInt(5),
displayImages: [],
userId: 'user-1',
user: { username: '安乔' },
...overrides,
}
}
function buildFormData(input: {
uploadId?: string
materialType?: string
existingFilePath?: string
} = {}) {
const uploadId = input.uploadId || 'upload-owned'
const formData = new FormData()
formData.append('uploadId', uploadId)
formData.append(
'existingFilePath',
input.existingFilePath || `uploads/temp/${uploadId}/merged-source.bin`,
)
formData.append('originalFileName', input.materialType === '品牌VI灵感' ? 'brand.pdf' : 'song.mp3')
formData.append('productModel', 'SPU-001')
formData.append('categoryName', '通用')
formData.append('sceneName', '通用')
formData.append('materialType', input.materialType || '音乐')
return formData
}
function createPostRequest(formData: FormData) {
return {
headers: new Headers({
'content-length': '1024',
'content-type': 'multipart/form-data; boundary=jest-boundary',
}),
formData: jest.fn().mockResolvedValue(formData),
} as any
}
describe('materials upload second-stage idempotency', () => {
beforeEach(() => {
jest.clearAllMocks()
;(getServerSession as jest.Mock).mockResolvedValue({ user: { id: 'user-1' } })
;(prisma.user.findUnique as jest.Mock).mockResolvedValue({ username: '安乔' })
;(prisma.material.findFirst as jest.Mock).mockResolvedValue(null)
;(prisma.material.create as jest.Mock).mockResolvedValue(createMaterial())
;(prisma.material.update as jest.Mock).mockResolvedValue(createMaterial())
;(validateCanonicalMaterialSource as jest.Mock).mockResolvedValue({
sourcePath: require('path').resolve('uploads/audio-source.mp3'),
size: BigInt(5),
})
;(resolveOwnedMergedUpload as jest.Mock).mockResolvedValue({
uploadId: 'upload-owned',
filePath: 'uploads/temp/upload-owned/merged-source.bin',
session: {
fileName: 'source.bin',
fileSize: 5,
totalChunks: 1,
chunks: new Map(),
createdAt: Date.now(),
userId: 'user-1',
},
})
;(deleteOwnedUploadSession as jest.Mock).mockResolvedValue(true)
;(getOwnedUploadSession as jest.Mock).mockResolvedValue(null)
;(processPdfUpload as jest.Mock).mockResolvedValue({ success: false, error: 'not a pdf' })
;(recoverMaterialUploadConsumptionJournal as jest.Mock).mockResolvedValue(null)
;(removeMaterialUploadConsumptionJournal as jest.Mock).mockResolvedValue(true)
;(syncMaterialUploadRootDirectory as jest.Mock).mockResolvedValue(undefined)
;(writeMaterialUploadConsumptionJournal as jest.Mock).mockImplementation(async (input) => ({
version: 1,
...input,
mergedPath: require('path').resolve(input.mergedPath),
finalPath: require('path').resolve(input.finalPath),
createdAt: Date.now(),
}))
;(acquireTaskMaterialUploadSessionLocks as jest.Mock).mockResolvedValue(
Object.assign(jest.fn().mockResolvedValue(undefined), { assertHeld: jest.fn() }),
)
mockFsPromises.mkdir.mockResolvedValue(undefined)
mockFsPromises.rename.mockResolvedValue(undefined)
mockFsPromises.stat.mockResolvedValue({ size: 5 })
})
test('Material.create 只写服务器从 merged 路径派生的 uploadId', async () => {
;(resolveOwnedMergedUpload as jest.Mock).mockResolvedValueOnce({
uploadId: 'server-derived-upload',
filePath: 'uploads/temp/server-derived-upload/merged-song.mp3',
session: {
fileName: 'song.mp3',
fileSize: 5,
totalChunks: 1,
chunks: new Map(),
createdAt: Date.now(),
userId: 'user-1',
},
})
const response = await POST(createPostRequest(buildFormData({ uploadId: 'client-claimed-upload' })))
expect(response.status).toBe(201)
expect(prisma.material.create).toHaveBeenCalledWith(
expect.objectContaining({
data: expect.objectContaining({ uploadId: 'server-derived-upload' }),
}),
)
expect(validateCanonicalMaterialSource).toHaveBeenCalledTimes(2)
expect(validateCanonicalMaterialSource).toHaveBeenNthCalledWith(1, {
storedPath: expect.stringContaining('audio-source-'),
expectedSize: BigInt(5),
})
})
test('canonical 源文件 stat 失败时提交前返回 503 且不建档', async () => {
;(validateCanonicalMaterialSource as jest.Mock).mockRejectedValueOnce(
new CanonicalMaterialSourceValidationError(
'CANONICAL_SOURCE_STAT_FAILED',
'source disappeared',
),
)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.code).toBe('CANONICAL_SOURCE_STAT_FAILED')
expect(prisma.material.create).not.toHaveBeenCalled()
// 第一次 rename 移入正式目录,第二次原子恢复到会话目录;不删除唯一源。
expect(mockFsPromises.rename).toHaveBeenCalledTimes(2)
expect(deleteFiles).not.toHaveBeenCalled()
})
test('分片实际大小与会话 zipFileSize 不一致时提交前返回 503', async () => {
;(validateCanonicalMaterialSource as jest.Mock).mockRejectedValueOnce(
new CanonicalMaterialSourceValidationError(
'CANONICAL_SOURCE_SIZE_MISMATCH',
'actual size differs from known zipFileSize',
),
)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.code).toBe('CANONICAL_SOURCE_SIZE_MISMATCH')
expect(validateCanonicalMaterialSource).toHaveBeenCalledTimes(1)
expect(prisma.material.create).not.toHaveBeenCalled()
expect(mockFsPromises.rename).toHaveBeenCalledTimes(2)
expect(deleteFiles).not.toHaveBeenCalled()
})
test('同用户重放在临时目录已删后直接返回原 material', async () => {
const winner = createMaterial({ id: 'material-from-first-response' })
;(prisma.material.findFirst as jest.Mock).mockResolvedValueOnce(winner)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(201)
expect(body.replayed).toBe(true)
expect(body.material.id).toBe('material-from-first-response')
expect(resolveOwnedMergedUpload).not.toHaveBeenCalled()
expect(mockFsPromises.rename).not.toHaveBeenCalled()
expect(deleteFiles).not.toHaveBeenCalled()
expect(prisma.material.findFirst).toHaveBeenCalledWith(expect.objectContaining({
where: { uploadId: 'upload-owned', userId: 'user-1', deletedAt: null },
}))
})
test('软删的同 uploadId 不是幂等赢家,允许创建新的 active material', async () => {
// helper 的 deletedAt:null 由数据库过滤;软删记录不会作为 replay 返回。
;(prisma.material.findFirst as jest.Mock).mockResolvedValueOnce(null)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(201)
expect(body.replayed).toBeUndefined()
expect(resolveOwnedMergedUpload).toHaveBeenCalledTimes(1)
expect(prisma.material.create).toHaveBeenCalledTimes(1)
expect(prisma.material.findFirst).toHaveBeenCalledWith(expect.objectContaining({
where: { uploadId: 'upload-owned', userId: 'user-1', deletedAt: null },
}))
})
test('幂等既有记录的 canonical 源损坏时不能返回 replayed 成功', async () => {
const winner = createMaterial({ id: 'material-broken-replay' })
;(prisma.material.findFirst as jest.Mock).mockResolvedValueOnce(winner)
;(validateCanonicalMaterialSource as jest.Mock).mockRejectedValueOnce(
new CanonicalMaterialSourceValidationError(
'CANONICAL_SOURCE_SIZE_MISMATCH',
'stored file truncated',
),
)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.code).toBe('CANONICAL_SOURCE_INTEGRITY_FAILED')
expect(body.replayed).toBeUndefined()
expect(prisma.material.update).toHaveBeenCalledWith(expect.objectContaining({
where: { id: 'material-broken-replay' },
data: expect.objectContaining({ deletedAt: expect.any(Date) }),
}))
expect(removeMaterialUploadConsumptionJournal).not.toHaveBeenCalled()
expect(resolveOwnedMergedUpload).not.toHaveBeenCalled()
expect(mockFsPromises.rename).not.toHaveBeenCalled()
expect(deleteFiles).not.toHaveBeenCalled()
})
test('提交后复核失败时软隔离记录、保留 canonical 源与 journal 并返回 503', async () => {
;(validateCanonicalMaterialSource as jest.Mock)
.mockResolvedValueOnce({ sourcePath: 'uploads/audio-source.mp3', size: BigInt(5) })
.mockRejectedValueOnce(new CanonicalMaterialSourceValidationError(
'CANONICAL_SOURCE_STAT_FAILED',
'post-commit stat failed',
))
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.code).toBe('CANONICAL_SOURCE_INTEGRITY_FAILED')
expect(prisma.material.create).toHaveBeenCalledTimes(1)
expect(prisma.material.update).toHaveBeenCalledWith(expect.objectContaining({
where: { id: 'material-created' },
data: expect.objectContaining({ deletedAt: expect.any(Date) }),
}))
expect(removeMaterialUploadConsumptionJournal).not.toHaveBeenCalled()
expect(deleteOwnedUploadSession).not.toHaveBeenCalled()
expect(deleteFiles).not.toHaveBeenCalled()
// 数据库提交后的复核失败不能把正式源移回或删除。
expect(mockFsPromises.rename).toHaveBeenCalledTimes(1)
})
test('其他用户不能查询或复用 uploadId', async () => {
;(resolveOwnedMergedUpload as jest.Mock).mockResolvedValueOnce(null)
const postResponse = await POST(createPostRequest(buildFormData()))
const postBody = await postResponse.json()
expect(postResponse.status).toBe(403)
expect(postBody.error).toBe('上传会话不存在、已失效或不属于当前用户')
expect(prisma.material.create).not.toHaveBeenCalled()
const getResponse = await GET(new NextRequest(
'http://localhost/api/materials/upload?uploadId=upload-owned',
))
const getBody = await getResponse.json()
expect(getResponse.status).toBe(404)
expect(getBody.status).toBe('not_found')
})
test('P2002 竞争返回同用户赢家且不删除赢家文件', async () => {
const winner = createMaterial({ id: 'material-winner' })
;(prisma.material.findFirst as jest.Mock)
.mockResolvedValueOnce(null)
.mockResolvedValueOnce(winner)
;(prisma.material.create as jest.Mock).mockRejectedValueOnce(
Object.assign(new Error('unique constraint'), { code: 'P2002' }),
)
const response = await POST(createPostRequest(buildFormData()))
const body = await response.json()
expect(response.status).toBe(201)
expect(body.replayed).toBe(true)
expect(body.material.id).toBe('material-winner')
expect(deleteFiles).not.toHaveBeenCalled()
expect(mockFsPromises.rename).toHaveBeenCalledTimes(1)
})
test('P2002 赢家暂不可见时保守留文件,不恢复或删除未知赢家', async () => {
;(prisma.material.findFirst as jest.Mock)
.mockResolvedValueOnce(null)
.mockResolvedValueOnce(null)
;(prisma.material.create as jest.Mock).mockRejectedValueOnce(
Object.assign(new Error('unique constraint'), { code: 'P2002' }),
)
const response = await POST(createPostRequest(buildFormData()))
expect(response.status).toBe(500)
expect(mockFsPromises.rename).toHaveBeenCalledTimes(1)
expect(deleteFiles).not.toHaveBeenCalled()
})
test('建档失败时原子移回 merged;PDF 预览 URL 纳入清理', async () => {
;(prisma.material.findFirst as jest.Mock)
.mockResolvedValueOnce(null)
.mockResolvedValueOnce(null)
;(prisma.material.create as jest.Mock).mockRejectedValueOnce(new Error('database unavailable'))
;(processPdfUpload as jest.Mock).mockResolvedValueOnce({
success: true,
previewUrl: 'https://s3.example/pdf-preview.jpg',
})
const response = await POST(createPostRequest(buildFormData({ materialType: '品牌VI灵感' })))
expect(response.status).toBe(500)
expect(mockFsPromises.rename).toHaveBeenCalledTimes(2)
expect(mockFsPromises.rename.mock.calls[1]).toEqual([
expect.stringContaining('material-'),
'uploads/temp/upload-owned/merged-source.bin',
])
expect(deleteFiles).toHaveBeenCalledWith(['https://s3.example/pdf-preview.jpg'])
})
test('owner-scoped GET 返回已确认 material', async () => {
const winner: MaterialFixture = createMaterial({ id: 'material-status-complete' })
;(prisma.material.findFirst as jest.Mock).mockResolvedValueOnce(winner)
const response = await GET(new NextRequest(
'http://localhost/api/materials/upload?uploadId=upload-owned',
))
const body = await response.json()
expect(response.status).toBe(200)
expect(body.status).toBe('completed')
expect(body.material.id).toBe('material-status-complete')
expect(body.material.zipFileSize).toBe('5')
expect(getOwnedUploadSession).not.toHaveBeenCalled()
})
test('owner-scoped GET 也不能把损坏 canonical 源报告为 completed', async () => {
const broken = createMaterial({ id: 'material-status-broken' })
;(prisma.material.findFirst as jest.Mock).mockResolvedValueOnce(broken)
;(validateCanonicalMaterialSource as jest.Mock).mockRejectedValueOnce(
new CanonicalMaterialSourceValidationError(
'CANONICAL_SOURCE_STAT_FAILED',
'source missing during status lookup',
),
)
const response = await GET(new NextRequest(
'http://localhost/api/materials/upload?uploadId=upload-owned',
))
const body = await response.json()
expect(response.status).toBe(503)
expect(body.status).toBeUndefined()
expect(body.code).toBe('CANONICAL_SOURCE_INTEGRITY_FAILED')
expect(prisma.material.update).toHaveBeenCalledWith(expect.objectContaining({
where: { id: 'material-status-broken' },
data: expect.objectContaining({ deletedAt: expect.any(Date) }),
}))
expect(getOwnedUploadSession).not.toHaveBeenCalled()
expect(deleteFiles).not.toHaveBeenCalled()
})
test('rename 后进程中断的重试通过 journal 复用同一正式源且不再次移动', async () => {
const path = require('path')
const mergedPath = path.resolve('uploads/temp/upload-owned/merged-source.bin')
const finalPath = path.resolve('uploads/audio-source-crash-recovery.mp3')
;(recoverMaterialUploadConsumptionJournal as jest.Mock).mockResolvedValueOnce({
sourceLocation: 'final',
sourcePath: finalPath,
record: {
version: 1,
uploadId: 'upload-owned',
ownerId: 'user-1',
mergedPath,
finalPath,
storedFinalPath: finalPath,
fileName: 'audio-source-crash-recovery.mp3',
size: 5,
createdAt: Date.now(),
},
})
const response = await POST(createPostRequest(buildFormData({ existingFilePath: mergedPath })))
expect(response.status).toBe(201)
expect(resolveOwnedMergedUpload).not.toHaveBeenCalled()
expect(writeMaterialUploadConsumptionJournal).not.toHaveBeenCalled()
expect(mockFsPromises.rename).not.toHaveBeenCalled()
expect(prisma.material.create).toHaveBeenCalledWith(expect.objectContaining({
data: expect.objectContaining({
uploadId: 'upload-owned',
zipFileName: 'audio-source-crash-recovery.mp3',
zipFilePath: finalPath,
}),
}))
expect(removeMaterialUploadConsumptionJournal).toHaveBeenCalledWith({
ownerId: 'user-1',
uploadId: 'upload-owned',
expectedFinalPath: finalPath,
})
})
test('同一 uploadId 的跨工作流锁覆盖 resolve、建档和会话清理', async () => {
const release = Object.assign(jest.fn().mockResolvedValue(undefined), { assertHeld: jest.fn() })
;(acquireTaskMaterialUploadSessionLocks as jest.Mock).mockResolvedValueOnce(release)
const response = await POST(createPostRequest(buildFormData()))
expect(response.status).toBe(201)
expect(acquireTaskMaterialUploadSessionLocks).toHaveBeenCalledWith(['upload-owned'])
expect((acquireTaskMaterialUploadSessionLocks as jest.Mock).mock.invocationCallOrder[0])
.toBeLessThan((resolveOwnedMergedUpload as jest.Mock).mock.invocationCallOrder[0])
expect((resolveOwnedMergedUpload as jest.Mock).mock.invocationCallOrder[0])
.toBeLessThan((prisma.material.create as jest.Mock).mock.invocationCallOrder[0])
expect((prisma.material.create as jest.Mock).mock.invocationCallOrder[0])
.toBeLessThan((deleteOwnedUploadSession as jest.Mock).mock.invocationCallOrder[0])
expect((deleteOwnedUploadSession as jest.Mock).mock.invocationCallOrder[0])
.toBeLessThan(release.mock.invocationCallOrder[0])
})
})

File diff suppressed because it is too large Load Diff

View File

@ -1,40 +1,14 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { completeTaskImageMaterial } from '@/lib/task-image-material-completion';
import { assignTaskNoOnComplete } from '@/lib/product-package';
import {
createZipFromFiles,
generatePreviewImages,
cleanupTempFiles,
cleanupOnError,
saveAssets,
} from '@/lib/task-material-processor';
import { createMaterialLog } from '@/lib/material-log';
import { TASK_DETAIL_UPLOAD_LIMITS } from '@/lib/task-detail-upload-policy';
import { stat } from 'fs/promises';
export const runtime = 'nodejs' export const runtime = 'nodejs';
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic';
export const maxDuration = 10800 export const maxDuration = 10800;
class TaskPageUploadValidationError extends Error {
constructor(message: string, readonly status: number = 400) {
super(message)
this.name = 'TaskPageUploadValidationError'
}
}
// POST - 完成任务-页面上传
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let materialId: string | null = null;
const tempFiles: string[] = []; // 需要清理的临时文件路径
let zipFilePath: string | null = null;
const uploadedPreviewUrls: string[] = []; // 新生成的S3预览图URLs
const uploadIds: string[] = []; // 需要清理的uploadId列表
let claimedForCompletion = false;
let completed = false;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -42,218 +16,26 @@ export async function POST(request: NextRequest) {
} }
const body = await request.json(); const body = await request.json();
const { materialId: matId, assets, taskNo, dingtalkApprovalNo } = body; const result = await completeTaskImageMaterial({
materialId: body?.materialId,
materialId = matId;
if (!materialId || !assets) {
return NextResponse.json(
{ error: '缺少必填字段' },
{ status: 400 }
);
}
const mainAssets = Array.isArray(assets.main) ? assets.main : [];
if (mainAssets.length === 0) {
return NextResponse.json(
{ error: '至少需要上传一张预览图' },
{ status: 400 }
);
}
console.log('[Task Page Complete] Material ID:', materialId);
console.log('[Task Page Complete] Preview images count:', mainAssets.length);
// 验证素材是否存在且属于当前用户
const material: any = await prisma.material.findUnique({
where: { id: materialId },
});
if (!material) {
return NextResponse.json(
{ success: false, error: '素材不存在' },
{ status: 404 }
);
}
if (material.userId !== (session.user as any).id) {
return NextResponse.json(
{ success: false, error: '无权限操作此素材' },
{ status: 403 }
);
}
if (material.materialType !== '任务-页面') {
return NextResponse.json(
{ success: false, error: '素材类型不匹配' },
{ status: 400 }
);
}
// 只有一个请求能把 uploading 原子地切换为 processing。重复或并发完成请求
// 不会进入清理分支,更不会误删已经完成的素材。
const claimResult = await prisma.material.updateMany({
where: {
id: materialId,
userId: (session.user as any).id, userId: (session.user as any).id,
materialType: '任务-页面', materialType: '任务-页面',
deletedAt: null, fallbackProductModel: 'task-page',
taskStatus: 'uploading', assets: Array.isArray(body?.assets?.main) ? body.assets.main : [],
}, taskNo: body?.taskNo,
data: { taskStatus: 'processing' }, dingtalkApprovalNo: body?.dingtalkApprovalNo,
});
if (claimResult.count !== 1) {
return NextResponse.json(
{ success: false, error: '该素材正在处理或已经完成,请勿重复提交' },
{ status: 409 },
);
}
claimedForCompletion = true;
if (mainAssets.length > TASK_DETAIL_UPLOAD_LIMITS.maxImagesPerGroup) {
throw new TaskPageUploadValidationError(
`预览图最多上传 ${TASK_DETAIL_UPLOAD_LIMITS.maxImagesPerGroup} 张,请分批提交`,
);
}
const requestedUploadIds = mainAssets.map((asset: any) => String(asset?.uploadId || '').trim());
if (requestedUploadIds.some((uploadId: string) => !uploadId)) {
throw new TaskPageUploadValidationError('上传会话编号无效');
}
if (new Set(requestedUploadIds).size !== requestedUploadIds.length) {
throw new TaskPageUploadValidationError('上传会话编号重复,请重新选择文件');
}
// 步骤1: 从chunk临时目录读取预览图文件
console.log(`[Task Page] Processing ${mainAssets.length} preview images...`);
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions');
const previewFiles: Array<{ path: string; name: string; size: number }> = [];
let totalSourceBytes = 0;
for (const asset of mainAssets) {
try {
const uploadId = String(asset.uploadId).trim();
const fileName = String(asset.fileName || 'image');
const filePath = await getOwnedUploadedFile(uploadId, (session.user as any).id);
if (filePath) {
tempFiles.push(filePath);
uploadIds.push(uploadId);
const fileStats = await stat(filePath);
if (!fileStats.isFile()) {
throw new TaskPageUploadValidationError(`上传文件无效: ${fileName}`);
}
if (fileStats.size > TASK_DETAIL_UPLOAD_LIMITS.maxSingleImageBytes) {
throw new TaskPageUploadValidationError(
`图片「${fileName}」超过单图 ${Math.round(TASK_DETAIL_UPLOAD_LIMITS.maxSingleImageBytes / 1024 / 1024)}MB 限制`,
413,
);
}
totalSourceBytes += fileStats.size;
if (totalSourceBytes > TASK_DETAIL_UPLOAD_LIMITS.maxTotalSourceBytes) {
throw new TaskPageUploadValidationError('本次图片总大小超过 1GB,请分批提交', 413);
}
previewFiles.push({
path: filePath,
name: fileName,
size: fileStats.size,
});
} else {
throw new Error(`找不到uploadId对应的文件: ${uploadId}`);
}
} catch (error) {
if (error instanceof TaskPageUploadValidationError) {
throw error;
}
console.error(`[Task Page] 获取预览图文件失败 ${asset.uploadId}:`, error);
throw new Error(`获取预览图文件失败: ${asset.fileName}`);
}
}
// 步骤2: 打包预览图成zip(保存到本地文件系统)
console.log(`[Task Page] Creating zip from ${previewFiles.length} preview files...`);
const zipInfo = await createZipFromFiles(
previewFiles,
materialId!,
material.productModel || 'task-page',
'任务-页面'
);
console.log(`[Task Page] Zip created: ${zipInfo.fileName} (${zipInfo.fileSize} bytes)`);
zipFilePath = zipInfo.filePath;
// 步骤3: 生成预览图并上传到S3
const previewUrls = await generatePreviewImages(
mainAssets.map((a: any) => ({ uploadId: a.uploadId, fileName: a.fileName })),
'任务-页面'
);
uploadedPreviewUrls.push(...previewUrls);
if (uploadedPreviewUrls.length === 0) {
throw new Error('生成预览图失败,至少需要一张预览图');
}
// 步骤4: 保存资产记录到数据库
await saveAssets(materialId!, uploadedPreviewUrls, zipInfo);
// 步骤4.5: 仅在素材成功完成后写入任务号
await assignTaskNoOnComplete(materialId!, taskNo, dingtalkApprovalNo);
completed = true;
// 步骤5: 清理临时文件
await cleanupTempFiles(tempFiles, uploadIds).catch((cleanupError) => {
console.warn('[Task Page] 上传已完成,但临时文件清理失败:', cleanupError);
});
// 步骤6: 记录上传日志
await createMaterialLog({
materialId: materialId!,
userId: (session.user as any).id,
action: 'UPLOAD',
changes: {
uploadType: {
oldValue: null,
newValue: '任务-页面',
},
previewCount: {
oldValue: 0,
newValue: uploadedPreviewUrls.length,
},
zipFileName: {
oldValue: null,
newValue: zipInfo.fileName,
},
},
}).catch((logError) => {
console.error('[Task Page] 上传已完成,但日志记录失败:', logError);
});
console.log('[Task Page] Complete success!');
return NextResponse.json({
success: true,
materialId,
}); });
return NextResponse.json({ success: true, ...result });
} catch (error: any) { } catch (error: any) {
console.error('[Task Page Complete Error]', error); console.error('[Task Page Complete Error]', error);
const status = Number.isInteger(error?.status) ? error.status : 500;
// 只有成功抢占 processing 状态的请求,才有资格清理由本请求创建的资源和素材。 const response = NextResponse.json(
// 重复请求、鉴权失败或查询异常绝不能根据用户传入的 materialId 删除数据。 { error: error?.message || '完成上传失败' },
if (claimedForCompletion && !completed) { { status },
await cleanupOnError(
uploadedPreviewUrls,
zipFilePath,
tempFiles,
materialId,
uploadIds,
true,
{ taskStatus: 'processing' },
); );
if (Number.isFinite(error?.retryAfterSeconds)) {
response.headers.set('Retry-After', String(error.retryAfterSeconds));
} }
return response;
return NextResponse.json(
{ error: error.message || '完成上传失败' },
{ status: error instanceof TaskPageUploadValidationError ? error.status : 500 }
);
} }
} }

View File

@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { purgeDeletedMaterialsByTaskNo, validateTaskNo } from '@/lib/product-package'; import { validateTaskNo } from '@/lib/product-package';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
// POST - 初始化任务-页面素材记录 // POST - 初始化任务-页面素材记录
@ -49,14 +49,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(`[Task Page Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条`);
}
} catch (cleanupError) {
console.error('[Task Page Init] 清理旧软删素材失败:', cleanupError);
}
} }
const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo); const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo);

View File

@ -1,29 +1,14 @@
import { NextRequest, NextResponse } from 'next/server'; import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { completeTaskImageMaterial } from '@/lib/task-image-material-completion';
import { assignTaskNoOnComplete } from '@/lib/product-package';
import {
createZipFromFiles,
generatePreviewImages,
cleanupTempFiles,
cleanupOnError,
saveAssets,
} from '@/lib/task-material-processor';
import { createMaterialLog } from '@/lib/material-log';
export const runtime = 'nodejs' export const runtime = 'nodejs';
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic';
export const maxDuration = 10800 export const maxDuration = 10800;
// POST - 完成视频截图上传
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let materialId: string | null = null;
const tempFiles: string[] = []; // 需要清理的临时文件路径
let zipFilePath: string | null = null;
const uploadedPreviewUrls: string[] = []; // 新生成的S3预览图URLs
const uploadIds: string[] = []; // 需要清理的uploadId列表
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -31,138 +16,26 @@ export async function POST(request: NextRequest) {
} }
const body = await request.json(); const body = await request.json();
const { materialId: matId, assets, taskNo, dingtalkApprovalNo } = body; const result = await completeTaskImageMaterial({
materialId: body?.materialId,
materialId = matId;
if (!materialId || !assets) {
return NextResponse.json(
{ error: '缺少必填字段' },
{ status: 400 }
);
}
if (!assets.main || assets.main.length === 0) {
return NextResponse.json(
{ error: '至少需要上传一张截图' },
{ status: 400 }
);
}
console.log('[Video Screenshot Complete] Starting, materialId:', materialId);
console.log('[Video Screenshot Complete] Screenshot count:', assets.main.length);
// 验证素材是否存在且属于当前用户
const material: any = await prisma.material.findUnique({
where: { id: materialId },
});
if (!material) {
return NextResponse.json(
{ success: false, error: '素材不存在' },
{ status: 404 }
);
}
if (material.userId !== (session.user as any).id) {
return NextResponse.json(
{ success: false, error: '无权限操作此素材' },
{ status: 403 }
);
}
// 步骤1: 从chunk临时目录读取截图文件
console.log(`[Video Screenshot] Processing ${assets.main.length} screenshots...`);
const { getOwnedUploadedFile } = await import('@/lib/upload-sessions');
const screenshotFiles: Array<{ path: string; name: string; size: number }> = [];
for (const asset of assets.main) {
try {
const filePath = await getOwnedUploadedFile(asset.uploadId, (session.user as any).id);
if (filePath) {
screenshotFiles.push({
path: filePath,
name: asset.fileName,
size: asset.fileSize,
});
tempFiles.push(filePath);
uploadIds.push(asset.uploadId);
} else {
throw new Error(`找不到uploadId对应的文件: ${asset.uploadId}`);
}
} catch (error) {
console.error(`[Video Screenshot] Failed to get file for uploadId ${asset.uploadId}:`, error);
throw new Error(`获取截图文件失败: ${asset.fileName}`);
}
}
// 步骤2: 打包截图成zip(保存到本地文件系统)
console.log(`[Video Screenshot] Creating zip from ${screenshotFiles.length} screenshots...`);
const zipInfo = await createZipFromFiles(
screenshotFiles,
materialId!,
material.productModel || 'screenshots',
'任务-视频截图'
);
console.log(`[Video Screenshot] Zip created: ${zipInfo.fileName} (${zipInfo.fileSize} bytes)`);
zipFilePath = zipInfo.filePath;
// 步骤3: 生成预览图并上传到S3
const previewUrls = await generatePreviewImages(
assets.main.map((a: any) => ({ uploadId: a.uploadId, fileName: a.fileName })),
'任务-视频截图'
);
uploadedPreviewUrls.push(...previewUrls);
if (uploadedPreviewUrls.length === 0) {
throw new Error('生成预览图失败,至少需要一张预览图');
}
// 步骤4: 保存资产记录到数据库
await saveAssets(materialId!, uploadedPreviewUrls, zipInfo);
// 步骤4.5: 仅在素材成功完成后写入任务号
await assignTaskNoOnComplete(materialId!, taskNo, dingtalkApprovalNo);
// 步骤5: 清理临时文件
await cleanupTempFiles(tempFiles, uploadIds);
// 步骤6: 记录上传日志
await createMaterialLog({
materialId: materialId!,
userId: (session.user as any).id, userId: (session.user as any).id,
action: 'UPLOAD', materialType: '任务-视频截图',
changes: { fallbackProductModel: 'video-screenshot',
uploadType: { assets: Array.isArray(body?.assets?.main) ? body.assets.main : [],
oldValue: null, taskNo: body?.taskNo,
newValue: '任务-视频截图', dingtalkApprovalNo: body?.dingtalkApprovalNo,
},
previewCount: {
oldValue: 0,
newValue: uploadedPreviewUrls.length,
},
zipFileName: {
oldValue: null,
newValue: zipInfo.fileName,
},
},
});
console.log('[Video Screenshot] Complete success!');
return NextResponse.json({
success: true,
materialId,
}); });
return NextResponse.json({ success: true, ...result });
} catch (error: any) { } catch (error: any) {
console.error('[Video Screenshot Complete Error]', error); console.error('[Video Screenshot Complete Error]', error);
const status = Number.isInteger(error?.status) ? error.status : 500;
// 清理失败时的资源 const response = NextResponse.json(
await cleanupOnError(uploadedPreviewUrls, zipFilePath, tempFiles, materialId); { error: error?.message || '完成上传失败' },
{ status },
return NextResponse.json(
{ error: error.message || '完成上传失败' },
{ status: 500 }
); );
if (Number.isFinite(error?.retryAfterSeconds)) {
response.headers.set('Retry-After', String(error.retryAfterSeconds));
}
return response;
} }
} }

View File

@ -2,7 +2,7 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { purgeDeletedMaterialsByTaskNo, validateTaskNo } from '@/lib/product-package'; import { validateTaskNo } from '@/lib/product-package';
import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no'; import { getDingtalkApprovalNoError } from '@/lib/dingtalk-approval-no';
// POST - 初始化视频截图素材记录 // POST - 初始化视频截图素材记录
@ -49,14 +49,6 @@ export async function POST(request: NextRequest) {
); );
} }
try {
const cleanupResult = await purgeDeletedMaterialsByTaskNo(taskNo);
if (cleanupResult.deletedMaterials > 0) {
console.log(`[Video Screenshot Init] 已清理任务号 ${taskNo} 的旧软删素材 ${cleanupResult.deletedMaterials} 条`);
}
} catch (cleanupError) {
console.error('[Video Screenshot Init] 清理旧软删素材失败:', cleanupError);
}
} }
const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo); const dingtalkApprovalNoError = getDingtalkApprovalNoError(dingtalkApprovalNo);
@ -91,6 +83,7 @@ export async function POST(request: NextRequest) {
previewUrl: '', // 临时空值,完成上传时更新 previewUrl: '', // 临时空值,完成上传时更新
zipFilePath: '', // 临时空值,完成上传时更新 zipFilePath: '', // 临时空值,完成上传时更新
zipFileName: '', // 临时空值,完成上传时更新 zipFileName: '', // 临时空值,完成上传时更新
taskStatus: 'uploading',
renderer: '', // 视频截图不需要渲染器 renderer: '', // 视频截图不需要渲染器
}, },
}); });

View File

@ -1,3 +1,4 @@
// LEGACY_READ_ONLY: 旧域名仅用于代理读取历史素材,禁止作为新写入目标。
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { DownloadErrorCode } from '@/lib/download-errors' import { DownloadErrorCode } from '@/lib/download-errors'
import { import {

View File

@ -0,0 +1,56 @@
/** @jest-environment node */
import { POST } from '@/app/api/restore-data/route'
import { getServerSession } from 'next-auth'
import { hasSystemAdminPermission } from '@/lib/department-permission'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({
authOptions: {},
}))
jest.mock('@/lib/department-permission', () => ({
hasSystemAdminPermission: jest.fn(),
}))
describe('/api/restore-data', () => {
beforeEach(() => {
jest.clearAllMocks()
})
test('匿名访问仍被拒绝', async () => {
;(getServerSession as jest.Mock).mockResolvedValue(null)
const response = await POST()
expect(response.status).toBe(401)
})
test('普通管理员不能进入历史恢复流程', async () => {
;(getServerSession as jest.Mock).mockResolvedValue({
user: { role: 'ADMIN', isSuperAdmin: false },
})
;(hasSystemAdminPermission as jest.Mock).mockReturnValue(false)
const response = await POST()
expect(response.status).toBe(403)
})
test('系统管理员也只能收到永久停用说明,不再触发任何写入', async () => {
;(getServerSession as jest.Mock).mockResolvedValue({
user: { role: 'SYSTEM_ADMIN', isSuperAdmin: false },
})
;(hasSystemAdminPermission as jest.Mock).mockReturnValue(true)
const response = await POST()
const body = await response.json()
expect(response.status).toBe(410)
expect(body.error).toBe('在线历史恢复接口已停用')
expect(body.action).toContain('dry-run')
})
})

View File

@ -1,264 +1,34 @@
// 数据恢复API - 从S3和本地文件恢复素材数据 import { NextResponse } from 'next/server'
import { NextRequest, NextResponse } from 'next/server'; import { getServerSession } from 'next-auth'
import { getServerSession } from 'next-auth'; import { authOptions } from '@/lib/auth'
import { prisma } from '@/lib/prisma'; import { hasSystemAdminPermission } from '@/lib/department-permission'
import https from 'https';
import fs from 'fs';
import path from 'path';
import { authOptions } from '@/lib/auth';
import { hasSystemAdminPermission } from '@/lib/department-permission';
const UPLOADS_DIR = '/app/uploads'; /**
const DEFAULT_CATEGORY = '默认店铺'; * 旧版在线恢复接口曾把历史桶地址直接写回新素材记录。
const DEFAULT_SCENE = '默认场景'; *
const MAX_RESTORE_ITEMS = 500; * 该流程无法保证对象已经迁移到当前存储桶,也没有逐条清单、备份和回滚,
const RATE_LIMIT_WINDOW_MS = 30 * 60 * 1000; * 因此永久停用。历史恢复必须改用默认 dry-run、显式来源和显式目标的离线脚本,
const RATE_LIMIT_MAX_REQUESTS = 2; * 并在写库前先完成对象迁移与可读性校验。
const requestCounts = new Map<string, { count: number; resetAt: number }>(); */
export async function POST() {
function consumeRateLimit(userId: string): number | null { const session = await getServerSession(authOptions)
const now = Date.now();
const current = requestCounts.get(userId);
if (!current || current.resetAt <= now) {
requestCounts.set(userId, { count: 1, resetAt: now + RATE_LIMIT_WINDOW_MS });
return null;
}
if (current.count >= RATE_LIMIT_MAX_REQUESTS) {
return Math.max(1, Math.ceil((current.resetAt - now) / 1000));
}
current.count += 1;
return null;
}
// 从S3获取预览图列表
async function fetchS3PreviewList(maxKeys: number): Promise<string[]> {
return new Promise((resolve, reject) => {
https.get(`https://s3.1080p.hk/jsxiangche/?max-keys=${maxKeys}`, (res) => {
let data = '';
res.on('data', (chunk) => { data += chunk; });
res.on('end', () => {
const keys = parseS3ListResponse(data);
resolve(keys.filter(key => key.includes('preview-')));
});
}).on('error', reject);
});
}
// 解析S3列表响应
function parseS3ListResponse(xml: string): string[] {
const keys: string[] = [];
const keyRegex = /<Key>([^<]+)<\/Key>/g;
let match;
while ((match = keyRegex.exec(xml)) !== null) {
keys.push(match[1]);
}
return keys;
}
// 从文件名提取时间戳
function extractTimestamp(filename: string): number | null {
const match = filename.match(/-(\d{13})-/);
return match ? parseInt(match[1]) : null;
}
// 扫描本地素材文件
function scanLocalMaterials(dir: string, limit: number): string[] {
try {
return fs.readdirSync(dir).filter(file =>
file.startsWith('zip-') &&
(file.endsWith('.zip') || file.endsWith('.rar') || file.endsWith('.7z'))
).slice(0, limit);
} catch (error: any) {
console.error('读取本地文件失败:', error.message);
return [];
}
}
// 匹配预览图和素材文件
function matchPreviewsWithMaterials(
previews: string[],
materials: string[]
): Array<{ previewFile: string | null; materialFile: string; timestamp: number | null }> {
const matches: Array<{ previewFile: string | null; materialFile: string; timestamp: number | null }> = [];
for (const material of materials) {
const materialTimestamp = extractTimestamp(material);
if (!materialTimestamp) continue;
// 查找时间戳相近的预览图(允许1秒误差)
const matchedPreview = previews.find(preview => {
const previewTimestamp = extractTimestamp(preview);
if (!previewTimestamp) return false;
return Math.abs(previewTimestamp - materialTimestamp) < 1000;
});
matches.push({
previewFile: matchedPreview || null,
materialFile: material,
timestamp: materialTimestamp,
});
}
return matches;
}
// 从文件名提取产品名称
function extractProductName(filename: string): string {
const parts = filename.split('-');
if (parts.length >= 3) {
return parts.slice(2).join('-').replace(/\.(zip|rar|7z)$/i, '');
}
return filename;
}
// POST - 执行数据恢复
export async function POST(request: NextRequest) {
try {
const session = await getServerSession(authOptions);
if (!session?.user) { if (!session?.user) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }); return NextResponse.json({ error: '请先登录' }, { status: 401 })
} }
const sessionUser = session.user as any; const sessionUser = session.user as {
role?: string | null
isSuperAdmin?: boolean
}
if (!hasSystemAdminPermission(sessionUser.role, Boolean(sessionUser.isSuperAdmin))) { if (!hasSystemAdminPermission(sessionUser.role, Boolean(sessionUser.isSuperAdmin))) {
return NextResponse.json({ error: '仅系统管理员可执行数据恢复' }, { status: 403 }); return NextResponse.json({ error: '仅系统管理员可执行数据恢复' }, { status: 403 })
} }
const retryAfter = consumeRateLimit(String(sessionUser.id || 'unknown'));
if (retryAfter) {
return NextResponse.json(
{ error: '数据恢复请求过于频繁,请稍后再试' },
{ status: 429, headers: { 'Retry-After': String(retryAfter) } },
);
}
const limit = Number(request.nextUrl.searchParams.get('limit') || '100');
if (!Number.isInteger(limit) || limit < 1 || limit > MAX_RESTORE_ITEMS) {
return NextResponse.json(
{ error: `limit 必须是 1 到 ${MAX_RESTORE_ITEMS} 之间的整数` },
{ status: 400 },
);
}
const adminUser = await prisma.user.findUnique({
where: { id: String(sessionUser.id) },
select: { id: true, username: true },
});
if (!adminUser) {
return NextResponse.json({ error: '当前管理员不存在' }, { status: 403 });
}
const logs: string[] = [];
logs.push('🚀 开始数据恢复...\n');
logs.push(`📝 步骤1: 使用当前管理员 ${adminUser.username} 作为恢复数据归属人`);
// 步骤2: 创建默认分类和场景
logs.push('\n📝 步骤2: 创建默认分类和场景');
try {
await prisma.category.create({ data: { name: DEFAULT_CATEGORY } });
logs.push(`✅ 创建默认分类: ${DEFAULT_CATEGORY}`);
} catch (error: any) {
if (error.code === 'P2002') {
logs.push(`ℹ️ 默认分类已存在`);
}
}
try {
await prisma.scene.create({ data: { name: DEFAULT_SCENE } });
logs.push(`✅ 创建默认场景: ${DEFAULT_SCENE}`);
} catch (error: any) {
if (error.code === 'P2002') {
logs.push(`ℹ️ 默认场景已存在`);
}
}
// 步骤3: 获取S3预览图列表
logs.push('\n📝 步骤3: 获取S3预览图列表');
const candidateLimit = Math.min(limit * 4, 2000);
const s3Previews = await fetchS3PreviewList(candidateLimit);
logs.push(`✅ 从S3找到 ${s3Previews.length} 个预览图`);
// 步骤4: 扫描本地素材文件
logs.push('\n📝 步骤4: 扫描本地素材文件');
const localMaterials = scanLocalMaterials(UPLOADS_DIR, candidateLimit);
logs.push(`✅ 从本地找到 ${localMaterials.length} 个素材文件`);
// 步骤5: 匹配预览图和素材文件
logs.push('\n📝 步骤5: 匹配预览图和素材文件');
const matches = matchPreviewsWithMaterials(s3Previews, localMaterials).slice(0, limit);
const withPreview = matches.filter(m => m.previewFile).length;
const withoutPreview = matches.length - withPreview;
logs.push(`✅ 总计 ${matches.length} 个素材 (${withPreview} 个有预览图, ${withoutPreview} 个无预览图)`);
// 步骤6: 恢复素材数据到数据库
logs.push('\n📝 步骤6: 恢复素材数据到数据库');
let successCount = 0;
let failCount = 0;
for (const match of matches) {
try {
const productName = extractProductName(match.materialFile);
const localPath = path.join(UPLOADS_DIR, match.materialFile);
const dockerPath = path.join('/mnt/PoolOne/apps/youyidian-3d/uploads', match.materialFile);
const stats = fs.statSync(localPath);
await prisma.material.create({
data: {
previewUrl: match.previewFile ? `https://s3.1080p.hk/jsxiangche/${match.previewFile}` : '',
productModel: productName.substring(0, 50),
categoryName: DEFAULT_CATEGORY,
sceneName: DEFAULT_SCENE,
materialType: '3D场景',
renderer: 'CR',
authorName: adminUser.username,
zipFileName: match.materialFile,
zipFilePath: dockerPath,
zipFileSize: BigInt(stats.size),
downloadCount: 0,
likeCount: 0,
userId: adminUser.id,
},
});
successCount++;
if (successCount % 10 === 0) {
logs.push(`✅ 已恢复: ${successCount}/${matches.length}`);
}
} catch (error: any) {
failCount++;
logs.push(`❌ 恢复失败: ${match.materialFile} - ${error.message}`);
}
}
logs.push(`\n📊 恢复完成统计:`);
logs.push(` ✅ 成功: ${successCount} 个`);
logs.push(` ❌ 失败: ${failCount} 个`);
logs.push(` 📁 总计: ${matches.length} 个文件`);
logs.push(`\n🎉 数据恢复完成!`);
logs.push(`\n📌 默认信息:`);
logs.push(` 数据归属管理员: ${adminUser.username}`);
logs.push(` 默认分类: ${DEFAULT_CATEGORY}`);
logs.push(` 默认场景: ${DEFAULT_SCENE}`);
logs.push(`\n⚠️ 提示: 请登录后台修改素材的SPU、分类、场景等信息`);
return NextResponse.json({
success: true,
message: logs.join('\n'),
stats: {
successCount,
failCount,
total: matches.length,
},
});
} catch (error: any) {
return NextResponse.json( return NextResponse.json(
{ {
success: false, error: '在线历史恢复接口已停用',
message: error.message, action: '请使用经过清单确认、默认 dry-run 且先迁移对象再写库的离线恢复流程',
}, },
{ status: 500 } { status: 410 },
); )
}
} }

View File

@ -4,6 +4,7 @@ import { GET, PUT } from '@/app/api/system-settings/route'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { prisma } from '@/lib/prisma' import { prisma } from '@/lib/prisma'
import { isSystemAdmin } from '@/lib/department-permission' import { isSystemAdmin } from '@/lib/department-permission'
import { proxyHttpUrl } from '@/lib/url-rewrite'
jest.mock('next-auth', () => ({ jest.mock('next-auth', () => ({
getServerSession: jest.fn(), getServerSession: jest.fn(),
@ -38,8 +39,8 @@ const storedSettings = {
id: 'settings-1', id: 'settings-1',
headerBgColor: '#123456', headerBgColor: '#123456',
siteName: '测试素材库', siteName: '测试素材库',
headerLogoUrl: 'https://static.example.com/header.png', headerLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/header.png',
loginLogoUrl: 'https://static.example.com/login.png', loginLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/login.png',
s3Endpoint: 'https://storage.internal.example.com', s3Endpoint: 'https://storage.internal.example.com',
s3AccessKey: 'test-access-key', s3AccessKey: 'test-access-key',
s3SecretKey: 'test-secret-key', s3SecretKey: 'test-secret-key',
@ -61,6 +62,7 @@ function createPutRequest(body: unknown) {
describe('/api/system-settings 安全响应', () => { describe('/api/system-settings 安全响应', () => {
beforeEach(() => { beforeEach(() => {
jest.clearAllMocks() jest.clearAllMocks()
;(proxyHttpUrl as jest.Mock).mockImplementation((url: string) => url)
;(isSystemAdmin as jest.Mock).mockImplementation( ;(isSystemAdmin as jest.Mock).mockImplementation(
(role: string | null | undefined) => (role: string | null | undefined) =>
role === 'ADMIN' || role === 'SYSTEM_ADMIN', role === 'ADMIN' || role === 'SYSTEM_ADMIN',
@ -80,8 +82,8 @@ describe('/api/system-settings 安全响应', () => {
expect(body).toEqual({ expect(body).toEqual({
headerBgColor: '#123456', headerBgColor: '#123456',
siteName: '测试素材库', siteName: '测试素材库',
headerLogoUrl: 'https://static.example.com/header.png', headerLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/header.png',
loginLogoUrl: 'https://static.example.com/login.png', loginLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/brand/login.png',
maintenanceEnabled: true, maintenanceEnabled: true,
maintenanceMessage: '计划维护', maintenanceMessage: '计划维护',
maintenanceStartTime: '2026-07-30T10:00:00.000Z', maintenanceStartTime: '2026-07-30T10:00:00.000Z',
@ -124,6 +126,29 @@ describe('/api/system-settings 安全响应', () => {
}) })
}) })
test('数据库遗留旧域名、第三方或代理地址时,GET 只返回本地安全品牌图', async () => {
;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'admin-1', role: 'SYSTEM_ADMIN' },
})
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue({
...storedSettings,
headerLogoUrl: 'https://s3.1080p.hk/jsxiangche/header.png',
loginLogoUrl: '/api/proxy-media?url=https%3A%2F%2Fstatic.example.com%2Flogo.png',
})
const response = await GET()
const body = await response.json()
expect(response.status).toBe(200)
expect(body.headerLogoUrl).toBe('/brand/header-logo.png')
expect(body.loginLogoUrl).toBe('/brand/login-logo.png')
expect(body.headerLogoPreviewUrl).toBe('/brand/header-logo.png')
expect(body.loginLogoPreviewUrl).toBe('/brand/login-logo.png')
expect(body.brandLogoConfigurationNeedsRepair).toBe(true)
expect(JSON.stringify(body)).not.toContain('s3.1080p.hk')
expect(JSON.stringify(body)).not.toContain('static.example.com')
})
test('普通用户不能更新系统设置', async () => { test('普通用户不能更新系统设置', async () => {
;(getServerSession as jest.Mock).mockResolvedValue({ ;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'user-1', role: 'USER' }, user: { id: 'user-1', role: 'USER' },
@ -207,4 +232,91 @@ describe('/api/system-settings 安全响应', () => {
expect(response.status).toBe(200) expect(response.status).toBe(200)
expect(prisma.systemSettings.update).toHaveBeenCalled() expect(prisma.systemSettings.update).toHaveBeenCalled()
}) })
test('管理员 GET 的长期地址与预览地址分离,GET 后保存不会把代理地址写入数据库', async () => {
const headerLogoUrl = 'https://s3.jiansun.vip/gongxiangpan/brand/header.png'
const loginLogoUrl = 'https://s3.jiansun.vip/gongxiangpan/brand/login.png'
;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'admin-1', role: 'SYSTEM_ADMIN' },
})
;(proxyHttpUrl as jest.Mock).mockImplementation(
(url: string) => `/api/proxy-media?url=${encodeURIComponent(url)}`,
)
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValueOnce({
...storedSettings,
headerLogoUrl,
loginLogoUrl,
})
const getResponse = await GET()
const getBody = await getResponse.json()
expect(getBody.headerLogoUrl).toBe(headerLogoUrl)
expect(getBody.loginLogoUrl).toBe(loginLogoUrl)
expect(getBody.headerLogoPreviewUrl).toBe(
`/api/proxy-media?url=${encodeURIComponent(headerLogoUrl)}`,
)
expect(getBody.loginLogoPreviewUrl).toBe(
`/api/proxy-media?url=${encodeURIComponent(loginLogoUrl)}`,
)
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValueOnce({
id: storedSettings.id,
})
;(prisma.systemSettings.update as jest.Mock).mockResolvedValue({
...storedSettings,
headerLogoUrl,
loginLogoUrl,
})
const putResponse = await PUT(createPutRequest({
headerLogoUrl: getBody.headerLogoUrl,
loginLogoUrl: getBody.loginLogoUrl,
}))
expect(putResponse.status).toBe(200)
expect(prisma.systemSettings.update).toHaveBeenCalledWith(
expect.objectContaining({
data: expect.objectContaining({ headerLogoUrl, loginLogoUrl }),
}),
)
expect(JSON.stringify((prisma.systemSettings.update as jest.Mock).mock.calls)).not.toContain(
'/api/proxy-media',
)
})
test.each([
'/api/proxy-media?url=https%3A%2F%2Fs3.jiansun.vip%2Fgongxiangpan%2Fbrand%2Fheader.png',
'http://s3.jiansun.vip/gongxiangpan/brand/header.png',
'https://s3.1080p.hk/jsxiangche/header.png',
'https://static.example.com/header.png',
'/uploads/header.png',
'/brand/../server.js',
'https://s3.jiansun.vip/gongxiangpan/brand/header.png?token=temporary',
])('拒绝把不受控或临时展示地址写成 LOGO 长期地址:%s', async (invalidUrl) => {
;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'admin-1', role: 'ADMIN' },
})
const response = await PUT(createPutRequest({ headerLogoUrl: invalidUrl }))
const body = await response.json()
expect(response.status).toBe(400)
expect(body.error).toContain('LOGO 地址无效')
expect(prisma.systemSettings.findFirst).not.toHaveBeenCalled()
expect(prisma.systemSettings.update).not.toHaveBeenCalled()
expect(prisma.systemSettings.create).not.toHaveBeenCalled()
})
test('无数据库设置时使用镜像内置的页头与登录品牌图', async () => {
;(getServerSession as jest.Mock).mockResolvedValue(null)
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue(null)
const response = await GET()
const body = await response.json()
expect(response.status).toBe(200)
expect(body.headerLogoUrl).toBe('/brand/header-logo.png')
expect(body.loginLogoUrl).toBe('/brand/login-logo.png')
})
}) })

View File

@ -6,11 +6,14 @@ import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { isSystemAdmin } from '@/lib/department-permission'; import { isSystemAdmin } from '@/lib/department-permission';
import { proxyHttpUrl } from '@/lib/url-rewrite'; import { proxyHttpUrl } from '@/lib/url-rewrite';
import {
DEFAULT_HEADER_LOGO_URL,
DEFAULT_LOGIN_LOGO_URL,
DEFAULT_SITE_NAME,
normalizeBrandLogoUrl,
} from '@/lib/brand-assets';
const DEFAULT_HEADER_BG_COLOR = '#1f2937'; const DEFAULT_HEADER_BG_COLOR = '#1f2937';
const DEFAULT_SITE_NAME = '由壹点3D素材';
const DEFAULT_LOGO_URL =
'https://s3.1080p.hk/shuzhou/2025/05/18/8ebadf0a7cc34eaea00e8619b8efb827.png';
const NO_STORE_HEADERS = { const NO_STORE_HEADERS = {
'Cache-Control': 'no-store, no-cache, max-age=0, must-revalidate', 'Cache-Control': 'no-store, no-cache, max-age=0, must-revalidate',
@ -59,8 +62,10 @@ function canManageSystemSettings(user: SettingsSessionUser | null | undefined) {
} }
function toPublicSettingsDto(settings: SystemSettingsRecord | null) { function toPublicSettingsDto(settings: SystemSettingsRecord | null) {
const headerLogoUrl = settings?.headerLogoUrl || DEFAULT_LOGO_URL; const headerLogoUrl =
const loginLogoUrl = settings?.loginLogoUrl || DEFAULT_LOGO_URL; normalizeBrandLogoUrl(settings?.headerLogoUrl) || DEFAULT_HEADER_LOGO_URL;
const loginLogoUrl =
normalizeBrandLogoUrl(settings?.loginLogoUrl) || DEFAULT_LOGIN_LOGO_URL;
return { return {
headerBgColor: settings?.headerBgColor || DEFAULT_HEADER_BG_COLOR, headerBgColor: settings?.headerBgColor || DEFAULT_HEADER_BG_COLOR,
@ -78,13 +83,32 @@ function toAdminSettingsDto(
settings: SystemSettingsRecord | null, settings: SystemSettingsRecord | null,
s3CredentialsConfigured: boolean, s3CredentialsConfigured: boolean,
) { ) {
const normalizedStoredHeaderLogoUrl = normalizeBrandLogoUrl(settings?.headerLogoUrl);
const normalizedStoredLoginLogoUrl = normalizeBrandLogoUrl(settings?.loginLogoUrl);
const headerLogoUrl = normalizedStoredHeaderLogoUrl || DEFAULT_HEADER_LOGO_URL;
const loginLogoUrl = normalizedStoredLoginLogoUrl || DEFAULT_LOGIN_LOGO_URL;
return { return {
...toPublicSettingsDto(settings), headerBgColor: settings?.headerBgColor || DEFAULT_HEADER_BG_COLOR,
siteName: settings?.siteName || DEFAULT_SITE_NAME,
// 管理页编辑的是数据库中的长期地址,预览地址只用于 <img> 展示。
headerLogoUrl,
loginLogoUrl,
headerLogoPreviewUrl: proxyHttpUrl(headerLogoUrl) || headerLogoUrl,
loginLogoPreviewUrl: proxyHttpUrl(loginLogoUrl) || loginLogoUrl,
maintenanceEnabled: settings?.maintenanceEnabled ?? false,
maintenanceMessage: settings?.maintenanceMessage ?? null,
maintenanceStartTime: settings?.maintenanceStartTime ?? null,
maintenanceEndTime: settings?.maintenanceEndTime ?? null,
s3Endpoint: settings?.s3Endpoint || '', s3Endpoint: settings?.s3Endpoint || '',
s3Bucket: settings?.s3Bucket || '', s3Bucket: settings?.s3Bucket || '',
s3Region: settings?.s3Region || 'us-east-1', s3Region: settings?.s3Region || 'us-east-1',
s3PublicUrl: settings?.s3PublicUrl || '', s3PublicUrl: settings?.s3PublicUrl || '',
s3CredentialsConfigured, s3CredentialsConfigured,
brandLogoConfigurationNeedsRepair: Boolean(
settings &&
(!normalizedStoredHeaderLogoUrl || !normalizedStoredLoginLogoUrl),
),
}; };
} }
@ -173,6 +197,28 @@ export async function PUT(request: NextRequest) {
); );
} }
const normalizedHeaderLogoUrl =
headerLogoUrl === undefined
? undefined
: normalizeBrandLogoUrl(headerLogoUrl);
const normalizedLoginLogoUrl =
loginLogoUrl === undefined
? undefined
: normalizeBrandLogoUrl(loginLogoUrl);
if (headerLogoUrl !== undefined && !normalizedHeaderLogoUrl) {
return jsonNoStore(
{ error: '导航栏 LOGO 地址无效,请使用内置品牌资源或当前素材存储桶地址' },
400,
);
}
if (loginLogoUrl !== undefined && !normalizedLoginLogoUrl) {
return jsonNoStore(
{ error: '登录页 LOGO 地址无效,请使用内置品牌资源或当前素材存储桶地址' },
400,
);
}
const updateData: Prisma.SystemSettingsUpdateInput = {}; const updateData: Prisma.SystemSettingsUpdateInput = {};
if (typeof headerBgColor === 'string') { if (typeof headerBgColor === 'string') {
updateData.headerBgColor = headerBgColor; updateData.headerBgColor = headerBgColor;
@ -180,11 +226,11 @@ export async function PUT(request: NextRequest) {
if (typeof siteName === 'string') { if (typeof siteName === 'string') {
updateData.siteName = siteName; updateData.siteName = siteName;
} }
if (typeof headerLogoUrl === 'string') { if (normalizedHeaderLogoUrl) {
updateData.headerLogoUrl = headerLogoUrl; updateData.headerLogoUrl = normalizedHeaderLogoUrl;
} }
if (typeof loginLogoUrl === 'string') { if (normalizedLoginLogoUrl) {
updateData.loginLogoUrl = loginLogoUrl; updateData.loginLogoUrl = normalizedLoginLogoUrl;
} }
if (typeof s3Endpoint === 'string') { if (typeof s3Endpoint === 'string') {
updateData.s3Endpoint = s3Endpoint; updateData.s3Endpoint = s3Endpoint;
@ -246,13 +292,9 @@ export async function PUT(request: NextRequest) {
? headerBgColor ? headerBgColor
: DEFAULT_HEADER_BG_COLOR, : DEFAULT_HEADER_BG_COLOR,
headerLogoUrl: headerLogoUrl:
typeof headerLogoUrl === 'string' && headerLogoUrl normalizedHeaderLogoUrl || DEFAULT_HEADER_LOGO_URL,
? headerLogoUrl
: DEFAULT_LOGO_URL,
loginLogoUrl: loginLogoUrl:
typeof loginLogoUrl === 'string' && loginLogoUrl normalizedLoginLogoUrl || DEFAULT_LOGIN_LOGO_URL,
? loginLogoUrl
: DEFAULT_LOGO_URL,
siteName: siteName:
typeof siteName === 'string' && siteName typeof siteName === 'string' && siteName
? siteName ? siteName

View File

@ -0,0 +1,187 @@
/** @jest-environment node */
import { POST } from '@/app/api/system-settings/upload-logo/route'
import { getServerSession } from 'next-auth'
import { prisma } from '@/lib/prisma'
import { uploadToS3, verifyUploadedS3Object } from '@/lib/s3'
import { hasSystemAdminPermission } from '@/lib/department-permission'
import { readFileSync } from 'node:fs'
import { join } from 'node:path'
jest.mock('next-auth', () => ({
getServerSession: jest.fn(),
}))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('@/lib/prisma', () => ({
prisma: {
systemSettings: {
findFirst: jest.fn(),
update: jest.fn(),
create: jest.fn(),
},
},
}))
jest.mock('@/lib/s3', () => ({
uploadToS3: jest.fn(),
verifyUploadedS3Object: jest.fn(),
}))
jest.mock('@/lib/department-permission', () => ({
hasSystemAdminPermission: jest.fn(),
}))
function pngBytes(): Buffer {
return readFileSync(join(process.cwd(), 'public', 'brand', 'favicon.png'))
}
function createLogo(bytes: Buffer, overrides: Partial<File> = {}): File {
return {
name: 'logo.png',
type: 'image/png',
size: bytes.length,
arrayBuffer: jest.fn().mockResolvedValue(
bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
),
...overrides,
} as unknown as File
}
function createRequest(logo: File, type = 'header') {
return {
headers: new Headers({
'content-type': 'multipart/form-data; boundary=test-boundary',
'content-length': String(logo.size + 512),
}),
formData: jest.fn().mockResolvedValue({
get: (key: string) => key === 'logo' ? logo : key === 'type' ? type : null,
entries: function* () {
yield ['logo', logo]
yield ['type', type]
},
}),
} as any
}
describe('/api/system-settings/upload-logo', () => {
beforeEach(() => {
jest.clearAllMocks()
;(getServerSession as jest.Mock).mockResolvedValue({
user: { id: 'system-admin', role: 'SYSTEM_ADMIN' },
})
;(hasSystemAdminPermission as jest.Mock).mockReturnValue(true)
;(uploadToS3 as jest.Mock).mockResolvedValue(
'https://s3.jiansun.vip/gongxiangpan/由壹点AI/header-logo.png',
)
;(verifyUploadedS3Object as jest.Mock).mockResolvedValue(undefined)
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue(null)
;(prisma.systemSettings.create as jest.Mock).mockResolvedValue({ id: 'settings-1' })
})
test('系统管理员可上传,按真实签名决定格式,并在写库前 HEAD 校验', async () => {
const bytes = pngBytes()
const response = await POST(createRequest(createLogo(bytes)))
expect(response.status).toBe(200)
expect(uploadToS3).toHaveBeenCalledWith(
bytes,
expect.stringMatching(/^header-logo-\d+\.png$/),
'image/png',
)
expect(verifyUploadedS3Object).toHaveBeenCalledWith(
'https://s3.jiansun.vip/gongxiangpan/由壹点AI/header-logo.png',
bytes.length,
'image/png',
)
expect(prisma.systemSettings.findFirst).toHaveBeenCalledWith({
select: { id: true },
})
expect(prisma.systemSettings.create).toHaveBeenCalledWith({
data: {
headerLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/由壹点AI/header-logo.png',
loginLogoUrl: '/brand/login-logo.png',
},
select: { id: true },
})
})
test('已有设置行只读取 id,更新结果也不把存储凭据读入进程', async () => {
const bytes = pngBytes()
;(prisma.systemSettings.findFirst as jest.Mock).mockResolvedValue({ id: 'settings-1' })
;(prisma.systemSettings.update as jest.Mock).mockResolvedValue({ id: 'settings-1' })
const response = await POST(createRequest(createLogo(bytes), 'login'))
expect(response.status).toBe(200)
expect(prisma.systemSettings.findFirst).toHaveBeenCalledWith({
select: { id: true },
})
expect(prisma.systemSettings.update).toHaveBeenCalledWith({
where: { id: 'settings-1' },
data: {
loginLogoUrl: 'https://s3.jiansun.vip/gongxiangpan/由壹点AI/header-logo.png',
},
select: { id: true },
})
})
test('伪造 image/png 的非图片内容会被拒绝', async () => {
const bytes = Buffer.from('not-a-png')
const response = await POST(createRequest(createLogo(bytes)))
expect(response.status).toBe(400)
expect(uploadToS3).not.toHaveBeenCalled()
})
test('只有 PNG 文件头的截断图片也会被拒绝', async () => {
const bytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3])
const response = await POST(createRequest(createLogo(bytes)))
expect(response.status).toBe(400)
expect(uploadToS3).not.toHaveBeenCalled()
})
test('超过 5MB 时在读取文件内容前拒绝', async () => {
const logo = createLogo(pngBytes(), { size: 5 * 1024 * 1024 + 1 })
const response = await POST(createRequest(logo))
expect(response.status).toBe(413)
expect(logo.arrayBuffer).not.toHaveBeenCalled()
})
test('对象 HEAD 校验失败时保持原数据库值', async () => {
;(verifyUploadedS3Object as jest.Mock).mockRejectedValue(new Error('HEAD failed'))
const response = await POST(createRequest(createLogo(pngBytes())))
expect(response.status).toBe(500)
expect(prisma.systemSettings.create).not.toHaveBeenCalled()
expect(prisma.systemSettings.update).not.toHaveBeenCalled()
})
test('对象存储返回非受控地址时不做 HEAD 也不写数据库', async () => {
;(uploadToS3 as jest.Mock).mockResolvedValue('http://storage.internal/logo.png')
const response = await POST(createRequest(createLogo(pngBytes())))
expect(response.status).toBe(500)
expect(verifyUploadedS3Object).not.toHaveBeenCalled()
expect(prisma.systemSettings.create).not.toHaveBeenCalled()
expect(prisma.systemSettings.update).not.toHaveBeenCalled()
})
test('普通用户无权上传系统 Logo', async () => {
;(hasSystemAdminPermission as jest.Mock).mockReturnValue(false)
const response = await POST(createRequest(createLogo(pngBytes())))
expect(response.status).toBe(403)
expect(uploadToS3).not.toHaveBeenCalled()
})
})

View File

@ -3,10 +3,24 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { uploadToS3 } from '@/lib/s3'; import { uploadToS3, verifyUploadedS3Object } from '@/lib/s3';
import { hasSystemAdminPermission } from '@/lib/department-permission';
import {
DEFAULT_HEADER_LOGO_URL,
DEFAULT_LOGIN_LOGO_URL,
normalizeBrandLogoUrl,
} from '@/lib/brand-assets';
import { MAX_BRAND_IMAGE_BYTES, validateBrandImage } from '@/lib/brand-image-validation';
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
// POST - 上传Logo(仅管理员) // POST - 上传Logo(仅管理员)
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session?.user || !(session.user as any).id) { if (!session?.user || !(session.user as any).id) {
@ -15,19 +29,32 @@ export async function POST(request: NextRequest) {
{ status: 401 } { status: 401 }
); );
} }
const userRole = (session.user as any).role; const sessionUser = session.user as any;
if (userRole !== 'ADMIN') { if (!hasSystemAdminPermission(sessionUser.role, Boolean(sessionUser.isSuperAdmin))) {
return NextResponse.json( return NextResponse.json(
{ error: '无权限' }, { error: '无权限' },
{ status: 403 } { status: 403 }
); );
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.logoRequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
const logo = formData.get('logo') as File; assertMultipartFiles(formData, {
fields: {
logo: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.logoFileBytes,
allowedMimeTypes: ['image/png', 'image/jpeg', 'image/webp'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.logoFileBytes,
});
const logo = formData.get('logo') as File | null;
const type = formData.get('type') as string; const type = formData.get('type') as string;
if (!logo) { if (!logo || typeof logo.arrayBuffer !== 'function') {
return NextResponse.json( return NextResponse.json(
{ error: '请选择LOGO文件' }, { error: '请选择LOGO文件' },
{ status: 400 } { status: 400 }
@ -41,10 +68,16 @@ export async function POST(request: NextRequest) {
); );
} }
// 验证文件类型 if (!Number.isFinite(logo.size) || logo.size <= 0 || logo.size > MAX_BRAND_IMAGE_BYTES) {
if (!logo.type.startsWith('image/')) {
return NextResponse.json( return NextResponse.json(
{ error: '只支持图片格式' }, { error: 'Logo 必须是 5MB 以内的有效图片' },
{ status: 400 }
);
}
if (!['image/png', 'image/jpeg', 'image/webp'].includes(logo.type.toLowerCase())) {
return NextResponse.json(
{ error: '只支持 PNG、JPEG 或 WebP 图片' },
{ status: 400 } { status: 400 }
); );
} }
@ -52,27 +85,56 @@ export async function POST(request: NextRequest) {
// 转换为Buffer // 转换为Buffer
const arrayBuffer = await logo.arrayBuffer(); const arrayBuffer = await logo.arrayBuffer();
const buffer = Buffer.from(arrayBuffer); const buffer = Buffer.from(arrayBuffer);
if (buffer.length !== logo.size || buffer.length > MAX_BRAND_IMAGE_BYTES) {
return NextResponse.json(
{ error: 'Logo 文件大小校验失败' },
{ status: 400 }
);
}
let detected
try {
detected = await validateBrandImage(buffer, logo.type)
} catch {
return NextResponse.json(
{ error: 'Logo 文件损坏、格式不一致或尺寸过大' },
{ status: 400 }
);
}
// 生成文件名 // 生成文件名
const timestamp = Date.now(); const timestamp = Date.now();
const ext = logo.name.split('.').pop(); const filename = `${type}-logo-${timestamp}.${detected.extension}`;
const filename = `${type}-logo-${timestamp}.${ext}`;
// 上传到S3 // 上传到S3
const logoUrl = await uploadToS3(buffer, filename, logo.type); const uploadedLogoUrl = await uploadToS3(buffer, filename, detected.contentType);
const logoUrl = normalizeBrandLogoUrl(uploadedLogoUrl)
if (!logoUrl) {
throw new Error('对象存储返回了不受信任的品牌图片地址')
}
await verifyUploadedS3Object(logoUrl, buffer.length, detected.contentType)
// 更新数据库 // 更新数据库
let settings = await prisma.systemSettings.findFirst(); const existingSettings = await prisma.systemSettings.findFirst({
select: { id: true },
});
const fieldName = type === 'header' ? 'headerLogoUrl' : 'loginLogoUrl'; const fieldName = type === 'header' ? 'headerLogoUrl' : 'loginLogoUrl';
if (settings) { if (existingSettings) {
settings = await prisma.systemSettings.update({ await prisma.systemSettings.update({
where: { id: settings.id }, where: { id: existingSettings.id },
data: { [fieldName]: logoUrl }, data: { [fieldName]: logoUrl },
select: { id: true },
}); });
} else { } else {
settings = await prisma.systemSettings.create({ const initialLogos = {
data: { [fieldName]: logoUrl }, headerLogoUrl: DEFAULT_HEADER_LOGO_URL,
loginLogoUrl: DEFAULT_LOGIN_LOGO_URL,
[fieldName]: logoUrl,
}
await prisma.systemSettings.create({
data: initialLogos,
select: { id: true },
}); });
} }
@ -83,9 +145,23 @@ export async function POST(request: NextRequest) {
}); });
} catch (error) { } catch (error) {
console.error('Upload logo error:', error); console.error('Upload logo error:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json( return NextResponse.json(
{ error: '上传LOGO失败' }, { error: '上传LOGO失败' },
{ status: 500 } { status: 500 }
); );
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -0,0 +1,186 @@
/** @jest-environment node */
import sharp from 'sharp'
import { getServerSession } from 'next-auth'
import { GET } from '@/app/api/taobao-detail-plus/image-proxy/route'
import { assertProcessableMetadata } from '@/lib/image-processor'
import { withImageProcessingSlot } from '@/lib/image-processing-limiter'
jest.mock('next-auth', () => ({ getServerSession: jest.fn() }))
jest.mock('@/lib/auth', () => ({ authOptions: {} }))
jest.mock('@/lib/image-processing-limiter', () => ({
withImageProcessingSlot: jest.fn((task: () => Promise<unknown>) => task()),
}))
jest.mock('@/lib/image-processor', () => ({
assertProcessableMetadata: jest.fn(),
}))
jest.mock('sharp', () => jest.fn())
describe('taobao detail image proxy', () => {
const originalFetch = global.fetch
const pngBuffer = Buffer.from('safe-png')
beforeEach(() => {
jest.clearAllMocks()
;(getServerSession as jest.Mock).mockResolvedValue({ user: { id: 'user-1' } })
const pipeline = {
metadata: jest.fn().mockResolvedValue({ width: 800, height: 1200, pages: 1 }),
flatten: jest.fn(),
png: jest.fn(),
toBuffer: jest.fn().mockResolvedValue(pngBuffer),
}
pipeline.flatten.mockReturnValue(pipeline)
pipeline.png.mockReturnValue(pipeline)
;(sharp as unknown as jest.Mock).mockReturnValue(pipeline)
})
afterEach(() => {
global.fetch = originalFetch
jest.useRealTimers()
})
function request(rawUrl: string) {
return {
nextUrl: new URL(`https://materials.example/api/proxy?url=${encodeURIComponent(rawUrl)}`),
} as any
}
function responseWithStream(
chunks: Uint8Array[],
options: { status?: number; headers?: Record<string, string>; cancel?: jest.Mock } = {},
) {
const stream = new ReadableStream<Uint8Array>({
start(controller) {
chunks.forEach((chunk) => controller.enqueue(chunk))
controller.close()
},
cancel: options.cancel,
})
return new Response(stream, {
status: options.status ?? 200,
headers: options.headers,
})
}
test('只接受白名单域名的 HTTPS 默认端口', async () => {
global.fetch = jest.fn() as any
const httpResponse = await GET(request('http://img.alicdn.com/a.jpg'))
const portResponse = await GET(request('https://img.alicdn.com:8443/a.jpg'))
const suffixTrickResponse = await GET(request('https://alicdn.com.attacker.example/a.jpg'))
expect(httpResponse.status).toBe(400)
expect(portResponse.status).toBe(400)
expect(suffixTrickResponse.status).toBe(400)
expect(global.fetch).not.toHaveBeenCalled()
})
test('fetch 必须关闭自动重定向,并拒绝上游重定向且取消响应体', async () => {
const cancel = jest.fn().mockResolvedValue(undefined)
global.fetch = jest.fn().mockResolvedValue(responseWithStream(
[new Uint8Array([1])],
{ status: 302, headers: { location: 'https://attacker.example/a.jpg' }, cancel },
)) as any
const response = await GET(request('https://img.alicdn.com/a.jpg'))
const body = await response.json()
expect(response.status).toBe(502)
expect(body.error.code).toBe('UPSTREAM_REDIRECT_REJECTED')
expect(global.fetch).toHaveBeenCalledWith(
'https://img.alicdn.com/a.jpg',
expect.objectContaining({ redirect: 'manual', signal: expect.any(AbortSignal) }),
)
expect(cancel).toHaveBeenCalled()
})
test('Content-Length 超过 20MB 时在读取前拒绝并取消响应体', async () => {
const cancel = jest.fn()
global.fetch = jest.fn().mockResolvedValue(responseWithStream(
[new Uint8Array([1])],
{
headers: {
'content-type': 'image/jpeg',
'content-length': String(20 * 1024 * 1024 + 1),
},
cancel,
},
)) as any
const response = await GET(request('https://img.alicdn.com/a.jpg'))
const body = await response.json()
expect(response.status).toBe(413)
expect(body.error.code).toBe('UPSTREAM_IMAGE_TOO_LARGE')
expect(cancel).toHaveBeenCalled()
expect(sharp).not.toHaveBeenCalled()
})
test('缺少 Content-Length 仍以实际流量硬上限终止读取并 cancel', async () => {
const cancel = jest.fn().mockResolvedValue(undefined)
const releaseLock = jest.fn()
const read = jest.fn()
.mockResolvedValueOnce({ done: false, value: new Uint8Array(12 * 1024 * 1024) })
.mockResolvedValueOnce({ done: false, value: new Uint8Array(9 * 1024 * 1024) })
global.fetch = jest.fn().mockResolvedValue({
status: 200,
ok: true,
headers: new Headers({ 'content-type': 'image/jpeg' }),
body: {
locked: false,
getReader: () => ({ read, cancel, releaseLock }),
},
}) as any
const response = await GET(request('https://img.alicdn.com/a.jpg'))
const body = await response.json()
expect(response.status).toBe(413)
expect(body.error.code).toBe('UPSTREAM_IMAGE_TOO_LARGE')
expect(cancel).toHaveBeenCalled()
expect(sharp).not.toHaveBeenCalled()
})
test('成功响应先按声明长度完整读取,再在全局图片锁内校验像素并转换', async () => {
const source = new Uint8Array([1, 2, 3])
global.fetch = jest.fn().mockResolvedValue(responseWithStream([source], {
headers: {
'content-type': 'image/jpeg; charset=binary',
'content-length': String(source.byteLength),
},
})) as any
const response = await GET(request('https://img.alicdn.com/a.jpg'))
expect(response.status).toBe(200)
expect(response.headers.get('content-type')).toBe('image/png')
expect(Buffer.from(await response.arrayBuffer())).toEqual(pngBuffer)
expect(withImageProcessingSlot).toHaveBeenCalledTimes(1)
expect(assertProcessableMetadata).toHaveBeenCalledWith(
{ width: 800, height: 1200, pages: 1 },
{
maxInputWidth: 12_000,
maxInputHeight: 20_000,
maxInputPixels: 24_000_000,
},
)
})
test('上游超过十秒未响应时中止请求并返回 504', async () => {
jest.useFakeTimers()
global.fetch = jest.fn((_url, init: RequestInit) => new Promise((_resolve, reject) => {
init.signal?.addEventListener('abort', () => {
reject(Object.assign(new Error('aborted'), { name: 'AbortError' }))
})
})) as any
const responsePromise = GET(request('https://img.alicdn.com/a.jpg'))
await jest.advanceTimersByTimeAsync(10_001)
const response = await responsePromise
const body = await response.json()
expect(response.status).toBe(504)
expect(body.error.code).toBe('UPSTREAM_IMAGE_TIMEOUT')
})
})

View File

@ -1,107 +1,219 @@
import sharp from 'sharp' import sharp from 'sharp'
import { NextRequest, NextResponse } from 'next/server' import { NextRequest, NextResponse } from 'next/server'
import { getServerSession } from 'next-auth' import { getServerSession } from 'next-auth'
import { authOptions } from '@/lib/auth' import { authOptions } from '@/lib/auth'
import { assertProcessableMetadata } from '@/lib/image-processor'
import { withImageProcessingSlot } from '@/lib/image-processing-limiter'
const ALLOWED_IMAGE_HOST_SUFFIXES = [ const ALLOWED_IMAGE_HOST_SUFFIXES = [
'alicdn.com', 'alicdn.com',
'taobaocdn.com', 'taobaocdn.com',
'tbcdn.cn', 'tbcdn.cn',
] ]
const UPSTREAM_TIMEOUT_MS = 10_000
const MAX_UPSTREAM_IMAGE_BYTES = 20 * 1024 * 1024
const MAX_OUTPUT_IMAGE_BYTES = 50 * 1024 * 1024
const MAX_IMAGE_PIXELS = 24_000_000
const MAX_IMAGE_WIDTH = 12_000
const MAX_IMAGE_HEIGHT = 20_000
const ALLOWED_IMAGE_CONTENT_TYPES = new Set([
'image/jpeg',
'image/png',
'image/webp',
])
class ImageProxyInputError extends Error {
constructor(
readonly code: string,
message: string,
readonly status: number,
) {
super(message)
this.name = 'ImageProxyInputError'
}
}
function isAllowedImageHost(hostname: string): boolean { function isAllowedImageHost(hostname: string): boolean {
const normalized = hostname.toLowerCase().replace(/\.$/, '')
return ALLOWED_IMAGE_HOST_SUFFIXES.some( return ALLOWED_IMAGE_HOST_SUFFIXES.some(
(suffix) => hostname === suffix || hostname.endsWith(`.${suffix}`), (suffix) => normalized === suffix || normalized.endsWith(`.${suffix}`),
)
}
export async function GET(request: NextRequest) {
try {
const session = await getServerSession(authOptions)
if (!session) {
return NextResponse.json(
{
success: false,
error: {
code: 'UNAUTHORIZED',
message: '请先登录后再导出图片',
},
},
{ status: 401 },
)
}
const rawUrl = request.nextUrl.searchParams.get('url')
if (!rawUrl) {
return NextResponse.json(
{
success: false,
error: {
code: 'MISSING_URL',
message: '缺少图片地址',
},
},
{ status: 400 },
) )
} }
function validateTargetUrl(rawUrl: string): URL {
let targetUrl: URL let targetUrl: URL
try { try {
targetUrl = new URL(rawUrl) targetUrl = new URL(rawUrl)
} catch { } catch {
throw new ImageProxyInputError('INVALID_URL', '图片地址不正确', 400)
}
if (
targetUrl.protocol !== 'https:'
|| targetUrl.username
|| targetUrl.password
|| (targetUrl.port && targetUrl.port !== '443')
|| !isAllowedImageHost(targetUrl.hostname)
) {
throw new ImageProxyInputError('IMAGE_HOST_NOT_ALLOWED', '图片来源不允许导出', 400)
}
return targetUrl
}
async function cancelResponseBody(response: Response, reason?: unknown): Promise<void> {
if (!response.body || response.body.locked) return
await response.body.cancel(reason).catch(() => undefined)
}
function parseContentLength(response: Response): number | null {
const raw = response.headers.get('content-length')
if (raw === null) return null
if (!/^\d+$/.test(raw)) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_SIZE_INVALID', '上游图片大小信息无效', 502)
}
const contentLength = Number(raw)
if (!Number.isSafeInteger(contentLength) || contentLength <= 0) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_SIZE_INVALID', '上游图片大小信息无效', 502)
}
if (contentLength > MAX_UPSTREAM_IMAGE_BYTES) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_TOO_LARGE', '上游图片超过安全大小限制', 413)
}
return contentLength
}
async function readBoundedUpstreamImage(response: Response): Promise<Buffer> {
let expectedBytes: number | null
try {
expectedBytes = parseContentLength(response)
} catch (error) {
await cancelResponseBody(response, error)
throw error
}
if (!response.body) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_EMPTY', '上游图片内容为空', 502)
}
// 未声明长度时也只预留固定硬上限;实际读取每一块前再次检查,绝不 concat
// 无界 chunks。声明长度时最终还必须与实际字节数完全一致。
const output = Buffer.allocUnsafe(expectedBytes ?? MAX_UPSTREAM_IMAGE_BYTES)
const reader = response.body.getReader()
let offset = 0
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value?.byteLength) continue
if (offset + value.byteLength > MAX_UPSTREAM_IMAGE_BYTES) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_TOO_LARGE', '上游图片超过安全大小限制', 413)
}
if (expectedBytes !== null && offset + value.byteLength > expectedBytes) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_SIZE_MISMATCH', '上游图片实际大小与声明不一致', 502)
}
output.set(value, offset)
offset += value.byteLength
}
} catch (error) {
await reader.cancel(error).catch(() => undefined)
throw error
} finally {
reader.releaseLock()
}
if (offset <= 0 || (expectedBytes !== null && offset !== expectedBytes)) {
throw new ImageProxyInputError('UPSTREAM_IMAGE_SIZE_MISMATCH', '上游图片响应不完整', 502)
}
return output.subarray(0, offset)
}
function jsonError(code: string, message: string, status: number) {
return NextResponse.json( return NextResponse.json(
{ { success: false, error: { code, message } },
success: false, { status },
error: {
code: 'INVALID_URL',
message: '图片地址不正确',
},
},
{ status: 400 },
) )
} }
if (!['http:', 'https:'].includes(targetUrl.protocol) || !isAllowedImageHost(targetUrl.hostname)) { export async function GET(request: NextRequest) {
return NextResponse.json( let timeout: ReturnType<typeof setTimeout> | null = null
{ let timedOut = false
success: false, try {
error: { const session = await getServerSession(authOptions)
code: 'IMAGE_HOST_NOT_ALLOWED', if (!session) return jsonError('UNAUTHORIZED', '请先登录后再导出图片', 401)
message: '图片来源不允许导出',
},
},
{ status: 400 },
)
}
const rawUrl = request.nextUrl.searchParams.get('url')
if (!rawUrl) return jsonError('MISSING_URL', '缺少图片地址', 400)
const targetUrl = validateTargetUrl(rawUrl)
const { pngBuffer, metadata } = await withImageProcessingSlot(async () => {
// 连下载 Buffer 也纳入整机唯一门闩;否则多个代理请求会在进入 Sharp 前
// 同时各自占用 20MB,跨蓝绿累计仍可能造成内存峰值。
const controller = new AbortController()
timeout = setTimeout(() => {
timedOut = true
controller.abort()
}, UPSTREAM_TIMEOUT_MS)
timeout.unref?.()
const upstream = await fetch(targetUrl.toString(), { const upstream = await fetch(targetUrl.toString(), {
method: 'GET', method: 'GET',
cache: 'no-store', cache: 'no-store',
redirect: 'manual',
signal: controller.signal,
headers: { headers: {
'User-Agent': 'Mozilla/5.0 Codex Taobao Image Proxy', 'User-Agent': 'Mozilla/5.0 Codex Taobao Image Proxy',
Referer: 'https://item.taobao.com/', Referer: 'https://item.taobao.com/',
}, },
}) })
if (upstream.status >= 300 && upstream.status < 400) {
await cancelResponseBody(upstream, 'redirect rejected')
throw new ImageProxyInputError(
'UPSTREAM_REDIRECT_REJECTED',
'上游图片发生重定向,已为安全起见拒绝',
502,
)
}
if (!upstream.ok) { if (!upstream.ok) {
return NextResponse.json( await cancelResponseBody(upstream, 'upstream failed')
{ throw new ImageProxyInputError(
success: false, 'UPSTREAM_IMAGE_FETCH_FAILED',
error: { `图片下载失败,上游返回 ${upstream.status}`,
code: 'UPSTREAM_IMAGE_FETCH_FAILED', 502,
message: `图片下载失败,上游返回 ${upstream.status}`,
},
},
{ status: 502 },
) )
} }
const buffer = Buffer.from(await upstream.arrayBuffer()) const contentType = String(upstream.headers.get('content-type') || '')
const pngBuffer = await sharp(buffer, { failOn: 'none' }) .split(';', 1)[0]
.trim()
.toLowerCase()
if (!ALLOWED_IMAGE_CONTENT_TYPES.has(contentType)) {
await cancelResponseBody(upstream, 'content type rejected')
throw new ImageProxyInputError('UPSTREAM_NOT_AN_IMAGE', '上游返回的不是支持的图片', 415)
}
const buffer = await readBoundedUpstreamImage(upstream)
if (timeout) {
clearTimeout(timeout)
timeout = null
}
const inputOptions = {
failOn: 'error' as const,
sequentialRead: true,
limitInputPixels: MAX_IMAGE_PIXELS,
}
const inputMetadata = await sharp(buffer, inputOptions).metadata()
assertProcessableMetadata(inputMetadata, {
maxInputWidth: MAX_IMAGE_WIDTH,
maxInputHeight: MAX_IMAGE_HEIGHT,
maxInputPixels: MAX_IMAGE_PIXELS,
})
const pngBuffer = await sharp(buffer, inputOptions)
.flatten({ background: '#ffffff' }) .flatten({ background: '#ffffff' })
.png() .png()
.toBuffer() .toBuffer()
const metadata = await sharp(pngBuffer).metadata() if (pngBuffer.length > MAX_OUTPUT_IMAGE_BYTES) {
throw new ImageProxyInputError('OUTPUT_IMAGE_TOO_LARGE', '转换后的图片超过安全大小限制', 413)
}
const metadata = await sharp(pngBuffer, inputOptions).metadata()
return { pngBuffer, metadata }
})
return new NextResponse(Uint8Array.from(pngBuffer), { return new NextResponse(Uint8Array.from(pngBuffer), {
status: 200, status: 200,
@ -113,16 +225,15 @@ export async function GET(request: NextRequest) {
}, },
}) })
} catch (error) { } catch (error) {
if (error instanceof ImageProxyInputError) {
return jsonError(error.code, error.message, error.status)
}
if (timedOut || (error as { name?: string } | undefined)?.name === 'AbortError') {
return jsonError('UPSTREAM_IMAGE_TIMEOUT', '图片下载超时,请稍后再试', 504)
}
console.error('[api/taobao-detail-plus/image-proxy] GET error:', error) console.error('[api/taobao-detail-plus/image-proxy] GET error:', error)
return NextResponse.json( return jsonError('IMAGE_PROXY_ERROR', '图片处理失败,请稍后再试', 500)
{ } finally {
success: false, if (timeout) clearTimeout(timeout)
error: {
code: 'IMAGE_PROXY_ERROR',
message: '图片处理失败,请稍后再试',
},
},
{ status: 500 },
)
} }
} }

View File

@ -5,6 +5,12 @@ import { authOptions } from '@/lib/auth';
import { uploadToS3 } from '@/lib/s3'; import { uploadToS3 } from '@/lib/s3';
import { compressImage, sanitizeFileName } from '@/lib/image-processor'; import { compressImage, sanitizeFileName } from '@/lib/image-processor';
import { randomBytes } from 'crypto'; import { randomBytes } from 'crypto';
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
export const runtime = 'nodejs' export const runtime = 'nodejs'
export const dynamic = 'force-dynamic' export const dynamic = 'force-dynamic'
@ -19,6 +25,7 @@ function generateUniqueFilename(prefix: string, ext: string): string {
// POST - 上传预览图 // POST - 上传预览图
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
@ -28,7 +35,20 @@ export async function POST(request: NextRequest) {
); );
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.previewRequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
file: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.previewFileBytes,
allowedMimeTypes: ['image/*'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.previewFileBytes,
});
const file = formData.get('file') as File; const file = formData.get('file') as File;
if (!file) { if (!file) {
@ -47,7 +67,7 @@ export async function POST(request: NextRequest) {
} }
// 验证文件大小(50MB限制) // 验证文件大小(50MB限制)
if (file.size > 50 * 1024 * 1024) { if (file.size > MULTIPART_ROUTE_LIMITS.previewFileBytes) {
return NextResponse.json( return NextResponse.json(
{ error: '预览图过大,请使用小于50MB的图片' }, { error: '预览图过大,请使用小于50MB的图片' },
{ status: 400 } { status: 400 }
@ -75,9 +95,23 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ url }); return NextResponse.json({ url });
} catch (error) { } catch (error) {
console.error('上传预览图失败:', error); console.error('上传预览图失败:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json( return NextResponse.json(
{ error: '上传失败,请稍后重试' }, { error: '上传失败,请稍后重试' },
{ status: 500 } { status: 500 }
); );
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -16,6 +16,10 @@ import {
import { randomBytes } from 'crypto'; import { randomBytes } from 'crypto';
import { writeFile } from 'fs/promises'; import { writeFile } from 'fs/promises';
import path from 'path'; import path from 'path';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
} from '@/lib/multipart-route-limits';
const MAX_PSD_FILE_BYTES = 128 * 1024 * 1024; const MAX_PSD_FILE_BYTES = 128 * 1024 * 1024;
const MAX_PSD_REQUEST_BYTES = MAX_PSD_FILE_BYTES + 2 * 1024 * 1024; const MAX_PSD_REQUEST_BYTES = MAX_PSD_FILE_BYTES + 2 * 1024 * 1024;
@ -41,6 +45,7 @@ function detectPhotoshopDocumentType(buffer: Buffer): '.psd' | '.psb' | null {
// POST - 上传 PSD 文件到本地 // POST - 上传 PSD 文件到本地
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
const userId = String((session?.user as { id?: string } | undefined)?.id || '').trim(); const userId = String((session?.user as { id?: string } | undefined)?.id || '').trim();
@ -56,6 +61,7 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ error: 'PSD 文件过大' }, { status: 413 }); return NextResponse.json({ error: 'PSD 文件过大' }, { status: 413 });
} }
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
const file = formData.get('file'); const file = formData.get('file');
const materialId = String(formData.get('materialId') || '').trim(); const materialId = String(formData.get('materialId') || '').trim();
@ -138,10 +144,24 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ path: relativePath }); return NextResponse.json({ path: relativePath });
} catch (error) { } catch (error) {
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
console.error('[PSD Upload] Error:', error); console.error('[PSD Upload] Error:', error);
return NextResponse.json( return NextResponse.json(
{ error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') }, { error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') },
{ status: 500 } { status: 500 }
); );
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -2,19 +2,42 @@ import { NextRequest, NextResponse } from 'next/server';
import { getServerSession } from 'next-auth'; import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { uploadToS3 } from '@/lib/s3'; import { uploadToS3 } from '@/lib/s3';
import {
assertBoundedMultipartHeaders,
assertMultipartFiles,
} from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
/** /**
* POST - 上传原图到 S3(不压缩) * POST - 上传原图到 S3(不压缩)
* 专门用于上传需要保留原始质量的图片(如任务原图) * 专门用于上传需要保留原始质量的图片(如任务原图)
*/ */
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }); return NextResponse.json({ error: '请先登录' }, { status: 401 });
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.legacyS3RequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
file: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.legacyS3FileBytes,
allowedMimeTypes: ['image/jpeg', 'image/png', 'image/webp', 'image/tiff', 'image/avif'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.legacyS3FileBytes,
});
const file = formData.get('file') as File; const file = formData.get('file') as File;
const type = formData.get('type') as string; // original-main/original-preview等 const type = formData.get('type') as string; // original-main/original-preview等
@ -23,7 +46,7 @@ export async function POST(request: NextRequest) {
} }
// 文件大小检查(原图最大100MB) // 文件大小检查(原图最大100MB)
if (file.size > 100 * 1024 * 1024) { if (file.size > MULTIPART_ROUTE_LIMITS.legacyS3FileBytes) {
return NextResponse.json( return NextResponse.json(
{ error: `文件过大,请使用小于100MB的图片` }, { error: `文件过大,请使用小于100MB的图片` },
{ status: 400 } { status: 400 }
@ -58,9 +81,23 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ url }); return NextResponse.json({ url });
} catch (error) { } catch (error) {
console.error('[S3 Original Upload] Error:', error); console.error('[S3 Original Upload] Error:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json( return NextResponse.json(
{ error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') }, { error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') },
{ status: 500 } { status: 500 }
); );
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -3,17 +3,37 @@ import { getServerSession } from 'next-auth';
import { authOptions } from '@/lib/auth'; import { authOptions } from '@/lib/auth';
import { uploadToS3 } from '@/lib/s3'; import { uploadToS3 } from '@/lib/s3';
import path from 'path'; import path from 'path';
import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
// POST - 上传文件到 S3 // POST - 上传文件到 S3
// 任务-详情类型:保持原图质量,不做压缩处理,确保素材完整性 // 任务-详情类型:保持原图质量,不做压缩处理,确保素材完整性
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
if (!session) { if (!session) {
return NextResponse.json({ error: '请先登录' }, { status: 401 }); return NextResponse.json({ error: '请先登录' }, { status: 401 });
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.legacyS3RequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
file: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.legacyS3FileBytes,
allowedMimeTypes: ['image/jpeg', 'image/png', 'image/webp', 'image/tiff', 'image/avif'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.legacyS3FileBytes,
});
const file = formData.get('file') as File; const file = formData.get('file') as File;
const type = formData.get('type') as string; // main/sku/detail const type = formData.get('type') as string; // main/sku/detail
@ -22,7 +42,7 @@ export async function POST(request: NextRequest) {
} }
// 文件大小检查(任务-详情允许较大文件,最大100MB) // 文件大小检查(任务-详情允许较大文件,最大100MB)
if (file.size > 100 * 1024 * 1024) { if (file.size > MULTIPART_ROUTE_LIMITS.legacyS3FileBytes) {
return NextResponse.json( return NextResponse.json(
{ error: `图片过大,请使用小于100MB的图片` }, { error: `图片过大,请使用小于100MB的图片` },
{ status: 400 } { status: 400 }
@ -62,9 +82,23 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ url }); return NextResponse.json({ url });
} catch (error) { } catch (error) {
console.error('[S3 Upload] Error:', error); console.error('[S3 Upload] Error:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json( return NextResponse.json(
{ error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') }, { error: '上传失败: ' + (error instanceof Error ? error.message : '未知错误') },
{ status: 500 } { status: 500 }
); );
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -4,11 +4,17 @@ import { authOptions } from '@/lib/auth';
import { prisma } from '@/lib/prisma'; import { prisma } from '@/lib/prisma';
import { uploadToS3 } from '@/lib/s3'; import { uploadToS3 } from '@/lib/s3';
import { normalizeUrl } from '@/lib/utils'; import { normalizeUrl } from '@/lib/utils';
import { compressImage, sanitizeFileName } from '@/lib/image-processor'; import { compressImage } from '@/lib/image-processor';
import path from 'path'; import { assertBoundedMultipartHeaders, assertMultipartFiles } from '@/lib/multipart-request-guard';
import {
boundedMultipartRequestLimiter,
getBoundedMultipartFailure,
MULTIPART_ROUTE_LIMITS,
} from '@/lib/multipart-route-limits';
// POST /api/user/avatar - 上传用户头像 // POST /api/user/avatar - 上传用户头像
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
let releaseRequestSlot: (() => void) | null = null;
try { try {
const session = await getServerSession(authOptions); const session = await getServerSession(authOptions);
const userId = session?.user ? (session.user as any).id : undefined; const userId = session?.user ? (session.user as any).id : undefined;
@ -20,7 +26,20 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ error: '未登录' }, { status: 401 }); return NextResponse.json({ error: '未登录' }, { status: 401 });
} }
assertBoundedMultipartHeaders(request.headers, MULTIPART_ROUTE_LIMITS.avatarRequestBytes);
releaseRequestSlot = await boundedMultipartRequestLimiter.acquire();
const formData = await request.formData(); const formData = await request.formData();
assertMultipartFiles(formData, {
fields: {
avatar: {
maxCount: 1,
maxFileBytes: MULTIPART_ROUTE_LIMITS.avatarFileBytes,
allowedMimeTypes: ['image/jpeg', 'image/png', 'image/webp'],
},
},
maxFiles: 1,
maxTotalBytes: MULTIPART_ROUTE_LIMITS.avatarFileBytes,
});
const avatar = formData.get('avatar') as File | null; const avatar = formData.get('avatar') as File | null;
if (!avatar) { if (!avatar) {
@ -39,7 +58,7 @@ export async function POST(request: NextRequest) {
} }
// 验证文件大小(5MB) // 验证文件大小(5MB)
if (avatar.size > 5 * 1024 * 1024) { if (avatar.size > MULTIPART_ROUTE_LIMITS.avatarFileBytes) {
return NextResponse.json( return NextResponse.json(
{ error: `头像过大,请使用小于5MB的图片(当前:${(avatar.size / 1024 / 1024).toFixed(2)}MB)` }, { error: `头像过大,请使用小于5MB的图片(当前:${(avatar.size / 1024 / 1024).toFixed(2)}MB)` },
{ status: 400 } { status: 400 }
@ -56,9 +75,7 @@ export async function POST(request: NextRequest) {
}); });
// 上传到S3 // 上传到S3
const sanitizedName = sanitizeFileName(avatar.name); const fileName = `avatar-${userId}-${Date.now()}.webp`;
const ext = path.extname(sanitizedName) || '.webp';
const fileName = `avatar-${userId}-${Date.now()}${ext}`;
console.log('[UserAvatar] Uploading to S3:', fileName, `(${(compressedBuffer.length / 1024).toFixed(2)}KB)`); console.log('[UserAvatar] Uploading to S3:', fileName, `(${(compressedBuffer.length / 1024).toFixed(2)}KB)`);
@ -81,6 +98,20 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ avatarUrl: normalizedUrl }); return NextResponse.json({ avatarUrl: normalizedUrl });
} catch (error) { } catch (error) {
console.error('[UserAvatar] POST error:', error); console.error('[UserAvatar] POST error:', error);
const boundedFailure = getBoundedMultipartFailure(error);
if (boundedFailure) {
return NextResponse.json(
{ error: boundedFailure.message, code: boundedFailure.code },
{
status: boundedFailure.status,
...(boundedFailure.retryAfterSeconds
? { headers: { 'Retry-After': String(boundedFailure.retryAfterSeconds) } }
: {}),
},
);
}
return NextResponse.json({ error: '上传头像失败' }, { status: 500 }); return NextResponse.json({ error: '上传头像失败' }, { status: 500 });
} finally {
releaseRequestSlot?.();
} }
} }

View File

@ -3,16 +3,15 @@
import { useEffect, useState, Suspense } from 'react' import { useEffect, useState, Suspense } from 'react'
import { useRouter, useSearchParams } from 'next/navigation' import { useRouter, useSearchParams } from 'next/navigation'
import Image from 'next/image' import Image from 'next/image'
import BrandLogo from '@/components/BrandLogo'
import { parseApiError, parseFetchError } from '@/lib/error-handler' import { parseApiError, parseFetchError } from '@/lib/error-handler'
import { DEFAULT_LOGIN_LOGO_URL, DEFAULT_SITE_NAME } from '@/lib/brand-assets'
interface Department { interface Department {
id: string id: string
name: string name: string
} }
const DEFAULT_LOGIN_LOGO_URL = '/api/favicon'
const DEFAULT_SITE_NAME = '由壹点3D素材'
function CompleteProfileContent() { function CompleteProfileContent() {
const router = useRouter() const router = useRouter()
const searchParams = useSearchParams() const searchParams = useSearchParams()
@ -174,8 +173,9 @@ function CompleteProfileContent() {
{/* Logo */} {/* Logo */}
<div className="text-center mb-8"> <div className="text-center mb-8">
<div className="flex justify-center mb-4"> <div className="flex justify-center mb-4">
<Image <BrandLogo
src={loginLogoUrl} src={loginLogoUrl}
fallbackSrc={DEFAULT_LOGIN_LOGO_URL}
alt={siteName} alt={siteName}
width={80} width={80}
height={80} height={80}

View File

@ -2,15 +2,13 @@
import { useState, useEffect } from 'react' import { useState, useEffect } from 'react'
import { signIn } from 'next-auth/react' import { signIn } from 'next-auth/react'
import Image from 'next/image'
import Link from 'next/link' import Link from 'next/link'
import { ensureHttpsUrl } from '@/lib/url-rewrite' import BrandLogo from '@/components/BrandLogo'
import { DEFAULT_LOGIN_LOGO_URL } from '@/lib/brand-assets'
import { parseApiError } from '@/lib/error-handler' import { parseApiError } from '@/lib/error-handler'
import { getPublicAppOrigin } from '@/lib/public-app-url' import { getPublicAppOrigin } from '@/lib/public-app-url'
import { normalizeInternalRedirectPath } from '@/lib/safe-redirect' import { normalizeInternalRedirectPath } from '@/lib/safe-redirect'
const DEFAULT_LOGIN_LOGO_URL = '/api/favicon'
export default function LoginPage() { export default function LoginPage() {
const [email, setEmail] = useState('') const [email, setEmail] = useState('')
const [password, setPassword] = useState('') const [password, setPassword] = useState('')
@ -58,16 +56,13 @@ export default function LoginPage() {
const response = await fetch('/api/system-settings') const response = await fetch('/api/system-settings')
if (response.ok) { if (response.ok) {
const settings = await response.json() const settings = await response.json()
// 确保 logo URL 使用 HTTPS,避免混合内容问题
const logoUrl = settings.loginLogoUrl || DEFAULT_LOGIN_LOGO_URL const logoUrl = settings.loginLogoUrl || DEFAULT_LOGIN_LOGO_URL
setLoginLogoUrl(ensureHttpsUrl(logoUrl) || logoUrl) setLoginLogoUrl(logoUrl)
setSiteName(settings.siteName || '由壹点3D素材') setSiteName(settings.siteName || '由壹点3D素材')
} }
} catch (error) { } catch (error) {
console.error('加载系统设置失败:', error) console.error('加载系统设置失败:', error)
// 即使加载失败,也确保默认 URL 使用 HTTPS setLoginLogoUrl(DEFAULT_LOGIN_LOGO_URL)
const defaultUrl = DEFAULT_LOGIN_LOGO_URL
setLoginLogoUrl(ensureHttpsUrl(defaultUrl) || defaultUrl)
} }
} }
loadSettings() loadSettings()
@ -284,9 +279,10 @@ export default function LoginPage() {
{/* Logo */} {/* Logo */}
<div className="text-center mb-8"> <div className="text-center mb-8">
<div className="flex justify-center mb-4"> <div className="flex justify-center mb-4">
<Image <BrandLogo
src={loginLogoUrl} src={loginLogoUrl}
alt={siteName} fallbackSrc={DEFAULT_LOGIN_LOGO_URL}
alt=""
width={80} width={80}
height={80} height={80}
unoptimized unoptimized

View File

@ -13,7 +13,11 @@ import { useToast } from '@/hooks/useToast'
import { parseDownloadError } from '@/lib/download-errors' import { parseDownloadError } from '@/lib/download-errors'
import { validateAndTriggerDownload } from '@/lib/client-download' import { validateAndTriggerDownload } from '@/lib/client-download'
import { parseApiError, handleError, parseFetchError } from '@/lib/error-handler' import { parseApiError, handleError, parseFetchError } from '@/lib/error-handler'
import { normalizeVideoMaterialType, isVideoMaterialType as isVideoMaterialTypeValue } from '@/lib/video-material-types' import {
normalizeVideoMaterialType,
isVideoMaterialType as isVideoMaterialTypeValue,
shouldPollVideoTranscode,
} from '@/lib/video-material-types'
import { buildPublicMaterialShareUrl } from '@/lib/public-app-url' import { buildPublicMaterialShareUrl } from '@/lib/public-app-url'
import { normalizeMaterialPreviewUrl, pickFirstMaterialPreviewUrl } from '@/lib/material-preview-url' import { normalizeMaterialPreviewUrl, pickFirstMaterialPreviewUrl } from '@/lib/material-preview-url'
// 首页拆分组件 // 首页拆分组件
@ -127,7 +131,6 @@ export default function Home() {
const [showVideoModal, setShowVideoModal] = useState(false) const [showVideoModal, setShowVideoModal] = useState(false)
const [videoSrc, setVideoSrc] = useState<string | null>(null) const [videoSrc, setVideoSrc] = useState<string | null>(null)
const transcodePollTimers = useRef<Map<string, NodeJS.Timeout>>(new Map()) const transcodePollTimers = useRef<Map<string, NodeJS.Timeout>>(new Map())
const transcodeStartTimes = useRef<Map<string, number>>(new Map())
const transcodePollingFlags = useRef<Map<string, boolean>>(new Map()) // 标记是否正在轮询 const transcodePollingFlags = useRef<Map<string, boolean>>(new Map()) // 标记是否正在轮询
const fetchMaterialsRef = useRef<typeof fetchMaterials | null>(null) // 存储fetchMaterials函数引用 const fetchMaterialsRef = useRef<typeof fetchMaterials | null>(null) // 存储fetchMaterials函数引用
const materialsRequestIdRef = useRef(0) const materialsRequestIdRef = useRef(0)
@ -135,55 +138,19 @@ export default function Home() {
const facetsRequestIdRef = useRef(0) const facetsRequestIdRef = useRef(0)
const facetsAbortRef = useRef<AbortController | null>(null) const facetsAbortRef = useRef<AbortController | null>(null)
// ✅ 清理转码Map的函数(直接操作Map,避免循环依赖) // 排队和单线程转码可能超过30分钟,不能按固定时长误停正常轮询。
const cleanupTranscodeMaps = useCallback(() => { // done/failed 会主动停止;素材离开当前列表时立即清理对应定时器。
const now = Date.now() useEffect(() => {
const MAX_AGE = 30 * 60 * 1000 // 30分钟
let cleanedCount = 0
// 清理超过30分钟的条目
transcodeStartTimes.current.forEach((startTime, materialId) => {
if (now - startTime > MAX_AGE) {
const timer = transcodePollTimers.current.get(materialId)
if (timer) {
clearTimeout(timer)
transcodePollTimers.current.delete(materialId)
}
transcodePollingFlags.current.delete(materialId)
transcodeStartTimes.current.delete(materialId)
cleanedCount++
}
})
// 清理不在当前素材列表中的条目
const currentMaterialIds = new Set(materials.map(m => m.id)) const currentMaterialIds = new Set(materials.map(m => m.id))
transcodePollTimers.current.forEach((timer, materialId) => { transcodePollTimers.current.forEach((timer, materialId) => {
if (!currentMaterialIds.has(materialId)) { if (!currentMaterialIds.has(materialId)) {
clearTimeout(timer) clearTimeout(timer)
transcodePollTimers.current.delete(materialId) transcodePollTimers.current.delete(materialId)
transcodePollingFlags.current.delete(materialId) transcodePollingFlags.current.delete(materialId)
transcodeStartTimes.current.delete(materialId)
cleanedCount++
} }
}) })
if (cleanedCount > 0) {
console.log(`[TranscodePoll] Cleaned up ${cleanedCount} old entries. Current size: ${transcodePollTimers.current.size}`)
}
}, [materials]) }, [materials])
// ✅ 定期清理转码Map(每10分钟)
useEffect(() => {
const interval = setInterval(() => {
if (transcodePollTimers.current.size > 0) {
console.log(`[TranscodePoll] Periodic cleanup check. Current size: ${transcodePollTimers.current.size}`)
cleanupTranscodeMaps()
}
}, 10 * 60 * 1000) // 10分钟
return () => clearInterval(interval)
}, [cleanupTranscodeMaps])
// 音频播放状态 // 音频播放状态
const audioRef = useRef<HTMLAudioElement | null>(null) const audioRef = useRef<HTMLAudioElement | null>(null)
const [playingAudioId, setPlayingAudioId] = useState<string | null>(null) const [playingAudioId, setPlayingAudioId] = useState<string | null>(null)
@ -644,9 +611,10 @@ export default function Home() {
setCurrentMaterialsTotalCount(data.total ?? data.totalCount ?? materialsList.length) setCurrentMaterialsTotalCount(data.total ?? data.totalCount ?? materialsList.length)
// 对需要转码的视频素材启动轮询 // 对需要转码的视频素材启动轮询
appendedItems.forEach((material) => { appendedItems.forEach((material, index) => {
if (isVideoMaterial(material.materialType) && material.videoSourceUrl) { if (shouldPollVideoTranscode(material.materialType, material.videoProcessingStatus)) {
startTranscodePoll(material, 0) // 避免一页多个视频在同一毫秒冲击状态接口。
startTranscodePoll(material, Math.min(index * 250, 3000))
} }
}) })
@ -688,7 +656,6 @@ export default function Home() {
transcodePollTimers.current.delete(materialId) transcodePollTimers.current.delete(materialId)
} }
transcodePollingFlags.current.delete(materialId) transcodePollingFlags.current.delete(materialId)
transcodeStartTimes.current.delete(materialId)
}, []) }, [])
// 启动转码轮询 // 启动转码轮询
@ -722,7 +689,16 @@ export default function Home() {
if (!response.ok) { if (!response.ok) {
const errorData = await response.json().catch(() => ({})) const errorData = await response.json().catch(() => ({}))
console.error(`[TranscodePoll] GET request failed for material ${materialId}:`, response.status, errorData) console.error(`[TranscodePoll] GET request failed for material ${materialId}:`, response.status, errorData)
if (response.status === 400) { if (response.status === 429) {
const retryAfterSeconds = Number(response.headers.get('Retry-After') || '15')
const retryDelay = Number.isFinite(retryAfterSeconds)
? Math.max(5000, Math.min(retryAfterSeconds * 1000, 60000))
: 15000
const timer = setTimeout(poll, retryDelay)
transcodePollTimers.current.set(materialId, timer)
return
}
if ([400, 401, 403, 404].includes(response.status)) {
stopTranscodePoll(materialId) stopTranscodePoll(materialId)
return return
} }
@ -730,15 +706,6 @@ export default function Home() {
} }
const data = await response.json() const data = await response.json()
const isProcessing = data.queue?.processingIds?.includes(materialId) ?? false
// 记录开始处理时间
if (isProcessing && !transcodeStartTimes.current.get(materialId)) {
transcodeStartTimes.current.set(materialId, Date.now())
}
if (!isProcessing && transcodeStartTimes.current.get(materialId)) {
transcodeStartTimes.current.delete(materialId)
}
// 更新素材状态 // 更新素材状态
setMaterials((prev) => setMaterials((prev) =>
@ -771,24 +738,6 @@ export default function Home() {
fetchMaterialsRef.current?.(1, false) fetchMaterialsRef.current?.(1, false)
} }
// 检查队列是否为空且状态为processing(可能是异常情况,尝试重试)
const queueEmpty = (data.queue?.processingIds?.length ?? 0) === 0 && (data.queue?.waitingTotal ?? 0) === 0
if (data.status === 'processing' && queueEmpty) {
const retryCount = transcodeStartTimes.current.get(materialId) ? 1 : 0
if (retryCount < 2) {
// 静默重试
try {
await fetch('/api/materials/transcode', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ materialId, forceRetry: true }),
})
} catch (e) {
// 静默失败
}
}
}
// 根据状态决定是否继续轮询 // 根据状态决定是否继续轮询
if (data.status === 'done') { if (data.status === 'done') {
stopTranscodePoll(materialId) stopTranscodePoll(materialId)
@ -798,8 +747,8 @@ export default function Home() {
} else if (data.status === 'failed') { } else if (data.status === 'failed') {
stopTranscodePoll(materialId) stopTranscodePoll(materialId)
} else { } else {
// 继续轮询,5秒后再次检查 // 转码是分钟级后台任务;15秒轮询既能及时刷新,也不会让一页多卡片打爆状态接口。
const timer = setTimeout(poll, 5000) const timer = setTimeout(poll, 15000)
transcodePollTimers.current.set(materialId, timer) transcodePollTimers.current.set(materialId, timer)
} }
} catch (error) { } catch (error) {
@ -816,17 +765,18 @@ export default function Home() {
// 延迟启动轮询 // 延迟启动轮询
const timer = setTimeout(poll, delay) const timer = setTimeout(poll, delay)
transcodePollTimers.current.set(materialId, timer) transcodePollTimers.current.set(materialId, timer)
}, [stopTranscodePoll]) }, [isVideoMaterial, stopTranscodePoll])
// 组件卸载时清理所有转码轮询 // 组件卸载时清理所有转码轮询
useEffect(() => { useEffect(() => {
const pollTimers = transcodePollTimers.current
const pollingFlags = transcodePollingFlags.current
return () => { return () => {
transcodePollTimers.current.forEach((timer) => { pollTimers.forEach((timer) => {
clearTimeout(timer) clearTimeout(timer)
}) })
transcodePollTimers.current.clear() pollTimers.clear()
transcodePollingFlags.current.clear() pollingFlags.clear()
transcodeStartTimes.current.clear()
} }
}, []) }, [])

View File

@ -0,0 +1,21 @@
import { readFileSync } from 'fs'
import path from 'path'
describe('首页视频转码状态轮询边界', () => {
test('轮询只读状态,不得静默触发强制重试写请求', () => {
const source = readFileSync(path.join(process.cwd(), 'app/page.tsx'), 'utf8')
const pollBody = source.split('const poll = async () => {')[1]?.split('// 标记为正在轮询')[0]
expect(pollBody).toBeTruthy()
expect(pollBody).not.toContain("method: 'POST'")
expect(pollBody).not.toContain('forceRetry')
})
test('不能用固定30分钟上限停止仍在排队或处理中的视频', () => {
const source = readFileSync(path.join(process.cwd(), 'app/page.tsx'), 'utf8')
expect(source).not.toContain('30 * 60 * 1000')
expect(source).not.toContain('transcodeStartTimes')
expect(source).toContain('done/failed 会主动停止')
})
})

View File

@ -1,5 +1,7 @@
'use client' 'use client'
// LEGACY_READ_ONLY: 旧域名仅用于展示历史考核素材。
import { useCallback, useEffect, useMemo, useRef, useState, type CSSProperties, type ReactNode } from 'react' import { useCallback, useEffect, useMemo, useRef, useState, type CSSProperties, type ReactNode } from 'react'
import Image from 'next/image' import Image from 'next/image'
import Link from 'next/link' import Link from 'next/link'

View File

@ -2,9 +2,10 @@
import { useState, useEffect } from 'react' import { useState, useEffect } from 'react'
import { useRouter } from 'next/navigation' import { useRouter } from 'next/navigation'
import Image from 'next/image'
import Link from 'next/link' import Link from 'next/link'
import DepartmentTreeSelect from '@/components/DepartmentTreeSelect' import DepartmentTreeSelect from '@/components/DepartmentTreeSelect'
import BrandLogo from '@/components/BrandLogo'
import { DEFAULT_LOGIN_LOGO_URL } from '@/lib/brand-assets'
import { parseFetchError, parseApiError } from '@/lib/error-handler' import { parseFetchError, parseApiError } from '@/lib/error-handler'
export default function RegisterPage() { export default function RegisterPage() {
@ -81,9 +82,10 @@ export default function RegisterPage() {
{/* Logo */} {/* Logo */}
<div className="text-center mb-8"> <div className="text-center mb-8">
<div className="flex justify-center mb-4"> <div className="flex justify-center mb-4">
<Image <BrandLogo
src="/api/favicon" src={DEFAULT_LOGIN_LOGO_URL}
alt="由壹点3D素材" fallbackSrc={DEFAULT_LOGIN_LOGO_URL}
alt=""
width={80} width={80}
height={80} height={80}
unoptimized unoptimized

View File

@ -0,0 +1,60 @@
import { fireEvent, render } from '@testing-library/react'
import BrandLogo from '@/components/BrandLogo'
describe('BrandLogo', () => {
test('远端失败只回退一次,本地也失败后显示稳定文字且不再生成图片', () => {
const handleError = jest.fn()
const { container } = render(
<BrandLogo
src="https://s3.jiansun.vip/gongxiangpan/brand/missing.png"
fallbackSrc="/brand/header-logo.png"
fallbackText="由壹点"
alt=""
width={36}
height={36}
unoptimized
onError={handleError}
/>,
)
const remoteImage = container.querySelector('img')
expect(remoteImage).toHaveAttribute('alt', '')
fireEvent.error(remoteImage!)
const localImage = container.querySelector('img')
expect(localImage?.getAttribute('src')).toMatch(/\/brand\/header-logo\.png$/)
fireEvent.error(localImage!)
expect(container.querySelector('img')).not.toBeInTheDocument()
expect(container.querySelector('[data-brand-logo-fallback="text"]')).toHaveTextContent('由壹点')
expect(handleError).toHaveBeenCalledTimes(2)
})
test('传入地址变化后会重新尝试新地址', () => {
const { container, rerender } = render(
<BrandLogo
src="/brand/header-logo.png"
fallbackSrc="/brand/header-logo.png"
alt="品牌"
width={36}
height={36}
unoptimized
/>,
)
fireEvent.error(container.querySelector('img')!)
expect(container.querySelector('[data-brand-logo-fallback="text"]')).toBeInTheDocument()
rerender(
<BrandLogo
src="/brand/login-logo.png"
fallbackSrc="/brand/header-logo.png"
alt="品牌"
width={36}
height={36}
unoptimized
/>,
)
expect(container.querySelector('img')?.getAttribute('src')).toMatch(/\/brand\/login-logo\.png$/)
})
})

87
components/BrandLogo.tsx Normal file
View File

@ -0,0 +1,87 @@
'use client'
import Image, { type ImageProps } from 'next/image'
import { useEffect, useState, type SyntheticEvent } from 'react'
import {
BRAND_LOGO_FALLBACK_TEXT,
DEFAULT_HEADER_LOGO_URL,
} from '@/lib/brand-assets'
type BrandLogoProps = Omit<ImageProps, 'src' | 'alt' | 'onError'> & {
src?: string | null
alt?: string
fallbackSrc?: string
fallbackText?: string
onError?: (event: SyntheticEvent<HTMLImageElement>) => void
}
function resolveInitialSource(src: string | null | undefined, fallbackSrc: string) {
return typeof src === 'string' && src.trim() ? src.trim() : fallbackSrc
}
/**
* 品牌 Logo 的单次回退组件:远端失败后只切换一次本地资源;
* 如果本地资源也失败,则显示稳定的文字标记,避免 onError 无限循环。
*/
export default function BrandLogo({
src,
alt = '',
fallbackSrc = DEFAULT_HEADER_LOGO_URL,
fallbackText = BRAND_LOGO_FALLBACK_TEXT,
onError,
className,
style,
width,
height,
...imageProps
}: BrandLogoProps) {
const initialSource = resolveInitialSource(src, fallbackSrc)
const [currentSource, setCurrentSource] = useState(initialSource)
const [showTextFallback, setShowTextFallback] = useState(false)
useEffect(() => {
setCurrentSource(initialSource)
setShowTextFallback(false)
}, [initialSource])
const handleImageError = (event: SyntheticEvent<HTMLImageElement>) => {
onError?.(event)
if (currentSource !== fallbackSrc) {
setCurrentSource(fallbackSrc)
return
}
setShowTextFallback(true)
}
if (showTextFallback) {
return (
<span
data-brand-logo-fallback="text"
role={alt ? 'img' : undefined}
aria-label={alt || undefined}
aria-hidden={alt ? undefined : true}
className={`inline-flex shrink-0 items-center justify-center overflow-hidden bg-white text-center text-[9px] font-semibold leading-tight text-gray-900 ring-1 ring-black/10 ${className || ''}`}
style={{
width: typeof width === 'number' ? width : undefined,
height: typeof height === 'number' ? height : undefined,
...style,
}}
>
{fallbackText}
</span>
)
}
return (
<Image
{...imageProps}
src={currentSource}
alt={alt}
width={width}
height={height}
className={className}
style={style}
onError={handleImageError}
/>
)
}

View File

@ -10,14 +10,14 @@ import { initializeNetwork, setupNetworkChangeDetection, addNetworkChangeListene
import UpdateLogModal, { useUpdateLog, showUpdateLog } from '@/components/UpdateLogModal' import UpdateLogModal, { useUpdateLog, showUpdateLog } from '@/components/UpdateLogModal'
import MaintenanceBanner from '@/components/MaintenanceBanner' import MaintenanceBanner from '@/components/MaintenanceBanner'
import HeaderTaobaoSearch from '@/components/HeaderTaobaoSearch' import HeaderTaobaoSearch from '@/components/HeaderTaobaoSearch'
import BrandLogo from '@/components/BrandLogo'
import { formatHumanSpeed, formatTransferProgressSummary } from '@/lib/human-readable-transfer' import { formatHumanSpeed, formatTransferProgressSummary } from '@/lib/human-readable-transfer'
import { DEFAULT_HEADER_LOGO_URL } from '@/lib/brand-assets'
interface HeaderProps { interface HeaderProps {
session: Session | null session: Session | null
} }
const DEFAULT_HEADER_LOGO_URL = '/api/favicon'
export default function Header({ session }: HeaderProps) { export default function Header({ session }: HeaderProps) {
const router = useRouter() const router = useRouter()
const role = session ? String((session.user as any)?.role || '') : '' const role = session ? String((session.user as any)?.role || '') : ''
@ -240,9 +240,10 @@ export default function Header({ session }: HeaderProps) {
<div className="flex items-center justify-between"> <div className="flex items-center justify-between">
{/* Logo */} {/* Logo */}
<Link href="/" onClick={handleLogoHomeClick} className="flex min-w-0 items-center space-x-2 sm:space-x-3"> <Link href="/" onClick={handleLogoHomeClick} className="flex min-w-0 items-center space-x-2 sm:space-x-3">
<Image <BrandLogo
src={headerLogoUrl} src={headerLogoUrl}
alt={siteName} fallbackSrc={DEFAULT_HEADER_LOGO_URL}
alt=""
width={36} width={36}
height={36} height={36}
unoptimized unoptimized

View File

@ -1,5 +1,5 @@
import { createRef } from 'react' import { createRef } from 'react'
import { render, screen } from '@testing-library/react' import { fireEvent, render, screen } from '@testing-library/react'
import MaterialCard, { type Material } from '@/components/home/MaterialCard' import MaterialCard, { type Material } from '@/components/home/MaterialCard'
@ -119,4 +119,65 @@ describe('MaterialCard', () => {
expect(screen.getByText('未设置')).toBeInTheDocument() expect(screen.getByText('未设置')).toBeInTheDocument()
}) })
test('15天策略隐藏源地址时仍显示视频转码进度', () => {
render(
<MaterialCard
{...defaultProps}
material={{
...baseMaterial,
materialType: '淘系-逛逛视频',
previewUrl: '',
videoSourceUrl: undefined,
videoProcessingStatus: 'pending',
videoPreviewBlocked: true,
videoPreviewRemainingDays: 15,
}}
/>,
)
expect(screen.getByRole('status')).toHaveTextContent('等待中')
expect(screen.getByRole('progressbar')).toHaveAttribute('aria-valuenow', '5')
expect(screen.getByText('15天后可查看详情')).toBeInTheDocument()
})
test('接口未返回并发上限时按当前单任务策略展示', () => {
render(
<MaterialCard
{...defaultProps}
material={{
...baseMaterial,
materialType: '淘系-逛逛视频',
previewUrl: '',
videoProcessingStatus: 'processing',
transcodeQueueProcessingIds: ['material-1'],
transcodeQueueWaitingTotal: 1,
}}
/>,
)
expect(screen.getByRole('status')).toHaveTextContent('1/1并发')
})
test('15天遮罩下的转码失败仍可见且可以重试', () => {
render(
<MaterialCard
{...defaultProps}
material={{
...baseMaterial,
materialType: '抖音-千川素材',
previewUrl: '',
videoSourceUrl: undefined,
videoProcessingStatus: 'failed',
transcodeLastError: '转码失败:源文件暂时不可读',
videoPreviewBlocked: true,
videoPreviewRemainingDays: 15,
}}
/>,
)
expect(screen.getByRole('status')).toHaveTextContent('转码失败:源文件暂时不可读')
fireEvent.click(screen.getByRole('button', { name: '重试转码' }))
expect(defaultProps.onRetryTranscode).toHaveBeenCalledWith('material-1')
})
}) })

View File

@ -278,11 +278,11 @@ export default function MaterialCard({
const queueIndex = material.transcodeQueueWaitingIndex const queueIndex = material.transcodeQueueWaitingIndex
const queueTotal = material.transcodeQueueWaitingTotal || 0 const queueTotal = material.transcodeQueueWaitingTotal || 0
const processingIds = material.transcodeQueueProcessingIds || [] const processingIds = material.transcodeQueueProcessingIds || []
const maxConcurrent = material.transcodeQueueMaxConcurrent || 2 const maxConcurrent = material.transcodeQueueMaxConcurrent || 1
const isProcessing = processingIds.includes(material.id) const isProcessing = processingIds.includes(material.id)
// 如果转码完成或没有视频源,不显示进度条 // 15天策略会隐藏 videoSourceUrl;进度只能依据公开的处理状态判断。
if (!material.videoSourceUrl || status === 'done' || (status !== 'pending' && status !== 'processing' && status !== 'failed' && attempts === 0)) { if (status === 'done' || (status !== 'pending' && status !== 'processing' && status !== 'failed')) {
return null return null
} }
@ -329,7 +329,12 @@ export default function MaterialCard({
} }
return ( return (
<div className="absolute bottom-0 left-0 right-0 bg-black/80 text-white text-xs p-2 z-20"> <div
className="absolute bottom-0 left-0 right-0 z-40 bg-black/85 p-2 text-xs text-white"
role="status"
aria-live={status === 'failed' ? 'assertive' : 'polite'}
aria-atomic="true"
>
<div className="flex items-center justify-between mb-1"> <div className="flex items-center justify-between mb-1">
<span className="truncate">{label}</span> <span className="truncate">{label}</span>
{status === 'failed' && !material.videoPreviewClipUrl && !material.videoPreview540Url && ( {status === 'failed' && !material.videoPreviewClipUrl && !material.videoPreview540Url && (
@ -344,7 +349,14 @@ export default function MaterialCard({
</button> </button>
)} )}
</div> </div>
<div className="w-full bg-gray-700 rounded-full h-1.5 overflow-hidden"> <div
className="h-1.5 w-full overflow-hidden rounded-full bg-gray-700"
role="progressbar"
aria-label={label}
aria-valuemin={0}
aria-valuemax={100}
aria-valuenow={progressValue}
>
<div <div
className={`h-full transition-all duration-300 ${bgColor}`} className={`h-full transition-all duration-300 ${bgColor}`}
style={{ width: `${progressValue}%` }} style={{ width: `${progressValue}%` }}
@ -549,7 +561,13 @@ export default function MaterialCard({
</> </>
) : ( ) : (
<div className="w-full h-full flex items-center justify-center bg-gray-300 dark:bg-gray-600 audio-placeholder"> <div className="w-full h-full flex items-center justify-center bg-gray-300 dark:bg-gray-600 audio-placeholder">
<span className="text-gray-500 dark:text-gray-400">暂无预览图</span> <span className="text-gray-500 dark:text-gray-400">
{material.videoProcessingStatus === 'failed'
? '预览生成失败'
: material.videoProcessingStatus === 'pending' || material.videoProcessingStatus === 'processing'
? '预览生成中'
: '暂无预览图'}
</span>
</div> </div>
)} )}

View File

@ -0,0 +1,314 @@
/** @jest-environment jsdom */
import { chunkUpload } from '@/lib/chunk-upload'
import {
completeBatchUpload,
uploadBatch3D,
uploadBatchBrandVI,
uploadBatchMusic,
} from './special-batch-uploaders'
jest.mock('@/lib/chunk-upload', () => ({
chunkUpload: jest.fn(),
}))
type FileItem = { file: File; productModel: string }
type BatchResult = { productModel: string; status: string }
type RunBatch = (files: FileItem[]) => Promise<BatchResult[]>
function flushMicrotasks(count = 10) {
return (async () => {
for (let index = 0; index < count; index += 1) {
await Promise.resolve()
}
})()
}
async function assertBoundedFilePool(
runBatch: RunBatch,
expectedStatus = 'processing',
expectedConcurrency = 3,
) {
const files = Array.from({ length: 10 }, (_, index) => ({
file: new File([`file-${index}`], `file-${index}.bin`, { type: 'application/octet-stream' }),
productModel: `SPU-${index}`,
}))
let chunkCallCount = 0
;(chunkUpload as jest.Mock).mockImplementation(async () => ({
uploadId: `upload-${chunkCallCount++}`,
confirmMaterialCreated: jest.fn().mockResolvedValue(undefined),
}))
let activeCompleteRequests = 0
let maxActiveCompleteRequests = 0
let completeCallCount = 0
const materialCreateUploadIds: string[] = []
const pendingCompleteResponses: Array<{
uploadId: string
resolve: (response: unknown) => void
}> = []
const fetchMock = jest.fn((url: string, init?: RequestInit) => {
if (url === '/api/materials/upload/complete') {
const body = JSON.parse(String(init?.body)) as { uploadId: string }
completeCallCount += 1
activeCompleteRequests += 1
maxActiveCompleteRequests = Math.max(maxActiveCompleteRequests, activeCompleteRequests)
return new Promise((resolve) => {
pendingCompleteResponses.push({
uploadId: body.uploadId,
resolve: (response) => {
activeCompleteRequests -= 1
resolve(response)
},
})
})
}
if (url === '/api/materials/upload') {
materialCreateUploadIds.push(String((init?.body as FormData).get('uploadId')))
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({ material: { id: 'material-id' } }),
})
}
return Promise.reject(new Error(`unexpected URL: ${url}`))
})
global.fetch = fetchMock as unknown as typeof fetch
const batchPromise = runBatch(files)
await flushMicrotasks(20)
expect(chunkUpload).toHaveBeenCalledTimes(expectedConcurrency)
expect(completeCallCount).toBe(expectedConcurrency)
expect(maxActiveCompleteRequests).toBeLessThanOrEqual(expectedConcurrency)
let safety = 0
while ((completeCallCount < files.length || pendingCompleteResponses.length > 0) && safety < 100) {
const pending = pendingCompleteResponses.shift()
if (pending) {
pending.resolve({
ok: true,
status: 200,
json: async () => ({
existingFilePath: `uploads/${pending.uploadId}.bin`,
fileName: `${pending.uploadId}.bin`,
}),
})
}
await flushMicrotasks(10)
safety += 1
}
expect(completeCallCount).toBe(files.length)
const results = await batchPromise
expect(results).toHaveLength(files.length)
expect(results.map((result) => result.productModel)).toEqual(files.map((file) => file.productModel))
expect(results.every((result) => result.status === expectedStatus)).toBe(true)
expect(activeCompleteRequests).toBe(0)
expect(maxActiveCompleteRequests).toBe(expectedConcurrency)
expect(materialCreateUploadIds).toEqual(
Array.from({ length: files.length }, (_, index) => `upload-${index}`),
)
}
describe('特殊批量上传的文件级并发池', () => {
const originalFetch = global.fetch
let consoleLog: jest.SpyInstance
let consoleWarn: jest.SpyInstance
let consoleError: jest.SpyInstance
beforeEach(() => {
jest.clearAllMocks()
consoleLog = jest.spyOn(console, 'log').mockImplementation(() => undefined)
consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => undefined)
consoleError = jest.spyOn(console, 'error').mockImplementation(() => undefined)
})
afterEach(() => {
global.fetch = originalFetch
consoleLog.mockRestore()
consoleWarn.mockRestore()
consoleError.mockRestore()
})
test('音乐 10 个文件最多同时处理 3 个 complete 请求', async () => {
await assertBoundedFilePool((files) => uploadBatchMusic(files, '', '', jest.fn()))
})
test('品牌 VI 10 个文件严格逐个处理,避免 PDF 解码与服务端排队叠加', async () => {
await assertBoundedFilePool(
(files) => uploadBatchBrandVI(files, '', jest.fn()),
'processing',
1,
)
})
test('3D 场景 10 个文件也最多同时处理 3 个 complete 请求', async () => {
await assertBoundedFilePool(
(files) => uploadBatch3D(files, '', '', '', '', jest.fn()),
'success',
)
})
test('complete 遇到 503 时用同一 uploadId 重试并恢复', async () => {
const completeBodies: Array<{ uploadId: string; originalFileName: string }> = []
const fetchMock = jest.fn((url: string, init?: RequestInit) => {
if (url === '/api/materials/upload/complete') {
completeBodies.push(JSON.parse(String(init?.body)))
if (completeBodies.length === 1) {
return Promise.resolve({
ok: false,
status: 503,
headers: { get: () => '0' },
text: async () => 'temporarily busy',
})
}
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({
existingFilePath: 'uploads/recovered.mp3',
fileName: 'recovered.mp3',
}),
})
}
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({ material: { id: 'material-recovered' } }),
})
})
global.fetch = fetchMock as unknown as typeof fetch
const result = await completeBatchUpload('upload-same-id', {
materialType: '音乐',
productModel: 'SPU-RECOVER',
fileName: 'song.mp3',
})
expect(result.material.id).toBe('material-recovered')
expect(completeBodies).toHaveLength(2)
expect(completeBodies[0].uploadId).toBe('upload-same-id')
expect(completeBodies[1].uploadId).toBe('upload-same-id')
expect(fetchMock.mock.calls.map(([url]) => url)).toEqual([
'/api/materials/upload/complete',
'/api/materials/upload/complete',
'/api/materials/upload',
])
})
test('complete 遇到磁盘空间不足 507 时不重试放大压力', async () => {
const fetchMock = jest.fn().mockResolvedValue({
ok: false,
status: 507,
headers: { get: () => null },
text: async () => 'disk full',
})
global.fetch = fetchMock as unknown as typeof fetch
await expect(
completeBatchUpload('upload-storage-low', {
materialType: '音乐',
productModel: 'SPU-LOW',
fileName: 'song.mp3',
}),
).rejects.toThrow('分片合并失败: 507')
expect(fetchMock).toHaveBeenCalledTimes(1)
})
test('第二阶段响应丢失时只查 owner-scoped 状态,不盲目再次 POST', async () => {
const fetchMock = jest.fn((url: string, init?: RequestInit) => {
if (url === '/api/materials/upload/complete') {
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({
existingFilePath: 'uploads/temp/upload-response-lost/merged-song.mp3',
fileName: 'song.mp3',
}),
})
}
if (url === '/api/materials/upload' && init?.method === 'POST') {
const formData = init.body as FormData
expect(formData.get('uploadId')).toBe('upload-response-lost')
return Promise.reject(new TypeError('connection closed after request'))
}
if (url === '/api/materials/upload?uploadId=upload-response-lost') {
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({
status: 'completed',
material: { id: 'material-already-created' },
}),
})
}
return Promise.reject(new Error(`unexpected URL: ${url}`))
})
global.fetch = fetchMock as unknown as typeof fetch
const result = await completeBatchUpload('upload-response-lost', {
materialType: '音乐',
productModel: 'SPU-LOST',
fileName: 'song.mp3',
})
expect(result.material.id).toBe('material-already-created')
expect(fetchMock.mock.calls.filter(([url, init]) => (
url === '/api/materials/upload' && (init as RequestInit | undefined)?.method === 'POST'
))).toHaveLength(1)
expect(fetchMock.mock.calls.map(([url]) => url)).toEqual([
'/api/materials/upload/complete',
'/api/materials/upload',
'/api/materials/upload?uploadId=upload-response-lost',
])
})
test('临时目录已删的断点重放通过状态确认,不再创建第二份素材', async () => {
const fetchMock = jest.fn((url: string) => {
if (url === '/api/materials/upload/complete') {
return Promise.resolve({
ok: false,
status: 404,
text: async () => 'session missing',
})
}
if (url === '/api/materials/upload?uploadId=upload-temp-deleted') {
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({
status: 'completed',
material: { id: 'material-existing' },
}),
})
}
return Promise.reject(new Error(`unexpected URL: ${url}`))
})
global.fetch = fetchMock as unknown as typeof fetch
const result = await completeBatchUpload('upload-temp-deleted', {
materialType: '品牌VI灵感',
productModel: 'SPU-EXISTING',
fileName: 'brand.pdf',
})
expect(result.material.id).toBe('material-existing')
expect(fetchMock.mock.calls.map(([url]) => url)).toEqual([
'/api/materials/upload/complete',
'/api/materials/upload?uploadId=upload-temp-deleted',
])
})
})

View File

@ -85,14 +85,149 @@ async function base64ToBlob(base64Data: string): Promise<Blob | null> {
}) })
} }
// complete 接口服务端只允许 1 个处理中请求并排队少量后续请求。
// 文件级并发保持很小,单个文件内部的 chunkUpload 并发仍由其自身策略控制。
const SPECIAL_BATCH_FILE_CONCURRENCY = 3
const BRAND_VI_FILE_CONCURRENCY = 1
const COMPLETE_MAX_ATTEMPTS = 3
const MATERIAL_CREATE_TIMEOUT_MS = 15 * 60 * 1000
// PDF 工具自身最多运行 10 分钟;单文件串行后仍需覆盖一个在途请求和排队开销。
const BRAND_VI_MATERIAL_CREATE_TIMEOUT_MS = 30 * 60 * 1000
const MATERIAL_STATUS_REQUEST_TIMEOUT_MS = 15 * 1000
const MATERIAL_STATUS_POLL_INTERVAL_MS = 2 * 1000
async function mapWithConcurrency<T, R>(
items: T[],
concurrency: number,
mapper: (item: T, index: number) => Promise<R>,
): Promise<R[]> {
if (items.length === 0) {
return []
}
const results = new Array<R>(items.length)
let nextIndex = 0
const worker = async () => {
while (true) {
const index = nextIndex
nextIndex += 1
if (index >= items.length) {
return
}
results[index] = await mapper(items[index], index)
}
}
const workerCount = Math.min(Math.max(1, concurrency), items.length)
await Promise.all(Array.from({ length: workerCount }, () => worker()))
return results
}
function isRetryableCompleteStatus(status: number): boolean {
// 507 是上传盘不足,501 是服务不支持;立即重试只会放大压力。
return status === 429 || status === 500 || status === 502 || status === 503 || status === 504
}
function getRetryAfterDelayMs(value: string | null | undefined): number | null {
if (!value) {
return null
}
const seconds = Number(value)
if (Number.isFinite(seconds) && seconds >= 0) {
return Math.ceil(seconds * 1000)
}
const retryAt = Date.parse(value)
return Number.isNaN(retryAt) ? null : Math.max(0, retryAt - Date.now())
}
function waitForRetry(delayMs: number): Promise<void> {
if (delayMs <= 0) {
return Promise.resolve()
}
return new Promise((resolve) => setTimeout(resolve, delayMs))
}
function getDefaultRetryDelayMs(attempt: number): number {
return Math.min(2_000, 500 * (2 ** Math.max(0, attempt - 1)))
}
type MaterialUploadStatusResult = {
status?: 'completed' | 'processing' | 'not_found'
material?: Record<string, any>
error?: string
}
async function recoverAmbiguousMaterialCreation(
uploadId: string,
waitTimeoutMs: number,
originalError: unknown,
): Promise<Record<string, any>> {
const deadline = Date.now() + waitTimeoutMs
while (true) {
let response: Response
try {
response = await fetchWithTimeout(
`/api/materials/upload?uploadId=${encodeURIComponent(uploadId)}`,
{
method: 'GET',
timeout: MATERIAL_STATUS_REQUEST_TIMEOUT_MS,
},
)
} catch (statusError) {
throw new Error(
`创建素材请求中断,且状态查询失败;已保留上传断点,请稍后重试确认(${
statusError instanceof Error ? statusError.message : '网络错误'
})`,
)
}
if (response.status === 404) {
throw new Error('创建素材请求中断,服务器尚未确认建档;已保留上传断点,请稍后重试')
}
const statusResult = await response.json().catch(() => ({})) as MaterialUploadStatusResult
if (!response.ok) {
throw new Error(
statusResult.error || `创建素材请求中断,状态查询失败: ${response.status}`,
)
}
if (statusResult.status === 'completed' && statusResult.material) {
console.log(`[CompleteBatchUpload] 状态查询确认建档完成: ${uploadId}`)
return {
message: '上传成功',
material: statusResult.material,
replayed: true,
}
}
if (statusResult.status !== 'processing' || Date.now() >= deadline) {
const reason = originalError instanceof Error ? originalError.message : '网络连接中断'
throw new Error(`创建素材结果尚未确认(${reason});已保留上传断点,请稍后重试`)
}
await waitForRetry(Math.min(MATERIAL_STATUS_POLL_INTERVAL_MS, Math.max(0, deadline - Date.now())))
}
}
export async function completeBatchUpload(uploadId: string, metadata: Record<string, any>) { export async function completeBatchUpload(uploadId: string, metadata: Record<string, any>) {
const fileName = metadata.fileName || metadata.originalFileName || 'unknown' const fileName = metadata.fileName || metadata.originalFileName || 'unknown'
console.log(`[CompleteBatchUpload] 开始处理: ${fileName}, uploadId: ${uploadId}`) console.log(`[CompleteBatchUpload] 开始处理: ${fileName}, uploadId: ${uploadId}`)
try { try {
// 第一步:合并分片(设置5分钟超时) // 第一步:合并分片(设置5分钟超时)
console.log(`[CompleteBatchUpload] 调用complete端点: ${uploadId}`) let completeResponse: Response | null = null
const completeResponse = await fetchWithTimeout('/api/materials/upload/complete', { for (let attempt = 1; attempt <= COMPLETE_MAX_ATTEMPTS; attempt += 1) {
console.log(`[CompleteBatchUpload] 调用complete端点: ${uploadId}(第${attempt}次)`)
try {
completeResponse = await fetchWithTimeout('/api/materials/upload/complete', {
method: 'POST', method: 'POST',
headers: { 'Content-Type': 'application/json' }, headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ body: JSON.stringify({
@ -101,13 +236,53 @@ export async function completeBatchUpload(uploadId: string, metadata: Record<str
}), }),
timeout: 300000, // 5分钟超时 timeout: 300000, // 5分钟超时
}) })
} catch (error) {
if (attempt >= COMPLETE_MAX_ATTEMPTS) {
throw error
}
const delayMs = getDefaultRetryDelayMs(attempt)
console.warn(
`[CompleteBatchUpload] complete请求中断,将用同一uploadId重试: ${uploadId}, `
+ `等待${delayMs}ms`,
error,
)
await waitForRetry(delayMs)
continue
}
if (completeResponse.ok) {
break
}
if (!completeResponse.ok) {
const errorText = await completeResponse.text().catch(() => 'Unknown error') const errorText = await completeResponse.text().catch(() => 'Unknown error')
const canRetry = isRetryableCompleteStatus(completeResponse.status) && attempt < COMPLETE_MAX_ATTEMPTS
if (!canRetry) {
if (completeResponse.status === 404) {
// 建档成功后会删除临时目录;若浏览器在删除 IndexedDB 前退出,
// 下一次恢复会先看到 complete=404。此时只查 owner-scoped 结果。
return await recoverAmbiguousMaterialCreation(
uploadId,
0,
new Error('上传临时目录已清理'),
)
}
console.error(`[CompleteBatchUpload] complete端点返回错误: ${completeResponse.status}, ${errorText}`) console.error(`[CompleteBatchUpload] complete端点返回错误: ${completeResponse.status}, ${errorText}`)
throw new Error(`分片合并失败: ${completeResponse.status}`) throw new Error(`分片合并失败: ${completeResponse.status}`)
} }
const retryAfter = getRetryAfterDelayMs(completeResponse.headers?.get?.('Retry-After'))
?? getDefaultRetryDelayMs(attempt)
console.warn(
`[CompleteBatchUpload] complete端点暂时不可用,将重试: ${completeResponse.status}, `
+ `等待${retryAfter}ms`,
)
await waitForRetry(retryAfter)
}
if (!completeResponse || !completeResponse.ok) {
throw new Error('分片合并失败')
}
const completeResult = await completeResponse.json() const completeResult = await completeResponse.json()
const existingFilePath = completeResult.existingFilePath const existingFilePath = completeResult.existingFilePath
const originalFileName = completeResult.fileName || metadata.fileName const originalFileName = completeResult.fileName || metadata.fileName
@ -161,13 +336,25 @@ export async function completeBatchUpload(uploadId: string, metadata: Record<str
} }
} }
// 调用upload端点(设置5分钟超时) // 调用 upload 端点。品牌 VI 的 PDF 处理上限为 10 分钟,30 分钟窗口
// 同时覆盖一个在途任务、服务端排队和收尾;请求结果不确定时只查状态。
console.log(`[CompleteBatchUpload] 调用upload端点: ${uploadId}`) console.log(`[CompleteBatchUpload] 调用upload端点: ${uploadId}`)
const uploadResponse = await fetchWithTimeout('/api/materials/upload', { formData.append('uploadId', uploadId)
const materialCreateTimeout = materialType === '品牌VI灵感'
? BRAND_VI_MATERIAL_CREATE_TIMEOUT_MS
: MATERIAL_CREATE_TIMEOUT_MS
let uploadResponse: Response
try {
uploadResponse = await fetchWithTimeout('/api/materials/upload', {
method: 'POST', method: 'POST',
body: formData, body: formData,
timeout: 300000, // 5分钟超时 timeout: materialCreateTimeout,
}) })
} catch (error) {
// POST 可能已经在服务器完成,不能盲目再次建档;owner-scoped GET
// 是唯一恢复路径,失败时保留 IndexedDB 供用户稍后确认。
return await recoverAmbiguousMaterialCreation(uploadId, materialCreateTimeout, error)
}
if (!uploadResponse.ok) { if (!uploadResponse.ok) {
const errorData = await uploadResponse.json().catch(() => ({ error: `HTTP ${uploadResponse.status}` })) const errorData = await uploadResponse.json().catch(() => ({ error: `HTTP ${uploadResponse.status}` }))
@ -201,8 +388,7 @@ export async function uploadBatchMusic(
renderer: renderer, renderer: renderer,
} }
const results = await Promise.all( const results = await mapWithConcurrency(files, SPECIAL_BATCH_FILE_CONCURRENCY, async (item, index) => {
files.map(async (item, index) => {
try { try {
const metadata = { const metadata = {
...baseMetadata, ...baseMetadata,
@ -217,6 +403,7 @@ export async function uploadBatchMusic(
onProgress(totalProgress) onProgress(totalProgress)
}, },
skipComplete: true, skipComplete: true,
retainStateOnSkipComplete: true,
maxRetries: 5, maxRetries: 5,
}) })
@ -224,6 +411,7 @@ export async function uploadBatchMusic(
if (result?.uploadId) { if (result?.uploadId) {
try { try {
await completeBatchUpload(result.uploadId, metadata) await completeBatchUpload(result.uploadId, metadata)
await result.confirmMaterialCreated?.()
} catch (err) { } catch (err) {
console.error(`[Upload] 音乐文件 ${item.file.name} 后台处理失败:`, err) console.error(`[Upload] 音乐文件 ${item.file.name} 后台处理失败:`, err)
throw err // 重新抛出错误,让上层知道失败 throw err // 重新抛出错误,让上层知道失败
@ -236,7 +424,7 @@ export async function uploadBatchMusic(
productModel: item.productModel, productModel: item.productModel,
fileName: item.file.name, fileName: item.file.name,
fileSize: item.file.size, fileSize: item.file.size,
status: 'processing', status: 'processing' as const,
} }
} catch (error) { } catch (error) {
console.error(`[Upload] 音乐文件 ${item.file.name} 上传失败:`, error) console.error(`[Upload] 音乐文件 ${item.file.name} 上传失败:`, error)
@ -246,11 +434,10 @@ export async function uploadBatchMusic(
productModel: item.productModel, productModel: item.productModel,
fileName: item.file.name, fileName: item.file.name,
fileSize: item.file.size, fileSize: item.file.size,
status: 'failed', status: 'failed' as const,
} }
} }
}) })
)
return results return results
} }
@ -313,6 +500,7 @@ export async function uploadBatchVideo(
onProgress(totalProgress) onProgress(totalProgress)
}, },
skipComplete: true, skipComplete: true,
retainStateOnSkipComplete: true,
maxRetries: 5, maxRetries: 5,
}) })
@ -320,6 +508,7 @@ export async function uploadBatchVideo(
if (result?.uploadId) { if (result?.uploadId) {
try { try {
await completeBatchUpload(result.uploadId, metadata) await completeBatchUpload(result.uploadId, metadata)
await result.confirmMaterialCreated?.()
} catch (err) { } catch (err) {
console.error(`[Upload] 视频文件 ${item.file.name} 后台处理失败:`, err) console.error(`[Upload] 视频文件 ${item.file.name} 后台处理失败:`, err)
throw err // 重新抛出错误,让上层知道失败 throw err // 重新抛出错误,让上层知道失败
@ -369,8 +558,10 @@ export async function uploadBatch3D(
renderer: renderer, renderer: renderer,
} }
const results = await Promise.all( const results = await mapWithConcurrency(
files.map(async (item, index) => { files,
SPECIAL_BATCH_FILE_CONCURRENCY,
async (item, index) => {
try { try {
const metadata: any = { const metadata: any = {
...baseMetadata, ...baseMetadata,
@ -391,6 +582,7 @@ export async function uploadBatch3D(
onProgress(totalProgress) onProgress(totalProgress)
}, },
skipComplete: true, skipComplete: true,
retainStateOnSkipComplete: true,
maxRetries: 5, maxRetries: 5,
}) })
@ -398,6 +590,7 @@ export async function uploadBatch3D(
// 等待completeBatchUpload完成,确保素材记录被创建 // 等待completeBatchUpload完成,确保素材记录被创建
try { try {
await completeBatchUpload(result.uploadId, metadata) await completeBatchUpload(result.uploadId, metadata)
await result.confirmMaterialCreated?.()
} catch (err) { } catch (err) {
console.error(`[Upload] 3D场景文件 ${item.file.name} 后台处理失败:`, err) console.error(`[Upload] 3D场景文件 ${item.file.name} 后台处理失败:`, err)
throw err // 重新抛出错误,让上层知道失败 throw err // 重新抛出错误,让上层知道失败
@ -423,7 +616,7 @@ export async function uploadBatch3D(
status: 'failed', status: 'failed',
} }
} }
}) },
) )
return results return results
@ -448,8 +641,17 @@ export async function uploadBatchBuyerShow(
renderer: renderer, renderer: renderer,
} }
const results = await Promise.all( const results: Array<{
batches.map(async (item, index) => { tempId: string
uploadId: string | null
productModel: string
status: 'processing' | 'failed'
}> = []
// 一个买家秀批次可能包含多张原图。逐批提交可避免多个大 multipart
// 同时进入 Node.js 并占满内存;单批内仍保留原图质量和原有字段语义。
for (let index = 0; index < batches.length; index += 1) {
const item = batches[index]
try { try {
const metadata: any = { const metadata: any = {
...baseMetadata, ...baseMetadata,
@ -496,23 +698,22 @@ export async function uploadBatchBuyerShow(
const totalProgress = Math.round(((index + 1) * 100) / batches.length) const totalProgress = Math.round(((index + 1) * 100) / batches.length)
onProgress(totalProgress) onProgress(totalProgress)
return { results.push({
tempId: `pending-${Date.now()}-${index}`, tempId: `pending-${Date.now()}-${index}`,
uploadId: result.material?.id || null, uploadId: result.material?.id || null,
productModel: item.productModel, productModel: item.productModel,
status: 'processing', status: 'processing',
} })
} catch (error) { } catch (error) {
console.error(`[Upload] 买家秀批次 ${item.productModel} 上传失败:`, error) console.error(`[Upload] 买家秀批次 ${item.productModel} 上传失败:`, error)
return { results.push({
tempId: `pending-${Date.now()}-${index}`, tempId: `pending-${Date.now()}-${index}`,
uploadId: null, uploadId: null,
productModel: item.productModel, productModel: item.productModel,
status: 'failed', status: 'failed',
}
}
}) })
) }
}
return results return results
} }
@ -536,8 +737,17 @@ export async function uploadBatchXiaohongshu(
renderer: renderer, renderer: renderer,
} }
const results = await Promise.all( const results: Array<{
batches.map(async (item, index) => { tempId: string
uploadId: string | null
productModel: string
status: 'processing' | 'failed'
}> = []
// 与买家秀保持同一安全边界:逐批提交,避免多个图片 multipart
// 同时解析;某一批失败不会阻止后续批次继续。
for (let index = 0; index < batches.length; index += 1) {
const item = batches[index]
try { try {
const metadata: any = { const metadata: any = {
...baseMetadata, ...baseMetadata,
@ -584,23 +794,22 @@ export async function uploadBatchXiaohongshu(
const totalProgress = Math.round(((index + 1) * 100) / batches.length) const totalProgress = Math.round(((index + 1) * 100) / batches.length)
onProgress(totalProgress) onProgress(totalProgress)
return { results.push({
tempId: `pending-${Date.now()}-${index}`, tempId: `pending-${Date.now()}-${index}`,
uploadId: result.material?.id || null, uploadId: result.material?.id || null,
productModel: item.productModel, productModel: item.productModel,
status: 'processing', status: 'processing',
} })
} catch (error) { } catch (error) {
console.error(`[Upload] 小红书批次 ${item.productModel} 上传失败:`, error) console.error(`[Upload] 小红书批次 ${item.productModel} 上传失败:`, error)
return { results.push({
tempId: `pending-${Date.now()}-${index}`, tempId: `pending-${Date.now()}-${index}`,
uploadId: null, uploadId: null,
productModel: item.productModel, productModel: item.productModel,
status: 'failed', status: 'failed',
}
}
}) })
) }
}
return results return results
} }
@ -622,8 +831,7 @@ export async function uploadBatchBrandVI(
productCategoryName: productCategoryName || '', productCategoryName: productCategoryName || '',
} }
const results = await Promise.all( const results = await mapWithConcurrency(files, BRAND_VI_FILE_CONCURRENCY, async (item, index) => {
files.map(async (item, index) => {
try { try {
const metadata = { const metadata = {
...baseMetadata, ...baseMetadata,
@ -638,6 +846,7 @@ export async function uploadBatchBrandVI(
onProgress(totalProgress) onProgress(totalProgress)
}, },
skipComplete: true, skipComplete: true,
retainStateOnSkipComplete: true,
maxRetries: 5, maxRetries: 5,
}) })
@ -645,6 +854,7 @@ export async function uploadBatchBrandVI(
if (result?.uploadId) { if (result?.uploadId) {
try { try {
await completeBatchUpload(result.uploadId, metadata) await completeBatchUpload(result.uploadId, metadata)
await result.confirmMaterialCreated?.()
} catch (err) { } catch (err) {
console.error(`[Upload] PDF文件 ${item.file.name} 后台处理失败:`, err) console.error(`[Upload] PDF文件 ${item.file.name} 后台处理失败:`, err)
throw err // 重新抛出错误,让上层知道失败 throw err // 重新抛出错误,让上层知道失败
@ -657,7 +867,7 @@ export async function uploadBatchBrandVI(
productModel: item.productModel, productModel: item.productModel,
fileName: item.file.name, fileName: item.file.name,
fileSize: item.file.size, fileSize: item.file.size,
status: 'processing', status: 'processing' as const,
} }
} catch (error) { } catch (error) {
console.error(`[Upload] PDF文件 ${item.file.name} 上传失败:`, error) console.error(`[Upload] PDF文件 ${item.file.name} 上传失败:`, error)
@ -667,11 +877,10 @@ export async function uploadBatchBrandVI(
productModel: item.productModel, productModel: item.productModel,
fileName: item.file.name, fileName: item.file.name,
fileSize: item.file.size, fileSize: item.file.size,
status: 'failed', status: 'failed' as const,
} }
} }
}) })
)
return results return results
} }

View File

@ -2,8 +2,13 @@
import { chunkUpload } from '@/lib/chunk-upload' import { chunkUpload } from '@/lib/chunk-upload'
import { completeBatchUpload, uploadBatchVideo } from './special-batch-uploaders' import {
import { compressVideoPreviewImage } from './image-processor' completeBatchUpload,
uploadBatchBuyerShow,
uploadBatchVideo,
uploadBatchXiaohongshu,
} from './special-batch-uploaders'
import { compressPreviewImage, compressVideoPreviewImage } from './image-processor'
jest.mock('@/lib/chunk-upload', () => ({ jest.mock('@/lib/chunk-upload', () => ({
chunkUpload: jest.fn(), chunkUpload: jest.fn(),
@ -127,3 +132,88 @@ describe('uploadBatchVideo preview image contract', () => {
expect(fetchMock).toHaveBeenCalledTimes(1) expect(fetchMock).toHaveBeenCalledTimes(1)
}) })
}) })
describe('image-only batch upload memory boundary', () => {
const originalFetch = global.fetch
beforeEach(() => {
;(compressPreviewImage as jest.Mock).mockImplementation(async (file: File) => file)
})
afterEach(() => {
global.fetch = originalFetch
jest.clearAllMocks()
})
test('买家秀必须等上一批请求完成后才提交下一批', async () => {
const firstResponse = createDeferred<any>()
const fetchMock = jest.fn()
.mockReturnValueOnce(firstResponse.promise)
.mockResolvedValueOnce({
ok: true,
json: async () => ({ material: { id: 'buyer-2' } }),
})
global.fetch = fetchMock as unknown as typeof fetch
const progress: number[] = []
const uploadPromise = uploadBatchBuyerShow(
[
{ productModel: 'BUYER-1', images: [new File(['one'], 'one.jpg', { type: 'image/jpeg' })] },
{ productModel: 'BUYER-2', images: [new File(['two'], 'two.jpg', { type: 'image/jpeg' })] },
],
'',
'',
'',
'',
(value) => progress.push(value),
)
for (let index = 0; index < 10 && fetchMock.mock.calls.length < 1; index += 1) {
await Promise.resolve()
}
expect(fetchMock).toHaveBeenCalledTimes(1)
firstResponse.resolve({
ok: true,
json: async () => ({ material: { id: 'buyer-1' } }),
})
const results = await uploadPromise
expect(fetchMock).toHaveBeenCalledTimes(2)
expect(results.map((item) => item.productModel)).toEqual(['BUYER-1', 'BUYER-2'])
expect(results.map((item) => item.status)).toEqual(['processing', 'processing'])
expect(progress).toEqual([50, 100])
})
test('小红书某一批失败后仍按顺序继续下一批', async () => {
const consoleError = jest.spyOn(console, 'error').mockImplementation(() => undefined)
const fetchMock = jest.fn()
.mockResolvedValueOnce({
ok: false,
status: 413,
json: async () => ({ error: '图片批次过大' }),
})
.mockResolvedValueOnce({
ok: true,
json: async () => ({ material: { id: 'redbook-2' } }),
})
global.fetch = fetchMock as unknown as typeof fetch
const results = await uploadBatchXiaohongshu(
[
{ productModel: 'RED-1', images: [new File(['one'], 'one.jpg', { type: 'image/jpeg' })] },
{ productModel: 'RED-2', images: [new File(['two'], 'two.jpg', { type: 'image/jpeg' })] },
],
'',
'',
'',
'',
jest.fn(),
)
expect(fetchMock).toHaveBeenCalledTimes(2)
expect(results.map((item) => item.productModel)).toEqual(['RED-1', 'RED-2'])
expect(results.map((item) => item.status)).toEqual(['failed', 'processing'])
consoleError.mockRestore()
})
})

View File

@ -1,5 +1,7 @@
'use client' 'use client'
// LEGACY_READ_ONLY: 旧域名仅用于展示历史审核素材。
import Image from 'next/image' import Image from 'next/image'
import { import {
type ReactNode, type ReactNode,

View File

@ -1,5 +1,7 @@
'use client' 'use client'
// LEGACY_READ_ONLY: 旧域名仅用于展示历史审核素材。
import Image from 'next/image' import Image from 'next/image'
import { useEffect, useMemo, useRef, useState } from 'react' import { useEffect, useMemo, useRef, useState } from 'react'
import { createPortal } from 'react-dom' import { createPortal } from 'react-dom'

View File

@ -1,5 +1,7 @@
'use client' 'use client'
// LEGACY_READ_ONLY: 旧域名仅用于展示历史审核素材。
import Image from 'next/image' import Image from 'next/image'
import { useEffect, useMemo, useState } from 'react' import { useEffect, useMemo, useState } from 'react'
import type { import type {

View File

@ -1,5 +1,7 @@
'use client' 'use client'
// LEGACY_READ_ONLY: 旧域名仅用于展示历史审核素材。
import Image from 'next/image' import Image from 'next/image'
import { useEffect, useMemo, useRef, useState } from 'react' import { useEffect, useMemo, useRef, useState } from 'react'
import type { import type {

View File

@ -82,10 +82,16 @@ services:
- DINGTALK_ROBOT_CODE=${DINGTALK_ROBOT_CODE:-} - DINGTALK_ROBOT_CODE=${DINGTALK_ROBOT_CODE:-}
- DINGTALK_PERFORMANCE_SURVEY_NOTIFICATION_CHANNEL=${DINGTALK_PERFORMANCE_SURVEY_NOTIFICATION_CHANNEL:-work_notice} - DINGTALK_PERFORMANCE_SURVEY_NOTIFICATION_CHANNEL=${DINGTALK_PERFORMANCE_SURVEY_NOTIFICATION_CHANNEL:-work_notice}
- DINGTALK_ROBOT_OTO_TEST_USER_IDS=${DINGTALK_ROBOT_OTO_TEST_USER_IDS:-} - DINGTALK_ROBOT_OTO_TEST_USER_IDS=${DINGTALK_ROBOT_OTO_TEST_USER_IDS:-}
- VIDEO_TRANSCODE_CODEC=libx265 # 视频转码配置:默认 720P H.265、10 秒片段;单实例单任务、单线程
- VIDEO_TRANSCODE_PRESET=medium - VIDEO_PREVIEW_RESOLUTION=720
- VIDEO_TRANSCODE_CRF=22 - VIDEO_CLIP_DURATION=10
- VIDEO_AUDIO_BITRATE=160k - VIDEO_PREVIEW_CRF=20
- VIDEO_CLIP_CRF=24
- VIDEO_TRANSCODE_THREADS=1
- VIDEO_TRANSCODE_MAX_CONCURRENT=1
- VIDEO_TRANSCODE_TIMEOUT=3600000
- VIDEO_MAX_TEMP_FILE_SIZE=10737418240
- VIDEO_TRANSCODE_RUNTIME_DIR=/app/uploads/.video-transcode-runtime
# 下载 Token 密钥(用于外部接口文件下载鉴权) # 下载 Token 密钥(用于外部接口文件下载鉴权)
- DOWNLOAD_TOKEN_SECRET=${SHADOW_DOWNLOAD_TOKEN_SECRET:?SHADOW_DOWNLOAD_TOKEN_SECRET is required} - DOWNLOAD_TOKEN_SECRET=${SHADOW_DOWNLOAD_TOKEN_SECRET:?SHADOW_DOWNLOAD_TOKEN_SECRET is required}
# Redis配置(影子系统使用独立的 Redis 实例) # Redis配置(影子系统使用独立的 Redis 实例)

View File

@ -1,5 +1,6 @@
# 当前生产发布不使用本文件作为入口。 # 当前生产发布不使用本文件作为入口。
# 生产发布以 scripts/git-incremental-deploy.ps1 为准;本文件仅作本地/单机/基础环境参考。 # 生产发布以 scripts/git-incremental-deploy.ps1 为准;本文件仅作单机/基础环境参考。
# 本文件仍按生产数据安全边界配置:必须显式传入真实 uploads 宿主机挂载,禁止回落到 ./uploads。
# 不要按此文件推断当前线上容器启动参数、uploads 挂载或健康检查。 # 不要按此文件推断当前线上容器启动参数、uploads 挂载或健康检查。
services: services:
app: app:
@ -30,6 +31,7 @@ services:
start_period: 40s start_period: 40s
environment: environment:
- NODE_ENV=production - NODE_ENV=production
- UPLOAD_DIR=/app/uploads
- DATABASE_URL=${DATABASE_URL:?DATABASE_URL is required} - DATABASE_URL=${DATABASE_URL:?DATABASE_URL is required}
- DATABASE_DRIVER=node - DATABASE_DRIVER=node
- KEY_VAULTS_SECRET=${KEY_VAULTS_SECRET:?KEY_VAULTS_SECRET is required} - KEY_VAULTS_SECRET=${KEY_VAULTS_SECRET:?KEY_VAULTS_SECRET is required}
@ -67,18 +69,28 @@ services:
- YYD_NOTIFICATION_CENTER_PROJECT_CODE=${YYD_NOTIFICATION_CENTER_PROJECT_CODE:-material_library} - YYD_NOTIFICATION_CENTER_PROJECT_CODE=${YYD_NOTIFICATION_CENTER_PROJECT_CODE:-material_library}
- YYD_NOTIFICATION_CENTER_PROJECT_SECRET=${YYD_NOTIFICATION_CENTER_PROJECT_SECRET:-} - YYD_NOTIFICATION_CENTER_PROJECT_SECRET=${YYD_NOTIFICATION_CENTER_PROJECT_SECRET:-}
- YYD_NOTIFICATION_CENTER_TEST_USER_IDS=${YYD_NOTIFICATION_CENTER_TEST_USER_IDS:-} - YYD_NOTIFICATION_CENTER_TEST_USER_IDS=${YYD_NOTIFICATION_CENTER_TEST_USER_IDS:-}
# 视频转码配置 # 视频转码配置:默认 720P H.265、10 秒片段;单实例单任务、单线程
- VIDEO_TRANSCODE_CODEC=libx265 - VIDEO_PREVIEW_RESOLUTION=720
- VIDEO_TRANSCODE_PRESET=medium - VIDEO_CLIP_DURATION=10
- VIDEO_TRANSCODE_CRF=22 - VIDEO_PREVIEW_CRF=20
- VIDEO_AUDIO_BITRATE=160k - VIDEO_CLIP_CRF=24
- VIDEO_TRANSCODE_THREADS=1
- VIDEO_TRANSCODE_MAX_CONCURRENT=1
- VIDEO_TRANSCODE_TIMEOUT=3600000
- VIDEO_MAX_TEMP_FILE_SIZE=10737418240
- VIDEO_TRANSCODE_RUNTIME_DIR=/app/uploads/.video-transcode-runtime
# 下载 Token 密钥(用于外部接口文件下载鉴权) # 下载 Token 密钥(用于外部接口文件下载鉴权)
- DOWNLOAD_TOKEN_SECRET=${DOWNLOAD_TOKEN_SECRET:?DOWNLOAD_TOKEN_SECRET is required} - DOWNLOAD_TOKEN_SECRET=${DOWNLOAD_TOKEN_SECRET:?DOWNLOAD_TOKEN_SECRET is required}
# Redis配置(用于上传会话存储) # Redis配置(用于上传会话存储)
- USE_REDIS=true - USE_REDIS=true
- REDIS_URL=redis://redis:6379 - REDIS_URL=redis://redis:6379
volumes: volumes:
- ./uploads:/app/uploads - type: bind
source: ${GONGXIANGP2_UPLOADS_MOUNT:?GONGXIANGP2_UPLOADS_MOUNT must point to the verified host uploads mount}
target: /app/uploads
bind:
# 路径写错或真实挂载未准备好时必须失败,不能静默创建空目录继续运行。
create_host_path: false
networks: networks:
- app-network - app-network
depends_on: depends_on:

View File

@ -70,7 +70,7 @@ AI 生图平台
- 展示图对象优先写入 `/gongxiangpan/public-display/` - 展示图对象优先写入 `/gongxiangpan/public-display/`
- `uploads` 保留为历史混合区,不作为默认展示资源路径 - `uploads` 保留为历史混合区,不作为默认展示资源路径
- 源文件 / ZIP / PSD / 视频源等下载型对象继续落在生产机 uploads:`/mnt/PoolOne/apps/youyidian-3d/uploads` - 源文件 / ZIP / PSD / 视频源等下载型对象继续落在生产机独立 ZFS uploads 数据集:`/mnt/PoolOne/youyidian-uploads`
- `s3.1080p.hk/jsxiangche` 只保留极少数老素材兼容,不作为新素材或新预览写入目标 - `s3.1080p.hk/jsxiangche` 只保留极少数老素材兼容,不作为新素材或新预览写入目标
- 当前不再以 `X-Cache-Status` 作为正式验证标准 - 当前不再以 `X-Cache-Status` 作为正式验证标准
- 当前优先验证 `s3.jiansun.vip -> 218.93.9.38 -> 192.168.58.20:9000` 是否可达 - 当前优先验证 `s3.jiansun.vip -> 218.93.9.38 -> 192.168.58.20:9000` 是否可达

View File

@ -25,7 +25,7 @@
默认部署: 默认部署:
- Git 仓库:`http://git.1080p.hk/WuYongJie/gongxiangp2.git` - Git 仓库:`https://git.1080p.hk/WuYongJie/gongxiangp2.git`(证书必须有效,禁止跳过 TLS 校验)
- 分支:`main` - 分支:`main`
- 构建机:`192.168.40.206` - 构建机:`192.168.40.206`
- 生产机:`192.168.58.20` - 生产机:`192.168.58.20`
@ -44,7 +44,8 @@
- 构建机能访问 Git 仓库和生产机 `192.168.58.20`。 - 构建机能访问 Git 仓库和生产机 `192.168.58.20`。
- 构建机本地 Registry 端口 `5000` 可用。 - 构建机本地 Registry 端口 `5000` 可用。
- 已通过环境变量或参数提供构建机密码:`BUILD_SERVER_PASSWORD` 或 `-BuildServerPassword`。 - 已通过环境变量或参数提供构建机密码:`BUILD_SERVER_PASSWORD` 或 `-BuildServerPassword`。
- 私有 Git 仓库认证通过 `GIT_DEPLOY_USERNAME` / `GIT_DEPLOY_PASSWORD`,或 `GIT_AUTH_HEADER` 提供;不要把密码写进仓库 URL。 - 已分别固定构建机和生产机的 SSH SHA256 主机指纹;OpenSSH 模式还必须使用预置、只读的 `known_hosts` 文件。
- 私有 Git 仓库认证必须预先配置在维护机和构建机的只读 credential helper 中。脚本拒绝 `GIT_DEPLOY_PASSWORD`、`GIT_AUTH_HEADER`、明文 HTTP 和 URL 内嵌凭据。
## 安全边界 ## 安全边界
@ -52,6 +53,7 @@
- 不在生产机上执行 `npm install` 或 `next build`。 - 不在生产机上执行 `npm install` 或 `next build`。
- 不改生产机的 `.env` 和 uploads 目录。 - 不改生产机的 `.env` 和 uploads 目录。
- 不把 Git 密码写入仓库或远端 URL。 - 不把 Git 密码写入仓库或远端 URL。
- Git HTTPS 证书失效、SSH 指纹不匹配或仓库地址偏离权威地址时,发布必须停止;不得使用 `-k` 或关闭主机校验绕过。
- 生产容器启动失败时,默认尝试回滚到旧镜像。 - 生产容器启动失败时,默认尝试回滚到旧镜像。
- 清理策略只清理无用镜像/容器,默认保留 Docker build cache。 - 清理策略只清理无用镜像/容器,默认保留 Docker build cache。
@ -79,11 +81,13 @@ DryRun 不会改代码、构建镜像或重启生产容器。
- `production-<commit>`:可回滚的确定版本 - `production-<commit>`:可回滚的确定版本
- `production-latest`:正式环境最新版本 - `production-latest`:正式环境最新版本
影子环境对应: 影子环境的历史标签为:
- `shadow-<commit>` - `shadow-<commit>`
- `shadow-latest` - `shadow-latest`
当前通用发布脚本会在任何 Git 扫描或远程操作前拒绝 `Target=shadow`,因为影子环境尚未证明 uploads、后台 runtime 和 Redis 与生产完全隔离。不得把这些历史标签说明理解为可执行发布入口。
## 回滚思路 ## 回滚思路
脚本启动新容器前会记录旧容器镜像。如果新容器健康检查失败,默认会重新用旧镜像启动容器。 脚本启动新容器前会记录旧容器镜像。如果新容器健康检查失败,默认会重新用旧镜像启动容器。

View File

@ -77,10 +77,12 @@
- 首期不做历史对象全量迁移。 - 首期不做历史对象全量迁移。
- 新生成展示图直接写 `public-display`,不再指望 `uploads` 承担正式展示路径。 - 新生成展示图直接写 `public-display`,不再指望 `uploads` 承担正式展示路径。
### 3.3 明确不作为展示路径的大对象 ### 3.3 不纳入 `public-display` 展示图/缓存规则的大对象
- 视频源文件 视频预览、预览片段和封面仍然会写入 S3 并通过对应的媒体 URL 提供播放;本节只说明它们不使用图片展示资源的 `public-display` 前缀和缓存规则。
- 视频预览大文件
- 视频源文件(canonical 源仍在生产机 uploads/NAS)
- 视频预览大文件(对象仍在 S3,但不走 `public-display` 前缀)
- 压缩包 - 压缩包
- 其它下载型源文件 - 其它下载型源文件

View File

@ -0,0 +1,81 @@
# 存储域名与品牌资源事故复盘(2026-08-11)
## 结论
生产 Header Logo 的旧地址不是 2026 年近期修复首次写入的,而是 2025 年遗留配置。近期系统设置安全重构虽然没有制造这条历史数据,但沿用了过期默认值,也没有识别“展示代理地址可能被管理页再次写回数据库”的问题,因此同类故障仍可能复发。
本次治理把存储引用分成两类:
- `LEGACY_READ_ONLY`:只允许读取、代理或迁移历史素材。
- 当前写入:默认值、新上传、恢复默认、系统设置和新数据库记录只能使用本地 `/brand/...` 或当前 `s3.jiansun.vip/gongxiangpan/...`。
2026-08-12 13:13(Asia/Shanghai)重新只读核查当前正式数据库,并与公开 `/api/system-settings` 结果逐字段比对:唯一设置行的 Header / Login Logo 已是 `https://s3.jiansun.vip/gongxiangpan/...` 当前可信桶地址,用户指出的旧精确 Header URL 已不在现有行中;两个公网对象均返回 HTTP 200、`image/png`,浏览器实际解码为 800×800,并在正式首页显示为 40×40 图标。因此“现有行仍保存旧地址”已不再是当前事实。
但这不等于数据库治理已经完成:同一只读核查显示 `headerLogoUrl` / `loginLogoUrl` 的数据库列默认值仍是退役 `http://s3.1080p.hk/...`,本轮三条目标 migration 也尚无已完成记录。仓库 migration 会在已有 `system_settings` 历史基线上补齐列、修正未来默认值,并且只把品牌两列中匹配退役前缀的值改为本地 `/brand/...`;当前可信桶中的合法自定义 Logo 不会被覆盖。仓库更早的历史 migration 本身不能从空数据库完整重放,本轮没有把这件事包装成“全新建库已修复”。
## 误导来源
1. Prisma 的 Logo 默认值仍指向旧域名。
2. 系统设置行自动创建时仍使用旧默认值。
3. 后台“恢复默认”按钮会重新保存旧地址。
4. 管理员 GET 返回展示代理 URL,页面保存其它设置时可能把代理 URL 当原值写回。
5. favicon 上游失败后返回透明 1×1 图片,接口看似成功但用户看不到 Logo。
6. Header、登录、注册、完善资料页没有稳定的本地图片回退。
7. 在线历史恢复接口会把历史桶 URL 直接写入新素材。
8. 旧项目记忆把已经下线的资源域名描述为新素材正式入口。
9. 现有存储迁移文档只核对素材预览,没有把 Header、登录 Logo、favicon、后台默认值和 Schema 默认值列入验收。
10. Git 历史中曾存在包含旧生产事实与敏感配置的报告;工作树删除不能消除历史风险,需单独纳入秘密扫描和凭据风险登记,不能在当前修复中贸然改写远端历史。
### Git 时间线证据
- `3fff87c`(2025-10-27)引入 Logo 分离上传;生产旧 Header 文件名中的时间戳也落在这一天。
- `2b7cd7f`(2025-12-27)的历史数据库报告已记录用户本次指出的精确 Header URL,证明它不是 2026-07 的安全修复首次写入。
- `3cba791` / `7c913b6`(2026-07-30/31)的系统设置脱敏修复没有主动更新生产 Logo,但仍复制了旧 Schema 默认值、API 默认值和后台重置值。这是本轮必须承认的遗漏:修复了凭据泄漏,却没有同时清理品牌资源的旧写入路径。
- `d69eb6e`(2026-08-07)只处理视频手动预览图,不涉及 Logo,也没有解决视频队列跨部署恢复。
因此,事故来源不是“某一次新提交把当前生产行直接改成旧 Header”,而是历史脏值长期存在,加上后续修复没有铲除所有复发入口。两部分都必须处理,不能只修数据库或只加页面回退。
| 文件 / 资料 | 造成的错误判断或复发风险 | 本次处理 |
| --- | --- | --- |
| `prisma/schema.prisma` | 新建设置行时可能自动填入旧 Logo | 改为本地品牌默认值并新增 migration |
| `app/api/system-settings/route.ts` | 设置不存在时写旧默认;管理员读取展示代理地址后可能再写回 | canonical 与 preview 分离,PUT 严格校验 |
| `app/admin/settings/page.tsx` | “恢复默认”重新选择旧域名 | 改为 `/brand/...` |
| `app/api/favicon/route.ts` | 旧地址失败后用透明 1×1 冒充成功 | 改为真实本地 favicon |
| `components/Header.tsx`、登录/注册/完善资料页 | 远端图坏时没有稳定回退,alt 被挤成竖排坏图文字 | 统一使用一次性回退组件 |
| `app/api/system-settings/upload-logo/route.ts` | 只信浏览器 MIME、未限制大小、上传后未确认对象已持久化且元数据一致,权限口径也不一致 | 限制字节/像素/尺寸并完整解码、核对 MIME、带凭据 HEAD 核对大小和类型后再写库、统一系统管理员权限 |
| `app/api/restore-data/route.ts` | 在线流程直接把旧桶 URL 写进新素材 | 永久停用在线写入入口 |
| `.kiro/steering/项目记忆库.md` | 把已下线资源域名写成新素材正式入口,并用“老素材不改”覆盖了系统 Logo | 更新当前拓扑,明确 Logo 不属于老素材 |
| `.kiro/specs/production-system-analysis.md` | 2025 年快照外观像当前正式说明 | 顶部标记为历史快照并指向当前文档 |
| 存储迁移文档 | 只验收素材,遗漏站点品牌资源和设置往返 | 补齐迁移门禁 |
| Git 历史中的旧生产报告 | 既会误导当前判断,也存在历史秘密风险 | 登记风险;当前不贸然改写远端历史 |
## 已采取的代码门禁
- 构建前执行 `npm run storage:legacy-boundary`。
- 旧域名只允许出现在明确列出的历史只读兼容代码、兼容性测试和不可修改的历史 migration 中。
- 在线历史恢复 API 永久停用;历史恢复改走显式来源、显式目标、默认 dry-run、先迁移对象再写库的离线流程。
- 品牌资源使用镜像内本地文件作为最终回退。
该门禁由两层组成:普通构建会扫描进入 Docker 构建上下文的源码、数据库 migration 和常见配置;正式增量发布会在工作站完整仓库上先扫描全部受支持源码/脚本,再允许联网构建。普通说明文档、未知扩展名及 Git 历史仍需独立审阅与秘密扫描,不能把规则匹配当成语义证明或历史清理工具。
## 合法的历史兼容范围
以下用途仍可识别旧域名,但不得生成新的旧域名数据:
- 媒体代理与 URL 改写
- 工作审核和考核调研中的历史素材展示
- 历史资源网络探测
- 旧对象迁移到当前桶的离线工具
- 对应的兼容性测试与不可变历史 migration
新增允许项必须在代码评审中说明用途并加入显式 allowlist,不能用“老素材兼容”作为默认值、上传目标、重置值或新写入的理由。
## 后续验收
1. 当前行值和首页显示已验收;仍须执行受控 migration,并确认三个目标 migration 均有成功记录、Logo 两列默认值已改为 `/brand/...`,再逐行复核只存在本地或当前可信桶地址。
2. Logo 配置 URL 404、超时或返回 HTML 时,所有入口仍显示可见本地品牌图。
3. 管理员只改颜色/维护通知并保存,不得改变数据库中的 canonical Logo URL。
4. 新建系统设置行和“恢复默认”均不得出现旧域名。
5. 构建门禁能阻止旧域名重新进入非只读代码;唯一受控写引用只能是本 migration 的精确品牌数据清洗 SQL。
6. 当前桶切换或存储迁移必须一并核对素材预览、品牌资源、favicon、设置 API、默认值和重置值。
7. Logo 上传验收必须再从公网 URL 实际 GET,确认 HTTP 200、图片 Content-Type 和浏览器可解码;带凭据 HEAD 成功不能替代用户侧可见性验收。

View File

@ -18,7 +18,7 @@
| 时间(北京时间) | 生产证据 | 业务对应 | | 时间(北京时间) | 生产证据 | 业务对应 |
| --- | --- | --- | | --- | --- | --- |
| 7 月 30 日 15:18 | `libvips worker` 触发 `global_oom`;内核杀掉 `next-server`,其匿名 RSS 约 1.405GiB | 同时段任务详情完成接口进入 `sharp.metadata()`、`sliceImage/compressImage`,Nginx 随后出现 502 | | 7 月 30 日 15:18 | `libvips worker` 触发 `global_oom`;内核杀掉 `next-server`,其匿名 RSS 约 1.405GiB | 同时段任务详情完成接口进入 `sharp.metadata()`、`sliceImage/compressImage`,Nginx 随后出现 502 |
| 7 月 30 日 16:18 | 同一应用容器同时有两个 `ffmpeg`,RSS 约 642MiB 和 372–408MiB;之后 MinIO 也被杀 | 同时段连续创建淘系视频任务;转码队列未配置时默认并发为 2 | | 7 月 30 日 16:18 | 同一应用容器同时有两个 `ffmpeg`,RSS 约 642MiB 和 372–408MiB;之后 MinIO 也被杀 | **事故时旧实现**未配置转码并发,默认并发为 2;后续已固定为同一容器 1 条任务、FFmpeg 1 线程(见下文) |
| 7 月 30 日 17:39 | `libvips worker` 再次触发 `global_oom`;同时还有一个约 325MiB 的 `ffmpeg`,最终 `next-server` 被杀 | 图片任务详情和抖音视频任务叠加,随后大量 502/504 | | 7 月 30 日 17:39 | `libvips worker` 再次触发 `global_oom`;同时还有一个约 325MiB 的 `ffmpeg`,最终 `next-server` 被杀 | 图片任务详情和抖音视频任务叠加,随后大量 502/504 |
三次事件都显示:宿主机没有 Swap,且是 `global_oom`,不是应用容器自己的 cgroup OOM。 三次事件都显示:宿主机没有 Swap,且是 `global_oom`,不是应用容器自己的 cgroup OOM。

View File

@ -6,7 +6,7 @@
## 1. 当前结论 ## 1. 当前结论
- 新素材预览 / 展示文件:写入生产 MinIO 新桶路径,公开 URL 统一为 `https://s3.jiansun.vip/gongxiangpan/由壹点AI/public-display/...`。 - 新素材预览 / 展示文件:写入生产 MinIO 新桶路径,公开 URL 统一为 `https://s3.jiansun.vip/gongxiangpan/由壹点AI/public-display/...`。
- 源文件 / 下载型文件:写入生产服务器 uploads 目录:`/mnt/PoolOne/apps/youyidian-3d/uploads`。 - 源文件 / 下载型文件:写入生产服务器独立 ZFS uploads 数据集:`/mnt/PoolOne/youyidian-uploads`。历史路径 `/mnt/PoolOne/apps/youyidian-3d/uploads` 仅作人工明确的兼容选项,不再是默认值。
- 旧 `1080p.hk` 桶:只用于极少数老素材兼容,不作为新素材写入目标。 - 旧 `1080p.hk` 桶:只用于极少数老素材兼容,不作为新素材写入目标。
- `/gongxiangpan/uploads/` 是历史混合区,不再作为新展示资源的默认路径。 - `/gongxiangpan/uploads/` 是历史混合区,不再作为新展示资源的默认路径。
@ -77,7 +77,7 @@ https://s3.jiansun.vip/gongxiangpan/由壹点AI/public-display/...
生产 uploads 目录: 生产 uploads 目录:
```text ```text
/mnt/PoolOne/apps/youyidian-3d/uploads /mnt/PoolOne/youyidian-uploads
``` ```
容器内路径: 容器内路径:
@ -110,13 +110,13 @@ https://s3.jiansun.vip/gongxiangpan/由壹点AI/public-display/...
- Windows 维护机读取 `Z:\...` 源素材。 - Windows 维护机读取 `Z:\...` 源素材。
- 预览图上传到新桶 `gongxiangpan` 的 `由壹点AI/public-display/...`。 - 预览图上传到新桶 `gongxiangpan` 的 `由壹点AI/public-display/...`。
- ZIP / PSD / 源文件先在本机生成,再通过远程同步写入生产机 `/mnt/PoolOne/apps/youyidian-3d/uploads`,并校验远程文件大小。 - ZIP / PSD / 源文件先在本机生成,再通过远程同步写入生产机 `/mnt/PoolOne/youyidian-uploads`,并校验远程文件大小。
正式 `--apply` 如果在 Windows 上连接生产库,必须同时配置: 正式 `--apply` 如果在 Windows 上连接生产库,必须同时配置:
```env ```env
HISTORY_IMPORT_REMOTE_UPLOADS_HOST=prod HISTORY_IMPORT_REMOTE_UPLOADS_HOST=prod
HISTORY_IMPORT_REMOTE_UPLOADS_DIR=/mnt/PoolOne/apps/youyidian-3d/uploads HISTORY_IMPORT_REMOTE_UPLOADS_DIR=/mnt/PoolOne/youyidian-uploads
``` ```
没有远程 uploads 同步时,脚本必须阻止正式导入。否则会把数据库写到生产库,但 ZIP / PSD 留在本机 `uploads`,线上下载会 404。 没有远程 uploads 同步时,脚本必须阻止正式导入。否则会把数据库写到生产库,但 ZIP / PSD 留在本机 `uploads`,线上下载会 404。
@ -131,7 +131,19 @@ HISTORY_IMPORT_REMOTE_UPLOADS_DIR=/mnt/PoolOne/apps/youyidian-3d/uploads
`s3.yydd.ai` 不再作为默认对接口径,只用于识别历史链接。 `s3.yydd.ai` 不再作为默认对接口径,只用于识别历史链接。
### 6.3 历史混合区 ### 6.3 品牌资源迁移门禁
存储切换不能只核对素材预览。每次迁移或域名切换必须同时核对:
- `SystemSettings.headerLogoUrl` 与 `loginLogoUrl`
- 首页 Header、登录页、注册页、完善资料页
- `/api/favicon`
- 后台“恢复默认”值与 Prisma 默认值
- 管理员 GET → 不修改 Logo → PUT 往返不会把 `/api/proxy-media?...` 写回数据库
上述配置只能使用镜像内 `/brand/...` 或当前 `https://s3.jiansun.vip/gongxiangpan/...`;旧域名仅允许历史素材只读兼容。
### 6.4 历史混合区
`/gongxiangpan/uploads/` 保留历史兼容,不再作为新展示图默认路径。 `/gongxiangpan/uploads/` 保留历史兼容,不再作为新展示图默认路径。
@ -152,7 +164,7 @@ ssh root@192.168.58.20 "docker inspect minio --format '{{json .Mounts}}'"
核对 uploads 目录: 核对 uploads 目录:
```powershell ```powershell
ssh root@192.168.58.20 "ls -la /mnt/PoolOne/apps/youyidian-3d/uploads | head -20" ssh root@192.168.58.20 "ls -la /mnt/PoolOne/youyidian-uploads | head -20"
``` ```
核对资源域名解析: 核对资源域名解析:

View File

@ -0,0 +1,381 @@
# 生产数据库安全增量迁移 SOP(2026-08-12)
## 结论
本次应用发布依赖四份增量 migration。普通蓝绿部署脚本**只读验证,不会自动修改数据库**。只有独立 migrator 在同时拿到精确 Git 提交、数据库身份和备份凭证后,才允许执行 `prisma migrate deploy`。
目标 migration:
- `20260811_add_video_transcode_lease`
- `20260811_update_brand_logo_defaults`
- `20260812_add_product_package_upload_job_lease`
- `20260812_align_material_upload_idempotency_index`
它们只新增可空租约字段/索引、修改未来 Logo 默认值、精确把 `system_settings` 品牌两列中的退役 `s3.1080p.hk` 值改为本地品牌图,并把 `materials_uploadId_key` 重建为“`uploadId` 非空且素材未软删除”范围内的唯一 btree 索引。索引迁移会先检测未删除素材中的重复 `uploadId`,发现重复就直接失败,不会自行改写或删除素材。迁移不会批量修改素材数据,也不会覆盖当前可信桶中的合法自定义 Logo。没有自动 down migration。执行失败时停止发布;恢复依赖执行前的数据库备份。
## 强制前提
1. 使用将要发布的**精确 40 位 Git SHA**;在新建的隔离检出目录构建,不清理或复用现有工作树。
2. 已用 `pg_dump --format=custom` 创建生产数据库备份,并在隔离数据库做过真实恢复验证;备份产物与 receipt 一起只读挂载。
3. 明确生产数据库的配置 host/port/database/user,以及数据库实际返回的 `inet_server_addr()` / `inet_server_port()`;不允许从旧文档猜测。
4. 本轮首次旧镜像发布必须先把 runner 与 migrator 固定到同一个精确 SHA/digest;先按 `scripts/legacy-worker-bootstrap-deploy.ps1` 完成 Audit,再 Apply 进入外部实测 503 维护态,并证明旧容器已停止但仍保留可回退。只有这些证据全部成立,才允许设置 `CONFIRM_MAINTENANCE_WINDOW=NO_WRITES_DURING_SNAPSHOT`。
5. 不使用历史 `execute-migration.ps1/.sh`;它们不登记本轮 Prisma migration。
不能以“当前无人使用”“已经通知大家”或只改一个维护开关代替 503 和旧容器停止证据。正确顺序固定为:
1. 构建并固定同一 exact SHA/digest 的 runner 与 migrator;
2. legacy bootstrap `Audit`;
3. legacy bootstrap `Apply`,外部验收 503,确认旧容器停止且保留;
4. 在同一维护窗内运行本文的真实备份、隔离恢复、四条 migration 和迁移后审计;
5. 仅在所有门禁通过后,运行普通同 SHA 应用部署。
顺序不得改为“先 migration,后 bootstrap”。本文备份工具的生产前后摘要若有变化,说明维护窗内仍可能存在写入,会直接失败且不生成 receipt。
## 1. 构建并传输精确 runner 与 migrator
runner 和 migrator 必须来自同一个 `TARGET_SHA`,但它们是两个不同的镜像,必须分别记录 image ID/digest。普通发布脚本的 `-SkipDeploy` 只构建并推送 runner,**不会**构建 migrator。
### 1.1 先准备 runner
在正式维护窗之前,用普通发布脚本的生产参数和 `-SkipDeploy` 构建、推送 runner。脚本会固定 Git SHA、基础镜像 digest、最终 image ID 和 registry manifest digest,但不会连接生产机或改流量。记录它输出的 runner `repo@sha256:...` 和 image ID;生产机只能按该 digest 预拉取并复核,不能按可变 tag 猜测。
### 1.2 独立构建 migrator
在构建机使用一次性的隔离 clone。下面不会执行 `reset --hard` 或 `clean`,因此不会破坏构建机已有工作区:
```bash
set -Eeuo pipefail
TARGET_SHA=<40位目标SHA>
SOURCE_REPO=<仓库地址>
BUILD_ROOT="$(mktemp -d /root/gongxiangp2-migrator-build.XXXXXX)"
trap 'rm -rf -- "$BUILD_ROOT"' EXIT
git clone --no-checkout --branch main --single-branch "$SOURCE_REPO" "$BUILD_ROOT/repo"
git -C "$BUILD_ROOT/repo" checkout --detach "$TARGET_SHA"
test "$(git -C "$BUILD_ROOT/repo" rev-parse HEAD)" = "$TARGET_SHA"
test -z "$(git -C "$BUILD_ROOT/repo" status --porcelain --untracked-files=all)"
cd "$BUILD_ROOT/repo"
DOCKER_BUILDKIT=1 docker build \
--target migrator \
--build-arg VCS_REF="$TARGET_SHA" \
-t "youyidian-3d-migrator:$TARGET_SHA" .
docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' \
"youyidian-3d-migrator:$TARGET_SHA"
MIGRATOR_IMAGE_ID="$(docker image inspect --format '{{.Id}}' "youyidian-3d-migrator:$TARGET_SHA")"
MIGRATOR_BAKED_SHA="$(docker run --rm --network none --read-only \
--entrypoint cat "youyidian-3d-migrator:$TARGET_SHA" /app/DEPLOYMENT_COMMIT)"
test "$MIGRATOR_BAKED_SHA" = "$TARGET_SHA"
printf '%s\n' "$MIGRATOR_IMAGE_ID"
```
revision label 和 baked SHA 必须精确等于目标 SHA。`MIGRATOR_IMAGE_ID` 是独立于 runner 的值,二者不得混用。
### 1.3 通过已固定主机指纹的 SSH 传输 migrator
不能经未加密 HTTP registry 或临时关闭 SSH 主机校验。下面使用构建机上已经由发布脚本预置并核验过的生产 `known_hosts`;实际用户名、主机名和文件路径按现场受控配置填写:
```bash
set -Eeuo pipefail
MIGRATOR_IMAGE="youyidian-3d-migrator:$TARGET_SHA"
MIGRATOR_ARCHIVE="$(mktemp /root/gongxiangp2-migrator-${TARGET_SHA}.XXXXXX.tar)"
trap 'rm -f -- "$MIGRATOR_ARCHIVE"' EXIT
docker save --output "$MIGRATOR_ARCHIVE" "$MIGRATOR_IMAGE"
MIGRATOR_ARCHIVE_SHA256="$(sha256sum "$MIGRATOR_ARCHIVE" | awk '{print $1}')"
test -n "$MIGRATOR_ARCHIVE_SHA256"
SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes \
-o UserKnownHostsFile=/root/.ssh/gongxiangp2-production-known_hosts)
PRODUCTION_SSH=<受控用户@生产主机>
REMOTE_ARCHIVE="/root/gongxiangp2-migrator-${TARGET_SHA}.tar"
scp "${SSH_OPTS[@]}" -- "$MIGRATOR_ARCHIVE" "$PRODUCTION_SSH:$REMOTE_ARCHIVE"
ssh "${SSH_OPTS[@]}" "$PRODUCTION_SSH" bash -s -- \
"$REMOTE_ARCHIVE" "$MIGRATOR_ARCHIVE_SHA256" "$MIGRATOR_IMAGE" \
"$MIGRATOR_IMAGE_ID" "$TARGET_SHA" <<'PRODUCTION_VERIFY'
set -Eeuo pipefail
archive="$1"; expected_archive_sha="$2"; image="$3"; expected_image_id="$4"; expected_sha="$5"
trap 'rm -f -- "$archive"' EXIT
[ -f "$archive" ] && [ ! -L "$archive" ]
[ "$(stat -c '%a' "$archive")" = "600" ]
[ "$(sha256sum "$archive" | awk '{print $1}')" = "$expected_archive_sha" ]
docker load --input "$archive" >/tmp/gongxiangp2-migrator-load.log
[ "$(docker image inspect --format '{{.Id}}' "$image")" = "$expected_image_id" ]
[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$image")" = "$expected_sha" ]
[ "$(docker run --rm --network none --read-only --entrypoint cat "$image" /app/DEPLOYMENT_COMMIT)" = "$expected_sha" ]
PRODUCTION_VERIFY
```
`scp` 目标目录必须预先由 root 创建且不可由其他账号写入;传输完成后生产端会重新核对 archive SHA、migrator image ID、OCI revision 和 baked commit。任一不一致都在进入维护窗前停止。
## 2. 用工具生成可核验的备份凭证
禁止手写或补填 receipt。`scripts/create-verified-production-backup.mjs` 会在同一次受控运行中自动完成:
1. 只读核对生产库与隔离库的配置身份和数据库实际身份;
2. 在生产库读取迁移表、素材、用户、系统设置、产品包作业的有界摘要;
3. 用 `pg_dump --format=custom` 写入本次唯一的 `.partial` 文件,刷新到磁盘后原子改名;
4. 再读一次生产摘要。维护窗内前后摘要不同就停止,不生成 receipt;
5. 在第二道明确授权下,只清空符合命名规则的隔离库 `public` schema,再用 `pg_restore` 恢复同一备份;
6. 在 30 秒 statement timeout 和每表最多 32 条样本的边界内,对比关键表计数与样本摘要;
7. 全部一致后,原子生成 receipt,并输出 receipt 自身 SHA-256。
### 2.1 准备真正隔离的恢复目标
隔离目标必须同时满足:
- 数据库名匹配 `gongxiangp2_restore_<唯一小写后缀>`;
- 数据库名与生产库不同;
- 配置 host/port 与生产不同;
- `inet_server_addr()` / `inet_server_port()` 返回的实际 PostgreSQL 服务器也与生产不同;
- 只供本次恢复验证,账号允许在这个隔离库内重建 `public` schema,不拥有生产库写权限。
因此,**同一台生产 PostgreSQL 上另建一个数据库也不合格**。先进入维护窗口,停止会修改关键业务表的入口;工具的前后摘要只是近似验证,不能代替维护窗口。
### 2.2 准备两份 root-only 环境文件
实际密码、证书路径和连接值不写在命令、Git、聊天或运行日志里。由密码管理工具在生产机生成两份 `0600` 环境文件;下面只列变量名,不展示值:
- 生产连接:`PRODUCTION_PGHOST`、`PRODUCTION_PGPORT`、`PRODUCTION_PGDATABASE`、`PRODUCTION_PGUSER`,以及 `PRODUCTION_PGPASSWORD` 或 `PRODUCTION_PGPASSFILE`;
- 生产期望身份:`EXPECTED_DATABASE_HOST`、`EXPECTED_DATABASE_PORT`、`EXPECTED_DATABASE_NAME`、`EXPECTED_DATABASE_USER`、`EXPECTED_DATABASE_SERVER_ADDRESS`、`EXPECTED_DATABASE_SERVER_PORT`;
- 隔离连接:`ISOLATED_PGHOST`、`ISOLATED_PGPORT`、`ISOLATED_PGDATABASE`、`ISOLATED_PGUSER`,以及 `ISOLATED_PGPASSWORD` 或 `ISOLATED_PGPASSFILE`;
- 隔离期望身份:`EXPECTED_ISOLATED_DATABASE_HOST`、`EXPECTED_ISOLATED_DATABASE_PORT`、`EXPECTED_ISOLATED_DATABASE_NAME`、`EXPECTED_ISOLATED_DATABASE_USER`、`EXPECTED_ISOLATED_DATABASE_SERVER_ADDRESS`、`EXPECTED_ISOLATED_DATABASE_SERVER_PORT`。
若现场要求 TLS,可在各自前缀下增加 `PGSSLMODE`、`PGSSLROOTCERT`、`PGSSLCERT`、`PGSSLKEY`。`PGPASSFILE` 和证书若是宿主机路径,还要只读挂载到容器内,并在环境文件中填写容器内路径。
期望的 server address/port 必须来自当次现场只读查询。使用已安全注入的标准 `PG*` 环境变量执行,不在命令行传密码或连接 URL:
```bash
psql --no-psqlrc --no-align --tuples-only --set=ON_ERROR_STOP=1 <<'SQL'
SELECT current_database(), current_user,
COALESCE(inet_server_addr()::text, ''),
COALESCE(inet_server_port()::text, '');
SQL
```
### 2.3 先运行备份工具的只读审计
下面的网络必须能访问生产 PostgreSQL 和真正隔离的 PostgreSQL,但隔离账号不得拥有生产写权限:
```bash
set -Eeuo pipefail
TARGET_SHA=<40位目标SHA>
MIGRATOR_IMAGE="youyidian-3d-migrator:$TARGET_SHA"
PRODUCTION_BACKUP_ENV=/root/secure/gongxiangp2-production-backup.env
ISOLATED_RESTORE_ENV=/root/secure/gongxiangp2-isolated-restore.env
test "$(stat -c '%a' "$PRODUCTION_BACKUP_ENV")" = 600
test "$(stat -c '%a' "$ISOLATED_RESTORE_ENV")" = 600
docker run --rm \
--network <可同时访问两套数据库的受控网络> \
--env-file "$PRODUCTION_BACKUP_ENV" \
--env-file "$ISOLATED_RESTORE_ENV" \
-e EXPECTED_COMMIT_SHA="$TARGET_SHA" \
--entrypoint node \
"$MIGRATOR_IMAGE" \
scripts/create-verified-production-backup.mjs --audit
```
审计模式只执行客户端版本探测和两套身份查询,不创建 dump、不清空隔离库、不恢复、不生成 receipt。输出只包含去密后的身份;任何字段不一致都先修正环境文件,不能绕过。
### 2.4 显式创建并验证备份
为本次发布创建一个全新的宿主机目录。工具会拒绝符号链接、已有同名文件和非目录;不会复用历史 receipt:
```bash
set -Eeuo pipefail
BACKUP_BATCH="$(date -u +%Y%m%dT%H%M%SZ)-${TARGET_SHA}"
HOST_BACKUP_DIR="/root/gongxiangp2-db-backup-receipt/$BACKUP_BATCH"
HOST_RESULT_DIR="/root/gongxiangp2-migration-audit/$BACKUP_BATCH"
HOST_RESULT_FILE="$HOST_RESULT_DIR/verified-backup-result.txt"
MIGRATOR_UID="$(docker run --rm --entrypoint id "$MIGRATOR_IMAGE" -u)"
install -d -m 0750 -o "$MIGRATOR_UID" -g "$MIGRATOR_UID" "$HOST_BACKUP_DIR"
install -d -m 0750 -o "$MIGRATOR_UID" -g "$MIGRATOR_UID" "$HOST_RESULT_DIR"
test ! -L "$HOST_BACKUP_DIR"
test -z "$(find "$HOST_BACKUP_DIR" -mindepth 1 -maxdepth 1 -print -quit)"
test ! -e "$HOST_RESULT_FILE"
docker run --rm \
--network <可同时访问两套数据库的受控网络> \
--env-file "$PRODUCTION_BACKUP_ENV" \
--env-file "$ISOLATED_RESTORE_ENV" \
-e EXPECTED_COMMIT_SHA="$TARGET_SHA" \
-e BACKUP_OUTPUT_DIR=/backup \
-e ALLOW_VERIFIED_BACKUP=CREATE_AND_RESTORE \
-e ALLOW_ISOLATED_DATABASE_RESET=RESET_VERIFIED_ISOLATED_TARGET \
-e CONFIRM_MAINTENANCE_WINDOW=NO_WRITES_DURING_SNAPSHOT \
-v "$HOST_BACKUP_DIR:/backup" \
--entrypoint node \
"$MIGRATOR_IMAGE" \
scripts/create-verified-production-backup.mjs --create-and-restore \
| tee "$HOST_RESULT_FILE"
```
只有末尾出现 `Production backup was created, restored to the isolated server and verified.` 才算成功。保存工具输出的 `artifactPath`、`artifactSize`、`artifactSha256`、`receiptPath`、`receiptSize`、`receiptSha256`。receipt 自动绑定:
- 精确 `sourceCommit`;
- 生产 configured/server database fingerprint;
- `/backup/...dump` 容器路径、字节数、SHA-256 和起止时间;
- 隔离库 fingerprint、恢复时间和同一 artifact SHA-256;
- 生产前后与恢复库的关键表计数、有限样本摘要及整体摘要 SHA-256。
`pg_dump`、`pg_restore`、摘要查询、身份核对或原子写入任一步失败,都不会产生 `.receipt.json`。可能已生成但未通过恢复验证的 `.dump` 只能作为失败现场证据,不能交给 migrator。运行 migrator 时把同一个 `HOST_BACKUP_DIR` 以 `/backup:ro` 挂载;工具会重新读取完整 artifact 和 receipt,核对类型、路径、字节数、SHA-256 与有效期。receipt 最多允许 `BACKUP_MAX_AGE_MINUTES` 指定的时长(范围 1–1440 分钟)。
在退出维护窗前,先从 helper 的最终 JSON 结果提取本批次精确路径和 SHA;不要按时间猜文件名,也不要把上级备份目录整体挂进 migrator。下面的 `node` 只读取结果和 receipt,不接触数据库:
```bash
readarray -t BACKUP_EVIDENCE < <(node --input-type=commonjs - "$HOST_RESULT_FILE" "$HOST_BACKUP_DIR" <<'NODE'
const fs = require('node:fs')
const path = require('node:path')
const [resultPath, hostBackupDir] = process.argv.slice(2)
const resultText = fs.readFileSync(resultPath, 'utf8')
const marker = '\nProduction backup was created, restored to the isolated server and verified.\n'
const markerIndex = resultText.lastIndexOf(marker)
if (markerIndex < 0) throw new Error('helper success marker is missing')
const resultJson = resultText.slice(0, markerIndex).trim()
if (!resultJson.startsWith('{') || !resultJson.endsWith('}')) throw new Error('helper final JSON is missing')
const result = JSON.parse(resultJson)
const entries = fs.readdirSync(hostBackupDir, { withFileTypes: true })
const receiptNames = entries.filter((entry) => entry.isFile() && entry.name.endsWith('.receipt.json')).map((entry) => entry.name)
const dumpNames = entries.filter((entry) => entry.isFile() && entry.name.endsWith('.dump')).map((entry) => entry.name)
if (receiptNames.length !== 1 || dumpNames.length !== 1 || entries.length !== 2) {
throw new Error('batch directory must contain exactly one regular receipt and one regular dump')
}
for (const name of [receiptNames[0], dumpNames[0]]) {
const candidate = path.join(hostBackupDir, name)
const stat = fs.lstatSync(candidate)
if (!stat.isFile() || stat.isSymbolicLink()) throw new Error('backup evidence must be regular non-symlink files')
}
const receiptPath = path.join(hostBackupDir, receiptNames[0])
const dumpPath = path.join(hostBackupDir, dumpNames[0])
const receiptBytes = fs.readFileSync(receiptPath)
const receipt = JSON.parse(receiptBytes.toString('utf8'))
const crypto = require('node:crypto')
const sha256 = (bytes) => crypto.createHash('sha256').update(bytes).digest('hex')
const receiptSha256 = sha256(receiptBytes)
const dumpStat = fs.lstatSync(dumpPath)
const dumpHash = crypto.createHash('sha256')
const dumpStream = fs.createReadStream(dumpPath)
dumpStream.on('data', (chunk) => dumpHash.update(chunk))
dumpStream.once('error', (error) => { throw error })
dumpStream.once('end', () => {
const dumpSha256 = dumpHash.digest('hex')
if (
result.receiptPath !== `/backup/${receiptNames[0]}`
|| result.receiptSize !== receiptBytes.length
|| result.receiptSha256 !== receiptSha256
|| result.artifactPath !== `/backup/${dumpNames[0]}`
|| result.artifactSize !== dumpStat.size
|| result.artifactSha256 !== dumpSha256
|| receipt.backup.artifactPath !== result.artifactPath
|| receipt.backup.artifactSize !== result.artifactSize
|| receipt.backup.artifactSha256 !== result.artifactSha256
) throw new Error('helper output, receipt and artifact evidence do not match')
process.stdout.write(`${receiptNames[0]}\n${receiptSha256}\n`)
})
NODE
)
test "${#BACKUP_EVIDENCE[@]}" -eq 2
BACKUP_RECEIPT_BASENAME="${BACKUP_EVIDENCE[0]}"
EXPECTED_BACKUP_RECEIPT_SHA256="${BACKUP_EVIDENCE[1]}"
test -n "$BACKUP_RECEIPT_BASENAME"
test -n "$EXPECTED_BACKUP_RECEIPT_SHA256"
```
> 当前状态(2026-08-12):代码和伪 PostgreSQL 测试已就绪,但尚未对真实生产库执行 `pg_dump`、隔离恢复或任何 migration。本文步骤不是已完成证明。
## 3. 先做只读审计
下例只展示参数结构;真实值从生产现场读取,不写进 Git:
```bash
PREFLIGHT_AUDIT_FILE="$HOST_RESULT_DIR/preflight-$(date -u +%Y%m%dT%H%M%SZ).json"
MIGRATION_AUDIT_FILE="$HOST_RESULT_DIR/applied-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$PREFLIGHT_AUDIT_FILE"
test ! -e "$MIGRATION_AUDIT_FILE"
docker run --rm \
--network youyidian-3d_app-network \
--env-file /mnt/PoolOne/apps/youyidian-3d/.env \
-e EXPECTED_COMMIT_SHA=<40位目标SHA> \
-e MIGRATION_AUDIT_OUTPUT="/audit/$(basename "$PREFLIGHT_AUDIT_FILE")" \
-v "$HOST_RESULT_DIR:/audit" \
--entrypoint node \
youyidian-3d-migrator:<40位目标SHA> \
scripts/safe-production-migrate.mjs --audit
test -f "$PREFLIGHT_AUDIT_FILE" && test ! -L "$PREFLIGHT_AUDIT_FILE"
test "$(stat -c '%a' "$PREFLIGHT_AUDIT_FILE")" = "600"
```
审计只挂载前一步已按 migrator UID 创建的本批次 `HOST_RESULT_DIR`,不挂载其 root 所有的上级目录。审计输出不得包含数据库密码。确认目标库、账号、服务器地址和当前 migration 状态,若出现失败 migration 或本轮以外的未执行 migration,立即停止。
先确认 migrator 容器内的 `node` 用户 UID,并预创建只属于本次发布的审计目录;目录必须可写,已存在的审计文件不得复用:
```bash
MIGRATOR_UID="$(docker run --rm --entrypoint id youyidian-3d-migrator:<40位目标SHA> -u)"
install -d -m 0750 -o "$MIGRATOR_UID" -g "$MIGRATOR_UID" /root/gongxiangp2-migration-audit/<发布批次>
```
## 4. 带备份凭证执行
把备份 receipt 只读挂载到容器,显式提供生产身份:
```bash
docker run --rm \
--network youyidian-3d_app-network \
--env-file /mnt/PoolOne/apps/youyidian-3d/.env \
-e ALLOW_PRODUCTION_MIGRATIONS=APPLY \
-e EXPECTED_COMMIT_SHA=<40位目标SHA> \
-e EXPECTED_DATABASE_HOST=<DATABASE_URL中的主机> \
-e EXPECTED_DATABASE_PORT=<DATABASE_URL中的端口> \
-e EXPECTED_DATABASE_NAME=<数据库名> \
-e EXPECTED_DATABASE_USER=<数据库账号> \
-e EXPECTED_DATABASE_SERVER_ADDRESS=<inet_server_addr结果> \
-e EXPECTED_DATABASE_SERVER_PORT=<inet_server_port结果> \
-e BACKUP_RECEIPT_PATH="/backup/$BACKUP_RECEIPT_BASENAME" \
-e BACKUP_ARTIFACT_ROOT=/backup \
-e BACKUP_MAX_AGE_MINUTES=240 \
-e EXPECTED_BACKUP_RECEIPT_SHA256="$EXPECTED_BACKUP_RECEIPT_SHA256" \
-e MIGRATION_AUDIT_OUTPUT="/audit/$(basename "$MIGRATION_AUDIT_FILE")" \
-v "$HOST_BACKUP_DIR:/backup:ro" \
-v "$HOST_RESULT_DIR:/audit" \
"$MIGRATOR_IMAGE" --apply
```
工具会在修改数据库前先验证审计目录可执行“新建→原子改名→删除”,再依次完成:核对镜像 SHA、配置端口与数据库实际端口、核对机器生成的 receipt 和完整备份产物、拒绝未知待执行/失败 migration、执行 `prisma migrate deploy`,最后核对四条 migration 的数据库 checksum、完成状态、字段类型/可空性、租约索引的精确列顺序、`materials_uploadId_key` 的唯一性/btree/单列/有效就绪状态/active-only 谓词、Logo 默认值,以及每条现存 Logo 地址均为本地品牌图或当前可信桶地址,并原子写审计 JSON。
apply 成功后立即固定审计文件本身;后续普通部署只能读取这一个文件,不能手工重写:
```bash
test -f "$MIGRATION_AUDIT_FILE" && test ! -L "$MIGRATION_AUDIT_FILE"
chmod 0600 "$MIGRATION_AUDIT_FILE"
MIGRATION_AUDIT_SHA256="$(sha256sum "$MIGRATION_AUDIT_FILE" | awk '{print $1}')"
test -n "$MIGRATION_AUDIT_SHA256"
```
## 5. 放行条件
只有以下全部成立,才进入应用首次安全上线步骤:
- `applied.json` 的 `verified=true`;
- 四条 target migration 均 `completed=true`,source/database checksum 一致;
- `_prisma_migrations.finished_at` 非空、`rolled_back_at` 为空;
- 视频租约列/索引、任务详情作业租约列/索引、active-only `uploadId` 唯一索引、品牌默认值精确匹配;
- 再次运行 `--audit` 仍通过;
- 备份 receipt、迁移审计文件、目标 SHA 和 migrator image ID 被记录在当次发布记录中。
任意一项不成立:不部署应用、不手工修改 `_prisma_migrations`、不批量改业务状态;先保留现场并依据备份制定恢复方案。
## 6. 用同一 SHA 执行普通蓝绿部署
保持 bootstrap 维护入口和全局部署锁不变,使用第 1 步固定的 runner,并把第 4 步真实生成的审计文件与 SHA-256 显式交给普通部署脚本:
```powershell
.\scripts\git-incremental-deploy.ps1 `
-Target production `
-Commit '<与 runner/migrator 完全相同的 40 位 SHA>' `
-MigrationAuditFile '/root/gongxiangp2-migration-audit/<发布批次>/applied-<UTC时间>.json' `
-MigrationAuditSha256 '<该文件的 64 位小写 SHA-256>' `
-KeepOldImages
```
脚本会在候选创建前核对审计文件的路径、普通文件属性、`0600` 权限和哈希;再把其中的数据库实际身份与候选容器真实连接逐项比较。切流前还必须通过 S3 认证、数据库配置 Logo 与镜像内品牌图完整解码。缺少审计参数、审计不属于同一提交、数据库身份变化或任一资源门禁失败,都不得切流。

View File

@ -49,7 +49,7 @@
- 视频大文件 - 视频大文件
- 压缩包 - 压缩包
- 源文件下载 - 源文件下载
- 生产机 uploads 源文件目录:`/mnt/PoolOne/apps/youyidian-3d/uploads` - 生产机 uploads 源文件独立 ZFS 数据集:`/mnt/PoolOne/youyidian-uploads`
## 4. 历史缓存策略 ## 4. 历史缓存策略

View File

@ -9,7 +9,7 @@
- 服务端访问 MinIO 继续走内网 `192.168.58.20:9000`。 - 服务端访问 MinIO 继续走内网 `192.168.58.20:9000`。
- 展示型图片统一写入 `/gongxiangpan/public-display/`。 - 展示型图片统一写入 `/gongxiangpan/public-display/`。
- `/gongxiangpan/uploads/` 保留为混合区,不再默认承担展示资源职责。 - `/gongxiangpan/uploads/` 保留为混合区,不再默认承担展示资源职责。
- 源文件 / 下载型文件默认落在生产机 uploads:`/mnt/PoolOne/apps/youyidian-3d/uploads`。 - 源文件 / 下载型文件默认落在生产机独立 ZFS uploads 数据集:`/mnt/PoolOne/youyidian-uploads`。
- `1080p.hk` 旧桶只用于极少数老素材兼容,不作为新素材写入目标。 - `1080p.hk` 旧桶只用于极少数老素材兼容,不作为新素材写入目标。
- `180.97.215.57` 已停用,不再作为正式资源入口或缓存入口。 - `180.97.215.57` 已停用,不再作为正式资源入口或缓存入口。
@ -27,7 +27,8 @@
### 3.1 公网用户 ### 3.1 公网用户
- 打开网站:走 `https://sc.jiansun.vip` - 打开网站:走 `https://sc.jiansun.vip`
- 看图:走 `https://s3.jiansun.vip/gongxiangpan/public-display/...` - 看展示图:走 `https://s3.jiansun.vip/gongxiangpan/public-display/...`
- 看视频预览 / 片段 / 封面:走 S3 中对应的媒体 URL,不要求位于 `public-display`
- 下载大文件:继续走业务下载接口或原有下载链路 - 下载大文件:继续走业务下载接口或原有下载链路
### 3.2 内网服务端 ### 3.2 内网服务端
@ -49,7 +50,9 @@
| 白底图预览 | `/gongxiangpan/public-display/white-background/...` | 是 | | 白底图预览 | `/gongxiangpan/public-display/white-background/...` | 是 |
| 任务-详情预览 | `/gongxiangpan/public-display/task-detail/...` | 是 | | 任务-详情预览 | `/gongxiangpan/public-display/task-detail/...` | 是 |
| 通用预览 / 展示图 | `/gongxiangpan/public-display/materials/...` | 是 | | 通用预览 / 展示图 | `/gongxiangpan/public-display/materials/...` | 是 |
| Header / 登录 Logo | 仓库内置 `/brand/...`;可选当前桶 `gongxiangpan/由壹点AI/...` | 是 |
| 历史混合对象 | `/gongxiangpan/uploads/...` | 否,默认不按展示缓存处理 | | 历史混合对象 | `/gongxiangpan/uploads/...` | 否,默认不按展示缓存处理 |
| 视频预览 / 片段 / 封面 | S3 `gongxiangpan/由壹点AI/...`(不走 `public-display`) | 否,按媒体预览规则访问 |
| 压缩包 / 视频源文件 / 大下载对象 | 生产机 uploads 目录 | 否 | | 压缩包 / 视频源文件 / 大下载对象 | 生产机 uploads 目录 | 否 |
## 5. 展示资源规则 ## 5. 展示资源规则
@ -60,10 +63,10 @@
- 现有素材库预览图 - 现有素材库预览图
- 现有 `product-package/*-preview` 图片对象 - 现有 `product-package/*-preview` 图片对象
### 5.2 明确不作为展示资源处理的对象 ### 5.2 不纳入 `public-display` 展示图/缓存规则的对象
- 视频预览大文件 - 视频预览大文件(对象仍在 S3,通过媒体预览 URL 访问)
- 视频源文件 - 视频源文件(canonical 源仍在生产机 uploads/NAS)
- 压缩包 - 压缩包
- 其它下载型大文件 - 其它下载型大文件
@ -96,6 +99,12 @@
- `2320` / `2330` 不再返回正式应用或正式 MinIO 内容 - `2320` / `2330` 不再返回正式应用或正式 MinIO 内容
### 7.5 品牌资源
- Header、登录页和 favicon 在远端对象不可用时必须回退到镜像内 `/brand/...` 的可见图片,不能返回透明像素冒充成功。
- 系统设置数据库只能保存 `/brand/...` 或 `https://s3.jiansun.vip/gongxiangpan/...`,不得保存代理包装地址、旧域名或第三方地址。
- 后台“恢复默认”、Schema 默认值、设置行自动创建都必须回到本地品牌资源。
## 8. 推荐排查顺序 ## 8. 推荐排查顺序
当用户反馈“公网慢”时,先按下面顺序判断: 当用户反馈“公网慢”时,先按下面顺序判断:

View File

@ -1,309 +1,146 @@
# 视频转码配置说明 # 视频转码配置与运行说明
## 概述 > 本文描述当前代码实现。数据库字段 `videoPreview540Url` 是历史命名,当前默认产物实际为 720P,不能按字段名推断分辨率。
由壹点3D素材平台支持视频素材的自动转码功能,可以将上传的视频转换为适合在线预览的格式。 ## 当前结论
## 环境要求 - 上传的正式源视频优先保存在生产机 `UPLOAD_DIR` 对应的 `uploads` 目录;预览视频、短片和封面写入当前 S3 存储桶。
- 当前编码器是代码中固定的 `libx265`(H.265/HEVC),不是可由 `VIDEO_TRANSCODE_CODEC` 切换的配置项。
- 每个应用实例最多并行处理 1 条视频;数据库租约和 PostgreSQL 事务锁保证蓝绿容器之间同一素材只有一个有效执行者。
- 应用启动时立即扫描未完成任务,之后每 60 秒重扫。Redis 用于上传会话和其它业务,不承担视频转码队列持久化。
- 正式源视频在成功、失败、重试和部署中断后都必须保留。任务只允许清理自己创建的临时文件。
- 转码产物从磁盘流式上传到 S3,不能先整体读入 Node.js 内存。
- 本地 canonical 源视频直接只读使用,不复制到系统临时盘;远端兼容输入和任务产物只能写入 `UPLOAD_DIR` 所在的 NAS 工作目录,并在开始前按该挂载点空间 fail-closed。
- FFmpeg 使用 `spawn(..., shell: false)` 直接启动;超时、租约丢失或 drain 时终止进程组,任意非零退出码都算失败。
- 普通素材建档请求最多 128MiB;源文件达到 5MiB 必须走分片,单个分片最多 20MiB,逻辑源文件最多 10GiB。分片合并前必须确认 uploads/NAS 至少可容纳两份源文件并保留 2GiB 余量。
### 1. FFmpeg ## 实际转码产物
视频转码需要 FFmpeg 工具。在 Docker 环境中,FFmpeg 已经在 Dockerfile 中安装: | 产物 | 当前实现 | 数据库字段 |
| --- | --- | --- |
| 主预览 | 默认高度 720、H.265、`medium`、AAC 128k | `videoPreview540Url`(仅字段名保留 540) |
| 首页短片 | 默认 10 秒、高度 360、H.265、`ultrafast`、无音轨 | `videoPreviewClipUrl` |
| 卡片封面 | 默认高度 720、JPEG;用户已上传封面时不重复截图 | `videoCoverUrl` / `previewUrl` |
```dockerfile 只有三类必需产物均已生成并上传、当前数据库租约仍有效时,任务才可以写成 `done`。FFmpeg 任意非零退出码、输出缺失、上传失败或租约失效都不得伪装成完成。转码期间用户后补的手动封面或首页预览优先,完成事务只能填补当时仍为空的字段,不能用任务开始时的旧快照覆盖用户结果。
RUN apk add --no-cache ffmpeg
```
### 2. 环境变量配置 ## 当前有效环境变量
在 `docker-compose.yml` 或 `.env` 文件中配置以下环境变量: | 变量 | 默认值 | 作用与边界 |
| --- | --- | --- |
| `VIDEO_PREVIEW_RESOLUTION` | `720` | 主预览和自动封面的目标高度 |
| `VIDEO_CLIP_DURATION` | `10` | 首页短片秒数 |
| `VIDEO_PREVIEW_CRF` | `20` | 主预览 H.265 质量,越小画质越高、文件越大 |
| `VIDEO_CLIP_CRF` | `24` | 首页短片 H.265 质量 |
| `VIDEO_TRANSCODE_TIMEOUT` | `3600000` | 单个 FFmpeg 命令超时,单位毫秒 |
| `VIDEO_MAX_TEMP_FILE_SIZE` | `10737418240` | 临时输入最大字节数,默认 10GB |
| `VIDEO_TRANSCODE_THREADS` | `1` | FFmpeg/x265 线程上限;代码钳制在 1–2 |
| `VIDEO_TRANSCODE_MAX_CONCURRENT` | `1` | 共享应用容器的并发数;代码强制最高为 1 |
| `VIDEO_TRANSCODE_RUNTIME_DIR` | `UPLOAD_DIR/.video-transcode-runtime` | 当前容器 capability、drain 和 active marker 目录 |
```bash 旧文档曾列出的 `VIDEO_TRANSCODE_CODEC`、`VIDEO_TRANSCODE_PRESET`、`VIDEO_TRANSCODE_CRF`、`VIDEO_AUDIO_BITRATE` 和“Redis 持久化视频队列”并不是当前视频链路的有效控制面,不得据此调整生产。
# 视频转码编码器(推荐使用 libx265,压缩率更高)
VIDEO_TRANSCODE_CODEC=libx265
# 转码预设(速度与质量的平衡) ## 状态与恢复
# 可选值:ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
# 推荐:medium(平衡速度和质量)
VIDEO_TRANSCODE_PRESET=medium
# 恒定质量因子(CRF,控制视频质量) 合法状态为:
# 范围:0-51,数值越小质量越高,文件越大
# 推荐:22-28(22 为高质量,28 为较低质量)
VIDEO_TRANSCODE_CRF=22
# 音频比特率 - `pending`:已建档,等待领取或等待周期恢复。
# 推荐:128k-192k(128k 为标准质量,192k 为高质量) - `processing`:某个未过期数据库租约正在处理。
VIDEO_AUDIO_BITRATE=160k - `done`:数据库已写入完整预览结果。
``` - `failed`:已确认处理失败,需要用户明确重试或人工检查源文件。
### 3. Redis 配置(可选) 应用通过 `instrumentation.ts` 启动恢复器;恢复器每 60 秒扫描:
如果需要使用 Redis 管理转码队列(推荐用于生产环境),需要配置: - 没有有效租约的 `pending` 视频;
- 租约已过期的 `processing` 视频;
- 历史版本留下且心跳已静默的 `processing` 视频。
```bash 恢复扫描失败只记录错误,下一轮继续;扫描之间不重叠。不得使用“批量把所有 processing 改回 pending”的裸 SQL,这会绕过租约、源文件核验和在途 FFmpeg,可能造成重复转码或覆盖结果。
# 启用 Redis
USE_REDIS=true
# Redis 连接地址 ## 上传入口的内存与磁盘边界
REDIS_URL=redis://localhost:6379
```
**注意**: - `/api/materials/upload` 必须在解析 multipart 前校验 `Content-Type`、`Content-Encoding` 和 `Content-Length`;缺失长度返回 `411`,超过 128MiB 返回 `413`。
- 如果不配置 Redis,系统会使用内存队列(重启后队列会丢失) - 直传 `zipFile` 必须小于 5MiB。达到 5MiB 后走 `init → chunk → complete → existingFilePath → 建档`,大文件网络传输仍使用分片并发,不会被普通建档的内存处理槽串行化。
- 生产环境强烈推荐使用 Redis,确保转码任务不会因为重启而丢失 - 普通建档/图片处理每个应用实例最多同时处理 1 个请求,另保留 8 个短队列位置;繁忙时返回带 `Retry-After` 的 `503`,避免多个大 multipart 与 Sharp 同时占满 Node.js 内存。
- 单分片最多 20MiB;初始化会话记录固定分片布局,每个位置的期望字节数必须精确匹配。断点续传复用原会话布局;服务器边读边计数,超限立即取消并删除半文件,不能靠大量“各自合法”的分片突破会话总大小。
- 服务器在 multipart 解析前进入最多 8 路的分片槽,并在每块落盘前再次确认 NAS 至少保留 2GiB 安全余量;每次请求先写唯一 partial,完整校验后原子替换 `chunk-N`。这与浏览器最高并发一致,可继续利用 300M 带宽,同时避免解析内存和重复请求写盘竞态。
- `init` 拒绝超过 10GiB 的逻辑源文件;无法确认 uploads/NAS 剩余空间时保守返回 `503`,空间不足返回 `507`,不得继续写盘。`complete` 全局只合并 1 个文件,合并前再次要求可用空间至少为文件大小加 2GiB;写入唯一 partial、同步并原子改名,异常必清半成品。
- `complete` 必须幂等:同一用户的成功结果即使在公网响应途中丢失,也能从受控元数据恢复归属并返回同一个 merged 文件,不得因分片已清理而要求整文件重传。
- 买家秀和小红书批次逐条提交,避免多个图片 multipart 同时解码;小红书附加图片必须进入 `displayImages`,不能静默丢弃。
- 本机稳定代理对主上传精确限制为 `129m`、分片精确限制为 `21m`、其余上传 JSON 子路由限制为 `2m`,相关上传路由关闭 `proxy_request_buffering`。这样活动上传会持续建立到应用容器的连接,蓝绿 drain 不会把仍在 Nginx 缓冲但尚未转发的请求误判为空闲。
## 转码流程 ## API 权限与用途
### 1. 上传视频 ### 查询单条状态
用户上传视频素材时,系统会:
1. 保存原始视频文件到 S3
2. 创建素材记录,设置 `videoProcessingStatus = 'pending'`
3. 将转码任务加入队列
### 2. 转码处理
转码队列会自动处理待转码的视频:
1. 从队列中取出任务
2. 下载原始视频
3. 使用 FFmpeg 转码:
- 生成 540p 预览视频(`videoPreview540Url`)
- 生成 10 秒预览片段(`videoPreviewClipUrl`)
- 提取视频封面(`videoCoverUrl`)
4. 上传转码后的文件到 S3
5. 更新素材记录,设置 `videoProcessingStatus = 'completed'`
### 3. 转码失败处理
如果转码失败:
1. 系统会记录错误信息到 `transcodeLastError` 字段
2. 增加重试次数(`transcodeAttempts`)
3. 如果重试次数超过限制(默认 3 次),标记为 `failed`
4. 用户可以在前端手动重试转码
## API 接口
### 查询转码状态
```http ```http
GET /api/materials/transcode?materialId=xxx GET /api/materials/transcode?materialId=<素材ID>
``` ```
**响应示例**: - 必须登录并拥有该素材的查看权限。
```json - 普通用户看不到其他素材 ID、全队列详情或后端错误全文。
{ - 首页只做只读轮询,不会因为看到队列暂时为空就静默发起强制重试。
"status": "processing",
"progress": 45,
"message": "正在转码中..."
}
```
### 启动转码任务 ### 明确启动或重试
```http ```http
POST /api/materials/transcode POST /api/materials/transcode
Content-Type: application/json Content-Type: application/json
{ {
"materialId": "xxx" "materialId": "<素材ID>",
"forceRetry": true
} }
``` ```
**响应示例**: - 普通用户只能操作自己的素材;系统管理员可按权限处理其它素材。
```json - 正在处理的素材返回 `409`,客户端不能绕过有效租约并行重试。
{ - 自动恢复由后台调度器负责,不依赖管理员打开这个接口。
"message": "转码任务已启动",
"status": "processing"
}
```
## 转码参数说明 ## 发布与数据安全门禁
### VIDEO_TRANSCODE_CODEC 生产发布前必须:
**libx264**(H.264): 1. 先构建并固定同一 40 位 Git SHA 的 runner、migrator、镜像 digest 与生产机 image ID;此时不启动候选、不改数据库。
- 优点:兼容性好,几乎所有设备都支持 2. 第一次从不支持新 drain 协议的旧镜像升级必须使用独立的 `scripts/legacy-worker-bootstrap-deploy.ps1`。先执行只读 `Mode=Audit`;只有同时提供 `-Apply -AllowLegacyDrainBootstrap` 才能进入已证明的 503 维护交接,不能删除门禁强推。
- 缺点:文件较大 3. 只有旧容器已停止并保留、503 维护入口已验证后,才在同一维护窗执行生产备份、隔离恢复验证及四份 additive migration;部署脚本不会偷偷迁移数据库,也不能以“当前没人使用”代替维护门禁。
- 适用场景:需要最大兼容性的场景 4. 候选容器切流前只读确认租约列、索引、品牌默认值、uploadId 幂等索引和全部 migration checksum 精确匹配。
5. 常态发布时旧容器进入 drain 后,等待等待队列和 active marker 同时归零;状态未知时拒绝停止旧容器。
6. 保留旧容器、旧镜像、数据库备份和正式源视频,直到 canary 全链路通过。
7. 候选容器必须带 drain 启动,在数据库、品牌资源、能力和切流门禁全部通过后才恢复后台任务;切流提交后旧槽继续保持 drain。
8. 未完成独立 uploads/runtime/Redis 隔离前,通用增量脚本拒绝 `Target=shadow`;正式发布发现运行中的影子容器也必须先人工确认无共享 FFmpeg 风险。
9. 通用脚本禁止自动执行历史 direct 容器到稳定代理的拓扑迁移,也禁止用未验证旧镜像做 direct 回滚;检测到该现场时必须在任何切流或停容器前 fail-closed,改走单独受控 SOP。
**libx265**(H.265/HEVC): ### 首次旧镜像过渡的职责边界
- 优点:压缩率高,文件小,画质好
- 缺点:编码速度较慢,部分老设备不支持
- 适用场景:追求高画质和小文件的场景(推荐)
### VIDEO_TRANSCODE_PRESET 一次性 bootstrap 只负责旧直连现场到稳定 503 维护入口的安全交接,不负责候选发布:
速度从快到慢: - 必须固定旧容器完整 ID、旧 image ID/RepoDigest、旧容器网络 upstream、完整 `docker port` 端点集合、uploads ZFS source/target/statfs,以及目标镜像 digest/image ID/40 位 Git SHA。
- `ultrafast`:最快,质量最差 - 在 deployment flock 与 proxy reconciliation flock 内重复核验全部 pins;发现已有蓝绿槽、稳定代理、影子容器或状态歧义立即退出。
- `superfast`:很快,质量较差 - 通过 IPv4/IPv6 规则阻断新连接,让已建立连接自然退出;同时检查宿主 `ss` 与通过宿主 `nsenter` 进入固定旧容器 network namespace 后的服务端口 3000,两侧都必须连续采样两次 `ESTABLISHED=0`。不能只看宿主到容器 IP 的连接,因为 Docker DNAT 且未启用 userland-proxy 时可能漏掉仍在上传的公网连接。
- `veryfast`:快,质量一般 - 冻结前后都只读核对视频 `pending/processing=0`、任务详情 job `queued/processing=0`,并列出和拒绝任何 `taskStatus=uploading` 素材;同时核对旧容器和整台宿主机没有 FFmpeg。
- `faster`:较快,质量较好 - 先预置候选蓝槽的两类 drain marker,但绝不写 capability;真实 capability 只能由目标容器用当前实例 ID 发布。
- `fast`:快,质量好 - `docker pause` 后再次用固定目标镜像的 Node 单次入口查库;确认 idle 后只能用 `docker stop --signal SIGKILL --time 0` 停止,避免普通 stop 为发送优雅退出信号而恢复 paused 容器、让旧用户代码短暂重跑。停止状态必须精确为 `exited`,并再次用固定目标镜像查数据库、检查宿主机无 FFmpeg,全部通过后才改名保留旧容器。不得删除旧容器或旧镜像。
- `medium`:平衡(推荐) - 成功只表示入口处于明确 503 维护状态。随后仍需运行普通 `git-incremental-deploy.ps1 -Target production -Commit <同一SHA>`;所有候选门禁照常执行。
- `slow`:慢,质量很好 - 自动恢复只允许在入口仍是 `legacy-bootstrap-maintenance` 且没有运行中的蓝绿容器时进行。候选可能接收过流量时保持维护状态并人工判定,不能盲启旧容器。
- `slower`:很慢,质量极好
- `veryslow`:最慢,质量最好
**推荐配置**: 完整取证命令、PowerShell 参数模板和恢复命令见《视频预览任务中断事故复盘(2026-08-11)》的“首次生产过渡门禁”。
- 开发环境:`fast` 或 `faster`(快速测试)
- 生产环境:`medium`(平衡速度和质量)
- 高质量需求:`slow`(追求极致画质)
### VIDEO_TRANSCODE_CRF 验收一条小视频时需要同时确认:
CRF(Constant Rate Factor)控制视频质量: - 数据库状态为 `done`,三类预览 URL 均非空;
- `0`:无损(文件极大,不推荐) - 三个 S3 对象实际 HEAD 200、类型和大小正确,浏览器可解码;
- `18-22`:高质量(推荐用于重要视频) - 卡片在 15 天保护期内仍显示处理状态,并在完成后自动出现封面;
- `23-28`:标准质量(推荐用于一般视频) - `uploads` 中 canonical 源视频仍存在,字节数未变化;
- `29-35`:较低质量(不推荐) - 当前只有一个有效租约和一个 FFmpeg 进程;
- `51`:最低质量(不推荐) - 部署过程中有活动转码时,旧容器没有被提前停止。
**推荐配置**: ## 故障排查顺序
- 高质量:`20-22`
- 标准质量:`23-25`(推荐)
- 节省空间:`26-28`
### VIDEO_AUDIO_BITRATE 1. 查素材的 `videoProcessingStatus`、租约 token/过期时间、三阶段进度和 `transcodeLastError`。
2. 核对 canonical 源文件是否存在、大小是否与数据库一致,并用 FFprobe 验证可解析。
3. 查当前容器 active marker、drain 状态、FFmpeg 进程和启动恢复日志。
4. 查三个预览对象是否存在;孤立对象不能在数据库引用不明时直接删除。
5. 只有源文件安全、没有有效租约且没有在途 FFmpeg 时,才允许按精确素材 ID 重试。
音频比特率控制音频质量: 详细事故原因、生产证据和首次上线步骤见《视频预览任务中断事故复盘(2026-08-11)》。
- `96k`:低质量(不推荐)
- `128k`:标准质量
- `160k`:较高质量(推荐)
- `192k`:高质量
- `256k`:极高质量(文件较大)
## 生产环境配置示例
### 正式系统(docker-compose.yml)
```yaml
environment:
# 视频转码配置
- VIDEO_TRANSCODE_CODEC=libx265
- VIDEO_TRANSCODE_PRESET=medium
- VIDEO_TRANSCODE_CRF=22
- VIDEO_AUDIO_BITRATE=160k
# Redis 配置(可选)
- USE_REDIS=true
- REDIS_URL=redis://redis:6379
```
### 影子系统(docker-compose.shadow.yml)
影子系统已经配置了完整的视频转码环境,包括:
- 独立的 Redis 实例(端口 6380)
- 完整的转码环境变量
- 独立的 uploads 目录
## 监控和排查
### 查看转码统计
管理员可以访问以下 API 查看转码统计:
```http
GET /api/admin/video-transcode-stats
```
**响应示例**:
```json
{
"pending": 5,
"processing": 2,
"completed": 150,
"failed": 3,
"recentFailures": [
{
"id": "xxx",
"productModel": "产品名称",
"transcodeLastError": "FFmpeg 转码失败:...",
"transcodeAttempts": 3,
"createdAt": "2025-12-20T10:00:00Z"
}
]
}
```
### 常见问题
#### 1. 转码失败:FFmpeg not found
**原因**:Docker 镜像中没有安装 FFmpeg
**解决**:
1. 确认 Dockerfile 中已添加 `ffmpeg` 到 apk 安装列表
2. 重新构建镜像:`docker compose build`
3. 重启容器:`docker compose up -d`
#### 2. 转码速度很慢
**原因**:转码预设太慢(如 `slow` 或 `slower`)
**解决**:
1. 调整 `VIDEO_TRANSCODE_PRESET` 为 `medium` 或 `fast`
2. 重启容器使配置生效
#### 3. 转码后文件太大
**原因**:CRF 值太小(质量太高)
**解决**:
1. 调整 `VIDEO_TRANSCODE_CRF` 为 `25-28`
2. 或使用 `libx265` 编码器(压缩率更高)
#### 4. 转码队列卡住
**原因**:
- 转码任务失败但状态未更新
- Redis 连接失败
- 内存队列因重启丢失
**解决**:
1. 查看转码统计 API,确认是否有 `processing` 状态的任务
2. 如果有卡住的任务,可以手动重置状态:
```sql
UPDATE materials
SET "videoProcessingStatus" = 'pending'
WHERE "videoProcessingStatus" = 'processing';
```
3. 重启容器,队列会自动恢复
## 性能优化建议
### 1. 使用 Redis
生产环境强烈推荐使用 Redis:
- 转码任务持久化,重启不丢失
- 支持分布式部署(多个转码节点)
- 更好的队列管理和监控
### 2. 调整转码参数
根据实际需求调整参数:
- 如果用户主要在移动端观看:使用 `libx265` + `CRF 25`
- 如果追求极致画质:使用 `libx265` + `CRF 20` + `slow`
- 如果需要快速转码:使用 `libx264` + `CRF 23` + `fast`
### 3. 分离转码服务
如果转码任务很多,可以考虑:
- 部署独立的转码服务器
- 使用消息队列(如 RabbitMQ)管理任务
- 使用 GPU 加速转码(需要支持 NVENC 的 FFmpeg)
## 总结
视频转码功能已经完整实现,包括:
- ✅ FFmpeg 已在 Dockerfile 中安装
- ✅ 环境变量已在 docker-compose.yml 中配置
- ✅ 转码 API 已实现(查询状态、启动任务)
- ✅ 转码队列已实现(支持 Redis 和内存队列)
- ✅ 错误处理和重试机制已实现
- ✅ 管理员监控 API 已实现
**下一步**:
1. 重新构建 Docker 镜像(包含 FFmpeg)
2. 部署到正式环境
3. 测试视频上传和转码功能
4. 监控转码队列状态

View File

@ -0,0 +1,183 @@
# 视频预览任务中断事故复盘(2026-08-11)
## 结论
本次“新视频一直没有封面”不是浏览器、源视频格式、对象存储或内存爆掉造成的。直接原因是:发布切换只等待网页连接结束,却没有等待后台视频队列结束;旧容器在 FFmpeg 转码过程中被正常停止,新容器又没有在启动时自动接回数据库里的未完成任务。
页面还有第二层问题:15 天保护规则会有意隐藏 `videoSourceUrl`,首页却把“是否有源地址”当成启动状态轮询的条件。因此后台即使稍后生成了封面,已经打开的页面也可能一直显示旧占位,直到用户手动刷新。
排查还发现同一稳定性链路的第三层风险:旧上传入口会在没有总量门禁时直接解析 multipart 并整体复制源文件;Nginx 又允许约 1GB 请求并先缓冲到临时盘。大请求可能同时推高 Node.js 内存和系统盘占用,而且缓冲期间没有旧容器 upstream 连接,发布脚本会误以为上传已经结束。
## 生产证据
- 截图中的 11 条素材在 10:21–10:24 已成功建档并进入队列,不是漏上传。
- 旧容器 10:58 开始处理第一条,720P 已上传、10 秒片段处理到约 72% 时,容器在发布切换中被正常停止;不是 OOM,也不是进程崩溃。
- 新容器启动后没有执行视频自动恢复。数据库留下 1 条 `processing` 和 10 条 `pending`,心跳停在旧容器停止时刻。
- 11/11 正式源视频仍在,字节数与数据库一致,FFprobe 均可正常解析。
- 本轮按精确名单安全恢复了 16 条可恢复素材的封面、720P 预览和短片,并逐条核对数据库、对象存储与源文件;已知缺失源文件的模型 290 未被盲目重试或伪造结果。
## 误导链路与复发入口
| 文件 | 隐患 | 用户可见结果 |
| --- | --- | --- |
| `scripts/git-incremental-deploy.ps1` | 旧版 drain 只看 HTTP/TCP 连接,不知道后台 FFmpeg 和等待队列 | 网页请求结束后就停止旧容器,后台任务被截断 |
| `app/api/materials/upload/route.ts` | 上传后异步入队;入队失败时旧逻辑还会删除正式源视频 | 数据库有素材,但源文件可能被删,无法安全重做 |
| `lib/video-transcode-queue.ts` | 主要状态在单个 Node 进程内;数据库记录不是跨容器唯一执行租约 | 重启后丢队列,蓝绿重叠时还可能重复执行 |
| `app/api/materials/transcode/route.ts` | 恢复只在系统管理员访问接口时懒触发 | 普通上传者无法让新容器接回中断任务 |
| `instrumentation.ts` | 过去只恢复“任务-详情”后台任务,没有恢复视频 | 服务健康但视频队列静默停住 |
| `lib/video-processor.ts` | 旧逻辑在 S3 预览上传后、数据库完成回写前删除正式源视频 | 此窗口崩溃会留下“预览未入库、源文件也没了” |
| `lib/video-download-policy.ts` | 15 天保护期隐藏源视频和预览视频地址,这是正确的访问控制 | 页面不能依赖这些字段判断后台任务状态 |
| `app/page.tsx` | 首页过去只有 `videoSourceUrl` 存在才轮询;上传后只刷新一次 | 保护期内的卡片不会自动拿到新封面 |
| `components/home/MaterialCard.tsx` | 没源地址就隐藏进度,15 天遮罩还盖住失败/重试 | 用户只看到“15天后可查看详情”,看不到真正失败原因 |
| `app/api/materials/upload/route.ts` 与 `/chunk` | 旧版缺少请求总量、分片、逻辑源和写盘硬上限 | 大请求可能拖高内存或写满上传盘,半文件残留 |
| 本机 Nginx 上传规则 | 旧版允许约 1GB 且默认先缓冲请求 | 系统临时盘承压;发布 drain 看不到尚未转发的上传 |
| 买家秀/小红书批量上传 | 多批次曾并发发送;小红书附加图片虽发送却未被后端处理 | 多个大 multipart 同时占内存,附加图片静默丢失 |
## 永久修复
### 后台任务
- 用 PostgreSQL advisory transaction lock 加素材级 token/过期时间,保证蓝绿容器之间同一素材只有一个有效执行者。
- 成功、失败、续租、进度和释放都必须匹配当前租约;旧容器租约失效后会中止 FFmpeg,不能回写覆盖新容器结果。
- 应用启动立即恢复,随后每 60 秒重扫;数据库暂时不可用时下一轮继续,不再依赖管理员打开页面。
- 正常完成和入队失败都不再删除 canonical 源视频;只清理本次任务创建的临时副本。
- FFmpeg 不可用或三类输出不完整时明确失败,禁止把空预览误标为 `done`。
- 本地源视频直接只读使用,远端兼容输入和中间产物落在 uploads/NAS 的任务目录;磁盘状态未知或空间不足时不启动 FFmpeg,避免把大视频复制到系统盘。
- 预览视频、片段和封面从文件流式上传,不再整体读入 Node.js 内存;FFmpeg 任意非零退出、超时或被取消都明确失败。
- 转码完成时重新锁定并读取数据库最新值,只填仍为空的封面/首页预览字段,不覆盖用户在排队或转码期间后补的手动结果。
### 发布切换
- 新 drain 协议由当前容器实例写入 capability;共享盘残留的旧文件不能冒充当前容器。
- 旧容器有活动任务、状态未知或协议不匹配时一律拒绝停止;失败路径会恢复旧队列。
- 候选容器带 drain 启动,所有切流门禁通过后才允许恢复视频任务;切流提交后旧槽保持 drain,避免回滚清理逻辑把旧 worker 再次唤醒。
- 意外残留的 direct 容器也必须走同一套 capability、drain、active 验证;禁止直接 `rm -f`。未完成独立隔离前,影子部署入口 fail-closed。
- 第一次从旧镜像升级时旧容器不支持新协议,自动部署会按设计 fail-closed,必须按本文“首次过渡”单独执行,不能删门禁强推。
### 页面
- 首页根据 `pending/processing` 状态轮询,不再依赖被 15 天规则隐藏的源视频地址。
- 多卡片请求错峰,常规间隔改为 15 秒,429 按 `Retry-After` 退避。
- 正常排队可能超过 30 分钟,页面不再用固定时长停止轮询;只在完成、失败、素材离开当前列表或页面关闭时清理。
- 处理中、失败和重试按钮位于 15 天遮罩之上,并增加实时状态和进度语义。
- 完成后重新获取素材列表,从标准列表接口取得经过权限策略和 URL 重写的封面地址。
### 上传与内存保护
- 普通 multipart 在解析前强制验证长度,最大 128MiB;源文件达到 5MiB 必须走分片。
- 分片单块最大 20MiB,初始化时固定分片布局,每个位置的声明与实际字节数都必须精确匹配;超限或不一致立即删除半文件,不能用很多合法小请求突破会话总大小。断点续传始终沿用原会话布局,不会因公网/内网切换改用另一种切片方式。
- 服务端在解析 multipart 分片前先进入最多 8 路的处理槽,每次重试写入唯一临时文件,完整校验后原子替换 `chunk-N`;既保留上传吞吐,也避免并发解析占满内存或同一分片交错写坏。
- 逻辑源最大 10GiB;初始化前确认 uploads/NAS 可同时容纳分片与合并文件并保留安全余量,状态未知时拒绝开始。合并阶段全局单槽并再次复查“完整文件大小 + 2GiB”,写入唯一 partial、同步落盘后才原子改名,任何失败都会清理本次半成品。
- `complete` 是幂等操作:首次合并成功但公网响应丢失时,可依据当前用户的 `meta.json` 和已校验 merged 文件返回同一结果,不要求重新上传整个源文件。
- 普通图片/建档处理限制同时活跃数并设置有界等待队列;大文件分片网络上传保持原有并发能力。
- 买家秀和小红书改为逐批提交;小红书附加图片进入标准展示图字段,不再被静默忽略。
- 本机代理按主上传 `129m`、分片 `21m`、其余上传 JSON 子路由 `2m` 精确分层,关闭上传请求缓冲,使上传期间 upstream 连接始终可被蓝绿 drain 观察。
- 通用部署脚本检测到历史 direct 拓扑时在任何变更前拒绝继续;一次性 direct→proxy 迁移必须另走停新上传、确认连接和后台任务均为空的受控流程。
## 数据安全边界
- 任何重做前先确认正式源文件存在、大小一致且可解析;缺源素材不得盲排。
- 修复预览时先做不可变保全副本并核对 SHA256;验收通过前不清理保全文件。
- 不把 S3 HEAD 成功、健康接口 200 或数据库状态 `done` 单独当作成功;必须同时核对数据库字段、三类对象 HEAD、实际解码和 canonical 源文件。
- 新增 migration 只增加可空租约列和索引,不删除或改写现有素材数据;生产执行前仍需数据库备份和迁移记录核对。
## 首次生产过渡门禁
第一次从不支持 drain capability 的旧镜像升级时,不能删除普通发布脚本的门禁,也不能在共享盘伪造 capability 文件。唯一允许的入口是 `scripts/legacy-worker-bootstrap-deploy.ps1`;它默认只做审计,写操作必须同时显式提供 `-Apply` 和 `-AllowLegacyDrainBootstrap`。
### 过渡前准备
1. 明确获得生产迁移和部署授权,确定短时 503 维护窗口,并冻结唯一的 40 位目标 Git SHA。
2. 在触碰流量和数据库前,先用普通发布脚本的 `-SkipDeploy` 路径构建并推送同一 SHA 的 runner;migrator 必须按《生产数据库安全增量迁移 SOP(2026-08-12)》独立构建,并通过已固定主机指纹的 SSH 传到生产机。runner 与 migrator 分别记录自己的 image ID/digest、OCI revision 和 baked commit,二者必须来自同一 SHA,但不能误写成同一个 digest。在生产机只预拉取/导入镜像,不启动应用。维护窗口内不得再临时找镜像或改目标提交。
3. 在生产机只读取得下面全部现场值;任何一项为空、存在多值歧义或随后发生变化,都停止过渡:
```bash
legacy=youyidian-3d-material
docker inspect -f '{{.Id}}' "$legacy"
docker inspect -f '{{.Image}}' "$legacy"
docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$(docker inspect -f '{{.Image}}' "$legacy")"
docker inspect -f '{{with index .NetworkSettings.Networks "youyidian-3d_app-network"}}{{.IPAddress}}{{end}}:3000' "$legacy"
docker port "$legacy" 3000/tcp
findmnt -rn -o TARGET,SOURCE,FSTYPE -T /mnt/PoolOne/youyidian-uploads
stat -f -c '%i:%t' -- /mnt/PoolOne/youyidian-uploads
```
4. 把上述值和目标镜像值逐项填入一个 PowerShell splat;不要把密码、Token 或 `.env` 内容写入命令、文档或 Git:
```powershell
$bootstrap = @{
TargetCommit = '<40位小写Git SHA>'
LegacyContainerId = '<64位旧容器ID,不是短ID>'
LegacyImageId = 'sha256:<64位旧镜像ID>'
LegacyImageRepoDigest = '<旧镜像仓库>@sha256:<64位digest>'
LegacyUpstream = '<旧容器IPv4>:3000'
UploadsStatfsSignature = '<stat -f输出>'
PublishedEndpoints = '0.0.0.0:4320,[::]:4320' # 必须按 docker port 实际输出填写
TargetImageReference = '<目标镜像仓库>@sha256:<64位digest>'
TargetImageId = 'sha256:<生产机上的64位目标镜像ID>'
TargetImageRepoDigest = '<目标镜像仓库>@sha256:<64位digest>'
}
```
如果线上仍使用兼容挂载 `/mnt/PoolOne/apps/youyidian-3d/uploads`,还必须显式传入 `UploadsMount`、`UploadsMountTarget=/mnt/PoolOne` 和 `UploadsMountSource=PoolOne`;不得让脚本从旧容器自动推断挂载路径。
### 只读审计与维护交接
先执行只读审计。它会核对完整容器 ID、image ID/RepoDigest、当前 upstream、全部发布端点、ZFS source/target/statfs、目标镜像 digest/提交号,并列出数据库中所有非终态视频、任务详情 job 和 `taskStatus=uploading` 素材;任一计数非零即失败。
```powershell
.\scripts\legacy-worker-bootstrap-deploy.ps1 @bootstrap -Mode Audit
```
审计通过并再次确认维护窗口后,才允许进入一次性维护交接:
```powershell
.\scripts\legacy-worker-bootstrap-deploy.ps1 @bootstrap `
-Mode EnterMaintenance `
-Apply `
-AllowLegacyDrainBootstrap
```
该步骤在生产机同时持有 bootstrap deployment flock 和代理 reconciliation flock;用带唯一注释的 IPv4/IPv6 防火墙规则只阻断新连接,并同时取得两套连接证据:宿主侧 `ss`,以及宿主 `nsenter` 进入固定旧容器 network namespace 后对服务端口 3000 的 `ss`。两侧都必须连续两次 `ESTABLISHED=0`;只看宿主到容器 IP 的连接在 Docker DNAT 且未启用 userland-proxy 时可能漏掉仍在上传的公网连接。随后再次检查数据库三类非终态计数和宿主机/旧容器 FFmpeg,再 `docker pause` 旧容器并用固定目标镜像的非应用 Node 入口只读查询数据库,以关闭“检查完又启动任务”的竞态。全部通过后使用 `docker stop --signal SIGKILL --time 0`:普通 stop 会在发送优雅退出信号后恢复 paused 容器,存在旧用户代码短暂重跑的窗口,因此这里不能使用普通 stop。脚本必须确认旧容器精确进入 `exited`,再用固定目标镜像复查数据库三类计数并确认宿主机无 FFmpeg,才允许改名保存旧容器;旧镜像不删除。蓝槽的视频和任务详情 drain marker 会预先创建,但 capability 必须由新容器启动后自行发布。
成功后的入口会明确返回 HTTP 503、`Retry-After` 和 `X-Material-Proxy-Target: legacy-bootstrap-maintenance`。这是短维护交接点,不是部署成功。只有到此时,写请求和旧后台 worker 才被技术手段停止;不能以“当前没人使用”代替这个门禁,也不能在 bootstrap 前重建生产索引。
保持 503 维护状态,按《生产数据库安全增量迁移 SOP(2026-08-12)》依次执行:机器生成的生产备份、隔离库恢复、源库与恢复库有界摘要一致性验证、四份 additive migration,以及 migration checksum/列/索引/default 的只读后验审计。任何一步失败都停止,保留旧容器、旧镜像、备份和维护入口;不得继续切流。
上述数据库证据全部通过后,才用普通脚本发布同一个精确 SHA,并保留旧镜像:
```powershell
.\scripts\git-incremental-deploy.ps1 `
-Target production `
-Commit $bootstrap.TargetCommit `
-MigrationAuditFile '/root/gongxiangp2-migration-audit/<发布批次>/applied-<UTC时间>.json' `
-MigrationAuditSha256 '<审计文件的64位小写SHA-256>' `
-KeepOldImages
```
普通脚本仍必须独立验证候选镜像内置提交号、数据库 schema/migration、真实视频与任务详情 capability、预置 drain、健康接口和切流目标。bootstrap 不会创建候选容器,也不会绕过普通门禁。
### 失败恢复边界
- bootstrap 在维护交接提交前的可确认安全状态失败,会在同一进程内恢复精确旧容器 ID 和原直连入口;旧容器或旧镜像绝不 `rm`。一旦已发出 SIGKILL,但停止状态、停止后的数据库 idle 或宿主机无 FFmpeg 不能被证明,脚本会保持入口阻断、保留旧容器并要求人工恢复,不会在证据未知时自动重启旧 worker。
- 普通部署失败且入口仍明确是 `legacy-bootstrap-maintenance`、蓝绿容器都未运行时,可用相同 pins 执行下面的恢复命令。恢复脚本会再次核对保存容器的完整 ID/image ID/RepoDigest;只要候选可能已经接收流量,就拒绝自动恢复,避免两个应用世界同时写生产数据。
```powershell
.\scripts\legacy-worker-bootstrap-deploy.ps1 @bootstrap `
-Mode RestoreLegacy `
-Apply `
-AllowLegacyDrainBootstrap
```
- 如果恢复门禁拒绝执行,保留维护入口、保存的旧容器和候选现场,先人工判定真实代理目标及是否已有写请求;不能强制启动旧容器。
- 首个新容器上线后验证自动恢复、60 秒重扫、租约唯一性、源文件保留、健康接口和公网真实页面。再用一条小视频做 canary:上传、排队、生成三类预览、卡片自动出现封面、源文件仍在;任一步失败立即停止继续放量,不回滚 additive 数据列。
## 验收清单
- 服务重启前上传的未完成视频能被新容器自动接回。
- 同一素材不存在两个有效租约或两个 FFmpeg。
- 部署期间有后台任务时旧容器不会被停止。
- 部署期间有活动上传或分片写入时,旧容器不会被误判为空闲。
- 转码成功、失败、重试和部署中断后 canonical 源视频都仍存在。
- 15 天保护期不暴露源视频,但卡片仍显示处理/失败状态并在完成后自动出现封面。
- 已知缺源素材保持明确失败/待人工恢复,不进入批量自动重试。

View File

@ -14,19 +14,19 @@
生产 uploads 目录: 生产 uploads 目录:
```text ```text
/mnt/PoolOne/apps/youyidian-3d/uploads /mnt/PoolOne/youyidian-uploads
``` ```
容器挂载: 容器挂载:
```bash ```bash
-v /mnt/PoolOne/apps/youyidian-3d/uploads:/app/uploads -v /mnt/PoolOne/youyidian-uploads:/app/uploads
``` ```
部署脚本的边界: 部署脚本的边界:
- 只检查该目录是否存在。 - 在创建候选容器前验证该路径确实是预期 ZFS 数据集,不是系统盘上的同名目录壳。
- 只把该目录挂载进应用容器。 - 验证容器内 `/app/uploads` 与宿主机数据集的文件系统标识一致,再允许切流。
- 不对该目录执行 `rm`、`mv`、`cp`、`find -delete` 或清空操作。 - 不对该目录执行 `rm`、`mv`、`cp`、`find -delete` 或清空操作。
- 不改变该目录内已有源文件、视频文件、压缩包和上传临时数据。 - 不改变该目录内已有源文件、视频文件、压缩包和上传临时数据。
@ -58,7 +58,7 @@
- 新素材预览 / 展示文件:写入新 MinIO 桶,公开 URL 统一为 `https://s3.jiansun.vip/gongxiangpan/public-display/...`。 - 新素材预览 / 展示文件:写入新 MinIO 桶,公开 URL 统一为 `https://s3.jiansun.vip/gongxiangpan/public-display/...`。
- 旧 `1080p.hk` 桶:只用于极少数老素材兼容,不作为新素材写入目标。 - 旧 `1080p.hk` 桶:只用于极少数老素材兼容,不作为新素材写入目标。
- 源文件 / 下载型文件:落在生产机 uploads 目录,即 `/mnt/PoolOne/apps/youyidian-3d/uploads`。 - 源文件 / 下载型文件:落在生产机独立 ZFS uploads 数据集,即 `/mnt/PoolOne/youyidian-uploads`。
## 5. 构建机操作范围 ## 5. 构建机操作范围
@ -170,7 +170,7 @@ ssh root@192.168.58.20 'curl -fsS http://127.0.0.1:4320/api/health'
检查 uploads 目录是否还在: 检查 uploads 目录是否还在:
```bash ```bash
ssh root@192.168.58.20 'ls -la /mnt/PoolOne/apps/youyidian-3d/uploads | head -20' ssh root@192.168.58.20 'ls -la /mnt/PoolOne/youyidian-uploads | head -20'
``` ```
检查 Docker 镜像和容器: 检查 Docker 镜像和容器:

View File

@ -25,14 +25,13 @@
默认参数: 默认参数:
- Git 仓库:`http://git.1080p.hk/WuYongJie/gongxiangp2.git` - Git 仓库:`https://git.1080p.hk/WuYongJie/gongxiangp2.git`(只允许有效 TLS,不得降级到 HTTP)
- 分支:`main` - 分支:`main`
- 构建机:`192.168.40.206` - 构建机:`192.168.40.206`
- 生产机:`192.168.58.20` - 生产机:`192.168.58.20`
- 正式容器:`youyidian-3d-material` - 正式容器:`youyidian-3d-material`
- 正式端口:`4320` - 正式端口:`4320`
- 影子容器:`youyidian-3d-shadow` - 影子发布:当前通用脚本已 fail-closed 禁用;必须使用 uploads/runtime/Redis 完全隔离的专用入口后才能恢复。
- 影子端口:`5320`
`WuYongJie/gongxiangp2` 是当前唯一权威仓库。`lunanting/gongxiangp2` 仅作为历史仓库识别,不得再用于构建或发布。 `WuYongJie/gongxiangp2` 是当前唯一权威仓库。`lunanting/gongxiangp2` 仅作为历史仓库识别,不得再用于构建或发布。
@ -47,9 +46,10 @@
- PowerShell 5.1+。 - PowerShell 5.1+。
- PuTTY 工具:`plink.exe`、`pscp.exe`。 - PuTTY 工具:`plink.exe`、`pscp.exe`。
- 构建机 SSH 凭据,通过 `BUILD_SERVER_PASSWORD` 或 `-BuildServerPassword` 提供。 - 构建机 SSH 凭据,通过 `BUILD_SERVER_PASSWORD` 或 `-BuildServerPassword` 提供。
- Git 仓库认证,通过 `GIT_DEPLOY_USERNAME` / `GIT_DEPLOY_PASSWORD` 或 `GIT_AUTH_HEADER` 提供。 - 构建机、生产机的固定 SSH SHA256 主机指纹;OpenSSH 模式还要提供受控 `known_hosts` 文件。
- 维护机和构建机已配置只读 Git credential helper。部署脚本禁止把 Git 密码/Header 写入参数、临时脚本或进程命令行。
不要把 Git 密码写进仓库 URL,也不要把临时部署密码写入文档或提交到仓库。 不要把 Git 密码写进仓库 URL,也不要把临时部署密码写入文档或提交到仓库。HTTPS 证书过期、主机指纹不匹配或权威仓库不可验证时,必须先修基础设施,不能关闭校验强行发布。
### 2.2 部署命令 ### 2.2 部署命令
@ -65,7 +65,7 @@
.\scripts\git-incremental-deploy.ps1 .\scripts\git-incremental-deploy.ps1
``` ```
部署影子环境: 影子环境(当前禁止通过通用脚本执行;下面命令会明确失败):
```powershell ```powershell
.\scripts\git-incremental-deploy.ps1 -Target shadow .\scripts\git-incremental-deploy.ps1 -Target shadow
@ -85,7 +85,7 @@
## 3. 参数说明 ## 3. 参数说明
- `-Target`:部署目标,`production` 或 `shadow`。 - `-Target`:当前只允许 `production`;`shadow` 会在任何 Git/远程操作前 fail-closed。
- `-RepoUrl`:Git 仓库地址,默认使用 `WuYongJie/gongxiangp2`。 - `-RepoUrl`:Git 仓库地址,默认使用 `WuYongJie/gongxiangp2`。
- `-Branch`:部署分支,默认 `main`。 - `-Branch`:部署分支,默认 `main`。
- `-Commit`:指定提交;为空时部署目标分支最新提交。 - `-Commit`:指定提交;为空时部署目标分支最新提交。
@ -93,7 +93,8 @@
- `-NoCache`:Docker 构建不使用缓存。 - `-NoCache`:Docker 构建不使用缓存。
- `-KeepOldImages`:跳过自动 Docker 镜像清理。 - `-KeepOldImages`:跳过自动 Docker 镜像清理。
- `-AggressiveBuildCacheCleanup`:额外清理较旧构建缓存;默认不启用。 - `-AggressiveBuildCacheCleanup`:额外清理较旧构建缓存;默认不启用。
- `-SkipBuild` / `-SkipPush` / `-SkipDeploy`:只用于排障,不作为常规发布方式。 - `-SkipBuild` / `-SkipPush`:只允许同时配合 `-SkipDeploy` 做本地排障;真实生产发布禁止跳过构建或推送。
- `-SkipDeploy`:只构建/验证,不触碰生产流量。
## 4. 安全边界 ## 4. 安全边界
@ -102,7 +103,7 @@
- 不修改生产机 `.env` 文件。 - 不修改生产机 `.env` 文件。
- 不删除、不移动、不覆盖生产机 uploads 目录内容。 - 不删除、不移动、不覆盖生产机 uploads 目录内容。
- 不直接操作 PostgreSQL、Redis 或 MinIO 容器。 - 不直接操作 PostgreSQL、Redis 或 MinIO 容器。
- 生产机只替换应用容器,并保留 `/mnt/PoolOne/apps/youyidian-3d/uploads:/app/uploads` 挂载。 - 生产机只替换应用容器,并保留 `/mnt/PoolOne/youyidian-uploads:/app/uploads` 独立 ZFS 数据集挂载。
- 新容器健康检查失败时,脚本默认用旧镜像回滚应用容器。 - 新容器健康检查失败时,脚本默认用旧镜像回滚应用容器。
详细安全说明见:[部署脚本安全说明](./部署脚本安全说明.md)。 详细安全说明见:[部署脚本安全说明](./部署脚本安全说明.md)。
@ -114,8 +115,7 @@
```text ```text
production-<commit> # 正式环境确定版本 production-<commit> # 正式环境确定版本
production-latest # 正式环境最新版本 production-latest # 正式环境最新版本
shadow-<commit> # 影子环境确定版本 shadow-* # 历史命名;通用发布入口当前不再生成
shadow-latest # 影子环境最新版本
``` ```
Registry UI: Registry UI:
@ -193,7 +193,7 @@ ssh root@192.168.58.20 'curl -fsS http://127.0.0.1:4320/api/health'
### uploads 目录检查 ### uploads 目录检查
```bash ```bash
ssh root@192.168.58.20 'ls -la /mnt/PoolOne/apps/youyidian-3d/uploads | head -20' ssh root@192.168.58.20 'ls -la /mnt/PoolOne/youyidian-uploads | head -20'
``` ```
## 9. 版本历史 ## 9. 版本历史

View File

@ -44,5 +44,23 @@ export async function register() {
// 迁移未执行或数据库暂不可用时只记录;周期恢复已提前注册,会继续重试。 // 迁移未执行或数据库暂不可用时只记录;周期恢复已提前注册,会继续重试。
console.error('[Instrumentation] 任务-详情后台队列恢复失败:', error) console.error('[Instrumentation] 任务-详情后台队列恢复失败:', error)
} }
const videoRecoveryGlobal = globalThis as typeof globalThis & {
__videoTranscodeRecoveryScheduler?: {
initial: Promise<number>
}
}
if (!videoRecoveryGlobal.__videoTranscodeRecoveryScheduler) {
const { createVideoTranscodeRecoveryScheduler } = await import('./lib/video-transcode-recovery')
const scheduler = createVideoTranscodeRecoveryScheduler()
videoRecoveryGlobal.__videoTranscodeRecoveryScheduler = scheduler
try {
const recovered = await scheduler.initial
console.log(`[Instrumentation] 视频转码启动恢复完成,重新入队 ${recovered} 个任务`)
} catch (error) {
// 周期调度器已先注册;数据库瞬时不可用时,60 秒后会继续重试。
console.error('[Instrumentation] 视频转码启动恢复失败,已保留周期重试:', error)
}
}
} }
} }

View File

@ -0,0 +1,97 @@
/** @jest-environment node */
import { mkdir, mkdtemp, rm, symlink, writeFile } from 'fs/promises'
import os from 'os'
import path from 'path'
import {
readBoundedRegularFile,
readBoundedResponseBuffer,
} from '@/lib/bounded-file-reader'
describe('bounded-file-reader', () => {
let root: string
beforeEach(async () => {
root = await mkdtemp(path.join(os.tmpdir(), 'bounded-file-reader-'))
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
})
test('在分配 Buffer 前拒绝超限、申报不一致和符号链接文件', async () => {
const filePath = path.join(root, 'source.bin')
await writeFile(filePath, Buffer.from('12345'))
await expect(readBoundedRegularFile(filePath, {
maxBytes: 4,
label: '测试文件',
})).rejects.toThrow('超过 1MB 限制')
await expect(readBoundedRegularFile(filePath, {
maxBytes: 10,
expectedBytes: 4,
label: '测试文件',
})).rejects.toThrow('实际大小与上传记录不一致')
const symlinkTarget = path.join(root, 'source-link-target')
const symlinkPath = path.join(root, 'source-link')
await mkdir(symlinkTarget)
await symlink(symlinkTarget, symlinkPath, process.platform === 'win32' ? 'junction' : 'dir')
await expect(readBoundedRegularFile(symlinkPath, {
maxBytes: 10,
expectedBytes: 5,
label: '测试文件',
})).rejects.toThrow('不是可读取的普通文件')
})
test('普通文件仅在实际大小与申报一致时返回完整内容', async () => {
const filePath = path.join(root, 'source.bin')
await writeFile(filePath, Buffer.from('bounded'))
await expect(readBoundedRegularFile(filePath, {
maxBytes: 64,
expectedBytes: 7,
label: '测试文件',
})).resolves.toEqual(Buffer.from('bounded'))
})
test('远程响应无 Content-Length 时仍按可信大小硬停,且不使用 arrayBuffer', async () => {
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array([1, 2]))
controller.enqueue(new Uint8Array([3, 4]))
controller.close()
},
})
const response = new Response(stream)
await expect(readBoundedResponseBuffer(response, {
maxBytes: 4,
expectedBytes: 3,
label: '远程文件',
})).rejects.toThrow('响应超过申报大小')
})
test('远程响应先校验 Content-Length,并按申报大小返回单个 Buffer', async () => {
const response = new Response(Buffer.from('remote'), {
headers: { 'Content-Length': '6' },
})
await expect(readBoundedResponseBuffer(response, {
maxBytes: 10,
expectedBytes: 6,
label: '远程文件',
})).resolves.toEqual(Buffer.from('remote'))
const mismatched = new Response(Buffer.from('remote'), {
headers: { 'Content-Length': '6' },
})
await expect(readBoundedResponseBuffer(mismatched, {
maxBytes: 10,
expectedBytes: 5,
label: '远程文件',
})).rejects.toThrow('响应大小与上传记录不一致')
})
})

View File

@ -88,9 +88,10 @@ function createIndexedDbMock(): IDBFactory {
class MockXMLHttpRequest { class MockXMLHttpRequest {
static instances: MockXMLHttpRequest[] = [] static instances: MockXMLHttpRequest[] = []
static defaultStatus = 413
timeout = 0 timeout = 0
status = 413 status = MockXMLHttpRequest.defaultStatus
responseText = '' responseText = ''
upload = { upload = {
onloadstart: null as (() => void) | null, onloadstart: null as (() => void) | null,
@ -129,6 +130,7 @@ describe('chunkUpload', () => {
beforeEach(() => { beforeEach(() => {
jest.clearAllMocks() jest.clearAllMocks()
MockXMLHttpRequest.instances = [] MockXMLHttpRequest.instances = []
MockXMLHttpRequest.defaultStatus = 413
Object.defineProperty(global, 'indexedDB', { Object.defineProperty(global, 'indexedDB', {
configurable: true, configurable: true,
@ -236,4 +238,139 @@ describe('chunkUpload', () => {
expect(getOptimalConcurrency).not.toHaveBeenCalled() expect(getOptimalConcurrency).not.toHaveBeenCalled()
expect(MockXMLHttpRequest.instances).toHaveLength(1) expect(MockXMLHttpRequest.instances).toHaveLength(1)
}) })
test('断点续传沿用服务端既有分片布局,不受公网内网切换影响', async () => {
;(global.fetch as jest.Mock)
.mockResolvedValueOnce({
ok: true,
json: async () => ({ uploadId: 'upload-resume-layout' }),
})
.mockResolvedValueOnce({
ok: true,
json: async () => ({ message: '清理成功' }),
})
const file = new File(['hello'], 'source.psb', { type: 'application/octet-stream' })
MockXMLHttpRequest.defaultStatus = 500
await expect(chunkUpload(
file,
{ materialType: '任务-详情' },
{ chunkSize: 2, chunkConcurrency: 1, maxRetries: 1, retryDelay: 0 },
)).rejects.toThrow('失败')
await expect(getAllPendingUploads()).resolves.toEqual([
expect.objectContaining({
uploadId: 'upload-resume-layout',
chunkSize: 2,
totalChunks: 3,
}),
])
MockXMLHttpRequest.defaultStatus = 413
await expect(chunkUpload(
file,
{ materialType: '任务-详情' },
// 模拟网络探测从公网2字节布局切到内网4字节布局。
{ chunkSize: 4, chunkConcurrency: 1, maxRetries: 1, retryDelay: 0 },
)).rejects.toThrow('上传分片超过入口服务器限制')
const resumedRequest = MockXMLHttpRequest.instances[1]
expect(resumedRequest.headers['X-Chunk-Size']).toBe('2')
expect(resumedRequest.headers['X-Chunk-Index']).toBe('0')
expect(global.fetch).toHaveBeenCalledTimes(2)
})
test('特殊批次跳过 complete 时保留断点,建档确认后才删除', async () => {
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => ({ uploadId: 'upload-retained-special-batch' }),
})
MockXMLHttpRequest.defaultStatus = 200
const file = new File(['hello'], 'brand-vi.pdf', { type: 'application/pdf' })
const result = await chunkUpload(
file,
{ materialType: '品牌VI灵感' },
{
chunkSize: 5,
chunkConcurrency: 1,
skipComplete: true,
retainStateOnSkipComplete: true,
},
)
await expect(getAllPendingUploads()).resolves.toEqual([
expect.objectContaining({
uploadId: 'upload-retained-special-batch',
uploadedChunks: new Set([0]),
}),
])
await result.confirmMaterialCreated()
await expect(getAllPendingUploads()).resolves.toEqual([])
})
test('同一文件并发选择两次时拒绝第二条,避免共用 uploadId 覆盖分片', async () => {
;(global.fetch as jest.Mock).mockResolvedValue({
ok: false,
status: 400,
json: async () => ({ error: '测试结束首条上传' }),
})
const file = new File(['hello'], 'duplicate.psb', {
type: 'application/octet-stream',
lastModified: 123456,
})
const firstUpload = chunkUpload(
file,
{ materialType: '任务-详情' },
{ maxRetries: 1, retryDelay: 0 },
)
await expect(chunkUpload(
file,
{ materialType: '任务-详情' },
{ maxRetries: 1, retryDelay: 0 },
)).rejects.toThrow('正在上传,请勿在同一批次重复选择')
await expect(firstUpload).rejects.toThrow('测试结束首条上传')
expect(global.fetch).toHaveBeenCalledTimes(1)
})
test('普通 complete 响应丢失时保留断点且不调用 destructive cleanup', async () => {
;(global.fetch as jest.Mock)
.mockResolvedValueOnce({
ok: true,
json: async () => ({ uploadId: 'upload-response-lost' }),
})
.mockRejectedValueOnce(new TypeError('network connection lost'))
MockXMLHttpRequest.defaultStatus = 200
const file = new File(['hello'], 'source.psb', { type: 'application/octet-stream' })
await expect(chunkUpload(
file,
{ materialType: '3D场景' },
{
chunkSize: 5,
chunkConcurrency: 1,
maxRetries: 1,
retryDelay: 0,
},
)).rejects.toThrow('已保留上传断点')
await expect(getAllPendingUploads()).resolves.toEqual([
expect.objectContaining({
uploadId: 'upload-response-lost',
uploadedChunks: new Set([0]),
}),
])
expect(global.fetch).toHaveBeenCalledTimes(2)
expect(global.fetch).not.toHaveBeenCalledWith(
'/api/materials/upload/cleanup',
expect.anything(),
)
})
}) })

Some files were not shown because too many files have changed in this diff Show More