diff --git a/AGENTS.md b/AGENTS.md index 13dd0bb0..a9eddbda 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,6 +46,7 @@ - 只发布移动端静态资源时,用 `npm --prefix mobile-uni run build:h5` 后备份远端 `/opt/wygj/www/h5`,再 `rsync -az --delete mobile-uni/dist/build/h5/ YCWY:/opt/wygj/www/h5/`;前端静态修复不需要重启后端。 - 原生 App 不复用 H5 静态发布流程;`npm --prefix mobile-uni run build:app` 仅生成 HBuilderX 出包资源,未提供法务 HTTPS 链接、签名或真机验收前不得称为已出包或上架。 - 定向后端发布只启用远端后端 + schema 的 `release-preflight`,不得因本轮未发布的管理端/H5 不匹配而制造假失败,也不得把结果称为完整包匹配;完整发布收口须同时启用远端静态、后端和 schema,只有三项都通过才可称当前本地构建已与线上完整包匹配。 +- 定向后端发布先运行只读 `scripts/release-backend.sh plan`;`deploy`/`rollback` 必须获得独立明确授权并使用计划输出的完整哈希授权串。脚本目标固定为 `YCWY:/opt/wygj/app/ruoyi-admin.jar` 和 `wygj-aihr.service`,不得绕过备份、原子切换、发布后预检和失败恢复链路手工覆盖。 - `aihr.practice.runtime-schema-bootstrap` / `AIHR_PRACTICE_RUNTIME_SCHEMA_BOOTSTRAP` 仅限本地开发逃生开关,Spring 默认关闭;生产请求只能校验表、列和关键索引,任何 DDL 都必须由正式 SQL 迁移完成,完整预检会拒绝开启该开关的服务。 ## 当前业务边界 diff --git a/docs/BRD_PRODUCTION_MIGRATION_RUNBOOK.md b/docs/BRD_PRODUCTION_MIGRATION_RUNBOOK.md index ccb38687..7a0f66c2 100644 --- a/docs/BRD_PRODUCTION_MIGRATION_RUNBOOK.md +++ b/docs/BRD_PRODUCTION_MIGRATION_RUNBOOK.md @@ -64,7 +64,7 @@ ## 发布前置条件 1. 确认发布窗口、DB 备份和回滚负责人;迁移脚本包含 `ALTER TABLE`,不能在无备份状态执行。 -2. 发布前先执行 `scripts/release-preflight.sh` 的本地只读检查。定向后端发布后使用 `RELEASE_VERIFY_REMOTE_BACKEND=true RELEASE_VERIFY_REMOTE_SCHEMA=true ./scripts/release-preflight.sh`,只核对 AIHR 模块、必需表和生产运行时 schema 保护,不因本轮未发布的静态资源产生假失败;该结果必须明确记录为“后端定向发布”。完整包发布后,必须按 `docs/DEV_SETUP.md` 再启用 `RELEASE_VERIFY_REMOTE_MATCH=true`,同时核对线上静态资源、AIHR 模块和必需表;schema 检查只读、不执行迁移。定向后端或单项 H5 校验不能替代完整包复核。 +2. 发布前先执行 `scripts/release-preflight.sh` 的本地只读检查。定向后端发布先运行 `scripts/release-backend.sh plan`;只有负责人明确授权后才运行 `deploy`,由脚本固定目标、备份旧 JAR、校验候选、原子切换并在失败时恢复。发布后脚本会使用 `RELEASE_VERIFY_REMOTE_BACKEND=true RELEASE_VERIFY_REMOTE_SCHEMA=true ./scripts/release-preflight.sh`,只核对 AIHR 模块、必需表和生产运行时 schema 保护,不因本轮未发布的静态资源产生假失败;该结果必须明确记录为“后端定向发布”。完整包发布后,必须按 `docs/DEV_SETUP.md` 再启用 `RELEASE_VERIFY_REMOTE_MATCH=true`,同时核对线上静态资源、AIHR 模块和必需表;schema 检查只读、不执行迁移。定向后端或单项 H5 校验不能替代完整包复核。 3. 确认迁移目标库、租户和字符集为 MySQL 8 / `utf8mb4`;脚本只允许在目标业务库执行。 4. 生产组织同步凭据、AI 供应商留存/费用口径和正式试点窗口仍未闭合时,不要把迁移完成宣称为 BRD 验收完成。 diff --git a/docs/DEV_SETUP.md b/docs/DEV_SETUP.md index af50a0d7..3c4d6fd8 100644 --- a/docs/DEV_SETUP.md +++ b/docs/DEV_SETUP.md @@ -324,6 +324,42 @@ Android 人工验收每完成一个步骤,用 `.\scripts\capture-android-accep 发布时必须保留 preflight 输出、远端备份目录和发布后浏览器回归结果;回滚优先使用对应备份目录恢复,再重启后端服务,不能直接覆盖当前线上目录而不留证据。 +### 定向后端发布与回滚 + +8 月 1 日独立 APK 如只需补齐候选后端语义,使用固定拓扑的 `release-backend.sh`,不要手工 `scp` 后直接覆盖。脚本只允许 `YCWY:/opt/wygj/app/ruoyi-admin.jar` 和 `wygj-aihr.service`,默认 `plan` 与 `rollback-plan` 只读;`deploy`/`rollback` 必须使用只读计划输出的完整 SHA-256 授权串并保持 Git 工作区干净。计划阶段会核对冻结提交、JAR 哈希与 AIHR 模块、生产目标/服务/空间和 schema;部署阶段先保留旧 JAR 与清单,再在同目录校验并原子切换、重启、执行后端 + schema 预检,任何激活或发布后预检失败都会尝试恢复旧 JAR。回滚前还会额外备份当时正在运行的 JAR。脚本不删除备份,不发布管理端/H5,不执行数据库迁移。 + +先只读生成计划: + +```bash +./scripts/release-backend.sh plan \ + --artifact output/aug1-release/0.1.10-110-dc20b920/ruoyi-admin-dc20b920.jar \ + --artifact-commit dc20b92061f10a356ed73eb105ef7b00c8b8a617 \ + --expected-sha256 8c676f2fbfccb4f4573cbca1259e8b26eb83ebdb469adcbaf714c5931ef3cf2d +``` + +只有负责人明确授权后,才能把计划输出的 `approval_token` 原样传给 `deploy`;仅运行 `plan` 不构成发布授权: + +```bash +./scripts/release-backend.sh deploy \ + --artifact output/aug1-release/0.1.10-110-dc20b920/ruoyi-admin-dc20b920.jar \ + --artifact-commit dc20b92061f10a356ed73eb105ef7b00c8b8a617 \ + --expected-sha256 8c676f2fbfccb4f4573cbca1259e8b26eb83ebdb469adcbaf714c5931ef3cf2d \ + --approval 'DEPLOY_BACKEND:<完整候选JAR-SHA256>' +``` + +发布成功会输出 `rollback_backup`、`rollback_sha256` 和回滚授权串。先只读核对备份,再在获得单独回滚授权后执行: + +```bash +./scripts/release-backend.sh rollback-plan \ + --backup-dir '' \ + --expected-sha256 '' + +./scripts/release-backend.sh rollback \ + --backup-dir '' \ + --expected-sha256 '' \ + --approval 'ROLLBACK_BACKEND:<完整备份JAR-SHA256>' +``` + ## MVP 页面验证 登录后侧栏应只展示以下入口: diff --git a/scripts/release-backend.sh b/scripts/release-backend.sh new file mode 100644 index 00000000..b620247e --- /dev/null +++ b/scripts/release-backend.sh @@ -0,0 +1,609 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REMOTE_SSH="YCWY" +REMOTE_PATH="/opt/wygj/app/ruoyi-admin.jar" +REMOTE_SERVICE="wygj-aihr.service" +REMOTE_URL="https://peilian.njzhmj.top" +REMOTE_BACKUP_ROOT="/opt/wygj/backups" +PREFLIGHT_SCRIPT="$ROOT_DIR/scripts/release-preflight.sh" +MIN_FREE_RESERVE_BYTES=$((512 * 1024 * 1024)) + +fail() { + echo "release-backend: $*" >&2 + exit 1 +} + +usage() { + cat <<'EOF' +Usage: + ./scripts/release-backend.sh plan \ + --artifact \ + --artifact-commit \ + --expected-sha256 + + ./scripts/release-backend.sh deploy \ + --artifact \ + --artifact-commit \ + --expected-sha256 \ + --approval DEPLOY_BACKEND: + + ./scripts/release-backend.sh rollback-plan \ + --backup-dir /opt/wygj/backups/backend-- \ + --expected-sha256 + + ./scripts/release-backend.sh rollback \ + --backup-dir /opt/wygj/backups/backend-- \ + --expected-sha256 \ + --approval ROLLBACK_BACKEND: + +plan and rollback-plan are read-only. deploy and rollback require an exact, +non-secret approval token and a clean Git worktree. The target is intentionally +fixed to YCWY:/opt/wygj/app/ruoyi-admin.jar and wygj-aihr.service. +EOF +} + +mode="${1:-plan}" +case "$mode" in + plan|deploy|rollback-plan|rollback) + shift || true + ;; + -h|--help) + usage + exit 0 + ;; + *) + usage >&2 + fail "unsupported mode: $mode" + ;; +esac + +artifact="" +artifact_commit="" +expected_sha256="" +backup_dir="" +approval="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --artifact) + [[ $# -ge 2 ]] || fail "--artifact requires a value" + artifact="$2" + shift 2 + ;; + --artifact-commit) + [[ $# -ge 2 ]] || fail "--artifact-commit requires a value" + artifact_commit="$2" + shift 2 + ;; + --expected-sha256) + [[ $# -ge 2 ]] || fail "--expected-sha256 requires a value" + expected_sha256="${2,,}" + shift 2 + ;; + --backup-dir) + [[ $# -ge 2 ]] || fail "--backup-dir requires a value" + backup_dir="$2" + shift 2 + ;; + --approval) + [[ $# -ge 2 ]] || fail "--approval requires a value" + approval="$2" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + fail "unknown argument: $1" + ;; + esac +done + +require_sha256() { + local value="$1" + local label="$2" + [[ "$value" =~ ^[0-9a-f]{64}$ ]] || fail "$label must be a 64-character SHA-256" +} + +require_clean_worktree() { + local status + status="$(git -C "$ROOT_DIR" status --porcelain)" + [[ -z "$status" ]] || fail "deploy and rollback require a clean Git worktree" +} + +require_approval() { + local expected="$1" + [[ "$approval" == "$expected" ]] \ + || fail "explicit approval required; rerun with --approval $expected" +} + +file_epoch() { + if stat -f %m "$1" >/dev/null 2>&1; then + stat -f %m "$1" + else + stat -c %Y "$1" + fi +} + +artifact_commit_sha="" +artifact_sha256="" +artifact_bytes="" +artifact_module_name="" +artifact_module_sha256="" +remote_current_sha256="" +remote_current_bytes="" + +validate_local_artifact() { + [[ -n "$artifact" ]] || fail "--artifact is required" + [[ -n "$artifact_commit" ]] || fail "--artifact-commit is required" + require_sha256 "$expected_sha256" "--expected-sha256" + + if [[ "$artifact" != /* && ! "$artifact" =~ ^[A-Za-z]:[/\\] ]]; then + artifact="$ROOT_DIR/$artifact" + fi + [[ -f "$artifact" ]] || fail "release JAR not found: $artifact" + + artifact_commit_sha="$(git -C "$ROOT_DIR" rev-parse --verify "${artifact_commit}^{commit}" 2>/dev/null)" \ + || fail "artifact commit is not a valid commit: $artifact_commit" + git -C "$ROOT_DIR" merge-base --is-ancestor "$artifact_commit_sha" HEAD \ + || fail "artifact commit must be an ancestor of HEAD: $artifact_commit_sha" + + local commit_epoch artifact_epoch + commit_epoch="$(git -C "$ROOT_DIR" show -s --format=%ct "$artifact_commit_sha")" + artifact_epoch="$(file_epoch "$artifact")" + [[ "$artifact_epoch" -ge "$commit_epoch" ]] \ + || fail "release JAR is older than its artifact commit: $artifact" + + artifact_sha256="$(sha256sum "$artifact" | awk '{print tolower($1)}')" + [[ "$artifact_sha256" == "$expected_sha256" ]] \ + || fail "release JAR SHA-256 mismatch: expected $expected_sha256, got $artifact_sha256" + artifact_bytes="$(stat -c %s "$artifact")" + + artifact_module_name="$( + unzip -Z1 "$artifact" | + sed -nE 's#BOOT-INF/lib/(ruoyi-aihr-[^/]+\.jar)#\1#p' | + head -1 + )" + [[ -n "$artifact_module_name" ]] || fail "release JAR does not contain ruoyi-aihr" + artifact_module_sha256="$( + unzip -p "$artifact" "BOOT-INF/lib/$artifact_module_name" | + sha256sum | + awk '{print tolower($1)}' + )" +} + +remote_value() { + local key="$1" + local text="$2" + printf '%s\n' "$text" | sed -n "s/^${key}=//p" | head -1 +} + +inspect_remote_target() { + local output + output="$( + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$REMOTE_PATH" "$REMOTE_SERVICE" "$REMOTE_BACKUP_ROOT" <<'REMOTE_INSPECT' +set -euo pipefail +target="$1" +service="$2" +backup_root="$3" +[[ -f "$target" && ! -L "$target" ]] +[[ "$(readlink -f "$target")" == "$target" ]] +[[ -d "$backup_root" && -w "$backup_root" ]] +printf 'target_type=%s\n' "$(stat -c %F "$target")" +printf 'target_realpath=%s\n' "$(readlink -f "$target")" +printf 'remote_user=%s\n' "$(id -un)" +printf 'target_sha256=%s\n' "$(sha256sum "$target" | awk '{print tolower($1)}')" +printf 'target_bytes=%s\n' "$(stat -c %s "$target")" +printf 'target_mode=%s\n' "$(stat -c '%a %U:%G' "$target")" +printf 'free_bytes=%s\n' "$(df --output=avail -B1 "$(dirname "$target")" | tail -1 | tr -d ' ')" +printf 'service_active=%s\n' "$(systemctl is-active "$service")" +printf 'service_exec_start=%s\n' "$(systemctl show "$service" --property=ExecStart --value --no-pager)" +REMOTE_INSPECT + )" || fail "read-only remote topology check failed" + + [[ "$(remote_value target_type "$output")" == "regular file" ]] \ + || fail "remote backend target is not a regular file" + [[ "$(remote_value target_realpath "$output")" == "$REMOTE_PATH" ]] \ + || fail "remote backend target resolves outside the fixed path" + [[ "$(remote_value remote_user "$output")" == "root" ]] \ + || fail "remote backend release requires the fixed root deployment account" + [[ "$(remote_value service_active "$output")" == "active" ]] \ + || fail "remote service is not active before the operation" + [[ "$(remote_value service_exec_start "$output")" == *"$REMOTE_PATH"* ]] \ + || fail "remote service does not start from the fixed backend target" + + remote_current_sha256="$(remote_value target_sha256 "$output")" + remote_current_bytes="$(remote_value target_bytes "$output")" + local free_bytes required_free + free_bytes="$(remote_value free_bytes "$output")" + [[ "$remote_current_sha256" =~ ^[0-9a-f]{64}$ ]] \ + || fail "remote target returned an invalid SHA-256" + [[ "$remote_current_bytes" =~ ^[0-9]+$ && "$free_bytes" =~ ^[0-9]+$ ]] \ + || fail "remote target returned invalid size metadata" + + if [[ -n "$artifact_bytes" ]]; then + required_free=$((artifact_bytes + remote_current_bytes + MIN_FREE_RESERVE_BYTES)) + [[ "$free_bytes" -ge "$required_free" ]] \ + || fail "remote filesystem free space is below backup + upload + 512 MiB reserve" + fi + + printf '%s\n' "$output" +} + +run_schema_preflight() { + RELEASE_REMOTE_URL="$REMOTE_URL" \ + RELEASE_VERIFY_REMOTE_SCHEMA=true \ + RELEASE_ARTIFACT_COMMIT="$artifact_commit_sha" \ + "$PREFLIGHT_SCRIPT" +} + +run_post_deploy_preflight() { + RELEASE_REMOTE_URL="$REMOTE_URL" \ + RELEASE_VERIFY_REMOTE_BACKEND=true \ + RELEASE_VERIFY_REMOTE_SCHEMA=true \ + RELEASE_ARTIFACT_COMMIT="$artifact_commit_sha" \ + RELEASE_LOCAL_BACKEND_PATH="$artifact" \ + "$PREFLIGHT_SCRIPT" +} + +run_public_health() { + local tenant_body mobile_body + curl -fsS --max-time 20 "$REMOTE_URL/" >/dev/null || { + echo "release-backend: root endpoint health check failed" >&2 + return 1 + } + tenant_body="$(curl -fsS --max-time 20 "$REMOTE_URL/prod-api/auth/tenant/list")" || { + echo "release-backend: tenant endpoint HTTP check failed" >&2 + return 1 + } + mobile_body="$(curl -fsS --max-time 20 "$REMOTE_URL/prod-api/api/aihr/mobile/home/user")" || { + echo "release-backend: mobile home endpoint HTTP check failed" >&2 + return 1 + } + printf '%s' "$tenant_body" | + LC_ALL=C grep -Eq '^[[:space:]]*\{[[:space:]]*"code"[[:space:]]*:[[:space:]]*200([[:space:]]*[,}])' \ + || { + echo "release-backend: tenant endpoint did not return business code 200" >&2 + return 1 + } + printf '%s' "$mobile_body" | + LC_ALL=C grep -Eq '^[[:space:]]*\{[[:space:]]*"code"[[:space:]]*:[[:space:]]*200([[:space:]]*[,}])' \ + || { + echo "release-backend: mobile home endpoint did not return business code 200" >&2 + return 1 + } +} + +run_deploy_plan() { + validate_local_artifact + inspect_remote_target + run_schema_preflight + + echo "mode=read-only-deploy-plan" + echo "artifact_commit=$artifact_commit_sha" + echo "artifact_path=$artifact" + echo "artifact_bytes=$artifact_bytes" + echo "artifact_sha256=$artifact_sha256" + echo "artifact_module=$artifact_module_name" + echo "artifact_module_sha256=$artifact_module_sha256" + echo "remote_target=$REMOTE_SSH:$REMOTE_PATH" + echo "remote_current_sha256=$remote_current_sha256" + echo "remote_service=$REMOTE_SERVICE" + echo "approval_token=DEPLOY_BACKEND:$artifact_sha256" + if [[ "$remote_current_sha256" == "$artifact_sha256" ]]; then + fail "remote backend already matches the candidate; no deploy is needed" + fi +} + +backup_remote_target() { + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$REMOTE_PATH" "$REMOTE_BACKUP_ROOT" "$REMOTE_SERVICE" \ + "$remote_current_sha256" "$artifact_sha256" "$artifact_commit_sha" <<'REMOTE_BACKUP' +set -euo pipefail +target="$1" +backup_root="$2" +service="$3" +expected_current_sha="$4" +candidate_sha="$5" +artifact_commit="$6" +[[ -f "$target" && ! -L "$target" ]] +actual_current_sha="$(sha256sum "$target" | awk '{print tolower($1)}')" +[[ "$actual_current_sha" == "$expected_current_sha" ]] +timestamp="$(date +%Y%m%d%H%M%S)" +backup_dir="$backup_root/backend-$timestamp-${expected_current_sha:0:12}" +mkdir "$backup_dir" +cp -a "$target" "$backup_dir/ruoyi-admin.jar" +backup_sha="$(sha256sum "$backup_dir/ruoyi-admin.jar" | awk '{print tolower($1)}')" +[[ "$backup_sha" == "$expected_current_sha" ]] +printf 'created_at=%s\nservice=%s\ntarget=%s\nold_sha256=%s\ncandidate_sha256=%s\nartifact_commit=%s\n' \ + "$(date --iso-8601=seconds)" "$service" "$target" "$expected_current_sha" "$candidate_sha" "$artifact_commit" \ + > "$backup_dir/release-manifest.txt" +printf 'backup_dir=%s\nbackup_sha256=%s\n' "$backup_dir" "$backup_sha" +REMOTE_BACKUP +} + +activate_remote_candidate() { + local staging_path="$1" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$REMOTE_PATH" "$staging_path" "$REMOTE_SERVICE" \ + "$remote_current_sha256" "$artifact_sha256" <<'REMOTE_ACTIVATE' +set -euo pipefail +target="$1" +staging="$2" +service="$3" +expected_current_sha="$4" +expected_candidate_sha="$5" +[[ -f "$target" && ! -L "$target" ]] +[[ -f "$staging" && ! -L "$staging" ]] +[[ "$(sha256sum "$target" | awk '{print tolower($1)}')" == "$expected_current_sha" ]] +[[ "$(sha256sum "$staging" | awk '{print tolower($1)}')" == "$expected_candidate_sha" ]] +chown --reference="$target" "$staging" +chmod --reference="$target" "$staging" +mv -T "$staging" "$target" +systemctl restart "$service" +for _ in $(seq 1 30); do + if [[ "$(systemctl is-active "$service" || true)" == "active" ]]; then + [[ "$(sha256sum "$target" | awk '{print tolower($1)}')" == "$expected_candidate_sha" ]] + exit 0 + fi + sleep 2 +done +systemctl status "$service" --no-pager >&2 || true +exit 1 +REMOTE_ACTIVATE +} + +restore_remote_backup() { + local source_jar="$1" + local expected_restore_sha="$2" + local expected_target_sha="$3" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$source_jar" "$REMOTE_PATH" "$REMOTE_SERVICE" \ + "$expected_restore_sha" "$expected_target_sha" <<'REMOTE_RESTORE' +set -euo pipefail +source_jar="$1" +target="$2" +service="$3" +expected_sha="$4" +expected_target_sha="$5" +[[ -f "$source_jar" && ! -L "$source_jar" ]] +[[ -f "$target" && ! -L "$target" ]] +[[ "$(sha256sum "$source_jar" | awk '{print tolower($1)}')" == "$expected_sha" ]] +[[ "$(sha256sum "$target" | awk '{print tolower($1)}')" == "$expected_target_sha" ]] +staging="${target}.restore-${expected_sha:0:12}-$(date +%Y%m%d%H%M%S)" +[[ ! -e "$staging" ]] +cp -a "$source_jar" "$staging" +chown --reference="$target" "$staging" +chmod --reference="$target" "$staging" +[[ "$(sha256sum "$staging" | awk '{print tolower($1)}')" == "$expected_sha" ]] +mv -T "$staging" "$target" +systemctl restart "$service" +for _ in $(seq 1 30); do + if [[ "$(systemctl is-active "$service" || true)" == "active" ]]; then + [[ "$(sha256sum "$target" | awk '{print tolower($1)}')" == "$expected_sha" ]] + exit 0 + fi + sleep 2 +done +systemctl status "$service" --no-pager >&2 || true +exit 1 +REMOTE_RESTORE +} + +get_remote_target_sha() { + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" \ + sha256sum "$REMOTE_PATH" | + awk '{print tolower($1)}' +} + +perform_deploy() { + local backup_output backup_path backup_sha staging_path after_failure_sha + backup_output="$(backup_remote_target)" || fail "remote backup failed; candidate was not activated" + printf '%s\n' "$backup_output" + backup_path="$(remote_value backup_dir "$backup_output")" + backup_sha="$(remote_value backup_sha256 "$backup_output")" + [[ "$backup_path" =~ ^/opt/wygj/backups/backend-[0-9]{14}-[0-9a-f]{12}$ ]] \ + || fail "remote backup returned an unsafe path" + [[ "$backup_sha" == "$remote_current_sha256" ]] \ + || fail "remote backup hash does not match the pre-deploy target" + + staging_path="${REMOTE_PATH}.candidate-${artifact_sha256:0:12}-$(date +%Y%m%d%H%M%S)" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" \ + test ! -e "$staging_path" || fail "remote candidate staging path already exists" + if ! scp -q -o BatchMode=yes -o ConnectTimeout=10 -- "$artifact" "$REMOTE_SSH:$staging_path"; then + fail "candidate upload failed; production target was not changed; backup is $backup_path" + fi + + if ! activate_remote_candidate "$staging_path"; then + after_failure_sha="$(get_remote_target_sha)" \ + || fail "candidate activation failed and the current target hash is unreadable; inspect $backup_path" + case "$after_failure_sha" in + "$artifact_sha256") + echo "release-backend: candidate activation failed after switch; restoring $backup_path" >&2 + restore_remote_backup \ + "$backup_path/ruoyi-admin.jar" "$remote_current_sha256" "$artifact_sha256" \ + || fail "automatic restore failed; use the recorded backup immediately: $backup_path" + fail "candidate activation failed and the original backend was restored" + ;; + "$remote_current_sha256") + fail "candidate activation failed before switching the production target; backup is $backup_path" + ;; + *) + fail "candidate activation saw a concurrent target change ($after_failure_sha); no automatic overwrite was attempted; inspect $backup_path" + ;; + esac + fi + + if ! run_post_deploy_preflight; then + after_failure_sha="$(get_remote_target_sha)" \ + || fail "post-deploy preflight failed and the current target hash is unreadable; inspect $backup_path" + case "$after_failure_sha" in + "$artifact_sha256") + echo "release-backend: post-deploy preflight failed; restoring $backup_path" >&2 + restore_remote_backup \ + "$backup_path/ruoyi-admin.jar" "$remote_current_sha256" "$artifact_sha256" \ + || fail "automatic restore failed; use the recorded backup immediately: $backup_path" + run_public_health \ + || fail "original backend was restored but public health verification failed" + fail "post-deploy preflight failed and the original backend was restored" + ;; + "$remote_current_sha256") + fail "post-deploy preflight failed after the original backend was already restored; inspect $backup_path" + ;; + *) + fail "post-deploy preflight saw a concurrent target change ($after_failure_sha); no automatic overwrite was attempted; inspect $backup_path" + ;; + esac + fi + + echo "mode=backend-deploy-complete" + echo "deployed_sha256=$artifact_sha256" + echo "rollback_backup=$backup_path" + echo "rollback_sha256=$remote_current_sha256" + echo "rollback_approval_token=ROLLBACK_BACKEND:$remote_current_sha256" +} + +validate_backup_path() { + [[ "$backup_dir" =~ ^/opt/wygj/backups/backend-[0-9]{14}-[0-9a-f]{12}$ ]] \ + || fail "--backup-dir must be a release-backend backup directory" + require_sha256 "$expected_sha256" "--expected-sha256" +} + +inspect_rollback() { + validate_backup_path + local output + output="$( + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$backup_dir/ruoyi-admin.jar" "$REMOTE_PATH" "$REMOTE_SERVICE" <<'REMOTE_ROLLBACK_INSPECT' +set -euo pipefail +backup_jar="$1" +target="$2" +service="$3" +[[ -f "$backup_jar" && ! -L "$backup_jar" ]] +[[ -f "$target" && ! -L "$target" ]] +printf 'backup_sha256=%s\n' "$(sha256sum "$backup_jar" | awk '{print tolower($1)}')" +printf 'current_sha256=%s\n' "$(sha256sum "$target" | awk '{print tolower($1)}')" +printf 'service_active=%s\n' "$(systemctl is-active "$service")" +REMOTE_ROLLBACK_INSPECT + )" || fail "read-only rollback inspection failed" + local backup_sha service_state + backup_sha="$(remote_value backup_sha256 "$output")" + service_state="$(remote_value service_active "$output")" + [[ "$backup_sha" == "$expected_sha256" ]] \ + || fail "backup JAR SHA-256 mismatch: expected $expected_sha256, got $backup_sha" + [[ "$service_state" == "active" ]] || fail "remote service is not active before rollback" + printf '%s\n' "$output" + echo "mode=read-only-rollback-plan" + echo "backup_dir=$backup_dir" + echo "approval_token=ROLLBACK_BACKEND:$expected_sha256" +} + +perform_rollback() { + local current_sha safety_output safety_path safety_sha after_failure_sha + current_sha="$(get_remote_target_sha)" \ + || fail "cannot read the current remote backend hash" + require_sha256 "$current_sha" "remote current backend hash" + [[ "$current_sha" != "$expected_sha256" ]] \ + || fail "remote backend already matches the requested rollback JAR" + + safety_output="$( + ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE_SSH" bash -s -- \ + "$REMOTE_PATH" "$REMOTE_BACKUP_ROOT" "$REMOTE_SERVICE" \ + "$current_sha" "$expected_sha256" <<'REMOTE_ROLLBACK_SAFETY' +set -euo pipefail +target="$1" +backup_root="$2" +service="$3" +current_sha="$4" +restore_sha="$5" +[[ "$(sha256sum "$target" | awk '{print tolower($1)}')" == "$current_sha" ]] +timestamp="$(date +%Y%m%d%H%M%S)" +safety_dir="$backup_root/backend-rollback-safety-$timestamp-${current_sha:0:12}" +mkdir "$safety_dir" +cp -a "$target" "$safety_dir/ruoyi-admin.jar" +[[ "$(sha256sum "$safety_dir/ruoyi-admin.jar" | awk '{print tolower($1)}')" == "$current_sha" ]] +printf 'created_at=%s\nservice=%s\ntarget=%s\nsafety_sha256=%s\nrestore_sha256=%s\n' \ + "$(date --iso-8601=seconds)" "$service" "$target" "$current_sha" "$restore_sha" \ + > "$safety_dir/rollback-manifest.txt" +printf 'safety_dir=%s\nsafety_sha256=%s\n' "$safety_dir" "$current_sha" +REMOTE_ROLLBACK_SAFETY + )" || fail "rollback safety backup failed; production target was not changed" + printf '%s\n' "$safety_output" + safety_path="$(remote_value safety_dir "$safety_output")" + safety_sha="$(remote_value safety_sha256 "$safety_output")" + [[ "$safety_path" =~ ^/opt/wygj/backups/backend-rollback-safety-[0-9]{14}-[0-9a-f]{12}$ ]] \ + || fail "rollback safety backup returned an unsafe path" + [[ "$safety_sha" == "$current_sha" ]] || fail "rollback safety backup hash mismatch" + + if ! restore_remote_backup \ + "$backup_dir/ruoyi-admin.jar" "$expected_sha256" "$current_sha"; then + after_failure_sha="$(get_remote_target_sha)" \ + || fail "rollback activation failed and the current target hash is unreadable; inspect $safety_path" + case "$after_failure_sha" in + "$expected_sha256") + echo "release-backend: rollback activation failed after switch; restoring safety copy $safety_path" >&2 + restore_remote_backup \ + "$safety_path/ruoyi-admin.jar" "$current_sha" "$expected_sha256" \ + || fail "rollback and safety restore both failed; use $safety_path immediately" + fail "rollback failed and the pre-rollback backend was restored" + ;; + "$current_sha") + fail "rollback activation failed before switching the production target; safety backup is $safety_path" + ;; + *) + fail "rollback activation saw a concurrent target change ($after_failure_sha); no automatic overwrite was attempted; inspect $safety_path" + ;; + esac + fi + if ! run_public_health; then + after_failure_sha="$(get_remote_target_sha)" \ + || fail "rollback health failed and the current target hash is unreadable; inspect $safety_path" + case "$after_failure_sha" in + "$expected_sha256") + echo "release-backend: rollback health failed; restoring safety copy $safety_path" >&2 + restore_remote_backup \ + "$safety_path/ruoyi-admin.jar" "$current_sha" "$expected_sha256" \ + || fail "health rollback and safety restore both failed; use $safety_path immediately" + fail "rollback health check failed and the pre-rollback backend was restored" + ;; + "$current_sha") + fail "rollback health failed after the pre-rollback backend was already restored; inspect $safety_path" + ;; + *) + fail "rollback health saw a concurrent target change ($after_failure_sha); no automatic overwrite was attempted; inspect $safety_path" + ;; + esac + fi + + echo "mode=backend-rollback-complete" + echo "restored_sha256=$expected_sha256" + echo "pre_rollback_safety_backup=$safety_path" + echo "pre_rollback_sha256=$current_sha" +} + +case "$mode" in + plan) + run_deploy_plan + ;; + deploy) + require_sha256 "$expected_sha256" "--expected-sha256" + require_approval "DEPLOY_BACKEND:$expected_sha256" + require_clean_worktree + run_deploy_plan + perform_deploy + ;; + rollback-plan) + inspect_rollback + ;; + rollback) + require_sha256 "$expected_sha256" "--expected-sha256" + require_approval "ROLLBACK_BACKEND:$expected_sha256" + require_clean_worktree + inspect_rollback + perform_rollback + ;; +esac diff --git a/scripts/tests/release-backend-safety.test.sh b/scripts/tests/release-backend-safety.test.sh new file mode 100644 index 00000000..ad63636e --- /dev/null +++ b/scripts/tests/release-backend-safety.test.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SCRIPT="$ROOT_DIR/scripts/release-backend.sh" +DEV_SETUP="$ROOT_DIR/docs/DEV_SETUP.md" + +bash -n "$SCRIPT" + +for remote_block in \ + REMOTE_INSPECT \ + REMOTE_BACKUP \ + REMOTE_ACTIVATE \ + REMOTE_RESTORE \ + REMOTE_ROLLBACK_INSPECT \ + REMOTE_ROLLBACK_SAFETY; do + awk -v marker="$remote_block" ' + index($0, "<<" SQ marker SQ) { capture = 1; next } + capture && $0 == marker { exit } + capture { print } + ' SQ="'" "$SCRIPT" | bash -n +done + +grep -Fq 'REMOTE_SSH="YCWY"' "$SCRIPT" +grep -Fq 'REMOTE_PATH="/opt/wygj/app/ruoyi-admin.jar"' "$SCRIPT" +grep -Fq 'REMOTE_SERVICE="wygj-aihr.service"' "$SCRIPT" +grep -Fq 'mode="${1:-plan}"' "$SCRIPT" +grep -Fq 'plan and rollback-plan are read-only' "$SCRIPT" +grep -Fq 'require_approval "DEPLOY_BACKEND:$expected_sha256"' "$SCRIPT" +grep -Fq 'require_approval "ROLLBACK_BACKEND:$expected_sha256"' "$SCRIPT" +grep -Fq 'deploy and rollback require a clean Git worktree' "$SCRIPT" +grep -Fq 'run_post_deploy_preflight' "$SCRIPT" +grep -Fq 'post-deploy preflight failed; restoring' "$SCRIPT" +grep -Fq 'rollback health failed; restoring safety copy' "$SCRIPT" +grep -Fq 'expected_target_sha="$5"' "$SCRIPT" +grep -Fq 'concurrent target change' "$SCRIPT" +grep -Fq 'remote backend release requires the fixed root deployment account' "$SCRIPT" +grep -Fq 'cp -a "$target" "$backup_dir/ruoyi-admin.jar"' "$SCRIPT" +grep -Fq 'mv -T "$staging" "$target"' "$SCRIPT" +grep -Fq 'RELEASE_VERIFY_REMOTE_BACKEND=true' "$SCRIPT" +grep -Fq 'RELEASE_VERIFY_REMOTE_SCHEMA=true' "$SCRIPT" + +if grep -Eq '(^|[[:space:]])rm([[:space:]]|$)' "$SCRIPT"; then + echo 'FAIL: backend release script must not delete release or backup files' >&2 + exit 1 +fi + +deploy_without_approval="$( + bash "$SCRIPT" deploy \ + --artifact missing.jar \ + --artifact-commit HEAD \ + --expected-sha256 0000000000000000000000000000000000000000000000000000000000000000 \ + 2>&1 || true +)" +grep -Fq 'explicit approval required' <<<"$deploy_without_approval" +if grep -Fq 'release JAR not found' <<<"$deploy_without_approval"; then + echo 'FAIL: deploy touched artifact or remote planning before approval' >&2 + exit 1 +fi + +rollback_without_approval="$( + bash "$SCRIPT" rollback \ + --backup-dir /opt/wygj/backups/backend-20260729000000-000000000000 \ + --expected-sha256 0000000000000000000000000000000000000000000000000000000000000000 \ + 2>&1 || true +)" +grep -Fq 'explicit approval required' <<<"$rollback_without_approval" +if grep -Fq 'read-only rollback inspection failed' <<<"$rollback_without_approval"; then + echo 'FAIL: rollback contacted the remote before approval' >&2 + exit 1 +fi + +grep -Fq './scripts/release-backend.sh plan' "$DEV_SETUP" +grep -Fq './scripts/release-backend.sh deploy' "$DEV_SETUP" +grep -Fq './scripts/release-backend.sh rollback-plan' "$DEV_SETUP" + +echo 'PASS: backend release and rollback require explicit approval and preserve recoverable backups'