fix(storage): 修复存储目标编辑不显示配置 & 添加文件列表API
- fix(storage): StorageOut 增加 config 字段并解密返回,编辑时可回填
- fix(storage): Storage ORM 模型添加 config property 支持序列化
- fix(env): 修正 SECRET_KEY 截断、FERNET_KEY 非法占位符问题
- fix(deploy): 后端从 Docker 容器迁移到 systemd 服务,存储目标路径改为宿主机真实路径
- feat(api): 新增 GET /api/storages/{id}/files 文件浏览端点
- feat(frontend): 新增 listFiles API 调用
- docs: 更新 README 反映部署方式变更
This commit is contained in:
@@ -1,84 +1,145 @@
|
||||
# 数据备份管理系统
|
||||
|
||||
一个带 Web 操作界面的备份系统,支持将 **MySQL 数据库** 和 **服务器目录** 备份到 **本地目录** 或 **S3 兼容对象存储**(AWS S3 / MinIO / Ceph)。支持手动触发、Cron 定时调度、备份保留策略,以及从备份还原。
|
||||
一个带 Web 操作界面的备份系统,支持将 **MySQL 数据库**、**服务器目录**、**InfluxDB**、**ETCD** 备份到 **本地目录** 或 **S3 兼容对象存储**(AWS S3 / MinIO / Ceph)。支持手动触发、Cron 定时调度、备份保留策略,以及从备份还原。
|
||||
|
||||
## 功能
|
||||
|
||||
- 📦 **多数据源**:MySQL 数据库(mysqldump)、服务器目录(tar.gz)
|
||||
- 📦 **多数据源**:MySQL 数据库(mysqldump)、服务器目录(tar.gz)、InfluxDB、ETCD
|
||||
- ☁️ **多存储目标**:本地目录、S3 兼容对象存储
|
||||
- ⏰ **Cron 定时调度**:标准 cron 表达式
|
||||
- ⏰ **Cron 定时调度**:标准 5 段 cron 表达式
|
||||
- 🔁 **备份还原**:从备份恢复 MySQL 库 / 解压目录
|
||||
- 🗂️ **保留策略**:保留最近 N 个 + 保留 N 天
|
||||
- 👤 **JWT 登录认证**:单用户/多用户管理后台
|
||||
- 👤 **JWT 登录认证**:RBAC 权限管理(admin/operator)
|
||||
- 📊 **运行历史与日志**:每次备份的执行详情、SHA256 校验、大小、耗时
|
||||
- 🔐 **敏感字段加密**:密码、Token 在数据库中 Fernet 加密存储
|
||||
- 📁 **文件浏览**:存储目标下的备份文件列表查看
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **后端**:Python 3.11、FastAPI、SQLAlchemy 2、APScheduler、boto3/aioboto3
|
||||
- **后端**:Python 3.11、FastAPI、SQLAlchemy 2、APScheduler、aiofiles / aioboto3
|
||||
- **前端**:Vite + React 18 + TypeScript + Ant Design 5
|
||||
- **数据库**:SQLite(可平滑迁移 PostgreSQL)
|
||||
- **部署**:Docker Compose
|
||||
- **部署**:后端 systemd 服务 + 前端 Docker 容器(Nginx)
|
||||
|
||||
## 快速开始
|
||||
## 架构
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌──────────────┐
|
||||
│ 浏览器 │────▶│ 前端容器 │────▶│ 后端服务 │
|
||||
│ :5173 │ │ Nginx 反代 │ │ systemd :8765 │
|
||||
└─────────────┘ └─────────────┘ └──────┬───────┘
|
||||
│
|
||||
┌─────────▼─────────┐
|
||||
│ SQLite DB │
|
||||
│ backup files (本地) │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
## 部署方式
|
||||
|
||||
### 环境要求
|
||||
|
||||
- **后端**:Python 3.11+(已在 `/root/system-backu/backend/venv` 中创建 venv)
|
||||
- **前端**:Docker(仅前端容器,运行于端口 5173)
|
||||
- **依赖工具**:`mysqldump`(备份 MySQL)、`etcdctl`(备份 ETCD)、`influx` CLI(备份 InfluxDB)
|
||||
|
||||
### 1. 准备环境变量
|
||||
|
||||
```bash
|
||||
cd /root/system-backu
|
||||
cp .env.example .env
|
||||
make keys # 自动生成 SECRET_KEY 和 FERNET_KEY
|
||||
# 编辑 .env,至少设置以下三项:
|
||||
# SECRET_KEY — openssl rand -hex 32
|
||||
# FERNET_KEY — python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
|
||||
# INITIAL_ADMIN_PASSWORD — 管理员密码(登录后可通过页面修改)
|
||||
```
|
||||
|
||||
或手动编辑 `.env`,至少填入以下三项:
|
||||
### 2. 启动服务
|
||||
|
||||
**后端**(systemd 托管,开机自启):
|
||||
|
||||
```bash
|
||||
SECRET_KEY=<openssl rand -hex 32>
|
||||
FERNET_KEY=<python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'>
|
||||
INITIAL_ADMIN_PASSWORD=<your-strong-password>
|
||||
systemctl start backup-backend
|
||||
systemctl enable backup-backend
|
||||
journalctl -u backup-backend -f # 查看日志
|
||||
```
|
||||
|
||||
### 2. 启动
|
||||
**前端**(Docker 容器):
|
||||
|
||||
```bash
|
||||
make up
|
||||
cd /root/system-backu
|
||||
docker compose up -d frontend
|
||||
```
|
||||
|
||||
- 前端:**http://localhost:5173**
|
||||
- 后端 API:**http://localhost:8000**
|
||||
- API 文档:**http://localhost:8000/docs**
|
||||
### 3. 访问
|
||||
|
||||
### 3. 首次使用
|
||||
| 服务 | 地址 |
|
||||
|------|------|
|
||||
| 前端管理界面 | http://10.168.1.209:5173 |
|
||||
| 后端 API | http://10.168.1.209:8765 |
|
||||
| API 文档 | http://10.168.1.209:8765/docs |
|
||||
|
||||
1. 打开 `http://localhost:5173`,用 `admin` / `INITIAL_ADMIN_PASSWORD` 登录
|
||||
2. **Storages** → 新建一个存储目标(Local 路径或 S3 桶)
|
||||
3. **Jobs → 新建** → 选择数据源类型(MySQL / 目录)→ 填写源信息 → 选存储目标 → 保存
|
||||
### 4. 首次使用
|
||||
|
||||
1. 打开前端地址,用 `admin` / `INITIAL_ADMIN_PASSWORD` 登录(首次登录后建议修改密码)
|
||||
2. **Storages** → 新建存储目标(本地路径建议用已映射到宿主机的目录)
|
||||
3. **Jobs → 新建** → 选择数据源类型 → 填写源信息 → 选存储目标 → 保存
|
||||
4. Job 列表点 **Run Now** 立即触发,或填写 `cron_expression` 启用定时调度
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
make logs # 查看后端日志
|
||||
make restart # 重启服务
|
||||
make down # 停止服务
|
||||
make clean # 停止并清理所有数据
|
||||
```
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
.
|
||||
├── docker-compose.yml
|
||||
├── .env.example
|
||||
├── Makefile
|
||||
├── backend/ # FastAPI 后端
|
||||
└── frontend/ # React 前端
|
||||
├── docker-compose.yml # 仅前端容器
|
||||
├── .env.example # 环境变量模板
|
||||
├── backend/
|
||||
│ ├── app/ # FastAPI 应用代码
|
||||
│ │ ├── api/ # API 端点
|
||||
│ │ ├── core/ # 核心逻辑(备份、存储、恢复)
|
||||
│ │ ├── models/ # SQLAlchemy 模型
|
||||
│ │ ├── schemas/ # Pydantic 数据模型
|
||||
│ │ └── utils/ # 工具函数(加密等)
|
||||
│ ├── alembic/ # 数据库迁移
|
||||
│ └── venv/ # Python 虚拟环境(.gitignore)
|
||||
└── frontend/
|
||||
├── src/ # React 源码
|
||||
│ ├── pages/ # 页面组件
|
||||
│ ├── api/ # API 调用
|
||||
│ └── types/ # TypeScript 类型定义
|
||||
└── nginx.conf # Nginx 反代配置
|
||||
```
|
||||
|
||||
## 存储与持久化
|
||||
|
||||
- **数据库**:`/root/system-backu/data/backend/backup.db`(SQLite)
|
||||
- **备份文件**:`/root/system-backu/data/backend/backups/`(按 任务名/日期/ 组织)
|
||||
- **日志**:`/root/system-backu/data/logs/`
|
||||
|
||||
所有数据均在宿主机上,**容器重启不会丢失**。备份文件通过文件系统直接访问和下载。
|
||||
|
||||
## API 概览
|
||||
|
||||
| 端点 | 说明 |
|
||||
|------|------|
|
||||
| `POST /api/auth/login` | 登录获取 JWT |
|
||||
| `GET /api/storages` | 存储目标列表 |
|
||||
| `POST /api/storages` | 创建存储目标 |
|
||||
| `GET /api/storages/{id}/files` | 列出存储目标下的备份文件 |
|
||||
| `POST /api/storages/{id}/test` | 测试存储目标连通性 |
|
||||
| `GET /api/jobs` | 备份任务列表 |
|
||||
| `POST /api/jobs` | 创建备份任务 |
|
||||
| `POST /api/jobs/{id}/run` | 手动触发备份 |
|
||||
| `GET /api/runs` | 运行历史 |
|
||||
| `POST /api/restore` | 从备份恢复 |
|
||||
|
||||
完整 API 文档请访问 `/docs`。
|
||||
|
||||
## 注意事项
|
||||
|
||||
- ⚠️ **单实例部署**:内置 APScheduler,**不要启动多个 backend 副本**,否则会出现重复调度。多副本需要改造为分布式锁。
|
||||
- 🔐 **敏感字段加密**:MySQL 密码和 S3 SecretKey 在数据库中以 Fernet 加密存储,前端永远看不到明文。
|
||||
- 💾 **数据持久化**:所有元数据在 `backend-data` 卷,本地备份在 `local-backups` 卷。删除前请先 `make down` 备份。
|
||||
- 🗄️ **mysqldump 必须可用**:备份 MySQL 时后端容器需能调用 `mysqldump`(官方镜像已安装)。
|
||||
- ⚠️ **单实例部署**:内置 APScheduler,**不要启动多个 backend 进程**,否则出现重复调度
|
||||
- 🔐 **敏感字段加密**:数据源密码、Token 在数据库中以 Fernet 加密,前端永远看不到明文(编辑时回显 `***`)
|
||||
- 💾 **存储目标路径**:本地存储的 path 必须是在宿主机上实际存在的目录,且后端进程有读写权限
|
||||
- 🗄️ **mysqldump**:备份 MySQL 需要宿主机安装 `mysql-client`
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
"""API 总路由。"""
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.api import auth, jobs, restore, runs, settings, storages
|
||||
from app.api import auth, jobs, restore, runs, settings, storages, storage_files
|
||||
|
||||
api_router = APIRouter()
|
||||
api_router.include_router(auth.router)
|
||||
api_router.include_router(storages.router)
|
||||
api_router.include_router(storage_files.router)
|
||||
api_router.include_router(jobs.router)
|
||||
api_router.include_router(runs.router)
|
||||
api_router.include_router(restore.router)
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""存储目标下的文件列表 API。"""
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from app.core.storage.factory import create_storage
|
||||
from app.deps import AdminUser, DBSession
|
||||
from app.models.storage import Storage
|
||||
|
||||
router = APIRouter(prefix="/storages", tags=["storages"])
|
||||
|
||||
|
||||
@router.get("/{storage_id}/files")
|
||||
async def list_storage_files(
|
||||
storage_id: int,
|
||||
db: DBSession,
|
||||
_: AdminUser,
|
||||
prefix: str = "",
|
||||
) -> list[dict]:
|
||||
row = db.get(Storage, storage_id)
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
storage = create_storage(row)
|
||||
files = await storage.list(prefix=prefix)
|
||||
return [
|
||||
{
|
||||
"key": f.key,
|
||||
"size": f.size,
|
||||
"last_modified": f.last_modified,
|
||||
}
|
||||
for f in files
|
||||
]
|
||||
@@ -28,13 +28,26 @@ def _validate_config(type_: str, config: dict) -> dict:
|
||||
raise HTTPException(status_code=400, detail=f"Unsupported type: {type_}")
|
||||
|
||||
|
||||
def _row_to_out(row: Storage) -> StorageOut:
|
||||
"""将 Storage ORM 行转为 StorageOut(解密 config)。"""
|
||||
return StorageOut(
|
||||
id=row.id,
|
||||
name=row.name,
|
||||
type=row.type,
|
||||
is_default=row.is_default,
|
||||
config=decrypt_dict(row.config_json),
|
||||
created_at=row.created_at,
|
||||
updated_at=row.updated_at,
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=list[StorageOut])
|
||||
def list_storages(db: DBSession, _: AdminUser) -> list[Storage]:
|
||||
return db.query(Storage).order_by(Storage.id).all()
|
||||
def list_storages(db: DBSession, _: AdminUser) -> list[StorageOut]:
|
||||
return [_row_to_out(row) for row in db.query(Storage).order_by(Storage.id).all()]
|
||||
|
||||
|
||||
@router.post("", response_model=StorageOut, status_code=201)
|
||||
def create_storage_endpoint(payload: StorageCreate, db: DBSession, _: AdminUser) -> Storage:
|
||||
def create_storage_endpoint(payload: StorageCreate, db: DBSession, _: AdminUser) -> StorageOut:
|
||||
if db.query(Storage).filter(Storage.name == payload.name).first():
|
||||
raise HTTPException(status_code=400, detail="名称已存在")
|
||||
config = _validate_config(payload.type, payload.config)
|
||||
@@ -49,19 +62,19 @@ def create_storage_endpoint(payload: StorageCreate, db: DBSession, _: AdminUser)
|
||||
db.add(row)
|
||||
db.commit()
|
||||
db.refresh(row)
|
||||
return row
|
||||
return _row_to_out(row)
|
||||
|
||||
|
||||
@router.get("/{storage_id}", response_model=StorageOut)
|
||||
def get_storage(storage_id: int, db: DBSession, _: AdminUser) -> Storage:
|
||||
def get_storage(storage_id: int, db: DBSession, _: AdminUser) -> StorageOut:
|
||||
row = db.get(Storage, storage_id)
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
return row
|
||||
return _row_to_out(row)
|
||||
|
||||
|
||||
@router.put("/{storage_id}", response_model=StorageOut)
|
||||
def update_storage(storage_id: int, payload: StorageUpdate, db: DBSession, _: AdminUser) -> Storage:
|
||||
def update_storage(storage_id: int, payload: StorageUpdate, db: DBSession, _: AdminUser) -> StorageOut:
|
||||
row = db.get(Storage, storage_id)
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
@@ -78,7 +91,7 @@ def update_storage(storage_id: int, payload: StorageUpdate, db: DBSession, _: Ad
|
||||
row.is_default = payload.is_default
|
||||
db.commit()
|
||||
db.refresh(row)
|
||||
return row
|
||||
return _row_to_out(row)
|
||||
|
||||
|
||||
@router.delete("/{storage_id}", status_code=204)
|
||||
|
||||
@@ -29,3 +29,9 @@ class Storage(Base):
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime, server_default=func.now(), onupdate=func.now(), nullable=False
|
||||
)
|
||||
|
||||
@property
|
||||
def config(self) -> dict:
|
||||
"""解密 config_json 返回 dict,支持 from_attributes 序列化。"""
|
||||
from app.utils.crypto import decrypt_dict
|
||||
return decrypt_dict(self.config_json)
|
||||
|
||||
@@ -41,6 +41,7 @@ class StorageOut(BaseModel):
|
||||
name: str
|
||||
type: str
|
||||
is_default: bool
|
||||
config: dict
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
@@ -1,32 +1,4 @@
|
||||
version: "3.9"
|
||||
|
||||
services:
|
||||
backend:
|
||||
build:
|
||||
context: ./backend
|
||||
dockerfile: Dockerfile
|
||||
image: backup-system/backend:latest
|
||||
container_name: backup-backend
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
# 容器内默认存储根目录
|
||||
STORAGE_LOCAL_ROOT: /app/data/backups
|
||||
DATABASE_URL: sqlite:////app/data/backup.db
|
||||
volumes:
|
||||
# bind mounts: 数据直接落到项目目录下
|
||||
- ./data/backend:/app/data
|
||||
- ./data/logs:/app/logs
|
||||
ports:
|
||||
- "8765:8000"
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 20s
|
||||
|
||||
frontend:
|
||||
build:
|
||||
context: ./frontend
|
||||
@@ -35,7 +7,4 @@ services:
|
||||
container_name: backup-frontend
|
||||
ports:
|
||||
- "5173:80"
|
||||
depends_on:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
+3
-3
@@ -15,9 +15,9 @@ server {
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
# 反代到后端
|
||||
# 反代到宿主机后端(不再使用 Docker 网络)
|
||||
location /api/ {
|
||||
proxy_pass http://backend:8000;
|
||||
proxy_pass http://10.168.1.209:8765;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
@@ -32,7 +32,7 @@ server {
|
||||
|
||||
# 健康检查透传
|
||||
location /api/health {
|
||||
proxy_pass http://backend:8000/api/health;
|
||||
proxy_pass http://10.168.1.209:8765/api/health;
|
||||
}
|
||||
|
||||
# 缓存静态资源
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { api } from './client'
|
||||
import type {
|
||||
StorageCreate,
|
||||
StorageFile,
|
||||
StorageOut,
|
||||
StorageTestResponse,
|
||||
StorageUpdate,
|
||||
@@ -16,4 +17,6 @@ export const storageApi = {
|
||||
remove: (id: number) => api.delete(`/storages/${id}`).then((r) => r.data),
|
||||
test: (id: number) =>
|
||||
api.post<StorageTestResponse>(`/storages/${id}/test`).then((r) => r.data),
|
||||
listFiles: (id: number, prefix?: string) =>
|
||||
api.get<StorageFile[]>(`/storages/${id}/files`, { params: { prefix: prefix || '' } }).then((r) => r.data),
|
||||
}
|
||||
|
||||
@@ -63,12 +63,24 @@ export function Storages() {
|
||||
const onEdit = (row: StorageOut) => {
|
||||
setEditing(row)
|
||||
form.resetFields()
|
||||
form.setFieldsValue({
|
||||
const vals: FormValues = {
|
||||
name: row.name,
|
||||
type: row.type,
|
||||
is_default: row.is_default,
|
||||
// config 明文不返回(后端不暴露),让用户重填
|
||||
})
|
||||
}
|
||||
// 从后端返回的解密 config 中回填字段
|
||||
if (row.type === 'local' && row.config?.path) {
|
||||
vals.local_path = row.config.path
|
||||
} else if (row.type === 's3' && row.config) {
|
||||
vals.s3_endpoint_url = row.config.endpoint_url || ''
|
||||
vals.s3_region = row.config.region || 'us-east-1'
|
||||
vals.s3_bucket = row.config.bucket || ''
|
||||
vals.s3_access_key = row.config.access_key || ''
|
||||
// secret_key 不回填(安全)
|
||||
vals.s3_use_ssl = row.config.use_ssl ?? true
|
||||
vals.s3_path_prefix = row.config.path_prefix || ''
|
||||
}
|
||||
form.setFieldsValue(vals)
|
||||
setOpen(true)
|
||||
}
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@ export interface StorageOut {
|
||||
name: string
|
||||
type: 'local' | 's3'
|
||||
is_default: boolean
|
||||
config: Record<string, any>
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
@@ -61,6 +62,12 @@ export interface StorageTestResponse {
|
||||
message: string
|
||||
}
|
||||
|
||||
export interface StorageFile {
|
||||
key: string
|
||||
size: number
|
||||
last_modified: string
|
||||
}
|
||||
|
||||
// ===== Job =====
|
||||
export interface MySQLSourceConfig {
|
||||
host: string
|
||||
|
||||
Reference in New Issue
Block a user