draft: commons upload

[DaxServer]

No description provided.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added chunked file uploads to a Wikimedia-like service with progress updates, retries, and overwrite handling.
  • Introduced file processing for local paths or URLs, including MIME detection, hashing, and image metadata extraction.
  • Enabled batch processing with per-file validation (size, MIME type, extension) and detailed, user-friendly error reporting.
  • Added ability to check if a target filename already exists before upload.

• Tests

  • Introduced comprehensive test coverage for single and multiple file processing, validations, and error scenarios.

• Style/Chores

  • Minor schema formatting updates with no functional impact.

Walkthrough

Adds a new file-processing pipeline (buffer extraction, MIME detection, hash/image metadata, validation, single/multi processing), a chunked Commons upload service with existence checks and retry/backoff, Wikibase upload-related runtime schemas/types, accompanying tests, and a minor import/formatting tweak in the project API module.

Changes

Cohort / File(s)	Summary
API import/formatting backend/src/api/project/index.ts	Imports GetProjectByIdResponse from schemas and reformats a request body schema field; no behavioral change.
Commons upload service backend/src/services/commons-upload.ts	New CommonsUploadService with chunked upload flow: session init, chunk uploads with retries/progress, finalization, overwrite handling, file existence check, and wikitext generation. Exposes types for options/progress/result/session.
File processing core backend/src/services/file-processor/processor.ts, .../multiple-files-processor.ts, .../file-buffer-extractor.ts, .../file-reader.ts, .../metadata-builder.ts, .../mime-detector.ts, .../hash-calculator.ts, .../image-extractor.ts, .../types.ts	Introduces end-to-end file processing: input reading (filepath/url), metadata assembly (extension, MIME, hash, image metadata), helpers for reading, MIME detection, hashing, and image extraction. Defines core types.
Validation pipeline backend/src/services/file-processor/validation.ts, .../validator/file-size-validator.ts, .../validator/mime-type-validator.ts, .../validator/extension-validator.ts	Adds validation defaults and validators for size, MIME type, and extension; aggregates errors.
Wikibase upload schemas/types backend/src/types/wikibase-upload.ts	Adds runtime schemas and TS types for upload jobs/files, statuses, configuration, validation error shape, and Commons upload response.
Tests for file processor backend/tests/upload/file-processor.test.ts	New tests covering single/multiple file processing, validation scenarios, error paths (URL and ENOENT), and image metadata extraction using temp JPEGs.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Caller
  participant FP as File Processor
  participant FE as extractFileBuffer
  participant MB as buildFileMetadata
  participant VD as validateFile

  Caller->>FP: processFile(uploadFile, options)
  FP->>FE: Extract buffer/size/filename
  FE-->>FP: { buffer, size, filename }
  FP->>MB: Build metadata (extension, MIME, hash, image)
  MB-->>FP: FileMetadata
  FP->>VD: validateFile(metadata, mergedOptions)
  VD-->>FP: errors[]
  FP-->>Caller: ProcessedFile { metadata, buffer, isValid, errors }

Loading

sequenceDiagram
  autonumber
  actor Client
  participant CU as CommonsUploadService
  participant Auth as AuthService
  participant API as Commons API

  Client->>CU: uploadFile(processedFile, options, onProgress)
  CU->>API: check file exists (query)
  API-->>CU: exists?/info
  CU->>API: init chunked upload (session key)
  API-->>CU: session key, offset=0
  loop for each chunk with retries
    CU->>API: upload chunk(session, offset, data)
    API-->>CU: ack new offset
    CU-->>Client: onProgress(update)
  end
  CU->>API: finalize upload(ignorewarnings, metadata)
  API-->>CU: result { url, pageUrl, warnings? }
  CU-->>Client: UploadResult { success, urls, warnings, errors? }

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60–90 minutes

Possibly related PRs

• feat: update Elysia 1.4 #157 — Refactors project API imports to use backend/src/api/project/schemas.ts, directly related to the import change in backend/src/api/project/index.ts.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch commons-upload

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 237827c and b314ea7.

📒 Files selected for processing (17)

• backend/src/api/project/index.ts (2 hunks)
• backend/src/services/commons-upload.ts (1 hunks)
• backend/src/services/file-processor/file-buffer-extractor.ts (1 hunks)
• backend/src/services/file-processor/file-reader.ts (1 hunks)
• backend/src/services/file-processor/hash-calculator.ts (1 hunks)
• backend/src/services/file-processor/image-extractor.ts (1 hunks)
• backend/src/services/file-processor/metadata-builder.ts (1 hunks)
• backend/src/services/file-processor/mime-detector.ts (1 hunks)
• backend/src/services/file-processor/multiple-files-processor.ts (1 hunks)
• backend/src/services/file-processor/processor.ts (1 hunks)
• backend/src/services/file-processor/types.ts (1 hunks)
• backend/src/services/file-processor/validation.ts (1 hunks)
• backend/src/services/file-processor/validator/extension-validator.ts (1 hunks)
• backend/src/services/file-processor/validator/file-size-validator.ts (1 hunks)
• backend/src/services/file-processor/validator/mime-type-validator.ts (1 hunks)
• backend/src/types/wikibase-upload.ts (1 hunks)
• backend/tests/upload/file-processor.test.ts (1 hunks)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}