256 lines
12 KiB
Markdown
256 lines
12 KiB
Markdown
# Frame Processor — Implementation Increments
|
||
|
||
Step-by-step build order for assembling the service described in `SPEC.md` and `PLAN.md`. Each increment is sized to compile, run, and be independently verified before moving to the next. Order is chosen so a minimal end-to-end path exists early; quality, resilience, and tests layer on after.
|
||
|
||
When working through this file, mark increments complete by changing `[ ]` to `[x]`. If the design deviates from `PLAN.md`/`SPEC.md` mid-stream, update those documents too.
|
||
|
||
---
|
||
|
||
## Phase 0 — Scaffolding
|
||
|
||
### [x] 0.1 Solution + projects
|
||
- Create `FrameProcessor.sln` at repo root.
|
||
- Create `src/FrameProcessor/FrameProcessor.csproj` targeting `net10.0`, `Microsoft.NET.Sdk.Web`.
|
||
- Create `tests/FrameProcessor.Tests/FrameProcessor.Tests.csproj` (xUnit) referencing the main project.
|
||
- Add `.gitignore` (dotnet template), `global.json` pinning .NET 10 SDK.
|
||
- **DoD:** `dotnet build` and `dotnet test` both succeed (no tests yet, but harness runs).
|
||
|
||
### [x] 0.2 Minimal Program.cs
|
||
- Minimal hosting + controllers wired up.
|
||
- `GET /health` returning `{ status: "Healthy", mqttConnected: false }` (mqttConnected hardcoded for now).
|
||
- **DoD:** `dotnet run --project src/FrameProcessor` then `curl localhost:8080/health` returns 200 with the JSON.
|
||
|
||
### [x] 0.3 Package references
|
||
- Add to `FrameProcessor.csproj`:
|
||
- `SixLabors.ImageSharp`
|
||
- `MQTTnet`
|
||
- `Serilog.AspNetCore`, `Serilog.Sinks.Console`, `Serilog.Sinks.File`
|
||
- **DoD:** restore + build clean.
|
||
|
||
---
|
||
|
||
## Phase 1 — Domain value types
|
||
|
||
Each type lives in `src/FrameProcessor/Domain/`. Tests in `tests/FrameProcessor.Tests/`.
|
||
|
||
### [x] 1.1 `MacAddress`
|
||
- `record struct` storing 6 bytes.
|
||
- `static bool TryParse(string, out MacAddress)` accepting `AA:BB:CC:DD:EE:FF`, `aa-bb-cc-dd-ee-ff`, `AABBCCDDEEFF`, surrounding whitespace.
|
||
- `ToString()` → canonical lowercase no-separator (`aabbccddeeff`).
|
||
- JSON converter (System.Text.Json) using the permissive parse + canonical write.
|
||
- ASP.NET route value binding (e.g., `IParsable<MacAddress>`).
|
||
- **Tests:** `MacAddressTests` covering round-trip from all four input forms, equality on bytes, rejection of bad input.
|
||
|
||
### [x] 1.2 `FrameName`
|
||
- `record struct` wrapping a validated string.
|
||
- Constructor / `TryParse` enforces RFC 3986 unreserved (`A-Z a-z 0-9 - . _ ~`), non-empty.
|
||
- JSON converter + `IParsable`.
|
||
- **Tests:** `FrameNameTests` — accepts URL-safe, rejects whitespace/reserved chars with clear message.
|
||
|
||
### [x] 1.3 `Orientation`
|
||
- `enum { Landscape, Portrait }`.
|
||
- JSON converter reading/writing kebab-case lowercase.
|
||
|
||
### [x] 1.4 `Resolution`
|
||
- `record(int Width, int Height)`.
|
||
- `Resolution ForOrientation(Orientation o)` — swaps when portrait.
|
||
|
||
### [x] 1.5 `PaletteEntry`
|
||
- `record(string Name, Color DisplayColor, Color DeviceColor)` using `SixLabors.ImageSharp.Color`.
|
||
- JSON converter parses hex strings into `Color` at deserialize time (don't store hex strings on the type).
|
||
|
||
### [x] 1.6 `DitherAlgorithm`
|
||
- `enum { FloydSteinberg, Atkinson, Stucki, Jarvis }`.
|
||
- JSON converter reading kebab-case (`floyd-steinberg`, etc.).
|
||
|
||
### [x] 1.7 `ApiKey`
|
||
- `record struct` wrapping a string.
|
||
- `bool Matches(string candidate)` using `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes.
|
||
|
||
---
|
||
|
||
## Phase 2 — Configuration binding
|
||
|
||
### [x] 2.1 `MqttOptions`, `StorageOptions`, `UrlFetchOptions`, `ApiKeyOptions`
|
||
- POCOs in `src/FrameProcessor/Configuration/`.
|
||
- Bound from `appsettings.json` via `builder.Services.Configure<T>(...)`.
|
||
- Validate on startup (`ValidateOnStart` + `IValidateOptions<T>` or DataAnnotations).
|
||
|
||
### [x] 2.2 `FramesOptions` (from `frames.json`)
|
||
- Top-level `{ Frames: Frame[] }` POCO.
|
||
- Register `frames.json` as an additional config source with `reloadOnChange: true`.
|
||
- Bind via `IOptionsMonitor<FramesOptions>`.
|
||
- Custom validator enforcing rules from `SPEC.md` §6.2 (URL-safe name, MAC parseable, palette ≥2, hex parseable, dithering known).
|
||
|
||
### [x] 2.3 Startup vs reload asymmetry
|
||
- On startup: throw on any invalid frame (fail-fast).
|
||
- On `OnChange`: log warning, skip invalid frame, keep valid ones serving (`PLAN.md` line 130, `CLAUDE.md` "frames.json reload asymmetry").
|
||
- Add a `FramesRegistry` service that exposes `TryGetByName(FrameName)` / `TryGetByMac(MacAddress)` over the current valid set.
|
||
|
||
### [x] 2.4 Drop sample configs in repo root
|
||
- `appsettings.json` from `PLAN.md` §"Settings schemas".
|
||
- `frames.json` with one example frame (Spectra 6 palette).
|
||
- **DoD:** Service starts and logs the loaded frames; an invalid `frames.json` fails startup with a clear error.
|
||
|
||
---
|
||
|
||
## Phase 3 — Image pipeline
|
||
|
||
### [x] 3.1 `PaletteFactory`
|
||
- `static ReadOnlyMemory<Color> BuildDisplay(IReadOnlyList<PaletteEntry>)` and `BuildDevice(...)`.
|
||
- **Tests:** `PaletteFactoryTests` — count, ordering preserved.
|
||
|
||
### [x] 3.2 `DitheringRegistry`
|
||
- `IReadOnlyDictionary<DitherAlgorithm, IDither>` mapping each enum to the corresponding ImageSharp `KnownDitherings.*`.
|
||
|
||
### [x] 3.3 `IImagePipeline` + `ImagePipeline`
|
||
- Single method: `byte[] Process(Stream input, Frame frame)`.
|
||
- Steps (`SPEC.md` §3.2): decode → resize (cover/center-crop to oriented resolution) → dither against display palette → remap pixels display→device → rotate 90° CW if portrait → encode indexed PNG (`PngColorType.Palette`, smallest bit depth fitting the palette, `PaletteQuantizer(deviceColors)`).
|
||
- Output dimensions always match the frame's *declared* `width × height` regardless of orientation.
|
||
|
||
### [x] 3.4 `ImagePipelineTests` golden fixtures
|
||
- Check in `tests/FrameProcessor.Tests/Fixtures/inputs/` (landscape.jpg, portrait.jpg, tiny-1x1.png, non-divisible.jpg).
|
||
- Generate expected outputs once, commit under `Fixtures/expected/<algo>/<input>.png`.
|
||
- Assert byte-equality of produced PNG vs golden.
|
||
- Run for all four dithering algorithms against the canonical Spectra 6 palette.
|
||
- **DoD:** `dotnet test` green; deleting a fixture output and rerunning fails loud.
|
||
|
||
---
|
||
|
||
## Phase 4 — Storage
|
||
|
||
### [x] 4.1 `ImageStore`
|
||
- Constructor takes `IOptions<StorageOptions>`; ensures `ImageDirectory` exists on startup.
|
||
- `Task WriteAsync(MacAddress, ReadOnlyMemory<byte>, CancellationToken)` — writes to `{mac}.png.tmp`, then `File.Move(tmp, final, overwrite: true)`.
|
||
- `bool TryGetPath(MacAddress, out string path)` returning the on-disk path if present.
|
||
- **Manual check:** call from a unit test or scratch endpoint, confirm atomic rename behavior.
|
||
|
||
---
|
||
|
||
## Phase 5 — First end-to-end happy path (no MQTT yet)
|
||
|
||
### [x] 5.1 `FramesController.UploadImage` (multipart)
|
||
- Route: `POST /api/frames/{name}/image`.
|
||
- Resolve frame by name → 404 if unknown.
|
||
- Read multipart file part `image` (return 400 if missing).
|
||
- Call `ImagePipeline.Process`, then `ImageStore.WriteAsync`.
|
||
- Return `200 { frame, mac, url, processedAt, mqttPublished: false }` (MQTT stubbed).
|
||
|
||
### [x] 5.2 `ImageController.GetImage`
|
||
- Route: `GET /i/{mac}.png`.
|
||
- Normalize `{mac}` via `MacAddress.TryParse` → 404 on bad form.
|
||
- Look up frame by MAC → 404 if unknown or file absent.
|
||
- Return `FileStreamResult` with `Content-Type: image/png`, `Cache-Control: no-store`, `ETag` derived from file mtime.
|
||
|
||
### [x] 5.3 Manual end-to-end smoke
|
||
- Start service, `curl -F image=@photo.jpg .../api/frames/living-room/image` → 200.
|
||
- `curl .../i/aabbccddeeff.png > out.png` → image opens and looks dithered + remapped.
|
||
- **DoD:** above two commands work.
|
||
|
||
---
|
||
|
||
## Phase 6 — MQTT
|
||
|
||
### [x] 6.1 `MqttPublisher` hosted service
|
||
- Singleton `IHostedService` wrapping `IMqttClient` (MQTTnet v4).
|
||
- `WithReconnectDelay`, `WithCleanSession(false)`, credentials/TLS from `MqttOptions`.
|
||
- Exposes `bool IsConnected` for `/health`.
|
||
- On `StartAsync`: connect; on failure, log and continue (background reconnect handles it).
|
||
|
||
### [x] 6.2 `PublishAsync(MacAddress, CancellationToken) → Result`
|
||
- Topic: `{BaseTopic}/{mac}`, payload UTF-8 `"update"`, QoS 1, retained false.
|
||
- Returns success/failure (no throw).
|
||
|
||
### [x] 6.3 Wire into `FramesController`
|
||
- After successful save, call `PublishAsync`; set `mqttPublished` accordingly in the response.
|
||
|
||
### [x] 6.4 `/health` reports MQTT status
|
||
- Replace hardcoded `mqttConnected` with `MqttPublisher.IsConnected`.
|
||
|
||
### [x] 6.5 Background retry queue
|
||
- In-memory `Channel<MacAddress>` (one slot per frame; newer publish supersedes older — per `SPEC.md` §5.1 "Multiple queued publishes for the same frame collapse to the most recent one").
|
||
- Background loop drains with backoff sequence from `MqttOptions.RetryBackoffSeconds`.
|
||
- On reconnect, drain immediately.
|
||
- **Manual check:** stop broker, upload → 200 with `mqttPublished: false`; restart broker → `mosquitto_sub` sees the message within backoff window.
|
||
|
||
---
|
||
|
||
## Phase 7 — URL-fetch ingestion
|
||
|
||
### [x] 7.1 `IImageUrlFetcher`
|
||
- Constructor takes `HttpClient` (via `IHttpClientFactory`) configured with `UrlFetch.TimeoutSeconds` and `MaxRedirects`.
|
||
- `Task<Stream> FetchAsync(Uri, CancellationToken)` — streams response, aborts if content exceeds `MaxBytes`.
|
||
- Throws a typed exception (`ImageFetchException`) for timeout / non-2xx / too-large / redirect-loop.
|
||
|
||
### [x] 7.2 `FramesController.UploadImageUrl`
|
||
- Route: `POST /api/frames/{name}/image-url`, body `{ "url": "..." }`.
|
||
- Fetch → pipeline → store → publish. Same response shape as 5.1.
|
||
- Map `ImageFetchException` to `502 Bad Gateway`.
|
||
|
||
### [x] 7.3 Manual check
|
||
- `curl -H "Content-Type: application/json" -d '{"url":"https://..."}' .../image-url` works end-to-end.
|
||
|
||
---
|
||
|
||
## Phase 8 — Auth + concurrency + robustness
|
||
|
||
### [x] 8.1 `ApiKeyMiddleware`
|
||
- Matches request path `/api/*`; reads `X-Api-Key` header; constant-time compare against `ApiKeyOptions` only if `ApiKeyOptions`is set to non-empty.
|
||
- 401 on mismatch. `/i/{mac}.png` and `/health` unaffected.
|
||
|
||
### [x] 8.2 `FrameLockProvider`
|
||
- `ConcurrentDictionary<FrameName, SemaphoreSlim>` (each `SemaphoreSlim(1, 1)`).
|
||
- `Task<IDisposable> AcquireAsync(FrameName, CancellationToken)` returning a disposable that releases on dispose.
|
||
|
||
### [x] 8.3 Wrap full pipeline in lock
|
||
- In `FramesController`, acquire the frame's lock before fetch/decode and release after publish-attempt completes (`CLAUDE.md` "Per-frame serialization").
|
||
- **Manual check:** fire two concurrent uploads to the same frame → both return 200, only one PNG on disk reflects whichever finished last, two MQTT publishes (or one if collapsed by retry queue).
|
||
|
||
---
|
||
|
||
## Phase 9 — Observability
|
||
|
||
### [x] 9.1 Serilog wiring
|
||
- `UseSerilog` with console + rolling file sinks (`logs/frame-processor-.log`, daily).
|
||
- Structured fields: `FrameName`, `MacAddress`, `InputSource` (file/url), `InputBytes`, `OutputBytes`, `ElapsedMs`, `MqttPublished`.
|
||
- Log one info line per upload at completion.
|
||
|
||
---
|
||
|
||
## Phase 10 — Packaging
|
||
|
||
### [x] 10.1 `Dockerfile`
|
||
- Multi-stage: SDK build → runtime (`mcr.microsoft.com/dotnet/aspnet:10.0`).
|
||
- Expose 8080; create `/data/images`.
|
||
- Non-root user.
|
||
|
||
### [ ] 10.2 `docker-compose.yml`
|
||
- `frame-processor` service (build from Dockerfile, mount `frames.json` and `/data/images`).
|
||
- `mosquitto` broker with anonymous access on the local network.
|
||
- Single network so the service can reach `mosquitto:1883`.
|
||
|
||
### [ ] 10.3 End-to-end via compose
|
||
- `docker compose up --build` → `/health` returns `mqttConnected: true`.
|
||
- Run the full `PLAN.md` §Verification checklist (steps 2–7).
|
||
|
||
---
|
||
|
||
## Phase 11 — Final verification
|
||
|
||
### [ ] 11.1 Run `PLAN.md` verification checklist top to bottom
|
||
- All 8 steps pass (step 9, real frame, is optional from this codebase's perspective).
|
||
|
||
### [ ] 11.2 README
|
||
- Short `README.md` pointing at `SPEC.md`, `PLAN.md`, `CLAUDE.md` and showing the `docker compose up` + smoke-test commands.
|
||
|
||
---
|
||
|
||
## Notes for future-me
|
||
|
||
- **Don't reorder dither/remap.** Dither against `DisplayColor`; remap to `DeviceColor` after. Reversing produces visibly wrong output.
|
||
- **Normalize MAC at every boundary.** Config load, route binding, MQTT topic, file path — all lowercase no-separator.
|
||
- **Atomic writes are non-negotiable.** Never write directly to `{mac}.png`; always tmp + rename.
|
||
- **MQTT failure ≠ upload failure.** A broker outage produces a 200 with `mqttPublished: false`, not a 5xx.
|
||
- **Hot-reload asymmetry.** Startup is strict; reload is lenient and skips bad frames.
|
||
- **No premature abstractions.** If a step says "single class," resist the urge to introduce interfaces beyond what `PLAN.md` lists.
|