frame-processor/IMPLEMENTATION.md

# Frame Processor — Implementation Increments

Step-by-step build order for assembling the service described in `SPEC.md` and `PLAN.md`. Each increment is sized to compile, run, and be independently verified before moving to the next. Order is chosen so a minimal end-to-end path exists early; quality, resilience, and tests layer on after.

When working through this file, mark increments complete by changing `[ ]` to `[x]`. If the design deviates from `PLAN.md`/`SPEC.md` mid-stream, update those documents too.

---

## Phase 0 — Scaffolding

### [x] 0.1 Solution + projects
- Create `FrameProcessor.sln` at repo root.
- Create `src/FrameProcessor/FrameProcessor.csproj` targeting `net10.0`, `Microsoft.NET.Sdk.Web`.
- Create `tests/FrameProcessor.Tests/FrameProcessor.Tests.csproj` (xUnit) referencing the main project.
- Add `.gitignore` (dotnet template), `global.json` pinning .NET 10 SDK.
- **DoD:** `dotnet build` and `dotnet test` both succeed (no tests yet, but harness runs).

### [x] 0.2 Minimal Program.cs
- Minimal hosting + controllers wired up.
- `GET /health` returning `{ status: "Healthy", mqttConnected: false }` (mqttConnected hardcoded for now).
- **DoD:** `dotnet run --project src/FrameProcessor` then `curl localhost:8080/health` returns 200 with the JSON.

### [x] 0.3 Package references
- Add to `FrameProcessor.csproj`:
  - `SixLabors.ImageSharp`
  - `MQTTnet`
  - `Serilog.AspNetCore`, `Serilog.Sinks.Console`, `Serilog.Sinks.File`
- **DoD:** restore + build clean.

---

## Phase 1 — Domain value types

Each type lives in `src/FrameProcessor/Domain/`. Tests in `tests/FrameProcessor.Tests/`.

### [x] 1.1 `MacAddress`
- `record struct` storing 6 bytes.
- `static bool TryParse(string, out MacAddress)` accepting `AA:BB:CC:DD:EE:FF`, `aa-bb-cc-dd-ee-ff`, `AABBCCDDEEFF`, surrounding whitespace.
- `ToString()` → canonical lowercase no-separator (`aabbccddeeff`).
- JSON converter (System.Text.Json) using the permissive parse + canonical write.
- ASP.NET route value binding (e.g., `IParsable<MacAddress>`).
- **Tests:** `MacAddressTests` covering round-trip from all four input forms, equality on bytes, rejection of bad input.

### [x] 1.2 `FrameName`
- `record struct` wrapping a validated string.
- Constructor / `TryParse` enforces RFC 3986 unreserved (`A-Z a-z 0-9 - . _ ~`), non-empty.
- JSON converter + `IParsable`.
- **Tests:** `FrameNameTests` — accepts URL-safe, rejects whitespace/reserved chars with clear message.

### [x] 1.3 `Orientation`
- `enum { Landscape, Portrait }`.
- JSON converter reading/writing kebab-case lowercase.

### [x] 1.4 `Resolution`
- `record(int Width, int Height)`.
- `Resolution ForOrientation(Orientation o)` — swaps when portrait.

### [x] 1.5 `PaletteEntry`
- `record(string Name, Color DisplayColor, Color DeviceColor)` using `SixLabors.ImageSharp.Color`.
- JSON converter parses hex strings into `Color` at deserialize time (don't store hex strings on the type).

### [x] 1.6 `DitherAlgorithm`
- `enum { FloydSteinberg, Atkinson, Stucki, Jarvis }`.
- JSON converter reading kebab-case (`floyd-steinberg`, etc.).

### [x] 1.7 `ApiKey`
- `record struct` wrapping a string.
- `bool Matches(string candidate)` using `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes.

---

## Phase 2 — Configuration binding

### [x] 2.1 `MqttOptions`, `StorageOptions`, `UrlFetchOptions`, `ApiKeyOptions`
- POCOs in `src/FrameProcessor/Configuration/`.
- Bound from `appsettings.json` via `builder.Services.Configure<T>(...)`.
- Validate on startup (`ValidateOnStart` + `IValidateOptions<T>` or DataAnnotations).

### [x] 2.2 `FramesOptions` (from `frames.json`)
- Top-level `{ Frames: Frame[] }` POCO.
- Register `frames.json` as an additional config source with `reloadOnChange: true`.
- Bind via `IOptionsMonitor<FramesOptions>`.
- Custom validator enforcing rules from `SPEC.md` §6.2 (URL-safe name, MAC parseable, palette ≥2, hex parseable, dithering known).

### [x] 2.3 Startup vs reload asymmetry
- On startup: throw on any invalid frame (fail-fast).
- On `OnChange`: log warning, skip invalid frame, keep valid ones serving (`PLAN.md` line 130, `CLAUDE.md` "frames.json reload asymmetry").
- Add a `FramesRegistry` service that exposes `TryGetByName(FrameName)` / `TryGetByMac(MacAddress)` over the current valid set.

### [x] 2.4 Drop sample configs in repo root
- `appsettings.json` from `PLAN.md` §"Settings schemas".
- `frames.json` with one example frame (Spectra 6 palette).
- **DoD:** Service starts and logs the loaded frames; an invalid `frames.json` fails startup with a clear error.

---

## Phase 3 — Image pipeline

### [x] 3.1 `PaletteFactory`
- `static ReadOnlyMemory<Color> BuildDisplay(IReadOnlyList<PaletteEntry>)` and `BuildDevice(...)`.
- **Tests:** `PaletteFactoryTests` — count, ordering preserved.

### [x] 3.2 `DitheringRegistry`
- `IReadOnlyDictionary<DitherAlgorithm, IDither>` mapping each enum to the corresponding ImageSharp `KnownDitherings.*`.

### [x] 3.3 `IImagePipeline` + `ImagePipeline`
- Single method: `byte[] Process(Stream input, Frame frame)`.
- Steps (`SPEC.md` §3.2): decode → resize (cover/center-crop to oriented resolution) → dither against display palette → remap pixels display→device → rotate 90° CW if portrait → encode indexed PNG (`PngColorType.Palette`, smallest bit depth fitting the palette, `PaletteQuantizer(deviceColors)`).
- Output dimensions always match the frame's *declared* `width × height` regardless of orientation.

### [x] 3.4 `ImagePipelineTests` golden fixtures
- Check in `tests/FrameProcessor.Tests/Fixtures/inputs/` (landscape.jpg, portrait.jpg, tiny-1x1.png, non-divisible.jpg).
- Generate expected outputs once, commit under `Fixtures/expected/<algo>/<input>.png`.
- Assert byte-equality of produced PNG vs golden.
- Run for all four dithering algorithms against the canonical Spectra 6 palette.
- **DoD:** `dotnet test` green; deleting a fixture output and rerunning fails loud.

---

## Phase 4 — Storage

### [x] 4.1 `ImageStore`
- Constructor takes `IOptions<StorageOptions>`; ensures `ImageDirectory` exists on startup.
- `Task WriteAsync(MacAddress, ReadOnlyMemory<byte>, CancellationToken)` — writes to `{mac}.png.tmp`, then `File.Move(tmp, final, overwrite: true)`.
- `bool TryGetPath(MacAddress, out string path)` returning the on-disk path if present.
- **Manual check:** call from a unit test or scratch endpoint, confirm atomic rename behavior.

---

## Phase 5 — First end-to-end happy path (no MQTT yet)

### [x] 5.1 `FramesController.UploadImage` (multipart)
- Route: `POST /api/frames/{name}/image`.
- Resolve frame by name → 404 if unknown.
- Read multipart file part `image` (return 400 if missing).
- Call `ImagePipeline.Process`, then `ImageStore.WriteAsync`.
- Return `200 { frame, mac, url, processedAt, mqttPublished: false }` (MQTT stubbed).

### [x] 5.2 `ImageController.GetImage`
- Route: `GET /i/{mac}.png`.
- Normalize `{mac}` via `MacAddress.TryParse` → 404 on bad form.
- Look up frame by MAC → 404 if unknown or file absent.
- Return `FileStreamResult` with `Content-Type: image/png`, `Cache-Control: no-store`, `ETag` derived from file mtime.

### [x] 5.3 Manual end-to-end smoke
- Start service, `curl -F image=@photo.jpg .../api/frames/living-room/image` → 200.
- `curl .../i/aabbccddeeff.png > out.png` → image opens and looks dithered + remapped.
- **DoD:** above two commands work.

---

## Phase 6 — MQTT

### [x] 6.1 `MqttPublisher` hosted service
- Singleton `IHostedService` wrapping `IMqttClient` (MQTTnet v4).
- `WithReconnectDelay`, `WithCleanSession(false)`, credentials/TLS from `MqttOptions`.
- Exposes `bool IsConnected` for `/health`.
- On `StartAsync`: connect; on failure, log and continue (background reconnect handles it).

### [x] 6.2 `PublishAsync(MacAddress, CancellationToken) → Result`
- Topic: `{BaseTopic}/{mac}`, payload UTF-8 `"update"`, QoS 1, retained false.
- Returns success/failure (no throw).

### [x] 6.3 Wire into `FramesController`
- After successful save, call `PublishAsync`; set `mqttPublished` accordingly in the response.

### [x] 6.4 `/health` reports MQTT status
- Replace hardcoded `mqttConnected` with `MqttPublisher.IsConnected`.

### [x] 6.5 Background retry queue
- In-memory `Channel<MacAddress>` (one slot per frame; newer publish supersedes older — per `SPEC.md` §5.1 "Multiple queued publishes for the same frame collapse to the most recent one").
- Background loop drains with backoff sequence from `MqttOptions.RetryBackoffSeconds`.
- On reconnect, drain immediately.
- **Manual check:** stop broker, upload → 200 with `mqttPublished: false`; restart broker → `mosquitto_sub` sees the message within backoff window.

---

## Phase 7 — URL-fetch ingestion

### [x] 7.1 `IImageUrlFetcher`
- Constructor takes `HttpClient` (via `IHttpClientFactory`) configured with `UrlFetch.TimeoutSeconds` and `MaxRedirects`.
- `Task<Stream> FetchAsync(Uri, CancellationToken)` — streams response, aborts if content exceeds `MaxBytes`.
- Throws a typed exception (`ImageFetchException`) for timeout / non-2xx / too-large / redirect-loop.

### [x] 7.2 `FramesController.UploadImageUrl`
- Route: `POST /api/frames/{name}/image-url`, body `{ "url": "..." }`.
- Fetch → pipeline → store → publish. Same response shape as 5.1.
- Map `ImageFetchException` to `502 Bad Gateway`.

### [x] 7.3 Manual check
- `curl -H "Content-Type: application/json" -d '{"url":"https://..."}' .../image-url` works end-to-end.

---

## Phase 8 — Auth + concurrency + robustness

### [x] 8.1 `ApiKeyMiddleware`
- Matches request path `/api/*`; reads `X-Api-Key` header; constant-time compare against `ApiKeyOptions` only if `ApiKeyOptions`is set to non-empty.
- 401 on mismatch. `/i/{mac}.png` and `/health` unaffected.

### [x] 8.2 `FrameLockProvider`
- `ConcurrentDictionary<FrameName, SemaphoreSlim>` (each `SemaphoreSlim(1, 1)`).
- `Task<IDisposable> AcquireAsync(FrameName, CancellationToken)` returning a disposable that releases on dispose.

### [ ] 8.3 Wrap full pipeline in lock
- In `FramesController`, acquire the frame's lock before fetch/decode and release after publish-attempt completes (`CLAUDE.md` "Per-frame serialization").
- **Manual check:** fire two concurrent uploads to the same frame → both return 200, only one PNG on disk reflects whichever finished last, two MQTT publishes (or one if collapsed by retry queue).

---

## Phase 9 — Observability

### [ ] 9.1 Serilog wiring
- `UseSerilog` with console + rolling file sinks (`logs/frame-processor-.log`, daily).
- Structured fields: `FrameName`, `MacAddress`, `InputSource` (file/url), `InputBytes`, `OutputBytes`, `ElapsedMs`, `MqttPublished`.
- Log one info line per upload at completion.

---

## Phase 10 — Packaging

### [ ] 10.1 `Dockerfile`
- Multi-stage: SDK build → runtime (`mcr.microsoft.com/dotnet/aspnet:10.0`).
- Expose 8080; create `/data/images`.
- Non-root user.

### [ ] 10.2 `docker-compose.yml`
- `frame-processor` service (build from Dockerfile, mount `frames.json` and `/data/images`).
- `mosquitto` broker with anonymous access on the local network.
- Single network so the service can reach `mosquitto:1883`.

### [ ] 10.3 End-to-end via compose
- `docker compose up --build` → `/health` returns `mqttConnected: true`.
- Run the full `PLAN.md` §Verification checklist (steps 2–7).

---

## Phase 11 — Final verification

### [ ] 11.1 Run `PLAN.md` verification checklist top to bottom
- All 8 steps pass (step 9, real frame, is optional from this codebase's perspective).

### [ ] 11.2 README
- Short `README.md` pointing at `SPEC.md`, `PLAN.md`, `CLAUDE.md` and showing the `docker compose up` + smoke-test commands.

---

## Notes for future-me

- **Don't reorder dither/remap.** Dither against `DisplayColor`; remap to `DeviceColor` after. Reversing produces visibly wrong output.
- **Normalize MAC at every boundary.** Config load, route binding, MQTT topic, file path — all lowercase no-separator.
- **Atomic writes are non-negotiable.** Never write directly to `{mac}.png`; always tmp + rename.
- **MQTT failure ≠ upload failure.** A broker outage produces a 200 with `mqttPublished: false`, not a 5xx.
- **Hot-reload asymmetry.** Startup is strict; reload is lenient and skips bad frames.
- **No premature abstractions.** If a step says "single class," resist the urge to introduce interfaces beyond what `PLAN.md` lists.