frame-processor/IMPLEMENTATION.md

Frame Processor — Implementation Increments

Step-by-step build order for assembling the service described in SPEC.md and PLAN.md. Each increment is sized to compile, run, and be independently verified before moving to the next. Order is chosen so a minimal end-to-end path exists early; quality, resilience, and tests layer on after.

When working through this file, mark increments complete by changing [ ] to [x]. If the design deviates from PLAN.md/SPEC.md mid-stream, update those documents too.

---

Phase 0 — Scaffolding

[x] 0.1 Solution + projects

• Create FrameProcessor.sln at repo root.
• Create src/FrameProcessor/FrameProcessor.csproj targeting net10.0, Microsoft.NET.Sdk.Web.
• Create tests/FrameProcessor.Tests/FrameProcessor.Tests.csproj (xUnit) referencing the main project.
• Add .gitignore (dotnet template), global.json pinning .NET 10 SDK.
• DoD: dotnet build and dotnet test both succeed (no tests yet, but harness runs).

[x] 0.2 Minimal Program.cs

• Minimal hosting + controllers wired up.
• GET /health returning { status: "Healthy", mqttConnected: false } (mqttConnected hardcoded for now).
• DoD: dotnet run --project src/FrameProcessor then curl localhost:8080/health returns 200 with the JSON.

[x] 0.3 Package references

• Add to FrameProcessor.csproj:
  • SixLabors.ImageSharp
  • MQTTnet
  • Serilog.AspNetCore, Serilog.Sinks.Console, Serilog.Sinks.File
• DoD: restore + build clean.

---

Phase 1 — Domain value types

Each type lives in src/FrameProcessor/Domain/. Tests in tests/FrameProcessor.Tests/.

[x] 1.1 MacAddress

• record struct storing 6 bytes.
• static bool TryParse(string, out MacAddress) accepting AA:BB:CC:DD:EE:FF, aa-bb-cc-dd-ee-ff, AABBCCDDEEFF, surrounding whitespace.
• ToString() → canonical lowercase no-separator (aabbccddeeff).
• JSON converter (System.Text.Json) using the permissive parse + canonical write.
• ASP.NET route value binding (e.g., IParsable<MacAddress>).
• Tests: MacAddressTests covering round-trip from all four input forms, equality on bytes, rejection of bad input.

[x] 1.2 FrameName

• record struct wrapping a validated string.
• Constructor / TryParse enforces RFC 3986 unreserved (A-Z a-z 0-9 - . _ ~), non-empty.
• JSON converter + IParsable.
• Tests: FrameNameTests — accepts URL-safe, rejects whitespace/reserved chars with clear message.

[x] 1.3 Orientation

• enum { Landscape, Portrait }.
• JSON converter reading/writing kebab-case lowercase.

[x] 1.4 Resolution

• record(int Width, int Height).
• Resolution ForOrientation(Orientation o) — swaps when portrait.

[x] 1.5 PaletteEntry

• record(string Name, Color DisplayColor, Color DeviceColor) using SixLabors.ImageSharp.Color.
• JSON converter parses hex strings into Color at deserialize time (don't store hex strings on the type).

[x] 1.6 DitherAlgorithm

• enum { FloydSteinberg, Atkinson, Stucki, Jarvis }.
• JSON converter reading kebab-case (floyd-steinberg, etc.).

[x] 1.7 ApiKey

• record struct wrapping a string.
• bool Matches(string candidate) using CryptographicOperations.FixedTimeEquals over UTF-8 bytes.

---

Phase 2 — Configuration binding

[x] 2.1 MqttOptions, StorageOptions, UrlFetchOptions, ApiKeyOptions

• POCOs in src/FrameProcessor/Configuration/.
• Bound from appsettings.json via builder.Services.Configure<T>(...).
• Validate on startup (ValidateOnStart + IValidateOptions<T> or DataAnnotations).

[x] 2.2 FramesOptions (from frames.json)

• Top-level { Frames: Frame[] } POCO.
• Register frames.json as an additional config source with reloadOnChange: true.
• Bind via IOptionsMonitor<FramesOptions>.
• Custom validator enforcing rules from SPEC.md §6.2 (URL-safe name, MAC parseable, palette ≥2, hex parseable, dithering known).

[x] 2.3 Startup vs reload asymmetry

• On startup: throw on any invalid frame (fail-fast).
• On OnChange: log warning, skip invalid frame, keep valid ones serving (PLAN.md line 130, CLAUDE.md "frames.json reload asymmetry").
• Add a FramesRegistry service that exposes TryGetByName(FrameName) / TryGetByMac(MacAddress) over the current valid set.

[x] 2.4 Drop sample configs in repo root

• appsettings.json from PLAN.md §"Settings schemas".
• frames.json with one example frame (Spectra 6 palette).
• DoD: Service starts and logs the loaded frames; an invalid frames.json fails startup with a clear error.

---

Phase 3 — Image pipeline

[x] 3.1 PaletteFactory

• static ReadOnlyMemory<Color> BuildDisplay(IReadOnlyList<PaletteEntry>) and BuildDevice(...).
• Tests: PaletteFactoryTests — count, ordering preserved.

[x] 3.2 DitheringRegistry

• IReadOnlyDictionary<DitherAlgorithm, IDither> mapping each enum to the corresponding ImageSharp KnownDitherings.*.

[x] 3.3 IImagePipeline + ImagePipeline

• Single method: byte[] Process(Stream input, Frame frame).
• Steps (SPEC.md §3.2): decode → resize (cover/center-crop to oriented resolution) → dither against display palette → remap pixels display→device → rotate 90° CW if portrait → encode indexed PNG (PngColorType.Palette, smallest bit depth fitting the palette, PaletteQuantizer(deviceColors)).
• Output dimensions always match the frame's declared width × height regardless of orientation.

[x] 3.4 ImagePipelineTests golden fixtures

• Check in tests/FrameProcessor.Tests/Fixtures/inputs/ (landscape.jpg, portrait.jpg, tiny-1x1.png, non-divisible.jpg).
• Generate expected outputs once, commit under Fixtures/expected/<algo>/<input>.png.
• Assert byte-equality of produced PNG vs golden.
• Run for all four dithering algorithms against the canonical Spectra 6 palette.
• DoD: dotnet test green; deleting a fixture output and rerunning fails loud.

---

Phase 4 — Storage

[x] 4.1 ImageStore

• Constructor takes IOptions<StorageOptions>; ensures ImageDirectory exists on startup.
• Task WriteAsync(MacAddress, ReadOnlyMemory<byte>, CancellationToken) — writes to {mac}.png.tmp, then File.Move(tmp, final, overwrite: true).
• bool TryGetPath(MacAddress, out string path) returning the on-disk path if present.
• Manual check: call from a unit test or scratch endpoint, confirm atomic rename behavior.

---

Phase 5 — First end-to-end happy path (no MQTT yet)

[x] 5.1 FramesController.UploadImage (multipart)

• Route: POST /api/frames/{name}/image.
• Resolve frame by name → 404 if unknown.
• Read multipart file part image (return 400 if missing).
• Call ImagePipeline.Process, then ImageStore.WriteAsync.
• Return 200 { frame, mac, url, processedAt, mqttPublished: false } (MQTT stubbed).

[x] 5.2 ImageController.GetImage

• Route: GET /i/{mac}.png.
• Normalize {mac} via MacAddress.TryParse → 404 on bad form.
• Look up frame by MAC → 404 if unknown or file absent.
• Return FileStreamResult with Content-Type: image/png, Cache-Control: no-store, ETag derived from file mtime.

[x] 5.3 Manual end-to-end smoke

• Start service, curl -F image=@photo.jpg .../api/frames/living-room/image → 200.
• curl .../i/aabbccddeeff.png > out.png → image opens and looks dithered + remapped.
• DoD: above two commands work.

---

Phase 6 — MQTT

[x] 6.1 MqttPublisher hosted service

• Singleton IHostedService wrapping IMqttClient (MQTTnet v4).
• WithReconnectDelay, WithCleanSession(false), credentials/TLS from MqttOptions.
• Exposes bool IsConnected for /health.
• On StartAsync: connect; on failure, log and continue (background reconnect handles it).

[x] 6.2 PublishAsync(MacAddress, CancellationToken) → Result

• Topic: {BaseTopic}/{mac}, payload UTF-8 "update", QoS 1, retained false.
• Returns success/failure (no throw).

[x] 6.3 Wire into FramesController

• After successful save, call PublishAsync; set mqttPublished accordingly in the response.

[x] 6.4 /health reports MQTT status

• Replace hardcoded mqttConnected with MqttPublisher.IsConnected.

[x] 6.5 Background retry queue

• In-memory Channel<MacAddress> (one slot per frame; newer publish supersedes older — per SPEC.md §5.1 "Multiple queued publishes for the same frame collapse to the most recent one").
• Background loop drains with backoff sequence from MqttOptions.RetryBackoffSeconds.
• On reconnect, drain immediately.
• Manual check: stop broker, upload → 200 with mqttPublished: false; restart broker → mosquitto_sub sees the message within backoff window.

---

Phase 7 — URL-fetch ingestion

[x] 7.1 IImageUrlFetcher

• Constructor takes HttpClient (via IHttpClientFactory) configured with UrlFetch.TimeoutSeconds and MaxRedirects.
• Task<Stream> FetchAsync(Uri, CancellationToken) — streams response, aborts if content exceeds MaxBytes.
• Throws a typed exception (ImageFetchException) for timeout / non-2xx / too-large / redirect-loop.

[x] 7.2 FramesController.UploadImageUrl

• Route: POST /api/frames/{name}/image-url, body { "url": "..." }.
• Fetch → pipeline → store → publish. Same response shape as 5.1.
• Map ImageFetchException to 502 Bad Gateway.

[x] 7.3 Manual check

• curl -H "Content-Type: application/json" -d '{"url":"https://..."}' .../image-url works end-to-end.

---

Phase 8 — Auth + concurrency + robustness

[x] 8.1 ApiKeyMiddleware

• Matches request path /api/*; reads X-Api-Key header; constant-time compare against ApiKeyOptions only if ApiKeyOptionsis set to non-empty.
• 401 on mismatch. /i/{mac}.png and /health unaffected.

[ ] 8.2 FrameLockProvider

• ConcurrentDictionary<FrameName, SemaphoreSlim> (each SemaphoreSlim(1, 1)).
• Task<IDisposable> AcquireAsync(FrameName, CancellationToken) returning a disposable that releases on dispose.

[ ] 8.3 Wrap full pipeline in lock

• In FramesController, acquire the frame's lock before fetch/decode and release after publish-attempt completes (CLAUDE.md "Per-frame serialization").
• Manual check: fire two concurrent uploads to the same frame → both return 200, only one PNG on disk reflects whichever finished last, two MQTT publishes (or one if collapsed by retry queue).

---

Phase 9 — Observability

[ ] 9.1 Serilog wiring

• UseSerilog with console + rolling file sinks (logs/frame-processor-.log, daily).
• Structured fields: FrameName, MacAddress, InputSource (file/url), InputBytes, OutputBytes, ElapsedMs, MqttPublished.
• Log one info line per upload at completion.

---

Phase 10 — Packaging

[ ] 10.1 Dockerfile

• Multi-stage: SDK build → runtime (mcr.microsoft.com/dotnet/aspnet:10.0).
• Expose 8080; create /data/images.
• Non-root user.

[ ] 10.2 docker-compose.yml

• frame-processor service (build from Dockerfile, mount frames.json and /data/images).
• mosquitto broker with anonymous access on the local network.
• Single network so the service can reach mosquitto:1883.

[ ] 10.3 End-to-end via compose

• docker compose up --build → /health returns mqttConnected: true.
• Run the full PLAN.md §Verification checklist (steps 2–7).

---

Phase 11 — Final verification

[ ] 11.1 Run PLAN.md verification checklist top to bottom

• All 8 steps pass (step 9, real frame, is optional from this codebase's perspective).

[ ] 11.2 README

• Short README.md pointing at SPEC.md, PLAN.md, CLAUDE.md and showing the docker compose up + smoke-test commands.

---

Notes for future-me

• Don't reorder dither/remap. Dither against DisplayColor; remap to DeviceColor after. Reversing produces visibly wrong output.
• Normalize MAC at every boundary. Config load, route binding, MQTT topic, file path — all lowercase no-separator.
• Atomic writes are non-negotiable. Never write directly to {mac}.png; always tmp + rename.
• MQTT failure ≠ upload failure. A broker outage produces a 200 with mqttPublished: false, not a 5xx.
• Hot-reload asymmetry. Startup is strict; reload is lenient and skips bad frames.
• No premature abstractions. If a step says "single class," resist the urge to introduce interfaces beyond what PLAN.md lists.