Hybrid Image Search Demo on ICE-ICE-BEAR-BLOG

hybrid-image-search-demo Dev Log #16 — Lens Preset Expansion, Hover Previews, OTLP Telemetry

Fri, 17 Apr 2026 00:00:00 +0900

Overview

Three commits, three themes. Lens presets expanded to 5 general options plus a beauty-specific Briese lighting preset. AnglePicker and LensPicker gained 31 hover-preview thumbnails so users can see what each preset actually produces before clicking. The headline work was wiring the production FastAPI backend’s traces, metrics, and logs to a local Grafana Alloy agent over OTLP, which forwards to Grafana Cloud. The same interval saw the telemetry’s first real use — debugging a user’s missing auto-fill tone image by following a trace through Loki. Two sessions, three commits, 5h 54m total.

Previous post: hybrid-image-search-demo Dev Log #15

graph TD
 A["FastAPI backend (prod)"] -->|"OTLP HTTP :4318"| B["Grafana Alloy (per EC2)"]
 B --> C["Grafana Cloud: Traces (Tempo)"]
 B --> D["Grafana Cloud: Metrics (Prometheus)"]
 B --> E["Grafana Cloud: Logs (Loki)"]
 F["logging.getLogger().error(...)"] -->|"stdlib LogRecord"| A
 G["FastAPI span"] -->|"auto-instrumented"| A
 C -. "trace_id link" .-> E

Lens Presets — 5 General + 1 Beauty

c4fb076 feat(gen): expand lens presets to 5 general + beauty w/ Briese lighting touched backend/src/generation/lens_presets.py. The previous three lens options weren’t enough to cover the generation scenarios our users wanted. This change did two things:

Expand to 5 general focal lengths — 24mm (wide), 35mm (street/environmental), 50mm (natural), 85mm (portrait), 135mm (tight). A standard photography focal-length ladder.
Add a beauty-specific preset — Briese lighting. Briese is the large reflector rig used heavily in advertising and beauty photography. This is the first time we’ve injected a lighting directive alongside focal length. prompt.py’s build_generation_prompt now combines the lens text with the lighting directive when the category is beauty.

Test coverage: one new unit test in backend/tests/test_lens_presets.py asserts each preset produces the expected string through the prompt builder.

Frontend LensPicker.tsx grew its radio options to five and grouped the beauty preset separately. GeneratedImageDetail.tsx surfaces the selected lens text in the info panel.

31 Hover-Preview Thumbnails

4b886a9 feat(ui): hover-preview examples for angle/lens pickers is a 31-file commit. Most of those files are the actual example images under frontend/public/preset-examples/angles/*.jpg and lens/*.jpg — bird’s-eye-view, close-up-cu, dutch-angle, extreme-close-up-ecu, extreme-long-shot-els, eye-level, high-angle, insert-shot, long-shot-ls, low-angle, master-shot, medium-close-up-mcu, and so on.

graph LR
 A["User hovers an angle/lens option"] --> B["AnglePicker/LensPicker computes preview URL"]
 B --> C["/preset-examples/{kind}/{slug}.jpg"]
 C --> D["Floating tooltip renders the image"]
 D --> E["User picks with visual context"]

A generator script at backend/scripts/generate_preset_examples.py batch-produced these thumbnails, calling the same generation pipeline from previous posts on a fixed reference character and dumping outputs into frontend/public/preset-examples/. .gitignore was updated to exclude the raw source materials.

AnglePicker.tsx and LensPicker.tsx share a floating tooltip pattern on hover. The UX call here is to stop making users pick by jargon alone — “extreme-long-shot (ELS)” is opaque if you’ve never shot cinema, but a thumbnail communicates it instantly.

Grafana OTLP Telemetry

The weight of the interval is 7a55e9b feat(telemetry): ship prod logs to Alloy/Grafana Cloud via OTLP. Only four files changed, but it’s a significant operational shift.

The Brief

User brief was precise: “I’m on the free Grafana tier. I’d like at least the API logs, or at minimum any API-level error logs. Confirm it’s possible under the free tier.” Collect prod only, manage the packages globally through pyproject.toml, enable prod-only via an .env variable.

Architecture

flowchart LR
 A["FastAPI app"] -->|"OTLP HTTP"| B["Alloy (localhost:4318)"]
 B --> C["Grafana Cloud OTLP endpoint"]
 C --> D["Tempo / Loki / Prometheus"]
 E["stdlib logging"] -->|"LoggingHandler"| A

The FastAPI app emits to a local Alloy agent over OTLP HTTP on port 4318. Alloy forwards to Grafana Cloud’s OTLP endpoint. This puts Grafana Cloud credentials in Alloy’s config instead of the app’s environment — rotating prod images doesn’t expose the Grafana token.

Implementation

backend/src/telemetry.py — initialization behind a _telemetry_enabled flag gated on DEPLOYMENT_ENV == "prod". Traces via OTLPSpanExporter, metrics via OTLPMetricExporter, logs via OTLPLogExporter. Auto-instrumentation through FastAPIInstrumentor, SQLAlchemyInstrumentor, and LoggingInstrumentor.
stdlib logging → OTLP bridge. The critical detail. A root-level LoggingHandler attaches to the stdlib root logger so every logging.getLogger(...) call (uvicorn access logs, SQLAlchemy chatter, app logger.errors) ships as an OTLP log. The handler reads the active span context on emit and attaches trace_id to each LogRecord — so clicking a log line in Grafana jumps to the trace that produced it.
Global pyproject.toml additions. opentelemetry-instrumentation-fastapi, -sqlalchemy, -logging, -exporter-otlp-proto-http, -exporter-otlp-proto-grpc, all pinned to >=0.54b0 / >=1.33.0.
infra/alloy/config.alloy — Alloy config. OTLP receiver opens grpc on 4317 and http on 4318, passes through a batch processor, forwards to Grafana Cloud. Short and boring, which is the right shape for infra config.
infra/alloy/SETUP.md — per-EC2 manual install: sudo apt install grafana-alloy, drop the config, enable via systemd.

Deploy and a PM2 Gotcha

Deployed dev → prod through the /deploy-diff workflow. Verified traces arriving in Grafana Cloud’s Explore view. One trap documented but not yet fixed:

ecosystem.config.js sets DEPLOYMENT_ENV: process.env.DEPLOYMENT_ENV || "" — which depends on the PM2 daemon’s shell environment. If prod EC2 reboots or pm2 kill + resurrect runs outside a login shell, DEPLOYMENT_ENV comes back empty, _telemetry_enabled flips to false, and telemetry silently dies in prod. Fix is to set Environment=DEPLOYMENT_ENV=prod in the systemd unit that launches PM2. Recorded for the next interval.

First Live Use — Debugging a User Issue

Session 4 was telemetry’s first real workout. A user (khk@diffs.studio) generated an image with the prompt “우주의 신비로운 모습” around 4/16 13:20, and the auto-fill tone image didn’t render in the detail view. Normally this would start with an SSH into the prod box and grep through files. This time the first query was in Grafana Loki: {service_name="hybrid-image-search"} |= "khk@diffs.studio" — which pulled the relevant generation logs immediately.

The chase turned up three intertwined issues:

A blob:http://... URL throwing an “insecure connection” warning — the EC2 host hadn’t moved to HTTPS yet.
A 502 Bad Gateway on a different request — likely to resolve together once the HTTPS + nginx upstream config lands.
A 401 on a third server from an expired session token that wasn’t being refreshed.

The workflow pattern was clean. Follow the trace link in Grafana → see the FastAPI span → jump to the connected log record and read the error. What used to be “SSH to prod and tail logs” became “click the trace in Grafana.” Fixes deferred to the next interval; the forensics part of this interval was the point.

Commit Log

Message	Changes
feat(gen): expand lens presets to 5 general + beauty w/ Briese lighting	5 files
feat(ui): hover-preview examples for angle/lens pickers	31 files (mostly images)
feat(telemetry): ship prod logs to Alloy/Grafana Cloud via OTLP	4 files

Insights

The best signal of this interval was that the telemetry work and the telemetry’s first real use overlapped in the same interval. “Set this up because it’ll be useful eventually” usually takes weeks to pay off, but the OTLP + Alloy stack got drafted into a real user debug on deployment day. Two effects. First, the shape of what Grafana captures and what it misses is now concrete — trace_id linking between logs and traces works; browser-side errors are not captured (OTLP covers server-side only). Second, the query “who ran what prompt when, with what error” now has a one-liner in Loki that takes an email and returns ordered log records — the next support ticket answers itself in 2 seconds. Being able to use a tool on the day it arrives is a signal that the tool landed on a real problem, not on a hypothetical one. This interval landed right.

hybrid-image-search-demo Dev Log #15 — Removing Tone Count, Unifying A/B Naming

Thu, 16 Apr 2026 00:00:00 +0900

Overview

This session focused on fully removing the tone_count system across the entire project and unifying generated image naming to a clean A/B pair. The change touched backend logic, existing DB rows, and the frontend UI, resulting in 7 commits. A deploy environment issue and an angle/lens-only regeneration bug were also fixed along the way.

Previous: hybrid-image-search-demo Dev Log #14

Summary of Changes

Why Remove Tone Count?

The original design managed the number of tone (color) variants per generation via a tone_count parameter. In practice, two variants (A and B) were always sufficient. The tone count concept added unnecessary complexity to both the UI and the prompt construction pipeline. This session removes it entirely.

flowchart LR
 A["Before: tone_count=N"] -->|"Removed"| B["Fixed A/B pair"]
 B --> C["Simpler prompts"]
 B --> D["Cleaner UI labels"]
 B --> E["DB migration"]

DB Migration (Alembic)

Existing rows in the injection_reason column carried suffixes like _tone2 or _tone3. An Alembic migration strips these suffixes from all existing rows. The parsing logic in app_utils.py was also updated to ignore any lingering suffixes.

Backend Changes

app_utils.py — Removed tone_count suffix appending logic; added suffix stripping during parsing
routes/generation.py — Removed tone_count parameter
generation/injection.py — Removed tone ratio logic
generation/prompt.py — Enriched the B variant with more detail in the prompt
routes/history.py — Added backward-compatible tone suffix handling for history queries
schemas.py — Removed tone_count field

Frontend Changes

App.tsx — Removed tone count badges, unified to A/B naming
GeneratedImageDetail.tsx — Removed tone-related labels
api.ts — Removed tone_count parameter

Angle/Lens-Only Regeneration Fix

When regenerating with only an angle or lens change (no attribute injection), the prompt was not constructed correctly. This was fixed by explicitly handling the angle/lens-only case in the generation pipeline.

Deploy Script Fix

The uv binary installs to ~/.local/bin on EC2, but the deploy script’s PATH did not include this directory, causing deployment failures. Fixed by adding it to PATH in the script.

Commit Log

#	Scope	Description
1	db	Alembic migration to strip tone_count suffix from existing injection_reason rows
2	gen	Stop appending tone_count to the reason string
3	history	Strip tone_count suffix before parsing category from reason
4	ui	Remove tone count badge from cards, use A/B only
5	ui	Replace remaining tone labels with A/B naming
6	deploy	Add ~/.local/bin to PATH for uv on EC2
7	gen	Remove tone ratio entirely, fix angle/lens-only regen, enrich B variant detail

Insights

Incremental removal is safer — Rather than deleting tone_count in one massive commit, the work was split into DB migration, backend logic, then frontend. Each step could be verified for backward compatibility with existing data.
A/B beats N variants — From a user perspective, “A / B” is far more intuitive than “Tone 3 images.” Reducing choice complexity improves UX.
PATH differences between dev and prod — A classic failure mode: works locally but breaks on EC2. Explicitly setting PATH in deploy scripts is a habit worth building.

hybrid-image-search-demo Dev Log #14 — Clock Skew, S3-First Ref Cache, Attribute-Aware Injection

Wed, 15 Apr 2026 00:00:00 +0900

Overview

Short but sharp week. Five commits, all production-hardening: a Google OAuth clock-skew fix blocking logins, ecosystem.config.js made host-portable for PM2 across dev/prod, the reference-image key cache moved off local filesystem onto S3 (so dev and prod see the same state), attribute-aware model auto-injection wired up, and personas relabeled with a 3-shot prompt that adds age estimates.

Previous: hybrid-image-search-demo Dev Log #13

graph TD
 A["Login: Invalid token 'used too early'"] --> B["clock_skew_in_seconds=10"]
 C["ref cache from local fs"] --> D[ref cache from S3]
 E[model auto-injection: any image] --> F[attribute-aware injection by tag]
 G[personas: old labels] --> H[3-shot relabel + age estimates]

Google OAuth clock skew

Context

Login blocked with Invalid Google token: Token used too early, 1776217862 < 1776217863. Check that your computer's clock is set correctly. The server’s clock was ~1 second ahead of Google’s — the JWT iat was in the future from the server’s perspective.

Fix

Added clock_skew_in_seconds=10 to id_token.verify_oauth2_token(...) in backend/src/auth.py:

id_token.verify_oauth2_token(
 token, google_requests.Request(), GOOGLE_CLIENT_ID,
 clock_skew_in_seconds=10,
)

Resolved immediately. A server should never trust its own clock to the second against a third party’s iat — 10 seconds of tolerance is standard practice for JWT validation and doesn’t open any meaningful attack surface.

S3-first reference key cache

Context

The model/reference-image cache was built from the local filesystem. This broke in production because prod’s S3-mounted paths didn’t always reflect the latest uploads, and because dev and prod had divergent local filesystem state. When a user regenerated in “tone only” mode, the UI showed the wrong reference image because the path resolved against local state, not S3 reality.

Fix

ce33906 fix(storage): build ref key cache from S3, not local filesystem — cache construction now enumerates S3 objects directly. All image retrieval paths resolved against S3 keys. Also backfilled existing generation history so old records point to the correct S3 URL.

Attribute-aware model auto-injection

Context

Previous injection logic would pull any image matching a loose condition, so the comparison mode (“tone + angle” vs “tone only”) sometimes injected a model image that didn’t match the tagged attribute. Users saw the wrong reference in the output grid.

Fix

d492ee1 feat(gen): attribute-aware model auto-injection — injection now keys on the tagged attributes (angle, tone) of the requested model folder. Subfolders under s3://diffs-studio-hybrid-search/.../01. Model are treated as attribute groups, one reference per group.

Pre-requisite: each model reference was relabeled so attributes are trustworthy. Grouping by folder means labels are a filesystem-visible schema, not a DB column, which matters because the ops team can audit and edit labels with just S3 browsing.

Persona relabeling with 3-shot prompt + age

Context

Persona labels were set earlier with a zero-shot prompt and did not include age estimates. User-facing filters needed age granularity.

Fix

2743eaf chore(labels): re-label personas with 3-shot prompt and age estimates — re-ran the labeler with three in-context examples per request and an age-range field. Labels pushed to the repo so every server picks them up, avoiding per-instance label drift.

PM2 / TSC fixes

95f8bbc fix(deploy): make ecosystem.config.js host-portable — removed hardcoded absolute paths so the same config works on dev and prod. PM2 now boots the same from any $HOME.
6ebab0d fix(ui): drop unused generatingCount state to unblock tsc build — dead state variable tripped the TypeScript build after a recent cleanup. Deleted and the build passed.

Commit log

Message	Area
fix(deploy): make ecosystem.config.js host-portable	PM2
fix(storage): build ref key cache from S3, not local filesystem	Storage
feat(gen): attribute-aware model auto-injection	Generation logic
fix(ui): drop unused generatingCount state to unblock tsc build	Frontend
chore(labels): re-label personas with 3-shot prompt and age estimates	Labeling

Insights

Two patterns worth locking in. First, “build cache from the source of truth” beats “sync cache with source of truth” every time. The ref-key cache was fragile as long as it started from local state and hoped to reconcile with S3 later; building directly from S3 removes a whole category of drift bugs. Second, the clock-skew fix is a reminder that production OAuth failures are almost always distributed-systems issues (clock sync, DNS propagation, key rotation) rather than crypto issues — a 1-line fix after 10 minutes of log reading, which is exactly how it should feel in a mature stack.