Sub-slice C — Queue endpoint, FE wiring, VCC poller

01 Context

Sub-slice B made deals processable end-to-end via an admin trigger. Sub-slice C closes the loop:

• Deals start arriving on their own (from HubSpot, the upstream source-of-truth).
• Operators see them live in the queue page Lovable already designed.

The Lovable mock (web/src/lib/queue-mock.ts + queue-store.ts) is the design target — same visual surface, real data. The FE today renders fields the backend doesn't model yet (step.params, step.output, step.logs[], deal input image, etc.); each one was triaged and either mapped, dropped for now, or flagged for a future slice (see §03).

The VCC poller is deliberately minimal: search HubSpot, create Deal rows, enqueue process_deal. No image-size gating, no terminal-stage skipping, no VCC appraisal-contents enrichment — those depend on tables (deal_images) and product knowledge that belong with later slices.

02 Scope

03 UI ↔ backend field audit

The Lovable mock renders many fields the new endpoint won't provide yet. Each row below: what the UI uses, whether the backend has it, and the chosen call. Anything dropped is recorded as a deferred UI capability with the slice that would unlock it.

Top-level (run / deal)

Per-step (pipeline_run)

Actions

04 Schema migration

One migration: alembic/versions/0007_add_hubspot_deal_id_and_queue_index.py.

ALTER TABLE deals ADD COLUMN hubspot_deal_id TEXT;
ALTER TABLE deals ADD COLUMN hubspot_checked_in_at TIMESTAMPTZ;
-- backfill seeds in the same migration step:
UPDATE deals SET hubspot_deal_id = 'seed-' || id::text WHERE hubspot_deal_id IS NULL;
UPDATE deals SET hubspot_checked_in_at = created_at WHERE hubspot_checked_in_at IS NULL;
ALTER TABLE deals ALTER COLUMN hubspot_deal_id SET NOT NULL;

CREATE UNIQUE INDEX ix_deals_hubspot_deal_id ON deals (hubspot_deal_id);
CREATE INDEX ix_deals_updated_at_desc ON deals (updated_at DESC);
CREATE INDEX ix_deals_hubspot_checked_in_at ON deals (hubspot_checked_in_at DESC);

TEXT for hubspot_deal_id, not bigint — HubSpot deal ids are documented as numeric strings. pipeline_runs unchanged.

05 HubSpot integration

Async HTTPX client ported from checkin-pipeline/app/hubspot.py — returns Pydantic models, raises a single HubspotError, no internal retry (periodic ticks are the retry strategy).

class HubspotDealSummary(BaseModel):
    hubspot_deal_id: str
    name: str | None = None
    country: str | None = None
    created_at: datetime | None = None
    deal_stage: str | None = None
    amount: float | None = None
    checked_in_at: datetime    # always present (the property we filter on)


class HubspotError(Exception):
    """Wraps HTTPX errors and non-2xx responses; carries response body when available."""


class HubspotClient:
    def __init__(self, access_token: str, base_url: str = "https://api.hubapi.com") -> None: ...

    async def search_checked_in_deals(
        self,
        since: datetime,
        until: datetime,
    ) -> list[HubspotDealSummary]:
        """Iterate by day to stay under HubSpot's 10K-per-query limit; follow pagination cursors."""

Settings: HUBSPOT_ACCESS_TOKEN: SecretStr | None = None, HUBSPOT_POLL_INTERVAL_MINUTES = 5, HUBSPOT_BOOTSTRAP_DAYS = 30. SecretStr so the token can't appear in pydantic-settings dumps or structlog output.

06 Periodic poller task

@app.periodic(cron="*/5 * * * *")
@app.task(queue="default", name="poll_hubspot")
async def poll_hubspot(timestamp: int) -> None:
    settings = get_settings()
    # No token-presence check here: worker startup validation guarantees it.
    sm = get_session_factory(settings.database_url)
    async with sm() as session:
        service = DealSyncService(session, settings)
        result = await service.sync_once()

    logger.info(
        "hubspot_poll_completed",
        fetched=result.fetched,
        new_deals=result.new_deals,
        skipped_existing=result.skipped_existing,
        cursor_since=result.cursor_since.isoformat(),
    )

DealSyncService.sync_once()

@dataclass
class SyncResult:
    fetched: int
    new_deals: int
    skipped_existing: int
    cursor_since: datetime

async def sync_once(self) -> SyncResult:
    since, until = await self._compute_window()
    summaries = await self._hubspot.search_checked_in_deals(since=since, until=until)

    new_deal_ids: list[UUID] = []
    for summary in summaries:
        deal_id, created = await self._upsert(summary)
        if created:
            new_deal_ids.append(deal_id)

    await self._session.commit()

    # Commit-first, then enqueue: guarantees jobs never reference uncommitted rows.
    for deal_id in new_deal_ids:
        await process_deal.defer_async(deal_id=str(deal_id))

    return SyncResult(...)

Upsert uses Postgres ON CONFLICT (hubspot_deal_id) DO UPDATE with xmin = txid_current() to distinguish insert vs. update. Only hubspot_checked_in_at and updated_at get overwritten on conflict; status / current_stage are pipeline-owned and never reset by sync.

Failure scenarios

07 Queue endpoint

@router.get("/queue/deals", response_model=DealQueuePage)
async def list_queue_deals(
    limit: int = Query(default=50, ge=1, le=200),
    before: datetime | None = Query(default=None),
    session: AsyncSession = Depends(db_session),
) -> DealQueuePage:
    return await DealQueueService(session).list_page(limit=limit, before=before)

Response schemas

class StageProgress(BaseModel):
    stage: Literal["intake", "segmentation", "categorization", "appraisal"]
    status: Literal["pending", "running", "succeeded", "failed"]
    started_at: datetime | None = None
    ended_at: datetime | None = None
    pipeline_run_id: UUID | None = None
    error: str | None = None


class DealQueueRow(BaseModel):
    id: UUID
    hubspot_deal_id: str
    status: DealStatus
    current_stage: PipelineStage
    created_at: datetime
    updated_at: datetime
    steps: list[StageProgress]   # exactly 4: intake, segmentation, categorization, appraisal


class DealQueuePage(BaseModel):
    items: list[DealQueueRow]
    next_before: datetime | None

_build_steps (latest-by-started_at wins for stages with multiple runs)

UI_STAGES = (
    PipelineStage.INTAKE,
    PipelineStage.SEGMENTATION,
    PipelineStage.CATEGORIZATION,
    PipelineStage.APPRAISAL,
)
by_stage: dict[PipelineStage, PipelineRun] = {}
for run in sorted(deal.pipeline_runs, key=lambda r: r.started_at):
    by_stage[run.stage] = run   # last write wins ⇒ latest

result = []
for stage in UI_STAGES:
    run = by_stage.get(stage)
    if run is None:
        result.append(StageProgress(stage=stage, status="pending"))
    else:
        result.append(StageProgress(
            stage=stage,
            status=run.status.value,
            started_at=run.started_at,
            ended_at=run.ended_at,
            pipeline_run_id=run.id,
            error=run.error if run.status == RunStatus.FAILED else None,
        ))
return result

Query uses selectinload on Deal.pipeline_runs — exactly two SQL statements regardless of row count. Keyset pagination via before=updated_at; limit+1 fetched to detect "is there a next page?". Drift on updates crossing the boundary is acceptable — polling is 2 s.

08 Frontend wiring

New adapter (web/src/lib/api/adapters/queue.ts)

import type { components } from "@/lib/api/generated/types.gen";
type ApiQueuePage = components["schemas"]["DealQueuePage"];

export interface QueueRow {
  dealId: string;            // hubspot_deal_id, display
  dealUuid: string;          // backend UUID, for actions
  createdAt: number;
  updatedAt: number;
  steps: QueueStep[];        // 4, canonical order
  status: "pending" | "running" | "done" | "failed";
  durationMs: number;
}

export function adaptQueuePage(page: ApiQueuePage): {
  rows: QueueRow[];
  nextBefore: string | null;
} { ... }

Edited

• admin.production.queue.tsx — useQueueStore replaced with useQuery; refetchInterval: isLive ? 2000 : false; isLive + filters become local state
• QueueTable.tsx — takes rows + filters as props; filter logic moves out of the table
• RunDetailSheet.tsx — selectedRunId from URL search-param; Timeline tab only; "Re-run from scratch" calls useEnqueueProcessDealMutation; "Retry step" removed
• PipelineTrail.tsx — typechange only
• TimelineTab.tsx — typechange; drop refs to logs[], params, output, model, retries, errorStack

09 Testing

Backend (~20 new test cases)

Frontend

Deliberately not testing: live HubSpot (separate contracts suite, later); 2 s polling under load (manual sanity); E2E via Playwright (deferred per architecture spec).

10 Verification gate

11 Risks

12 Open decisions for user review

1. §01 queue-mock.ts deleted entirely. Types come from generated. Aligns with the README's type-source policy. Implicit from "frontend completely only use the generated types"; flagging in case you want a phased delete.
2. §04 Defer the (current_stage, status) index on deals. Revisit when filter behavior is measured.
3. §05 Cursor lives in deals.hubspot_checked_in_at rather than a dedicated poll-state table. Fewer tables; more debuggable shape.
4. §05 Fail at worker boot when HUBSPOT_ACCESS_TOKEN is missing. Worker entrypoint runs _validate_worker_settings() and raises RuntimeError. The API service is unaffected (it doesn't import the poller). Tests inject a placeholder via the settings fixture. Changed from earlier silent-no-op default per user feedback.
5. §06 Commit-first then enqueue. Guarantees no orphan jobs; trades a vanishingly rare "row without job" failure (admin endpoint recovers).
6. §07 Trim REVIEW / DONE from DealQueuePage.steps[]. Backend stays consistent internally; FE sees exactly its 4 visible stages.
7. §07 Keyset pagination via before=updated_at rather than opaque cursor or offset.
8. §07 Filters (status, stage, search) stay FE-only for this slice. Revisit when 50-row pages stop being enough.
9. §08 URL-based selectedRunId via TanStack Router search params (rather than local state). Makes the detail-sheet view shareable.
10. §08 PipelineTrail kept as-is (typechange only). Simplification deferred until it causes drag.
11. §08 Single adaptQueuePage adapter as the codified API↔UI tech debt. README explicitly anticipates this place.