Skip to main content

Reliability

Ujex keeps durable intent in Firestore so a function retry or provider outage does not have to invent what happened.

State and recovery​

StateSystem of recordRecovery responsibility
Agents, mandates, approvals, audit, queuesFirestoreFirebase backup/PITR plus tested export.
Artifact and memory objectsCloud StorageBucket versioning/replication configured by operator.
Trust signing and bridge secretsSecret Manager / Cloud KMSRotation runbook and offline recovery copy where required.
Mailboxes and SMTP deliveryRuzayo plus Ujex projectionsReconcile provider state; never claim readiness before confirmation.

Durable retry paths​

  • Agent creation reserves a stable idempotency attempt and mailbox claim before provider work.
  • Mailbox provisioning writes a retry job first, uses leases/backoff, and has scheduled retry and reconciliation functions.
  • Outbound messages remain in Firestore until the dispatcher can claim an eligible row.
  • Approval expiry is reaped every five minutes; expired approval-bound sends are not dispatched.
  • Scheduler jobs and DLQ records remain inspectable and replayable.
  • Identity cleanup jobs preserve ownership checks and retry ambiguous provider cleanup.

Not every callable is idempotent. Retry only operations documented as idempotent, or reuse the same idempotency key.

Scheduled controls​

Important schedules include hourly chain verification and anchoring, one-minute outbound and scheduler dispatch, five-minute approval expiry, periodic mailbox reconciliation, budget/anomaly checks, certificate renewal, artifact cleanup, memory-diff cleanup, and runtime reaping. The deployed inventory—not this prose—is authoritative for exact schedule state.

Failure responses​

Callable unavailable or rate limited​

Do not reinterpret uncertainty as permission. Retry unavailable, aborted, and resource-exhausted with bounded exponential backoff and the same idempotency key. Never retry a policy denial as an allow.

Mail provider unavailable​

Identity creation still succeeds. The mailbox stays pending; send/read paths refuse to imply a working address. Reconciliation retries transient failures and surfaces terminal reasons.

Chain verification failure​

Stop relying on records after the first bad sequence, preserve the project and logs for forensics, and investigate before resuming evidence claims. Exported packs identify the affected range rather than hiding it.

Secret or KMS denial​

Treat this as cloud-owner configuration, not an application validation error. Check Secret Manager access, KMS decrypt permission, secret version state, and billing/quota before changing product code.

Public webhook becomes unreachable after deploy​

Re-check the Gen 2 Cloud Run invoker binding and run the public endpoint smoke. Deploys can recreate the underlying service; the binding is an explicit release check.

Agent offboarding​

Use the owner Settings action and hold to confirm. The backend suspends the agent and increments its session version before cleanup, retires the provisioned mailbox/claim, then removes working agent data. The owner account and sealed global audit evidence remain. If cleanup reports an error, keep the agent suspended and retry the same offboarding operation; do not recreate it with the same mailbox until cleanup converges.

Known limits​

  • Production restores and provider disaster recovery require operator drills.
  • The public chain roots are not independently retained outside Ujex yet.
  • Pod soft-delete has a grace period but no user-facing restore callable.
  • A formal external penetration test has not been completed.