ProphetAI
A probability engine for arbitrary real-world questions — five free-tier providers behind one fallback chain, and a JSON contract that decides which answers count.
- Type
- Artifact
- Where
- Personal
- When
- Dec 2024 → Apr 2025
- Duration
- 4 mo
- Status
- Shipped
- Source
- GitHub
The problem
Ask a single language model how likely something is and you get a confident number with nothing behind it. Ask it twice and you often get two different numbers. Neither is calibrated, and neither tells you how much to trust it.
I wanted a consistent interface to several providers, with fallback when a request failed or returned invalid data. The current version returns one provider's estimate; it does not combine votes or establish calibration.
What I built
Five providers — Together, OpenRouter, Groq, Cohere and Mistral — sit behind one interface. A question walks the chain in order and the first provider that returns a usable answer wins; the rest are never called. That keeps each successful request to one provider rather than paying for an ensemble.
"Usable" is the load-bearing word. Every provider is prompted for the same four-key JSON object — probability, short_explanation, paragraph_explanation, math_explanation — and a response that is malformed, empty, or missing a key is discarded and the chain advances. A rate limit and a model that ignored the schema fail identically, which is why one vendor having a bad afternoon is invisible from the outside.
Answers are cached against the question that produced them — SQLite on device, an in-memory map on web — so a repeat question costs nothing and returns immediately, flagged in the UI as cached rather than fresh.
Screens
What I considered
Rejected. No API cost and instant offline answers, but even a 2B-parameter model is around 500 MB, and iOS restrictions make cross-platform local inference impractical.
Rejected. Full control and no rate limits, but the cheapest GPU is $20–50 a month out of my own pocket, and the bill scales with demand rather than with value.
Chosen. Five vendors' free tiers, tried in sequence until one honours the JSON contract. It is O(n) in the worst case and it will not survive real scale — and it is the right answer for a project that has to cost nothing and stay up when any single vendor throttles.
Results
- Runs on five vendors' free tiers with no paid plan anywhere in the stack.
- Repeat questions return from the local cache instead of the network, and are labelled as cached in the UI.
- Flutter web client shipped through GitHub Actions to Firebase Hosting, with preview deploys on every pull request.
Architecture
In hindsight
Fallback improves availability, not the accuracy of a probability estimate. The current implementation returns the first valid response and has no ensemble weighting or outcome-based calibration. A next version would store predictions, grade them against resolved outcomes, and compare providers before deciding whether combining their estimates helps.