Two small career products shipped end to end, and what they taught me about where a model belongs

Problem

In 2025 I shipped two small products for people preparing for their next role.

Evolvee answers "how exposed is my job to automation, and where could I move?" You enter a job title, experience level, industry, and skills, or upload a résumé, and get an automation-risk rating, the tasks most and least exposed, three adjacent roles, and three skills to build.

AI Mock Interview (live) runs a timed practice interview for a domain track or a target company, scores your answers, and puts you on a leaderboard.

Each was small enough to build alone in one to two weeks, which made them a clean place to learn what an LLM feature costs beyond the API call.

Architecture

Evolvee extracts text from the uploaded file on the server, then makes a small extraction call to pull out job title, skills, experience level, and industry. The user reviews those fields in the form. The assessment call takes the profile plus a résumé excerpt and returns one JSON document: risk rating and explanation, risk indicators (automated tasks, human tasks, adoption evidence, skill volatility), three role recommendations, and three skill suggestions with learning resources. Both calls go through OpenRouter, so the model is an environment setting; the code defaults are Claude 3 Haiku for extraction and GPT-4 Turbo for assessment.

The mock interview draws seven questions from a curated bank: 8 domain tracks (General, Full Stack, DSA, Startups, MAANG, AI/ML, Data, Design) and 12 company tracks, each at Easy, Medium, or Hard. If a company has no bank, it falls back to MAANG questions. MCQs score by correct selections. Free-text answers score as a weighted mix: keyword coverage of each expected key point (40%), string similarity against the key points (30%), answer structure (15%), and completeness (15%), scaled by a length factor. Answers under 50 characters get 10% of the points.

The mock interview's stack is small on purpose: Next.js 15, Supabase for users, results, and the leaderboard, Zustand for interview state, string-similarity for the similarity score, and Recharts for the stats views. There is no model in it.

Key decisions

1. Extraction and judgment get different settings. The résumé extraction call runs at temperature 0.2 with a 400-token cap and the provider's JSON mode. Then code has the last word: experience level and industry are snapped to fixed option lists with keyword maps and a default. The assessment call runs warmer with a 1,500-token cap, because it is meant to reason and write. The lesson I carried forward: let the model propose, let code decide what is valid.

2. Bound the input to bound the cost. The assessment prompt includes only the first 1,000 characters of the résumé, on top of the structured profile. That caps prompt size per request. The trade-off is that anything deep in a long résumé never reaches the assessment.

3. Not every "AI" feature needs a model call. The mock interview scores in the browser with deterministic heuristics. Feedback is instant, free per request, and the same answer always gets the same score. The trade-off is real: keyword coverage rewards vocabulary, not understanding, and a well-reasoned answer in different words can score low.

4. Fresh over consistent, on purpose, and I would reverse it. Evolvee disables caching, adds a unique request ID to every prompt, and nudges temperature randomly between 0.7 and 0.8 so every run is a fresh assessment. That also means the same profile can get a different risk percentage on a second click, which undermines a number people are meant to trust.

5. Let people start without an account. Evolvee saves anonymous assessments separately from signed-in ones. The mock interview lets guests pick a username and finish an interview first, then converts that same user to a full account on sign-up, so their results stay attached. A results dedupe check (same user, track, difficulty, and score within two minutes) stops double submissions from flooding the leaderboard.

What broke

• Prompt-only JSON is fragile. The assessment asks for JSON in the prompt and calls JSON.parse on the raw reply with no schema. Any stray text or a reply cut off at the token cap fails the whole request. NearbyLens, which I built later, validates model output against Zod schemas line by line instead.

• The privacy copy did not match the data path. Evolvee's landing page says "We don't store any of your PI data on our servers." The code does: /api/assess saves the career info it receives to Supabase, in assessments for signed-in users and anonymous_assessments for everyone else. From the upload form, that career info includes the full résumé text, even though the model only ever sees the first 1,000 characters of it. The lesson I took: privacy copy is a claim about the code, and it needs the same review as the code.

• Browser state read on the server. Anonymous assessments were meant to be grouped by a session ID kept in localStorage. But the ID is created inside the save, which runs in the /api/assess server route, where localStorage does not exist. Every anonymous save got a fresh ID that no browser held, so anonymous history never worked. The results page ended up loading an assessment by its ID alone. The fix is to send the browser's session ID with the request instead of reading it where it cannot exist.

Results

• Both products are live. The mock interview was built in about a week in March 2025; Evolvee in about two weeks in May 2025, ending with PostHog analytics.
• Evolvee grew beyond the assessment: a directory of AI tools by role, a waitlist, and a public roadmap.

What I'd change

• Validate every model response against a schema and retry or degrade gracefully instead of failing the request.
• Make the risk score stable. Low temperature and caching per profile for the score, with variety only in the prose.
• Match the privacy copy to the data path, or change the data path: send the excerpt to the model, store only what the user chose to save.
• Use a model where heuristics fall short. Grading free-text interview answers is the one place in these two products where a model with a rubric and structured output would clearly beat keyword matching.