¯\_(ツ)

Architecture

Multi-stage prompt engineering pipeline. Each question passes through three sequential LLM calls:

Seed stage — topic + target difficulty → raw question stem + 1 correct answer. Uses multi-shot templates seeded with Bloom’s taxonomy keywords (remember → understand → apply → analyze → evaluate → create).
Distractor stage — stem + correct answer → 3 plausible distractors. Validates against lexical overlap, semantic similarity, and surface-form diversity via embedding cosine distances.
Validation stage — full question → discrimination index, difficulty score, option validity flags. Rejected questions cycle back through Seed with temperature adjusted ±0.2.

Difficulty calibration maps directly to Bloom’s taxonomy levels via a 6-tier rubric. Each tier constrains verb choice, cognitive load, and acceptable answering strategies. Output is a structured JSON envelope.

Question Pool Structure

Every question writes to a local SQLite-backed pool with this metadata schema:

question_id | bloom_tier | subject_area | source_topic |
difficulty_score (0.0–1.0) | discrimination_index | usage_count |
avg_response_time | last_reviewed

Pool supports filtering by any combination of fields. Used downstream by quiz generation, spaced-repetition schedulers, and analytics dashboards. Duplicate detection runs on stem embeddings — cosine similarity > 0.85 triggers manual review flag.

Technical Implementation

Async Python pipeline orchestrated with asyncio + httpx for concurrent LLM API calls. Dual-provider routing:

GPT-4 (OpenAI) — primary generation engine. Temperature = 0.7 ± 0.1, max_tokens = 1024.
Gemini Pro (Google) — fallback + cross-validation. Used during distractor validation to catch adversarial answer patterns.

NLP preprocessing layer strips domain-irrelevant artifacts: tokenizes via spaCy, normalizes lemmas, filters stop words, and extracts domain-specific keyphrase candidates via YAKE for topic anchoring. This step runs before Seed stage to prime the prompt with relevant context tokens.

Evaluation Methodology

Two-layer quality assessment:

Automated metrics — computed over generated question pools (n > 500):

Discrimination index: point-biserial correlation between score and question outcome. Target > 0.30.
Difficulty index: proportion of correct responses. Target 0.30–0.80.
Option validity: distractor response rate distribution. No option should attract < 5% of responses.
Internal consistency: Cronbach’s alpha across pool subsets. Target > 0.70.

Human evaluation protocol — 5 domain experts per batch, inter-rater reliability via Fleiss’ kappa. Rated on relevance (1–5), clarity (1–5), difficulty alignment (binary), and distractor plausibility (binary). Minimum acceptable kappa = 0.60.

Results & Impact

Key performance figures from evaluation runs:

98% prompt-adherence rate across 50+ subject areas
0.72 mean difficulty index — aligns with midpoint calibration target
0.34 mean discrimination index — exceeds 0.30 threshold
87% expert-rated relevance (≥4/5)
4.2/5 mean clarity rating
TEKNOFEST 2024 — Best Presentation Award in Education Technologies category, Istanbul

Pipeline reduces manual content creation time by ~70% while matching or exceeding human-authored question quality on all measured dimensions. Ongoing work: multi-language support and real-time adaptive difficulty tuning.

Code

Repository available on GitHub. Core pipeline is ~1,200 LOC of Python, no heavy frameworks — just asyncio, httpx, spaCy, and scipy.