Questionnaires¶

The questionnaire system allows event organizers to gate participation behind customizable questionnaires with support for AI-powered evaluation, prompt injection detection, and flexible scoring rules.

Data Model¶

erDiagram
    Questionnaire ||--o{ QuestionnaireSection : "has sections"
    Questionnaire ||--o{ MultipleChoiceQuestion : "has MC questions"
    Questionnaire ||--o{ FreeTextQuestion : "has FT questions"
    Questionnaire ||--o{ FileUploadQuestion : "has FU questions"
    Questionnaire ||--o{ QuestionnaireSubmission : "receives submissions"
    QuestionnaireSection ||--o{ MultipleChoiceQuestion : "contains"
    QuestionnaireSection ||--o{ FreeTextQuestion : "contains"
    QuestionnaireSection ||--o{ FileUploadQuestion : "contains"
    QuestionnaireSection }o--o| MultipleChoiceOption : "depends_on_option"
    MultipleChoiceQuestion ||--o{ MultipleChoiceOption : "has options"
    MultipleChoiceQuestion ||--o{ MultipleChoiceAnswer : "receives answers"
    MultipleChoiceQuestion }o--o| MultipleChoiceOption : "depends_on_option"
    FreeTextQuestion ||--o{ FreeTextAnswer : "receives answers"
    FreeTextQuestion }o--o| MultipleChoiceOption : "depends_on_option"
    FileUploadQuestion ||--o{ FileUploadAnswer : "receives answers"
    FileUploadQuestion }o--o| MultipleChoiceOption : "depends_on_option"
    QuestionnaireSubmission ||--o{ MultipleChoiceAnswer : "contains"
    QuestionnaireSubmission ||--o{ FreeTextAnswer : "contains"
    QuestionnaireSubmission ||--o{ FileUploadAnswer : "contains"
    QuestionnaireSubmission ||--o| QuestionnaireEvaluation : "evaluated by"
    FileUploadAnswer }o--o{ QuestionnaireFile : "references files"

    Questionnaire {
        string name
        string description
        decimal min_score
        string llm_guidelines
        string llm_backend
        string status
        string evaluation_mode
        boolean shuffle_questions
        boolean shuffle_sections
        int max_attempts
        duration can_retake_after
    }

    QuestionnaireSection {
        string name
        string description
        int order
        uuid depends_on_option FK
    }

    MultipleChoiceQuestion {
        string question
        string hint
        string reviewer_notes
        decimal positive_weight
        decimal negative_weight
        boolean is_fatal
        boolean is_mandatory
        int order
        boolean allow_multiple_answers
        boolean shuffle_options
        uuid depends_on_option FK
    }

    FreeTextQuestion {
        string question
        string hint
        string reviewer_notes
        decimal positive_weight
        decimal negative_weight
        boolean is_fatal
        boolean is_mandatory
        int order
        string llm_guidelines
        uuid depends_on_option FK
    }

    FileUploadQuestion {
        string question
        string hint
        string reviewer_notes
        decimal positive_weight
        decimal negative_weight
        boolean is_fatal
        boolean is_mandatory
        int order
        string[] allowed_mime_types
        int max_file_size
        int max_files
        uuid depends_on_option FK
    }

    MultipleChoiceOption {
        string option
        boolean is_correct
        int order
    }

    MultipleChoiceAnswer {
        uuid submission FK
        uuid question FK
        uuid option FK
    }

    FreeTextAnswer {
        uuid submission FK
        uuid question FK
        string answer
    }

    FileUploadAnswer {
        uuid submission FK
        uuid question FK
        uuid[] files
    }

    QuestionnaireFile {
        uuid uploader FK
        binary file
        string original_filename
        string file_hash
        string mime_type
        int file_size
    }

    QuestionnaireSubmission {
        uuid questionnaire FK
        uuid user FK
        string status
        datetime submitted_at
        json metadata
    }

    QuestionnaireEvaluation {
        uuid submission FK
        decimal score
        string status
        string proposed_status
        string comments
        boolean automatically_evaluated
        uuid evaluator FK
        json raw_evaluation_data
    }

Hierarchy¶

Questionnaire → QuestionnaireSection (optional) → Questions → Options/Answers

Question Types¶

Common Fields (BaseQuestion)¶

All question types inherit from BaseQuestion and share these fields:

FileUploadQuestion Extra Fields¶

QuestionnaireFile Model¶

Uploaded files are stored in a user-scoped library (QuestionnaireFile) and can be reused across multiple questions and questionnaires via M2M relationship with FileUploadAnswer. Files are stored with UUID-based paths to prevent enumeration attacks, and duplicate uploads per user are prevented via a SHA-256 hash constraint.

Questionnaire Status Lifecycle¶

Questionnaires follow a three-stage status lifecycle:

stateDiagram-v2
    [*] --> DRAFT
    DRAFT --> READY : All questions configured
    READY --> PUBLISHED : Ready for users

Submission Status¶

Submissions also have their own status flow:

Conditional Sections and Questions¶

Both QuestionnaireSection and all question types support conditional display via the depends_on_option foreign key to MultipleChoiceOption.

• A section with depends_on_option set is only shown if the referenced option was selected by the user.
• A question with depends_on_option set is only shown if the referenced option was selected.
• The referenced option must belong to the same questionnaire (validated in clean()).

During evaluation, the SubmissionEvaluator computes which questions are applicable based on the user's selected options. Questions that were not shown (because their condition was not met) are excluded from scoring and mandatory checks.

Retry Mechanics¶

Questionnaires support configurable retry behavior:

For admission questionnaires, max_attempts alone controls whether retakes are possible; can_retake_after only controls the cooldown duration between attempts. For example, max_attempts=3 with can_retake_after=null allows up to 3 attempts with no waiting period between failures.

Both readings live in one place — questionnaires/utils/retake_policy.py — selected by the null_cooldown_is_terminal flag. The eligibility gates and the submit endpoints share it, so a gate can never promise a submission the endpoint would reject. The cooldown blocks only while retry_on is still in the future: at exactly retry_on the retake is allowed.

Evaluation Modes¶

Questionnaires support three evaluation modes:

QuestionnaireEvaluation¶

Each submission can have at most one QuestionnaireEvaluation (OneToOne relationship). The evaluation stores:

The raw_evaluation_data field stores an EvaluationAuditData Pydantic model containing:

• mc_points_scored / max_mc_points: multiple choice scoring breakdown
• ft_points_scored / max_ft_points: free text scoring breakdown
• missing_mandatory: list of UUIDs for unanswered mandatory questions
• llm_response: the complete EvaluationResponse from the LLM batch evaluation

Scoring System¶

Weights and Thresholds¶

Each question has a positive_weight (points for correct answers) and a negative_weight (points deducted for incorrect answers). The questionnaire defines a min_score (0 to 100 scale) that the percentage score must meet or exceed.

total_points_scored = mc_points_scored + ft_points_scored
total_max_points   = max_mc_points + max_ft_points

score_percent = (total_points_scored / total_max_points) * 100
pass          = score_percent >= min_score

Multiple Choice Scoring¶

For each MultipleChoiceAnswer:

• If option.is_correct is True: mc_points_scored += question.positive_weight
• If option.is_correct is False: mc_points_scored -= question.negative_weight

Free Text Scoring¶

Free text answers are evaluated in a single batch call to the configured LLM backend. For each answer:

• If is_passing is True: ft_points_scored += question.positive_weight
• If is_passing is False: ft_points_scored -= question.negative_weight

LLM guidelines are provided at two levels:

1. Questionnaire-level: Questionnaire.llm_guidelines: global guidelines sent to every LLM call
2. Question-level: FreeTextQuestion.llm_guidelines: per-question guidelines that supplement the questionnaire-level guidelines

Fatal Questions¶

Mandatory Questions¶

Questions marked as is_mandatory=True must be answered. If any applicable mandatory question is left unanswered, the score is set to -100.0 and the submission fails. Only questions that are applicable (based on conditional dependencies) are checked for mandatory compliance.

LLM Evaluation¶

Free text questions are evaluated by a pluggable LLM backend. The backend is configured per questionnaire via the llm_backend field, which stores the fully-qualified Python path to the evaluator class.

The system uses Instructor with an OpenAI-compatible client for vendor-agnostic LLM access. Any OpenAI-compatible provider (OpenAI, Ollama, Azure, vLLM, etc.) can be used — only Django settings need to change.

Backend Pipeline¶

flowchart TD
    Answer["User's Free Text Answers<br/>(batch)"] --> Backend["LLM Backend"]

    Backend --> Mock["MockEvaluator<br/>(testing)"]
    Backend --> Sanitizing["SanitizingLLMEvaluator<br/>(input sanitization + defense)"]
    Backend --> Sentinel["SentinelLLMEvaluator<br/>(ML detection + sanitization)"]

    Mock -->|"keyword match"| Result["EvaluationResponse"]
    Sanitizing -->|"strip tags + defensive prompting"| Result
    Sentinel -->|"ML check first"| Detection{"Injection<br/>detected?"}
    Detection -->|"Yes"| Reject["Automatic Fail"]
    Detection -->|"No"| SanitizeFirst["Sanitize + evaluate"]
    SanitizeFirst --> Result

    style Reject fill:#c62828,color:#fff
    style Sentinel fill:#1565c0,color:#fff

Available Backends¶

All backends implement the FreeTextEvaluator protocol and accept a batch of AnswerToEvaluate objects.

LLM Provider Configuration¶

The LLM provider is configured via Django settings (see revel/settings/llm.py), all sourced from environment variables:

# Model identifier including provider prefix: "provider/model-name"
LLM_DEFAULT_MODEL="ollama/llama3.1:8b"  # Ollama (default for local dev)
# LLM_DEFAULT_MODEL="openai/gpt-4o-mini"  # OpenAI (hosted provider example)

# API key — not needed for local providers like Ollama
# LLM_API_KEY=sk-...

# Override the provider's default base URL (typically not needed)
# LLM_BASE_URL=

# Max retries on validation failure (Instructor handles this)
# LLM_MAX_RETRIES=3

# Instructor mode for structured output extraction (see below)
# LLM_INSTRUCTOR_MODE=JSON

See Instructor integrations for all supported providers.

Instructor Mode (JSON vs TOOLS)¶

Instructor extracts structured output from LLMs in two ways:

• JSON mode (LLM_INSTRUCTOR_MODE=JSON): LLM returns JSON in the message content, validated against the Pydantic schema. Works universally.
• TOOLS mode (LLM_INSTRUCTOR_MODE=TOOLS): Uses the provider's native function/tool-calling API. More reliable with large models that support it well (GPT-4o, Claude, etc.).

The default is JSON because Instructor's auto-detection can misfire for smaller models. For example, ollama/llama3.1:8b is listed as "tool capable" in Instructor's internal registry, but the 8B variant doesn't handle TOOLS mode reliably — it fails with Instructor does not support multiple tool calls. Forcing JSON mode avoids this.

Inheritance Hierarchy¶

FreeTextEvaluator (Protocol)
├── MockEvaluator
└── BaseLLMEvaluator (defensive prompting)
    └── SanitizingLLMEvaluator (sanitizes input, then calls super) — DEFAULT
        └── SentinelLLMEvaluator (ML check, then delegates to sanitize + evaluate)

Prompt Injection Protection¶

The SentinelLLMEvaluator provides the strongest protection. It works in three stages:

1. Detection: Each answer is classified by the local Sentinel ML model as benign or jailbreak
2. Sanitization: XML-like tags and their content are stripped from user answers
3. Evaluation: Defensive prompting instructs the LLM to ignore any injected instructions

Evaluation Pipeline (SubmissionEvaluator)¶

The SubmissionEvaluator class orchestrates the full evaluation of a submitted questionnaire:

flowchart TD
    Start["SubmissionEvaluator.evaluate()"] --> Applicable["Compute applicable questions<br/>(based on conditional deps)"]
    Applicable --> Mandatory["Check for missing<br/>mandatory answers"]
    Mandatory --> MC["Score multiple choice answers"]
    MC --> FT["Evaluate free text answers<br/>(batch LLM call)"]
    FT --> Score["Calculate score_percent"]
    Score --> Mode{"Evaluation<br/>mode?"}
    Mode -->|AUTOMATIC| Final["Set status = proposed_status"]
    Mode -->|MANUAL / HYBRID| Pending["Set status = PENDING_REVIEW"]
    Final --> Save["Create/Update QuestionnaireEvaluation"]
    Pending --> Save

The entire evaluation runs inside a database transaction (@transaction.atomic) to ensure it is an all-or-nothing operation.

Service Layer (QuestionnaireService)¶

The QuestionnaireService class provides CRUD operations for questionnaires and handles submission:

• build(): Constructs the questionnaire schema for API responses, applying shuffle logic for questions, options, and sections
• submit(): Validates answers against the questionnaire, checks mandatory questions, validates file uploads, and creates the submission with all answers
• create_questionnaire(): Creates a complete questionnaire with sections, questions, and options from a single schema
• create_section() / create_mc_question() / create_ft_question() / create_fu_question(): Individual creation methods supporting nested conditional content
• evaluate_submission(): Creates or updates a manual evaluation for a submission

Security¶

Beyond prompt-injection protection, two access-control invariants guard the questionnaire system.

Admin reads are org-admin gated¶

The admin questionnaire schema exposes the answer key — is_correct on each option, per-question positive_weight / negative_weight, the questionnaire's min_score, reviewer_notes, and llm_guidelines. None of this may reach a candidate who is about to take the questionnaire.

Accordingly, the admin detail (GET /{org_questionnaire_id}) and list (GET /) endpoints are restricted to organization administrators:

• The detail endpoint requires QuestionnairePermission("evaluate_questionnaire") (owner or staff with that flag).
• The list endpoint is scoped to OrganizationQuestionnaire.objects.for_admin(self.user()), so only organizations the user owns or staffs are returned.

Submission invariants enforced in the service layer¶

Answers are persisted with bulk_create, which skips Django's Model.clean(). So model-level checks (MultipleChoiceAnswer.clean()) are not a sufficient guard on the write path. The QuestionnaireService re-enforces the invariants explicitly at submission time:

• Single-answer questions (allow_multiple_answers=False) may receive at most one selected option. Options are aggregated per question across all answer entries (de-duplicated), so a client cannot split selections into several entries to slip past a per-entry length check. A violation raises DisallowedMultipleAnswersError.
• Options must belong to their question. Each (question_id, option_id) pair is checked against the options actually owned by that question (queried fresh, since the prefetched cache can be stale). A foreign option id raises CrossQuestionnaireSubmissionError.

Information-Gathering Mode (requires_evaluation)¶

The OrganizationQuestionnaire wrapper has a requires_evaluation boolean field (default True). When set to False, the questionnaire gates access based on submission existence alone — no LLM evaluation, no manual review, no scoring.

This is useful for admission questionnaires that collect information without judgment (e.g., dietary preferences, accessibility needs, emergency contacts).

Integration with Eligibility¶

Questionnaires integrate with the Eligibility Pipeline through the Questionnaire Gate:

1. Event is configured to require one or more questionnaires
2. When a user attempts to RSVP/purchase a ticket, the eligibility pipeline checks questionnaire status
3. If questionnaires are incomplete, the user receives next_step: COMPLETE_QUESTIONNAIRE
4. If requires_evaluation=False, submission existence is sufficient — no evaluation checks are performed
5. If questionnaires are submitted but failed, the user is rejected with the failure reason

flowchart LR
    User["User"] -->|"RSVP"| Eligibility["Eligibility Pipeline"]
    Eligibility -->|"Questionnaire Gate"| Check{"Questionnaires<br/>complete?"}
    Check -->|"Not submitted"| Redirect["next_step:<br/>COMPLETE_QUESTIONNAIRE"]
    Check -->|"Submitted &<br/>requires_evaluation=False"| Continue
    Check -->|"Submitted & passed"| Continue["Continue pipeline"]
    Check -->|"Submitted & failed"| Reject["Rejected"]

    style Reject fill:#c62828,color:#fff
    style Continue fill:#2e7d32,color:#fff