Logan's Analytics Bootcamp
Dashboard › Phase 5: AI for Analytics
M5.4

Validating AI Output and AI-Assisted Workflows

⏱ 4 hours📅 Week 29❓ 3 quiz questions
Phase 5 goal

Use AI to speed up every layer you've learned (formulas, M, DAX, flows, Python, narrative) while keeping data security airtight and checking every output.

The golden rule (government and defense): Only use AI tools your organization has approved, for the data types they are approved for. Never put CUI, PII, proprietary, export-controlled, source selection, pre-decisional, or classified information into public or consumer AI tools. When unsure, use schema-only prompts with fictional sample data, and ask your security or CUI office.

🎯 Learning objectives

Tick each one when you can actually do it.

💡 Key concepts

  • The V.E.R.I.F.Y. checklist:
    • Validate against a known answer (golden dataset or hand calculation)
    • Edge cases: blanks, zeros, negatives, missing periods, duplicate keys
    • Reference check: every function exists in the official docs and version
    • Intent: does it match the business rule (fiscal year, grain, latest-period BAC)?
    • Footprint: performance and maintainability (is there a simpler approach?)
    • You own it: you can explain every line to your supervisor
  • Golden dataset: A small fictional dataset (e.g., 3 programs × 6 months) with hand-calculated expected outputs. Use it to test any AI-written logic.
  • AI for documentation: Generate first drafts of README files, measure descriptions, data dictionaries, and flow documentation, then edit.
  • AI for code comprehension: "Explain this script's data flow and list its inputs/outputs" for your coworker's (sanitized or internal-approved) code.
  • AI-drafted narratives: Status-report narrative drafts from computed metrics (not raw data), using approved tools only, always reviewed by a human. Never let AI invent causes ("driven by vendor delays") that aren't in your source data.

🛠 Hands-on exercise: Break the AI's Code

⬇ Practice data for this module
D1 programsProgram dimension (12 rows)CSVXLSX
D2 milestonesMilestones (~150 rows)CSVXLSX
D7 risksRisk register (80 rows)CSVXLSX

Ask an approved AI tool (fictional schema) for: (a) a DAX EAC measure, (b) a pandas cumulative EVM function. Build a golden dataset of 3 programs × 6 months with hand-calculated cumulative CPI and EAC. Run both outputs against it. Find at least one flaw or edge case (e.g., summing BAC across periods, divide-by-zero, wrong sort before cumsum) and fix it.

📦 Mini-project: AI-Assisted Narrative Generator (Guardrailed)

Extend your Phase 4 pipeline so it produces a structured metrics summary per program (JSON: CPI, SPI, trend, top late milestones, top risks). Then either (a) generate narrative text with rule-based templates (no AI), or (b) if you have an approved AI tool with an approved integration, send only the computed metrics and a strict prompt ("use only provided facts, no speculation on causes"). Compare the outputs, and document the review step that a human must sign off.

🧠 Self-check quiz

Answer in your own words, then check. Grade yourself honestly; your score is saved.

Q5.4.1
What's a golden dataset and why use it?
Write at least a few words to unlock.
Q5.4.2
Name three edge cases to test in EVM logic.
Write at least a few words to unlock.
Q5.4.3
Why send computed metrics rather than raw data to an AI narrative step?
Write at least a few words to unlock.
← M5.3 Copilot in Excel and Power BICAP Capstone: Portfolio Status Pipeline →