Python: What is Artificial Intelligence?

Example 1 - Same Model, Two Systems, Two Authorities

Scenario

A company plans two assistants on the same large language model. The first answers employees' questions about the travel policy. The second handles customers' refund requests for a subscription product. In the planning meeting both are called "our AI assistant". Describe each on the six axes, find where they differ, and say what that difference demands.

Six-Axis Descriptions

The first four rows are nearly identical, which is why "our AI assistant" sounded like one thing. The authority row is where they part.

What the Authority Row Demands

Travel-policy assistant (inform). A wrong answer can still cost money, as Moffatt v. Air Canada showed for a customer-facing chatbot; with employees the exposure is smaller but real. The design response is to ground each answer in the policy text it quotes, link to that section, and tell the employee that the policy page, not the answer, is authoritative. The measurement is answer accuracy on a set of questions with known answers.

Refund assistant (execute). Now a wrong output moves money without anyone looking. On the Sheridan–Verplank scale the proposal sits at level 7 or above: the system acts, then someone may find out. Three questions decide whether that authority is defensible:

1. Is the action reversible? A refund can be clawed back only at the cost of an angry customer, so in practice it is not.
2. How fast would a mistake be noticed? Only when finance reconciles, days later.
3. What does a mistake cost? Up to $200 each, multiplied by however often the model is wrong.

None of the three favours execute authority on day one. The alternative is draft authority: the assistant prepares the refund with its reasoning and a link to the rule it applied, and an agent approves it with one click. Once the error rate has been measured on real requests, refunds under a lower limit that match a simple rule exactly can move to execute, with the rest staying at draft.

Checking the Descriptions

Run each description through the chapter's program by adding it as a profile. The travel assistant passes if its controls include answers quote canonical policy. The refund assistant, as proposed, fails the irreversible-action check until its controls include human approval or a working rollback.

Lesson

"Which model?" was the least informative question in that meeting. Two systems on the same model need different evidence, different controls and a different launch plan, and the six axes make that visible before anything is built. Write the authority row first; it tells you what the rest of the design has to prove.

Example 2 - Pricing the Review Decision by Hand

Scenario

A customer-service assistant answers 100,000 conversations a month, some of them about refund and cancellation policy. Your manager asks a simple question with a numeric answer: should a person review the assistant's answers, all of them, some of them, or none? Work it out with the chapter's formulas before touching the calculator.

Step 1: Write Down the Inputs

Only one number here comes from a real case; the rest are assumptions you would replace with measurements.

N   conversations per month                         100,000
e   share of answers wrong on a policy point        0.5% and 1% (two scenarios)
a   share of wrong answers a customer acts on       10%
L   cost of one relied-upon wrong answer            C$812.02  (Moffatt v. Air Canada award)
r_c cost of reviewing one conversation              1.5 min at C$40/h = C$1.00
k   share of wrong answers a reviewer catches       90%
s   share of conversations routed to review         10%  (refunds and policy exceptions)
q   share of costly errors inside routed ones       80%

Step 2: Price the Three Policies at e = 1%

No review:    N × e × a × L = 100,000 × 0.01 × 0.10 × 812.02          = C$81,202
Full review:  N × r_c + 81,202 × (1 − 0.90) = 100,000 + 8,120         = C$108,120
Routed:       N × s × r_c + 81,202 × (1 − 0.90 × 0.80)
            = 10,000 + 81,202 × 0.28 = 10,000 + 22,737               = C$32,737

Routed review is cheapest by a factor of about 2.5 over no review and 3.3 over full review.

Step 3: Repeat at e = 0.5%

No review:    C$40,601
Full review:  100,000 + 4,060  = C$104,060
Routed:       10,000 + 11,368  = C$21,368

Halving the error rate halves the error costs but not the reviewer costs, so no review closes some of the gap, but routed review still wins.

Step 4: Find the Flip Points

Set two policies' costs equal and solve for e. The volume N cancels every time.

Full beats none when      N × r_c < N × e × a × L × k
                          e > r_c / (a × L × k)             = 1.00 / 73.08  = 1.37%
Routed beats none when    e > s × r_c / (a × L × k × q)     = 0.10 / 58.47  = 0.17%
Full beats routed when    e > (1 − s) × r_c / (a × L × k × (1 − q))
                                                            = 0.90 / 14.62  = 6.16%

So: no review below 0.17%, routed review from 0.17% to 6.16%, full review above 6.16%.

Step 5: Ask What It Would Take to Choose "No Review"

The flip point is 0.17%, so choosing no review means showing that the error rate is below it. Chapter 1's zero-failure bound says how many consecutive correct answers that takes at 95% confidence:

n ≥ ln(0.05) / ln(1 − 0.00171) ≈ 1,750 clean answers

That is 1,750 policy answers checked against the policy with none wrong. Until the team has that evidence, routed review is the defensible choice.

Lesson

A review decision is three straight lines that cross at two points, and the crossings depend on rates, not volume. The expensive part is not the arithmetic but measuring the error rate precisely enough to know which side of 0.17% you are on.

Runnable Example - Six-Axis Profiles and the Review Decision

The program describes four real systems and one flawed proposal on the chapter's six axes, records each system's authority as the effects its outputs may have, checks every description for the category errors the chapter warns about, and prices three review policies for a customer-service assistant. It uses the Python standard library only, runs offline in a fraction of a second, and prints exactly what the Java version prints.

Expected output:

=== Parts 1 and 2: Five systems on six axes, with consistency checks ===
Gmail spam filtering
  field        AI: natural language processing
  method       machine learning
  model family neural text classifier (RETVec vectorizer, 2023)
  capability   classify
  application  email client
  authority    execute, reversible; Sheridan-Verplank level 7-8
  effects      act_reversible
  checks       consistent
GitHub Copilot inline suggestions
  field        AI: code generation
  method       machine learning
  model family large language model
  capability   generate
  application  code editor
  authority    draft; Sheridan-Verplank level 5
  effects      display, prepare
  checks       consistent
Waymo Driver
  field        AI: robotics and perception
  method       machine learning, planning, rules
  model family neural networks for perception and prediction
  capability   perceive, predict, plan, act
  application  ride-hailing
  authority    execute; Sheridan-Verplank level 9-10
  effects      act
  checks       consistent
Air Canada website chatbot (2022)
  field        not disclosed
  method       not disclosed
  model family not disclosed
  capability   answer questions
  application  airline customer service
  authority    execute: a statement the company must honour; Sheridan-Verplank level 7
  effects      display, state_policy
  CHECK FAILED customer-facing policy statements bind the company; ground them in the canonical policy or route them to a person
Refund bot (a flawed proposal)
  field        AI
  method       prompting
  model family Generative AI
  capability   generate, act
  application  customer refunds
  authority    execute; Sheridan-Verplank level 9-10
  effects      state_policy, act
  CHECK FAILED 'Generative AI' is a capability, not a model family
  CHECK FAILED irreversible action with no approval step, rollback or operating envelope
  CHECK FAILED customer-facing policy statements bind the company; ground them in the canonical policy or route them to a person

=== Part 3: Expected monthly cost of three review policies (C$) ===
100,000 conversations a month; 10% of wrong answers acted on at C$812.02 each; review 1.5 min at C$40/h catching 90%.
error rate      no review    full review   risk-routed   cheapest
      0.1%         8,120       100,812        12,274   no review
      0.5%        40,601       104,060        21,368   risk-routed
      1.0%        81,202       108,120        32,737   risk-routed
      2.0%       162,404       116,240        55,473   risk-routed
      5.0%       406,010       140,601       123,683   risk-routed
      8.0%       649,616       164,962       191,892   full review
Full review beats no review above an error rate of 1.37%.
Risk-routed review beats no review above 0.17%, and full review beats risk-routed above 6.16%.

What to notice:

• Authority is computed, not declared. Each profile lists the effects its outputs may have (display, prepare, state_policy, act_reversible, act), and the program takes the most powerful one as the system's authority. Copilot's suggestions only prepare a change, so it sits at draft even though its model is the same family as a chatbot's.
• The Air Canada check fires without knowing the technology. The chatbot's method and model are "not disclosed", yet its state_policy effect, with only a link to the policy page as a control, fails the rule that customer-facing policy statements must be grounded in the canonical policy or routed to a person.
• The flawed proposal fails three ways. It names a capability as a model family, takes an irreversible action with no approval or rollback, and makes policy statements with no grounding.
• The cheapest policy flips twice. No review is cheapest only below 0.17% wrong answers; routed review wins up to 6.16%; full review wins above that. All costs are proportional to the 100,000 conversations, so changing the volume changes every number and none of the flip points. Every rate except the C$812.02 award is illustrative.

Try this:

• Add a "answers quote canonical policy" control to the Air Canada profile and confirm the check passes. What would you need to build to make that control true?
• Describe a system you use at work as a new profile. Which axes did you have to mark "not disclosed", and does the authority check still work?
• Set ROUTED_ERRORS to 0.1, so routing concentrates nothing, and rerun Part 3. At which error rate does routed review stop being worth it?
• Raise ERROR_COST to 5,000 to represent an error that also costs a customer, and find the new flip points.