Input Matters: Evaluating Input Structure’s Impact on LLM Summaries of Sports Play-by-Play

Error Categories (Annotation Protocol)

Shorthand used in annotations: U=Number, N=Name, WO=Word Objective, WS=Word Subjective, C=Context, X=Not Checkable, O=Other.

Input Structures (Partial game excerpts from the NBA play-by-play logs)

{
      "time": {
        "period": "OT1",
        "clock": "2:13.0"
      },
      "team": "Los Angeles Lakers",
      "play_details": {
        "description": "LeBron James makes 3-pt jump shot from 25 ft",
        "event": {
          "type": "Shot",
          "action": "Jump Shot",
          "outcome": "Made",
          "distance": "25 ft",
          "points": 3
        },
        "players": {
          "primary_player": "LeBron James"
        }
      },
      "score": {
        "points_scored": "LAL: 3",
        "cumulative": {
          "LAL": 127,
          "ATL": 125
        }
      }
    }

Summary Examples with Annotations

Unstructured input: Llama3.1-70B (partial summary)

The Lakers fought back, however, with James and Davis leading the charge. James hit a pair of three-pointers^WO, and Davis added a dunk and a layup^WO to tie the game at 130^U-129^U. Throughout the game, James and Davis were the Lakers’ top performers, combining for 63^U points. Young and Bogdanovic^N led the Hawks, combining for 61^U points.

• a pair of three-pointers^WO: incorrect word objective phrases from events, correct event/phrase is ‘a three-pointer’.
• a dunk and a layup^WO: incorrect events (word objective), the player made a ‘free-throw’.
• 130 - 129^U: incorrect scores (numbers), the correct scores are ‘123-123.’
• 63^U: incorrect score (number), the correct number/score is ‘77.’
• Bogdanovic^N: incorrect player name, the correct player name is ‘Hunter.’
• 61^U: incorrect score (number), the correct number/score is ‘75.’

Row input: Qwen2.5-72B (partial summary)

The game saw several lead changes, with Anthony Davis making a pair of free throws^WO to tie the game at 123. LeBron James then hit a three-pointer to give the Lakers a 127–125 lead, but Trae Young answered with a layup^WO to tie it again. The Lakers regained the lead with a three-pointer from Gabe Vincent, but the Hawks stayed close with a jump shot from Dyson Daniels. LeBron James finished with a double^WO-double, contributing significantly on both ends of the court.

• a pair of free throws^WO: the game was tied at 123 when Davis ‘made a free-throw.’
• with a layup^WO: incorrect events (word objective), the player made ‘a jump shot.’
• double^WO: incorrect word (objective), LeBron James finished with a ‘triple-double.’

JSON input: Qwen2.5-72B (partial summary)

The Hawks took an early lead with a layup from Onyeka Okongwu, but the Lakers quickly tied it^WO with a three-pointer from Max Christie. Throughout the game, LeBron James and Anthony Davis were the standout players for the Lakers, contributing significantly with scoring, rebounds, and assists.

• tied it^WO: word objective error, the Lakers ‘responded with’ a three-pointer, they did not tie the game.

Dataset

• Row-structured play-by-play (Google Drive)
• JSON-structured play-by-play (Google Drive)

Please cite the paper if these resources are used.

BibTeX