Skip to content

3. Epic News Architectural Patterns

This document describes key architectural patterns and solutions implemented in the Epic News system, with a strong focus on ensuring reliable, maintainable, and high-quality outputs.

1. Report Generation (DOCX)

Reports are DOCX files and nothing else (ADR-017). The core principle is to separate content generation from presentation: the crew produces structured data, and an assembler turns it into a document.

1.1. Assemblers

Pattern:

  1. Execute Crew: The CrewAI flow runs as usual to generate the core content. The .kickoff() method returns a CrewOutput object.
  2. Parse to Pydantic Model: The raw output from the crew is parsed into a structured Pydantic model (e.g., SaintData, FinancialReport).
  3. Assemble the DOCX: The model goes to the crew's assembler (utils/docx_report/crews/). Some sections are written by one LLM call each, the others are filled in deterministically. build_docx() converts the Markdown fragments with pandoc, using the styles in reference.docx, and refuses any path outside output/.
  4. Record the path: emit_report(state, assemble_docx) stores the file path in state.output_file. The email step attaches it and the Streamlit app offers it for download.
# Example of the flow (from ReceptionFlow.generate_poem)
self._run_standard(CREW_REGISTRY[CrewKey.POEM], PoemCrew(), inputs)

# _run_standard does, in order:
#   kickoff_flow(crew, inputs)                  # run the crew
#   load_or_parse_model(json_path, ...)         # JSON written by output_pydantic, or the raw output
#   emit_report(state, assembler closure)       # build the DOCX; an error here stops the run
#   state.report, state.raw_output = model, output   # set only after the DOCX is built

_run_standard takes its model class, paths, title and assembler from the crew's CrewSpec in src/epic_news/crew_registry.py.

1.2. Data Contract Between Crew and Assembler

A common issue arises when the data structure a crew produces does not match what the assembler expects.

Best Practice:

  • Ensure the data structure passed from the crew to the assembler is consistent.
  • Create a clear contract (e.g., via Pydantic models) between the data source and the assembler.
  • Map fields explicitly to prevent mismatches (e.g., link → url, published → date).

1.3. Failures Stop the Run

A report that cannot be built raises. There is no placeholder report, and nothing partial is emailed. If more than half the narrated sections of a report fail, the assembler aborts instead of writing a document full of placeholders.

Issue: Incorrect regex can fail to parse Markdown links [text](url).

Solution: Use re.search() with a non-greedy pattern.

# ✅ CORRECT - Finds a link anywhere in the string
import re
match = re.search(r"\[(.*?)\]\((.*?)\)", text_with_link)
if match:
    link_text = match.group(1)
    link_url = match.group(2)

2. Information Retrieval Strategy

The project prioritizes data freshness over a static knowledge base. Instead of a traditional RAG that can become stale, agents use a suite of real-time tools to fetch information on demand.

2.1. Core Principle: Real-Time Retrieval

Financial markets are highly dynamic. To provide accurate and timely analysis, agents retrieve live data from the web for every task.

2.2. Key Retrieval Tools

  • HybridSearchTool: Cascading search with Perplexity (primary) → Brave → Serper fallback chain for reliable web searches.
  • ScraperFactory-selected scraper (get_scraper()): Centralized website scraping; defaults to ScrapeNinjaTool. Override via WEB_SCRAPER_PROVIDER (scrapeninja, firecrawl). Direct Firecrawl usage is deprecated in crews.
  • YahooFinanceNewsTool: For fetching the latest financial news for a specific ticker, providing timely market-moving information.

2.3. Why Not a Traditional RAG?

  • Data Staleness: A vector database would require constant, resource-intensive updates.
  • Scope Limitation: A pre-populated database is limited, whereas live tools can access the entire public web.

Agents share information within a crew through task context, not through a vector store.

3. Case Study: Refactoring the Sales Prospecting Report

The evolution of the SalesProspectingReport provides a clear example of applying architectural principles to solve real-world data challenges.

3.1. The Initial Problem

The first version of the SalesProspectingReport relied on a highly complex, nested Pydantic model called StructuredDataReport. This model was designed to capture a wide array of metrics, KPIs, data series, and tables.

Challenge: The data generated by the LLM agents did not align with this rigid and complex structure. The output was a simple, flat JSON with fields like company_overview and key_contacts. As a result, the Pydantic validation failed, and the rendering process broke.

3.2. The Solution: Data-Centric Refactoring

Instead of forcing the agents to conform to an overly complex model, the architecture was adapted to the data.

  1. Model Simplification: The SalesProspectingReport Pydantic model was completely redesigned to match the actual data being produced. The dependency on StructuredDataReport was removed, and a new, simpler structure was implemented:

    • company_overview: str
    • key_contacts: List[KeyContact] (with a new KeyContact sub-model)
    • approach_strategy: str
    • remaining_information: str

  2. Assembler Update: The sales prospecting assembler was rewritten to work with the new, simpler model. This involved:

    • Removing the logic for rendering metrics and KPIs.
    • Adding sections for company_overview, key_contacts (a table), and remaining_information.

  3. Data File Correction: The debug/repair_attempt...json file was updated to conform to the new, simpler Pydantic model, ensuring that tests and local development would work correctly.

3.3. Architectural Lessons

  • Model the Data You Have: Design Pydantic models that reflect the actual data being generated, not an idealized version.
  • Simplicity Over Complexity: A simpler, flatter data structure is often more robust and easier to work with than a deeply nested one.
  • Decouple Rendering from Data Structure: While the assembler needs to understand the data, the refactoring was made easier because the report logic was contained in one assembler and not scattered across the application.