1. Epic News Development Guide¶

Light as a Haiku 🌸 Epic News is designed to be elegant, minimalist, and simple. Every line of code should serve a purpose.

1. Philosophy & Core Principles¶

Epic News operates on the CrewAI Flow paradigm - a revolutionary approach to multi-agent AI systems that emphasizes:

CrewAI-First: Use framework capabilities before writing custom Python.
Configuration-Driven: Separate code from configuration using YAML.
Context-Driven: Let data flow naturally through the system.
Async by Default: Maximize performance with parallel execution.

1.1. General Principles¶

KISS (Keep It Simple, Stupid): Favor simple solutions over complex ones.
YAGNI (You Aren't Gonna Need It): Don't build features until they're needed.
DRY (Don't Repeat Yourself): Eliminate code duplication through abstraction.
Fail Fast: Validate inputs early and provide clear error messages. Use type hints and Pydantic models for data validation.
Immutable State: State flows through the system without mutation. Use Pydantic models for structured data with clear state transitions.

1.2. Code Style and Quality¶

Imports: ALL imports must be placed at the top of Python files, never inside functions or methods. This ensures clarity, prevents circular imports, and follows PEP 8 standards.
Whitespace: Avoid trailing whitespace (W291) and whitespace on blank lines (W293). These are enforced by ruff and help maintain a clean, readable codebase.

1.3. Development Tools & Environment¶

Package Management¶

ALWAYS use uv: Epic News exclusively uses uv for all Python package management operations. Never use pip, poetry, or other package managers.

# ✅ CORRECT - Use uv for all operations
uv add package-name
uv remove package-name
uv run pytest
uv run python script.py
uv sync

# ❌ WRONG - Never use other package managers
# pip install package-name
# poetry add package-name

Why uv?

Speed: Significantly faster than pip/poetry
Reliability: Better dependency resolution
Consistency: Ensures reproducible environments
Modern: Built for Python 3.10+ with latest standards

Logging¶

Epic News uses Loguru for all logging. It provides a more powerful and flexible logging system than the standard logging module.

Configuration:

The main logging configuration is located in src/epic_news/utils/logger.py.
By default, logs are sent to the console and to a file in the logs/ directory.
The setup_logging() function in src/epic_news/utils/logger.py is called at the beginning of the kickoff function in src/epic_news/main.py to configure the log sinks.

Usage:

To use the logger in any module, simply import it:

from loguru import logger

Then, you can use the logger to log messages at different levels:

logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

Commit Messages¶

Well-crafted commit messages are crucial for maintaining a clean and understandable project history. They should be clear, concise, and follow a consistent style.

Multiline Commit Messages¶

When writing multiline commit messages, avoid passing the message directly via the command line, as it can be complex and error-prone with quoting. Instead, use a temporary file.

Workflow¶

Create a temporary file with your detailed commit message.
Use the -F or --file flag with git commit to read the message from the file.
Delete the temporary file after the commit is successful.

Example¶

# 1. Write the commit message to a temporary file
echo "feat: Add new feature

- Detailed description of the feature.
- Rationale behind the implementation.
- Closes #123" > commit_message.tmp

# 2. Commit using the file
git commit -F commit_message.tmp

# 3. Clean up the temporary file
rm commit_message.tmp

This approach ensures that your commit messages are well-formatted and easy to write, especially for more complex changes.

Makefile - Primary Development Interface¶

Epic News uses a comprehensive Makefile as the primary interface for all development tasks. This ensures consistency between local development and CI/CD environments.

Philosophy: The Makefile wraps common development operations with simple, memorable commands. All targets use uv internally.

Getting Started¶

# See all available targets
make help

# First-time setup
make dev              # Install all dependencies + pre-commit hooks

# Daily development workflow
make lint             # Check code style (ruff + yamllint)
make format           # Format code (ruff + yamlfix)
make fix              # Auto-fix linting issues
make test             # Run tests quickly
make coverage         # Run tests with coverage

Available Target Categories¶

Installation & Setup: - make install - Install production dependencies - make dev - Install development dependencies (includes mypy, bandit, safety) - make build - Build package distribution - make clean - Remove all artifacts

Code Quality: - make lint - Run ruff check + yamllint - make format - Format code with ruff + yamlfix - make fix - Auto-fix linting issues - make test - Quick test run without coverage - make coverage - Tests with HTML + terminal coverage report - make pre-commit - Run pre-commit hooks on all files - make validate - Quick validation (lint + test) - make ci-checks - All CI checks (lint + test)

Advanced Quality Checks: - make type-check - Run mypy type checking - make security - Run bandit (code security) + safety (dependency vulnerabilities)

Docker Operations: - make docker-build-api - Build FastAPI image - make docker-build-streamlit - Build Streamlit image - make docker-build-combined - Build combined image - make docker-build-all - Build all Docker images - make docker-run-* - Run respective containers

Application Runtime: - make run-streamlit - Start Streamlit app (port 8501) - make run-api - Start FastAPI server (port 8000) - make run-crew - Run CrewAI flow kickoff - make update-kb - Update knowledge base

Dependency Management: - make show-deps - Show dependency tree - make show-outdated - Show outdated packages - make sync - Sync dependencies with lock file - make lock - Update lock file

Why Use the Makefile?¶

Consistency: Same commands work locally and in CI/CD
Simplicity: Memorable targets instead of complex uv commands
Documentation: make help shows all available operations
Efficiency: Composite targets (e.g., make fix runs format + auto-fix)
Quality Gates: Conditional checks (security/type-check warn if tools not installed)

Integration with CI/CD¶

The GitHub Actions CI workflow (.github/workflows/ci.yml) uses Makefile targets:

- name: Install dependencies using Makefile
  run: make install

- name: Run linting checks using Makefile
  run: make lint

- name: Run tests with coverage using Makefile
  run: make coverage

This ensures developers run the exact same commands locally as CI does.

Common Development Workflows¶

Before committing:

make fix              # Auto-fix issues
make test             # Verify tests pass
make lint             # Check for remaining issues

Before pushing:

make pre-commit       # Run all pre-commit hooks
make type-check       # Optional: check type hints
make security         # Optional: security scan

Full quality check:

make all              # clean + install + lint + test

Testing & Quality Assurance¶

IMPORTANT: Use Makefile targets instead of direct commands for consistency.

Testing: make test (quick) or make coverage (with coverage report)
Linting: make lint (check) or make fix (auto-fix)
Type Checking: make type-check (requires mypy)
Security: make security (bandit + safety)
Coverage: Maintain comprehensive test coverage for all utilities

Advanced Testing Libraries¶

To enhance our testing capabilities, Epic News integrates the following libraries for more robust and realistic tests:

Faker: For generating realistic mock data (e.g., names, addresses, dates). This helps create tests that more closely resemble real-world scenarios.
pytest-mock: A wrapper around the standard unittest.mock library, providing a more convenient and pytest-friendly interface for mocking objects and functions.
pendulum: For precise control over date and time in tests. This is crucial for testing time-sensitive logic, allowing you to "freeze" time or travel to specific points in time.

Example Usage¶

import pendulum
from faker import Faker

def test_faker_and_pendulum():
    """
    This test demonstrates the use of Faker and Pendulum.
    """
    fake = Faker()
    name = fake.name()
    now = pendulum.now()

    assert isinstance(name, str)
    assert isinstance(now, pendulum.DateTime)

def test_mocking_with_pytest_mock(mocker):
    """
    This test demonstrates the use of pytest-mock.
    """
    # Create a mock object
    mock_object = mocker.Mock()

    # Configure the mock to return a specific value when a method is called
    mock_object.get_name.return_value = "Test Name"

    # Call the method on the mock object
    result = mock_object.get_name()

    # Assert that the method was called and returned the expected value
    mock_object.get_name.assert_called_once()
    assert result == "Test Name"

Using `contextlib` for Cleaner Tests¶

To further improve the readability and maintainability of our tests, we use Python's built-in contextlib library to create reusable context managers for mocking dependencies. This is particularly useful when multiple tests require the same set of mocks.

Benefits¶

Encapsulation: Bundles related setup and teardown logic into a single, reusable function.
Readability: Replaces repetitive mocker.patch() calls with a clean with statement, making the test's intent clearer.
Maintainability: Allows you to update mocking logic in one central place, rather than in every test.

Example¶

Instead of this:

def test_some_function_with_many_mocks(mocker):
    mocker.patch("module.a.dependency_one")
    mocker.patch("module.b.dependency_two")
    mocker.patch("module.c.dependency_three")
    # ... test logic ...

We can create a reusable context manager:

# In a helper file like tests/utils/context_managers.py
from contextlib import contextmanager

@contextmanager
def mock_app_dependencies(mocker):
    """A context manager to mock core application dependencies."""
    mocks = {
        "dep1": mocker.patch("module.a.dependency_one"),
        "dep2": mocker.patch("module.b.dependency_two"),
        "dep3": mocker.patch("module.c.dependency_three"),
    }
    try:
        yield mocks
    finally:
        # Teardown is handled automatically by pytest-mock
        pass

# The test becomes much cleaner:
def test_some_function_with_many_mocks(mocker):
    with mock_app_dependencies(mocker) as mocks:
        # ... test logic using mocks['dep1'], etc.

This approach is highly encouraged for complex test setups.

Development Workflow¶

Environment Setup:
Run uv sync to install dependencies
Install the package in editable mode: uv pip install -e .
This creates a special link to your source code and prevents import errors like ModuleNotFoundError: No module named 'epic_news'
Make Changes: Edit code, add features, fix bugs
Test Changes:
Run uv run pytest to ensure tests pass
For specific components: uv run python tests/bin/regenerate_holiday_plan.py
Quality Checks: Run uv run yamlfix src ; uv run ruff check --fix to ensure code quality
Execute Crews:
Use crewai flow kickoff to run CrewAI flows
This validates the complete pipeline including Pydantic model validation

CrewAI Flow: Engagement Rules & Best Practices¶

Execution Method (CRITICAL)¶

ALWAYS use the CrewAI Flow command to run crews. This is the single most important rule for maintaining a consistent and manageable architecture.

# ✅ CORRECT - Use CrewAI Flow command
crewai flow kickoff

# ❌ WRONG - Never run Python modules directly
# python -m src.epic_news.crews.fin_daily.run
# python src/epic_news/main.py

Why this matters¶

CrewAI Flow manages the complete execution lifecycle.
It ensures proper state management and context injection.
It handles async execution and crew coordination.
It guarantees that all crews follow the same execution pattern.

2.2. Anti-Patterns (What to Avoid)¶

NEVER bypass CrewAI Flow with direct Python execution (MyCrew().crew().kickoff()).
NEVER call crew methods directly from main.py.
NEVER reinvent the wheel. Use existing utilities like ensure_output_directories() instead of os.makedirs().
NEVER pass context via constructor parameters. Use CrewAI's input mechanism.

# ✅ CORRECT - Context-driven
crew_inputs = self.state.to_crew_inputs()
result = MyCrew().crew().kickoff(inputs=crew_inputs)

# ❌ WRONG - Constructor injection
crew = MyCrew(topic=topic, objective=objective)

2.3. Python Type Compatibility (CRITICAL)¶

CrewAI Pydantic Schema Parser Limitation: CrewAI's internal schema parser is incompatible with Python 3.10+ Union syntax (X | Y).

ALWAYS use the legacy Union[X, Y] or Optional[X] syntax for all Pydantic models used with CrewAI.

from typing import Union, Optional

# ✅ CORRECT - Compatible with CrewAI
field: Union[str, None] = None
field: Optional[str] = None

# ❌ WRONG - Causes AttributeError in CrewAI
field: str | None = None

Rationale: CrewAI's PydanticSchemaParser attempts to access field_type.__name__ on types.UnionType objects, which don't have this attribute. This causes runtime errors during schema generation for task outputs.

2.4. Data Normalization for Pydantic Validation¶

When working with external data (especially from LLMs) that needs to be validated against Pydantic models:

Field Name Mapping: Create normalization functions to map field names:

# Map alternative field names to expected schema names
if "name" in source and "title" not in source:
    source["title"] = source["name"]
if "type" in contact and "service" not in contact:
    contact["service"] = contact["type"]

Multilingual Key Handling: Handle alternative language keys:

# Map French keys to English
if "numéro" in contact and "number" not in contact:
    contact["number"] = contact["numéro"]
if "fr" in phrase and "french" not in phrase:
    phrase["french"] = phrase["fr"]

Structure Conversion: Convert data structures as needed:

# Convert strings to objects with required fields
if isinstance(phrase, str):
    phrase = {"french": phrase, "local": phrase}

Default Values: Provide sensible defaults for missing required fields:

# Ensure required fields exist
if "address" not in accommodation:
    accommodation["address"] = "Address not provided"

This approach ensures robust validation while maintaining flexibility with various input formats.

3. Agent Handbook¶

This section establishes the core principles, ethical standards, and research methodologies for all AI agents.

3.1. Agent Code of Conduct¶

Core Principles¶

Accuracy and Thoroughness: Always provide complete, accurate information. Cite authoritative sources and acknowledge limitations.
Output Quality: Structure information logically. Use formatting to enhance readability and follow specified output formats exactly.
Ethical Guidelines: Present balanced perspectives. Avoid biased language and respect intellectual property.
Collaboration: Pass complete context to other agents. Document reasoning and maintain a consistent tone.
Technical Best Practices: Follow project design principles (KISS, DRY). Write clean, documented, and tested code.

Agent Responsibilities¶

Research Agents: Provide exhaustive, factual information (20+ data points). Use tools freely, as they do not write to the final output file.
Reporting Agents: Transform research into well-structured, professional reports. DO NOT USE TOOLS to prevent action traces from contaminating the final HTML output.

3.2. Shared Research Guidelines¶

Objectivity: Analysis must be impartial and backed by verifiable evidence.
Clarity: Use clear, concise language.
Rigor: Triangulate information from multiple sources.
Adherence: Strictly follow task instructions and constraints.
Professionalism: Format output in a clean, readable manner.

4. Output Standards¶

The primary output of each crew is a comprehensive HTML report.

4.1. Critical Reminders¶

DO NOT return raw API responses.
DO NOT include placeholder text like 'TODO'.
ALWAYS ensure your final output is a single, complete, and well-formed HTML document.

4.2. HTML Report Standards¶

Structure: Use proper HTML5 (<!DOCTYPE html>) with UTF-8 encoding (<meta charset="UTF-8">).
Content: Present information in logical sections with clear headings, data sources, and actionable conclusions.
Prohibited: No raw JSON, API responses, or malformed HTML.
Emoji Usage: Use emojis strategically to highlight key points (e.g., 📈 for growth, ⚠️ for risks).

4.3. HTML Output Architecture (CRITICAL)¶

CrewAI has a known issue where agents with tools write action traces to output files instead of the final result. To prevent this, use a Two-Agent Pattern.

Research Agent(s): Equipped with tools to gather information. They do not have an output_file and pass data to the next agent via context.
Reporting Agent: Has NO TOOLS. This agent's sole purpose is to take the data from the context and generate a clean HTML report.

# ✅ CORRECT - Separate research and reporting agents
@agent
def researcher(self) -> Agent:
    return Agent(
        tools=[WikipediaSearchTool(), HybridSearchTool()],  # Research with tools
    )

@agent
def reporter(self) -> Agent:
    return Agent(
        tools=[],  # NO TOOLS = No action traces
        # Generates clean HTML output
    )

5. Configuration & Project Structure¶

5.1. YAML-First Approach¶

Define agents, tasks, and crews in YAML files to separate configuration from code.

# agents.yaml
product_researcher:
  role: "Product Research Specialist"
  goal: "Research {topic} comprehensively"
  backstory: "Expert in analyzing products and market trends"

5.2. Tool Assignment (CRITICAL)¶

NEVER specify tools in agents.yaml. Tools must be assigned programmatically in code using centralized factories.

# ✅ CORRECT - Tools assigned in code
@agent
def researcher(self) -> Agent:
    return Agent(
        config=self.agents_config["researcher"],
        tools=self.search_tools,  # Tools from factory functions
        verbose=True,
    )

Why: Hybrid YAML/code tool configuration causes KeyError when CrewAI tries to map YAML tool names to non-existent functions.

5.3. Path and Directory Management¶

Paths: All file paths must be project-relative and managed programmatically to avoid nested path issues.
Directories: Directory creation should be centralized using ensure_output_directories() at application startup. Do not use os.makedirs within individual crew or task logic.

5.4. Project Structure¶

epic_news/
├── src/epic_news/
│   ├── crews/           # Crew implementations
│   ├── tools/           # Tool implementations
│   ├── models/          # Pydantic models
│   └── main.py          # Flow orchestration
├── docs/                # Documentation
├── output/              # Generated reports
└── tests/               # Unit tests

The following best-practices were distilled while integrating the MenuDesignerCrew and its HTML pipeline:

Pydantic Union Syntax – CrewAI’s internal schema parser still requires legacy typing.Union / Optional[...] annotations. Ensure every field in menu_designer_report.py (and any new model) avoids the X | Y pipe syntax to prevent runtime parsing errors.
Factory Pattern Consistency – Each crew must expose a small, deterministic factory (e.g. menu_to_html) that:
Normalises CrewOutput, Pydantic model or dict into a serialisable dict.
Delegates body creation to TemplateManager.render_report() using the correct crew identifier.
Optionally persists the resulting HTML.
Renderer Construction – All concrete subclasses of BaseRenderer must implement __init__ (even if empty) to avoid Python treating them as abstract, otherwise TemplateManager fails to instantiate them.
Nested Meal Rendering – For structured meals (starter / main_course / dessert), MenuRenderer now:
Detects nested dicts within lunch / dinner entries.
Renders readable bullet-lists with appropriate emojis.
Falls back gracefully to plain strings when structure is missing.
DailyMenu Iteration – MenuRenderer supports both legacy {day: meals} mappings and the new list-of-objects (DailyMenu) produced by the Pydantic model.
Debugging Pattern – Use dump_crewai_state() to capture problematic crew outputs. Enable by setting DEBUG_STATE=true in the environment.

Adhering to these guidelines keeps new menu-style crews consistent with the broader HTML rendering architecture.

6. Development Workflow¶

Package Management: Use uv for all Python operations.
Testing:
Focus on testing deterministic components (tools).
Use crewai flow kickoff for end-to-end validation.
Do not write unit tests for non-deterministic agents.
Documentation: Update the relevant handbooks when adding new tools or patterns.

1. Epic News Development Guide¶

1. Philosophy & Core Principles¶

1.1. General Principles¶

1.2. Code Style and Quality¶

1.3. Development Tools & Environment¶

Package Management¶

Logging¶

Commit Messages¶

Multiline Commit Messages¶

Workflow¶

Example¶

Makefile - Primary Development Interface¶

Getting Started¶

Available Target Categories¶

Why Use the Makefile?¶

Integration with CI/CD¶

Common Development Workflows¶

Testing & Quality Assurance¶

Advanced Testing Libraries¶

Example Usage¶

Using contextlib for Cleaner Tests¶

Benefits¶

Example¶

Development Workflow¶

CrewAI Flow: Engagement Rules & Best Practices¶

Execution Method (CRITICAL)¶

Why this matters¶

2.2. Anti-Patterns (What to Avoid)¶

2.3. Python Type Compatibility (CRITICAL)¶

2.4. Data Normalization for Pydantic Validation¶

3. Agent Handbook¶

3.1. Agent Code of Conduct¶

Core Principles¶

Agent Responsibilities¶

3.2. Shared Research Guidelines¶

4. Output Standards¶

4.1. Critical Reminders¶

4.2. HTML Report Standards¶

4.3. HTML Output Architecture (CRITICAL)¶

5. Configuration & Project Structure¶

5.1. YAML-First Approach¶

5.2. Tool Assignment (CRITICAL)¶

5.3. Path and Directory Management¶

5.4. Project Structure¶

4.4. Menu Designer & HTML Renderer Learnings¶

6. Development Workflow¶

Using `contextlib` for Cleaner Tests¶