apunkt/flakestorm
1
0
Fork 0
mirror of https://github.com/flakestorm/flakestorm.git synced 2026-04-25 00:36:54 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
flakestorm/docs/CONTRIBUTING.md

8.2 KiB
Raw Blame History

Contributing to flakestorm

Thank you for your interest in contributing to flakestorm! This document provides guidelines and instructions for contributing.

Code of Conduct

Please be respectful and constructive in all interactions. We welcome contributors of all experience levels.

Getting Started

Development Setup

  1. Clone the repository

    git clone https://github.com/flakestorm/flakestorm.git
cd flakestorm

  2. Set up Python environment

    # 1. Verify Python version (3.10+ required)
python3 --version

# 2. Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# 3. Upgrade pip (required for pyproject.toml editable installs)
pip install --upgrade pip

# 4. Verify pip version (should be 21.0+)
pip --version

# 5. Install in editable mode
pip install -e ".[dev]"

  3. Install Ollama (for mutation generation)

    macOS:

    # Option 1: Homebrew (recommended)
brew install ollama

# Option 2: Official installer
# Visit https://ollama.ai/download and download the .dmg file

    Windows:

    Linux:

    curl -fsSL https://ollama.com/install.sh | sh

    Then pull the model:

    ollama pull qwen3:8b

  4. Set up Rust (optional, for performance module)

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cd rust && cargo build --release

  5. Install pre-commit hooks

    # Make sure venv is activated first!
source venv/bin/activate  # If not already activated

# Verify pre-commit is installed
which pre-commit  # Should show: .../venv/bin/pre-commit

# Install git hooks (this will update .git/hooks/pre-commit)
pre-commit install

# Verify hooks are installed
pre-commit run --all-files  # Optional: test all files

    Note: If you get pre-commit not found when committing, the hooks need to be reinstalled:

    source venv/bin/activate
pre-commit install --overwrite

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=src/flakestorm --cov-report=html

# Run specific test file
pytest tests/test_config.py

# Run specific test
pytest tests/test_config.py::TestFlakeStormConfig::test_create_default_config

Code Style

We use:

  • black for Python formatting
  • ruff for linting
  • mypy for type checking
# Format code
black src tests

# Lint
ruff check src tests

# Type check
mypy src

Project Structure

flakestorm/
├── src/flakestorm/           # Main package
│   ├── cli/                # CLI commands
│   ├── core/               # Core logic
│   ├── mutations/          # Mutation engine
│   ├── assertions/         # Invariant checkers
│   ├── reports/            # Report generators
│   └── integrations/       # External integrations
├── rust/                   # Rust performance module
├── tests/                  # Test suite
├── docs/                   # Documentation
└── examples/               # Example configurations

How to Contribute

Finding Good First Issues

New to contributing? Look for issues labeled good first issue on GitHub. These are specifically curated for beginners and include:

  • Clear problem statements
  • Well-defined scope
  • Helpful guidance in the issue description
  • Good learning opportunities

To find them:

  1. Go to Issues
  2. Filter by label: good first issue
  3. Read the issue description and ask questions if needed
  4. Comment on the issue to let others know you're working on it

Note for Maintainers: To add good first issue labels to beginner-friendly issues:

  • Look for issues that are well-scoped and have clear acceptance criteria
  • Add the good first issue label via GitHub's web interface
  • Ensure the issue description includes context and guidance for new contributors
  • Consider 5-10 issues at a time to give beginners options

Reporting Bugs

  1. Check existing issues first
  2. Include:
    • flakestorm version
    • Python version
    • Operating system
    • Steps to reproduce
    • Expected vs actual behavior
    • Error messages/logs

Suggesting Features

  1. Open an issue with the "enhancement" label
  2. Describe the use case
  3. Explain why existing features don't meet the need
  4. If possible, outline an implementation approach

Submitting Pull Requests

  1. Fork the repository

  2. Create a feature branch

    git checkout -b feature/my-feature

  3. Make your changes

    • Write clear, documented code
    • Add tests for new functionality
    • Update documentation as needed

  4. Run checks locally

    black src tests
ruff check src tests
mypy src
pytest

  5. Commit with clear messages

    git commit -m "feat: Add new mutation type for XXX"

    Use conventional commits:

    • feat: New feature
    • fix: Bug fix
    • docs: Documentation
    • test: Tests
    • refactor: Code refactoring
    • chore: Maintenance

  6. Push and create PR

    git push origin feature/my-feature

  7. PR Description should include

    • What the change does
    • Why it's needed
    • How it was tested
    • Any breaking changes

Development Guidelines

Adding a New Mutation Type

  1. Add to MutationType enum in mutations/types.py
  2. Add template in mutations/templates.py
  3. Add weight in core/config.py
  4. Add tests in tests/test_mutations.py
  5. Update documentation

Adding a New Invariant Checker

  1. Create checker class in assertions/ (deterministic, semantic, or safety)
  2. Implement check(response, latency_ms) -> CheckResult
  3. Register in assertions/verifier.py CHECKER_REGISTRY
  4. Add to InvariantType enum if new type
  5. Add tests
  6. Document in CONFIGURATION_GUIDE.md

Adding a New Agent Adapter

  1. Create adapter class implementing AgentProtocol
  2. Add to core/protocol.py
  3. Add to AgentType enum if new type
  4. Update create_agent_adapter() factory
  5. Add tests
  6. Document usage

Testing Guidelines

Test Structure

class TestMyFeature:
    """Tests for MyFeature."""

    def test_happy_path(self):
        """Test normal operation."""
        ...

    def test_edge_case(self):
        """Test edge case handling."""
        ...

    def test_error_handling(self):
        """Test error conditions."""
        ...

Async Tests

import pytest

@pytest.mark.asyncio
async def test_async_function():
    result = await some_async_function()
    assert result is not None

Mocking Ollama

from unittest.mock import AsyncMock, patch

@patch('flakestorm.mutations.engine.AsyncClient')
async def test_mutation_generation(mock_client):
    mock_client.return_value.generate = AsyncMock(
        return_value={"response": "mutated text"}
    )
    # Test code...

Documentation

Docstring Format

def function_name(param1: str, param2: int = 10) -> bool:
    """
    Brief description of function.

    Longer description if needed. Explain what the function
    does, not how it does it.

    Args:
        param1: Description of param1
        param2: Description of param2

    Returns:
        Description of return value

    Raises:
        ValueError: When param1 is empty

    Example:
        >>> result = function_name("test")
        >>> print(result)
        True
    """

Updating Documentation

  • README.md: High-level overview and quick start
  • CONFIGURATION_GUIDE.md: Detailed config reference
  • API_SPECIFICATION.md: Python SDK reference
  • ARCHITECTURE_SUMMARY.md: System design

Release Process

  1. Update version in pyproject.toml and __init__.py
  2. Update CHANGELOG.md
  3. Create release PR
  4. After merge, tag release
  5. CI automatically publishes to PyPI

Getting Help

  • Open an issue for questions
  • Join Discord community (coming soon)
  • Check existing documentation

Recognition

Contributors are recognized in:

  • CONTRIBUTORS.md
  • Release notes
  • GitHub contributors page

Thank you for contributing to flakestorm!