flakestorm/docs/PUBLISHING.md

Publishing flakestorm to PyPI

This guide explains how to publish flakestorm so users can install it with pip install flakestorm.

---

Table of Contents

1. Understanding PyPI
2. Prerequisites
3. Project Structure for Publishing
4. Step-by-Step Publishing Guide
5. Automated Publishing with GitHub Actions
6. Publishing the Rust Extension
7. Version Management
8. Testing Before Publishing
9. Common Issues

---

Understanding PyPI

What is PyPI?

PyPI (Python Package Index) is the official repository for Python packages. When users run:

pip install flakestorm

pip downloads the package from PyPI (https://pypi.org).

What Gets Published?

A Python package is distributed as either:

• Source Distribution (sdist): .tar.gz file with source code
• Wheel (bdist_wheel): .whl file, pre-built for specific platforms

For flakestorm:

• Pure Python code: Published as universal wheel (works everywhere)
• Rust extension: Published as platform-specific wheels (separate process)

---

Prerequisites

1. PyPI Account

Create accounts on:

• Test PyPI: https://test.pypi.org/account/register/ (for testing)
• PyPI: https://pypi.org/account/register/ (for production)

2. API Tokens

Generate API tokens (more secure than username/password):

1. Go to https://pypi.org/manage/account/token/
2. Create a token with scope "Entire account" or project-specific
3. Save the token securely (you'll only see it once!)

3. Install Build Tools

pip install build twine hatch

---

Project Structure for Publishing

flakestorm is already set up correctly. Here's what makes it publishable:

pyproject.toml (Key Sections)

[build-system]
requires = ["hatchling", "hatch-fancy-pypi-readme"]
build-backend = "hatchling.build"

[project]
name = "flakestorm"                    # Package name on PyPI
version = "0.1.0"                    # Version number
description = "The Agent Reliability Engine"
readme = "README.md"                 # Shown on PyPI page
license = "Apache-2.0"
requires-python = ">=3.10"
dependencies = [                     # Auto-installed with package
    "typer>=0.9.0",
    "rich>=13.0.0",
    # ...
]

[project.scripts]
flakestorm = "flakestorm.cli.main:app"  # Creates `flakestorm` command

[tool.hatch.build.targets.wheel]
packages = ["src/flakestorm"]         # What to include in wheel

Directory Structure

flakestorm/
├── pyproject.toml      # Package metadata (required)
├── README.md           # PyPI description
├── LICENSE             # License file
├── src/
│   └── flakestorm/       # Your package code
│       ├── __init__.py # Must exist for package
│       ├── core/
│       ├── mutations/
│       └── ...
└── tests/              # Not included in package

src/flakestorm/__init__.py (Package Entry Point)

"""flakestorm - The Agent Reliability Engine"""

__version__ = "0.1.0"

from flakestorm.core.config import load_config, FlakeStormConfig
from flakestorm.core.runner import FlakeStormRunner

__all__ = ["load_config", "FlakeStormConfig", "FlakeStormRunner", "__version__"]

---

Step-by-Step Publishing Guide

Step 1: Verify Package Metadata

# Check pyproject.toml is valid
# NOTE: Use editable mode for development, regular install for testing wheel builds
pip install -e .  # Editable mode (recommended for development)

# OR test the wheel build process:
python -m pip install build
python -m build --wheel
python -m pip install dist/*.whl

# Verify the package works
flakestorm --version

Important: If you get ModuleNotFoundError: No module named 'flakestorm.reports' when using pip install . (non-editable), it means the wheel build didn't include all subpackages. Use pip install -e . for development, or ensure pyproject.toml has the correct packages configuration.

Step 2: Build the Package

# Install build tools (if not already installed)
pip install build

# Clean previous builds
rm -rf dist/ build/ *.egg-info src/*.egg-info

# Build source distribution and wheel
python -m build

# You should see:
# dist/
#   flakestorm-0.1.0.tar.gz      (source)
#   flakestorm-0.1.0-py3-none-any.whl  (wheel)

# Verify all subpackages are included (especially reports)
unzip -l dist/*.whl | grep "flakestorm/reports"

Step 3: Check the Build

# Install twine for checking (if not already installed)
pip install twine

# Verify the package contents
twine check dist/*

# List files in the wheel
unzip -l dist/*.whl

# Ensure it contains all subpackages:
# - flakestorm/__init__.py
# - flakestorm/core/*.py
# - flakestorm/mutations/*.py
# - flakestorm/reports/*.py  (important: check this exists!)
# - flakestorm/assertions/*.py
# - etc.

# Quick check for reports module:
unzip -l dist/*.whl | grep "flakestorm/reports"

Step 4: Test on Test PyPI (Recommended)

# Upload to Test PyPI first
twine upload --repository testpypi dist/*

# You'll be prompted for:
# Username: __token__
# Password: pypi-your-test-token-here

# Install from Test PyPI to verify
pip install --index-url https://test.pypi.org/simple/ flakestorm

Step 5: Publish to Production PyPI

# Upload to real PyPI
twine upload dist/*

# Username: __token__
# Password: pypi-your-real-token-here

Step 6: Verify Installation

# In a fresh virtual environment
python -m venv test_env
source test_env/bin/activate

pip install flakestorm
flakestorm --version

🎉 Congratulations! Users can now pip install flakestorm!

---

Automated Publishing with GitHub Actions

Set up automatic publishing when you create a release:

.github/workflows/publish.yml

name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install build tools
        run: pip install build twine

      - name: Build package
        run: python -m build

      - name: Check package
        run: twine check dist/*

      - name: Publish to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
        run: twine upload dist/*

Setting Up the Secret

1. Go to your GitHub repo → Settings → Secrets → Actions
2. Add a new secret named PYPI_TOKEN
3. Paste your PyPI API token as the value

Creating a Release

1. Go to GitHub → Releases → Create new release
2. Create a new tag (e.g., v0.1.0)
3. Add release notes
4. Publish release
5. GitHub Actions will automatically publish to PyPI

---

Publishing the Rust Extension

The Rust extension (if implemented) would be published separately because it requires platform-specific binaries.

Using maturin

cd rust/

# Build wheels for your current platform
maturin build --release

# The wheel would be in: ../target/wheels/flakestorm_rust-0.1.0-cp311-*.whl
# (or cp310, cp312 depending on Python version used)

Multi-Platform Publishing with GitHub Actions

# .github/workflows/rust-publish.yml
name: Publish Rust Extension

on:
  release:
    types: [published]

jobs:
  linux:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          manylinux: auto
          command: build
          args: --release --manifest-path rust/Cargo.toml -o dist
      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: wheels-linux
          path: dist

  macos:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          command: build
          args: --release --manifest-path rust/Cargo.toml -o dist
      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: wheels-macos
          path: dist

  windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          command: build
          args: --release --manifest-path rust/Cargo.toml -o dist
      - name: Upload wheels
        uses: actions/upload-artifact@v4
        with:
          name: wheels-windows
          path: dist

  publish:
    needs: [linux, macos, windows]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: dist
          merge-multiple: true
      - name: Publish to PyPI
        uses: PyO3/maturin-action@v1
        with:
          command: upload
          args: --skip-existing dist/*
        env:
          MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}

---

Version Management

Semantic Versioning

Follow Semantic Versioning:

MAJOR.MINOR.PATCH

0.1.0 - Initial release
0.1.1 - Bug fixes
0.2.0 - New features (backward compatible)
1.0.0 - Stable release / Breaking changes

Where Version is Defined

Update version in TWO places:

1. pyproject.toml:

   [project]
   version = "0.2.0"
   

2. src/flakestorm/__init__.py:

   __version__ = "0.2.0"
   

Automating Version Sync (Optional)

Use hatch-vcs to automatically get version from git tags:

# pyproject.toml
[build-system]
requires = ["hatchling", "hatch-vcs"]

[tool.hatch.version]
source = "vcs"

Then just create a git tag and the version is set automatically:

git tag v0.2.0
git push --tags

---

Testing Before Publishing

Local Testing

# Create a fresh virtual environment
python -m venv test_install
source test_install/bin/activate

# Install from local build
pip install dist/flakestorm-0.1.0-py3-none-any.whl

# Test it works
flakestorm --help
flakestorm init
python -c "from flakestorm import load_config; print('OK')"

Test PyPI

Always test on Test PyPI first:

# Upload to Test PyPI
twine upload --repository testpypi dist/*

# Install from Test PyPI
pip install --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/ \
            flakestorm

The --extra-index-url is needed because Test PyPI may not have all dependencies.

---

Common Issues

"Package name already taken"

Package names on PyPI are unique. If flakestorm is taken:

• Check https://pypi.org/project/flakestorm/
• Choose a different name: flakestorm-cli, py-flakestorm, etc.

"Invalid distribution file"

# Check what's wrong
twine check dist/*

# Common fixes:
# - Ensure README.md is valid markdown
# - Ensure LICENSE file exists
# - Ensure version is valid format

"Missing files in wheel"

# List wheel contents
unzip -l dist/*.whl

# If files are missing, check pyproject.toml:
[tool.hatch.build.targets.wheel]
packages = ["src/flakestorm"]  # Make sure path is correct

"Command not found after install"

Ensure project.scripts is set in pyproject.toml:

[project.scripts]
flakestorm = "flakestorm.cli.main:app"

---

Quick Reference

One-Time Setup

# Install tools
pip install build twine

# Create PyPI account and token
# Store token securely

Each Release

# 1. Update version in pyproject.toml and __init__.py
# 2. Commit and push
git add -A && git commit -m "Release 0.2.0" && git push

# 3. Build
python -m build

# 4. Check
twine check dist/*

# 5. Test (optional but recommended)
twine upload --repository testpypi dist/*
pip install --index-url https://test.pypi.org/simple/ flakestorm

# 6. Publish
twine upload dist/*

# 7. Tag release
git tag v0.2.0
git push --tags

With GitHub Actions

Just create a release on GitHub and everything happens automatically!

---

Next Steps After Publishing

1. Announce: Post on social media, Reddit, Hacker News
2. Documentation: Update docs with install instructions
3. Monitor: Watch for issues and PyPI download stats
4. Iterate: Fix bugs, add features, release new versions

---

Happy publishing! 🚀