Indexly Developer Guide

This guide is for contributors who want to ship reliable changes without breaking CLI stability.

Development Principles

Indexly is maintained with these priorities:

• Keep the core install lightweight and brew-friendly.
• Preserve CLI backward compatibility where possible.
• Fail gracefully when optional dependencies are missing.
• Prefer clear modules over tightly coupled logic.
• Keep changes testable and easy to review.

Local Setup

Clone and create a virtual environment:

git clone https://github.com/kimsgent/project-indexly.git
cd project-indexly
python -m venv .venv

Activate:

• macOS/Linux: source .venv/bin/activate
• Windows (PowerShell): .venv\Scripts\Activate.ps1

Install editable package:

python -m pip install --upgrade pip
python -m pip install -e .

Install optional packs for full feature development:

python -m pip install -e ".[documents,analysis,visualization,pdf_export,backup]"

Install dev tooling:

python -m pip install pytest pytest-cov flake8 black isort mypy build twine hatch

Quick verification:

python -m indexly --help
indexly --version

Windows Contributor Shortcut

On Windows, this repository also ships a repo-native setup script:

.\setup.ps1 -CheckOnly
.\setup.ps1

That script currently:

• validates winget, Python, and expected repo files
• applies system dependencies from winget.yaml
• creates or reuses .venv
• installs both requirements.txt and requirements-dev.txt

For platform install notes, see Install Indexly. For maintained workstation setup, see Windows Development Environment Setup and Linux Development Environment Setup.

Repository Structure

project-indexly/
├── pyproject.toml
├── README.md
├── README_PYPI.md
├── scripts/
│   └── generate_brew_formula.py
├── tests/
├── docs/
│   └── content/documentation/
└── src/indexly/
    ├── __main__.py
    ├── indexly.py
    ├── cli_utils.py
    ├── optional_deps.py
    ├── filetype_utils.py
    ├── db_utils.py
    ├── fts_core.py
    ├── backup/
    ├── compare/
    ├── inference/
    ├── observers/
    ├── organize/
    ├── perf/
    ├── visualization/
    └── assets/

Key Modules And Responsibilities

Dependency Policy (Important)

Indexly is designed for lightweight core installation and optional feature packs. When adding dependencies:

• Keep core dependencies minimal, pure Python where possible.
• Put heavy/compiled libraries in extras (documents, analysis, visualization, pdf_export).
• Never import optional dependencies at module import time for core paths.
• Use lazy imports and user-friendly install hints.

Use this pattern for optional imports:

try:
    import pandas as pd
except ModuleNotFoundError as exc:
    raise ModuleNotFoundError(
        "Feature requires optional dependency 'pandas'. "
        "Feature requires: pip install indexly[analysis]"
    ) from exc

How Commands Are Wired

Typical command flow:

1. src/indexly/__main__.py starts the CLI.
2. src/indexly/indexly.py builds/dispatches command handlers.
3. Handler calls the feature module (for example indexing, analysis, compare).
4. Output helpers print results and optional exports.

When adding a command:

1. Add parser arguments in cli_utils.py or command parser section.
2. Add handler logic in indexly.py (or a dedicated module).
3. Keep default behavior safe and non-destructive.
4. Add tests and update relevant docs page(s).

Common Extension Points

• New file type extraction: filetype_utils.py, extract_utils.py
• Search behavior: fts_core.py, search_core.py, delete_search.py
• CSV/data features: csv_analyzer.py, analysis_orchestrator.py, inference/
• Export and rendering: output_utils.py, export_utils.py, visualization/
• Ignore behavior: ignore/ and ignore_defaults/
• Backup behavior: backup/

Observer Internals

The observer system lives in src/indexly/observers/.

Core modules:

• base.py: observer contract (applies_to, extract, compare, format_event)
• registry.py: built-in registration, enable/disable controls, event handlers
• runner.py: execution order, dependency wiring, snapshot lifecycle, logging, metrics
• snapshot_store.py: latest generic observer snapshots in observer_snapshots
• csv/csv_snapshot_store.py: historical CSV snapshots in csv_snapshots
• aggregator.py: optional event aggregation for programmatic runs
• metrics.py: in-process observer execution metrics

Built-in observer names are identity, field, state, health_identity, health_fields, health_events, and csv.

The public CLI surface is intentionally small:

indexly observe --help
indexly observe run /path/to/file
indexly observe run /path/to/folder --recursive
indexly observe audit
indexly observe audit --id 20260201-patient-00001

Programmatic controls such as disable_observer(), enable_observer(), register_event_handler(), run_observers_batch(), event aggregation, and MetricsCollector.get_summary() are Python APIs rather than CLI flags.

When changing observer behavior, run:

python -m pytest tests/test_observers_config.py tests/test_health_event_observer.py tests/test_csv_snapshot_store.py tests/test_csv_observer.py tests/test_observer_runner.py
python -m indexly observe --help
python -m indexly observe run --help
python -m indexly observe audit --help

Clear Search Internals

The clear-search command is implemented in src/indexly/delete_search.py and wired through cli_utils.py. It is intentionally separate from fts_core.py because deletion has different safety requirements than indexing.

Responsibility Boundary

delete_search.py only operates on the FTS search database configured by config.DB_FILE, normally fts_index.db. It does not delete source files and does not modify the separate cleaned-data stats database used by analysis commands.

The deletion surface is limited to:

• file_index: FTS5 virtual table rows
• file_tags: tag rows for deleted paths
• file_metadata: structured metadata rows for deleted paths
• search_cache.json: cache entries referencing deleted paths

Control Flow

The high-level flow is:

1. Validate that exactly one criterion is supplied: path, tag, or all.
2. Resolve matching paths using normalized path, prefix, basename, or exact tag semantics.
3. Build a deletion plan with per-table counts and an operation ID.
4. Print the plan and, in CLI mode, request confirmation unless --yes is set.
5. Log SEARCH_DELETE_INITIATED before changing the database.
6. Delete rows inside one SQLite transaction.
7. Verify deleted counts against the plan.
8. Invalidate search cache entries on a best-effort basis.
9. Log completion events and print the final summary.

Safety Guarantees

Destructive behavior should stay conservative:

• Keep --dry-run read-only.
• Keep --yes as the only way to skip confirmation in non-dry-run CLI use.
• Keep path and tag deletion inside a transaction.
• Treat cache and logging failures as warnings after database success, not as rollback triggers.
• Preserve operation IDs in user output and logs for auditability.

Path And Tag Semantics

Path matching uses normalized strings from path_utils.normalize_path():

• exact path first
• directory-like prefix next
• basename fallback for legacy compatibility

Tag matching reads comma-separated values from both file_tags.tags and file_index.tag. Multiple tags are OR logic. Do not change this to AND logic without a CLI flag and migration note because existing help and tests document OR behavior.

Testing Requirements

When changing delete_search.py, update or run:

python -m pytest tests/test_delete_search.py -q
python -m pytest tests/test_search.py tests/test_tagging.py -q

Add tests for:

• confirmation and cancellation
• dry-run read-only behavior
• cache save failures
• database lock or corruption diagnostics
• transaction rollback when a table delete fails
• large batch progress output
• path normalization edge cases

Performance Internals

The performance command routes before normal Indexly initialization in both supported executable entry points. That boundary matters because ordinary configuration and database helpers may create runtime directories or initialize tables.

Responsibility Boundary

The src/indexly/perf/ package separates:

• model.py: versioned record and JSON contracts
• probe.py: bounded read-only SQLite, file-size, and indexing-log probes
• baseline.py: pure calculations and conservative status classification
• state.py: strict validation, atomic replacement, and previous-record recovery
• evidence.py: pure action-specific eligibility from validated records
• actions.py: live preconditions, verified backup, bounded SQLite actions, postconditions, and numeric audit
• cli.py: early command routing and text/JSON rendering
• __init__.py: the narrow status API consumed by Doctor

perf --show may write only the performance-record files. perf --read must not open SQLite or create a directory, temporary file, lock, cache, database, or record. A live --show rejects WAL mode before opening SQLite so the read-only contract cannot create or update a shared-memory sidecar. Plan-only perf --opti is non-mutating.

Doctor invokes only the performance package’s conservative status API. It never calls probe, baseline, state-write, CLI, evidence, or action behavior, calculates performance metrics, or invokes a maintenance action.

Optimization and Apply Boundary

evidence.plan_optimizations() does no I/O. It evaluates planner-optimize from SQLite 3.46+ dry-run evidence using mask 0x10013. If that debug-only pragma returns SQLITE_READONLY on an FTS database, the probe reports an Indexly-derived count of known relational tables whose indexes lack sqlite_stat1 coverage; it never inspects FTS shadow tables. It evaluates fts-merge only from three latest comparable FTS p95/generation observations with strictly advancing generations and two latest baseline-relative degradations. Status, database size, and generic capacity metrics never establish action eligibility.

Before either action-specific branch, evidence evaluates fts_schema_action_ready conservatively:

• absent legacy metric → collect_evidence
• present but non-measured, unavailable, or invalid evidence → unavailable
• measured 0 → repair_required
• measured 1 → continue with action-specific evidence

Only measured 0 establishes an inspected noncanonical, missing, drifted, external-content, malformed, unsupported, or uninspectable definition and routes the operator to Doctor. Unavailable readiness does not infer schema damage. The action layer repeats the structured match inspection under its writer reservation.

Applied actions require a validated primary report no older than 24 hours. actions.execute_action() then matches database identity, schema fingerprint, Indexly and SQLite versions, journal mode, page size, page count, main-file bytes, the observed big-endian SQLite header change counter at offset 24, document count, size bucket, and search index generation. It refuses WAL mode or a -wal sidecar, validates a free-space-qualified existing non-symlink backup directory distinct from the live database directory, acquires BEGIN IMMEDIATE, and creates a verified generic SQLite backup.

planner-optimize applies mask 0x10012; fts-merge issues one positive 500-page merge. Source, backup, and post-action verification use SQLite-side quick_check plus bounded logical invariants. The FTS action additionally runs FTS5 integrity-check against the completed backup and the post-action live database before commit. The CLI appends a numeric-only audit, retains at most 30 outcomes, refreshes the report, and returns a before/after/delta comparison. It never implements VACUUM, FTS rebuild, schema or tag migration, cache deletion, journal-mode changes, or automatic re-indexing.

The backup becomes reportable only after file fsync, atomic rename, and backup-directory fsync all succeed. A directory synchronization failure triggers candidate cleanup, raises a backup failure, and prevents action execution.

If candidate cleanup itself fails, ActionBackupError carries only the candidate path internally. The CLI emits indexly.performance-error/v1 with mutation_applied: false, backup_verified: false, cleanup_incomplete: true, and backup_filename; it never exposes the directory path. The operator must resolve that filename within the configured backup directory and remove it before retrying.

baseline.build_record() preserves action outcomes across a reset when the database identity is unchanged, including version, schema, journal, page-size, or size-bucket transitions. It clears outcomes only when identity changes. The CLI’s post-action comparability check includes size bucket, so a bucket transition retains the audit but fails the comparison rather than publishing misleading deltas.

Failure output preserves transaction state. Preflight failures and rolled-back actions exit 2. A rollback with a retained verified backup uses indexly.performance-action/v1, sets mutation_applied: false and rolled_back: true, and exposes only the backup filename. If the action applies a mutation but audit persistence or the post-action report fails, the CLI sets mutation_applied: true and exits 3; failed follow-up after a no_op remains exit 2. Callers must not treat exit 3 as safe to retry blindly.

Evidence and Privacy Rules

Every reported metric is labelled Observed, Indexly-derived, or Theoretical. Theoretical formulas are descriptive and never authorize a database action. Timeouts report an unavailable measurement instead of expanding into an unbounded scan.

The record may contain bounded numeric samples, calculation context, a salted non-reversible database identity, and a reserved bounded action-outcomes list. It must not contain paths, filenames, indexed content, metadata JSON, query terms, usernames, hostnames, raw log records, or network telemetry.

Testing Requirements

Performance changes require focused coverage for:

• both early executable routes and parser/help consistency
• no-write behavior for --read and database-read-only behavior for --show
• bounded probe and global deadlines
• formulas, units, comparable-baseline rules, and status transitions
• record checksum/schema rejection, atomic replacement, and previous-copy recovery
• absence of sensitive values in text, JSON, and local records
• Doctor’s status-only consumption and inability to invoke an optimizer
• action-specific planner and FTS eligibility, including unavailable evidence
• legacy/unavailable/measured-zero FTS readiness disposition boundaries
• exact stale/identity/schema/journal/page/size/generation precondition failures
• exact SQLite database-header change-counter binding
• WAL/header/sidecar refusal without checkpoint or journal-mode change
• backup, free-space, writer-lock, rollback, audit-retention, and post-action comparison behavior
• backup-directory synchronization failure before action execution
• sanitized cleanup-incomplete reporting for an unverified backup candidate
• same-identity audit retention and size-bucket postcheck rejection
• backup and live FTS5 integrity verification for fts-merge
• exit 2 rollback versus exit 3 committed partial-success output

See Performance Diagnostics and Optimization for the public contract.

Doctor Internals

The indexly doctor command is implemented in src/indexly/doctor.py. It is a health and maintenance orchestration layer, not a replacement for search, indexing, analysis, or migration modules.

Responsibility Boundary

Plain indexly doctor must stay read-only. It may inspect:

• runtime paths from config.py
• search database health for fts_index.db or an explicit --db
• analysis persistence at ~/.indexly/indexly.db
• search_cache.json
• optional dependency availability
• external tools such as ExifTool and Tesseract
• conservative performance status from the validated local record

State-changing actions require explicit flags:

• --clear-cache writes {} to the search cache file
• --fix-db applies schema migrations after preflight checks and confirmation unless --auto-fix is used
• --rebuild-fts allows FTS5 virtual table rebuilds during repair

--full-integrity is intentionally read-only. It enables SQLite PRAGMA integrity_check for inspected databases and should not imply repair.

Doctor must not run performance probes, calculate baselines, display detailed performance metrics, or invoke perf --opti. Slow performance evidence does not establish corruption.

Command Wiring

Doctor flags are declared in cli_utils.py and forwarded by handle_doctor() in indexly.py. When adding or renaming a Doctor flag:

1. update the parser in cli_utils.py
2. forward the value in indexly.py
3. include the flag in show-help --details if it is high-signal
4. update docs/content/documentation/indexly-doctor.md
5. add or update tests/test_doctor.py

FTS5 Safety Rule

Do not silently rebuild FTS5 virtual tables. The repair layer uses one structured semantic definition for the FTS5 module, ordered columns and optional UNINDEXED markers, tokenizer, prefix sequence, and supported options. uninspectable definitions are never treated as matching and are never automatically rebuilt.

db_update.py skips FTS5 rebuilds unless allow_fts_rebuild=True, which is exposed through:

indexly doctor --fix-db --rebuild-fts

The common rebuild engine must keep full integrity, definition, writable-path, free-space, verified SQLite snapshot, and lock checks ahead of mutation. It must keep data movement in SQLite, bound client memory, validate logical-row preservation and FTS behavior before swap, increment search_index_generation in the same transaction, and roll back transfer or verification failures. It must not run VACUUM.

If post-commit reopening or verification fails, return the verified snapshot path and recovery guidance rather than success.

Testing Requirements

Run the focused Doctor suite after changes:

python -m pytest tests/test_doctor.py tests/test_search.py::test_search_cli_defaults_to_runtime_db_unless_db_is_explicit

For path deletion or cache-adjacent changes, also run:

python -m pytest tests/test_delete_search.py

On Windows, use an explicit writable --basetemp when local ACLs make default pytest temp folders unreadable.

Testing And Quality Checks

Run fast checks during development:

pytest -q
flake8 src tests
black --check src tests
isort --check-only src tests
mypy src/indexly

Smoke-test critical commands after larger changes:

indexly --help
indexly show-help
indexly doctor

If you modify indexing, analysis, compare, backup, or migration behavior, run targeted command tests in a local sandbox folder.

Build, Package, And Brew Formula

Build package artifacts:

python -m build
twine check dist/*

Generate Homebrew formula:

python scripts/generate_brew_formula.py --out Formula/indexly.rb

Dry-run formula generation with local source artifact:

python scripts/generate_brew_formula.py --dry-run --source dist/indexly-<version>.tar.gz --out Formula/indexly.rb

Brew-oriented review checklist:

• Formula uses virtualenv_install_with_resources.
• Dependency resource list stays small and stable.
• No heavy scientific stack in core runtime dependencies.
• CLI starts correctly with only core dependencies installed.

Documentation Responsibilities

When behavior changes, update docs in the same PR:

• User-facing install/usage: README.md, README_PYPI.md
• Website docs: docs/content/documentation/
• Packaging behavior: scripts/generate_brew_formula.py docs and examples

Keep examples copy-paste ready and aligned with indexly --help.

When you change AutoDoctor-related analysis behavior, update both sides of the documentation boundary:

• Indexly-side operational usage: docs/content/documentation/analyze-autodoctor-artifacts.md
• AutoDoctor-side artifact meaning: docs/content/documentation/autodoctor/

This keeps “how to analyze the artifact” separate from “what the artifact means inside AutoDoctor,” which mirrors the current code separation.

Contribution Workflow

1. Create a feature branch from latest main.
2. Keep commits focused and descriptive.
3. Run quality checks locally.
4. Include a risk note in your PR for production-sensitive changes.
5. Document any compatibility impact (especially brew/package/install changes).

See Contributing for collaboration details.