Atlas - semantic knowledge and retrieval

[maximus]

Hi everyone,

I’m releasing Atlas, a lightweight semantic knowledge and retrieval module for ProcessWire.

Atlas converts selected ProcessWire content into a searchable knowledge base. It creates embeddings through Squad, stores vectors, text and metadata in MySQL/MariaDB, and retrieves relevant entries using cosine similarity.

The module is intended for projects where semantic search and grounded AI features should remain close to ProcessWire without requiring a separate vector database such as Qdrant, Pinecone or pgvector.

What Atlas can be used for

Some possible applications:

• semantic website search;
• FAQ and support retrieval;
• related-content recommendations;
• editorial knowledge bases;
• contextual information for AI assistants;
• retrieval-augmented generation;
• searching content by meaning rather than exact keywords.

Atlas is designed for small and medium collections—roughly up to 10,000 entries. Larger datasets will usually benefit from a dedicated vector database.

Main features

• Independent named collections.
• Provider-independent embeddings through Squad.
• Semantic search using cosine similarity.
• Metadata filtering.
• Literal keyword boosting.
• Optional required-keyword matching.
• Maximal Marginal Relevance for more diverse results.
• Paragraph- and sentence-aware document chunking.
• Atomic batch writes and document replacement.
• Atomic full-site reindexing through a staging collection.
• Incremental reindexing that reuses unchanged vectors.
• Model and vector-dimension consistency checks.
• Deterministic references and chunk-aware deletion.
• ProcessWire hooks for custom indexing and external integrations.

Atlas stores everything in a single MySQL/MariaDB table. No additional database service is required.

ProcessWire content indexing

Atlas includes a domain-neutral ProcessWire page adapter.

It can extract labelled content from common ProcessWire field types, including:

• Text and Textarea fields;
• numeric and date fields;
• Checkbox and Toggle fields;
• Options fields;
• Page references;
• Images and Files;
• Repeaters and PageTable;
• ProFields Table;
• ProFields Combo;
• ProFields Textareas;
• ProFields Multiplier;
• Repeater Matrix.

Nested values are stored as readable, structured text instead of serialized field data. Image and file values are converted to absolute URLs, while internal file metadata and presentation-only fields are excluded.

For example:

Agent:
  Bio: Local real-estate agent specialising in residential property.
  Photo:
    https://example.com/site/assets/files/123/portrait.jpg
  Role: Realtor
  Languages: English, German
  Phone: +1 555 0100
  Email: agent@example.com

The default adapter is intentionally general-purpose. Project-specific field handling can be added through hooks without modifying Atlas itself.

Admin interface

The optional ProcessAtlas companion module adds Setup → Atlas to the ProcessWire admin.

The workspace includes:

• Overview — collection totals, provider readiness and index status;
• Collections — collection statistics and management;
• Entries — structured content and source metadata;
• Search — semantic, hybrid and MMR retrieval;
• Reindex — safe batched indexing of public ProcessWire pages.

A full-site rebuild is created in an internal staging collection. The existing live collection remains available until every batch succeeds. If embedding fails, the previous collection is preserved.

Reindexing is resumable, displays real progress and supports cancellation.

Basic API

$atlas = $modules->get('Atlas');

$atlas->add(
    'faq',
    'password-reset',
    'Use the password reset form on the account page.',
    [
        'title' => 'Password reset',
        'url' => '/account/reset/',
        'source' => 'website',
    ]
);

$hits = $atlas->search(
    'faq',
    'I forgot my password',
    5
);

Hybrid retrieval and metadata filtering are available through search options:

$hits = $atlas->search('articles', $query, 5, [
    'filter' => [
        'language' => 'en',
        'source' => 'website',
    ],
    'keyword' => $literalTerm,
    'mmr' => true,
    'mmrLambda' => 0.7,
]);

Long documents can be split and stored atomically:

$atlas->addChunked(
    'articles',
    'page-' . $page->id,
    $page->title . "\n\n" . $page->body,
    [
        'page_id' => $page->id,
        'title' => $page->title,
        'url' => $page->url,
    ]
);

Integration hooks

Other modules can customise individual field projections or complete page documents:

• ProcessAtlas::buildPageFieldText
• ProcessAtlas::buildPageDocument
• ProcessAtlas::reindexReady
• ProcessAtlas::pageIndexed
• ProcessAtlas::collectionReindexed

External applications can contribute documents to the same atomic reindex before the staging collection is promoted.

This makes it possible to integrate Atlas with project-specific modules without patching its core.

What Atlas is not

Atlas is not intended to replace:

• ProcessWire selectors for exact structured queries;
• normal access-control checks;
• the original ProcessWire page content;
• a large-scale vector database;
• an AI model that generates final responses.

Search results should be treated as candidate source material. Applications must still verify current page status and user access before displaying retrieved content or sending it to another model.

Part of a wider series

Atlas is one part of a broader group of ProcessWire modules I have been working on—covering provider-independent AI, semantic retrieval, content intelligence, website generation and operational workflows.

Each module remains independent, but they can work together when a project needs more than one capability.

I’ll introduce the other modules separately throughout this month.

Requirements

• ProcessWire 3.0.210 or newer;
• PHP 8.1 or newer;
• MySQL or MariaDB with InnoDB and utf8mb4;
• Squad with an embedding-capable provider configured.

Installation

1. Install and configure Squad.
2. Copy the Atlas directory to /site/modules/Atlas/.
3. Refresh modules in the ProcessWire admin.
4. Install Atlas.
5. Optionally install ProcessAtlas for the admin workspace.
6. Configure an embedding provider before indexing or searching.

Download and documentation

• GitHub: github.com/mxmsmnv/Atlas

The repository includes installation instructions, full API documentation, integration examples, an AI-agent integration guide and dependency-free smoke tests.

Atlas 1.0.0 is the first public release. Feedback, testing and real-world use cases are very welcome.