Ingestion Layer

EdgarClient queries the EDGAR Full-Text Search (EFTS) API and Submissions API with async HTTP, rate-limited to SEC guidelines. Results are filtered by form type (10-K, 10-Q) and date range.

FilingParser performs section-aware HTML parsing using BeautifulSoup. It identifies Items (1, 1A, 7, 7A, 8) via regex patterns against a FilingSection enum, applies TOC deduplication (keeping the longest match per item), and calls chunk_text() to split sections into overlapping chunks (1500 chars, 200 overlap) for embedding.

EDGAR EFTS API ──▶ EdgarClient.search_filings()
                        │
                        ▼
              EdgarClient.fetch_filing_document()
                        │
                        ▼
              FilingParser.parse()
              ├── extract_sections()  [regex Item matching]
              ├── deduplicate_toc()   [keep longest per Item]
              └── chunk_text()        [1500 char chunks, 200 overlap]
                        │
                        ▼
              Snowflake RAW tables
              ├── RAW_COMPANIES
              ├── RAW_FILINGS
              └── RAW_FILING_SECTIONS

Processing Layer

AI_PARSE_DOCUMENT (LAYOUT mode) extracts structured text and embedded images from filing PDFs. Images are stored in FILING_IMAGES and analyzed via CORTEX.AI_EXTRACT to produce chart metadata in CHART_EXTRACTS.

Dual Embedding Pipeline

SecSignal uses two embedding models in the same 1024-dimensional space:

• Arctic embed (text-to-text) — via EMBED_TEXT_1024('snowflake-arctic-embed-l-v2.0'). Populates TEXT_EMBEDDINGS and SEARCH_CORPUS for Cortex Search.
• Voyage multimodal-3 (cross-modal) — via EMBED_TEXT_1024('voyage-multimodal-3'). Encodes chart image descriptions into IMAGE_EMBEDDINGS for cross-modal retrieval.

Text chunks ──▶ Arctic embed (1024d) ──▶ TEXT_EMBEDDINGS
                                      └──▶ SEARCH_CORPUS (Cortex Search)

Image descriptions ──▶ Voyage multimodal-3 (1024d) ──▶ IMAGE_EMBEDDINGS

dbt Transformation Pipeline

A 12-model dbt project transforms raw data through three layers:

STAGING (stg_*)                INTERMEDIATE (int_*)           MARTS (fct_*, dim_*)
─────────────                  ────────────────               ──────────────────
stg_companies          ──▶     int_parsed_sections    ──▶     fct_risk_factors
stg_filings            ──▶     int_chart_extracts     ──▶     fct_financials
stg_sections                                          ──▶     fct_chart_data
stg_filing_images                                     ──▶     dim_filing_images
stg_chart_extracts

LangGraph Agent System

The agent system is a LangGraph StateGraph with a FilingState TypedDict. List fields use Annotated[list, operator.add] so multiple nodes can contribute items that auto-merge.

                    ┌─────────────────┐
                    │  classify_query  │
                    │  (Supervisor)    │
                    └────────┬────────┘
                             │ conditional routing
               ┌─────────────┼─────────────┐
               ▼             ▼             ▼
     ┌──────────────┐ ┌───────────┐ ┌───────────────┐
     │ trend_agent  │ │ comparison│ │ anomaly_agent │
     │              │ │ _agent    │ │               │
     └──────┬───────┘ └─────┬─────┘ └──────┬────────┘
            │               │              │
            └───────────────┼──────────────┘
                            ▼
                  ┌──────────────────┐
                  │ web_search_agent │
                  │ (Cortex Agent +  │
                  │  AGENT_RUN)      │
                  └────────┬─────────┘
                           ▼
                  ┌──────────────────┐
                  │   synthesizer    │
                  │  (Cortex        │
                  │   COMPLETE)      │
                  └──────────────────┘

  State fields (operator.add annotation):
  ├── sources: list[dict]         # filing citations
  ├── charts: list[dict]          # retrieved chart data
  ├── anomaly_scores: list[dict]  # z-score anomalies
  ├── generated_charts: list[dict]# LLM-extracted charts
  ├── web_sources: list[dict]     # web search citations
  └── trajectory: list[str]       # step-by-step trace

API Layer

FastAPI serves two endpoints: POST /api/query for synchronous responses and POST /api/query/stream for Server-Sent Events (SSE). The streaming endpoint uses an asyncio.Queuebridge — run_query_stream() yields trajectory steps and the final result into the queue, while FastAPI's StreamingResponse drains it to the client.

Frontend

Next.js 16 App Router with a single-page chat interface. The useQueryStream hook connects to the SSE endpoint via EventSource, parsing trajectory and result events into React state. Charts are rendered with recharts (BarChart, LineChart, ResponsiveContainer), and markdown answers use react-markdown with remark-gfm for pipe table support. The UI is styled with shadcn/ui + Tailwind 4 using oklch colors (warm off-black background, muted teal primary).

Airflow Orchestration

A weekly Airflow DAG orchestrates the end-to-end pipeline:

fetch_metadata ──▶ process_filings ──┬──▶ process_images ──▶ embed_images
                                     │
                                     └──▶ run_dbt ──────────▶ embed_text

Schedule: Weekly (Sunday 2:00 AM UTC)
Fan-out: process_filings triggers both image and dbt branches in parallel
Fan-in: embed_images and embed_text run after their respective upstream tasks

1. Agentic RAG (Supervisor → Specialist → Synthesizer)

Rather than a monolithic RAG pipeline, SecSignal uses a multi-agent architecture. A supervisor node classifies the query type, then conditionally routes to a specialist agent (trend, comparison, or anomaly). Each specialist retrieves different data and produces structured artifacts. A final synthesizer merges everything into a coherent answer. This separation of concerns allows each agent to be optimized independently.

2. State Accumulation via operator.add

LangGraph's Annotated[list, operator.add] annotation on state fields (sources, charts, trajectory, etc.) enables multiple nodes to append items without overwriting each other. When a specialist adds 3 sources and the web agent adds 2 more, the synthesizer sees all 5 automatically.

3. Dual Embedding Strategy

Text-to-text retrieval uses Arctic embed for precise semantic search over filing sections. Cross-modal retrieval uses Voyage multimodal-3 to match text queries against chart image descriptions. Both produce 1024-dimensional vectors, enabling unified similarity searches across modalities.

4. SSE Bridge Pattern (asyncio.Queue)

The LangGraph stream generator yields trajectory steps synchronously. An asyncio.Queuebridges this into FastAPI's async world — a background thread runs the generator and enqueues events, while the SSE response handler dequeues them. This prevents blocking the event loop during long Cortex LLM calls.

5. TTL Dictionary Cache

Web search and chart extraction results are cached in module-level dictionaries with a 5-minute TTL. Cache keys combine the normalized query and sorted ticker list. This avoids redundant Cortex AGENT_RUN and COMPLETE calls for repeated or similar queries within a short window.

6. Section-Aware Parsing with TOC Deduplication

SEC filings often have a Table of Contents that duplicates section headers. The parser uses regex to match Item headings, then deduplicates by keeping only the longest text match per Item. This ensures embedding and search indices contain the full section content rather than TOC stubs.

7. Conditional Routing via LangGraph Edges

add_conditional_edges dispatches to the correct specialist based on the classified query type. The routing function returns a node name string, keeping the graph definition declarative and the routing logic testable in isolation.

8. Fan-Out / Fan-In Orchestration (Airflow)

The weekly DAG fans out after filing processing: one branch extracts and embeds images, the other runs dbt transforms and embeds text. Both branches converge implicitly at DAG completion. This parallelizes the two heaviest operations (image processing and dbt) for faster pipeline throughput.