moonboard.ioResearch & Data
Data

From source to score

The logical architecture: eight stages between an upstream API and a number on a dashboard, and what each stage can and cannot repair.

Logical, not operational

This describes what happens to the data and in what order. It deliberately omits infrastructure detail — hostnames, schedules, credentials, internal service topology — which would be a security liability and is of no use in interpreting a score.

Stage 1 — Sources

  • Market aggregator
  • Social analytics
  • News feeds
  • Code hosting
  • Community platforms

Stage 2 — Ingestion

  • Scheduled collection
  • Rate-limit budgeting
  • Retry on failure
  • Raw capture

Stage 3 — Normalisation

  • Unit alignment
  • Symbol resolution
  • Type coercion
  • Range validation

Stage 4 — Deduplication

  • News matching
  • Asset identity
  • Observation collapse

Stage 5 — Enrichment

  • Derived ratios
  • Sector tagging
  • Window aggregation

Stage 6 — Scoring

  • Cross-section build
  • Percentiles
  • Z-scores
  • Composites

Stage 7 — Storage

  • Current snapshot
  • Historical series

Stage 8 — Delivery

  • Internal API
  • Caching

Consumption

  • moonboard.ai
Processing order. Each stage depends only on the output of the one above it.

Stage 2 — Ingestion

Collection runs on fixed schedules sized to each provider's rate allowance rather than on demand, so all users are served from the same snapshot between cycles. Responses are captured before any transformation, which means a parsing change can be re-applied to the raw payload without re-fetching.

On a failed fetch the previous successful observation is retained rather than writing a gap. This keeps every downstream metric computable and has the consequence described in missing data: a stalled source looks like an unchanging asset.

Stage 3 — Normalisation

Where most real-world data problems get resolved — or become permanent.

  • Unit alignment. Sources disagree on conventions: volatility as fraction or percent, dominance as fraction or percent, timestamps in several formats. Each field is coerced to one documented convention, listed in data categories. Nearly every reproduction error traces back to this step.
  • Symbol resolution. The same asset carries different identifiers across providers, and the same ticker is reused by unrelated assets. Records are keyed on an internal identifier rather than a symbol. Resolution failures are the main cause of an asset appearing with partial data — market fields present, social fields absent.
  • Range validation. Values outside plausible bounds — a negative market cap, a dominance above 100 — are rejected rather than clamped, because clamping would silently manufacture a plausible-looking number from a broken one.

Stage 4 — Deduplication

News is collapsed by title and content similarity, since one story typically appears across many outlets within minutes. Multiple observations of the same asset in a cycle collapse to the most recent.

Social posts are not deduplicated here. Whatever deduplication exists happens upstream, so identical promotional posts can be counted more than once — a mechanism relevant to synthetic activity.

Stage 5 — Enrichment

Derived quantities are computed once and stored, rather than recomputed at every read: turnover ratios, engagement rate, the social-to-market ratio, and sector tags.

Sector tags deserve a caveat. They are assignments, not observations, and assets legitimately belong to several categories. Every sector-level view on the platform inherits the tagging decision — see narratives.

Stage 6 — Scoring

The stage where a per-asset record stops being independent of the others.

Four of the seven metrics are percentile ranks and one uses z-scores, so all five require the full cross-section to be assembled first. Nothing here can be computed for a single asset in isolation: the universe has to be built, non-finite values excluded, and only then can any asset be placed within it.

for each metric: build cross-section over all valid assets drop non-finite values compute percentile or (mean, std) assign each asset its position then compute composites (OPS, COR, WSD)

This ordering is why the ranked metrics are universe-dependent, and why reconstructing a historical score requires reconstructing the historical universe — the obstacle described in research notes.

Stage 7 — Storage

Two shapes are kept: the current snapshot, read by nearly every request, and a historical series of observations that supports the platform's time-series views.

Historical records preserve the fields as collected. Scores stored alongside them were computed against the universe as it stood at that moment, which is the correct behaviour and also means a stored historical score cannot be recomputed from the stored fields alone.

Stage 8 — Delivery

Serving is cache-first: the application reads a prepared snapshot rather than triggering collection. Reloading a page does not fetch newer upstream data, and a value that has not changed between two loads may simply be within the same refresh cycle.

What the pipeline cannot fix

Processing improves consistency. It does not create information that was never collected:

  • Coverage gaps stay gaps. A community on an uncollected platform is absent at every stage.
  • Unverified inputs stay unverified. Wash-traded volume and coordinated posting are normalised and ranked exactly like genuine activity.
  • Upstream latency is inherited. No stage makes a figure more current than the source made it.
  • Classification errors propagate. A sentiment misread at the source is standardised and ranked with full confidence downstream.

Related