Why Ticks Are Irregular — and Why the Gaps Are Signal

A tick — a trade print or a quote update — happens when a market participant decides to act. Nobody coordinates those decisions with a clock. A liquid US equity might print thousands of trades in the minute after the open, then go quiet for seconds at a time mid-day; an illiquid small-cap or a long-tail crypto pair might not trade for minutes. The result is a sequence of events at timestamps t₁ < t₂ < t₃ ... whose inter-arrival times (durations) vary by orders of magnitude across the trading day.

Almost every classical time-series tool — ARIMA, GARCH, standard LSTMs trained on fixed windows — assumes observations arrive on a uniform grid. So the first instinct is to force tick data onto a grid and move on. The reason this deserves more care than a resample() call is that the spacing itself is informative in two distinct ways:

• Durations cluster. Short gaps follow short gaps: bursts of trading beget more trading. Engle and Russell (1998) documented this duration clustering in NYSE transaction data and built an entire model class (ACD, covered below) around the analogy with volatility clustering in GARCH. Throwing away durations discards a predictable, economically meaningful series.
• Markets run on event time, not clock time. The old "time deformation" literature — Clark (1973) is the classic reference — models prices as a Brownian motion subordinated to a market-activity clock. Sample returns by units of trading activity instead of wall-clock seconds and they come out much closer to the IID-Gaussian world your models implicitly assume.

Everything that follows is one of two strategies: (1) resample onto a clock chosen so that standard tools work well (bars, the pragmatic majority choice), or (2) keep the irregular event sequence and use a model that understands it (point processes and continuous-time neural networks).

The Two Failure Modes of Naive Regularization

Before the good options, the two ways teams routinely get this wrong.

Failure 1: Linear interpolation (a look-ahead bug wearing a statistics costume)

Linear interpolation fills the grid point at time t using the ticks on both sides of t. The tick after t had not happened yet. Any feature computed from linearly interpolated prices leaks future information into your training data — a textbook look-ahead bias that inflates backtests and evaporates live. (Our piece on why most backtests fail covers this family of bugs in depth.)

Interpolation also breaks volatility estimation for a subtler reason: a linearly interpolated path is smooth (of bounded variation), so its realized variance shrinks toward zero as you sample it more finely — a degenerate side effect noted in the realized-variance literature, e.g. Hansen and Lunde's work on realized variance and microstructure noise. The standard convention in that literature is previous-tick sampling: at each grid point, take the last observed price at or before that time. It is causal, simple, and what "5-minute realized volatility" means in practice.

Failure 2: Forward-filling onto a too-fine grid

Previous-tick sampling is the right primitive, but pushed to a grid much finer than the actual trading frequency it manufactures artifacts: long runs of identical prices produce zero returns that deflate volatility per-observation, induce spurious autocorrelation, and inflate your sample size without adding information. A model trained on millisecond grids of a stock that trades every few seconds is mostly learning the shape of your padding. There is also a bias–variance trade-off from microstructure noise (bid-ask bounce, price discreteness): sampling too finely makes realized-variance estimates worse, which is why the empirical literature converged on conventions like 5-minute sampling with previous-tick prices, or noise-robust estimators at higher frequencies. If bid-ask bounce is polluting your features, see our companion article on filtering bid-ask bounce with signal decomposition.

Rules of thumb: never interpolate; previous-tick only. Choose grid resolution near the asset's median inter-trade duration or coarser. And carry n_ticks and volume per window as features so the model can tell a stale window from an active one.

Bar Construction: Time, Tick, Volume, and Dollar Bars

The mainstream answer to irregular spacing is to aggregate ticks into bars — OHLCV summaries of consecutive chunks of activity. The question is what defines a chunk. López de Prado's Advances in Financial Machine Learning (2018) popularized the taxonomy most practitioners now use:

The argument for activity-based bars (tick/volume/dollar) is the time-deformation logic from above: by sampling in event time, each bar contains a roughly comparable amount of "market information," so bar-to-bar returns are closer to identically distributed. López de Prado argues — building on Clark's subordination result — that returns sampled this way exhibit lower serial correlation and heteroskedasticity and are closer to Gaussian than time-bar returns, which matters because most ML methods quietly assume something like IID inputs. Open-source implementations exist if you'd rather not roll your own: Hudson & Thames' mlfinlab data structures module implements standard and information-driven bars from AFML.

Choosing thresholds: a practical convention is to target a bar count, not a threshold. Decide you want roughly 50 bars per day, then set the dollar threshold to (average daily dollar volume ÷ 50), recalibrated monthly or quarterly. Dollar bars need this least often — that robustness to price drift is their main selling point.

When time bars are still right: activity bars are not a free lunch. Bars close at different wall-clock times for different assets, which complicates multi-asset alignment (next section), calendar features (time-of-day seasonality), and anything that must sync with external events. Realized-volatility estimation conventions are also defined in calendar time. Many production stacks keep both: dollar bars for model features, time bars for risk and reporting.

Building Bars in pandas and polars

Assume a tick table with columns ts (timestamp), price, and size — the shape you get from most vendors after normalization (see our guide to LOBSTER and other HFT datasets for the raw formats).

Time bars with previous-tick closes (pandas)

Keeping n_ticks matters: a 1-second bar built from 400 trades and one built from a forward-filled stale price are very different observations, and the model can only tell them apart if you say so.

Tick, volume, and dollar bars (pandas + numpy, vectorized)

One honest caveat about the cumsum // threshold trick: the tick that crosses a threshold is assigned entirely to one bar rather than split, so bar sizes wobble slightly around the target. Exact implementations (like mlfinlab's) iterate and carry remainders. For feature engineering the fast approximation is almost always fine; for accounting it is not.

The same in polars

On serious tick volumes (hundreds of millions of rows), polars treats resampling as a group-by and is typically the better tool — lazy scanning means you never hold raw ticks in memory:

Note that unlike pandas, polars does not emit rows for empty windows — if you need a dense grid, chain .upsample() and forward-fill the close explicitly.

Multi-Asset Alignment: The Epps Effect and Refresh Time

Everything above was one asset. Correlations and covariance matrices introduce a problem that catches nearly everyone the first time: estimate the correlation between two assets from finely sampled returns and it collapses toward zero as the sampling interval shrinks. Epps documented this in 1979 for stock returns, and the Epps effect is largely an artifact of asynchronous trading: at fine resolutions, asset A's latest tick and asset B's latest tick almost never coincide, so each grid return on A is being compared against a stale price on B, smearing genuine co-movement across neighboring intervals.

Two standard remedies:

• Refresh-time sampling. Instead of a fixed clock, sample every time all assets in the basket have traded at least once since the last sample. Introduced as the synchronization scheme for multivariate realized kernels by Barndorff-Nielsen, Hansen, Lunde and Shephard (2011), it produces a common, irregular grid on which every return is computed from genuinely fresh prices. The cost: the slowest-trading asset sets the pace for the whole basket.
• The Hayashi–Yoshida estimator. Hayashi and Yoshida (2005) sidestep synchronization entirely: sum the products of all pairs of tick-to-tick returns whose time intervals overlap. No grid, no interpolation, and unbiased under their assumptions — though microstructure noise still requires care in practice.

Refresh time is short enough to show in full:

Then previous-tick sample each asset at these times and compute covariance on the synchronized returns. If you are building correlation features for an ML model rather than a formal covariance estimator, the same logic applies in spirit: compute cross-asset features on refresh-time or coarse grids, not on millisecond forward-filled ones.

Models That Consume Irregular Ticks Directly

Bars deliberately compress away the durations. When the durations are the signal — forecasting trade intensity, modeling order-flow reactions, anticipating liquidity droughts — you want models built for marked point processes.

ACD: GARCH for waiting times

The autoregressive conditional duration model of Engle and Russell (1998, Econometrica) treats the duration xᵢ between events as xᵢ = ψᵢ·εᵢ, where the expected duration follows a GARCH-like recursion ψᵢ = ω + α·xᵢ₋₁ + β·ψᵢ₋₁ and εᵢ is a positive IID innovation (exponential or Weibull in the original paper). It captures duration clustering the same way GARCH captures volatility clustering, and after fitting deterministic time-of-day effects it gives a clean conditional forecast of "how long until the next trade" — directly useful for liquidity timing and as an input to execution schedulers. The model family has spawned dozens of extensions (log-ACD, stochastic conditional duration); Cavaliere, Mikosch, Rahbek and Vilandt's treatment of the econometrics of financial duration modeling is a good map of the modern territory.

Hawkes processes: self-exciting arrivals

Where ACD models the gaps, a Hawkes process models the arrival intensity: λ(t) = μ + Σ φ(t − tᵢ), summing a decaying kernel φ (often αe^⁻ᵖᵗ) over past events, so every trade temporarily raises the probability of more trades. Multivariate versions let buy trades excite sell quotes, cancellations excite price moves, and so on — a natural language for order-book dynamics. Bacry, Mastromatteo and Muzy (2015) is the standard review of Hawkes processes in finance, covering applications from market impact to order-flow modeling. For implementation, the tick library provides fast parametric and non-parametric Hawkes estimation in Python; neural variants (e.g. the Transformer Hawkes Process) replace the parametric kernel with a learned one.

Continuous-time neural networks

Deep learning's answer to irregular sampling is to make the hidden state evolve in continuous time:

• GRU-D (Che et al., 2018) augments a GRU with learned exponential decay of the hidden state and inputs toward empirical means as the gap since the last observation grows, taking (value, mask, elapsed-time) triplets as input.
• ODE-RNNs and latent ODEs (Rubanova, Chen and Duvenaud, 2019) define the hidden dynamics by a neural ODE integrated across each inter-event gap, updating the state only when an observation arrives — arbitrary spacing handled by construction.
• Time-aware attention: transformers handle irregularity by encoding actual timestamps or inter-arrival gaps (rather than integer positions) in the positional encoding, an approach that scales better than ODE solvers on long sequences.

An honest caveat before reaching for any of these: this corner of the literature is benchmark-driven and the benchmarks are mostly medical, not financial. Recent re-evaluation work has found that well-tuned classical recurrent models with simple time features remain competitive with ODE-based architectures on irregular time-series tasks, at a fraction of the training cost. In practice the highest-value-per-line-of-code option is unglamorous: keep your standard sequence model and append Δt since the last event, event counts, and rolling intensity as input features. Graduate to ODE machinery only if that baseline measurably saturates. (If you go down the deep-sequence-model road for finance, our Temporal Fusion Transformer walkthrough covers the adjacent design choices.)

Recommendations by Use Case

That last row deserves a sentence: whatever representation your model consumes, your backtest should replay events in native event time, because fill prices and queue positions depend on the within-bar sequence that aggregation destroys. Our article on replay frameworks for HFT strategy testing covers that side of the problem.

Bottom Line

Irregular spacing in tick data is not a defect to be cleaned away — it is the market's activity clock showing through. The practical hierarchy: aggregate into dollar or volume bars for most ML work; use previous-tick sampling (never interpolation) when calendar time is unavoidable; synchronize multiple assets with refresh time rather than fine grids; and reach for ACD, Hawkes, or continuous-time neural models only when the arrival process itself is what you are trying to predict. Each step up that ladder buys fidelity at the cost of complexity — and as usual in this field, the boring middle rung, honestly implemented, beats the exotic top rung with a look-ahead bug in it.

Related reading: Regime-Switching Models with Hidden Markov Chains for Volatility Clustering • Bid-Ask Bounce Filtering with Signal Decomposition • Creating Synthetic Tick Data to Augment Sparse Crypto Pairs