If you do any kind of high-frequency trading research, market microstructure analysis, or machine learning on order book data, you've almost certainly encountered the LOBSTER dataset. It's the de facto standard for academic work on NASDAQ limit order books, and for good reason: it reconstructs the full order book from raw exchange messages, saving researchers months of preprocessing work.

But choosing the right HFT dataset is more nuanced than picking the most-cited option. Your choice depends on which exchange you need, how deep into the order book you need to see, whether you need raw order IDs or aggregated levels, and what you can afford. This guide covers everything: LOBSTER's exact data format and fields, how to preprocess it properly, what it does and doesn't capture, and how it compares to every major alternative from NYSE TAQ to Databento to free public sources.

Contents

  1. What Is LOBSTER? Origin, Coverage, and Access
  2. LOBSTER Data Format: Message File, Order Book File, and Event Types
  3. Preprocessing LOBSTER Data in Python
  4. Feature Engineering from Order Book Data
  5. What LOBSTER Doesn't Capture (and Why It Matters)
  6. NASDAQ TotalView-ITCH: The Raw Feed Behind LOBSTER
  7. TickData: Multi-Asset Commercial Alternative
  8. Every Other HFT Data Source Worth Knowing
  9. Interactive Comparison: Choose the Right Dataset
  10. Backtesting with Order Book Data: Pitfalls and Best Practices
  11. Decision Framework: Which Dataset Should You Use?

1. What Is LOBSTER? Origin, Coverage, and Access

LOBSTER stands for Limit Order Book System — The Efficient Reconstruction. It was created by Ruihong Huang and Tomas Polak, with supporting research from Nikolaus Hautsch and others at Humboldt University Berlin and the University of Vienna. The original paper was published in 2011 (SSRN #1977207), and the system has been serving the academic community as a data provider since 2013.

What LOBSTER does, in essence, is take NASDAQ's raw TotalView-ITCH binary feed—a firehose of millions of message-level data records per day describing every order submission, cancellation, and execution—and reconstruct the limit order book state after each event. The output is a pair of clean CSV files (one for messages, one for the order book) that you can load directly into Python, R, or MATLAB without touching a byte of binary protocol parsing.

Exchange Coverage

LOBSTER covers NASDAQ only. It reconstructs order books from NASDAQ's Historical TotalView-ITCH files, which means you get data for all NASDAQ-traded securities—both NASDAQ-listed stocks and ETFs that trade on the NASDAQ venue. This is the single most important constraint to understand: if you need NYSE, CBOE, or international exchange data, LOBSTER cannot help you.

Historical Depth

Data is available from April 27, 2010 to the present, updated daily (typically through two business days ago). That gives you over 15 years of continuous history as of early 2026—long enough to cover the Flash Crash (May 2010), the post-Volcker Rule regime, the COVID crash (March 2020), the meme stock era (January 2021), and several Fed tightening and easing cycles.

Number of Stocks

LOBSTER provides data for the entire universe of NASDAQ-traded stocks on any given day. You are not limited to a preset list—you can request data for any ticker that was active on the exchange during your desired time period. This includes delisted securities if they were trading during the period, which is critical for avoiding survivorship bias in research.

Order Book Depth

You can request anywhere from 1 to 200 price levels of depth. Each additional level adds four columns to the order book file (bid price, bid size, ask price, ask size). Common choices are 1 level (best bid/ask only), 5 or 10 levels (typical for mid-frequency research), and 50+ levels (for deep liquidity analysis). The LOBSTER team's own documentation notes that depth beyond 200 levels is unlikely to be informative because algorithmic traders rarely react to the deep book and most trading platforms don't display it.

Pricing and Access

LOBSTER offers an academic subscription at £4,897 per year (plus a one-time £500 setup fee). This includes 10 user accounts for your research institute, unlimited tickers and time periods, up to 200 order book levels, and 1 TB of storage on LOBSTER's servers. Additional sub-accounts cost £100 each. Commercial access for hedge funds, investment banks, and asset managers is available at custom pricing—you'll need to contact them directly. Sample data files are freely available on lobsterdata.com for anyone wanting to explore the format before committing.

2. LOBSTER Data Format: Message File, Order Book File, and Event Types

LOBSTER generates two CSV files per ticker per trading day: a message file describing every event, and an order book file capturing the state of the book after each event. The k-th row of the message file describes the event that caused the book to transition from state k−1 to state k in the order book file. This one-to-one correspondence is what makes the data so clean to work with.

Message File Structure (N × 6)

Each row is one order event. There are exactly six columns:

Message File Columns Timestamp, EventType, OrderID, Size, Price, Direction 34200.183746000, 1, 8349571, 200, 1850000, 1 34200.183746000, 1, 8349572, 500, 1849500, -1 34200.284951000, 4, 8349571, 100, 1850000, 1 34200.301822000, 2, 8349572, 200, 1849500, -1

Timestamp is seconds after midnight, with decimal precision ranging from milliseconds to nanoseconds depending on the historical period. EventType is an integer 1–7 (described below). OrderID is a unique identifier for the order on that day—it resets at market close. Size is the number of shares. Price is the dollar price multiplied by 10,000 (so $185.00 is stored as 1850000). Direction is 1 for buy orders and −1 for sell orders.

The Seven Event Types

Understanding the event types is essential for any preprocessing or feature engineering work:

Type 1Submission — A new limit order is placed on the book. This is the most common event type and represents a trader expressing willingness to buy or sell at a specific price. Type 2Partial Cancellation — An existing order is reduced in size but not fully removed. The OrderID and remaining size are recorded. Type 3Deletion — An order is completely removed from the book. Common when traders withdraw liquidity, often within milliseconds of placing it. Type 4Execution (Visible) — A visible limit order is matched and executed. This is a confirmed trade. The price and size tell you the execution details. Type 5Execution (Hidden) — A hidden order is executed. LOBSTER reports the executed portion, but the remaining hidden volume is not revealed. More on this in the limitations section. Type 6Cross Trade — A non-standard execution, typically from crossing sessions or auction processes. Type 7Trading Halt — The Price field encodes the halt status: −1 = halt begins, +1 = halt ends, 0 = potential quoting resumption. During halts, the order book file duplicates the previous state to maintain row alignment.

Order Book File Structure (N × 4L)

Where L is the number of levels you requested. For each level, there are four columns repeating in this order:

Order Book File Columns (example with 3 levels) AskPrice1, AskSize1, BidPrice1, BidSize1, AskPrice2, AskSize2, BidPrice2, BidSize2, AskPrice3, AskSize3, BidPrice3, BidSize3 1850500, 300, 1850000, 200, 1851000, 1500, 1849500, 500, 1851500, 800, 1849000, 1200

Prices are in the same 10,000× integer format as the message file. When the book is thinner than the number of requested levels, LOBSTER fills empty positions with dummy values: −9999999999 for empty bid prices, 9999999999 for empty ask prices, and 0 for the corresponding volumes. Always check for these sentinels before computing features like weighted mid-price or order book imbalance—they'll corrupt your calculations silently if you don't filter them.

File Sizes and Storage

File sizes vary enormously by stock and depth. A 10-level order book for an actively traded ticker like AAPL or MSFT can easily exceed 5 GB for a single day. Less active stocks are much smaller. At the academic subscription's 1 TB storage limit, you can hold roughly a few months of the most liquid names at deep levels, or years of a moderate universe at fewer levels. Plan your storage and compute accordingly—this is not data you want to load entirely into memory on a laptop.

3. Preprocessing LOBSTER Data in Python

LOBSTER handles the hardest part—reconstructing the order book from raw ITCH messages—but there are still several critical preprocessing steps you need to do before the data is ready for research. Getting these wrong is one of the most common sources of bugs in HFT research.

Loading the Data

import pandas as pd import numpy as np def load_lobster(msg_file, ob_file, num_levels=10): # Message file: always 6 columns msg_cols = ['Timestamp', 'EventType', 'OrderID', 'Size', 'Price', 'Direction'] messages = pd.read_csv(msg_file, header=None, names=msg_cols) # Order book file: 4 columns per level ob_cols = [] for i in range(1, num_levels + 1): ob_cols += [f'AskPrice_{i}', f'AskSize_{i}', f'BidPrice_{i}', f'BidSize_{i}'] orderbook = pd.read_csv(ob_file, header=None, names=ob_cols) # Combine: row k of messages caused book state k return pd.concat([messages, orderbook], axis=1)

LOBSTER also provides official Python (py4lobster) and R (lobsteR) packages that handle authentication, downloading, and basic data loading. The LOBFrame toolkit from UCL's Financial Computing group provides an end-to-end pipeline including ingestion, preprocessing, normalization, and deep learning model training.

Timestamp Handling

LOBSTER timestamps are seconds since midnight in US Eastern Time (EST during winter, EDT during summer—the exchange follows US daylight saving rules). Regular trading hours span 34,200 (9:30 AM) to 57,600 (4:00 PM).

# Convert seconds-since-midnight to proper datetime import pandas as pd base_date = pd.Timestamp('2025-01-15') df['Datetime'] = base_date + pd.to_timedelta(df['Timestamp'], unit='s') df['Datetime'] = df['Datetime'].dt.tz_localize('US/Eastern')

Filtering to Regular Trading Hours

LOBSTER files may include pre-market and post-market events. Most research filters to regular hours, and many researchers further trim the first and last 10 minutes to avoid auction effects:

# Standard regular hours: 9:30 AM - 4:00 PM ET df_regular = df[(df['Timestamp'] >= 34200) & (df['Timestamp'] <= 57600)] # Conservative: skip opening/closing auctions (9:40 AM - 3:50 PM) df_clean = df[(df['Timestamp'] >= 35400) & (df['Timestamp'] <= 56400)]

Handling Trading Halts

Event Type 7 messages signal trading halts (triggered by circuit breakers, pending news, or SEC orders). During halts, the order book file simply duplicates the previous state. You should filter these out for most analyses, but you may want to study them separately if you're researching halt dynamics or volatility around news events.

# Remove halt events (and optionally the stale book states during halts) df = df[df['EventType'] != 7]

Detecting Crossed and Locked Books

A crossed book (best bid > best ask) or locked book (best bid = best ask) indicates a transient data anomaly. These are rare but will produce nonsensical spread and mid-price calculations if left in.

# Flag and remove crossed/locked books spread = df['AskPrice_1'] - df['BidPrice_1'] df = df[spread > 0] # Keep only valid positive spreads

Handling Dummy Values in Thin Books

# Replace LOBSTER's sentinel values with NaN for col in df.columns: if 'Price' in col: df[col] = df[col].replace({-9999999999: np.nan, 9999999999: np.nan}) if 'Size' in col: df.loc[df[col.replace('Size', 'Price')].isna(), col] = np.nan

Converting Prices

# LOBSTER stores prices as dollar × 10,000 # $185.00 → 1850000. Divide to get real dollars: price_cols = [c for c in df.columns if 'Price' in c] df[price_cols] = df[price_cols] / 10000

Memory Management for Large Files

A single day of 10-level AAPL data can have millions of rows. Naively loading it all into a DataFrame may consume 10+ GB of RAM. Key strategies:

Downcast numeric types (float64 to float32 saves ~50% memory). Use chunked reading with pd.read_csv(..., chunksize=100000) and process in batches. For multi-day or multi-stock studies, consider Dask for out-of-core parallel processing, or write intermediate results to Parquet files (columnar storage with excellent compression for this type of data).

Adjusting for Stock Splits

LOBSTER does not adjust historical prices for corporate actions. If you're studying a stock that split during your sample period (e.g., Google's 20:1 split in July 2022, Amazon's 20:1 in June 2022), you must adjust pre-split prices and volumes manually. Multiply pre-split share counts by the split ratio and divide prices by it. The CRSP or Yahoo Finance corporate actions databases can tell you exactly when splits occurred.

4. Feature Engineering from Order Book Data

The whole point of having order book data is to extract features that capture market dynamics invisible in trade-and-quote data. Here are the most commonly used features in HFT research, with formulas and implementation.

Mid-Price and Micro-Price

The mid-price is the simplest fair value estimate: (best bid + best ask) / 2. The micro-price (or volume-weighted mid-price) adjusts for the relative sizes at the top of book, producing a more informative estimate when liquidity is asymmetric:

# Standard mid-price df['MidPrice'] = (df['BidPrice_1'] + df['AskPrice_1']) / 2 # Micro-price (volume-adjusted) df['MicroPrice'] = ( df['BidPrice_1'] * df['AskSize_1'] + df['AskPrice_1'] * df['BidSize_1'] ) / (df['BidSize_1'] + df['AskSize_1'])

When the ask side has more volume than the bid, the micro-price shifts toward the bid (reflecting that the weight of resting sell liquidity makes an uptick slightly more likely). This feature has significant short-horizon predictive power—it's one of the first things any LOB-based ML model should include.

Order Book Imbalance (OBI)

OBI measures the relative pressure between buyers and sellers at one or more levels of the book. Values near +1 indicate heavy buy-side interest; values near −1 indicate selling pressure. It's a strong predictor of short-term price direction and appears in nearly every LOB machine learning paper:

# Single-level imbalance (best bid/ask) df['OBI_1'] = (df['BidSize_1'] - df['AskSize_1']) / \ (df['BidSize_1'] + df['AskSize_1']) # Multi-level imbalance (e.g., top 5 levels) bid_vol = sum(df[f'BidSize_{i}'] for i in range(1, 6)) ask_vol = sum(df[f'AskSize_{i}'] for i in range(1, 6)) df['OBI_5'] = (bid_vol - ask_vol) / (bid_vol + ask_vol)

Order Flow Imbalance (OFI)

While OBI measures the state of the book at a point in time, OFI measures the flow—the net impact of order submissions, cancellations, and executions over a time window. It captures the directionality of market activity, not just the resulting snapshot. Multi-level OFI (MLOFI) extends this to a vector across multiple price levels, typically reduced via PCA.

Realized Volatility

With tick-level data, you can compute realized volatility at far higher frequency than daily returns allow. The standard estimator sums squared log returns over a window:

# 5-minute realized volatility from tick-level mid-prices df['LogReturn'] = np.log(df['MidPrice'] / df['MidPrice'].shift(1)) df['TimeBucket'] = (df['Timestamp'] // 300).astype(int) rv = df.groupby('TimeBucket')['LogReturn'].apply( lambda x: np.sqrt((x**2).sum()) )

VPIN (Volume-Synchronized Probability of Informed Trading)

VPIN estimates the likelihood that informed traders are active in the market, aggregated over volume buckets rather than time buckets. Values above ~0.7 historically precede volatility spikes. It requires careful implementation to avoid look-ahead bias—the volume buckets must be formed strictly from past data.

Kyle's Lambda (Price Impact)

Kyle's lambda estimates how much prices move per unit of signed order flow. Higher values mean the market is less liquid (each trade moves the price more). It's computed by regressing returns on signed square-root dollar volume over fixed time periods.

5. What LOBSTER Doesn't Capture (and Why It Matters)

Understanding LOBSTER's blind spots is just as important as knowing its strengths. Publishing results without acknowledging these limitations is a common peer review red flag.

Hidden and Iceberg Orders

Event Type 5 reports executions against hidden orders, but only the executed portion is revealed. If a hidden order for 10,000 shares has 500 filled, you see the 500-share execution but have no idea that 9,500 shares remain hidden. This means your order book snapshot systematically understates true liquidity. Research suggests that hidden orders can represent 10–40% of total resting liquidity depending on the stock and time period.

Dark Pool Activity

LOBSTER captures only the lit NASDAQ order book. Trades that execute in dark pools (ATS venues like Crossfinder, SIGMA X, IEX's D-Limit mechanism) or are reported through FINRA's TRFs (Trade Reporting Facilities) are invisible. In many stocks, 40–50% of total volume executes off-exchange. If you're studying price discovery, you're missing roughly half the picture.

Odd Lots

Orders for fewer than 100 shares (odd lots) are not included in the ITCH feed that LOBSTER processes. This is an increasingly significant limitation: as stock prices have risen (particularly for names like AMZN pre-split at $3,000+ and BRK.A), odd-lot trading has grown to represent the majority of trades in some securities. SEC data shows that odd lots account for over 50% of all trades in many NASDAQ stocks. Your volume and imbalance calculations are systematically understated.

Other Exchanges

LOBSTER covers only NASDAQ. For stocks that are listed on NYSE but also trade on NASDAQ (which is most of the large-cap universe), you're seeing only the NASDAQ portion of the consolidated order book. The same stock's order book on NYSE, CBOE, IEX, and other venues is invisible. This matters particularly for fragmented stocks where NASDAQ's market share may be only 15–25% of total volume.

Pre-Market and After-Hours

Extended-hours trading (4:00–9:30 AM and 4:00–8:00 PM ET) is lightly documented in LOBSTER. Many researchers simply filter to regular hours, but if you're studying overnight information incorporation or earnings announcements, the data quality and completeness in extended hours requires careful validation.

6. NASDAQ TotalView-ITCH: The Raw Feed Behind LOBSTER

LOBSTER's source data is NASDAQ's TotalView-ITCH protocol, a binary application-level protocol that describes every order-related event on the exchange. Understanding ITCH helps you appreciate both what LOBSTER gives you and what it filters out.

ITCH is a message-level protocol that carries over 20 message types including system event messages (market open/close), stock directory information (which securities are available), order-related messages (add, execute, cancel, replace), trade break messages, net order imbalance indicators (NOII for auction pricing), and various administrative messages. The current version is ITCH 5.0, with historical data also available in versions 3.0 and 4.1.

LOBSTER filters this stream to keep only the order-book-relevant messages: submissions, cancellations, deletions, and executions (Types 1–5). System events, stock directory messages, NOII messages, and other administrative traffic are discarded. The surviving events are applied to the book reconstruction algorithm to produce the clean CSV output.

If you need the raw ITCH feed directly (for example, to study NOII signals or to build your own book reconstruction with different assumptions), you can purchase NASDAQ's Historical TotalView-ITCH files. The tradeoff is that you'll be parsing binary data and handling all the edge cases yourself—exactly the work that LOBSTER was built to eliminate.

7. TickData: Multi-Asset Commercial Alternative

TickData has gone through several ownership changes. Originally an independent vendor, it was acquired by OneMarketData in 2015. Then in September 2025, OneMarketData merged with KX (owned by TA Associates), creating a combined entity competing with Bloomberg and LSEG/Refinitiv in the market data space.

Where LOBSTER is narrow and deep (one exchange, full order book), TickData is broad: it covers equities across NYSE, NASDAQ, AMEX, and regional exchanges; futures on 150+ global contracts from CME, ICE, Eurex, and others going back to 1974; options on all US equity and index options via OPRA; and forex with 2,000+ spot currency pairs. Data is delivered as delimited text files and timestamps reach nanosecond resolution for NASDAQ data from late 2016 onward.

The key distinction from LOBSTER is that TickData provides trade-and-quote data, not reconstructed order books. You get the national best bid and offer (NBBO) at each timestamp, but not the full depth of book. For research that requires seeing resting liquidity at multiple price levels—order book imbalance, queue position estimation, hidden order detection—TickData is insufficient. For research that needs cross-asset analysis, multi-exchange coverage, or futures data, it's often the better choice.

Pricing starts at a $1,000 minimum for new clients ($500 for returning clients), with per-symbol-month tiered pricing and volume discounts at 100+ symbol-years. The TickAPI streaming service starts at $250/month with a one-year commitment.

8. Every Other HFT Data Source Worth Knowing

NYSE TAQ (Trades and Quotes)

The longest-running US equity tick dataset, available from January 1993 through WRDS (Wharton Research Data Services). Daily TAQ provides millisecond-stamped trades and quotes across all US exchanges and off-exchange TRFs—over 10,000 securities on 16+ exchanges. Academic access is through your institution's WRDS subscription; direct access costs $3,800/month with a 12-month minimum. NYSE TAQ is the standard for published research that doesn't require full order book depth.

Databento

A modern, cloud-native vendor that's becoming a serious alternative for quantitative researchers. Databento provides data from 60+ trading venues in multiple schemas: L1 (top of book), L2 (aggregated depth), and L3 (order-by-order)—including full NASDAQ TotalView-ITCH data. Every record carries up to four nanosecond-precision timestamps, including a PTP-synchronized wire capture time. Data comes in CSV, JSON, or their open-source DBN binary format. Pricing starts at $199/month for equities and $179/month for CME futures, with $125 in free credits for new users. For researchers who need ITCH-level data without LOBSTER's academic infrastructure, Databento is the strongest new entrant.

AlgoSeek

A specialist HFT data vendor with co-located ticker plant servers in Equinix NY2 and NY4. They offer survivorship-bias-free tick data for ~27,500 US securities since 1998, collected live from the SIP feed. Their TAQ products include trade+quote, trade+NBBO, and trade+top-of-book variants. Infrastructure-oriented pricing starts at $250/month for leases, with discounts for startups and academics.

FirstRate Data

Institutional-grade tick data for 4,500+ equity tickers from 2010 onward, covering NASDAQ, NYSE, and 10+ other venues plus 4 dark pools. Rigorous quality screening (gaps, duplicates, spikes) and same-day delivery by 11:30 PM ET. Used by hedge funds and institutions including NBER, Boston Fed, and several top universities.

SEC MIDAS

The SEC's Market Information Data Analytics System collects approximately 1 billion records daily from proprietary feeds of all 13 national equity exchanges, timestamped to the microsecond. It was built for market surveillance (detecting flash crashes, analyzing market structure), but the data is freely available to the public. Coverage includes all posted orders, modifications, cancellations, and trade executions across both on- and off-exchange venues. If you need broad market surveillance data and don't mind working with the SEC's interface, this is the only free source of comprehensive US equity microstructure data.

Kibot

An inexpensive option covering 62,000+ instruments with 17+ years of tick data. However, quality is widely reported as poor: users document missing dividend adjustments, data holes, and incorrect prices. Updates lag 8–12 hours behind real time. Suitable only for rough prototyping at higher timeframes (5-minute bars and above); not reliable for serious microstructure research.

Crypto Exchange APIs (Free)

If you're researching order book dynamics and don't need traditional equities, cryptocurrency exchanges offer free real-time data. Binance provides up to 1,000 levels of depth via WebSocket streams. Coinbase offers full Level 3 (order-by-order) data natively. The tradeoff is different market structure (no Reg NMS, different tick rules, 24/7 trading) and the need to build your own historical database. Third-party aggregators like Tardis.dev and CoinAPI provide historical order book replay across multiple crypto exchanges.

Public Academic Datasets

The FI-2010 benchmark dataset (from the Finnish stock exchange) provides order book data for 5 stocks over 10 trading days—about 4 million samples—and is freely downloadable. It's tiny compared to LOBSTER but widely used as a benchmark for LOB prediction models (DeepLOB, TransLOB, etc.). There are also various LOB datasets on Kaggle and GitHub, though quality varies significantly.

9. Interactive Comparison: Choose the Right Dataset

HFT Data Source Comparison

Filter by your requirements to find the right data source:

Source Depth Exchange Coverage History From Timestamp Cost
LOBSTER L3 (1–200 levels) NASDAQ only Apr 2010 ms–ns £4,897/yr academic
TickData (KX) L1–L2 + order IDs NYSE, NASDAQ, AMEX, Global Futures, FX Dec 1974 (futures) ns (2016+) $1,000+ minimum
NYSE TAQ L1 (NBBO + trades) All US exchanges + TRFs Jan 1993 ms (2003+) $3,800/mo
Databento L1 / L2 / L3 60+ venues (NASDAQ, NYSE, CME, ICE, Eurex) Varies by venue ns (4 timestamps) $179–199/mo
AlgoSeek L1–L2 (TAQ) 16 US exchanges + 3 TRFs 1998 ms $250/mo lease
FirstRate Data L1 (tick trades) 12+ US exchanges + 4 dark pools 2010 μs Custom
SEC MIDAS Full book (reconstructed) All 13 US equity exchanges Jan 2013 μs Free
Binance API L2 (1,000 levels) Binance (crypto only) Real-time + limited history ms Free (rate limits)
Coinbase API L3 (order-by-order) Coinbase (crypto only) Real-time + limited history ms Free (rate limits)
FI-2010 Benchmark L2 (10 levels) NASDAQ Nordic (5 stocks) 2010 (10 days only) Event-driven Free download
Kibot L1 US equities + futures 2009 (tick), 1998 (minute) ms ~$350

10. Backtesting with Order Book Data: Pitfalls and Best Practices

Having order book data enables much more realistic backtesting than trade-and-quote data alone. But it also introduces new categories of errors that don't exist in daily bar backtests. Here are the most important ones to get right.

Look-Ahead Bias

The most insidious bug in HFT backtests. With millions of rows of tick data, it's easy to accidentally use information from time t+1 to make decisions at time t. Common mistakes include: computing features using the order book state after a trade executes rather than before; using future volume to bucket VPIN calculations; and training ML models on data that leaks forward through improper cross-validation splits. Always verify that your feature matrix at row k uses only data from rows 0 through k−1.

Survivorship Bias

If you backtest only on stocks that are in the current NASDAQ index, you're excluding every company that was delisted, acquired, or went bankrupt during your sample period. This inflates backtested returns by an estimated 1–4% annually. LOBSTER's ability to provide data for any historical ticker (including those no longer trading) is a significant advantage here—but you need to actively request delisted tickers, not just use the current universe.

Phantom Liquidity and Flickering Quotes

High-frequency market makers routinely place and cancel orders within milliseconds. These "flickering quotes" inflate the apparent depth of the order book but are not available for you to trade against. If your backtest assumes you can execute at the displayed depth, you'll overestimate fill rates and underestimate market impact. A practical filter: flag quotes with lifetimes under 100 milliseconds and exclude them from your available liquidity calculation.

Realistic Execution Simulation

The most critical and most frequently botched aspect of HFT backtesting. At minimum, your simulator needs to model: queue position (at a given price level, you fill in FIFO order—not instantly), market impact (your own order changes the book state), latency (you see the market 1–100 ms in the past, and your orders arrive 1–100 ms in the future), and partial fills (the book may not have enough depth to fill your entire order at the displayed price). Libraries like hftbacktest (on GitHub) provide frameworks for this, but there's no substitute for understanding the mechanics yourself.

Data Snooping

With high-frequency data, you have millions of observations per day. Testing many hypotheses on this much data virtually guarantees you'll find "significant" patterns by chance. Use out-of-sample holdout periods (never touch them during development), apply Bonferroni or BH corrections for multiple comparisons, and pre-register your hypotheses before running tests.

11. Decision Framework: Which Dataset Should You Use?

After reviewing everything above, here's how to think about the choice:

You need full NASDAQ order book depth for academic microstructure research: LOBSTER is the clear choice. It's the most cited, most reproducible, and most thoroughly validated source for this specific use case. The academic pricing is reasonable, and your peers will be using the same data, making results directly comparable.

You need multi-exchange US equity data (trades and quotes) for published research: NYSE TAQ through WRDS. It's the oldest, most widely cited trade-and-quote dataset, and your institution likely already has access. You won't get order book depth, but you get comprehensive coverage across all venues.

You need L3 order book data and modern tooling on a budget: Databento. Their ITCH data gives you the same raw feed LOBSTER uses, their API is well-designed, and at $199/month it's accessible to independent researchers and small funds. The tradeoff is that you'll do your own book reconstruction.

You need multi-asset coverage (equities + futures + FX): TickData is the traditional answer. Databento is catching up fast with 60+ venues and significantly lower pricing.

You're a student or independent researcher with no budget: Start with SEC MIDAS for US equities (free, microsecond-stamped, comprehensive). Use Coinbase's L3 API or Binance's depth streams for crypto order book data. Download the FI-2010 dataset for benchmarking ML models. Apply for LOBSTER sample data to learn the format.

You're building a production trading system: You likely need direct exchange feeds (NASDAQ TotalView-ITCH, NYSE OpenBook) with co-located infrastructure. LOBSTER and similar services add latency that's unacceptable for live trading, but they're invaluable for the research that informs your strategy design.

The best approach for serious research programs is often a combination: LOBSTER for deep order book analysis on NASDAQ names, NYSE TAQ for cross-exchange trade studies, and a vendor like Databento or AlgoSeek for filling specific gaps. Understanding the strengths and limitations of each source—and being transparent about them in your methodology sections—is what separates rigorous research from the rest.