Archive
Trades: 7 days (core and top-100). Core ticker, mark, OI, stats, depth-derived: 10-second samples for 6 months. Core archived as Parquet (backfilling). A backtest wants months of trades, and it reads them end to end rather than at a point in time. The archive is that history as files: one Parquet file per listing, layer and UTC day, written the day after the day is over, checked against the storage it was written to, and listed and fetched through the API.
The principle is the platform’s own: stored as seen. A file holds exactly the rows the platform observed. Nothing is filled, nothing is interpolated, no row is repaired into a smoother series. Where the platform was not listening, the file has no rows, and the gaps of the day are in the list beside it.
What is archived, and for how long
Section titled “What is archived, and for how long”| group | layers | in the database | in the archive |
|---|---|---|---|
| core | every layer below | days: see Retention | from the day after each day is anchored, with no end |
| top-100 | every layer below | days: see Retention | from the day after each day is anchored, with no end |
| tail, new | every layer the listing is collected for | days: see Retention | from the day after each day is anchored, with no end (backfilling from 2026-10-30) |
The layers are trade, ticker, mark, depth-derived, open-interest, stats, funding, candle (the 1-minute candles of every price type) and venue-depth. Candle rollups
(5 minutes and longer) are computed from the 1-minute candles and are not archived; liquidations and funding settlements stay in the database and are not archived.
A day’s row in the database is dropped only after its file has been read back from the storage and its row count and hash are what was written, the database still
holds as many rows of that listing and day as the file, and (for every layer but venue-depth) the day’s root is anchored on Base and recorded beside the file. There is no day that is
neither in the database nor in the archive because a copy failed.
depth-derived is the book’s derived figures (mid, spread, liquidity in the bands around the mid and the value resting in them, the number of levels), not the book itself:
levels are never stored.
Listing the files
Section titled “Listing the files”curl -s -H "Authorization: Bearer $DEBYKO_KEY" \ "https://api.debyko.com/v2/archive?listing=BINANCE-USDM:BTCUSDT&layer=trade&from=2026-09-28&to=2026-09-30"{ "listing": "BINANCE-USDM:BTCUSDT", "from": "2026-09-28", "to": "2026-09-30", "truncated": false, "items": [ { "layer": "trade", "day": "2026-09-30", "group": "core", "available": true, "rows": 4182113, "bytes": 187004211, "sha256": "9f2c…e1", "gaps": 1, "anchored": false, "download": "/v2/archive/BINANCE-USDM%3ABTCUSDT/trade/2026-09-30", "coverage": [ { "cause": "ws_disconnected", "start": "2026-09-30T03:12:09.114000Z", "end": "2026-09-30T03:12:31.480000Z", "end_unknown": false, "scope": "segment", "datasets": null, "repaired_at": null } ] } ]}(The figures are illustrative.) The list is free with any key and counted against nothing. It shows only files that were read back and checked. A day on which nothing was observed
has available: false, no hash and no download, and its coverage says why.
Fetching a file
Section titled “Fetching a file”curl -sL -H "Authorization: Bearer $DEBYKO_KEY" -o btcusdt-trade-2026-09-30.parquet \ "https://api.debyko.com/v2/archive/BINANCE-USDM%3ABTCUSDT/trade/2026-09-30"The answer is a 302 to a link that reads that one file for fifteen minutes. The response carries X-Debyko-Sha256 (the file’s hash) and Link (the day’s entry in the list). The link
needs no key and carries none: a client that drops the Authorization header when it follows a redirect to another host, as curl -L and requests do, never sends your key to the storage.
The contents of a file are never paged through the API: the redirect is the whole of it.
- With a key, a file counts against the plan’s files for the calendar month (UTC): Free and Mini none, Standard 100, Pro 1 000, Pro Plus 10 000. Past them the answer is
402with the price of the file;X-Debyko-Archive-Remainingsays what is left. - Without a key, when paying per call is on, a file costs 0.10 USDC (core) or 0.05 USDC (top-100): one payment, one file, one
302. - A day that is not archived yet is a
404: a day is written once it has been over for two days.
When as_of or a history request reaches a raw day the database no longer holds, the answer is not_retained as ever, and when a file exists for that day it carries
archive: { "path": …, "sha256": … } beside the reason. That is a pointer and nothing more; the file is fetched as above.
The file
Section titled “The file”- Path in the storage:
archive/{venue}/{listing}/{layer}/{yyyy}/{mm}/{listing}-{layer}-{yyyy-mm-dd}.parquet, the venue in lower case, the symbol with every character outsideA-Za-z0-9._-written as%XX. - Rows: in time order (trades: then by the venue’s trade id), one row per observation, exactly as stored. A file has the rows of one listing and one UTC day.
- Row groups: one per hour, a very busy hour in groups of at most 25 000 rows, each with minimum and maximum statistics on every column, so a reader can skip the hours it does not need. Compression: zstd.
- Types: timestamps are
INT64with the logical type timestamp, UTC, nanosecond unit (the platform keeps microseconds; they are written as they are). The manifest recordsschema_version; v2 writes it in the Parquet footer too. Older v1 footers have no version key and are treated as v1. In v2, venue-published numeric columns (including the mixed-provenancepremium) are exact decimal strings, with no float conversion or rounding. Derived-only columns, includingmidandspread, remainDECIMAL(38, 18). Existing v1 files stay unchanged: their numeric columns areDECIMAL(38, 18).generation,sequenceareINT64, level countsINT32, words are strings, andsuspendedisBOOLEAN. - Columns, in this order, per layer.
?marks a column that can be null (the venue did not publish it).
trade — event_time, price, size, side, venue_uid, received_at, generation, sequence, origin
ticker — bucket, bid, ask, last?, bid_size?, ask_size?, received_at, source_time?, generation, sequence, transport, origin, cadence?
mark — bucket, price, index_price?, premium?, received_at, source_time?, generation, sequence, transport, origin, cadence?
depth-derived — bucket, mid, spread, spread_bps, imbalance?, bid_liquidity_01?, bid_liquidity_025?, bid_liquidity_05?, bid_liquidity_1?, ask_liquidity_01?, ask_liquidity_025?,
ask_liquidity_05?, ask_liquidity_1?, bid_value_01?, bid_value_025?, bid_value_05?, bid_value_1?, ask_value_01?, ask_value_025?, ask_value_05?, ask_value_1?, bid_levels, ask_levels,
bid_band_levels_01?, bid_band_levels_025?, bid_band_levels_05?, bid_band_levels_1?, ask_band_levels_01?, ask_band_levels_025?, ask_band_levels_05?, ask_band_levels_1?,
received_at, source_time?, generation, sequence, transport, origin, cadence?
open-interest — bucket, value, received_at, source_time?, generation, sequence, transport, origin, cadence?
funding — bucket, rate, relative_rate?, predicted_rate?, predicted_relative_rate?, next_funding_at?, short_rate?, predicted_short_rate?, received_at, source_time?,
generation, sequence, transport, origin, cadence?
candle — open_time, price_type, open_price, high_price, low_price, close_price, volume?, trade_count?, received_at, source_time?, generation, sequence,
transport, origin
venue-depth — bucket, bid_01?, ask_01?, bid_025?, ask_025?, bid_05?, ask_05?, bid_1?, ask_1?, received_at, source_time?, generation, sequence,
transport, origin, cadence?
stats — bucket, volume_24h?, quote_volume_24h?, open_24h?, high_24h?, low_24h?, change_percent?, suspended?, received_at, source_time?,
generation, sequence, transport, origin, cadence?
Derived columns are the platform’s own calculation, not a venue’s figure, and are listed in the file’s footer (debyko.derived_columns). What the platform computes has the precision of its unit, half to even, fixed where it is
computed (from 2026-10-02; before that a division carried up to 28 digits): a fraction or ratio has 12 digits after the decimal point, basis points have 6, and a price or quantity is exact arithmetic, as stored, with
no rounding of ours. Where a stored historical value still has more digits than its unit, the file holds it rounded half to even to the digits of the unit (to 18, the column’s scale, for an amount): that is the only rounding there is, and only these
columns have it. The mark layer’s premium is the one derived column that is a venue’s own figure on some rows (Hyperliquid, OKX and Kraken Futures state it; the platform divides it for Bybit, Binance, Coinbase and Weex, to 12 digits when it computes it),
so v2 holds its stored decimal text exactly. A v1 file, written before this version, holds the value at the old 18-digit scale.
| Layer | Columns | Kind |
|---|---|---|
trade, ticker |
every number | venue figure, never rounded |
open-interest, stats, funding, candle, venue-depth |
every number | venue figure, never rounded |
mark |
price, index_price |
venue figure, never rounded |
mark |
premium |
derived: fraction, 12 digits when the platform computes it (a stated premium is kept to 18) |
depth-derived |
mid, spread |
derived: amount, as stored (at most 18 digits); exact arithmetic on venue prices |
depth-derived |
spread_bps |
derived: basis points, 6 digits |
depth-derived |
imbalance |
derived: fraction, 12 digits |
depth-derived |
bid_liquidity_01, bid_liquidity_025, bid_liquidity_05, bid_liquidity_1, ask_liquidity_01, ask_liquidity_025, ask_liquidity_05, ask_liquidity_1 |
derived: quantity, as stored (at most 18 digits) |
depth-derived |
bid_value_01, bid_value_025, bid_value_05, bid_value_1, ask_value_01, ask_value_025, ask_value_05, ask_value_1 |
derived: amount in the quote currency, as stored (at most 18 digits) |
bucket is the start of the sampling window the row stands for (cadence says how long it is: 1s, 10s, 15s or 1m); received_at is when the platform received the value and source_time when the venue says it
happened. The same words mean the same as in the snapshots. Not in a file: the instrument (the file is of one listing) and the storage class a row was filed under, which is the platform’s own routing.
Coverage: rows observed, gaps in the list
Section titled “Coverage: rows observed, gaps in the list”A file is not a claim that the market was quiet where it has no rows. It is the rows the platform saw. The gaps of the day (the platform’s connection to the venue broke, a poll was rate limited,
the collector was down) are in coverage of the list entry with their causes, copied from the platform’s gap record when the file was written: scope says whether the whole venue connection was
affected or this listing alone, datasets which layers, repaired_at whether a replay later filled it (such rows are in the file, with their own origin). A day with a gap is still written, rows as observed. A
day with no rows at all has no file, and an entry that says why.
Reading it
Section titled “Reading it”Each takes the downloaded file or, for the first two and DuckDB, the link itself (the Location of the 302; it is good for fifteen minutes).
# pandasimport pandas as pdimport decimaldf = pd.read_parquet("btcusdt-trade-2026-09-30.parquet") # or the https link; v2 prices are strings, v1 prices are decimal.Decimaldf["price"] = df["price"].map(decimal.Decimal) # exact for both versions# polarsimport polars as pldf = pl.read_parquet("btcusdt-trade-2026-09-30.parquet") # or pl.scan_parquet(link) to read only the row groups a filter needsdf = df.with_columns(pl.col("price").cast(pl.Decimal(38, 30))) # only when this precision and scale fit your values-- DuckDBSELECT count(*), min(event_time), max(event_time) FROM read_parquet('btcusdt-trade-2026-09-30.parquet'); -- or the https linkSELECT TRY_CAST(price AS DECIMAL(38, 30)) AS price FROM read_parquet('btcusdt-trade-2026-09-30.parquet'); -- NULL when it does not fitDecimal casts in Polars and DuckDB require a precision and scale that fit your data; keep the original string for values outside their range.
Hashes, and the anchor
Section titled “Hashes, and the anchor”Every file has a SHA-256 in the list and in the X-Debyko-Sha256 header of the download; what you download hashes to it (sha256sum file.parquet). The hash is what the manifest says the platform wrote and then read back
from storage. New files also have a rows_hash: SHA-256 of the canonical row stream (JCS row arrays in schema column order, one per line), with numeric values represented as decimal strings. This hash is independent of the v1/v2 Parquet encoding and is the planned anchor leaf;
the file SHA-256 checks the downloaded bytes. Existing v1 files are not rewritten and their manifest rows_hash remains empty until a separate read-only backfill. Anchoring is not live yet: anchored is false in the list.