Back to home

Iceberg REST catalog · DuckDB · Polars

API

Versão em português

Attach the dartic lake from your own tools. The catalog at https://dartic.io/catalog speaks the Apache Iceberg REST protocol (read-only) and lists the same tables as the web console. Table data is read straight from the public bucket; the catalog needs a personal access token.

Access tokens

Sign in, open the API page in the console and create a token. It starts with dpat_ and is shown once; store it like a password (for example in DARTIC_TOKEN). You can hold up to 10 tokens, choose an expiry, and revoke any of them at any time.

Create a token

HTTP

Every request carries Authorization: Bearer <token>.

export DARTIC_TOKEN=dpat_...

# Namespaces
curl -H "Authorization: Bearer $DARTIC_TOKEN" https://dartic.io/catalog/v1/namespaces

# Tables in a namespace
curl -H "Authorization: Bearer $DARTIC_TOKEN" https://dartic.io/catalog/v1/namespaces/equities/tables

# Current Iceberg metadata of one table
curl -H "Authorization: Bearer $DARTIC_TOKEN" https://dartic.io/catalog/v1/namespaces/equities/tables/raw_cvm_cgvn_geral

DuckDB

CLI or Python. ATTACH exposes every table as lake.<namespace>.<table>, the same names the web editor uses.

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs;

-- Data files are public: an S3 secret without keys, pointed at the lake domain.
CREATE SECRET dartic_data (TYPE S3, KEY_ID '', SECRET '', REGION 'auto',
  ENDPOINT 'data.dartic.io', URL_STYLE 'vhost', USE_SSL true);

-- Your personal access token for the catalog.
CREATE SECRET dartic (TYPE iceberg, TOKEN 'dpat_...');

ATTACH 'dartic' AS lake (TYPE iceberg, ENDPOINT 'https://dartic.io/catalog',
  SECRET dartic, ACCESS_DELEGATION_MODE 'none');

SHOW ALL TABLES;
SELECT * FROM lake.equities.raw_cvm_cgvn_geral LIMIT 10;

Polars

Through PyIceberg. scan_iceberg is lazy: filters and projections push down.

# pip install polars 'pyiceberg[pyarrow]'
import os

import polars as pl
from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    "dartic",
    uri="https://dartic.io/catalog",
    token=os.environ["DARTIC_TOKEN"],
    **{
        "s3.endpoint": "https://data.dartic.io",
        "s3.force-virtual-addressing": "true",
        "s3.anonymous": "true",
        "s3.region": "auto",
    },
)

print(catalog.list_tables("equities"))
df = pl.scan_iceberg(catalog.load_table("equities.raw_cvm_cgvn_geral")).head(10).collect()
print(df)

Without a token

The lake is open data. The list of tables and their current metadata files is published at https://hyoko-raw.data.dartic.io/dartic/manifest.json and any of them can be read with iceberg_scan. The token adds discovery and ATTACH.

-- No token: read one table straight from its metadata file listed in manifest.json.
SELECT count(*) FROM iceberg_scan(
  's3://hyoko-raw/__r2_data_catalog/.../metadata/00013-....gz.metadata.json',
  metadata_compression_codec => 'gzip');

Limits

120 catalog requests per minute per token (an ATTACH costs about ten). The catalog is refreshed with the lake, once a week. Writes are rejected with 405.