Frequently asked questions

What can I do with LSDB?#

LSDB is for analyzing sky catalogs that are too big to download and too big to hold in memory (think: hundreds of millions to billions of rows). With LSDB you can:

• Read a catalog you never downloaded. You get only the columns and the region of sky you ask for.
• Cut a billion rows down to something workable. Filter by region, by column, or by expression, before any data is read.
• Cross-match against another survey. LSDB pairs up overlapping tiles for you, at any scale.
• Run your own function over the whole sky. It runs in parallel across partitions, and the result can be written back out as a catalog of your own.

A whole workflow is usually one chain, and none of it reads data until you ask:

import lsdb

ztf = lsdb.open_catalog(
    "https://data.lsdb.io/hats/ztf_dr14/ztf_object",
    columns=["ra", "dec", "mean_mag_r"],
)
gaia = lsdb.open_catalog(
    "https://data.lsdb.io/hats/gaia_dr3/gaia",
    margin_cache="https://data.lsdb.io/hats/gaia_dr3/gaia_10arcs",
    columns=["ra", "dec", "phot_g_mean_mag"],
)

result = (
    ztf.cone_search(ra=180.0, dec=10.0, radius_arcsec=3600)
       .crossmatch(gaia, radius_arcsec=1.0, require_right_margin=True)
       .query("mean_mag_r < 19")
)

What it is not: a plotting library, a fitting library, or a replacement for pandas. It gets you a manageable table out of an unmanageable one; what you do next is ordinary Python.

Why use HATS and LSDB?#

As data sizes grow, new and interesting catalogs no longer fit anywhere convenient. Gaia DR3 is about 1.8 billion rows. You cannot download it before lunch, and you cannot load it into memory at all.

The usual escapes each give something up. Downloading a subset means deciding what you need before you know. A cone-search service is excellent for a hundred objects and hopeless for ten million. Neither lets you ask a question of the whole catalog.

HATS and LSDB take a different route, built on one idea: objects near each other on the sky live in the same file. Everything follows from that.

• Spatial questions read almost nothing. "Everything within a degree of here" touches three files instead of thirty thousand.
• Cross-matching becomes tractable. Two catalogs tiled the same way can be matched tile against overlapping tile, rather than comparing every row to every other row.
• The work is embarrassingly parallel. Partitions are independent, so every core, or every machine, can take one.
• The same code scales. What you prototype on a laptop over conference wifi runs unchanged on a cluster.

The cost is that the data has to be tiled that way in the first place. That is what HATS is, and what hats-import does for data of your own.

What is HATS?#

HATS, or Hierarchical Adaptive Tiling Scheme, is the file format LSDB reads. It is a way of splitting a sky catalog into files such that objects near each other on the sky end up in the same file.

It starts with HEALPix

HEALPix divides the sphere into 12 equal-area base tiles. Split each into four, and you have 48. Split again: 192. How many times you have split is the tile's order. Every tile has an order and a pixel number, and that pair names it uniquely.

Why adaptive tiling?

Sky catalogs are wildly uneven: the Galactic plane holds orders of magnitude more sources than the poles. Tiling at one fixed order would give you some files with a few thousand rows and others with tens of millions.

So HATS varies the order across the sky: dense regions get subdivided further, sparse regions stay coarse, until every tile holds roughly the same number of rows.

What it looks like on disk

A HATS catalog is a directory of parquet files named by order and pixel, plus metadata describing the partitioning. Nothing exotic — you can open any single file with pandas.

gaia/
├── hats.properties         # what this catalog is: rows, columns, provenance
├── partition_info.csv      # which pixels exist, at which orders
├── skymap.fits             # coverage, for quick answers about where data is
└── dataset/
    ├── _metadata           # parquet footers for every file, in one place
    ├── Norder=2/Dir=0/Npix=51.parquet     # a sparse region, split twice
    ├── Norder=5/Dir=0/Npix=2341.parquet   # a dense region, split five times
    └── ...

Because the partitioning is spatial, LSDB can answer "everything within one degree of here" by reading three files instead of thirty thousand — and can crossmatch two catalogs by pairing up tiles that overlap on the sky.

What is a Catalog Collection?#

A collection is a main catalog published together with its supplementary tables, described by a collection.properties file. Opening the collection gets you all of it; opening the bare catalog gets you only the rows.

So: if you open a collection, the margin arrives attached at .margin. If you open a plain catalog, you must supply margin_cache yourself, or quietly go without one.

What is a nested column?#

Some data is one measurement per object (e.g. a position, a magnitude). Some is many per object: a light curve, a set of detections, a spectrum. A nested column holds that second kind, by putting a whole sub-table inside a single cell.

The point is that one row stays one object, no matter how many epochs it has.

Do I need to download the data?#

No, and for most catalogs you should not try. LSDB reads HATS catalogs directly from https:// and s3:// endpoints, fetching only the pieces your workflow actually touches.

Two things make that work:

• Because the catalog is tiled by sky position, a spatial filter (e.g. a cone search) eliminates whole files before anything is opened.
• And because the files are parquet, the reader pulls only the columns you named, using HTTP range requests rather than downloading each file whole.

So columns=[...] and a cone search are not merely tidy — they are what keeps the transfer small enough to work over conference wifi.

A local copy can be worth it when you are running the same heavy workflow repeatedly, or when your cluster already hosts a mirror. In this case, you can point LSDB at the local path and nothing else changes.

How does this relate to Pandas?#

A Catalog is meant to feel like a dataframe you have not run yet, and compute() hands you a real one: a nested-pandas NestedFrame, which is a subclass of pandas.DataFrame. Anything you know how to do to a dataframe works on the result.

import pandas as pd

frame = gaia.cone_search(ra=180.0, dec=10.0, radius_arcsec=600).compute()

isinstance(frame, pd.DataFrame)   # True
frame.index.name                  # '_healpix_29'

The habit that transfers worst is reaching for a global view: sorting the whole catalog, or indexing into an arbitrary row. Those assume everything is in memory at once, which is exactly the assumption LSDB is built to avoid. Filter first, compute second, and be ordinary pandas from there.

How to migrate from astropy#

from_astropy takes the Table you already have and hands back a catalog.

from astropy.table import Table
import lsdb

data = {
    "ra": [10.0, 20.0, 30.0],
    "dec": [-10.0, -20.0, -30.0],
    "magnitude": [15.0, 16.5, 14.2],
}
table = Table(data)
catalog = lsdb.from_astropy(table, ra_column="ra", dec_column="dec")
catalog.head()

# From here it is an ordinary LSDB catalog: crossmatch it against a survey,
# filter it, or write it out. See Do for the recipes.

Coordinate columns are detected by name when you leave ra_column and dec_column out, and a margin is built for you, so the result is safe to crossmatch straight away.

Spatial filtering, both ways

Narrowing an astropy table to a region runs centre-first: build a SkyCoord for the position, build another for every row of the table, then keep the rows whose separation is small enough.

# Pure astropy

from astropy import units as u
from astropy.coordinates import SkyCoord

coord_single = SkyCoord(10.0, -10.0, unit="deg")
coord_table = SkyCoord.guess_from_table(table, unit="deg")
matches = table[coord_single.separation(coord_table) < 1 * u.arcsec]

None of that scaffolding is needed on the LSDB side. There is no intermediate coordinate object to build, because the catalog already carries its positions:

# Astropy table in lsdb

import lsdb

matches = lsdb.from_astropy(table).cone_search(ra=10.0, dec=-10.0, radius_arcsec=1)

This lets us avoid excessive computation: astropy computes a separation for every row because the table is just numbers until you wrap it. A HATS catalog already knows where in the sky each of its rows are, so a cone search discards entire files before computing any distance at all.

Which package do I need?#

There are four main packages in the LSDB ecosystem. They divide up cleanly once you see the split: one is a file format, one analyses data in it, one creates it, and one is the table type underneath.

For a hack day, the answer is almost always just pip install lsdb. This will bundle in hats and nested-pandas as dependencies, and you meet them only when you need them.

What went wrong?#

Why is nothing happening?

Probably because nothing is supposed to be happening yet. LSDB is lazy: each operation records what you want and returns immediately. No file is opened, no row is read. You are building a plan, not running one.

That is why open_catalog on a billion-row survey returns instantly, and why a cross-match of two catalogs takes no time at all until the moment you ask for the answer.

"Instant" and "cheap" are not the same as "lazy": the second row runs straight away, it just never touches your data. On Do, any snippet that reads rows is tagged ⚡ triggers computation.

Why did my kernel die?

The same reason, from the other side. compute() means "give me all of this, in memory, now". If the catalog is a billion rows, that request is not slow, it is impossible — and Python is killed for asking.

Three habits avoid it entirely:

• Ask what it would cost first. estimate_size() prints the expected memory without reading any data.
• Use head() to look. Looking at values and materialising a result are different jobs.
• Cut down before computing. A cone search first, the full sky later.

If work is genuinely running and you want to see it, start a Dask client — the dashboard shows exactly which partitions are in flight and how much memory each worker is holding.

Don't see your question?#

The LINCC Frameworks team is friendly and genuinely wants to hear from you — especially if you have a gnarly dataset.

• Drop-in office hours, Fridays 10–11am US Pacific. Bring a problem you're stuck on, or just work alongside the people who build this.
• Slack: #lincc-frameworks-lsdb on LSSTC Slack, where most day-to-day discussion happens.
• Bugs and feature ideas: open an LSDB issue.
• Rubin HATS data questions: the LSST Community Forum.