Andy Garcia · freelance dev
FRENES
Back to the blog
Space

Gaia DR4 Is Scheduled for December 2 at About 400 TB. ESA Changed the Archive Rules First.

October 20, 2026 7 min read

On Wednesday, October 14, the Gaia ESA Archive, where astronomers query the catalogues of ESA's Gaia mission, shipped version 3.15 with two breaking changes. Two DataLink data structures were renamed: RAW became DATAMODEL_GAIA, INDIVIDUAL became DATAMODEL_STANDARD. And "to support a fair and equitable distribution of resources," no user can have more than five jobs running at once. Each user has a queue of 50 jobs; anything submitted beyond it is rejected.

The reason is on the calendar. Gaia's fourth data release, DR4, is scheduled for Wednesday, December 2, 2026, at about 12:00 CET. ESA announced the date on May 7. On October 8, it gave the cutover: the archive is scheduled to be offline from Wednesday, November 25 at 12:00 CET until December 2 at 12:00 CET, when it reopens with DR4. Release dates can shift, and ESA's content page still describes the expected content as "pending certain processing and validation activities." ESA gives the order of magnitude: about 400 TB, against 10 TB for DR3.

What astronomers will find in it is theirs to say. My side of the problem is the other one: I am CTO of a LegalTech built on Claude, and I built BidForge, an AI that reads French public tender files. My products live on data that other people publish. So I read the archive's release notes rather than the countdown. They read like a migration plan run over nearly two years, in public.

What Gaia measured, in plain terms

Gaia scanned the whole sky, again and again, from July 2014. Each time a star crossed its detectors, it recorded the star's position and brightness, and spectra for many of them. Science observations ended on January 15, 2025, as its cold gas propellant ran low, and the spacecraft was switched off on March 27, 2025.

From those repeated measurements, pipelines fit a model for each star: where it is, how it drifts across the sky, and its parallax, the small yearly back-and-forth shift caused by Gaia's own orbit around the Sun, which it shared with Earth. Parallax gives distance, and the angles are tiny. They are counted in milliarcseconds, and a single degree holds 3.6 million of them. A star with a parallax of 1 milliarcsecond is about 1,000 parsecs away, roughly 3,260 light-years.

DR4 uses 66 months of data, from July 25, 2014 to January 20, 2020, against 34 months for DR3. Each release replaces the previous one rather than adding to it. And with DR4, ESA says, the full Gaia time series for those first 5.5 years become available: the individual measurements underneath the fitted answers, which ESA calls epoch data. Earlier releases carried only subsets.

Why that matters has a name: Gaia BH3. A star with an unseen companion wobbles, and a single-star model fits it poorly. During the preparation of DR3, this star did produce two candidate solutions hinting at a companion, and both were discarded for low significance: the orbit takes 11.6 years, much longer than DR3's 34 months. It turned up while the team validated preliminary DR4 binary solutions: a black hole of 32.70 ± 0.82 solar masses, about 590 parsecs away. The collaboration took what its paper calls "the exceptional step" of publishing it in 2024, ahead of the release. It is one of 12 sources whose epoch astrometry ESA released on June 29, so users could prepare. With the measurements, anyone can fit a model the pipeline did not choose.

The arithmetic of 400 TB

The DR4 content page lists the expected products, each with its row count and volume. The main table, gaia_source, holds 2,000,033,276 sources "considered to have both high-quality astrometry and photometry." It weighs 586 GB, about 0.15% of the release. In all, data will be published for about 2.8 billion processed sources.

The weight is in the epoch data. Four products: epoch BP/RP spectra, 119 TB; epoch images, 113 TB; epoch astrometry, 62 TB for 129.6 billion records; epoch RVS spectra, 49 TB. Together about 343 TB, more than four-fifths of the total.

The two kinds of product are not served the same way. Catalogue tables are queried through TAP with ADQL, a SQL dialect. The big epoch products go through DataLink and are fetched by source: up to 5,000 sources per query for a registered user, 50 for an anonymous one. A 586 GB table is something you query. 343 TB is something you fetch piece by piece.

Nearly two years of migration, in public

The release notes show how the archive got there.

  • December 10, 2024, version 3.6: a "BREAKING CHANGE" deprecated the COMBINED structure, which packed one product for many sources into a single file. ESA explained that serving the largest DR4 products that way "would be unfeasible." DR3 light curves moved to a new serialization that DR4 will reuse, so users could "get prepared well in advance."
  • June 4, 2025, version 3.8.1: the back end serving DataLink products from Parquet files was activated. An entry dated June 12 says the archive "has been rolled back to version 3.7." The December notes list, under bug fixes, an "Analysis of the Gaia Archive overload in OPS after deployment of GACS 3.8." They say nothing more.
  • December 2, 2025, version 3.10: what the news page called a major backend infrastructure upgrade. Fifteen days later, ESA listed the side effects (a web interface taking up to dozens of seconds, some joined queries timing out, DataLink products taking up to several minutes) and published workarounds: programmatic access, box searches instead of cone searches, the bulk download repository.
  • April 15, 2026, version 3.12: the bulk download area moved from file-based to object-based storage, and queries combining joins with cone searches got better handling. Two patch releases followed within a week.
  • October 14, 2026, version 3.15: the database system migrated from Greenplum to EDB WarehousePG, alongside the renames and the job caps.

Note what the October changes land on: DR3, the data everyone already uses. Users fix their scripts against a system they know, seven weeks before the new data arrives.

Identifiers belong to a release

One warning on the content page deserves a sticky note. DR4's source list "should be treated as independent" from earlier releases: a DR4 record with the same source_id as a DR3 source is not necessarily the same star. The release is set to include a table, dr3_neighbourhood, with 2.4 billion rows, listing none, one or (rarely) several possible DR3 counterparts for each DR4 source.

The main table also changes meaning. In DR3, gaia_source carried the default single-star model. In DR4, its parameters are consolidated so that "the best-suited model" applies to each source, a binary-star model where that fits better. The results of the core pipelines, for all sources, come in separate all_source tables.

What I take from it

A space agency's archive is not a SaaS product, and what follows are analogies, not claims about ESA's stack. But anyone shipping a data product that is about to grow can copy the sequence:

  1. Ship breaking changes on the old data, weeks before the new data. People debug best against what they already know.
  2. Put quotas in before the rush, and say why. If an agent or a script of yours fires queries in parallel, only five run at once, and beyond a queue of 50 new jobs are rejected. Handle the rejection.
  3. Separate what people query from what they fetch. A 586 GB table and 343 TB of epoch files are two products with two access paths.
  4. Publish the measurements, not only your answer, when your model may be wrong for some records. BH3 sat in the data before it sat in a catalogue.
  5. Scope identifiers to a release and ship the mapping table, instead of letting users assume a stable key.
  6. Announce the window with its reopening time, and when an upgrade degrades service, publish the workaround the same month.

What this does not show

Nothing here says DR4 will open on time, or that the archive will hold its first-day load. The content page calls its table content "under development." I do not know why the June 2025 rollback happened; the notes do not say. The scientific value of DR4 is for astronomers to judge, starting with the release papers: more than 80 in preparation, according to ESA's Gaia bulletin #09, and documentation that bulletin #10 says now exceeds 2,500 pages.

The last observation in DR4 dates from January 20, 2020, and processing started that year. Most of what has happened since is what makes data usable: validation, documentation and, for nearly two years, getting the doors ready.

Sources

A project like this one?

I design and deploy products like this. Let's talk.

Let's talk