DR7: Bulk Data Formats

Some projects using the DASCH data might be accomplished most efficiently by obtaining copies of the data that underlie the various DASCH data access APIs. This route is generally discouraged because these data are formatted in ways that are less user-friendly than what comes out of the APIs. Before seeking access to the bulk data, contact the DASCH Astrophysics email list to request an increase to your API rate limit, and follow up with a request for bulk access if it truly seems necessary.

This page gives some guidance as to the file formats. In addition, the GitHub repository pkgw/dasch-bulk-data provides sample Rust code that demonstrates basic usage of some of these files. However, this sample code is not designed to support efficient local analysis — you will need to use the sample code, the other DASCH documentation, and above all your judgment to decide how to use the bulk files to accomplish your research goals. Furthermore, unfortunately there is not currently funding for anyone to be able to provide support to users who wish to explore the DASCH bulk data products. You should expect to have to figure a lot of things out on your own.

The bulk data files that may be provided include:

The following kinds of files are also available for bulk access but do not (yet) have additional documentation:

Amazon S3 URLs are provided below for reference, but the objects in these buckets are not publicly accessible, so you can't just fetch the files without prior arrangement.

Plates Table Dump

A dump of the AWS DynamoDB table that supports the POST /dasch/dr7/queryexps Web API endpoint. It can be delivered as a single file:

  • S3 url: s3://dasch-prod-user/dasch-dr7-dumps/plates_dynamodb_data.jsonl.gz
  • Size: 1052530528 bytes (~1.0 GiB)
  • MD5 digest: e0dc4068fe10c5c52e3c044cd39c47e1

This file is formatted as gzipped JSON Lines data. That is, each line is a parseable as JSON that describes information about a single plate in the Harvard collection. There are 556,406 lines (database rows) in the file.

The JSON records use the DynamoDB typed JSON format. This is a somewhat verbose encoding that uses JSON object structures to express a slightly larger range of types than are defined in the JSON standard. These data can be parsed with a standard JSON parser, but it is helpful to use a DynamoDB-aware decoder that can assist with more complex types like binary blobs, which are represented as base64-encoded strings.

The individual plate records can have a complex structure. Available data include

Refer to plates_dump.rs in the pkgw/dasch-bulk-data sample code for a more formalized schema of the file contents (but not their semantics). Each plate is associated with one or more “exposures”, with information derived from observing logbooks, and zero or more “solutions”, with information derived from astrometric solutions of the scanned DASCH image. As is usual, there can be exposures that do not match to solutions, as well as solutions that do not match to exposures.

The file plates_dynamodb_data_tiny_sample.jsonl.gz provides a tiny sample of the format of this dump file.

Reference Catalog Dumps

Dumps of the AWS DynamoDB tables that support the POST /dasch/dr7/querycat Web API endpoint. There are two files:

  • APASS dump:
    • S3 url: s3://dasch-prod-user/dasch-dr7-dumps/apass_dynamodb_data.jsonl.gz
    • Size: 11564842182 bytes (~11 GiB)
    • MD5 digest: 789c6df3de8c3fa5d7cd6182633618a1
    • Number of lines / rows: 287,864,850
  • ATLAS dump:
    • S3 url: s3://dasch-prod-user/dasch-dr7-dumps/atlas_dynamodb_data.jsonl.gz
    • Size: 34155080603 bytes (~32 GiB)
    • MD5 digest: d44d603c3f9385474828fed1bc7c7c19
    • Number of lines / rows: 992,637,840

These files have the same overall structure as the plates table dump: gzipped, JSON Lines structure, DynamoDB typed JSON.

The individual rows have a fairly flat structure that fairly directly provides most of the columns described in the Catalog Data Table Columns documentation. There is also file-offset information needed to efficiently retrieve lightcurve information from the bulk lightcurve data. Refer to refcat_dump.rs in the pkgw/dasch-bulk-data sample code for a more formalized schema of the file contents (but not their semantics).

The file apass_dynamodb_data_tiny_sample.jsonl.gz provides a tiny sample of the format of this dump file.

Photometry Data

The raw DASCH DR7 photometry data that support the POST /dasch/dr7/lightcurve and POST /dasch/dr7/platephot Web API endpoints. These are stored in custom binary formats. Deliverable products include:

  • Limiting-magnitude index files
    • APASS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-apass/lim_aws.idx
      • Size: 2637248 bytes (~2.5 MiB)
      • MD5 digest: 1b6422f9b468b63e7adc4f58e52bef2c
    • ATLAS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-atlas/lim_aws.idx
      • Size: 2637248 bytes (~2.5 MiB)
      • MD5 digest: 8323c5f33e10959c02a7770ef43e5c8f
  • Limiting-magnitude data payload files
    • APASS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-apass/limiting.dat
      • Size: 23968216608 bytes (~23 GiB)
      • MD5 digest: dd8373fbf4223fe8fe3f829e9a0f64cf
    • ATLAS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-atlas/limiting.dat
      • Size: 29562366048 bytes (~28 GiB)
      • MD5 digest: ca1dc21a9e534397c89cffd29f818d63
  • "Mega-index" files (needed for per-plate photometry extraction)
    • APASS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-apass/megaindex.idx
      • Size: 675865548 bytes (~645 MiB)
      • MD5 digest: 823f74ea49de7b45984c10aa4829640c
    • ATLAS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-atlas/megaindex.idx
      • Size: 675865548 bytes (~645 MiB)
      • MD5 digest: baca9082b12db29e394b70fac55e3dde
  • Photometry data payload files
    • APASS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-apass/mags/*
      • Number of files: 165,007
      • Total size: about 7.2 TiB
      • For validation information, consult the DASCH Digital Inventory
    • ATLAS:
      • S3 url: s3://dasch-prod-user/dasch-dr7-phot-atlas/mags/*
      • Number of files: 165,007
      • Total size: about 8.6 TiB
      • For validation information, consult the DASCH Digital Inventory

These data combine to allow the extraction of most of the data columns mentioned in the Lightcurve Data Table Columns documentation. If you only care about limiting-magnitude data, you do not need the mega-index or photometry data payload files.

To understand how to use the files, consult the pkgw/dasch-bulk-data sample code, especially phot_data.rs, extract_lightcurve.rs, and extract_exp_phot.rs.

Note that these files are structured to support efficient retrieval of the lightcurves of cataloged sources. (This requires that you also obtain the reference catalog database dumps, because they store important file-offset information.) In order to analyze photometry on a plate-by-plate or exposure-by-exposure basis, which is the only way to retrieve photometry of uncataloged (i.e. transient) sources, you will likely need to process the provided files to restructure their contents into a format that is better optimized for such queries.