This page gives the recommended procedures for reducing DASCH lightcurve data. The guidance assumes that you are using daschlab but the same steps can be recreated in other analysis systems.

Warning! This guide is still under development and we are still figuring out the best procedures! This page is guaranteed to provide the best-available official reduction instructions … but there is plenty of room for improvement. Use common sense and reach out to the scientific contact to discuss any concerns.

Introduction

Raw DASCH lightcurves must be reduced before being used for scientific analysis. Not only can plates be challenging to work with (there’s a reason we don't use them anymore), but the HCO plate collection is extremely heterogeneous, and it is difficult to implement automated analyses that work reliably across all kinds of plates.

Experience shows that DASCH lightcurves typically achieve RMS precisions of 0.15 mag or so. Any individual photometric measurement is unlikely to be more reliable than this. But, the long-term stability of DASCH measurements (using the APASS refcat) is good, and DASCH often provides thousands of measurements per source.

Rejection Paradigm

At the moment, DASCH lightcurve reduction is primarily a matter of identifying and rejecting bad data points. Once the bad data points have been excised, the remaining measurements should have good accuracy (in the statistical sense) even if their individual precision (once again, in the statistical sense) is low.

In daschlab, lightcurve tables (which are instances of the Lightcurve class) contain an integer column named reject which is initially filled with zeros. daschlab allows you to do whatever you want with this column, but makes the following assumption: a zero value for reject means that the point is good; any other value reject value means that it is bad. Points with non-zero reject values will be omitted from the majority of analyses by default.

(Your humble correspondent calls this the Anna Karenina principle of error handling. You may be familar with the famous first line of the novel; in the Pevear & Volokhonsky translation: “All happy families are alike; each unhappy family is unhappy in its own way”. Likewise, if I know that a data point is good, there isn’t anything more to know about it; but bad points can go bad in any number of ways. By the same token, Unix programs return zero on success, and anything else means an error; the same holds for many C library functions too.)

daschlab does provide infrastructure to help you use the reject column as a set of bit-flags. Several built-in actions allow you to associate a textual “tag” with a bit number that will be logically OR’ed into the reject values of certain points. For instance:

from astropy import units as u

# [ ... standard daschlab setup; `lc` is a Lightcurve object ... ]

lc.reject.sep_above(15 * u.arcsec, tag="sep")

This code will select any lightcurve points whose fitted separation is farther than 15 arcsec from the mean source position, and set one of the bits in their reject column to non-zero. The exact bit chosen will depend on how many other “tags” have already been used, but all points rejected with the "sep" tag will have the same bit set.

1. Source Splitting

Before analyzing a particular lightcurve, you should consider whether your target may be affected by the “source splitting” known issue.

In short, the DASCH photometric reference catalogs (like any other catalog) may contain entries for a group of nearby sources that blend together in most DASCH plates. When this happens, the photometric pipeline has to choose which catalog entry it will assign the detection to, and it doesn't always do this consistently. The result may be that a set of valid detections of a single astronomical object may be split across several separate lightcurve tables.

In order to evaluate whether this may affect you, it helps plot the contents of the reference catalog near your target. In daschlab this is accomplished with:

# make sure that the session is connected to WWT
await sess.connect_to_wwt()

sess.refcat().show()

Zoom in on your target and keep an eye out for catalog entries hiding near your target of interest.

If there are any candidate objects that might be “stealing” your lightcurve points, you should fetch their lightcurves and check their contents. Since the daschlab catalog extract is ordered by proximity to your target position, the “local IDs” of these items will be the first few integers. For instance, to print summary information about the lightcurve of the catalog object that is second-closest to your target position, run:

sess.lightcurve(1).summary()

You should assess the positions and magnitudes of the detections in such a lightcurve to decide whether they correspond to your actual target or not.

As of this writing (April 2024), daschlab does not provide any facilities to merge together lightcurves when the source-splitting phenomenon occurs. It should! Please reach out if you would like to help implement this feature.

See also the Source Splitting Known Issue.

2. daschlab standard rejections

The daschlab package provides a method, Lightcurve.apply_standard_rejections(), that does what it says: it applies a variety of rejections that can be done automatically and reliably.

lc.apply_standard_rejections()

Warning: Like this guide, this method is a work in progress. Even when it is more developed, it will never be able to fully clean your data. Its behavior will change as this guide is refined.

3. AFLAG rejections

The AFLAGS column includes a variety of data quality flags that you can use to assess the trustworthiness of points.

TODO: documentation of the AFLAGS is still lacking. The flags() method on an individual lightcurve row will print out a textual report of the AFLAGS and other flag fields in the row:

# Get a flags report for the last point in the lightcurve
>>> lc[-1].flags()
<<< AFLAGS: 0x10088080
<<<   bit  8: BAD_PLATE_QUALITY                - Plate fails general quality checks
<<<   bit 16: BIN_DRAD_UNKNOWN                 - Object's spatial bin has unmeasured `drad`
<<<   bit 20: UNCERTAIN_CATALOG_MAG            - Magnitude of the catalog source is uncertain/variable
<<<   bit 29: LARGE_SMOOTHING_CORRECTION       - Smoothing correction is suspiciously large
<<< BFLAGS: 0x406A0000
<<<   bit 18: LOWESS_CAL_APPLIED               - Lowess calibration has been applied
<<<   bit 20: EXTINCTION_CAL_APPLIED           - Extinction calibration has been applied
<<<   bit 22: COLOR_CORRECTION_APPLIED         - Color correction has been applied
<<<   bit 23: COLOR_CORRECTION_USED_METROPOLIS - Color correction used the Metropolis algorithm
<<<   bit 31: PROMO_APPLIED                    - Catalog position has been corrected for proper motion
<<< LocalBinReject: 0x00000005
<<<   bit  1: LARGE_CORRECTION                 - Local correction is out of range
<<<   bit  3: LARGE_DRAD                       - Majority of stars have a high `drad`
<<< PlateQuality: 0x00000291
<<<   bit  1: MULTIPLE_EXPOSURE                - Multiple-exposure plate
<<<   bit  5: PICKERING_WEDGE                  - Pickering Wedge plate
<<<   bit  8: MAG_DEP_CAL_UNAVAILABLE          - Magnitude-dependent calibration unavailable
<<<   bit 10: NARROW_TELESCOPE                 - Narrow-field-telescope plate

Historically, DASCH tools have default to rejecting points with any of the following AFLAGS:

• bit 7, HIGH_BACKGROUND
• bit 12, LARGE_ISO_RMS
• bit 13, LARGE_LOCAL_SMOOTH_RMS
• bit 14, CLOSE_TO_LIMITING
• bit 16, BIN_DRAD_UNKNOWN

These AFLAGS are rejected by Lightcurve.apply_standard_rejections().

All AFLAGS bits are “bad”, in the sense that zero values are associated with more-reliable data and one values are associated with less-reliable data. However, these flags are subject to both false positives and false negatives. In general, very few points have AFLAGS exactly equal to zero.

To reject points associated with a particular AFLAG bit, use code like:

from daschlab.lightcurves import AFlags

# [...]

lc.reject.any_aflags(AFlags.SUSPECTED_DEFECT, tag="aflags")

4. Astrometric rejections

Some DASCH FITS files have incorrect WCS (Incorrect Astrometry known issue) and in other cases the pipeline fails to reject spurious detections. Both of these issues can be mitigated by rejecting points that have fitted positions far from the mean source position.

To investigate this issue, try plotting magnitude versus separation:

lc["sep"] = lc.mean_pos().separation(lc["pos"]).arcsec
lc.plot(x_axis="sep")

If the magnitudes seem to go haywire above a certain separation, you can flag them:

from astropy import units as u

# [ ... ]

lc.reject.sep_above(15 * u.arcsec, tag="sep")

The separation at which the data go bad is typically around 10 or 20 arcsec.

5. Source shape rejections

With as many plates as DASCH has, there are going to be some number that have a speck of dust exactly where your source should be (Undetected Plate Defects known issue). These can be rejected by comparing the shape of the source image to the shapes of nearby stars.

FIXME! We can't provide contextual information right now, so you have to do this by looking at cutout images! That is lame!

6. Artificially deep limits

Most DASCH lightcurves will include a few upper limits that are well below the typical brightness of your target. These generally occur for one of two known reasons. Either it is the Incorrect Astrometry known issue, and the pipeline has failed to detect your source because it actually lies at a different image location; or it is the Missing Lightcurve Points known issue, and the pipeline has detected your source at the right location, but the detection record is missing from the compiled lightcurve database.

With the current DASCH data processing architecture, neither problem can be easily solved without staff intervention. The best available course of action is to reject these points. If a potential measurement is critical to your science, contact the team to discuss reprocessing of the plate in question.

7. Low-resolution plates

The plates in DASCH are quite heterogeneous in terms of their spatial resolution, and plates with poor spatial resolution can have various issues with their photometry. In particular, while the DASCH photometry pipeline attempts to handle blends judiciously, blended sources may appear systematically brighter in low-resolution plates. This phenomenon can induce false time-dependent trends in lightcurves because there is time clustering in when low-resolution plates were obtained — e.g., the “meteor” campaigns of the 1950’s–70’s.

To a very good approximation, the series designation of a plate tells you about its spatial resolution. The plates are broadly grouped into three series kinds: “narrow-field” series (with plate scales less than 400 arcsec/mm), “patrol” series (plate scales 400–900 arcsec/mm), and “meteor” series (scales 900 arcsec/mm and larger). The method Plates.series_info() reports information about the different plate series represented in a plate list, including plate-scale data and a series-kind designation. So, for a given target, you can get a summary in an interactive program by running:

sess.plates().series_info()

The lightcurve selection system includes selectors narrow(), patrol(), and meteor(), as does the plate-list selection system. To blanket-reject plates from low-resolution series, you can use constructs like:

# reject meteor plates; allow narrow and patrol
lc.reject.meteor(tag="meteor")

# reject meteor and patrol plates; allow narrow
lc.reject.not.narrow(tag="non-narrow")

See also the Undetected Blends Known Issue.

8. Non-blue color responses

Most plates in the DASCH corpus were made with blue-sensitive photographic emulsions, and their photometric response is close to the Johnson B band, whose definition in fact traces back to the Harvard plate collection. However, some plates used different emulsions and/or filters. While the DASCH pipeline attempts to account for different color responses, color corrections are inevitably dicey. Visual inspection can also end up being misleading: what appears to be a stellar flare may in fact be a red-emulsion observation of a red object.

While the DASCH photometric pipeline computes color sensitivity parameters empirically, as of this writing (April 2024) the DASCH data products unfortunately do not surface this information very effectively.

One hint as to the emulsion used for a plate is to look at its class, if it is logged. Classes involving the digit “2” generally indicate the use of a red emulsion and/or filter: C2, D2, F2, L2, L2S, M2, M2S, MS2, S2, d2. Classes involving the digit “1” generally indicate the use of a yellow emulsion and/or filter: C1, D1, F1, FD1, L1, M1, S1, d1.

Writing on the plate jacket can also indicate which emulsion was used. A letter “E” may indicate a red-sensitive IIa-E emulsion, while “O” might indicate the usual blue-sensitive IIa-O. Newer emulsion designations, whose usage started around the 1970’s, are abbreviated “J”, “F”, “N”, and “D”.

If a certain individual lightcurve point is important to your science, the current recommendation is to examine the plate image, class, and jacket to check whether it may be derived from a non-blue plate.

See also the Unspecified Emulsions Known Issue.

99. Everything else

This guide is incomplete! It will grow over time as we continue codifying best practices. If you have any suggestions, please contact the team and let them know!