[gnuastro-devel] [task #15047] Server-client Gnuastro operation

[Mohammad Akhlaghi]

Follow-up Comment #1, task #15047 (project gnuastro):

This is just further elaboration on this task.

The compression ratios reported in the previous commit are relative to the
actual input file size (2.9Gb), not NoiseChisel's output on it. Comparing with
the actual NoiseChisel output prior to compression (which was 724Mb), the
compression ratios become: 36.6 and 49.3.

With this example, I was trying to emphasize that only one higher-level step
(detection segmentation map from NoiseChisel) allows for such a large
compression level, compared to getting the raw data and running NoiseChisel
over it. 

These segmentation maps allow fast server-side access to the data to produce
high-level measurements over any interesting object. See arXiv:1611.06387
<https://arxiv.org/abs/1611.06387v1> or its corresponding slides
<http://www.adass2016.inaf.it/images/presentations/01_Akhlaghi.pdf> in the
26th ADASS (2016). As described there, the segmentation maps can also account
for de-blending, I also have some thoughts on accounting for the PSF.

In other words, it won't  be necessary to do a very large set of pre-defined
measurements over the survey in one run to generate one catalog (heavy for the
server, with catalogs sometimes becoming larger than the actual images). A
single generic measurement forces users to limit their science to the
information in that pre-defined catalog. Their only alternative  is to that
catalog is accessing the raw images (which need a lot of expertise and time to
process on the server). 

In this scenario, using Gnuastro's MakeCatalog, people can define their own
custom measurement over any part of the images they need: at different times,
with different filters, or even using custom segmentation maps (like circular
apertures). 

MakeCatalog is a very cheap operation for the server (it is multi-threaded,
and will only read the pixels over the segmentation map into the server's
memory, not a whole image). Note that the majority of the sources are
faint/small (galaxy luminosity function). Also see task #13557 on an efficient
way to find images for mapping of the segmentation map over an image.

As a summary, once we implement this server-client operation within
MakeCatalog, the only intermediary file necessary for archiving (possibly
locally) to start many high-level science applications (certainly not all!) is
the segmentation map (which can be highly compressed). 

Of course, the statement above doesn't reject the idea of producing an
over-all generic survey catalog. That is certainly necessary/efficient for
initial sample selections. The discussion here allows customization (removing
the need for columns in the generic catalog which differ very slightly), and
allows users to be more creative and complement those initial measurements
with more accurate ones for their particular science case.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/task/?15047>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/