Load ionosphere time series from topsStack processor

[yuankailiu]

Outline

This PR is meant to better handle the ionosphere correction in MintPy. Before this PR, the ionosphere data was loaded into MintPy as ionospheric interferograms. We then do an SBAS inversion to get an ionospheric time series to do the correction (history).

As MintPy users, for simplicity, now, we want to load the ionosphere time series products from ISCE2/topStack processor directly. Once this #600 PR on ISCE is merged into topsStack code, the ionosphere final products contain the following files storing under the directory of topsStack processor, $STACK_DIR:

• $STACK_DIR/ion_dates/*.ion: the smooth ionospheric time series for each epoch
• $STACK_DIR/ion_burst_ramp_merged_dates/*.float: the burst ramp time series for each epoch

Once the mintpy .cfg template file is filled with the above two paths, we can load those ionospheric time series directly into two separate time-series files ion.h5 and ionBurstRamp.h5, respectively during the load_data.py stage.

## EXAMPLE
load_data.py -t smallbaselineApp.cfg                   # this will load ifgrams, geometry, and ionosphere time series
load_data.py -t smallbaselineApp.cfg  -l ifg geom ion  # equivalent to above
load_data.py -t smallbaselineApp.cfg  -l ion           # this will just load the ionosphere time series
load_data.py -t smallbaselineApp.cfg  -l geom          # this will just load the geometry

The loaded ionosphere files ion.h5 and ionBurstRamp.h5 will be under inputs/. Later, we can apply the correction by the following:

add.py inputs/ion.h5 inputs/ionBurstRamp.h5 -o inputs/ionTotal.h5
diff.py timeseries.h5 inputs/ionTotal.h5 -o timeseries_ion.h5

Commits

1. new template keywords for the ion time series:

mintpy.load.ionFile: ../ion_dates/*.ion #[path pattern of ionosphere timeseries files]
mintpy.load.ionBurstRampFile = ../ion_burst_ramp_merged_dates/*.float #[path pattern of ionosphere burst ramp timeseries files]

Based on above, I also update the defaults/auto_path.py, defaults/smallbaselineApp.cfg, objects/stack.py.

2. load_data: more flexible loading datasets (changes in load_data.py, cli/load_data.py)

• allow specifying which dataset to load with the load_data.py -l option. choice from {ifg, geom, ion}

3. prep_isce.py and stackDict.py: read ion time-series files from topsStack files

• class timeseriesDict():
• class timeseriesAcqDict(): read(): data_unit & phase2range

4. changes in readfile.py to read these isce .float and .ion files

Description of proposed changes

Reminders

• Fix #xxxx
• Pass Pre-commit check (green)
• Pass Codacy code review (green)
• Pass Circle CI test (green)
• Make sure that your code follows our style. Use the other functions/files as a basis.
• If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
• If adding new functionality, add a detailed description to the documentation and/or an example.

Summary by Sourcery

Add functionality to load ionosphere time series directly from the ISCE2/topStack processor into MintPy, streamlining the process of handling ionospheric corrections. Introduce new configuration options and enhance data loading flexibility.

New Features:

• Enable loading of ionosphere time series products directly from the ISCE2/topStack processor into MintPy, allowing users to handle ionospheric corrections more efficiently.

Enhancements:

• Introduce new template keywords for specifying paths to ionosphere time series and burst ramp files, improving configuration flexibility.
• Enhance the load_data.py script to allow selective loading of datasets, including ionosphere time series, geometry, and interferograms.

[yunjunz]

Thank you @yuankailiu for the big PR. I want to let you know that it may take some time for me to find time and review it. In the meanwhile, I will cut for a new release (version 1.5.2) before merging this PR.

[yunjunz]

@sourcery-ai review and summary

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request introduces significant changes to handle ionosphere correction in MintPy. The main focus is on loading ionosphere time series products directly from ISCE2/topStack processor, rather than loading ionospheric interferograms and performing SBAS inversion. The changes include updates to data loading, new template keywords, and modifications to handle the new ionosphere data format.

Sequence diagram for loading ionosphere time series

sequenceDiagram
    participant User
    participant MintPy
    participant ISCE2
    User->>MintPy: load_data.py -t smallbaselineApp.cfg -l ion
    MintPy->>ISCE2: Request ionosphere time series files
    ISCE2-->>MintPy: Return .ion and .float files
    MintPy->>MintPy: Process and load ionosphere time series
    MintPy-->>User: Save ion.h5 and ionBurstRamp.h5

Loading

Class diagram for new timeseriesDict and timeseriesAcqDict classes

classDiagram
    class timeseriesDict {
        -name: str
        -datesDict: dict
        -dsName0: str
        +get_size(box, xstep, ystep, geom_obj): tuple
        +get_date_list(): list
        +get_dataset_list(): list
        +get_metadata(): dict
        +get_dataset_data_type(dsName): str
        +write2hdf5(outputFile, access_mode, box, xstep, ystep, mli_method, compression, extra_metadata, geom_obj): str
    }

    class timeseriesAcqDict {
        -name: str
        -datasetDict: dict
        -metadata: dict
        +read(family, box, xstep, ystep, mli_method, resize2shape): tuple
        +get_size(family): tuple
        +get_perp_baseline(family): float
        +get_metadata(family): dict
        +get_dataset_list(): list
    }

    timeseriesDict --> timeseriesAcqDict : uses

Loading

File-Level Changes

Change	Details	Files
Introduce new timeseriesDict and timeseriesAcqDict classes for handling ionosphere time series data	Add timeseriesDict class to handle a set of InSAR acquisitions Add timeseriesAcqDict class to handle timeseries, date, and bperp data Implement methods for reading, writing, and manipulating time series data	src/mintpy/objects/stackDict.py
Update load_data.py to handle new ionosphere time series format	Modify read_inps_dict2timeseries_dict_object function to handle ionosphere data Update load_data function to process ionosphere time series separately Add support for loading specific datasets (ifg, geom, ion) based on user input	src/mintpy/load_data.py
Modify prep_isce.py to handle ionosphere time series files	Update prepare_stack function to process ionosphere time series files Add support for reading metadata from .ion and .float files	src/mintpy/prep_isce.py
Update CLI interface for load_data.py	Add -l/--listDset option to specify which datasets to load Remove --geom/--geometry option in favor of the new -l option	src/mintpy/cli/load_data.py
Update configuration files and default settings	Add new template keywords for ionosphere time series files Update auto_path.py with new ionosphere file patterns Modify smallbaselineApp.cfg to include new ionosphere file settings	src/mintpy/defaults/auto_path.py src/mintpy/defaults/smallbaselineApp.cfg
Enhance readfile.py to support new ionosphere data formats	Update read_binary_file function to handle ionosphere phase data Modify get_slice_list function to support new timeseries formats	src/mintpy/utils/readfile.py
Update stack.py to include new ionosphere-related dataset names	Add 'ionosphericDelay' and 'ionosphericBurstRamp' to TIMESERIES_DSET_NAMES	src/mintpy/objects/stack.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @yuankailiu - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider adding unit tests for the new ionospheric correction functionality to ensure robustness and catch potential regressions.
• Review error handling in new functions, especially for file I/O operations, to improve reliability and provide clearer error messages to users.

Here's what I looked at during the review

• 🟡 General issues: 2 issues found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

suggestion: Consider refactoring timeseriesDict to reduce code duplication

The new timeseriesDict class seems to duplicate functionality from ifgramStackDict. Consider refactoring to create a common base class or use composition to reduce code duplication and improve maintainability.

class TimeSeriesDict(IfgramStackDict):
    """
    TimeSeriesDict object for a set of InSAR acquisitions from the same platform and track.
    Inherits from IfgramStackDict to reduce code duplication.
    """

[sourcery-ai]

issue: Reconsider exception handling in read_inps_dict2timeseries_dict_object

Raising an exception with a warning message is an unusual pattern. Consider using a logging framework to log the warning and handle the error condition more gracefully, possibly by returning None or a partial result.