Simulate a plan and optionally download the results.


List uploaded plans.


Clean and Create New Configuration for a Given Plan.


Update Configuration for a Given Plan.


Delete an activity plan by its id.


Delete all activity plans.


Upload a single mission model from a .jar file.


Delete a mission model by its model id.


Delete all mission models.


List uploaded mission models.


Create decorator to mark a method as the handler of a VCS.


TAG[.postDISTANCE[.dev0]] .

    The ".dev0" means dirty.

    Exceptions:
    1: no tags. 0.postDISTANCE[.dev0]


Do main VCS-independent setup function for installing Versioneer.


check if the TPF file exists and download it if not


reads file with values True or False


help function to load header from tpf


    ## inputs:
    - `target_flux` - target flux
    - `target_kplr_mask` - kepler mask of the target star
    - `neighbor_flux_matrix` - fitting matrix of neighbor flux
    - `time` - array of time
    - `epoch_mask` - epoch mask
    - `covar_list` - covariance list
    - `margin` - size of the test region
    - `poly` - number of orders of polynomials of time(zero order is the constant level)
    - `l2_vector` - array of L2 regularization strength
    - `thread_num` - thread number
    - `prefix` - output file's prefix

    ## outputs:
    - prefix.npy file - fitting fluxes of pixels


    print(dif_cpm.shape)
    plt.plot(dif_cpm, '.k')
    plt.show()


    ## inputs:
    - `predictor_flux_matrix` - fitting matrix of neighbor flux
    - `l2_vector` - array of L2 regularization strength
    ## outputs:


read multiple grids from a single file


find index that is closest to given BJD


Calculate pixel coordinates for given (RA,Dec) for all epochs.
        (RA,Dec) can be floats, lists, or numpy.arrays.

        For input of length n_stars the output is 2 arrays (first for X,
        second for Y), each of shape (n_epochs, n_stars).
        If inputs are floats than output is 2 1D arrays.


for given (RA,Dec) calculate position for every epoch
        and report weighted average


calculate pixel coordinates for given (RA,Dec) which can be floats, lists, or numpy.arrays


get list of EPIC IDs in numerical order


search for all pixels within given radius around RA/Dec position and store resulting mask mask


find indexes of all matches with given column and row


loads header information and data from given file


path to the TPF file


check if file is where it should and download if not


return array with a list of all pixels


check if given (column,row) pixel is inside the area covered by this TPF file


check if we have data for given (column,row) pixel


prepare vectors with some numbers


gives array that translates index pixel into row number


gives array that translates index pixel into column number


finds index of given (row, column) pixel in given file - information necessary to extract flux


saves the time vector and the flux for a single pixel into a file


saves:
        the time vector, flux vector, and flux_err vector
        for a single pixel into a file


find in which triangle given point is located and
        calculate weights for barycentric interpolation


barycentric interpolation on a traiangle grid


get predictor_matrix, run CPM, calculate dot product and difference of target_flux and fit_flux


queries for EPID ids that are within radius from given ra/dec


finds at least minimum_n EPIC ids closest to given ra/dec; up to n=except_largest largest files are removed in order not to read huge files with almost no data


evaluate powers of given values and return as a table: [1, x, y, x^2, xy, y^2] for deg = 2


evaluate 2d polynomial without summing up


evaluate 2d polynomial


fits 2 polynomials: x_out = f(x_in, y_in, coefs) and same for y_out


    Plot given 3D matrix in subpanels. Note that 3rd component of matrix.shape
    must be the same as time.size i.e., matrix.shape[2]==time.size

    TO_BE_DONE: add options to specify symbols, add errorbars, add mask options

    example usage:
    import matplotlib.pyplot as plt
    fig = plt.gcf()
    fig.set_size_inches(50,30)
    plot_matrix(fig, time, matrix)
    plt.savefig("file_name.png")

    to construct input data use e.g.:
    tpfdata.TpfData.directory = TPF_DIRECTORY_NAME
    tpf = tpfdata.TpfData(epic_id=SOME_epic_id, campaign=SOME_campaign)
    time = tpf.jd_short
    matrix = tpf.get_fluxes_for_square(pix_y, pix_x, half_size=3) # 3 gives 7x7 sublots


        Solve a linear system as fast as possible.

        :param A: ``(ndata, nbasis)``
        The basis matrix.

        :param y: ``(ndata)``
        The observations.

        :param yvar:
        The observational variance of the points ``y``.

        :param l2:
        The L2 regularization strength. Can be a scalar or a vector (of length
        ``A.shape[1]``).


reads file with matrix in format like:
    0 0 123.454
    0 1 432.424
    ...
    into numpy array


add one more instance of TpfData


returns an instance of TpfData corresponding to given epic_id


for each epic_id in the list, construct TPF object and add it to the set


read data from file and apply add_tpf_data_from_epic_list()


limit self._epic_ids to ones in epic_list


    This is a copy from the pymagicc package. Credits to Robert Giesecke.
    Input: Any MAGICC .OUT file
    Output: Pandas DataFrame


solid ice discharge as used in Nauels et al. (2017, ERL).


 handles several scenarios for one parameter set.
        We assume that there is one global maximum ice volume that can be lost
        (not dependent on parameter set.)


 OLD for reference:
    solid ice discharge as used in Nauels et al. 2017 for the
    Greenland solid ice discharge. Ice loss is exponentially dependent
    on the driving temperature in Nauels et al. 2017.


A safe, general minimisation function.


    Create a plot showing the walker values for each parameter at every step.

    :param xs:
        The samples. This should be a 3D :class:`numpy.ndarray` of size
        (``n_walkers``, ``n_steps``, ``n_parameters``).

    :type xs:
        :class:`numpy.ndarray`

    :param labels: [optional]
        Labels for all the parameters.

    :type labels:
        iterable of strings or None

    :param truths: [optional]
        Reference values to indicate on the plots.

    :type truths:
        iterable of floats or None

    :param truth_color: [optional]
        A ``matplotlib`` style color for the ``truths`` markers.

    :param burn: [optional]
        Reference step to indicate on the plots.

    :type burn:
        integer or None

    :param alpha: [optional]
        Transparency of individual walker lines between zero and one.

    :type alpha:
        float

    :param fig: [optional]
        Overplot onto the provided figure object.

    :type fig:
        :class:`matplotlib.Figure` or None

    :raises ValueError:
        If a ``fig`` is provided with the incorrect number of axes.

    :returns:
        The chain figure.

    :rtype:
        :class:`matplotlib.Figure`


    Plot the meana cceptance fractions for each MCMC step.

    :param mean_acceptance_fractions:
        The acceptance fractions at each MCMC step.

    :type mean_acceptance_fractions:
        :class:`numpy.array`

    :param burn: [optional]
        The burn-in point. If provided, a dashed vertical line will be shown at
        the burn-in point.

    :type burn:
        int

    :param ax: [optional]
        The axes to plot the mean acceptance fractions on.

    :type ax:
        :class:`matplotlib.axes.AxesSubplot`

    :returns:
        The acceptance fractions figure.


    Plot the autocorrelation function for each parameter of a sampler chain.

    :param chain:
        The sampled parameter values.

    :type chain:
        :class:`numpy.ndarray`

    :param index: [optional]
        Index to calculate the autocorrelation from.

    :type index:
        int

    :param limit: [optional]
        Maximum number of MCMC steps to display. By default half of the chain
        will be shown.

    :type limit:
        int

    :param fig: [optional]
        Figure class to use.

    :type fig:
        :class:`matplotlib.Figure` or None

    :param figsize: [optional]
        The figure size (x-dimension, y-dimension) in inches.

    :type figsize:
        tuple or None


    Project the maximum likelihood values and sampled posterior points as
    spectra.

    :param model:
        The model employed.

    :type model:
        :class:`sick.models.Model`

    :param data:
        The observed spectra.

    :type data:
        iterable of :class:`sick.specutils.Spectrum1D` objects

    :param theta: [optional]
        The optimised model parameters given the data. Either theta
        or chain should be given.

    :type theta:
        dict

    :param chain: [optional]
        The chain of sampled parameters.

    :type chain:
        :class:`numpy.ndarray`

    :param extents: [optional]
        The wavelength extents to plot for each channel in the form of
        [(min_chan_1, max_chan_1), ..., (min_chan_N, max_chan_N)]

    :type extents:
        tuple or None

    :param uncertainties: [optional]
        Show uncertainty of the data points.

    :type uncertainties:
        bool

    :param title: [optional]
        Title to set for the top axes.

    :type title:
        str

    :param fig: [optional]
        Overplot onto the provided figure object.

    :type fig:
        :class:`matplotlib.Figure` or None

    :param figsize: [optional]
        The figure size (x-dimension, y-dimension) in inches.

    :type figsize:
        tuple or None

    :raises ValueError:
        If a ``fig`` is provided with the incorrect number of axes.

    :raise TypeError:
        If the ``data`` are not provided in the correct type.

    :returns:
        The projection figure.

    :rtype:
        :class:`maplotlib.Figure`


helper function for simple_round (a factory for simple_round functions)


decorator for rounding a function's input argument and keywords to the
    given precision *tol*.  This decorator always rounds to a floating point
    number.
    Rounding is only done for arguments or keywords that are floats.
    For example:
    >>> @simple_round(tol=1)
    ... def add(x,y):
    ...   return x+y
    ...
    >>> add(2.54, 5.47)
    8.0
    >>>
    >>> # does not round elements of iterables, only rounds at the top-level
    >>> add([2.54, 5.47],['x','y'])
    [2.54, 5.4699999999999998, 'x', 'y']
    >>>
    >>> # does not round elements of iterables, only rounds at the top-level
    >>> add([2.54, 5.47],['x',[8.99, 'y']])
    [2.54, 5.4699999999999998, 'x', [8.9900000000000002, 'y']]


Remove the downloaded files, and remove the created figures.


Creates a copy of the object


Load a Spectrum1D from a given filename.

        :param filename:
            The path of the filename to load. Can be either simple 1D FITS files
            or an ASCII filename.

        :type filename:
            str


        Save the `Spectrum1D` object to the specified filename.

        :param filename:
            The filename to save the Spectrum1D object to.

        :type filename:
            str

        :param clobber: [optional]
            Whether to overwrite the `filename` if it already exists.

        :type clobber:
            bool

        :raises IOError:
            If the filename exists and we were not asked to clobber it.


Cross correlate the spectrum against a set of templates.


Cross-correlate the observed spectrum against template fluxes.


    Resample a spectrum to a new wavelengths map while conserving total flux.

    :param old_wavelengths:
        The original wavelengths array.

    :type old_wavelengths:
        :class:`numpy.array`

    :param new_wavelengths:
        The new wavelengths array to resample onto.

    :type new_wavelengths:
        :class:`numpy.array`


        Save the model configuration to a YAML-formatted file.

        :param filename:
            The filename to save the model configuration to.

        :type filename:
            str

        :param clobber: [optional]
            Clobber the filename if it already exists.

        :type clobber:
            bool

        :returns:
            True

        :raises:
            IOError


        Return a scaling factor to convert the redshift into the user's
        preferred units.


LaTeX-ify labels based on information provided in the configuration.


Return the model parameters.


Return the channels that match the given wavelength array.


        Sort the data in blue wavelengths to red, and ignore any spectra that
        have entirely non-finite or negative fluxes.


Apply pre-defined masks to the data.


Apply pre-defined model masks.


        Match observed data to a channel, and return possibly superfluous model
        parameters.


Generate an initial proposal distribution around the point theta.


        Cast a new model from this model. The new model might have a restricted
        wavelength range, different channels, larger sampling/binning, and/or a
        lower spectral resolution.


Return parameters specific to a channel for a model.


    Create a new *sick* model from files describing the parameter names, fluxes,
    and wavelengths.


Estimate the model parameters, given the data.


        Infer the model parameters, given the data.
        auto_convergence=True,
        walkers=100, burn=2000, sample=2000, minimum_sample=2000,
        convergence_check_frequency=1000, a=2.0, threads=1,


        Pre-create binning matrix factories. The following options need to be
        followed on a per-matched channel basis.

        Options here are:

        1) If we have no redshift or resolution parameters to solve for, then
           we can just produce a matrix that will be multiplied in.
           Inputs: none
           Outputs: matrix

        If fast_binning is turned on:

        2) Provide the wavelengths, redshift, resolution (single value) and
           flux. Returns the convolved, interpolated flux.
           Inputs: flux, wavelengths (e.g., redshift), resolution
           Outputs: normalised flux.

        If fast_binning is turned off:

        3) If we *just* have redshift parameters to solve for, then we can
           produce a _BoxFactory that will take a redshift z and return
           the matrix
           [This function will have a LRU cacher and should be removed after]
           Inputs: redshift
           Outputs: matrix

        4) If we have redshift parameters and resolution parameters, then we
           can produce a _BlurryBoxFactory that will take a redshift z and
           Resolution (or resolution coefficients!) and return a binning matrix
           [This function will have a LRU cacher and should be removed after]
           Inputs: redshift, Resolution parameter(s)
           Outputs: matrix

        5) If we *just* have resolution parameters to solve for, then we
           can produce a _BlurryBoxFactory as well, because by default z=0.
           [This function will have a LRU cacher and should be removed after]
           Inputs: resolution parameter(s)
           Outputs: matrix

        Because these options require different input/outputs, and the LRU
        cachers need to remain wrapped around simple functions, we may need a

        Consistent lambda function:
        lambda(obs_wavelength, obs_flux, z=0, *R)


Optimise the model parameters, given the data.


Initialise a spectrum interpolator.


Intepolate model intensities at the given data points.


        Train the model in a Cannon-like fashion using the grid points as labels
        and the intensities as normalised rest-frame fluxes.


        Train the model in a Cannon-like fashion using the grid points as labels
        and the intensities as normalsied rest-frame fluxes within some local
        regime.


Train the Cannon coefficients.


        Return a human-readable representation of the Cannon label vector.

        label_vector should be [[(1,2), (2,1)], [(1,3)]] etc.


    Recursively update a nested dictionary.

    :param original:
        The original nested dictionary to update.

    :type original:
        dict

    :param new:
        The nested dictionary to use to update the original.

    :type new:
        dict

    :returns:
        The updated original dictionary.

    :rtype:
        dict


    Return the unique items of a list in their original order.

    :param original_list:
        A list of items that may have duplicate entries.

    :type original_list:
        list

    :returns:
        A list with unique entries with the original order preserved.

    :rtype:
        list


    Return a digit in a human-readable string form.

    :param number:
        The obfuscated number.

    :type number:
        float

    :returns:
        A more human-readable version of the input number.

    :rtype:
        str


Estimate the exponential auto-correlation time for all parameters in a chain.


Estimate the integrated auto-correlation time for all parameters in a chain.


Command line parser for *sick*.


Announce theta values to the log.


    Return a default filename prefix for output files based on the input files.

    :param filenames:
        The input filename(s):

    :type filenames:
        str or list of str

    :returns:
        The extensionless common prefix of the input filenames:

    :rtype:
        str


Return a point estimate of the model parameters theta given the data.


Optimise the model parameters.


Infer the model parameters.


Aggregate the results from multiple analyses into a single file.


Parse arguments and execute the correct sub-parser.


Command line parser for dealing with *sick* models.


Create a model by recasting an existing model.


Create a model from wavelength and flux files.


        Round down minimums and up maximums to the nearest resolution in m.
        Returns new Bounds object.


Return middle point coordinate for the bounds.


Define bounding box for provided coordinates.


Create new grid given the spacing and bounding box for the region.


Transform coordinates from proj1 to proj2 (EPSG num).


Create command-line argument parser and parse arguments.


Clear current set of cube layers.


Reset all internal data structures.


        Send request to ITS_LIVE API to get a list of granules to satisfy polygon request.
        Or instead the testing purposes use a list of provided granules through input
        JSON file.

        num_granules: int
            Number of first granules to examine.
            (ATTN: This is a temporary solution to a very long time to open remote granules.
            Should not be used when running the code in production mode.)


        Skip duplicate granules (the ones that have earlier processing date(s))
        for the same path row granule for Landsat8 and Landsat9 data only.

        Examples of the Landsat image pair filename with one of the images from L89 mission group:
        LC08_L1GT_007011_20130819_20200912_02_T2_X_LC08_L1GT_007011_20140806_20200911_02_T2_G0120V02_P044.nc
        LC08_L1TP_013010_20130330_20200913_02_T1_X_LE07_L1TP_012010_20130627_20200907_02_T1_G0120V02_P003.nc


        * Exclude datacube granules, and all skipped granules in existing datacube
        (empty data, wrong projection, duplicate middle date) from found granules.
        * Identify if any of the skipped double mid_date granules from "found_urls"
        are already existing layers in the datacube. Need to mark such layers
        to be deleted from the datacube.
        * Identify if current cube layers and remaining found_urls have duplicate
        mid_date - register these for deletion from the datacube if they appear
        as datacube layers.

        Return:
            granules: list
                List of granules to update datacube with.
            layers_to_delete: list
                List of existing datacube layers to remove.


Examine the layer if it qualifies to be added as a cube layer.


        Initialize output store for the datacube. It removes existing local
        store if it exists already. This method is useful only if create_* methods
        are called directly by the user - to guarantee that datacube is created
        from the scratch.


        Check if datacube exists. The datacube can be on a local file system or
        in AWS S3 bucket.


        Read datacube from provided store. The method detects if S3 bucket
        store or local Zarr archive is provided, and reads xarray.Dataset from
        the Zarr store.


Create new or update existing datacube.


        Update velocity pair datacube by reading and pre-processing new cube layers in parallel.

        output_dir: str
            Local datacube Zarr store to write updated datacube to.
        output_bucket: str
            AWS S3 bucket if datacube Zarr store resides in the cloud.
        num_granules: int
            Number of first granules to examine.
            TODO: This is a temporary solution to a very long time to open remote granules.
                    Should not be used when running the code in production mode.


        Create velocity pair datacube by reading and pre-processing cube layers in parallel.

        output_dir: str
            Directory to write datacube to.
        output_bucket: str
            AWS S3 bucket if datacube Zarr store resides in the cloud.
        num_granules: int
            Number of first granules to examine.
            ATTN: This is a temporary solution to a very long time to open remote granules. Should not be used
                    when running the code in production.


        Pre-process ITS_LIVE dataset in preparation for the cube layer.

        ds: xarray dataset
            Dataset to pre-process.
        ds_url: str
            URL that corresponds to the dataset.

        Returns:
        cube_v:     Filtered data array for the layer.
        mid_date:   Middle date that corresponds to the velocity pair (uses date
                    separation as milliseconds)
        empty:      Flag to indicate if dataset does not contain any data for
                    the cube region.
        projection: Source projection for the dataset.
        url:        Original URL for the granule (have to return for parallel
                    processing: no track of inputs for each task, but have output
                    available for each task).


        Helper method to collect attributes for v-related data variables.

        Inputs:
        =======
        var_name: Name of the variable (vx, vy, va, vr).
        mid_date_coord: Middle date coordinate for collected data.

        Returns:
        =======
        List of new data variables names that were created to correspond to
        the attributes of "var_name" variable. These variables names are used
        to set their encoding parameters when storing datacube to the Zarr store.


Helper method to clean up attributes for M1[12]-related data variables.


Display current memory usage.


        Combine selected layers into one xr.Dataset object and write (append) it
        to the Zarr store.


        Format statistics of the run. Don't display statistics if using
        granules as provided in the input JSON file.


Read Dataset from the file and pre-process for the cube layer.


        Read Dataset from the S3 bucket and pre-process it for the cube layer.
        Return re-tried exceptions messages, if any, and cube layer information.

        each_url: Granule S3 URL.
        s3: s3fs.S3FileSystem object to access the granule from.
        total_retries: Number of retries in a case of exception
        num_seconds: Number of seconds to sleep between retries.


        Plot cube's layers data. All layers share the same x/y coordinate labels.
        There is an option to display only a subset of layers by specifying
        start and end index through "boundaries" input parameter.


        Validate just written to disk datacube. This method is introduced because
        of observed corrupted datacube properties:
        1. Validate X and Y coordinates values: not to include NaN's.
        2. Validate datetime objects of the cube against start_date of the cube.

        This check is introduced to capture corrupted datacubes as early as
        possible in the cube generation.


        Remove Zarr store and corresponding json file (with records of skipped
        granules for the cube) in S3 if they exists - this is done to replace existing
        cube with newly generated one:
            * at the beginning of the processing if --removeExistingCube command-line
            option is provided
            * at the end of the processing if destination location of created
            cube is in S3 bucket. This is done to avoid lingering Zarr objects
            generated with other settings which will result in different
            "directory" structure of the Zarr store.


        Read shape file in with ice mask information required for processing.

        Inputs:
        =======
        shapefile: URL to the shapefile.

        Returns:
        ========
        Object representing the shapefile.


        Read ice mask as stored in "column_name" field of the shapefile's row.

        Inputs:
        =======
        found_row: Row from the shape file that corresponds to the datacube's EPSG code
        column_name: Name of the shape file column that represents the land ice mask.
        grid_x: X coordinates of the datacube grid.
        grid_y: Y coordinates of the datacube grid.

        Returns: A tuple of:
                 * None if there is no overlap between land ice mask and datacube polygon,
                    or land ice mask for the same grid as datacube polygon.
                 * URL to the mask file as provided in the shapefile.


Compute median and median absolute deviation (MAD) for the vector x.


    Project vx and vy onto the median flow vector for the given spacial point.

    Inputs:
    =======
    x_in: x component of the velocity vector.
    y_in: y component of the velocity vector.
    dt:   Day separation vector.

    Return:
    =======
    Projected velocity onto median flow vector.


    Filter one spacial point by dt (date separation) between the images.

    Inputs:
    =======
    vp_in: Projected velocity to median flow unit vector.
    dt_in: Day separation vector.

    Return: a tuple of
    maxdt:   Maximum dt as determined by the filter.
    invalid: Mask for invalid values of the input vector based on maxdt.


    Filter data cube by dt (date separation) between the images for the sensor type.

    Input:
    ======
    vp:            Velocity projected to median flow unit vector.
    dt:            Day separation vector.
    mad_std_ratio: Scalar relation between MAD and STD.
    current_sensor_group: Current sensor to filter by.
    exclude_sensor_groups: List of sensors that should be excluded from the filter
                           (per spatial cell).

    Return:
    =======
    invalid: Mask for invalid values.
    maxdt:   Maximum date separation.
    sensor_include: Mask for included sensors.


    Return weighted standard deviation.

    Reference: https://stackoverflow.com/questions/2413522/weighted-standard-deviation-in-numpy


Make matrix of percentages of years corresponding to each displacement measurement


LSQ fit iteration for a single spacial point of the datacube.


    Get indices of "select_years" into "all_years" array.
    This is to replace built-in numpy.intersect1d() which does not work with
    numba.


    Initialize variables for LSQ fit.

    Return:
    results_valid: Boolean flag set to True if results are valid, False otherwise meaning that
                    further computation should be skipped. Computations should be
                    skipped if identified data validity mask is empty which results
                    in no data to be processed.
                    This flag has to be introduced in order to use numba compilation
                    otherwise numba-compiled code fails when using empty mask (pure
                    Python code does not).
    start_year, stop_year, v_in, v_err_in, dyr, totalnum, M_in: Filtered by data validity mask
                    and sorted by mid_date all input data variables.


    Initialize variables for LSQ fit.

    Return:
    results_valid: Boolean flag set to True if results are valid, False otherwise meaning that
                    further computation should be skipped. Computations should be
                    skipped if identified data validity mask is empty which results
                    in no data to be processed.
                    This flag has to be introduced in order to use numba compilation
                    otherwise numba-compiled code fails when using empty mask (pure
                    Python code does not).
    start_year, stop_year, v_in, dyr, w_v, w_d, d_obs, y1, M_in: Filtered by data
                    validity mask and pre-processed for LSQ fit input data variables.


    Create a mask based on the median date which falls within v0_years.

    Inputs:
    =======
    start_year: Decimal year corresponding to the start date.
    stop_year: Decimal year corresponding to the stop date.
    v0_years: Years within which middle date should fall into.


    Populates [A,ph,A_err,t_int,v_int,v_int_err,N_int,count_image_pairs] data
    variables.
    Computes the amplitude and phase of seasonal velocity
    variability, and also gives interannual variability.

    From original Matlab code:
    % [A,ph,A_err,t_int,v_int,v_int_err,N_int] = itslive_sinefit_lsq(t,v,v_err)
    % also returns the standard deviation of amplitude residuals A_err. Outputs
    % t_int and v_int describe interannual velocity variability, and can then
    % be used to reconstruct a continuous time series, as shown below. Output
    % Output N_int is the number of image pairs that contribute to the annual mean
    % v_int of each year. The output |v_int_err| is a formal estimate of error
    % in the v_int.
    %
    %% Author Info
    % Chad A. Greene, Jan 2020.
    %

    Inputs:
    =======
    TODO: ...
    v0_years: List of years to filter data by for calculations of climatological data


    Computes and returns the annual mean, error, count, and outlier fraction
    from component values projected on the unit flow vector defined by vx0 and vy0.

    Inputs:
        vx_fit: annual mean flow in x direction
        vy_fit: annual mean flow in y direction
        vx_fit_err: error in annual mean flow in x direction
        vy_fit_err: error in annual mean flow in y direction
        vx_fit_count: number of values used to determine annual mean flow in x direction
        vy_fit_count: number of values used to determine annual mean flow in y direction

    Outputs:
        self.mean.v[start_y:stop_y, start_x:stop_x, :]
        self.error.v[start_y:stop_y, start_x:stop_x, :]
        self.count.v[start_y:stop_y, start_x:stop_x, :]

    Outputs map to:
        * v_fit
        * v_fit_err
        * v_fit_count


    Computes and populates the mean, trend, seasonal amplitude, error in seasonal amplitude,
    and seasonal phase from component values projected on the unit flow  vector defined by vx0 and vy0

    Input:
    ======
    vx0: mean flow in x direction
    vy0: mean flow in y direction
    dvx_dt: trend in flow in x direction
    dvy_dt: trend in flow in y direction
    vx_amp: seasonal amplitude in x direction
    vy_amp: seasonal amplitude in y direction
    vx_amp_err: error in seasonal amplitude in x direction
    vy_amp_err: error in seasonal amplitude in y direction
    vx_phase: seasonal phase in x direction [day of maximum flow]
    vy_phase: seasonal phase in y direction [day of maximum flow]
    vx_se: standard error in x direction
    vy_se: standard error in y direction
    v_limit: maximum limit for the flow magnitude

    Correlation to actual inputs:
    =============================
    self.offset.vx[start_y:stop_y, start_x:stop_x],
    self.offset.vy[start_y:stop_y, start_x:stop_x],
    self.slope.vx[start_y:stop_y, start_x:stop_x],
    self.slope.vy[start_y:stop_y, start_x:stop_x],
    self.amplitude.vx[start_y:stop_y, start_x:stop_x],
    self.amplitude.vy[start_y:stop_y, start_x:stop_x],
    self.sigma.vx[start_y:stop_y, start_x:stop_x],
    self.sigma.vy[start_y:stop_y, start_x:stop_x],
    self.phase.vx[start_y:stop_y, start_x:stop_x],
    self.phase.vy[start_y:stop_y, start_x:stop_x],
    self.std_error.vx[start_y:stop_y, start_x:stop_x],
    self.std_error.vy[start_y:stop_y, start_x:stop_x]

    Output:
    =======
    v
    dv_dt
    v_amp
    v_amp_err
    v_phase
    v_se

    Correlation to actual outputs:
    =============================
    self.offset.v[start_y:stop_y, start_x:stop_x]
    self.slope.v[start_y:stop_y, start_x:stop_x]
    self.amplitude.v[start_y:stop_y, start_x:stop_x]
    self.sigma.v[start_y:stop_y, start_x:stop_x]
    self.phase.v[start_y:stop_y, start_x:stop_x]
    self.std_error.v[start_y:stop_y, start_x:stop_x]


    Returns the offset, slope, and error for a weighted linear fit to v with an intercept of datetime0.

    t: date (decimal year) of input estimates offset by the CENTER_DATE
    v: estimates
    v_err: estimate errors


        dims: a tuple of dimension indices for new data layout, i.e. if original
                dimension indices are [y, x, t], then to get [t, y, x] dimensions,
                tuple has to be (2, 0, 1).

        Traspose data variables to new dimensions.
        This is used to switch from continuous memory layout approach (for
        time dimension calculations) to end result data access ([time, y,x]).


Convert data to uint16 datatype to store to output file.


        Return mapping of sensor to its corresponding sensor group.

        This method builds mapping of the individual sensor to the group
        it belongs to:
            {
                '4.':  LANDSAT45,
                '5.':  LANDSAT45,
                4.0:   LANDSAT45,
                5.0:   LANDSAT45,
                '4.0': LANDSAT45,
                '5.0': LANDSAT45,
                '7.':  LANDSAT7,
                '7.0': LANDSAT7,
                7.0:   LANDSAT7,
                '8.':  LANDSAT89,
                '9.':  LANDSAT89,
                8.0:   LANDSAT89,
                9.0:   LANDSAT89,
                '8.0': LANDSAT89,
                '9.0': LANDSAT89,
                '1A':  SENTINEL1,
                '1B':  SENTINEL1,
                '2A':  SENTINEL2,
                '2B':  SENTINEL2
            }


        Return mapping of sensor to its corresponding sensor group name.

        This method builds mapping of the individual sensor to the group
        it belongs to:
            {
                '4.':  'L45',
                '5.':  'L45',
                4.0:   'L45',
                5.0:   'L45',
                '4.0': 'L45',
                '4.0': 'L45',
                '7.':  'L7',
                '7.0': 'L7',
                7.0:   'L7',
                '8.':  'L89',
                '9.':  'L89',
                8.0:   'L89',
                9.0:   'L89',
                '8.0': 'L89',
                '9.0': 'L89',
                '1A':  'S1',
                '1B':  'S1',
                '2A':  'S2',
                '2B':  'S2'
            }


        Map each of the granule's first sensor to the mission group it belongs to.

        Inputs:
        =======
        sensors: List of first sensors in the granules.


        Identify unique sensors within provided set and collect mission groups
        these sensors belong to: to know which missions are represented by the set.

        Inputs:
        =======
        sensors: List of sensors (as stored within datacube).

        Returns:
        ========
        List of mission groups that correspond to provided individual sensors.


        Returns list of sensor groups to exclude based on the timeseries for
        the spacial point.

        Inputs:
        =======
        ds_date_dt: date_dt timeseries for spacial point
        ds_sensors: individual sensors timeseries for spacial point
        ds_vx:      vx timeseries
        ds_vy:      vy timeseries
        ds_mid_date: mid_date timeseries

        Returns list of sensor groups to exclude for the spacial point.


        Exclude granules, if any are detected by the filter, from the data.

        ATTN: We had to introduce this method because of the
        "pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects"
        exception we are getting if calling
        xr.Dataset.drop_isel()
        for the datacube with layers with duplicates of "mid_date" values.

        Inputs:
        =======
        data: Data to exclude granules from.

        Returns:
        ========
        Updated or original data if no exclusions are required.


        Apply stable_shift corrections to the datacube's vx and vy variables and
        remove excluded granules if any.

        Inputs:
        =======
        vx: VX data
        vy: VY data

        Returns:
        ========
        Updated vx and vy data or original data if no corrections are required.


Create datacube composite: cube time mean values.


        Project valid velocity values to median flow unit vector.

        Inputs:
        =======
        ds_vx: 3d block of vx values.
        ds_vy: 3d block of vy values.
        ds_date_dt: day separation for velocity image pairs.
        ds_sensors: Current sensors for the datacube.
        exclude_sensors: 2d "map" of sensors to exclude from calculations (one list per each [y, x] point).


        Compute time average for the datacube [:, :, start_x:stop_index] coordinates.
        Update corresponding entries in output data variables.


Store datacube annual composite to the Zarr store.


    Cube LSQ fit with 2 iterations.

    Populate: [amp, phase, mean, err, sigma, cnt]

    Inputs:
    =======
    TODO:...


    Return mapping of sensor to its corresponding unique sensor ID.

    This method builds mapping of the individual sensor to the group
    it belongs to:
        {
            '4.':  4,
            4.0:   4,
            4:     4,
            '4.0': 4,
            '5.':  5,
            5.0:   5,
            '5.0': 5,
            '7.':  7,
            '7.0': 7,
            7.0:   7,
            '8.':  8,
            '9.':  9,
            8.0:   8,
            9.0:   9,
            '8.0': 8,
            '9.0': 9,
            '1A':  11,
            '1B':  12,
            '2A':  21,
            '2B':  22
        }


Function that returns description string to include all unique sensors and their meanings.


    Download R-tree index file from AWS S3 bucket.

    Args:
    - s3_bucket: Name of the S3 bucket.
    - s3_key: Key of the R-tree file in the S3 bucket.
    - local_path: Local path to save the downloaded file. If None is provided,
            the file will be saved in the current directory with the same name as the S3 key.

    Returns: R-tree index object.


    Query the R-tree for all files overlapping with the query bounding box.
    The query returns only files with at least min_percent_valid_pix valid pixels and
    with the same EPSG code as the query box.

    Args:
    - rtree_idx: R-tree index.
    - query_box: Bounding box to query (min_lon, min_lat, max_lon, max_lat)
    - epsg_code: Original EPSG code of the longitude, latitude coordinates in the query box.
    - min_percent_valid_pix: Minimum percentage of valid pixels in the granule to be considered.
        Default is 1%.

    Returns:
    - List of files names whose extents overlap with the query box.


Copy file to/from aws s3 bucket.

    Args:
    command_line (list): List tokens for the command-line to invoke.
    env_copy (dict): Dictionary of environment variables set for the compute environment.
    is_quiet (bool): Flag if using "quiet" mode to reduce output clutter. Default is True.

    Raises:
        RuntimeError: Failure to copy the store if NUM_AWS_COPY_RETRIES attempts failed.


    Returns a string (for example, N78W124) for directory name based on
    granule centerpoint lat,lon


    Define 5 points per each polygon side. This is done before re-projecting
    polygon to longitude/latitude coordinates.
    This function assumes rectangular polygon where min/max x/y define all
    4 polygon vertices.

    polygon: list of lists
        List of polygon vertices.


Representation for the composite.


        Create annual mosaics from datacubes composites for specified region.

        cubes_file: GeoJson catalog file with existing datacube's Zarr S3 URLs.
        s3_bucket: S3 bucket that stores all data (assumes that all datacubes,
                    composites, and mosaics are stored in the same bucket).
        cube_dir:  Directory path within S3 bucket that stores datacubes.
        composite_dir: Directory path within S3 bucket that stores datacubes' composites.
        mosaics_dir:   Directory path within S3 bucket that stores datacubes' mosaics.


        Collect datacube composites for specified region.
        Iterate over existing cubes that qualify for the search criteria and
        use cube's corresponding composites to create annual mosaics.

        cubes_file: GeoJson catalog file with existing datacubes' Zarr S3 URLs.
        s3_bucket: S3 bucket that stores all data (assumes that all datacubes,
                    composites, and mosaics are stored in the same bucket).
        cube_dir:  Directory path within S3 bucket that stores datacubes.
        composite_dir: Directory path within S3 bucket that stores datacubes' composites.


        Invoke reprojection code as subprocess - to allow for taichi import by
        re-projection code.
        NOTE: Can't import numba and taichi into the same module as both use
        JIT (Just-In-Time) compiler (compiles code at runtime the very first it's called).

        To invoke:
        reproject_mosaics_taichi.py -i mosaics_file
                                    -o reproject_mosaics_filename
                                    -p epsg
                                    -m reproject_matrix_filename


        Build annual and static mosaics from collected datacube composites per each EPSG code.
        Store "robust" composite attributes to the dataset if mosaics are not copied
        to the target AWS S3 bucket. Store "robust" mosaics attributes to the standalone
        JSON file if mosaics are to be copied to the target AWS S3 bucket.
        (ATTN: should change the logic to always store to JSON file? - than need
        to add more logic when merging re-projected mosaics stored on local drive as an
        intermittent step).

        When merging multi-EPGS codes mosaics into one, create mask of maximum count0 values
        based on the static mosaics. Use that mask to select pixels (will affect overlapping regions)
        that match maximum count. This is to avoid high noise when averaging values from multiple EPSGs
        with low count pixels.

        epsg_code:      Current EPSG code the mosaics are built for.
        epsg:           Dictionary of center_x->center_y->s3_path_to_composite format to
                        provide composites that should contribute to mosaics.
        s3_bucket:      S3 bucket that stores all data (assumes that all datacubes,
                        composites, and mosaics are stored in the same bucket).
        mosaics_dir:    Directory path within S3 bucket that stores datacubes' mosaics.
        check_for_epsg: Boolean flag to indicate if source data should be of the same EPSG code as
                        requested target EPSG for mosaics.
        copy_to_s3:     Boolean flag to indicate if generated mosaics files should be copied
                        to the target S3 bucket.
        rename_zero_based_data_vars:
                        Flag to indicate if these are the "final" mosaics and we need
                        to rename zero-based names of the data variables (for example,
                        'v0' with 'v') in summary mosaics and corresponding attributes
                        when writing final mosaics to NetCDF format files. Default is False,
                        meaning not to replace data variable names.


        Re-project "proj_polygon" attribute of original mosaic and save it to
        new JSON format file that corresponds to re-projected mosaic.

        Inputs:
        =======
        mosaics_epsg: Source EPSG of original mosaic.
        mosaic_filename: Filename of original mosaic.
        target_epsg: Target EPSG of re-projected mosaic.
        reproject_mosaic_filename: Filename of re-projected mosaic.


        Combine re-projected to the target EPSG projection annual and static mosaics
        for the region.
        Apply "average" to overlapping cells due to re-projection of mosaics that
        originate from different EPSG projections (original projections of composites).

        epsg_mosaics_files: Dictionary of re-projected mosaics per EPSG code. All mosaics
                            in this dictionary correspond to the same EPSG target code.
                            It's in the format: {epsg: {'year' or '0000': mosaic_file}}.
        s3_bucket: S3 bucket that stores all data (assumes that all datacubes,
                    composites, and mosaics are stored in the same bucket).
        mosaics_dir: Directory path within S3 bucket that stores datacubes' mosaics.
        copy_to_s3: Flag if result mosaics need to be copied to the AWS S3 bucket.


        Create summary mask based on max_count0=max(count0) across all mosaics. This mask is used to select
        cell values from the composites that correspond to the highest mask value when merging the rest of
        data variables for final composites.

        Args:
            ds: Dataset to add mask to. Need to remove the mask variable from the annual mosaics dataset since
                it is only present in summary mosaics.
            raw_ds: Mapping of static mosaic filename to open corresponding file and its metadata.


        Merge summary mosaics that were generated by re-projection of EPSG mosaics into
        target projection. Store result to NetCDF format file in S3 bucket if provided.
        Store robust attributes that contain information about all composites and datacubes
        contributed to the mosaic into separate json file (for traceability).
        Create mask with maximum count0 across all mosaics to be merged. Add data variables
        values for overlapping regions (by multiple EPGSs) should include pixels from the
        EPSG with maximum count0. This mask should be applied to annual mosaics as well.

        Args:
        first_ds: xarray.Dataset object that represents any (first) composite dataset.
                It's used to collect global attributes that are applicable to the
                mosaics.
        s3_bucket: AWS S3 bucket to place result mosaics file in.
        mosaics_dir: AWS S3 directory to place mosaics in.
        copy_to_s3: Boolean flag to indicate if generated mosaics files should be copied
                    to the target S3 bucket.


        Merge mosaics for a specific year that were generated by re-projection of
        EPSG mosaics into target projection. Store result to NetCDF format file in
        S3 bucket if provided.
        This method relies on the fact that it's called after static mosaics were
        merged - thus all metadata is set when static mosaics are processed.

        static_raw_ds: Mapping of static mosaics name to the correspondingopen file and metadata information
                        (needed for "count0").
        first_ds: xarray.Dataset object that represents any (first) composite dataset.
                It's used to collect global attributes that are applicable to the
                mosaics.
        year: Year for the mosaic to create.
        s3_bucket: AWS S3 bucket to place result mosaics file in.
        mosaics_dir: AWS S3 directory to place mosaics in.
        copy_to_s3: Boolean flag to indicate if generated mosaics files should be copied
            to the target S3 bucket.


        Create mosaics for a specific year and store it to NetCDF format file in
        S3 bucket if provided.

        ds_projection: EPSG projection for the current mosaics.
        first_ds: xarray.Dataset object that represents any (first) composite dataset.
                It's used to collect global attributes that are applicable to the mosaics.
        year_date: Datetime object for the mosaic to create.
        s3_bucket: AWS S3 bucket to place result mosaics file in.
        mosaics_dir: AWS S3 directory to place mosaics in.
        copy_to_s3: Boolean flag to indicate if generated mosaics files should be copied
            to the target S3 bucket.


        Create path to the annual mosaics filename which is based on the EPSG
        code the mosaic is created for.


Store datacube annual mosaics to NetCDF store.


        Create summary mosaics and store it to NetCDF format file in
        S3 bucket if provided.

        ds_projection: EPSG projection for the current mosaics.
        first_ds: xarray.Dataset object that represents any (first) composite dataset.
                It's used to collect global attributes that are applicable to the
                mosaics.
        s3_bucket: AWS S3 bucket to place result mosaics file in.
        mosaics_dir: AWS S3 directory to place mosaics in.
        copy_to_s3: Boolean flag to indicate if generated mosaics files should be copied
            to the target S3 bucket.
        rename_zero_based_data_vars:
                        Flag to indicate if these are the "final" mosaics and we need
                        to rename zero-based names of the data variables (for example,
                        'v0' with 'v') in summary mosaics and corresponding attributes
                        when writing final mosaics to NetCDF format files.


        Strip zeros from variable's metadata.

        Inputs:
        ds: xr.Dataset object containing the data variable.
        ds_var: name of the data variable.


        Unite provided polygons into one.

        Inputs:
        =======
        key: Attribute type that corresponds to the polygon being processed.
        polygons: collected "geometry" objects that represent polygons

        Returns:
        ========
        values: List of united polygons coordinates as lists (to be consistent with
                other attributes formats).
        geo_polygon: List of geo polygons external coordinates to compute center
                longitude and latitude for.


        Store datacube summary mosaics to NetCDF store and "robust" attributes
        to standalone JSON format file (for traceability).


Convert filename from NetCDF format to JSON.


Copy local NetCDF file to S3 bucket.


    Format filename for the datacube's composite:
    ITS_LIVE_velocity_EPSG3413_120m_X-3250000_Y250000.zarr

    Inputs:
    =======
    epsg_format: String representation of the EPSG code in "EPSGXXXXX" format
    grid_size: Grid size
    mid_x: X coordinate of datacube centroid
    mid_y: Y coordinate of datacube centroid


    Format filename for the annual mosaics of the region:
    ITS_LIVE_velocity_120m_ALA_2013_v02.nc

    Inputs:
    =======
    grid_size: Size of the grid cell (assumes the same in X and Y dimentions)
    region: Region for which mosaic file is created.
    year_date: Year for which mosaic file is created. Can be a string or a
                datetime object.


    Format filename for the summary mosaics of the region:
    ITS_LIVE_velocity_120m_ALA_2013_v02.nc


    Convert data to requested integer datatype. "fill_value" must correspond
    to the "data_type" to replace NaNs with corresponding to the datatype missing_value:
    -32767 for int16/32
    32767 for uint16/32
    etc.

    Inputs:
    =======
    data: Data to convert to new datatype to. It can be of numpy.ndarray or
            xarray.DataArray data type.
    data_type: numpy data type to convert data to. Default is np.uint16.
    fill_value: value to replace NaN's with before conversion to integer type.


Generate references from a single granule. Data is loaded directly
    into memory first.


Process a batch of granules. Data are downloaded into memory
    concurrently (async) in order to better saturate available bandwidth.


Get list of urls for geojson catalogs from master file


Build granule URLs from GeoJSON features


Generate kerchunk references JSON for granules in batches.


Convert elevation data from NetCDF to Zarr format file.


Store datacube to NetCDF format file.


Convert datacube composites Zarr store to NetCDF format file.


Display current memory usage.


Convert datacube Zarr store to NetCDF format file.


Driver to submit multiple Batch jobs to AWS.


Driver to re-submit multiple Batch jobs to AWS.


Check if datacube exists in AWS S3 bucket.


    Remove Zarr store and corresponding json file (with records of skipped
    granules for the cube) in S3 if they exists - this is done to replace existing
    cube with newly generated one.


Check if Zarr store exists in AWS S3 bucket.


    Remove Zarr store from S3 if it exists - this is done to replace existing
    composites with newly generated one.
    Also identify which AWS Batch queue the cube should be processed with: OnDemand
    or SPOT.


Replace Nan's with NoDataValue before converting data to np.int16 to avoid Nan->0 replacement


        Correct S1 data for the granule residing in S3 bucket. Copy corrected granule to the new
        S3 location.

        Inputs:
        =======
        bucket: S3 bucket holding granule for correction.
        granule_path: Granule path within S3 bucket.
        ref_bucket: S3 bucket with reference data for correction.
        ref_dir: S3 directory path that holds reference data for correction.
        s3: s3fs.S3FileSystem object to access data for correction.


Command line parser.


Main function to re-calculate stable_shift for the image velocity pair.


Fix everything in the granule.


    Validate datacube's datatime variables against original values in
    corresponding granules.


    Validate min values of datacube's datatime variables to make sure
    they fall into expected datetime range.


        Crop the granule to X/Y extend of non-None "v" values, store
        new granule in NetCDF format to the local directory, and push new granule
        to the target location in S3 bucket.


Copy the granule and corresponding PNG files to the target location in S3 bucket (without any modifications).


Given target granule URL find corresponding original granule in s3://its-live-project bucket. Granule sub-directories
            can be different due to the fact that target granules were cropped to valid X/Y extends and could end up in
            different "10x10 degree grid" subdirectory.

        Args:
            granule_url (str): Target granule URL

        Returns:
            Original granule URL that corresponds to the target URL.


        Crop corrsponding to "granule_url" original granule to X/Y extend of non-None "v" values,
        and restore M11 and M12 values in target granule (which has all metadata already fixed).
        Store new granule in NetCDF format to the local directory, and push new granule
        to the target location in S3 bucket along with corresponding *png files.
        If "restore_only" flag is set to True, it means that processing is done for zero coverage granules.
        Zero coverage granules were not cropped to the valid data X/Y extends and cropping should be
        skipped when restoring M11 and M12 data.

        Args:
            granule_url (str): Target URL to correct.
            original_granule_url (str): Original granule that has correct M11 and M12 values.
            restored_granule_url (str): S3 bucket fullpath filename to store corrected granule to.
            s3 (s3fs.S3FileSystem): s3fs object to access target granules stored in s3 bucket.
            s3_original (s3fs.S3FileSystem): s3fs object to access original granules stored in s3 bucket.
            restore_only (bool): Flag if cropping of the granule should be skipped when restoring M11 and M12
                (applies to zero coverage granules only).
            target_bucket_name: Name of target S3 bucket that stores granules to correct, and restored granules.
            local_dir (str): local directory to store corrected granules to


        Fix mapping.GeoTransform of ITS_LIVE datacubes stored in S3 bucket.
        Strip suffix from original granules names as appear within 'granule_url'
        data variable and skipped_* datacube attributes.


        Add new variables to the datacubes and copy them to S3 bucket's new target destination.

        Args:
        cube_url (str): Original cube URL in S3 bucket to add new variables to.
        cube_bucket_dir (str): AWS S3 bucket for original datacube.
        target_bucket_dir (str): AWS S3 target bucket for updated datacube.
        local_original_cube_dir (str): Local directory to store copy of the original datacube to update.
        local_dir (str): Local directory to store updated datacube before copying it to the target S3 bucket.
        s3 (s3fs.S3FileSystem): s3fs FileSystem object to access datacubes and granules.


Translate Cartesian polygons back to geospacial coordinates


    Function to define ITS_LIVE data cubes based on provided shape file and grid
    size in meters.


Identify bounding box for original dataset.


        Run reprojection of ITS_LIVE granule into target projection.

        This methods warps X and Y components of v and vp velocities, and
        adjusts them by rotation for new projection.


Write dataset to the netCDF format file.


        Rotate stable_shift_* and vx_error_*/vy_error_* attributes for
        corresponding data variables.


Warp variable into new projection.


        Re-project velocity's X and Y components, compute velocity magnitude and
        velocity error.


Reproject variables in ITS_LIVE granule into new projection.


Format spatial_ref attribute value for the UTM_Projection.


Convert x, y dimensions of the dataset into numpy grid array in (y, x) order.


Fix v_error of ITS_LIVE datacubes' composites stored in S3 bucket.


Fix composites and copy them back to S3 bucket.


Create COG for the "var_name" data variable of composites sequentially.


        Generate COG for the data variable of interest and copy the file to the S3 bucket in
        provided target directory.

        Use CLI command to generate COG. For example:
        gdal_translate -of COG 'ZARR:"/vsicurl/https://its-live-data.s3.amazonaws.com/composites/annual/v2/N50W140/ITS_LIVE_velocity_EPSG3413_120m_X-3350000_Y350000.zarr":/v0' junk_cog.tif -a_srs epsg:3413 -co TILING_SCHEME=GoogleMapsCompatible

        where "3413" for the "epsg:3413" command-line option is the projection attribute of the composite.

        Output COG filename should be formatted:
         s3://its-live-data/composites/annual/v2/cog/v0/ITS_LIVE_velocity_EPSG32735_120m_X750000_Y10050000_v0.tif


Given granule filename find its full path in s3://its-live-data bucket.

        Args:
            granule_basename (str): Granule filename.

        Returns:
            Original granule S3 path that corresponds to the target URL.


        Correct S1 data for the granule residing in S3 bucket. Copy corrected granule along with corresponding PNG files
        to the new S3 location.

        Inputs:
        =======
        bucket: S3 bucket holding granule for correction.
        granule_path: Granule path within S3 bucket.
        ref_bucket: S3 bucket with reference data for correction.
        ref_dir: S3 directory path that holds reference data for correction.
        s3: s3fs.S3FileSystem object to access original ITS_LIVE granules for correction.
        s3_ref: s3fs.S3FileSystem object to access data for correction.
        all_original_granules: List of all V2 S1 granules as stored in S3 bucket (on user end).


        Fix datacubes and copy them to S3 bucket's new location.

        Args:
        =====
        cube_url (str): Original cube URL in S3 bucket to add new variables to.
        cube_bucket_dir (str): AWS S3 bucket for original datacube.
        target_bucket_dir (str): AWS S3 target bucket for updated datacube.
        local_original_cube_dir (str): Local directory to store copy of the original datacube to update.
        local_dir (str): Local directory to store updated datacube before copying it to the target S3 bucket.
        s3 (s3fs.S3FileSystem): s3fs FileSystem object to access datacubes and granules.


        Reproject static mosaic to new projection.

        output_file: Output file to write reprojected data to.


        Reproject annual mosaic to new projection.

        output_file: Output file to write reprojected data to.


Write static mosaic dataset to the netCDF format file.


        Re-project variable's X and Y components, compute its magnitude and
        error if required.

        vx_var: name of the variable's X component
        vy_var: name of the variable's Y component
        v_var:  name of the variable
        vx_error_var: name of X component of error
        vy_error_var: name of Y component of error
        v_error_var: name of the error variable


        Re-project:
        * dv_dt's X and Y components, and compute dv_dt based on re-projected unit flow vector.
        * v_amp's X and Y components, and compute v_amp based on re-projected unit flow vector.
        * v_amp_error's X and Y components (compute v_amp_error outside of this method as need to
            compute re-projected v_phase first).
        * v_phase's X and Y components, and compute v_phase in direction of re-projected unit flow.

        Inputs:
        =======
        vx0: numpy array that stores re-projected vx0 data, used to compute unit flow vector
        vy0: numpy array that stores re-projected vy0 data, used to compute unit flow vector

        Outputs:
        ========
        dvx_dt
        dvy_dt
        dv_dt
        vx_amp
        vy_amp
        v_amp
        vx_amp_error
        vy_amp_error
        v_amp_error
        vx_phase
        vy_phase
        v_phase

        Example:
        dvx_dt, \
        dvy_dt, \
        dv_dt, \
        vx_amp, \
        vy_amp, \
        v_amp, \
        vx_amp_error, \
        vy_amp_error, \
        v_amp_error, \
        vx_phase, \
        vy_phase, \
        v_phase = self.reproject_static_vars(vx0, vy0)


Wrap phase and amplitude to be within valid ranges.


        Rotate v_phase and v_amp in the direction of v which is defined by vx0 and vy0.

        % seasonal_velocity_rotation gives the amplitude and phase of seasonal
        % velocity components. (Only the x and y components change when rotation is
        % applied, i.e., v_amp and v_phase are unchanged).
        %
        % Inputs:
        % theta (degrees) rotation of the coordinate system.
        % vx_amp (m/yr) x component of seasonal amplitude in the original coordinate system.
        % vx_phase (doy) day of maximum x velocity in original coordinate system.
        % vy_amp (m/yr) y component of seasonal amplitude in the original coordinate system.
        % vy_phase (doy) day of maximum y velocity in original coordinate system.
        %
        % Outputs:
        % vx_amp_r (m/yr) x component of seasonal amplitude in the original coordinate system.
        % vx_phase_r (doy) day of maximum x velocity in original coordinate system.
        % vy_amp_r (m/yr) y component of seasonal amplitude in the original coordinate system.
        % vy_phase_r (doy) day of maximum y velocity in original coordinate system.
        %
        % Written (in Matlab) by Alex Gardner and Chad Greene, July 2022.

        Returns:
        v_phase (doy) - day of maximum velocity in original coordinate system
        v_amp (m/yr) - seasonal amplitude in the original coordinate system


        This method creates transformation matrix for each point of the grid.

        vx_var - Name of the X component of the variable to decide if there is
                 data in the cell.
        vy_var - Name of the Y component of the variable to decide if there is
                 data in the cell.
        v_var - Name of the variable to decide if there is data in the cell.


    Main function of the module to be able to invoke the code from
    another Python module.


        1. Correct percent valid for the granule in NetCDF format to the local directory:
        * Rename granule
        * Rename corresponding PNG files
        * Update img_pair_info.roi_valid_percentage to int(new_pvalue)
        * Push new granule to the target location in S3 bucket.
        * Copy original PNG files to the target location in S3 bucket using new filenames

        2. Crop granule's X/Y grid only to the valid data extends

        Inputs:
        granule_url - HTTPS URL for the granule to correct.
        new_pvalue - Newly computed percent valid value.
        v1_pvalue - V1 percent valid value.
        s3 - s3fs.S3FileSystem object to access granules in AWS S3 bucket.


        Fix composites and copy them back to S3 bucket.

        Per Slack chat with Alex on July 12, 2022:
        "we need to rotate the vx/y_amp and vx/y_phase into the direction of v,
        which is defined by vx0 and vy0. If you replace the rotation matrix in
        the sudo code (coordinate projection rotation) by the rotation matrix
        defined by vx0 and vy0 then one of the rotated component is in the
        direction of v0 and the other is perpendicular to v0.
        We only want to retain the component that is in the direction of v0.""


        Run the transfer of granules from ASF to ITS_LIVE S3 bucket.

        If provided, don't process job IDs listed in exclude_job_ids_file (
        as previously processed).


        Copy granule from source to target bucket if it does not exist in target
        bucket already.


        Compute 2d unit vector for each cell of the grid.

        Inputs:
        =======
        xy: Re-projected to original grid cell points (this is 3d vectors because
            re-projection code generates 3d output).
        xy0: Original grid cell points (this is 3d vectors) (this is 3d vectors because
            re-projection code generates 3d output).
        cell_size: Size of the grid cell (either X or Y dimension).


Fill matrix data with initial values.


Compute transformation matrix for provided valid indices.


        Warp variable into new projection.

        Inputs:
        =======
        var: Name of the variable to warp
        warp_options: gdal.WarpOptions object to use for warping
        is_binary_data: Flag if the data is binary, and should enforce 0|1 values
            for warped data.


        Wrap phase and amplitude to be within valid ranges.

        Args:
        v_phase: Input phase in degrees.
        v_amp: Input amplitude.

        Ouputs:
        v_phase, v_amp: Wrap-corrected input values.


        Rotate v_phase and v_amp given "theta" rotation angle and already scaled
        vx_amp and vy_amp (according to the transformation matrix).

        % seasonal_velocity_rotation gives the amplitude and phase of seasonal
        % velocity components. (Only the x and y components change when rotation is
        % applied, i.e., v_amp and v_phase are unchanged).
        %
        % Inputs:
        % theta (degrees) rotation of the coordinate system.
        % vx_amp (m/yr) x component of seasonal amplitude in the original coordinate system.
        % vx_phase (doy) day of maximum x velocity in original coordinate system.
        % vy_amp (m/yr) y component of seasonal amplitude in the original coordinate system.
        % vy_phase (doy) day of maximum y velocity in original coordinate system.
        %
        % Outputs:
        % vx_amp_r (m/yr) x component of seasonal amplitude in the original coordinate system.
        % vx_phase_r (doy) day of maximum x velocity in original coordinate system.
        % vy_amp_r (m/yr) y component of seasonal amplitude in the original coordinate system.
        % vy_phase_r (doy) day of maximum y velocity in original coordinate system.
        %
        % Written (in Matlab) by Alex Gardner and Chad Greene, July 2022.

        Returns:
        vx_phase_r (doy) - day of maximum velocity in original coordinate system
        vy_phase_r (doy) - day of maximum velocity in original coordinate system
        vx_amp_r (m/yr) - x component of seasonal amplitude in the original coordinate system
        vy_amp_r (m/yr) - y component of seasonal amplitude in the original coordinate system


        This method creates transformation matrix for each point of the grid.

        vx_var - Name of the X component of the variable to decide if there is
                data in the cell.
        vy_var - Name of the Y component of the variable to decide if there is
                data in the cell.
        v_var - Name of the variable to decide if there is data in the cell.


        Fix metadata, store it to the local NetCDF file, and copy to the
        destination S3 bucket.


    Remove granule and corresponding *png files from S3 bucket if they exist.
    This is done in preparation to replace buggy granules with newly generated ones.


Fix static mosaics and copy them back to S3 bucket.


        Correct S1 data for the granule residing in S3 bucket. Copy corrected granule along with corresponding PNG files
        to the new S3 location.

        Inputs:
        =======
        granule_url: Granule path within S3 bucket.
        s3: s3fs.S3FileSystem object to access original ITS_LIVE granules for correction.


    Rename v*_error* and flag_stable_shift* attributes of velocity data variables.
    For example, for "vx" data variable:

    vx_error -> error
    vx_error_description -> error_description
    vx_error_mask -> error_mask
    vx_error_mask_description -> error_mask_description
    vx_error_slow -> error_slow
    vx_error_slow_description -> error_slow_description
    vx_error_modeled -> error_modeled
    vx_error_modeled_description -> error_modeled_description

    flag_stable_shift -> stable_shift_flag
    flag_stable_shift_description -> stable_shift_flag_description


Fix "sensor_flag" of ITS_LIVE composites stored in S3 bucket.


        Correct S1 data for the granule residing in S3 bucket. Copy corrected granule along with corresponding PNG files
        to the new S3 location.

        Inputs:
        =======
        granule_basename: Granule path within S3 bucket.
        ref_bucket: S3 bucket with reference data for correction.
        ref_dir: S3 directory path that holds reference data for correction.
        s3: s3fs.S3FileSystem object to access original ITS_LIVE granules for correction.
        s3_ref: s3fs.S3FileSystem object to access data for correction.
        granule_s3_path: full path of original granule in the s3 bucket.


    Driver to update general datacube definition file with existing
    datacube NetCDF S3 URLs.


Skip duplicate granules (the ones that have earlier processing date(s)).


Create catalog geojson file.


Generate geojson feature for the image pair, and create NSIDC metadata files.
           Publish metadata files to the target directory in S3 bucket.

        Args:
            infilewithpath (str): Fullpath to the granule file in S3 bucket.
            s3 (s3fs.S3FileSystem): s3fs object to access S3 bucket.

        Returns:
            geojson.Feature: Geojson feature for the granule.

        Raises:
            RuntimeError: If an error occurs while processing the granule file.


        Validate composite and determine what kind of processing it should go through.

        Returns:
        msgs: Messages created during Processing
        change_type: Flag if dtype of "count" and "count0" data variables should be just changed to np.uint32
        spot_queue: Flag if composite should be re-created with SPOT queue
        ondemand_queue: Flag if omposite should be re-created with OnDemand queue
        composite_file: Name of the composite store in AWS


Fix datacubes and copy them back to S3 bucket.


      Create premet file that corresponds to the input image pair velocity granule.

      Inputs
      ======
      ds: xarray.Dataset object that represents the granule.
      infile: Filename of the input ITS_LIVE granule


      Create spatial file that corresponds to the input image pair velocity granule.

      Inputs
      ======
      ds: xarray.Dataset object that represents the granule.
      infile: Basename of the granule.


   Upload file to the AWS S3 bucket, and remove local file if requested.

   Inputs:
   filename: Filename to be uploaded.
   target_dir: Target directory in the S3 bucket.
   target_bucket: Target bucket in the S3.
   s3_client: S3 client object.
   remove_original_file: Flag to remove the original file after uploading. Default is True.


   Create premet and spatial files for the input ITS_LIVE granule.

   Inputs
   ======
   infilewithpath: Filename of the input ITS_LIVE granule.
   s3: S3FileSystem object to access the S3 bucket. Default is None.

   Returns:
   ========
   A list of premet and spatial filenames generated locally.


Enable compression for the granule when storing it to the NetCDF file.


Read catalog geojson file and extract granules to build R-tree.

    Args:
        filename (str): Catalog geojson file name
        s3_read (s3fs.S3FileSystem): s3fs object to read the file


    Replace 'landsat/v00.0' directory with 'landsatOLI/v01' for all v01 granules listed in
    found catalog geojson files to correspond to the granules location within S3 bucket.


    Fix all attributes: acquisition datetime attribute for image1 and image2,
    chip_size_* description of the granule.


Fix acquisition datetime attribute for image1 and image2 of the granule


Fix description attributes for chip_size_width and chip_size_height of the granule


        Fix order of listed coordinates.
        This is sequential processing used for debugging only.


Fix order of the coordinates.


        Create premet file that corresponds to the input mosaics file.

        Inputs
        ======
        infile: Filename of the input ITS_LIVE granule
        url_tokens_1: Parsed out filename tokens that correspond to the first image of the pair
        url_tokens_2: Parsed out filename tokens that correspond to the second image of the pair


        Create spatial file that corresponds to the input image pair velocity granule.

        Inputs
        ======
        infile: Filename of the input ITS_LIVE granule


        Fix ITS_LIVE granules and create corresponding NSIDC meta files (spacial
        and premet).


Fix granule format and create corresponding metadata files as required by NSIDC.


Returns true if file exists in the bucket, False otherwise.


        Upload file to the AWS S3 bucket. If dryrun is enabled,
        just report the action.

        Inputs:
        filename: Filename to be uploaded.
        target_dir: Target directory in the S3 bucket.
        target_bucket: Target bucket in the S3.
        s3_client: S3 client object.
        remove_original_file: Flag to remove the original file after uploading. Default is True.


        Create corresponding metadata files for the granule as required by NSIDC.

        Inputs:
        target_bucket: Target AWS S3 bucket to copy metadata files to to.
        infilewithpath: Path to input granule file.


        Create premet file that corresponds to the input elevation file.

        Inputs
        ======
        infile: Filename of the input ITS_LIVE elevation file
        time_vals: Datetime values that correspond to the elevation.


        Fix ITS_LIVE elevation and create corresponding NSIDC meta files (spacial
        and premet) in parallel.
        Not used for now as we only have 1 elevation file to fix.


Fix data format and create corresponding metadata files as required by NSIDC.


        ATTN: This method implements sequetial processing for debugging purposes only.

        Fix ITS_LIVE granules and create corresponding NSIDC meta files (spatial
        and premet).


Fix "mapping" attributes for ingest by NSIDC.


        Fix "mapping" attributes of "ds" Dataset:
            Remove :CoordinateAxisTypes = "GeoX GeoY";
            Remove :CoordinateTransformType = "Projection";
            Rename spatial_proj4 to proj4text :proj4text = "+proj=lcc +lat_1=15 +lat_2=65 +lat_0=30 +lon_0=95 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs";
            :spatial_ref = "PROJCS["Asia_North_Lambert_Conformal_Conic",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",95],PARAMETER["Standard_Parallel_1",15],PARAMETER["Standard_Parallel_2",65],PARAMETER["Latitude_Of_Origin",30],UNIT["Meter",1],AUTHORITY["ESRI","102027"]]";
            Add (this is a suggestion only, so optional! adding crs_wkt (redundant with spatial_ref) expands interoperability with geolocation tools.) :crs_wkt = "PROJCS["Asia_North_Lambert_Conformal_Conic",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",95],PARAMETER["Standard_Parallel_1",15],PARAMETER["Standard_Parallel_2",65],PARAMETER["Latitude_Of_Origin",30],UNIT["Meter",1],AUTHORITY["ESRI","102027"]]";
            Add required attribute; add attribute and set to 95.0 :longitude_of_central_meridian = 95.0;
            Add required attribute; add attribute and set to 30.0 :latitude_of_projection_origin = 30.0; // double
            Add required attribute; add attribute and set to 0.0 :false_easting = 0.0; //double
            Add required attribute; add attribute and set to 0.0 :false_northing = 0.0; //double
            Add required attribute; add attribute and set to 15.0 and 65.0 :standard_parallel = 15.0, 65.0; // double

        Inputs:
        =======
        ds - xr.Dataset object
        new_filename - Local filename to store fixed dataset to
        encoding_params - Encoding settings for the dataset to store to the file
        chunk_size - Chunk size to use when storing dataset to the file.


        Fix "mapping" attributes of "ds" Dataset:
            Remove :CoordinateAxisTypes = "GeoX GeoY";
            Remove :CoordinateTransformType = "Projection";
            Rename spatial_proj4 to proj4text :proj4text = "+proj=lcc +lat_1=15 +lat_2=65 +lat_0=30 +lon_0=95 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs";
            :spatial_ref = "PROJCS["Asia_North_Lambert_Conformal_Conic",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",95],PARAMETER["Standard_Parallel_1",15],PARAMETER["Standard_Parallel_2",65],PARAMETER["Latitude_Of_Origin",30],UNIT["Meter",1],AUTHORITY["ESRI","102027"]]";
            Add (this is a suggestion only, so optional! adding crs_wkt (redundant with spatial_ref) expands interoperability with geolocation tools.) :crs_wkt = "PROJCS["Asia_North_Lambert_Conformal_Conic",GEOGCS["GCS_WGS_1984",DATUM["WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["False_Easting",0],PARAMETER["False_Northing",0],PARAMETER["Central_Meridian",95],PARAMETER["Standard_Parallel_1",15],PARAMETER["Standard_Parallel_2",65],PARAMETER["Latitude_Of_Origin",30],UNIT["Meter",1],AUTHORITY["ESRI","102027"]]";
            Required attribute; add attribute and set to 0.9996 :scale_factor_at_central_meridian = 0.9996;
            Required attribute; add attribute and set to -75.0 :longitude_of_central_meridian = -75.0
            Required attribute;  add attribute and set to 0.0 :latitude_of_projection_origin = 0.0;
            Optional; add attribute and set to 500000 :false_easting = 500000;
            Optional; add attribute and set to 10000000 :false_northing = 10000000;

        Inputs:
        =======
        ds - xr.Dataset object
        new_filename - Local filename to store fixed dataset to
        encoding_params - Encoding settings for the dataset to store to the file
        chunk_size - Chunk size to use when storing dataset to the file.


        Fix "mapping" attributes of "ds" Dataset:
            Change value to -90 :latitude_of_projection_origin = -90.0; // double
            Optional attribute; but if to be included, set to 6378137.0 :semi_major_axis = 6378137.0; // double
            Optional attribute; remove attribute (I can't find the correct value) :semi_minor_axis = 6356.752; // double
            Add new: this is a suggestion only, so optional! adding crs_wkt (redundant with spatial_ref) expands interoperability with geolocation tools. :crs_wkt = "PROJCS["WGS 84 / Antarctic Polar Stereographic",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Polar_Stereographic"],PARAMETER["latitude_of_origin",-71],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","3031"]]";
            Edit attribute name from spatial_proj4 to proj4text :proj4text = "+proj=stere +lat_0=-90 +lat_ts=-71 +lon_0=0 +k=1 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs";
            Required attribute; add attribute and set to -71.0 (standard_parallel is aka latitude_of_origin, but is not the same as latitude_of_projection_origin). :standard_parallel = -71.0; // double

        Inputs:
        =======
        ds - xr.Dataset object
        new_filename - Local filename to store fixed dataset to
        encoding_params - Encoding settings for the dataset to store to the file
        chunk_size - Chunk size to use when storing dataset to the file.


        Fix "mapping" attributes of "ds" Dataset:
            Change attribute value and set to 90.0 :latitude_of_projection_origin = 90.0; // double
            Optional attribute; but if to be included, set to 6378137.0 :semi_major_axis = 6378137.0; // double
            Remove :semi_minor_axis = 6356.752; // double
            Add new: this is a suggestion only, so optional! adding crs_wkt (redundant with spatial_ref) expands interoperability with geolocation tools. :crs_wkt = "PROJCS["WGS 84 / NSIDC Sea Ice Polar Stereographic North",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Polar_Stereographic"],PARAMETER["latitude_of_origin",70],PARAMETER["central_meridian",-45],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["X",EAST],AXIS["Y",NORTH],AUTHORITY["EPSG","3413"]]";
            Edit attribute name from spatial_proj4 to proj4text :proj4text = "+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +k=1 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs";
            Required attribute; add attribute and set to 70.0 (standard_parallel is aka latitude_of_origin, but is not the same as latitude_of_projection_origin) :standard_parallel = 70.0; // double

        Inputs:
        =======
        ds - xr.Dataset object
        new_filename - Local filename to store fixed dataset to
        encoding_params - Encoding settings for the dataset to store to the file
        chunk_size - Chunk size to use when storing dataset to the file.


Fix "mapping" attributes.


        Create premet file that corresponds to the input image pair velocity granule.

        Inputs
        ======
        infile: Filename of the input ITS_LIVE granule
        url_tokens_1: Parsed out filename tokens that correspond to the first image of the pair
        url_tokens_2: Parsed out filename tokens that correspond to the second image of the pair


    Additional fixes to dataset's mapping variable attributes.

    mapping: Mapping data variable of the dataset.


Upload file to the AWS S3 bucket.


        Fix granule format and create corresponding metadata files as required by NSIDC.

        Inputs:
        target_bucket: Target AWS S3 bucket to copy fixed granule to.
        target_dir: Directory in AWS S3 bucket to copy fixed granule to.
        infilewithpath: Path to input granule file.


Fix "mapping" attribute.


        Create spatial file that corresponds to the input image pair velocity granule.

        Inputs
        ======
        ds: xarray.Dataset object that represents the granule.
        infile: Filename of the input ITS_LIVE granule


        Fix granule format and create corresponding metadata files as required by NSIDC.
        Place fixed granule and metadata files in the target S3 bucket.


        See github issue #38 for details on the changes required for the file
        to be ingested by NSIDC.

        Inputs:
        ds: xarray.Dataset object that represents the NetCDF file content (granule
            or mosaic).


    takes an l1c scene id (e.g. "S2B_7VEG_20210417_0_L1C") and looks for it in l1c STAC returns - if not there (because catalog not updated)
    looks in l2a STAC returns, pulls ESA tileinfo.json to get full id, builds necessary info for its_live processing for this scene
    Entirely made to work around STAC catalog that isn't being updated (L2A is, L1C is not)


Collect granule percent differences for original and newly reconstructed M11/M12 workflow


Collect granule differences for original and newly reconstructed M11/M12 workflow


Generates data containing batch_size samples


Pads the 2nd and 3rd dimensions of a 4D tensor.


Write fields common to all EOS types in HDF5 file


Write tabulated barotropic EOS data into hdf5 file


Autogenerate description text for piecewise polytropic EOS


Write piecewise polytropic EOS data into hdf5 file


Autogenerate description text for polytropic EOS


Write polytropic EOS data into hdf5 file


Create hdf5 file with hybrid thermal EOS


r"""Create hdf5 file with tabulated barotropic EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param comment: Description of the EOS
    :param isentropic: Whether the EOS is isentropic.
    :param poly_n: Adiabatic index used for extrapolating below
                   lowest tabulated density
    :param rho: Array with baryonic mass density
                :math:`\rho\, [\mathrm{kg}/\mathrm{m}^3]`
                at the sample points
    :param eps: Array with specific internal energy
                :math:`\epsilon` [dimensionless]
                at sample points
    :param press: Array with pressure :math:`P\,[\mathrm{Pa}]`
                  at sample points
    :param csnd: Array with adiabatic sound speed
                 :math:`c_s\,[\mathrm{m}/\mathrm{s}]`
                 at sample points
    :param gm1: Array with pseudo-enthalpy
                :math:`g-1` [dimensionless]
                at sample points
    :param ye: Optionally, array with electron fraction
               :math:`Y_e` [dimensionless]
               at sample points.
    :param ye: Optionally, array with temperature
               :math:`T \,[\mathrm{MeV}]`
               at sample points.


r"""Create hdf5 file with polytropic barotropic EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param rho_poly: Polytropic density scale
                     :math:`\rho_p\, [\mathrm{kg}/\mathrm{m}^3]`
    :param rho_max: Maximum allowed baryonic mass density
            :math:`\rho_\mathrm{max} \, [\mathrm{kg}/\mathrm{m}^3]`
    :param n_adiab: Adiabatic index :math:`n`
    :param comment: Optional additional description of the EOS,
                    appended to autogenerated description


r"""Create hdf5 file with piecewise poly barotropic EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param rho_poly: Polytropic density scale
                     :math:`\rho_p\, [\mathrm{kg}/\mathrm{m}^3]`
                     of first segment
    :param rho_max: Maximum allowed baryonic mass density
            :math:`\rho_\mathrm{max} \, [\mathrm{kg}/\mathrm{m}^3]`
    :param rho_bnd: Array with lower boundary
                    :math:`\rho_{b,i}\, [\mathrm{kg}/\mathrm{m}^3]`
                    of baryonic mass densities for each segment.
                    Must start at zero.
    :param gammas: Array with adiabatic exponents :math:`\Gamma_i`
                   for each segment
    :param comment: Optional additional description of the EOS,
                    appended to autogenerated description


r"""Create a file for a hybrid EOS based on a tabulated cold EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param cold_name:  Name of the underlying cold barotropic EOS
    :param cold_comment: Description of underlying cold EOS
    :param gamma_th: :math:`\Gamma_{th}` defining the thermal part
    :param eps_max: Maximum allowed specific internal energy
                    :math:`\epsilon`
    :param poly_n: Adiabatic index used for extrapolating below
                   lowest tabulated density
    :param rho: Array with baryonic mass density
                :math:`\rho\, [\mathrm{kg}/\mathrm{m}^3]`
                at the sample points
    :param eps: Array with specific internal energy
                :math:`\epsilon` [dimensionless]
                at sample points
    :param press: Array with pressure :math:`P\,[\mathrm{Pa}]`
                  at sample points
    :param csnd: Array with adiabatic sound speed
                 :math:`c_s\,[\mathrm{m}/\mathrm{s}]`
                 at sample points
    :param gm1: Array with pseudo-enthalpy
                :math:`g-1` [dimensionless]
                at sample points
    :param ye: Optionally, array with electron fraction
               :math:`Y_e` [dimensionless]
               at sample points.


r"""Create a file for a hybrid EOS based on a piecewise polytropic
    cold EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param cold_name:  Name of the underlying cold barotropic EOS
    :param gamma_th: :math:`\Gamma_{th}` defining the thermal part
    :param eps_max: Maximum allowed specific internal energy
                    :math:`\epsilon`
    :param rho_poly: Polytropic density scale
                     :math:`\rho_p\, [\mathrm{kg}/\mathrm{m}^3]`
                     of first segment
    :param rho_max: Maximum allowed baryonic mass density
            :math:`\rho_\mathrm{max} \, [\mathrm{kg}/\mathrm{m}^3]`
    :param rho_bnd: Array with lower boundary
                    :math:`\rho_{b,i}\, [\mathrm{kg}/\mathrm{m}^3]`
                    of baryonic mass densities for each segment.
                    Must start at zero.
    :param gammas: Array with adiabatic exponents :math:`\Gamma_i`
                   for each segment
    :param cold_comment: Optionally, additional description of the
                         cold EOS.


r"""Create a file for a hybrid EOS based on a polytropic cold EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param cold_name:  Name of the underlying cold barotropic EOS
    :param gamma_th: :math:`\Gamma_{th}` defining the thermal part
    :param eps_max: Maximum allowed specific internal energy
                    :math:`\epsilon`
    :param rho_poly: Polytropic density scale
                     :math:`\rho_p\, [\mathrm{kg}/\mathrm{m}^3]`
    :param rho_max: Maximum allowed baryonic mass density
            :math:`\rho_\mathrm{max} \, [\mathrm{kg}/\mathrm{m}^3]`
    :param n_adiab: Adiabatic index :math:`n`
    :param cold_comment: Optionally, additional description of the
                         cold EOS.


r"""Create a file for a classical ideal gas EOS

    :param path:  Path of the file to create. Ending should be .eos.h5
    :param name:  Name of the EOS
    :param n_adiab: Adiabatic index :math:`n`
    :param eps_max: Maximum allowed specific internal energy
                    :math:`\epsilon`
    :param rho_max: Maximum allowed baryonic mass density
            :math:`\rho_\mathrm{max} \, [\mathrm{kg}/\mathrm{m}^3]`
    :param comment: Optional additional description of the EOS,
                    appended to autogenerated description


    Plot mass-radius and compactness-deformability diagrams for
    a list of TOV sequences.


    Set up a family of piecewise polytropic EOS by varying one
    of the segment's adiabatic exponent. Compute TOV sequence
    for each EOS, plot them, save plot in current directory.


Check if the test datasets are downloaded. If not, download them from Zenodo.


Execute the test routines for the line plot postprocessing module

    Returns:
        int: success status of the test


Executes the test routine for the 2D postprocessing module

    Returns:
        boolean: success status of the test


Executes the test routine for the 3D postprocessing module

    Returns:
        boolean: success status of the test


main function to run VisIt plot routines

	Raises:
		RuntimeError: No plot variables specified!


    Main function to plot the data

    Raises:
        IOError: Data path not found
        IOError: Data file not found
        RuntimeError: No plot variables specified
        RuntimeError: Number of linestyles does not match number of plot variables.
        RuntimeError: Number of xlims does not match number of plot variables.


Generates a single slice plot for a given variable

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters
		variableToPlot (str): name of the variable to be plotted
		setplotbounds (bool, optional): flag to specify whether bounds for the plotting variable should be specified. Defaults to False.
		plotbounds (list, optional): list of two values that specify the minimum and maximum value of the plot variable. Defaults to [0,1].

	Raises:
		RuntimeError: DrawPlots() failed. Aborting.


    Make a line plot, given pandas dataframe and plot variable

    Args:
        data_obj (pd.DataFrame): a pandas dataframe holding the data
        plot_variable (str): the name of the variable to be plotted
        plotbounds (list, optional): the plot bounds. Defaults to None.
        linestyle (str, optional): the plot line style. Defaults to '-'.


    Make multiple plots

    Args:
        data_obj (pd.DataFrame): a pandas dataframe holding the data
        plot_variables (list): the names of the variables to be plotted, in a list
        plotbounds (list, optional): the plot bounds. Defaults to None.
        linestyles (list, optional): the plot line styles in a list format. Defaults to None.


Set the annotation settings.

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

	Raises:
		RuntimeError: Could not set Annotation attributes. Aborting.


Setup a pseudocolor plot

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters
		variableToPlot (str): name of the variable to be plotted
		setplotbounds (bool): flag to specify whether bounds for the plotting variable should be specified
		plotbounds (list): list of two values that specify the minimum and maximum value of the plot variable

	Raises:
		RuntimeError: Could not set Pseudocolor attributes. Aborting.


Setup a volume plot

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters
		variableToPlot (str): name of the variable to be plotted

	Raises:
		RuntimeError: Could not set Volume attributes. Aborting.


Setup the attributes for the slice operator

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

	Raises:
		RuntimeError: Could not set slice attributes. Aborting.


Setup the attributes for the mesh

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

	Raises:
		RuntimeError: Could not set mesh attributes. Aborting.


Setup the 2D view

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

	Raises:
		RuntimeError: Failed to set 2D view.


Setup the 3D view

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

	Raises:
		RuntimeError: Failed to set 3D view.


Save the current window

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters
		variableToPlot (str): name of the variable to be plotted


 This function iterates over all hdf5 files and generates the plots according to the plot_type

	Args:
		config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters
		variableToPlot (str): name of the variable to be plotted
		hdf5files (list): list of paths to the hdf5 files
		plot_type (str, optional): type of plot to generate. Defaults to "2d". Options are "2d" and "3d"
		setplotbounds (bool, optional): flag to specify whether bounds for the plotting variable should be specified. Defaults to False.
		plotbounds (list, optional): list of two values that specify the minimum and maximum value of the plot variable. Defaults to [0,1].

	Raises:
		RuntimeError: TimeSlider could not advance
		RuntimeError: Window could not be saved
		SystemError: Plot could not be cleared from memory


create output directories if they don't exist

    Args:
        config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters


selector function for type of plot to generate

    Args:
        type (str): string defining type of plot function to choose. Options: '2d' and '3d'

    Raises:
        ValueError: Invalid plot type

    Returns:
        func: appropriate plotting function


	Find all the plot files and return them as list of absolute path strings

    Args:
        config (configparser.ConfigParser): instance of a ConfigParser class that holds the users parameters

    Raises:
        FileNotFoundError: Error raised if no plot files were found

    Returns:
        list: list of plot files