Process a single month of footprints for a given domain, site, height,
    year, month.

    If you want to process all files for a given domain + site
    use process_all.

    This routine finds all fields files and particle location files which match
    the timestamp in the file name. The date strings are stored in the datestr
    variable, and these strings must match exactly for the fields and the particle
    locations. Hourly netcdf output must be .nc and not .nc.gz!

    At the moment, the entire Met folder is read each time.

    Args:
        domain (str):
            Domain of interest
        site (str):
            Observation site
        height (str):
            Height of observation, e.g. "10magl"
        year (int):
            The year
        month (int):
            1,2,3,4,5,6,7,8,9,10,11,12
        met_model (str):
            Met model used to run NAME
            Default is None and implies the standard global met model
            Alternates include 'UKV' and this met model must be in the outputfolder NAME
        base_dir (str, optional):
            Base directory containing NAME output
            Default (i.e. on BP1) ="/work/chxmr/shared/NAME_output/",
        process_dir (str, optional):
            Base directory for where processed netcdf files go
            Default (i.e. on BP1)="/work/chxmr/shared/LPDM/fp_NAME/",
        fields_folder (str, optional):
            Folder containing fields data
            Default="Fields_files",
        particles_folder (str, optional):
            Folder containing particles data.
            Default = "Particle_files",
        met_folder (str or list, optional):
            Folder(s) containing met data
            Default = "Met"
        force_met_empty (bool, optional):
             Force the met data to be empty?
             Default = False.
        use_surface_conditions (bool, optional) :
            Use default expected surface conditions for meteorological values
            if converting from gs/m3 to mol/mol / mol/m2/s units.
            This involves fixing the P/T ratio to 345 based on typical surface conditions
            in Europe and not using the meteorlogical data extracted from NAME release
            points.
            Default = True.
        satellite (bool, optional):
            NAME run was performed over satellite data.
            This means satellite_vertical_profile function will be used to combined data
            from multiple levels for each point as defined by upper_level parameter.
            Default = False.
        obs_folder (str, optional) :
            Folder for netCDF format observations. Won't be contained in the original
            NAME results folder and will need to be copied into the subfolder defined
            by this parameter.
            Default = "Observations"
        upper_level (int/None):
            Only needed when satellite=True. Highest level number from within the
            NAME run for the satellite data.
            Default = None.
        max_level (int, optional):
            Only needed when satellite=True. The max level to
            process the footprints.
            Any levels above are replaced by the prior profile.
            This can be <= upper_level.
            If max_level is not specified, this will match upper_level.
            Default = None.
        force_update (bool, optional):
            By default, any existing netCDF files are NOT overwritten.
            To explicitly over-write a file, set force_update = True
        perturbed_folder (str, optional)
            Process a subfolder which has all of the required folders
            (Fields_files, etc). This is for perturbed parameter ensembles for
            a particular site. E.g. for
            EUROPE_BSD_110magl/Perturbed/PARAMETERNAME_VALUE
            you'd set: perturbed_folder = "Perturbed/PARAMETERNAME_VALUE".
            Default = None.
        vertical_profile (bool, optional):
            If set to True will look for vertical potential temperature met file
            and incorporate into footprint file.
            This is a separate file from the normal met file, and is not mandatory.
            Default=False.
        transport_model (str, optional):
            Defaults to "NAME". If "STILT", reads footprints in the
            ncdf format created by the STILT model. Other values are invalid.
            Notset up to read satellite column footprints from STILT format.
            Default="NAME".
        species (str,optional):
            Defaults to None which will process footprints for an inert species
            Otherwise will look in json file for lifetime and process a species-
            specific footprint with the species name in the .nc file
        user_max_hour_back (float, required when species = 'CO2'):
            Defaults to 24 hours. This is the maximum amount of time back from release time that
            hourly footprints are calculated.

    Returns:
        None.
        This routine outputs a copy of the xarray dataset that is written to file.


    This script processes multiple months in parallel (but loops through years
    in serial). It basically just calls process.process in parallel.

    Args:
        domain (str):
            Domain of interest
        site (str):
            Observation site
        height (str):
            Height of observation, e.g. "10magl"
        years (list):
            List of years you want to process
        month (int):
            List of months to process (e.g. np.arange(12)+1)
        kwargs (dict):
            Dictionary of optional arguments that you would pass to process, e.g.
            kwargs = {"met_folder":"Met_daily", "fields_folder" : "MixR_files"}
        nprocess (int):
            Number of threads (and number of months to run in parallel).
            Default = 4

    Returns:
        None


    Function to process the site specific vertical pressure and temperature gradients.
    Relies on the process_met function
    In order to fit into this function the header information needs to be edited
    in the output files NAME generates.
    Required structure given in vp_met_dict.

    Args:
        vp_name (str, optional):
            Name of vertical profiles file name

    Returns:
        xarray dataset containing:
         - theta_slope - the potential temperature gradient at each time point
         - slope_error - the error in this calculated gradient


    Extract the time and location of release from STILT output based on the
    identifier string.

    Args:
        nc (netCDF4.Dataset):
            netCDF4.Dataset in STILT format.

    Returns:
        release_lat (float):
            Release latitude.
        release_lon (float):
            Release longitude.
        release_ht (float):
            Release height (units??).
        time (float):
            Release time.
        time_seconds (float):
            ???
        time_reference (float):
            ???

    Todo:
        Update outputs with units and what they actually are.


    Extract particle data from STILT output and assign it the given Id.

    Args:
        nc (netCDF4.Dataset):
            netCDF4.Dataset in STILT format.
        ID (int):
            Identifier ID
        domain_N (float):
            N latitude
        domain_S (float):
            S latitude
        domain_E (float):
            E longitude
        domain_W (float):
            W longitude
        exitfile (str):
            Name of out file
        append (bool):
            Append header True/False

        Returns:
            part (dataframe):
                Particle data
            partnames (list):
                Variable names


    Read data from STILT netcdf output files into arrays in an xray dataset.
    The corresponding function for NAME output is footprint_array.
    This function reads multiple files at once because STILT output is stored
    as a separate file for each release time. To read a month of output, give
    a prefix like stilt2016x07x.

    Args:
        prefix (str):
            File pattern prefix (including path) for output files to read
        exitfile (str, optional):
            File in which to temporarily store particle data.
            Default = None
        met (dict):
            not used to interpret STILT footprints, which are already given
            in ppm/(mol/m^2/s), but can be included if available.
            Default = None.
        satellite (bool, optional):
            Not implemented; will produce error if True.
            Default=False
        time_step (float, optional):
            Ignored; not needed to interpret STILT footprints.
            Default=None.

    Returns:
        fp (xarray datset):
            An xarray dataset as from footprint_concatenate
            (see help(process.footprint_concatenate))


    Opens the satellite observations file. This should match
    the points within the NAME input csv and with the output of NAME.
    Used for satellite data as NAME input is not precise enough (minute precision).

    Can either specify satellite_obs_file or specify subfolder,datestr,obs_folder so obs_fils can be found.

    Args:
        satellite_obs_file (str) :
            Observation filename.
        subfolder (str) :
            Only include if satellite_obs_file is not specified.
            Folder containing the output from NAME. Folder containing observations should be
            within this directory.
        datestr (str) :
            Only include if satellite_obs_file is not specified.
            The date string being used to search for files. This should be of the form
            'YYYYMMDD' or 'YYYYMMDD-NNN'
        obs_folder (str, optional) :
            Only include if satellite_obs_file is not specified.
            Name of the folder containing the observations data. Will be appended to subfolder
            to get the full path information.

     Returns:
        xarray.dataset:
            satellite file


2D mass-conservative regrid

    Args:
        array_in (arrray):
            2D field to regrid
        lat_in (array):
            latitudes corresponding to array_in
        lon_in (array):
            longitude corresponding to array_in
        lat_out (array):
            latitude to regrid onto
        lon_out (array):
            longitude to regrid onto
        global_grid (boolean):
            True if grid wraps around Earth, otherwise it does not realise
            that 0 and 360 deg longitude are the same, and your grid will have
            a strange gap in.

    Returns:
        array:
            regridded 2D array of dimensions [lat_out, lon_out]
        iris 'Cube': regridded iris 'Cube' object

    Example:
        new_array, newcube = regrid2d(array_in, lat_in, lon_in, lat_out, lon_out)


3D mass-conservative regrid using a cached regridder

    Args:
        array_in (array):
            3D field to regrid -- Dimensions of [lat_in, lon_in, time_in]
        lat_in (array):
            latitudes corresponding to array_in
        lon_in (array):
            longitude corresponding to array_in
        lat_out (array):
            latitude to regrid onto
        lon_out (array):
            longitude to regrid onto
        time (array):
            times to regrid onto
        global_grid (boolean):
            True if grid wraps around Earth, otherwise it does not realise
            that 0 and 360 deg longitude are the same, and your grid will have
            a strange gap in.

    Returns
        array:
            regridded 3D array of dimensions [lat_out, lon_out, time]

    Example:
        array_out = regrid3d(array_in, lat_in, lon_in, lat_out, lon_out, time)


Creates a Dataset ready to be used by the xesmf regridder from 1d arrays of lat and lon


    Regrid data between two uniform, cell centered grids using a conservative method. Data must have dimensions [lat, lon] if 2d or [time, lat, lon] if 3d.

    inputs
        data - numpy array
        input_lat - 1d numpy array of cell centre latitudes for data
        input_lon - 1d numpy array of cell centre longitudes for data
        output_lat - 1d numpy array of cell centre latitudes for desired output
        output_lon - 1d numpy array of cell centre longitudes for desired output

    returns
        data regridded as numpy array


    Regrid data from predefined input_grid and output_grid

    Inputs
        data - numpy array

        input_grid, output_grid -  Dataset of the form:

            xr.Dataset({'lat': (['x', 'y'], LAT),
                         'lon': (['x', 'y'], LON),
                         'lat_b': (['x_b', 'y_b'], LAT_b),
                         'lon_b': (['x_b', 'y_b'], LON_b)})

            where lat and lon give cell centre locations, and lat_b and lon_b give the cell bounds (corners)

        method - string describing method to use.
            Should be one of "conservative" or "conservative_normed".
            Note that to use the conservative_normed method you need to have installed the "masking" development
            branch of the xesmf package (contact Daniel for more details).

            With the 'masking' branch of xESMF you can include a mask in the input_grid to ignore nan values
    returns
        regridded numpy array


Coarsen a fine grid by some integer factor.
    E.g. if the input array is 1000 x 500, and is coarsened by a factor of 2
    the output array will be 500 x 250. By default, the output is the mean
    of the inputs.

    Args:
        arrayFine (array):
            array of values for fine resolution grid
        latFine (array):
            1D latitudes of fine resolution grid
        lonFine (array):
            1D longitudes of fine resolution grid
        factor (int, optional):
            Factor by which to coarsen array. Default is 2
        mean (bool):
            True/False.
            True calculates the mean of the fine array values; False calculates the sum.
            Default is True

    Returns:
        out (array):
            grid of coarsened values
        outLat (array):
            Latitudes of coarsened grid
        outLon (array):
            Longitudes of coarsened grid

    Example:
        out, outLat, outLon = coarsen(fineArray, fineLat, fineLon, factor = 10, mean = True)


Calculates grid of areas (m2) given arrays of latitudes and longitudes

  Args:
      lat (array):
          1D array of latitudes
      lon (array):
          1D array of longitudes

  Returns:
      area (array):
          2D array of areas of of size lat x lon

  Example:
    import acrg_grid
    lat=np.arange(50., 60., 1.)
    lon=np.arange(0., 10., 1.)
    area=acrg_grid.areagrid(lat, lon)


    Gets a list of files given a species, start date and end date.

    Searches for path and filename of the format:
        MOZART_DIRECTORY/SPECIES/*runname*year-month*.nc

    year and month are extracted from list of dates created from start, end values with frequency of a month.

    Args:
        species (str) :
            Species name.
            One of "CH4","CO2","HFC-125","HFC-134A","HFC-143A","HFC-152A","HFC-227EA","HFC-23","HFC-32","N2O"
        start (str) :
            Start date. Format "YYYY-MM-DD".
        end (str) :
            End date. Same format as start.
        runname (str) :
            Name of MOZART run. Valid options are dependent on the species.
            "CH4": "NewEDGAR"
            "CO2","N2O": "NewEDGAR", "Scaled"
            No runname for other gases, just specify an empty string ''.

    Returns:
        list :
            List of extracted filenames


    Converts variables with a longitude dimension to the -180-180 convention
    rather than the 0-360 convention.
    WARNING: variable must have dimensions ('height','lat','lon','time') in that order.

    Args:
        DS (xarray.Dataset) :
            Dataset containing data_var
        data_var (str) :
            Name of data variable within dataset with dimensions ('height','lat','lon','time')

    Returns:
        None

        Updates DS in place with new co-ordinate system


    Created to convert MOZART heights to NAME heights at boundaries.
    Interpolates the heights of the VMR variable 'vmr_var_name' in the xarray dataset
    'DS' to the heights specified in 'interp_heights'.

    Args:
        DS (xarray.Dataset) :
            Dataset containing (at least) data variables:
                vmr_var_name (see vmr_var_name definition)
                "Alt" (for altitudes) (height, lat_or_lon, time)?
            Must be an altitude variable present for each value of the VMR variable.
        vmr_var_name (str) :
            Name of data variable for vmr (volume mixing ratio) quanity. Must have dimensions of:
                (height, lat_or_lon, time) in that order.
        interp_height (numpy.array?) :
            Array of heights to interpolate to.

    Returns:
        xarray.Dataset:
            New dataset with the VMRs recalculated at interpolated heights, a 'height' dimension
            replaced with the interpolated values and the 'Alt' variable removed.


    Created to convert MOZART lons/lats to NAME lons/lats at boundaries.
    Make sure that the heights have already been interpolated using 'interp_heights'.
    Interpolates the heights of the VMR variable 'vmr_var_name' in the xray dataset
    'DS' to the longitude or latitude specified in 'lon_or_lat'. The variable must
    have dimensions (height, lat_or_lon, time) in that order.

    Returns:
        xarray.Dataset:
            Updated dataset with interpolated latitudes and longitudes


    Returns an xray dataset with (height,lat,lon,time) coordinates and 2 data variables:
    concentration of species (SPECIES_vmr_mozart) and an array of altitudes calculated
    from the pressure levels given in the original MOZART file (Alt). The lons are converted
    from the 0 - 360 convention used in MOZART to the -180 to 180 convention used in NAME.


    Gets an xray dataset with 4 variables, each of one side of the domain boundary
    (n,e,s,w) and with height and lat/lon interpolated to the NAME grid.
    MZ is a mozart xray dataset created using MOZART_vmr.


    Specify end date as 2 months after the month of the last file
    (because the date specified is actually the first day of the next month and
    the range goes up to but doesn't include the last date). Only monthly
    frequency because this is the frequency of the mozart files we have so far.


    Args:
      start (str)
        Start date
      end (str)
        End date
      species (str, optional)
        Defaults to 'ch4'
      cams_directory (str, optional)
        Path to CAMS files
      version (str, int, optional))
        Version number of inversion, e.g. 'v19'
        if an int is entered, it will be converted to a str starting with 'v'
        defaults to 'latest'
      variables (dict, optional)
        Dictionary containing variable names including:
        - 'altitude',
        - 'file_start_str',
        - 'height',
        - 'hlevel',
        - 'lat',
        - 'lon',
        - 'time',
        - 'z'
        if not given variables will be assumed to be the same as for CAMS CH4 v19

    Returns:
        xarray dataset of combined CAMS inversions


    Convert altitude coordinate to level

    Args
        ds (xarray.Dataset)
            CAMS data
        species, version (str, optional)
            used to infer the variable names from the 'default_inputs' dictionary
            default to species='ch4' and version='latest'
        variables (dict, optional)
            dictionary containing variable names including:
                'altitude', 'file_start_str', 'height', 'hlevel', 'lat', 'lon', 'time', & 'z'
            if not given variables will be assumed to be the same as for CAMS CH4 v19

    Returns
        xarray.Dataset


    Interpolates the CAMS data to the NAME heights

    Args:
        nesw (dataset)
            The N, E, S or W BC CAMS boundary data
        fp_height (array)
            NAME footprint heights
        lonorlat (string)
            Whether you're interpolating along the longitude (N or S) or the latitude (E or W).
        species (string)
            The gas species of interest, defaults to 'ch4'
        version (string)
            The version to use, used to infer the variable names from the 'default_inputs' dictionary
        variables (dict, optional)
            dictionary containing variable names including:
                'altitude', 'file_start_str', 'height', 'hlevel', 'lat', 'lon', 'time', & 'z'
            if not given variables will be assumed to be the same as for CAMS CH4 v19
        reverse (bool/None)
            Whether height values within is nesw input are in reverse order
            (i.e. nesw["z"] level 1 values > nesw["z"] level 2 values).
            Default = None. If this is set to None this will be automatically determined.

    Returns:
        dataset : CAMS BC data interpolated to NAME heights


    Interpolates the CAMS data to the NAME longitudes and latitudes

    Args:
        nesw (dataset)
            The N, E, S or W BC CAMS boundary data
        fp_lonorlat (array)
            NAME footprint longitudes or latitudes
        species (string)
            The gas species of interest
        lonorlat (string)
            Whether you're interpolating along the 'longitude'
            (N or S) or the 'latitude' (E or W).
        reverse (bool/None)
            Whether lon or lat values within nesw input are in reverse order
            (i.e. nesw[lonorlat][0] > nesw[lonorlat][1]).
            Default = None. If this is set to None this will be automatically determined.
        version (string)
            The version to use, used to infer the variable names from the 'default_inputs' dictionary
        variables (dict, optional)
            dictionary containing variable names including:
                'altitude', 'file_start_str', 'height', 'hlevel', 'lat', 'lon', 'time', & 'z'
            if not given variables will be assumed to be the same as for CAMS CH4 v19

    Returns:
        dataset : CAMS BC data interpolated to NAME longitudes or latitudes


    Create a standardised filename for CAMS boundary conditions files

    Args:
        domain (str)
            Georgaphical domain, e.g. 'EUROPE', 'PACIFIC', 'USA'
        start_date (str)
            Start date of BC, e.g. '2015-01-01'
        time_res (str)
            Time resolution of the output file; one of "montlhy", "daily", "3h"
        species (str, optional)
            Gas species, e.g. 'ch4', 'co2'
            defaults to 'ch4'
        from_climatology (bool)
            Whether the BC has been produced from climatologies

    Returns:
        A standardised filename (str)


    Writes the CAMS BC data to a netcdf file.
    This uses bc_filename to produce a standardised file name for the boundary conditions file.

    Args:
        vmr_n (array)
            Molar ratio at northern boundary
        vmr_e (array)
            Molar ratio at eastern boundary
        vmr_s (array)
            Molar ratio at western boundary
        vmr_w (array)
            Molar ratio at southern boundary
        st_date (str)
            Start date of form "YYYY-MM-dd"
        time_res (str)
            Time resolution of the output file; one of "montlhy", "daily", "3h"
        domain (str)
            The domain which you want the boundary conditions for.
        outdir (str)
            Directory for writing output file.
        gridsize (int/float)
            Resolution of CAMS output in degrees.
            Possible are: 0.125, 0.25, 0.4, 0.5, 0.75, 1, 1.125, 1.5, 2, 2.5, 3
        species (str, optional)
            The gas species e.g. 'ch4', 'co2'
            defaults to 'ch4'
        version (str, optional)
            CAMS version to use, used to infer the variable names from the 'default_inputs' dictionary
        variables (dict, optional)
            dictionary containing variable names including:
                'altitude', 'file_start_str', 'height', 'hlevel', 'lat', 'lon', 'time', & 'z'
            if not given variables will be assumed to be the same as for CAMS CH4 v19
        from_climatology (bool)
            Whether BCs were produced from climatologies

    Returns
        netcdf file: Boundary conditions at domain boundaries


    Create CAMS boundary conditions and write to a netcdf file

    This uses interplonlat and interpheight to regrid the boundary conditions to match the NAME grid.
    Then writes to a netcdf file using write_CAMS_BC_tonetcdf.

    Args
      ds (xarray.Dataset)
         CAMS dataset
      fp_lat, fp_lon (numpy.ndarray)
        NAME footprint latitudes and longitudes
      date (str)
        Start date of form "YYYY-MM-dd"
      domain (str)
        Geographical region of interest, e.g. 'EUROPE', 'PACIFIC', 'USA'
      species (str)
        Gas species of interest, e.g. 'ch4', 'co2'
        defaults to 'ch4'
      time_res (str):
        Time resolution of the output file; one of "montlhy", "daily", "3h"
        defauts to "monthly"
      version (str, optional)
        CAMS version to use, used to infer the variable names from the 'default_inputs' dictionary
      variables (dict, optional)
        dictionary containing variable names including:
         - 'altitude',
         - 'file_start_str',
         - 'height',
         - 'hlevel',
         - 'lat',
         - 'lon',
         - 'time',
         - 'z'
        if not given variables will be assumed to be the same as for CAMS CH4 v19
      outdir (str, optional)
        Directory in which to save outputs
        Outputs will be saved to a subdirectory : outdir/LPDM/bc/<domain>
      from_climatology (bool, optional)
        Whether BCs are being produced from climatologies
      verbose (bool, optional)
        Whether to print any updates
      test (bool, optional)
        If creating boundary conditions in a test units will not be converted
        Should never be True outside of testing
     Returns
        netcdf file
            Writes CAMS BC xarray.Datasets to a netcdf file in outdir/LPDM/bc/<domain>
            with a standardised filename:
            <species>_<domain>_<start_year><start_month>.nc'


    Make boundary condition files from the CAMS inversion product

    This uses:
        - readCAMSInversion to find and open the CAMS inversion files for the given date range
        - climatology.monthly_cycle and climatology.add_full_time to produce climatologies (if required)
        - create_CAMS_BC to produce boundary conditions for multiple date-time intervals within the date range

    Args:
        domain (str):
            Domain name, e.g., "EUROPE"
        start (str):
            Start date of output, e.g., "2015-01-01"
        end (str):
            End date of output, e.g., "2016-01-01"
        species (str, optional)
            Gas species, defaults to 'ch4'
        time_res (str, optional)
            Time resolution of the output BC file including:
                "3h", "daily", "monthly"
            Default "monthly"
        cams_version (str, int)
            version number of CAMS inversion, e.g. 'v19'
            if an int is entered, it will be converted to a str starting with 'v'
        outdir (str, optional)
            Output directory to save output. Output will automatically be written to outdir/LPDM/bc/DOMAIN
        cams_directory (str, optional)
            Location of CAMS inversion output
        clim_start (str, optional):
            Start date to average fields into a climatology, e.g., "2010-01-01"
            Default of none will assume start
        clim_end (str, optional):
            End date to average fields into a climatology, e.g., "2010-01-01"
            Default of none will assume end
        make_climatology (bool, optional)
            If True climatologies will be created from the dates clim_start & clim_end, or start & end if clim_start and clim_end are None
        fp_directory (str/None, optional):
            Footprint folder to use when extracting domain volume. By default, this will use your data_path and expected file structure by default.
            If specified, sub-folder of domain name will be assumed.
        variables (dict, optional)
            dictionary containing variable names including:
                'altitude', 'file_start_str', 'height', 'hlevel', 'lat', 'lon', 'time', & 'z'
            if not given variables will be assumed to be the same as for CAMS CH4 v19
        verbose (bool, optional)
            If False do not print updates
        overwrite (bool, optional)
            If True, files will automatically be overwritten
            If False and if the file already exists, the user will be asked to confirm that the file should be overwritten
        test (bool, optional)
            If creating boundary conditions in a test units will not be converted
            Should never be True outside of testing

    Returns:
        netcdf files of monthly boundary condition curtain files corresponding to the edges of the corresponding NAME domain

    Example:
        makeCAMSBC(domain, start, end, clim_start, clim_end)

    Todo:
        This code currently works as such:  if you want real CAMS inversion for a month (not a climatology), then you can only specify one year
        at a time. If multiple years are given in start and end, it will average them together. This code needs to be split up with a
        climatology keywork so that it does not automatically do averaging.


Create iterator over tuples (start_date, end_date), where end_date - start_date is
    one month, and the start dates begin at the initial year, and continue for n_years at
    the given frequency.

    Use "QS" for quarterly frequency (e.g. for testing) and "MS" for (start of) monthly frequency.


Create DataFrame with MHD baseline mean and std over each month starting at 1 Jan `initial_year`,
    for `n_years` with frequency `freq` (default freq. is quarterly).

    You might need to specify the inlet height for some species (e.g. CH4); it is usually "10m", but might
    be "24m" (or something else).


Create a flat BC prior with the given baseline values based on a template file.


    Expand out climatology dataset to include full date range from start to end.
    Expect input to be a climatology dataset with dimension of "month". This dimension will be expanded
    and replaced with the "time" dimension.

    Args:
        ds_seasonal (xarray.Dataset)
            Monthly averages with 'month' coordinate
        start, end (str)
            Start and end date range required to be added to ds_seasonal

    Returns:
        xarray.Dataset:
            dataset with a time coordinate in the range start:end, and the month coordinate removed


    Calculate mean across month axis only for each data variable. Do not collapse other axes.
    Used on Grouper object created by ds.groupby("month").

    Args:
        ds (xarray.Dataset)
            Emissions data
        time_col (str)
            Name of the time coordinate
            Defaults to 'time'

    Returns:
        xarray.Dataset
            Average emissions per month
            "time" dimension is replaced by integer "month" coordinates.


    Calculate the monthly cycle across time period of input object.

    Args:
        ds (xarray.Dataset)
            Dataset object to look at seasonal cycle. Must contain a time coordinate.
        time_col (str)
            Name of time co-ordinate in dataset.
            Default = "time".

    Output:
        xarray.Dataset
            Dataset with each variable averaged for each repeated month.
            "time" dimension is replaced by "month".
            "time" coordinate still contained within dataset but no longer attached to any data variables.


        Create boundary conditions which match to the NAME grid and save to a netcdf file

        Args
            fp_directory (str, optional)
                Path to the NAME footprint files which the input data will be matched to
            fp_height_coord (str, optional)
                Name of the heght coordinate n the footprint file
                Defaults to 'height'
            reverse : (bool/None, optional)
                Whether height values within th dataset input are in reverse order
                (i.e. nesw["gph"] level 1 values > nesw["gph"] level 2 values).
                Default = None. If this is set to None this will be automatically determined.
            convert_units (bool, optional)
                If True, units will be converted from mol/mol to parts per e.g. ppt, ppb, ppm
                The units to be converted to will be extracted from species_info.json
            datasource (str, optional)
                source from which data was taken
            out_path : str, optional)
                path to save outputs
            glob_attrs (dict, optional)
                global attributes to add to Dataset with:
                 - key : title of attributes
                 - value : description of attribute
                data will already include
                 - a title with the data source and self.species
                 - the author
                 - the date created
            copy_glob_attrs (bool/list, optional)
                If True, all global attributes from the original dataset will be copied
                to the new file
                Or a list of attributes can be given which will be copied
                If False, none will be copied
            verbose (bool, optional)
                Whether to print any updates


        Cut the vmr array to the edge of the region needed

        Args:
            fp_directory (str, optional)
                Path to the NAME footprint files which the input data will be matched to
                Defualts to <data_path>/LPDM/fp_NAME'
            fp_height_coord (str, optional)
                Name of the heght coordinate n the footprint file
                Defaults to 'height'
            convert_units (bool, optional)
                If True, units will be converted from mol/mol to parts per e.g. ppt, ppb, ppm
                The units to be converted to will be extracted from acrg_species_info.json
            verbose (bool, optional)
                Whether to print any updates

        Outputs:
            dict
                Adds a dictionary to the BoundaryConditions object which contans the edge vmrs
                for north, south, east, and west
                Can be accessed using BoundaryConditions_obj.edges


        Interpolates the data to the NAME heights, latitudes, and longitudes

        Args
            reverse (bool/None, optional)
                Whether height values within th dataset input are in reverse order
                (i.e. nesw["gph"] level 1 values > nesw["gph"] level 2 values).
                Default = None. If this is set to None this will be automatically determined.
            verbose (bool, optional)
                    Whether to print any updates

        Output
            xarray.Dataset
                Adjusts the edges dataset, replacing the 3D VMR arrays for all directions
                with 3D VMR arrays interpolated to the NAME heights, latitudes, and longitudes
                The adjusted dataset can be accessed using BoundaryConditions_obj.edges


        Interpolates the data to the NAME heights

        Args
            direction (str, optional)
                The compass direction of nesw
                Used for naming the output array : {self.species}_{direction}
            verbose (bool, optional)
            reverse (bool/None, optional)
                Whether height values within is nesw input are in reverse order
                (i.e. nesw["gph"] level 1 values > nesw["gph"] level 2 values).
                Default = None. If this is set to None this will be automatically determined.
            new_vmr_var (str, optional)
                name for interpolated vmr variable
                defaults to None, inwhich case the original variable name is used
            verbose (bool, optional)
                Whether to print any updates

        Output
            xarray.Dataset
                Adjusts the edges dataset, replacing the 3D VMR array for the given direction
                with a 3D VMR array interpolated to the NAME heights
                Can be accessed using BoundaryConditions_obj.edges


        Interpolates the data to the NAME latitudes and longitudes

        Args
            direction (str, optional)
                The compass direction of nesw
                Used for naming the output array : {self.species}_{direction}
            verbose (bool, optional)
                    Whether to print any updates
            reverse : bool/None
                Whether height values within is nesw input are in reverse order
                (i.e. nesw["gph"] level 1 values > nesw["gph"] level 2 values).
                Default = None. If this is set to None this will be automatically determined.
                name for interpolated vmr variable
                defaults to None, inwhich case the original variable name is used
            new_vmr_var (str, optional)
                name for interpolated vmr variable
                defaults to None, inwhich case the original variable name is used
            verbose (bool, optional)
                Whether to print any updates

        Output
            xarray.Dataset
                Adjusts the edges dataset, replacing the 3D VMR array for the given direction
                with a 3D VMR array interpolated to the NAME latitudes and longitudes
                Can be accessed using BoundaryConditions_obj.edges


        Create a standardised filename for boundary conditions file

        Args
            verbose (bool, optional)
                Whether to print any updates

        Returns
            A standardised filename (str)


        Args
            datasource (str, optional)
                source from which data was taken
            out_path (str, optional)
                path to save outputs
            glob_attrs (dict, optional)
                global attributes to add to Dataset with:
                 - key : title of attributes
                 - value : description of attribute
                data will already include
                 - a title with the data source and self.species
                 - the author
                 - the date created
            from_climatology (bool, optional)
                If True, a climatology tag will be added to the name of the output file
            copy_glob_attrs (bool/list, optional)
                If True, all global attributes from the original dataset will be copied
                to the new file
                Or a list of attributes can be given which will be copied
                If False, none will be copied
            verbose (bool, optional)
                Whether to print any updates

        Output
            Creates a netcdf file containing boundary conditions with the heights and latlon
            coordinates interpolated to match the NAME grid coordinates

            Includes DataArrays for the n, s, e, and w boundaries


Flatten any iterable values in dictionary.

    For example:

    >>>flatten({"a": (1, 2), "b": "XY"})
    [{"a": 1, "b": "X"}, {"a": 1, "b": "Y"}, {"a": 2, "b": "X"}, {"a": 2, "b": "Y"}]

    >>>flatten({"a": (1, 2), "b": 3})
    [{"a": 1, "b": 3}, {"a": 2, "b": 3}]

    Args:
        x: dictionary to flatten

    Returns:
        list of dictionaries with same keys as input dictionary and no iterable values;
    all combinations of iterable values in the input dictionary appear as a value in
    some dictionary in the output.


Create a DataFrame containing `n_periods` start and end dates starting at the given
    `year` and initial month (`initial_month` defaults to 1).

    Args:
        year: year to start first period
        n_periods: number of periods (months or years) in result
        frequency: length of periods, "annual" or "monthly"
        initial_month: month to start first period (default 1)

    Returns:
        DataFrame containing columns for start and end dates.


Read ini file and update key-value pairs according to `new_kwargs`.

    If a key in `new_kwargs` matches a key in the .ini file, the value of that
    key is set to the value in `new_kwargs`.

    Any items in `new_kwargs` that are not matched are added to the end of the
    the result in a new section.

    Args:
        ini_file: path to .ini file
        new_kwargs: dictionary of new key-value pairs to use.

    Returns:
        list of lines of the updated ini file.


If s is a match to toml_key_pat, say "<general.species>", then
    this will return the list ["general", "species"]


Find next substitution whose root is not in `skip`.


Recursively apply _format


Return a pandas Series by filling substitutions in `self.value`
        with values from `df`.


Take string of Python code for getting attributes and items and return
    function that does the same sequence of "getters" applied to first object.

    For instance 'obj.attr1[key1].attr2.attr3[key2]' --> function f so that
    f(obj) = obj.attr1[key1].attr2.attr3[key2].


Find params that have substitutions matching the array_row.


Make experiment directory with associated files.


Make script to run array job on SLURM.

    TODO: need to set output paths for .out, .err, and .ini


Search for surface and footprint data as specified by an .ini file.

    This searches for all surface and footprint data that would be used in an inversion.


Load template ini file and update species and output paths, as well as sites,
    averaging_period, inlet, instrument, and fp_height with options provided by site_info


    Define input parameters for satellite data to test
    Keys include:
        "domain" (str),"site" (str),"height" (str),"year" (int),"month" (int),"satellite" (bool), "upper_level" (bool), "max_level" (bool)


Define subfolder name for satellite data


    Define input parameters for dummy satellite data (based on sat_param)

    Want simplified
     - upper_level
     - max_level
    values


    Define subfolders within NAME output directory structure
    Keys includes:
        "fields_folder","particles_folder","met_folder","processed_folder","obs_folder"


Read first satellite by day field file using process.read_file() function and create output.


Read first satellite by point field file using process.read_file() function and create output.


    Defining dummy values for fields files. Should be consistent with
    the dummy fields file data which we can check against.

    Matching to values within:
      files/LPDM/raw_output/Satellite_ByDay/SOUTHAMERICA_GOSAT-BRAZIL_column/MixR_files/Fields_SOUTHAMERICA_GOSAT-BRAZIL_column_synthetic_20130101.txt.gz

    Data include 4 fields entries for 2 lat, lon points.
    Overall grid is 100 x 100


Create output from process.define_grid() function for one satellite by day field file


Create output from process.define_grid() function for one satellite by point field file


Define height range for input into particle_locations function.


    Defining dummy values for particle histograms. Should be consistent with
    the dummy particle location data which we can check against.

    Data include 4 id values, 2 points with 2 levels for each, 16 particles (2x2x4).
    Point 001 (id 1, 2): 2 particles at each of the south and west boundary (none at north and east)
    Point 002 (id 3, 4): 1 particle at each boundary (n, e, s, w)


    Defining dummy values for the satellite met data. Should be consistent with
    the dummy satellite met data which we can check against

    TODO: May want to read this in in some way rather than just define values


    Read met data for satellite run with points grouped by day.
    Returns: list of dataframes.


    Read met data for satellite data is separated into points.
    Returns: list of dataframes.


    Read met data for satellite run with points grouped by day.
    Separates points into individual dataframes.??
    Returns: nested list of dataframes.


    Read met data for satellite run is separated into points.
    Separates points into individual dataframes.
    This is the input format expected by the original process script.
    Returns: nested list of dataframes.


Calculate the area using the dataframe extracted from a fields file


    Extract fields files values from dummy file and calculate unit conversion.
    Outputs a dataframe which has been stacked to include all values in one column
    and has the associated converted values.

    Also adds the correct indices for the associated fp dataset (time,lev,lat,lon)
    as:
        coord"_index" i.e. "lat_index","lon_index","time_index","lev_index"

    Returns:
        pandas.DataFrame,str:
            Dataframe containing the values, name of the new column containing
            the converted values.


    Define parameters for input into footprint_concatenate function for satellite files when grouped by day.
    **NOTE: At the moment "datestr" and "met" keys are linked to a lists of datestr and met data rather
    than one value. Need to re-assign "datestr" and "met" keys to singular values in param dictionary before
    passing to function **


    Define parameters for input into footprint_concatenate function for satellite files when separated by point.
    **NOTE: At the moment "datestr" and "met" keys are linked to a lists of datestr and met data rather
    than one value. Need to re-assign "datestr" and "met" keys to singular values in param dictionary before
    passing to function **


    Create output from footprint_concatenate function for satellite data seperated by point.
    Returns list of xarray.Dataset objects. One for each footprint, one footprint per point.


    Create output from footprint_concatenate function for satellite data grouped by day.
    Returns list of xarray.Dataset objects. One for each footprint, one footprint per day.


    Define input parameters for process.process function for satellite data grouped by day.
    Additional "base_dir" parameter added to point to Satellite_ByDay folder.


    Define input parameters for process.process function for satellite data separated by point.
    Additional "base_dir" parameter added to point to Satellite_ByPoint folder.


Create dataset from gosat data with no processing.


Create gosat dataset with coordinate values added (necessary for most additional processing steps).


 Create a dataset with a date offset from the first of the month to check pressure values are
    being selected correctly from the first day.


Create gosat dataset object from data with repeated entries (2014-12-05).


    Define input parameters for site data to test.
    Keys include:
        "domain" (str),"site" (str),"height" (str),"year" (int),"month" (int),"satellite" (bool)


    Define input parameters for site data to test.
    Keys include:
        "domain" (str),"site" (str),"height" (str),"year" (int),"month" (int),"satellite" (bool), "species" (str)


Define full subfolder path for site data.


    Define subfolders within NAME output directory structure
    Keys includes:
        "fields_folder", "fields_folder_hourly", "particles_folder","met_folder","obs_folder",
        (also "mixr_fields_folder","mixr_hourly_folder" but largely deprecated)


Define values in the test fields file


Benchmark infromation from the dummy particle file


Benchmark infromation from the dummy particle file, replacing values on E and W directions with 0 to test this feature


Read first site field file using process.read_file() function and create output.


Create output from process.define_grid() function for one site field file


    Reads in met files based on extracted met filenames. Expects nested list of met files and processes each
    list of met_files together so they are output as one dataframe per set of met files.
    Args:
        met_files (nested list):
            List of each set of met files. One set of met files extracted for each input date string
            (date string is based on fields filenames).
        satellite (bool, optional):
            Whether to process as satellite data or site data.
    Returns:
        list(pandas.Dataframe):
            List of dataframes, one for each set of met_files


    Reads in met files based on extracted met filenames. Expects nested list of met files but processes each
    met_file separately so they are output as one dataframe per met file.
    Args:
        See read_met_files function
    Returns:
        nested list(pandas.Dataframe):
            List of dataframes, one list for set of met files, one dataframe for every met file.


    Define parameters for input into footprint_concatenate() function.
    Args:
        subfolder (str) :
            Path to subfolder of NAME output. The final folder is based on the domain_site_height/ syntax
            e.g. SOUTHAMERICA_GOSAT-BRAZIL_column.
        read_met (list) :
            Output from any read_met_files() or read_met_files_separate function. Should contain list of
            dataframes (or nested list of dataframes)
        folder_names (dict) :
            Dictionary of folder_names - see folder_names() fixture
        parameters (dict) :
            Dictionary of parameters relevant to run - see satellite_param() or site_param() fixtures
        satellite (bool) :
            If the input data is satellite data
        get_time_step (float / None, optional) :
            Optional if satellite=True; not needed if satellite=False.
            Time step value - see get_time_step function
        #get_obs_files (list, optional)
        #    Only needed if satellite=True.
        #    List of observations files.

    Returns:
        dict :
            Key-value pairs matching input into footprint_concatenate function.
            **NOTE: At the moment "datestr" key is linked to a list of datestr rather than one datestr value.
            Need to re-assign "datestr" key to one datestr value to param dictionary before passing to function **


    Define parameters for input into footprint_concatenate function for site files.
    **NOTE: At the moment "datestr" and "met" keys are linked to a lists of datestr and met data rather
    than one value. Need to re-assign "datestr" and "met" keys to singular values in param dictionary before
    passing to function **


    Define input parameters for process.process function.
    Args:
        input_param (dict) :
            Dictionary of input parameters - see satellite_param and site_param fixtures
        folder_names (dict) :
            Dictionary of folder names within subfolder - see folder_names fixture
        satellite (bool) :
            If input parameters are related to satellite data.
            If so, additional parameters for "max_level" and "upper_level" will be added.
    Returns:
        dict :
            Key-value pairs to pass to process function


    Define input parameters for process.process function for site data.
    Additional "base_dir" parameter added to point to Site folder.


Define an example prior to be used throughout the test suite


Define hbmcmc config file input to run hbmcmc code


Import the fields files for the sperate inventories and embedded version


Define set of measurement parameters to be used throughout the test suite


Define set of parameters to be used with tests on the SMALL-DOMAIN


Define set of parameters for basis functions to be used throughout the test suite


Define set of satellite measurement parameters to be used throughout the test suite


Define set of input parameters for the filenames() function. Based on measurement_param


Define set of input parameters for footprints() function. Based on measurement_param


Define set of input parameters for flux() function. Based on measurement_param


Define set of input parameters for flux_HiTRes() function.


Create flux dataset using flux_for_HiTRes for associated tests


Define set of input parameters for boundary_conditions() function. Based on measurement_param


Define set of input parameters for basis() function. Based on measurement_param


Define set of input parameters for basis_boundary_conditions() function. Based on measurement_param


    Define set of input parameters to use with footprint_data_merge() function.
    Note: cannot specify data directory directly.


 Define set of input parameters to use with footprint_data_merge() function. Edited for use with satellite data.
    Note: cannot specify data directory directly.


Create fp_and_data dictionary from footprints_data_merge() function


Define parameters for fp_sensitivity() function


Define MCMC parameter type nested dictionary for tdmcmc config input


Define parameter type nested dictionary for example config file


Function to check correct parameters and types are being returned


    Define parameters for input into footprint_concatenate() function.
    Args:
        subfolder (str) :
            Path to subfolder of NAME output. The final folder is based on the domain_site_height/ syntax
            e.g. SOUTHAMERICA_GOSAT-BRAZIL_column.
        read_met (list) :
            Output from any read_met_files() or read_met_files_separate function. Should contain list of
            dataframes (or nested list of dataframes)
        folder_names (dict) :
            Dictionary of folder_names - see folder_names() fixture
        parameters (dict) :
            Dictionary of parameters relevant to run - see satellite_param() or site_param() fixtures
        satellite (bool) :
            If the input data is satellite data
        fields (str, optional) :
            What are the input fields:
                "mixr" or "conc"
            Will use to determine folder to look for.
            Default = "mixr"
        get_time_step (float / None, optional) :
            Optional if satellite=True; not needed if satellite=False.
            Time step value - see get_time_step function

    Returns:
        dict :
            Key-value pairs matching input into footprint_concatenate function.
            **NOTE: At the moment "datestr" key is linked to a list of datestr rather than one datestr value.
            Need to re-assign "datestr" key to one datestr value to param dictionary before passing to function **


    Find NAME processed file based on search string:
        "{site}-{height}[_{species}]_{domain}_{year}{month:02}*.nc"
    Args:
        #subfolder (str) :
        #    Main sub-folder containing NAME output files
        #processed_folder (str) :
        #    Directory containing the processed file.
        param (dict) :
            Parameter dictionary containing
            'site','height','domain','year','month', 'species', 'process_dir'
    Returns
        str / list:
            Filename
        None :
            If no files or more than one file is found


    Check and remove any files matching datestr extracted from param input from processed files folder.
    This is to allow test of process function to be run multiple times.
    Processed files are found within processed files folder based on datestr from
    param["year"] + param["month"].zfill(2)


    Define benchmark bc file

    vmr_n, vmr_s, vmr_e, and vmr_w are 2D arrays of ones (height and lat or lon)


Define output directory


Define output filename


Empty real-space array (backend-dependent)


Transform from real- to Fourier-space.


Transform from Fourier- to real-space.


Pixel window functions including first-order alias corrections (copied from nbodykit)


Set-up the relevant FFT modules


        Generate a Gaussian periodic map with a given set of P(k) multipoles (or the fiducial power spectrum monopole, if unspecified).
        The input Pk are expected to by in the form {k, P_0, [P_2], [P_4]}, where P_2, P_4 are optional.

        No mask is added at this stage (except for a pixelation window, unless include_pixel_window=False), and the output can be in real- or Fourier-space.


Apply a trivial filter to the data, converting to real/Fourier-space as appropriate.


Apply the exact inverse weighting A^{-1} to a map. This assumes a diagonal-in-ell C_l^{XY} weighting, as produced by generate_data.

        Note that the code has two input and output options: "fourier" or "real", to avoid unnecessary transforms.


Compute Sum_m Y_lm(k) a_lm^*(k) for some a(x)


Compute Sum_m Y_lm(x) a_lm^*(x) for some a(k)


Apply xi(x,y) to a map (symmetrically), given a gridded power spectrum. This is used to compute the covariance matrices.


Compute the numerator of the unwindowed power spectrum estimator, using the custom weighting function S^-1.


This computes the contribution to the Fisher matrix from a single GRF simulation, created internally.

        An optional input is a function that applies (S^-1)^dagger to a map. The code will assume (S^-1)^dagger = S^-1 if this is not supplied.


This computes the contribution to the Gaussian covariance matrix from a single GRF simulation, created internally.

        This requires an input theory power spectrum, Pk_cov (without shot-noise).

        An optional input is a function that applies (S^-1)^dagger to a map. The code will assume (S^-1)^dagger = S^-1 if this is not supplied.


This computes the a rectangular matrix used to window-convolve the theory, using a single GRF simulation, created internally.
        We also compute and output the contribution to the (square) Fisher matrix, used for normalization.

        We must specify the desired theory binning and lmax.

        Finally, we can optionally add wide-angle effects (for the theory matrix, not the Fisher matrix).


Apply the pointing matrix to a map. This multiplies by the mask and optionally includes IC effects.


Internal function to compute the contribution to the Fisher matrix, covariance, or theory matrix.


Apply S^-1 P or S^-1 C S^-dagger to a map. Note that the input is either in Fourier-space if const_mask=True else in real-space.


Apply (S^-1 P)^dagger weighting to maps. Input is either a Fourier-space map or a real-space map.


Compute coefficients for numerical derivative of k * dP(k)/dk


Compute a single row of the Fisher matrix (and optionally the binning matrix), given a Q map.


Integrate two maps over all k-bins


This computes the contribution to the shot-noise from a single GRF simulation, created internally.


Compute the Fisher matrix using N_it realizations. Since the calculation is already parallelized, this is run in serial.


        Compute the unwindowed power spectrum estimator.

        Note that the Fisher matrix and shot-noise terms must be computed before this is run, or they can be supplied separately.

        The Poisson shot-noise can be optionally subtracted. We return the imaginary part of any ell=odd spectra.


Internal function to compute the power spectrum numerator. This is used by the Pk_numerator and Pk_numerator_ideal functions.


Compute the numerator of the idealized power spectrum estimator, weighting by 1/P_fid(k) within each bin.

        The estimator does *not* use the mask or S^-1 weighting schemes. It also applies only for even ell. We normalize by the mean of the squared mask.

        This can compute the ideal power spectrum of simulation volumes, or, for suitably normalized input, the FKP power spectrum.


This computes the idealized Fisher matrix for the power spectrum, weighting by 1/P_fid(k) within each bin.

        We optionally include discreteness correction for ell>0. We can use a new fiducial power spectrum by specifying Pk_fid (which is None for unit spectrum).

        If 'local' sightlines are being used, this assumes the distant-observer approximation.


Compute the (normalized) idealized power spectrum estimator, weighting by 1/P_fid(k) within each bin.

        The estimator does *not* use the mask or S_inv weighting schemes (except for normalizing by < mask^2 >. It also applies only for even ell.

        We optionally include discreteness corrections in the Fisher matrix for ell>0.

        This can compute the ideal power spectrum of simulation volumes, or, for suitably normalized input, the FKP power spectrum.


Compute Sum_k d(k)^2 L_l1(k.n)L_l2(k.n) across all k in range for real d(k).


Compute Legendre polynomial L_l(mu) for l in [0, 6], using explicit formulas.


Compute pointing matrix with radial integral constraint.


Compute pointing matrix transpose with radial integral constraint.


Compute the real spherical harmonics on the coordinate grid. This is performed in numpy (not jax) to avoid memory errors.


Assemble the bispectrum numerator from a set of g_lb(x) maps.


Return one if modes in the bin satisfy the triangle conditions, or zero else.

        This is used either for all triangles in the bin, or just the center of the bin.


Compute symmetry factor giving the degeneracy of each bin. For l>0 this is computed using FFTs.


        Process a single input simulation. This is used for the linear term of the bispectrum estimator.

        We return g_{b,l}(r) maps for this simulation.


        Load in and preprocess N_sim Monte Carlo simulations used in the linear term of the bispectrum estimator.
        These can alternatively be generated with a fiducial spectrum using the generate_sims script.

        The input is a function which loads the simulations in real- or Fourier-space given an index (0 to N_sims-1).

        If preload=False, the simulation products will not be stored in memory, but instead accessed when necessary. This greatly reduces memory usage, but is less CPU efficient if many datasets are analyzed together.


        Generate N_sim Monte Carlo simulations used in the linear term of the bispectrum generator.
        These are pure GRFs simulations, generated with the pointing matrix and pixel-window-function, as well as the power spectrum defined in the base class (or Pk_input).

        If preload=True, we create N_it such simulations and store the relevant transformations into memory.
        If preload=False, we store only the function used to generate the sims, which will be processed later. This is cheaper on memory, but less CPU efficient if many datasets are analyzed together.

        We can alternatively load custom simulations using the load_sims script.


Generate a single GRF simulation and produce g_bl maps


        Compute the numerator of the unwindowed bispectrum estimator, using the custom weighting function S^-1.

        We can optionally include a linear term to reduce the large-scale variance.


This computes the contribution to the Fisher matrix from a single pair of GRF simulations, created internally.


This computes the contribution to the Gaussian covariance matrix from a single pair of GRF simulations, created internally.

        This requires an input theory power spectrum, Pk_cov (without shot-noise). We can feed in a (two-point) shot-noise mask.


Internal function to compute the contribution to the Fisher matrix or covariance.


            Assemble and return the Q [ = partial_alpha zeta_ijk (weighting a)_j (weighting a)_k] maps in Fourier-space, given a weighting scheme.

            The outputs are Q_alpha arrays.


        This computes the shot-noise from a single pair of GRF simulations, created internally.

        We can optionally drop the quadratic term (~ P(k)/nbar).

        The quadratic piece involves both the singly- and doubly-weighted data (data and data2). If unspecified, we set data2 = data.


        Compute the unwindowed bispectrum estimator, using the custom weighting function S^-1.

        Note that the Fisher matrix must be computed before this is run, or it can be supplied separately.

        We can optionally drop the linear term (at the cost of slightly enhanced large-scale variance).


Internal function to compute the bispectrum numerator. This is used by the Bk_numerator and Bk_numerator_ideal functions.


Compute the numerator of the idealized bispectrum estimator, weighting by 1/P_fid(k) within each bin.

        The estimator does *not* use the mask or S_inv weighting schemes, and does not remove the linear term (which vanishes under ideal circumstances). It also applies only for even ell.

        This can compute the ideal bispectrum of simulation volumes, or, for suitably normalized input, the FKP bispectrum.


This computes the idealized Fisher matrix for the power spectrum, weighting by 1/P_fid(k) within each bin.

        We optionally include discreteness correction for ell>0.

        If 'local' sightlines are being used, this assumes the distant-observer approximation.


Compute the (normalized) idealized bispectrum estimator, weighting by 1/P_fid(k) within each bin.

        The estimator does *not* use the mask or S_inv weighting schemes (except for normalizing by < mask^3 >, and does not remove the linear term (which vanishes under ideal circumstances)

        We optionally include
        discreteness corrections for ell>0.

        This can compute the ideal bispectrum of simulation volumes, or, for suitably normalized input, the FKP power spectrum.


 Returns a MixtureOfGaussians profile that has been
        affine-transformed into the pixel space of the image.


    Produce a ball of walkers around an initial parameter value 'p0'
    with axis-aligned standard deviation 'stdev', for 'nw' walkers.


	MP_BAD  =                    0
	Bit 0 has 2500 pixels set
	MP_SAT  =                    1
	Bit 1 has 5771 pixels set
	MP_INTRP=                    2
	Bit 2 has 11269 pixels set
	MP_CR   =                    3
	Bit 3 has 136 pixels set
	MP_EDGE =                    4
	Bit 4 has 11856 pixels set
	HIERARCH MP_DETECTED =       5
	Bit 5 has 37032 pixels set


        Compute the black-body formula, for a given lnT.

        'lam' is wavelength in meters
        'lnT' is log-temperature in Kelvin

        Return value is in [J s^{-1} m^{-2} m^{-1} sr^{-1}],
        power radiated per square meter (area), per m of wavelength, per steradian

        BUGS: formula needs to be audited


        Compute the black-body formula, for a given lnT.

        'lam' is wavelength in meters
        'lnT' is log-temperature in Kelvin

        Return value is in [J s^{-1} m^{-2} Hz^{-1} sr^{-1}],
        power radiated per square meter (area), per Hz of frequency, per steradian

        BUGS: formula needs to be audited


        Compute the derivative of the black body formula with
        respect to lnT.  Same inputs as `black_body_nu()`.

        Return value is in [J s^{-1} m^{-2} Hz^{-1} sr^{-1}],

        BUGS: formula needs to be audited


        Returns an image giving the brightness of this DustSheet in
        the given image.  This is a grid the same shape as the
        DustSheet, but in the image's units.


        Pre-compute the "grid-spread function" transformation matrix
        for this parameter grid to the given image pixels.

        The result is a home-brewed sparse matrix representation: a
        dictionary mapping from model grid pixel indices (integers) to
        the tuple (I, G, nz, NZI), where:

        I: numpy index array (integers) in the image
        G: grid weights for those pixels
        nz = ((x0,y0), (h,w)): the subimage with non-zero weights
        NZI: numpy index array (integers) within the "nz" subimage


    Brittle function to read Groves data sample and make the
    things we need for fitting.

    # issues:
    - Need to read the WCS too.
    - Need, for each image, to construct the rectangular
    nearest-neighbor interpolation indices to image 0.
    - Can I just blow up the SPIRE images with interpolation?


    Read photoObjs that are inside the given 'wcs', plus 'margin' in degrees.

    If 'pa' is not None, assume it is a PrimaryArea object.


select probpsf, psfmag_i, devmag_i, expmag_i, cmodelmag_i, fracDev_i, ra, dec into mydb.MyTable from photoprimary
	where
	  ra between 333.77 and 334.77
	    and dec between -0.14 and 0.91


	select run,camcol,field, RA,Dec,probpsf, nchild,
	psfmag_u,psfmag_g,psfmag_r,psfmag_i,psfmag_z,
	modelmag_u, modelmag_g, modelmag_r, modelmag_i, modelmag_z
	into mydb.MyTable_2 from PhotoObjAll
	where ra between 15.77 and 16.77
	and dec between -0.11 and 0.95
	-> cas-DR7-S82-p18p.fits


Compare parallax (1) vs no-parallax (2).


	select fracdev_r,exprad_r,expab_r,expmag_r,expphi_r,run,camcol,field into mydb.exp3818
	from Galaxy where run=3818
	and clean=1 and probpsf=0
	and (flags & (dbo.fPhotoFlags('BINNED1'))) = (dbo.fPhotoFlags('BINNED1'))
	and (flags & (dbo.fPhotoFlags('INTERP') | dbo.fPhotoFlags('BLENDED') |
    dbo.fPhotoFlags('BINNED2') | dbo.fPhotoFlags('MOVED') | dbo.fPhotoFlags('DEBLENDED_AT_EDGE') |
    dbo.fPhotoFlags('MAYBE_CR') | dbo.fPhotoFlags('MAYBE_EGHOST'))) = 0
    and fracdev_r < 0.5
	=> exp3818_dstn.fit

	select fracdev_r,exprad_r,expab_r,expmag_r,expphi_r,run,camcol,field,ra,dec into mydb.exp3818
	from Galaxy where
	run=3818
	and camcol=2
	and clean=1 and probpsf=0
	and (flags & (dbo.fPhotoFlags('BINNED1'))) = (dbo.fPhotoFlags('BINNED1'))
	and (flags & (dbo.fPhotoFlags('INTERP') | dbo.fPhotoFlags('BLENDED') |
    dbo.fPhotoFlags('BINNED2') | dbo.fPhotoFlags('MOVED') | dbo.fPhotoFlags('DEBLENDED_AT_EDGE') |
    dbo.fPhotoFlags('MAYBE_CR') | dbo.fPhotoFlags('MAYBE_EGHOST'))) = 0
	and fracdev_r < 0.5
	and exprad_r between 0.4 and 0.55
	and expmag_r between 21 and 22
	=> exp3818b.fits

	select fracdev_i,exprad_i,expab_i,expmag_i,expphi_i,run,camcol,field,ra,dec into mydb.exp3818i
	from Galaxy where run=3818 and camcol=2
    and clean=1 and probpsf=0
	and (flags & (dbo.fPhotoFlags('BINNED1'))) = (dbo.fPhotoFlags('BINNED1'))
	and (flags & (dbo.fPhotoFlags('INTERP') | dbo.fPhotoFlags('BLENDED') |
	dbo.fPhotoFlags('BINNED2') | dbo.fPhotoFlags('MOVED') | dbo.fPhotoFlags('DEBLENDED_AT_EDGE') |
	dbo.fPhotoFlags('MAYBE_CR') | dbo.fPhotoFlags('MAYBE_EGHOST'))) = 0
	and fracdev_i < 0.5
	=> exp3818i.fits


	select fracdev_i,exprad_i,expab_i,expmag_i,expphi_i,run,camcol,field,ra,dec into mydb.exp3818i_dr7
	from Galaxy where run=3818
    and probpsf=0
	and (flags & (dbo.fPhotoFlags('BINNED1'))) = (dbo.fPhotoFlags('BINNED1'))
	and (flags & (dbo.fPhotoFlags('INTERP') | dbo.fPhotoFlags('BLENDED') |
	dbo.fPhotoFlags('BINNED2') | dbo.fPhotoFlags('MOVED') | dbo.fPhotoFlags('DEBLENDED_AT_EDGE') |
	dbo.fPhotoFlags('MAYBE_CR') | dbo.fPhotoFlags('MAYBE_EGHOST'))) = 0
	and fracdev_i < 0.1
	## In DR7 context of SDSS3 CAS
	=> exp3818i_dr7.fits


	select fracdev_i,exprad_i,expab_i,expmag_i,expphi_i,run,camcol,field,ra,dec,flags
	into mydb.exp5 from Galaxy where
	exprad_i > 3
	and expmag_i < 20
	and clean=1 and probpsf=0
	and fracdev_i < 0.5

	select * from mydb.exp5 into mydb.exp5b
	where (flags & 0x10000000) = 0x10000000
	and (flags & 0x3002000a0020008) = 0


    Given filenames for CFHT coadd image and weight files, produce
    a tractor.Image object.

    *roi*: (x0,x1, y0,y1): a region-of-interest in pixel space;
           returns the subimage [x0,x1), [y0,y1).
    *radecroi*: (ra0, ra1, dec0, dec1): a region-of-interest in RA,Dec space;
           returns the subimage bounding the given RA,Dec box [ra0,ra1], [dec0,dec1].
    *filtermap*: dict, eg,  { 'i.MP9701': 'i' }, to map from the FILTER header keyword to
           a standard filter name.


Locate all files matching supplied filename pattern in and below
    supplied root directory.


slc: slice of image to read


    mu: shape (C,2) or (2,)
    sigma: shape (C,) or scalar
    weights: shape (C,) or 1.
    C: number of Gaussian components

    X: (N,2)


    Nice PR image for NOAO.  Make a synthetic brick centered on a nice galaxy in 374451

    bricks.txt:
    # brickid ra dec ra1 ra2 dec1 dec2
    1 244.70 7.41 244.5739 244.8261 7.285 7.535
    2 244.70 7.41 244.5739 244.8261 7.285 7.535

    text2fits.py -f jdddddd PR/bricks.txt PR/decals-bricks.fits
    cp decals/decals-ccds.fits PR
    (cd PR; ln -s ~/cosmo/work/decam/versions/work/calib .)
    (cd PR; ln -s ~/cosmo/work/decam/versions/work/images .)
    export DECALS_DIR=$(pwd)/PR
    python -u projects/desi/tunebrick.py -b      1 -s primage -P "pickles/PR-%(brick)06i-%%(stage)s.pickle"
    python -u projects/desi/tunebrick.py -b 374441 -s primage -P "pickles/PR-%(brick)06i-%%(stage)s.pickle"


        This is used during fitting.  When used afterwards, you just
        want to use the getPointSourcePatch() and similar methods
        defined in the parent class.


        Produce a ball of walkers around an initial parameter value 'p0'
        with axis-aligned standard deviation 'stdev', for 'nw' walkers.


    Build a co-add, find discrepant pixels in the individual exposures
    and mask them, then redo the coadd.


    Given a list of tractor sources *cat*
    and a list of unWISE tiles *tiles* (a fits_table with RA,Dec,coadd_id)
    runs forced photometry, returning a FITS table the same length as *cat*.


 cov:
    BAND    =                    1 / wavelength band number
    WAVELEN =                3.368 / [microns] effective wavelength of band
    COADDID = '3342p000_ab41'      / atlas-image identifier
    MAGZP   =                 20.5 / [mag] relative photometric zero point
    MEDINT  =      4.0289044380188 / [DN] median of intensity pixels


    select
      run, rerun, camcol, field, nChild, probPSF,
      psfFlux_u, psfFlux_g, psfFlux_r, psfFlux_i, psfFlux_z,
      deVRad_u, deVRad_g, deVRad_r, deVRad_i, deVRad_z,
      deVAB_u, deVAB_g, deVAB_r, deVAB_i, deVAB_z,
      deVPhi_u, deVPhi_g, deVPhi_r, deVPhi_i, deVPhi_z,
      deVFlux_u, deVFlux_g, deVFlux_r, deVFlux_i, deVFlux_z,
      expRad_u, expRad_g, expRad_r, expRad_i, expRad_z,
      expAB_u, expAB_g, expAB_r, expAB_i, expAB_z,
      expPhi_u, expPhi_g, expPhi_r, expPhi_i, expPhi_z,
      expFlux_u, expFlux_g, expFlux_r, expFlux_i, expFlux_z,
      fracDeV_u, fracDeV_g, fracDeV_r, fracDeV_i, fracDeV_z,
      flags_u, flags_g, flags_r, flags_i, flags_z,
      probPSF_u, probPSF_g, probPSF_r, probPSF_i, probPSF_z,
      ra, dec
      from PhotoPrimary
      where ra between 333.5 and 335.5 and dec between -0.5 and 1.5


    imgshape = (H,W): full image size

    If not None, roi = (x0, x1, y0, y1) defines a region-of-interest
    in the image, in zero-indexed pixel coordinates.  x1,y1 are
    NON-inclusive, ie, x in [x0,x1);
    roi=(0,100,0,100) will yield a 100 x 100 image.

    "roiradecsize" = (ra, dec, half-size in pixels) indicates that you
    want to grab a ROI around the given RA,Dec.

    "roiradecbox" = (ra0, ra1, dec0, dec1) indicates that you
    want to grab a ROI containing the given RA,Dec ranges.


        Returns True if the current parameter values are legal; ie,
        have > 0 prior.


d: dict of (string, int) parameter names->indices


Yields  (name,val) tuples, where "name" is None if the parameter is not named.


This is a (non-recursive) basic implementation


Returns the raw index of the i-th liquid parameter.


Yields (i,j), for i, the liquid parameter and j, the raw index.


        Yields *index-ignoring-freeze-state*,sub
        for unfrozen subs.


Count unpinned (active) params.


Scales this sky model by a factor of *s*.


        This class does lazy evaluation of the spatial-variation fit.
        This methods does the evaluation if necessary.


Returns a PSF model at the given pixel position (x, y)


Return PSF model parameters at the given pixel position x,y.


Returns the shape of the PSF


Returns the number of eigen-PSFs -- the number of terms in the expansion.


Returns the N x H x W eigen-PSF images


Returns an image of the PSF at the given pixel coordinates.


        Parameters
        ----------
        term : None or int
            If None, plot all components.  If an integer, plot that PSF component.


        Shifts this PSF model so it applies to the subimage starting at (dx,dy).

        Returns
        -------
        None


If merge: merge params "amp0", "amp1", ... into an "amp" array.


    Computes the area of the rectangular region in an integral image.

    (xx0,yy0) are one less than the first pixel included in the sum.
    (xx1,yy1) are the last pixel included in the sum.

    (Note that the inclusiveness is opposite from usual.)

    These can be (1-d) arrays, but they must be the same shape.

    The return value is:

    sum_{i=xx0 + 1}^{xx1} sum_{j = yy0 + 1}^{yy1} I[j,i]

where I is the original image

    >>> if True:
    ...     from numpy.random import random_sample, seed
    ...     seed(42)
    ...     I = random_sample((10,10))
    ...     # This is the explicit sum
    ...     R0 = sum(I[3:5, 4:8])
    ...     print round(R0, 5)
    ...     # This is the integral-image sum
    ...     intimg = integral_image(I)
    ...     R1 = intimg_rect(intimg, 4-1, 8-1, 3-1, 5-1)
    ...     print round(R1, 5)
    3.9294
    3.9294


        brightness: SdssFlux object
        returns: float


        extras: [ ('key', [eim0,eim1,eim2]), ... ]

        Extra fields to add to the "FitStats" object, populated by
        taking the profile-weighted sum of 'eim*'.  The number of
        these images *must* match the number and shape of Tractor
        images.


Returns position angle in *radians*


Returns minor/major axis ratio.


Returns the "usual" ellipticity e in [0,1]


Returns the "softened" ellipticity ee in [0, inf]


        Converts a traditional magnitude zeropoint to a scale factor
        by which nanomaggies should be multiplied to produce image
        counts.


        Converts a scale factor (by which nanomaggies should be
        multiplied to produce image counts) into a traditional
        magnitude zeropoint.


        NOTE, "scale" is transposed vs earlier versions of this code!

        NOTE, does NOT make a copy of amplitude!!

        shift: D-vector offset
        scale: DxD-matrix transformation


        NOTE, does NOT make a copy of amplitude and mean!!

        shift: D-vector offset
        scale: DxD-matrix transformation


        [x0,x1): (int) X values to evaluate
        [y0,y1): (int) Y values to evaluate
        (cx,cy): (float) pixel center of the MoG


        minval: small value at which to stop evaluating

        [x0,x1): (int) X values to evaluate
        [y0,y1): (int) Y values to evaluate
        (cx,cy): (float) pixel center of the MoG

        Returns: numpy array of shape (y1-y0, x1-x0)


mask: np array of booleans (NOT Patch object!)


        minval: small value at which to stop evaluating

        [x0,x1): (int) X values to evaluate
        [y0,y1): (int) Y values to evaluate
        (fx,fy): (float) pixel offset of the MoG; ie, evaluate MoG shifted by
                this amount.

        'maxmargin': don't render sources more than this distance outside the box.

        If 'doslice' is True, slices the images down to the non-zero
        bounding-box.

        If 'derivs' is True, computes and returns x and y derivatives too.

        Unlike evaluate_grid_approx, returns a Patch object.


    `mixture`: a MixtureOfGaussians
    `x0,x1,y0,y1`: integer bounds [x0,x1), [y0,y1) of the grid to evaluate

    Returns: a Patch object


        Embeds this PSF image into a larger or smaller image of shape H,W.

        Return (img, cx, cy), where *cx*,*cy* are the coordinates of the PSF
        center in *img*.


        optional P0 = (w,mu,var): initial parameter guess.

        w has shape (N,)
        mu has shape (N,2)
        var (variance) has shape (N,2,2)

        optional xy0 = int x0,y0 origin of stamp.


        optional P0 = (list of floats): initial parameter guess.

        (parameters of a GaussianMixtureEllipsePSF)


    Computes the area of a given rectangle, given an integral image for
    an image that is assumed to be mirror-symmetric in both axes.

    That is, if the image size is N x N, then I[i, j] == I[N-i-1, j]
    and I[i, j] == I[i, N-j-1].

    The image must be odd-sized and square.

    "midline" is (image size - 1)/2; if the image is 2N-1, midline == N.

    "img" must be an integral image of size at least midline+1 x midline+1.

    (xx0,yy0) are one less than the first pixel included in the sum.
    (xx1,yy1) are the last pixel included in the sum.

    (Note that the inclusiveness is opposite from convention.)

    These can be (1-d) arrays, but they must be the same shape.

    The return value is:

        sum_{i=xx0 + 1}^{xx1} sum_{j = yy0 + 1}^{yy1} I[j,i]

    >>> if True:
    ...     from numpy import fliplr, flipud
    ...     from numpy.random import random_sample, seed
    ...     from integral_image import *
    ...     seed(42)
    ...     I = random_sample((11,11))
    ...     # symmetrize by copying the first quadrant
    ...     I[:,6:] = fliplr(I[:,0:5])
    ...     I[6:,:] = flipud(I[0:5,:])
    ...     intimg = integral_image(I)
    ...     symmimg = integral_image(I[:6,:6])
    ...     for (x0,x1,y0,y1) in [(-1,0,-1,0), (3,7,4,8), (-1,10,-1,10), (5,6,-1,0)]:
    ...         R0 = intimg_rect(intimg, x0, x1, y0, y1)
    ...         R1 = symm_intimg_rect(symmimg, x0, x1, y0, y1, 5)
    ...         print round(R0,5), round(R1,5), round(R0,5) == round(R1,5)
    0.37454 0.37454 True
    5.76586 5.76586 True
    58.09467 58.09467 True
    0.15602 0.15602 True


        Returns an "OptResult" duck with fields:

        .ims0, .ims1         (if wantims=True)
        .IV                  (if variance=True)
        .fitstats            (if fitstats=True)

        ims0, ims1:
        [ (img_data, mod, ie, chi, roi), ... ]

        ASSUMES linear brightnesses!

        ASSUMES all source parameters except brightness are frozen.

        If sky=False,
        ASSUMES image parameters are frozen.
        If sky=True,
        ASSUMES only the sky parameters are unfrozen

        ASSUMES the PSF and Sky models are position-independent!!

        PRIORS probably don't work because we don't setParams() when
        evaluating likelihood or prior!


        Performs *one step* of optimization.

        (Exactly what that entails depends on the optimizer; by
        default (LsqrOptimizer) it means one linearized least-squares
        + line search iteration.)

        Returns (delta-logprob, parameter update X, alpha stepsize)

        If variance=True,

        Returns (delta-logprob, parameter update X, alpha stepsize, variance)

        If just_variance=True,
        Returns variance.


        Performs multiple steps of optimization until convergence.

        Returns a dict of results (exact contents varying by optimizer).


This runs the mixture-of-Gaussians convolution method.


 Returns a MixtureOfGaussians profile that has been
        shear-transformed into the pixel space of the image.
        At px,py (but not offset to px,py).


        Returns the ``CD`` matrix at the given ``x,y`` pixel position.

        (Returns the constant ``CD`` matrix elements)


        Converts an :class:`tractor.RaDecPos` to a pixel position.
        Returns: tuple of floats ``(x, y)``


        Converts floats ``x``, ``y`` to a
        :class:`tractor.RaDecPos`.


        Returns the local pixel scale at the given *x*,*y* pixel coords,
        in *arcseconds* per pixel.


        Returns a tuple containing the state of this `Params` object
        for use as a cache key.

        All elements must be hashable: see
        http://docs.python.org/glossary.html#term-hashable


        Add the sky to the input synthetic image `mod`, a 2-D numpy
        array.


Subtracts a constant value from this sky model.


Shifts this sky model so that it applies to the subimage starting at x0,y0.


Converts `brightness`, a `Brightness` duck, into counts.

        Returns: float


        Converts a :class:`tractor.Position` *pos* into ``x,y`` pixel
        coordinates.

        Note that the :class:`tractor.Source` may be passed in; your
        :class:`tractor.WCS` could have color-specific behavior, for
        example.

        Returns tuple `(x, y)`, where `x` and `y` are floats, and `0,
        0` is the first pixel.

        Pixels are funny things.  Our convention is shifted by 1 from
        the FITS convention, so 0,0 is the *center* of the first
        ("zeroth", says Hogg) pixel, if you think of pixels as little
        boxes.  (What is the emoticon for "point and laugh"?)


        Returns a local affine relationship between `Position` and
        (x,y) pixel coordinates.  This is used, for example, to
        convert tensor shapes of galaxies from `Position` space to
        image space.

        Returns a numpy array of shape (2,2).

        In FITS celestial coordinates language, this is the CD matrix
        at pixel x,y:

        [ [ dRA/dx * cos(Dec), dRA/dy * cos(Dec) ],
          [ dDec/dx          , dDec/dy           ] ]

        in FITS these are called:

        [ [ CD11             , CD12              ],
          [ CD21             , CD22              ] ]

        The units of these things are degrees per pixel.


        Returns a new WCS object appropriate for the subimage starting
        at (dx,dy) with respect to the current WCS origin.


Class decorator that fills in missing ordering methods


        Performs *one step* of linearized least-squares + line search.

        Returns (delta-logprob, parameter update X, alpha stepsize)

        If variance=True,

        Returns (delta-logprob, parameter update X, alpha stepsize, variance)

        If just_variance=True,
        Returns variance.


    If set,

    radecrad = (ra,dec,rad)

    returns sources within "rad" degrees of the given RA,Dec (in degrees)
    center.

    WARNING, this method alters the "objs" argument, if given.
    Consider calling objs.copy() before calling.

    -"bandname" is the SDSS band used to cut on position, select
     star/gal/exp/dev, and set galaxy shapes.

    -"bands" are the bands to include in the returned Source objects;
     they will be initialized from the SDSS bands.

    -"extrabands" are also included in the returned Source objects;
     they will be initialized to the SDSS flux for either the first of
     "bands", if given, or "bandname".


    Returns a rebinned "tim" by integer scale "S" (ie S=2 reduces the
    image size by a factor of 2x2).

    The rescaling preserves intensity.

    We drop remainder pixels.


        negfluxval: when 'nonneg' is set, the flux value to give sources that went
        negative in an unconstrained fit.


mask: True to use pixel.  None for no masking.


    Produce an ellipsoid of walkers around an initial parameter value,
    according to a covariance matrix.

    :param p0: The initial parameter value.
    :param covmat:
        The covariance matrix.  Must be symmetric-positive definite or
        it will raise the exception numpy.linalg.LinAlgError
    :param size: The number of samples to produce.


    Comoving matter density, not a function of time [Msun/h / (Mpc/h)^3]
    args:
        Om_m: Cosmological matter density (at z=0)


    Fourier transform of a tophat function.
    args:
        x: Usually kR


    Derivative of the tophat Fourier transform function
    args:
        x: Usually kR


    Get the square-root of the variance, sigma(R), in the density field
    at comoving Lagrangian scale R


    Integrand for calculating sigma(R)
    Note that k can be a float or an arraay here
    args:
        k: Fourier wavenumber (or array of these) [h/Mpc]
        R: Comoving Lagrangian radius [Mpc/h]
        Pk: Function of k to evaluate the linear power spectrum


    Brute force integration, this is only slightly faster than using a loop
    args:
        R: Comoving Lagrangian radius [Mpc/h]
        Pk: Function of k to evaluate the linear power spectrum
        kmin: Minimum wavenumber [h/Mpc]
        kmax: Maximum wavenumber [h/Mpc]
        nk: Number of bins in wavenumber


    Quad integration
    args:
        R: Comoving Lagrangian radius [Mpc/h]
        Pk: Function of k to evaluate the linear power spectrum


Calculates d(ln sigma^2)/d(ln R) by integration


    Radius [Mpc/h] of a sphere containing mass M in a homogeneous universe
    args:
        M: Halo mass [Msun/h]
        Om_m: Cosmological matter density (at z=0)


Mass [Msun/h] contained within a sphere of radius 'R' [Mpc/h] in a homogeneous universe


    Initialise the emulator for a given set of cosmological parameters
    cpar: My verion of Dark Quest cosmology object


Minimum halo mass for the set of cosmological parameters [Msun/h]


Comoving matter density [(Msun/h)/(Mpc/h)^3]


Mass enclosed within comoving radius R [Msun/h]


    Linear halo bias: b(M, z)
    Taken from the pre-release version of Dark Quest given to me by Takahiro
    I am not sure why this functionality was omitted from the final version


    Calculate the logarithmic derivative of f(x) at the point x: dln(f)/dln(x)
    Here f(x) is a callable function and x is the value at which to evaluate the derivative


    Calculates the derivative of the function f(x) which is sampled as fs at values xs
    Approximates the function as quadratic using the samples and Legendre polynomials
    TODO: Can this be made to work for array x argument? Vectorize?
    Args:
        x: Point at which to calculate the derivative
        xs: Sample locations
        fs: Value of function at sample locations


Return a logarithmically spaced range of numbers


    Two-dimensional trapezium rule
    First integrates along x for each y, and then y


Find the index, value pair of the closest values in array 'arr' to value 'x'


Returns True iff the array contains monotonically increasing values


        Halo mass function g(nu) with nu=delta_c/sigma(M)
        Integral of g(nu) over all nu is unity


        Halo linear bias b(nu) with nu=delta_c/sigma(M)
        Integral of b(nu)*g(nu) over all nu is unity


        Calculates the halo mass function as a function of halo mass
        This is the comoving number-density of haloes per halo mass
        Sometimes referred to as n(M) or dn/dM in the literature
        Args:
            M: Halo masses [Msun/h]
            sigmaM: Standard deviation in overdensity on mass scale M


        Calculates M^2 n(M) / rhobar, the so-called halo multiplicity function
        Note that this is dimensionless
        Args:
            M: Halo masses [Msun/h]
            sigmaM(M): Standard deviation in overdensity on mass scale M


        Calculates the linear halo bias as a function of halo mass
        Args:
            M: Halo masses [Msun/h]
            sigmaM(M): Standard deviation in overdensity on mass scale M


        Radius [Mpc/h] of a sphere containing mass M in a homogeneous universe
        Args:
            M: Halo masses [Msun/h]


        Calculate the mean of some f(M) over halo mass <f>: int f(M)n(M)dM where n(M) = dn/dM in some notations.
        Note that the units of n(M) are [(Msun/h)^{-1} (Mpc/h)^{-3}] so the units of the result are [f (Mpc/h)^{-3}].
        This corresponds to the mean of f(M) over haloes, weighted by their mass.
        Common use cases:
            <M> = rho: The mean matter density in the universe
            <M/rho> = 1: over all halo mass (equivalent to int g(nu)dnu = 1)
            <M^2/rho> = M_NL: non-linear halo mass that maximall contributes to one-halo term (not M*)
            <(M/rho)^2> = P_1h(k->0): amplitude of the one-halo term in the matter-matter spectrum [(Mpc/h)^3]
            <b(M)M/rho> = 1: over all halo mass (equivalent to int g(nu)b(nu)dnu = 1)
            <N(M)> = rhog: with N the number of galaxies in each halo of mass M; gives mean number density of galaxies, rhog
            <b(M)N(M)>/<N(M)> = bg: with N the number of galaxies in each halo of mass M; gives mean bias of galaxies, bg
        Args:
            M: Halo masses [Msun/h]
            sigmaM(M): Standard deviation in overdensity on mass scale M
            func(M): Function values, evaluated at M, of which to calculate mean


        Computes power spectra given that halo model. Returns the two-halo, one-halo and sum terms.
        Args:
            k: Comoving wavenumbers [h/Mpc]
            Pk_lin(k): Linear power at each wavenumber [(Mpc/h)^3]
            M: Halo masses [Msun/h]
            sigmaM(M): Standard deviation of density field at scale corresponding to halo mass M
            profiles: Halo profiles from halo_profile class and corresponding field names
            beta(k, M1, M2): Optional array of beta_NL values at points M, M, k
            simple_twohalo: Should the scale-dependence of the two-halo term be ignored?
            subtract_shotnoise: Should shot noise be subtracted from discrete spectra?
            correct_discrete: Properly treat discrete tracers with <N(N-1)> rather than <N^2>?
            k_trunc: None or wavenumber [h/Mpc] at which to truncate the one-halo term at large scales
            verbose: verbosity


Two-halo term at a specific wavenumber


One-halo term at a specific wavenumber


Evaluate the integral that appears in the two-halo term


Evaluates the beta_NL double integral


        Computes the power spectrum of a single tracer crossed with haloes of a single specific mass
        Args:
            k: Comoving wavenumbers [h/Mpc]
            Pk_lin(k): Linear power at each wavenumber [(Mpc/h)^3]
            M: Halo masses [Msun/h]
            sigmaM(M): Standard deviation of density field at scale corresponding to halo mass M
            profiles: Dictionary of halo profiles (halo_profile class) with associated field names
            M_halo: Halo mass at which to evaluatae cross spectra [Msun/h]
            beta(k, M): Optional array of beta_NL values at points M, k
            simple_twohalo: Should the scale-dependence of the two-halo term be ignored?
            k_trunc: None or wavenumber [h/Mpc] at which to truncate the one-halo term at large scales
            verbose: verbosity


Two-halo term for halo-u at a specific wavenumber


Evaluates the beta_NL integral for halo-u


Calculate peak-height (nu) values from array of halo masses


        Halo virial radius based on the halo mass and overdensity condition
        Args:
            M: Halo masses [Msun/h]


    Wrapper for various beta_NL interpolation schemes to go from coarse grid of halo masses
    to a finer grid of halo masses.
    Args:
        k: Comoving wavenumbers [h/Mpc]
        M: Desired halo masses [Msun/h]
        M_small: Current halo masses at which beta_NL_small is evaluated already
        beta_NL_small: Small beta_NL(k, M_small, M_small) array, which needs to be expanded
        scheme: Interpolation scheme
        **kwargs: To be passed to interpolation schemes


    Interpolate beta_NL from a small grid to a large grid for halo-model calculations.
    TODO: Can I remove inefficient loops here?


Supported methods are 'linear', 'nearest', 'slinear', 'cubic', and 'quintic'


        Class initialisation for Fourier space halo profiles.
        Uk is assumed to be an array Uk(k, M) of the profile Fourier transform.
        If amplitude=None then Uk is assumed to be dimensionful, otherwise Uk is assumed dimensionless with Uk(k->0, M) = 1
        Matter profiles have amplitude=M (halo mass), galaxy profiles have amplitude=N (number of galaxies)
        Args:
            k: Comoving wavenumbers [h/Mpc]
            M: Halo masses [Msun/h] going from low to high
            Uk: Two-dimensional array of halo Fourier transform (order k, M)
            amplitude: Halo profile amplitudes at halo masses 'M' (e.g., M for mass; N for galaxies)
            normalisation: Field normalisation (e.g., rhom for mass, mean galaxy number density for galaxies)
            variance: Variance in the profile amplitude at each halo mass (e.g., N for Poisson galaxies)
            mass_tracer: Are contributions expected for M < M[0] (e.g., matter)?
            discrete_tracer: Does the profile correspond to that of a discrete tracer (e.g., galaxies)?


        Alternative class initialisation for configuration-space haloes.
        differential_profile is a halo (density or whatever) profile multiplied by 4pi r^2.
        If amp=None then Prho is assumed to be normalised correctly, otherwise Phro can
        have any normalisation and will be renormalaised by amp to have the correct 'mass'.
        TODO: Are loops necessary here?
        Args:
            k: Comoving wavenumbers [h/Mpc] going from low to high
            M: Halo masses [Msun/h] going from low to high
            differential_profile: Function to evaluate halo profile as a function of radius: P(r, M, rv, c)
            rv: Comoving halo virial radii [Mpc/h]
            c: Halo concentration
            amp: Halo profile amplitudes at halo masses 'M' (e.g., M for mass; N for galaxies)
            norm: Float for field normalisation (e.g., rhom for mass, ng for galaxies)
            var: Var(N(M)) (auto)variance in the profile amplitude at each halo mass (e.g., N for Poisson galaxies)
            mass_tracer: Are contributions expected for M < M[0] (e.g., matter)?
            discrete_tracer: Does the profile correspond to that of a discrete tracer (e.g., galaxies)?


    Compute the halo window function via integration given a differential 'density' profile P(r) = 4*pi*r^2*rho(r)
    TODO: This should almost certainly be done with a dedicated integration routine
    TODO: Can this be made to accept arrays of M, rv, c as arguments? This would avoid loops


    Fourier transforms of halo profiles
    Args:
        k: Comoving wavenumbers [h/Mpc]
        rv: Comoving halo virial radii [Mpc/h]
        *args: Additional arguments for halo profiles (e.g., concentration)
        profile: Profile name:
            'delta': delta function concentrated at halo centre
            'isothermal': 1/r^2
            'NFW': Navarro, Frenk & White (1997)


Normalised Fourier transform for an isothermal profile


Normalised Fourier transform for an NFW profile


Factor from normalisation that always appears in NFW equations


    Pre-configured matter NFW profile
    Args:
        k: Comoving wavenumbers [h/Mpc]
        M: Halo masses [Msun/h]
        rv: Comoving halo virial radii [Mpc/h]
        c: Halo concentrations
        Om_m: Cosmological matter density


    Halo concentration as a function of halo mass and redshift
    Args:
        M: Halo masses [Msun/h]
        z: redshift
        method: Name of concentration-mass relataion
            Duffy et al. (2008)
        halo_definition: Halo overdensity definition
            M200: 200 times the mean background density
            Mvir: 'virial' density according to spherical-collapse
            M200c: 200 times the critical density


    Duffy et al (2008; 0804.2486) c(M) relation for WMAP5, See Table 1
    Appropriate for the full (rather than relaxed) samples


    Returns the expected (mean) number of central and satellite galaxies as a tuple
    Halo occupation distribution (HOD)
    Args:
        M: Halo masses [Msun/h]
        method: Name of
            'simple': Very basic HOD, see code
            'Zehavi et al. (2004)'
            'Zheng et al. (2005)'
            'Zhai et al. (2017)'
        **kwargs: For specific HOD


Simple HOD model


    HOD model from Zehavi et al. (2004; https://arxiv.org/abs/astro-ph/0703457)
    Same as Zheng model in the limit that sigma=0 and M0=0
    Mean number of central galaxies is only ever 0 or 1 in this HOD


Zheng et al. (2005; https://arxiv.org/abs/astro-ph/0408564) HOD model


HOD model from Zhai et al. (2017; https://arxiv.org/abs/1607.05383)