    Applys Tikinhov regularisation to covariance matrix so its inverse can be sensibly computed.
    This is sometimes called ridge regression.
    Params: covariance - a NxN symmetric matrix whose diagonal values will be modified.
            cond_number -- the target condition number relative to the condition number of the diagonal matrix
            initial_scale -- initial scaling on the diagonal elements (default value 1e-4)
            trace -- provide some diagnostics when True (default is False)

    Returns: Regularised covariance matrix

    Algorithm: The scaling is doubled each iteration until the condition
    number is less than target value. Target value is cond_number*cond(diagonal covariance)


    calculate error for simulated observed values given targets and covariance.

    param: simulated: A numpy array of simulated observations. Each row
      corresponds to a different simulation and each column to a
      different observations.

    param: observations: A numpy 1D array of observations.

    opt-param: cov: Covariance matrix. Default value is the identity matrix.


    the function doGaussNewton does the  calculation for the n+1 initial runs in an iteration,
    and can be invoked from (usually) makeNewRuns, using data from studies, or from
    test python scripts for arbitrary inputs.

    :param param_value: numpy array of parameter values.
    :param param_range: 2D numpy  array of min/max of range for each parameter.
      param_range[:,0] is the min value, param_range[:,1] the maximum value.
      Q: Should this be passed through studyJSON as property of optimisation rather than framework?
    :param UM_value: array of simulated observables
    :param obs: array of target values -- should contain same number of elements as columns in UM_value

     param_value & UM_value are ordered such that rows correspond to different simulations and
     columns to parameters and simulated observations respectively.

    :opt-param cov: covariance array (header defines the observables in use)
                    Default is identity matrix
    :opt-param scalings: observables are related arrays get scaled before statistical analysis
                Default value is 1
    :opt-param olist: name of the observables. Default values are obs1..obsN where N is number of oBs.
    :opt-param constraint: Value of simulated constraint.
    :opt-param constraint_target: Value of observed constraint. BOTH constraint and constraint_target should be   present if studyJSON has sigma True
    :opt-param studyJSON The optimisation control specified as a dict. Contains the following values:
        alphas        -- alpha values used in linesearch. Default values are 0.3, 0.7 and 1.
        sigma         -- If true then constraint is used. Default is False
        mu            -- weighting on constraint. (wt is sqrt(1/2mu)). Default is 1.
        covar_cond    -- if specified used by regularize_cov (qv) to regularize covariance
        reg_crit_cond -- critical condition number for regularisation of Hessian. Default is 10e10.
        reg_pow_range -- range of powers used to generate list of powers in regularisation. Default is (-7,-2)

    :opt-param trace: turn on/off trace of what happening. Default is False. Set True to get some tracing.

    :returns: linesearch: array defining the next set of parameter values. Each row corresponds to parameter set.
    :returns: err: Error values for each simulation.
    :returns: err_constraint: Constrained error for each simulation
    :returns info: dictionary of data destined for json file.
       hessian: An array of the regularised hessian matrix.
       jacobian: An array of the Jacobian matrix
       InvCov: An array of the Inverse Covariance matrix (after possible regularisation)
       condnum: The Condition Number of the regularized hessian matrix.
       software: A string with info on the software
       scalings: Scalings applied to data.
       olist: names of variables
       params: A dictionaary of the  actual parameters (or defaults used) from studyJSON


    Randomly selects nrandom subset of perturbations from full set. Should be called from  doLineSearch
      if required. Will then modify next set of runs to be ran. Should also modify optClimBegin

    :param paramValues: parameter values for next iteration -- each case corresponds to a row with 0th row being
              the base value. The base value should always be included in the set.
    :param nrandom: number of random cases to select
    :param deterministic: optional with default True -- if True RNG initialised with first set of param values
              deterministic will be added to the values in order to allow a little user control over the RNG
    :param trace: optional with default False -- if true more diagnostic information provided
    :return: sub-set of nrandom+1 param_values to actually run and  indx into initial parameter array.


 Run Linesearch Algorithm and decide when to stop.
     :param param_value: numpy array of parameter values.
     !param param_range: numpy array of parameter ranges.
     :param UM_value: numpy array of simulated observables for the entire iteration. The last nalpha will be used.
     :param obs: 1D array of target observations
     :param step: 1D array of perturbation steps.

      param_value & UM_value are ordered such that rows correspond to different simulations and
      columns to parameters and simulated observations respectively.

     :opt-param cov: covariance array used to compute error with default the identity matrix
     :opt-param cov_iv: covariance array used to decide if states are different -- depends on internal var.
       Default is 1
     :opt-param scalings: observables and related arrays get scaled before statistical analysis
                 Default value is 1
     :opt-param olist: name of the observables. Default values are obs1..obsN where N is number of oBs.
     :opt-param constraint: Value of simulated constraint.
     :opt-param constraint_target -- observed value for constraint.

     BOTH constraint and constraint_target should be present if studyJSON has simga set to True

     :opt-param studyJSON The optimisation control specified as a dict. Uses the following values:
         alphas        -- alpha values used by doGaussNewton -- only number used. Default is 3.
         sigma         -- If True then constraint is used. Default is False
         mu            -- weighting on constraint. (wt is sqrt(1/2mu)). Default is 1.
         covar_cond    -- if specified used by regularize_cov (qv) to regularize covariance.
                 Termination Conditions
         prob_int      -- critical probability that previous and current best states are different. Default is 0.5
         prob_obs      -- critical probability that current state is different from observations. Default is 0.5

         nrandom        -- if not none then chose (at random) nrandom permutations rather than full set. Default is None
         deterministicPertub -- if True then make random choice is deterministic function of parameters.
         minImprovement -- the minimum value by which the error should be reduced to continue iteration.

     :opt-param trace: turn on/off trace of what happening. Default is False. Set True to get some tracing.
     :returns StatusInfo -- status of optimisation. 'continue' means keep going
     :returns: err: Error values for each simulation.
     :returns: err_constraint: Constrained error for each simulation
     :returns: GaussNewton: nparam+1, nparam  array defining the next set of parameter values.
               Or None if LineSearch has terminated.
     :returns index  -- which one of the linesearch values is the best value.
     :returns: bestParam -- the best parameters in the current iteration.
     :returns info: dictionary of data destined for json file.
        cov: An array of the  Covariance matrix (after possible regularisation)
        cov_iv: An array of the covariance matrix of int var (after possible regularisation)
        software: A string with info on the software
        scalings: Scalings applied to data.
        olist: names of variables
        params: A dictionary of the  actual parameters (or defaults used) from studyJSON
        BestIndex: Best index of the linesearch values
        StatusInfo: Additional information on the status


    Support function to run function in a way that all cases are ran before error happens..
    :param function: function to run
    :param params: parameter set for function
    :param npt -- number of obs values but not actually used.
    :param constraint_target: if not none then have a constraint
    :return: array of obs values and constraint.


    Apply guassNewton/Linesearch algorithm to specified function.
    :param function: function to be optimised. Should take a numpy array of N values and return a M length array of observations
    :param startParam: a length N numpy array of the starting parameters for the optimisation
    :param paramRange: a Nx2 numpy array of the minimum parameter values [*,0] and maximum parameter values [*,1]
    :param paramStep: The perturbation to be made to each parameter -- note that algorithm development could work this out automatically.
    :param target: The target for the optimisation. A len M numpy array.
    :param optimise: A dict of information used by optimisation.
    :param cov: (default None)  Covariance matrix for scaling cost function. Default values sets by doGaussNewton and doLineSearch
    :param cov_iv: (default None) Covariance  matrix used in doLinesearch to determine if values changed enough.
    :param scalings : (default is 1) Scalings to apply to simulated observations and targets. A len M numpy array
    :param constraint_target : (Optional -- default is None) If provided the target value for the constraint
    :param trace: provide more trace information
    :return: Returns minimal error param values,  status of termination & information on GN/LS cpts of algorithm


    Generate perturbations towards the centre of the valid range for each
    parameter

    Inputs

         baseVals: array of parameters for the first of the GN runs (unperturbed)

         parLimits: max,min as read from JSON into 2-D array,
                    and passed through to here unaltered

         steps: array of stepsizes for each parameter.

         nrandom: perturb nrandom parameters rather than the full set.
         deterministic: optional argument with default value True -- make random choice of parameters deterministic.
            deterministic will be added to the RNG seed. So to modify what you get change the value of deterministic.
            It is a bit of a hack though...

         trace: Optional argument with default value False.
                Boolean -- True to get output, False no output

    Note inputs are all arrays, with data in order of the parameter list.

    Returns
          y: array defining the pertrubed parameter  parameter values
             with dimension nparameters


    compute Jacobian from given function. Really a fairly thin layer to allow optimise dict to be unpicked.
    :param function: function to be ran -- should call appropriate model simulations
    :param startParam:  start parameter as a pandas Series
    :param deltaParam:  perturbations as a pandas Series
    :param paramRange: paramRange as a pandas dataFrame

    Other unnamed arguments are passed to jacobian (and then to function)

    :keyword args

    :param obsNames (default None). Names of observations -- used to label the xarray. If not passed then will be 0,1,2...
    :param nEnsemble (default 1). Jacobian will be computed multiple times with parameter ensembleMember=X being passed to function.
    :param verbose (default False). If set then more verbose information
    :param returnVar (default False). If true then return the variance.

    other keyword arguments are passed to function.

    :return: jacobian (wrapped as an xarray)


    Compute Jacobian for specified function.
    :param function: function to be used. Should take a numpy array of NxM values and return a N*Y Jacobian array
    :param startParam: a length M numpy array of the reference parameters for the Jacobian calculation
    :param deltaParam: a length N array of parameter perturbations.

    Optional parameters

    :param verbose: provide more trace. Default False
    :param extraArgs: List of extra arguments -- passed through to function call.

    :return: Returns Jacobian as a NxY array


        Find and return the closet model in the (current) model trace to specified model.
        Close is defined as the smallest min difference for the normalised params that are fp or int.
         A more sophisticated implementation would normalise difference by the param ranges.
        :param model: model we are looking for closest match.
        :return:  Closest model match.


        Setup state for restart. Copies trace to prev_trace and sets trace to empty list.
        :param test_nondetermin -- verify that trace and prev_trace are consistent --
            all elements in prev_trace should be in trace.
            test_nondetermin should be one of 'error or 'warning'. If 'warning' warnings will be given.
            if 'error' then ValueError will be raised.
        If None then will be set to 'error'
        :return: None


        Standard Function used for running model . Returns values from cache if already got it.
          If not got cached model then raises runModelError exception or returns NaN
           after generating new model.

        It is a method rather than a class function as it makes use of some information in the object.
        So to actually get used as a function it needs to be converted to a function. This is done by genOptFunction which
          uses partial.function to do it. But this function does all the work.

        :param params -- a numpy array with the parameter values.
                These parameters should  be ordered as in self.paramNames()
        :param df  -- If True return all obs (read in after processing) as a dataframe.
        :param raiseError  -- If True raise optclim_exceptions.runModelError  if any requested models do not exist.
                This should cause generation & submission of models that need running.
                Else return array full of nans.
        :param ensemble_average -- If True average the ensemble members.

        The four  parameters below they are applied in the order: scale, residual, transform, sumSquare
        :param scale   -- if True scale obs  by self.scales()

        :param residual  -- if True remove target (self.target()) from obs

        :param transform  -- if provided transform the model obs by matrix multiplying them by this matrix.
                  It should be  N*nobs where nobs are the number of sim ons and  0 <= N <= nobs.
                  One application of this is to transform the data into a basis of eigenvectors of an
                    error covariance matrix. The transform matrix should be provided as a pandas datarray.
                  Column names being the obs names. Index being sensible labels for rows which will be new obs names.
                  Do be careful to make sure transform has same scaling as here...

        :param sumSquare (defaultFalse) -- if True return the sum of squares of the observations after any processing.

        Using stdFunction -- this  is a method as it needs to know various bits of information contained in ModelSubmit..
        To actually use it with optimisation function you need to call runSubmit.genOptFunction(**kwargs).
        That will give you a function suitable for many optimisation functions.  If you have more complex needs you
          likely need to add the runSubmit object to the list of arguments and then do runSubmit.stdFunction(.... )
          This might require you to create  a partial function. Your life will probably be easier if you set df=True
          and work with dataframes in your function.


        :return:A function suitable for use in an optimisation  algorithm (or anything else that uses the framework).
        by generating a partial function which converts stdFunction from a method to a function in the argument list
        and any adds extra args (as named arguments) that are present. You can use this to give you something suitable
        for most optimisation methods or use your own function to wrap stdFunction.


        :arg self
        Run optimised case using reference model which may be potentially different from configuration used to optimize.
        Cares about number of ensemble members (default is 1 if not set) and the optimum parameters which should be set.


        Generate perturb param values  towards the centre of the valid range for each
        parameter. Used in runJacobian

        Inputs

            arg: baseVals: parameters at which Jacobian computed at.
            arg: parLimits: dataset max,min limits. parLimits.loc[:,'maxParm'] are the max values;
              parLimits.loc[:,'minParam'] are the min values;
            arg: steps: stepsizes for each parameter.

        Inputs, except parLimits, are all pandas series
        Returns pandas series array defining the perturbed parameter values


        Run Jacobian cases.
        Rather crude (first order accurate) estimate. Evaluate functions at
          optimum values +/- delta where delta is specified in the StudyConfig used to generate Submit
          +/- is to keep perturbations towards the centre pf the domain.
        A more accurate version would run 2nd order differences around the base point.
        But (as yet) no need for this so not implemented. Probably best done with a centred option.
        The Jacobian computed is the transformed Jacobian. (Apply Transpose matrix).

        :arg self -- a Submit object.
        :param scale -- If True apply scalings.
        :returns a configuration. The following methods should work on it:
                return: finalConfig -- a studyConfig. The following methods should give you useful data:
                finalConfig.transJacobian() -- the  transformed Jacobian matrix at the optimum pt
                finalConfig.hessian() -- the  hessian computed from J^T J at the optimum pt.

            runs runConfig to provide generic info. (See documentation for that)


        run DFOLS algorithm. It runs until new models need to be ran or DFOLS complets.
                If new models are needed then runModelError will be raised.
                DFOLS raises np.linalg.linalg.LinAlgError if it tries to do calculations with nan (which is what it gets
                when model has not been ran). This then triggers a runModelError. The callee of this method
                should trap that error and then run the necessary models using Submit.submit.

        :param scale (default True). If True scale data for transform matrix and in calculations of obs
        See StudyConfig.scalings()
        It should not make any difference if scale is True or False. But if regularisation is done
        then it is better to have all (diagonal) elements of the covariance matrix have roughly the same mag
        which is really what scaling does for you. (for example converting kg/sec/m^2 to mm/day). Current implementation
        of tranMatrix truncates removing all eigenvectors and eigenvalues when egivenvalues < 1E-6 * max(evalues)

        return: finalConfig -- a studyConfig. The following methods should give you useful data:

                # Generic stuff (that is probably more useful)
                finalConfig.transJacobian() -- the final transformed Jacobian matrix at the optimum pt
                finalConfig.hessian() -- the final hessian computed from J^T J at the optimum pt.
                finalConfig.get_dataFrameInfo('diagnostic_info') -- Diagnostic info from DFOLS. See DFOLS documentation.
                finalConfig.optimumParams() -- optimum parameters.

            also can get generic info and cost info:
            runs runCost & runConfig to provide  info. (See documentation of those methods for what they provide)


        param: verbose if True produce more verbose output.
        param: scale if True apply scaling (default is True)
        Run Gauss Newton algorithm as used in Tett et al, 2017.
        return: finalConfig -- a studyConfig. The following methods should give you useful data:
                finalConfig.GNparams() -- the best parameters (min error)
                finalConfig.GNcost() -- the cost for each function evaluation
                finalConfig.GNalpha() -- the values of alpha used for next optimisation in the linesearch
                finalConfig.GNhessian() -- the diagnosed Hessians (in transformed space) at each iteration
                # Generic stuff (that is probably more useful)
                finalConfig.transJacobian() -- the final transformed Jacobian matrix at the optimum pt
                finalConfig.hessian() -- the final hessian computed from J^T J at the optimum pt.
                finalConfig.optimumParams() -- optimum parameters.

            also can get generic info and cost info:
            as runs runCost & runConfig to provide  info. (See documentation of those methods for what they provide)

            As the Gauss-Newton  component of this algorithm is a deterministic  perturbation to the minima over a small
               number of line-search values then provisional running  is not reliable. To make it work provisional running
               needs modification to run several times (more than twice) and only go if have "hits" = number of times ran.


        Run PYSOT algorithm

        Not been ran or tested. Likely needs various things set up to make it work..
        :params scale -- if True scale data internally by scalings.

        :returns finalConfig (a studyConfig)
            from which you can get generic info and cost info:
            See documentation of runCost & runConfig methods  to see what they provide.


    Read a configuration and return object of the appropriate version.
    :param filename: name of file (or filepath) to read.
    :return: Configuration of appropriate type


        if input object is a ndarray it will be converted into a OrderedDict
        holding dtype, shape and the data


    Decodes a previously encoded numpy ndarray
    with proper shape and dtype
    :param dct: (dict) json encoded ndarray
    :return: (ndarray) if input was an encoded ndarray


        Print out top-level keys
        :return: None


        saves dict to specified filename.
        :param filename to save file to. Optional and if not provided will use
           private variable filename in object. If provided filename that self uses
           subsequently will be this filename
        :param verbose (optional, default False). If true be verbose

        :return: status of save.


        :param key: key for which value we are incrementing
        :param increment: how much we increment key by. Be careful what you increment..
        :return: incremented value


:return: path to filename


        Convert a dictfile to an OptClimVn3
        :param config_dir: dictfile rep of config.
        :return: OptClimConfigVn3 (or with development later version)


:return: the current version of the configuration file.


        :param name -- if not None then set name in self to name
        :return: a short name for the configuration file


        :param paramNames (default None). If not none set ParamList in config to this. paramNames should be a list.
        :return: a list of parameter names from the configuration files


        :param (optional default is None). If not None set obsNames to values
        :param (optional default = False) add_constraint -- if True add the constraint name to the list of obs
        :return: a list  of observation names from the configuration files


        :param paramNames -- a list of the parameters to extract ranges for.
        If not supplied the paramNames method will be used.
        :return: a pandas array with rows names minParam, maxParam, rangeParam


        Extract standard parameter values for study
        :param paramNames: Optional names of parameters to use.
        :param values (default None). If values is not None set standardParam to values passed in and return them
        :param all (default False). Return the whole default parameters dict.
        :param scale (default False). If True scale the values by the range
        :return: pandas series (unless all is set)


        Extract standard obs values for study
        :param obsNames: Optional names of observations to use.
        :param scale (default False). If true scale values
        :return: pandas series


        Extract constraint value from standard model values.
        :param constraintName (default None): value to extract
        :return: constrain as a pandas Series


        get the begin parameter values for the study. These are specified in the JSON file in begin block
        Any values not specified use the standard values
        :param paramNames: Optional names of parameters to use.
        :param scale (default False). If True scale parameters by their range so 0 is minimum and 1 is maximum
        :return: pandas series of begin parameter values.


        :param studyDir the study directory
        :return: a studyVars object for the current study


        Get the name of the file used to store and update information for the whole study.

        :return: filename relative to studyDir


        Get the pathname relative to the study directory of the cache file.
        This file holds information (directories in current design) on model simulation directories.
        :return:  filename relative to studyDir


        Get (or set) the target values for specific obs names. If some obsNames not present in target then a ValueError is raised
        :param targets -- tgt values as pandas array or None.
         Note that this does not change the obsNames which should be set using obsNames()
        :param obsNames: list of observations to use. If not None the self.obsNames() will be used.
        :param scale: if True  scale target values by scaling
        :return: target values as a pandas series


        Check that observation related stuff is OK.
          Check targets, scalings and covariances. Trapping errors and then reporting at end
        :return: True if OK/False if not.


        Work out if doing constrained optimisation or not.
        :return: True if constraint active.


        Extract the name of the constraint (if any) variable from the configuration file

        :return: the name of the constraint variable


        extract the value for the constraint variable target returning it as a pandas series.
        :param constraintName (optional) if not specified then constraintName() will be used
        :param scale (optional; default False) if True then constraint value will be appropriately scaled.
        :return: constraint target as a pandas series


        Get the scales for specified obsNames
        :param scalings If not None then set scales to these values. Should be dict like.
            Any value of 1 will not be stored.
        :param obsNames: list of observations to use. If None then self.obsNames will be used
        :return: scales as a pandas series


        :param value. If not None set maxFails to this.

        :return: the maximum number of fails allowed. If nothing set then return 0.


        If CovObsErr and CovIntVar are both specified then CovTotal will be computed from
        CovObsErr+2*CovIntVar overwriting the value of CovTotal that may have been specified.
        Unspecified values will be set equal to None.
        If CovIntVar is not present it will be set to diag(1e-12)
        If CovTotal is not present it will be set to the identity matrix

        :param obsNames: Optional List of observations wanted and in order expected.
        :param trace: optional with default False. If True then additional output will be generated.
        :param dirRewrite: optional with default None. If set then rewrite directory names used in readCovariances.
        :param scale: if set true (default is false) then covariances are scaled by scaling factors derived from self.scales()
        :param constraint: is set to True  (default is None) then add constraint weighting into Covariances. If set to None then
           if configuration asks for constraint (study.sigma set True) then will be set True. If set False then no constraint will be set.
            Total and ObsErr covariances for constraint will be set to 1/(2*mu) while IntVar covariance will be set to 1/(100*2*mu)
            This is applied when data is returned. If you don't want constraint set then see StudyConfig.constraint method.

        :param CovTotal -- if not None set CovTotal to  value overwriting any existing values.
           Should be a pandas datarrray
        :param CovIntVar -- if not None set CovIntVar to value overwriting any existing values.
        :param CovObsErr -- if not None set CovObsErr to value overwriting any existing values.
         In setting values you can make CovTotal inconsistent with CovIntVar and CovObsErr.
         This method does not check this. You should also pass in unscaled values as scaling is applied on data
        No diagonalisation  is done to these value. Constraint, if requested, added on.
         Scaling is then applied to these values (or original values)
        :param read (default True)-- if True use readCovariances to read in the data in essence resetting covariances
        :return: a dictionary containing CovTotal,CovIntVar, CovObsErr-  the covariance matrices and ancillary data.
         None if not present.

        TODO: Modify internal var covariance matrix as depends on ensemble size.


        Return matrix that projects data onto eigenvectors of total covariance matrix
        :param scale: (Default False) Scale covariance.
        :param verbose: (default False) Be verbose.
        :param dataFrame: wrap result up as a dataframe
        :param minEvalue: evalues less than minEvalue * max(eigenvalues) are removed. Meaning a non-square transMatrix
        :return: Transformation matrix that makes Total covariance matrix I.


        Compute perturbation  for all parameters supplied. If value specified use that. If not use 10% of the range.
        Quasi-scientific in that 10% of range is science choice but needs knowledge of the structure of the JSON file
             so in this module.
        :param steps (optional). If not None then set the steps.
        :param paramNames -- optional the parameter names for step sizes. If not defined uses self.paramNames() to work
                them out
        :return: the step sizes for the parameters as a pandas Series.


         rewrite dir using keys in dir_rewrite
        :param dir: list or string of directories to be rewritten
        :param dir_rewrite:  dictionary of files to be rewritten
        :return: rewritten dict


        :param covFile: Filename for the covariance matrix. Env variables and ~ expanded
        :param obsNames: List of Observations wanted from covariance file. If None then self.obsNames() will be used though constraint name will be omitted.
        :param trace: (optional) if set True then some handy trace info will be printed out.
        :param dirRewrite (optional) if set to something then the first key in dirRewrite that matches in covFile
              will be replaced with the element.
        :return: cov -- a covariance matrix sub-sampled to the observations

        Returns a covariance matrix from file optionally sub-sampling to named observations.
        Note if obsName is not specified ordering will be as in the file.


        :param postProcessScript (defaultNone). If not None set the postProcessScript to this value
        :return: the full path for the postprocessing script


            :param (optional) postProcessOutput -- if not None (which is default) value will be used to set
        :return: relative  path for output from post processing script. Path is taken relative to model directory


        :param referenceConfig  -- set referenceConfig if not None (Should be a string).
        :return: full path to the reference configuration of model being used


        Extract and package all optimisation information into one directory
        Note this is not a copy
        :param optimise_values -- if not None
        :return: a dict


:return: a dict of all the fixed parameters. All names ending _comment or called comment will be excluded


:return: the runCode (or None)


:return: the run time (or None)


        lookup and return model creation function from value of modelName in fnTable.
        :return:the function that creates a model instance


Return name of Machine in config file.


        Use value of fakeFunction (default is 'default') to look up and return function in fnTable

        :param: fnTable -- dict  of functions
        :return: fake function


        Common method to set/get stuff
        :return:


        Set/Return the Jacobian array as an xarray DataArray
        :param jacobian: (default None). If not None should be a numpy 3D array which will be stored in the config
        :return: xarray version of the Jacobian array


        Return the Hessian array as an xarray DataArray
        :param hessian: (default None). If not None should be a numpy 3D array which will be stored in the config
        :return: xarray version of the Hessian array


        Return (and optionally set) the best parameter values as an xarray from the GN optimisation.
        :param params: (default None). If not None should be a 2D numpy array which will be stored in the config.
        :return: parameter array as xarray.DataArray


        Return (and optionally set) the cost values as a pandas Series from the GN optimisation.
        :param cost:  (default None)). If not None should a 1D numpy array which will be stored in the config.
        :return: cost as a pandas.Series


        Return and optionally set the the status of the Gauss-Newton aglorithm
        :param status: if not None should be a string which will stored.


        Return (and optionally set) the alpha values as a pandas Series from the N optimisation.
        :param alpha:  (default None)). If not None should a 1D numpy array which will be stored in the config.
        :return: alpha as a pandas.Series


r"""
        Compute the covariance for parameter error.
        Theory is that (to first order) $$\vect{\delta O}= \matrix{J}\vect{\delta p}$$
          where $$\delta O$$ are perturbations in observations, $$\delta p$$ are perturbations to parameters and J is the Jacobian.
          Then multiple by J^+ (the pseudo-inverse) to give $$\vect{\delta p} = \matrix{J}^+  \vect{\delta O}$$.
        Then the covariance is $$\matrix{J}^+ C {\matrix{J}^+}^T$$

         or alternatively with covariance...
         $$P=(J^TC^{-1}J)^{-1}J^TC^{-1}$$
        :param normalise (default = True) -- compute the normalised covariance error (fraction of range)
        :return: covariance of parameters


        Set/get the optimum parameters.  (VN1 variant)
        :param values: default None -- if set then parameter values in configuration gets updated with values
        :param paramNames -- return specified parameters.
        :param scale (default False). If True scale parameters over range (0 is min, 1 is max)
        :return: values as pandas series.


        Set/get the simulated observations from the best value in each iteration of the Gauss Newton algorithm.
        :param obsNames: Names of observations wanted
        :param set: a pandas array or xarray (or anything that hass a to_dict method) that will set values of the observations.
          Should only be the best values on each iteration. Perhaps this routne should have all values...
        :return: Observations as a pandas array.


        :param obsNames -- the observations to extract
        :return: Optimum observations as a pandas series.


        : named arguments used to set algorithm information in config
           Information is stored in config information based on name of algorithm
        :return:  specific algorithm information.


        :param diagnostic: A pandas dataframe. If set add this to the configuration
        :return: the values as a  dataframe


        Set jacobian if set. Return jacobian as pandas dataframe
        :param jacobian: a pandas dataframe containing the jacobian information
        :return: jacobian -- converted to a float (the json conversion loses type info)


        Set transformed jacobian if set. Return transJacobian as pandas dataframe
        :param transJacobian: a pandas dataframe containing the transformed jacobian information.
           If not None will set values.
        :return: transformed jacobian (J.Trans^T) where Trans is the transform matrix (see transMatrix)
               converted to a float (the json conversion loses type info)


        :param hessian. If not None set hessian to this value. Should be a pandas dataframe.
        return -- returns the hessian matrix converted to a float (the json conversion loses type info).


r"""
        Compute the covariance for parameter error. TODO -- consider removing.
          to store in the final json file. Then regardless of algorithm we can get it out again...
        Theory is that (to first order) \vect{\delta O}= \matrix{J}\vect{\delta p}
          where \delta O are perturbations in observations, \delta p are perturbations to parameters and J is the Jacobian.
          Then multiple by J^+ (the pseudo-inverse) to give \vect{\delta p} = \matrix{J}^+  \vect{\delta O}.
        Then the covariance is \matrix{J}^+ C {\matrix{J}^+}^T

         or alternatively with covariance...
         P=(J^TC^{-1}J)^{-1}J^TC^{-1}
        :param normalise (default = True) -- compute the normalised covariance error (fraction of range)
        :param useCov
        :return: covariance of parameters


        Set best_obs if set. Return best_obs as pandas series
        :param best_obs: a pandas series containing the best observations info
        :return: best_obs


        :param diagnosticInfo: (optional -- defult None). If set add ths to the confguration
        :return: the values as a  dataframe. Temp code for backward compatability


        Extract (and optionally set) the DFOLS specific information
        :param dfolsConfig -- a dict of dfoLS config data which replaces existing values
        :return: the configuration which (like most python is a ptr tothe data. If you change


        Extract the user parameters for DFOLS
        :param userParams (default None) -- default user params which are overwritten by config values.
        :param updateParams (default None) -- modify userparams in configuration.
        :return: user parameters


        :param diagnosticInfo: (optional -- defult None). If set add ths to the confguration
        :return: the vlues as a  dataframe


        :param simObs: optional -- default None and if passed should be a pandas dataframe
        :param best: optional-- defaultFalse. If true return the obs for the best iteration
        :return:  dataframe of simulated observations in order in which they were ran (which may not be
           the same as the algorithm sees them)

        TODO: Modify to use set/get_dataframe


        :param parameters: optional -- default None and if passed should a pandas dataframe
        : param (optional) normalise -- default False. If True the parameters
           returned are normalised to 0 (min allowed) to 1 (max allowed)
        :return:  dataframe of simulated observations
        :TODO rewrite to use set/get_dataframe.


        :param cost: (optional) the cost for each model evaluation as a pandas series.
        :param best (optional) -- if True return only the best value
        :return: cost as a pandas series.


        :param directories: (optional -- default = None). The directory (currently full path) where each simulation was run as a pandas series
        :param best (optional) -- if True return only the best value
        :return: directories as a pandas series


        Get/set the ensemble size
        :param value: If not None (default is None) then set the value for number of ensemble members in each simln
        :return: ensemble size


        Get/set the maximum numbers of runs
        :param value: If not None (default is None) then set the value f
        :return: maximum number of runs to be done.


        Store or return jacobian.
        :param jacobian: jacobian as dataset with mean (Jacobian) and variances (Jacobian_var)
        :return: jacobian as xaarry dataset.
        TODO : Consider removing dependance on xarray -- which means storing as dataframe.
        TODO: Remove -- probably not needed.


        Return (and optionally set) the baseRunID.
        If not defined return 'aa'
        :return: baseRunId


        Return (and optionally set) the maxDigits for model run names

        If not defined return None -- up to the user of this to deal with it.

        :param value -- value to be set -- default is not to set it


        :param filename (optional default None): Name of filename to save to.
        :return: a copy of the configuration but set _fileName  to avoid accidental overwriting of file.


        :param paramNames -- a set of paramNames to overwrite existing values. Should be a list
        :return: a list of parameter names from the configuration files


        Extract standard parameter values for study
        :param paramNames: Optional names of parameters to use.
        :param values (default None). If values is not None set standardParams to values passed in and return them
        :param scale (default False). If True scale values by parameter range so that 0 is min, 1 is max
        :param all (default False). Return the whole parameters dict. Used when want to copy
        :return: pandas series (unless all is set)


        get the begin parameter values for the study. These are specified in the JSON file in begin block
        Any values not specified use the standard values
        :param begin -- if not None then set begin values to this.
           No scaling is done  and initScale will be set False.
        :param paramNames: Optional names of parameters to use.
        :param scale (default False). If True scale parameters by their range so 0 is minimum and 1 is maximum
        :return: pandas series of begin parameter values.


        Set/get the optimum parameters. (VN2 version)
        :param normalise (default False). If True then normalise parameters.
        :param values: default None -- if set then parameter values in configuration gets updated with values
        :param paramNames -- name of parameters
        :return: values as pandas series.


        Compute perturbation  for all parameters supplied. If value specified use that. If not use 10% of the range.
        Quasi-scientific in that 10% of range is science choice but needs knowledge of the structure of the JSON file
             so in this module.

        :param paramNames -- optional the parameter names for step sizes. If not defined uses self.paramNames() to work
                them out
        :param normalise -- if True normalise by the range regardless of the value of scale in the configuration
        :param steps (optional). If not None then update the steps with values here. Should be a pandas series or dict.
              These values should be consistent with scale_step which determines the ... Which you can pass in.
        :return: the step sizes for the parameters as a pandas Series.


        plot cost, normalised parameter & obs values for runs.
        :param figName: name of figure to make -- default is monitor
        :param monitorFile: name of file to save figure to if not None. Default is None
        :return: figure, (costAxis, paramAxis, obsAxis)

        Note needs matplotlib


        Extract documentation from configuration. Done through finding any key that ends in "_comment"
        :return: configuration with key -- the name and doc the doc string


            Extract comments from list or list like things
            :param inputList: input list
            :param tgt_end:  what is a comment (anythoing ending in this)
            :return: extract list.


        Print out documentation within a study configuration. Uses extractDoc and pretty print.
        :param stream -- stream to print to -- see print.pprint for details
        :param width: -- width of screen for printing. Default is 120.
        :return: Nada


        Expand any env vars, convert to path and then expand any user constructs. (Copy from model_base which does things we don't want to do!)
        :param filestr: path like string or None
        :return:expanded path or, if filestr is None, None.


        Convert covariance (as a pandas dataframe) to a dict containing scale factor
          and converted scaled dataframe
        :param cov:covariance
        :return:jsonised version of df & scale factor.
           Values will be scaled by 1/abs(min) excluding zero values.


        Convert a dict representation back to a dataframe.
        :param dct: dict to be be convered. SHould contain scale and dataframe keys
        :return: Decoded dataframe


        saves dict to specified filename.
        :param filename to save file to. Optional and if not provided will use
           private variable filename in object. If provided filename that self uses
           subsequently will be this filename
        :param verbose (optional, default False).
            Does not do anything -- just for compatibility with older versions of code.

        :return: Nade


        Normalize a parameters by range such that min is 0 and max is 1
        :param pandas_obj: dataframe or series of parameters to normalise
        :return: normalised values


        Set best_obs if set. Return best_obs as pandas series with name from self
        :param best_obs: a pandas series containing the best observations info
        :return: best_obs


        Store or get extra stuff in alg_info
        :param getargs -- arguments to get.
        :param **setargs -- arguments to set.
        example: config.set_dataFrameInfo(result=result,flaming=flaming,success=pd.Dataframe(data))


        Set/get the optimum parameters. (VN3 version)
        :param normalise (default False). If True then normalise parameters.
        :optimum -- optimal parameter values as panda Series
        :param paramNames -- name of parameters
        :return: values as pandas series.


        :param referenceConfig  -- set referenceConfig if not None .
        :return: full path to the reference configuration of model being used


        Store/return solution to DFOLS run.
        :param solution: solution (from DFOLS) which if not None will be converted to
           something that can be converted to json
        :return: solution


Convert StudyConfig to a dict suitable for writing out


:return run_info dict


        value: If not None runCode will be set to this
        :return: the runCode (or None)


:return: the run time (or None)


Return name of Machine in config file. Fails if not defined


:return name of the model to be created. Should have been registered. And must exist in run_info.


        Returns name of module to be loaded.
         Uses run_info['module_name']. If not present (or None) then model_name will be used
        :param value: If not None then 'module_name' will be set to this value.
        :param model_name: If not None then rather than using self.model_name() then model_name will be used
        :return: Module to be loaded.


        Get/set the maximum numbers of runs
        :param value: If not None (default is None) then set the value
        :return: maximum number of runs to be done.


        The maximum total number of simulations that should be submitted. Used by runSubmit
          Taken from run_info/max_model_sims
        :param value: If not None set max_model_sims to this value. Should be integer > 0
        :return: The value of max_model_sims or None if not found.


        Recursively remove all keys ending in _comment from a dct.
        :param: dct -- dict to have all _comment keys removed.
        Defined by self.comment_end
        :return dct with all keys ending with _comment removed.


        Extract logging configuration from studyConfig.
        You can pass this straight into logging.config.dictConfig() to set up logging.
        Users are responsible that this dict is correct etc.
        All comments will be removed.

        This is designed for codes that make use of the library.

        :param: cfg -- if not None set the value of logging to this value.


        get the begin parameter values for the study. These are specified in the JSON file in initial block
        Any values not specified use the standard values
        :param begin -- if not None then set begin values to this.
           No scaling is done  and initScale will be set False.
        :param paramNames: Optional names of parameters to use.
        :param scale (default False). If True scale parameters by their range so 0 is minimum and 1 is maximum
        :return: pandas series of begin parameter values.


:return: a dict of all the fixed parameters. All names ending _comment or called comment will be excluded


        If CovObsErr and CovIntVar are both specified then CovTotal will be computed from
        CovObsErr+2*CovIntVar overwriting the value of CovTotal that may have been specified.
        Unspecified values will be set equal to None.
        If CovIntVar is not present it will be set to diag(1e-12)
        If CovTotal is not present it will be set to the identity matrix

        :param obsNames: Optional List of observations wanted and in order expected.
        :param trace: optional with default False. If True then additional output will be generated.
        :param dirRewrite: optional with default None. If set then rewrite directory names used in readCovariances.
        :param scale: if set true  then covariances are scaled by scaling factors derived from self.scales()
        :param constraint: is set to True  (default is None) then add constraint weighting into Covariances. If set to None then
           if configuration asks for constraint (study.sigma set True) then will be set True. If set False then no constraint will be set.
            Total and ObsErr covariances for constraint will be set to 1/(2*mu) while IntVar covariance will be set to 1/(100*2*mu)
            This is applied when data is returned. If you don't want constraint set then see StudyConfig.constraint method.

        :param CovTotal -- if not None set CovTotal to  value overwriting any existing values.
           Should be a pandas datarrray
        :param CovIntVar -- if not None set CovIntVar to value overwriting any existing values.
        :param CovObsErr -- if not None set CovObsErr to value overwriting any existing values.
         In setting values you can make CovTotal inconsistent with CovIntVar and CovObsErr.
         This method does not check this. You should also pass in unscaled values as scaling is applied on data
        No diagonalisation  is done to these value. Constraint, if requested, added on.
         Scaling is then applied to these values (or original values)
        :param read -- if True use readCovariances to read in the data in essence resetting covariances
        :return: a dictionary containing CovTotal,CovIntVar, CovObsErr-  the covariance matrices and ancillary data.
         None if not present. Also may modify the configuration.

        TODO: Modify internal var covariance matrix as depends on ensemble size.


        Add config to self and derive what needed from config to include in self.

        :param config: A study config used to set. This is deep copied into self
        :return: Nothing


            Clean repr of a series
            :param series: series to be represented
            :return: clean string!


        Generate key from model
        :param model: model for which key gets generated. Uses parameters to generate the key.
        :param fpFmt: floating point format for floats
        :return: key


        Generate key from keys and values in parameters. There should include all fixed and variable parameters.
        This should be unique (to some rounding on float parameters)
        :param parameters -- a dictionary (or something that behaves likes a dict) of variable parameters.
        :param fpFmt -- format to convert float to string. (Default is %.4g)
        :return: a tuple as an index. tuple is key_name, value in sorted order of key_name.


        Read all files that look like model config files.
        :param pattern: glob pattern to match for model config
        :param direct: directory to look in -- all subdirectories will be looked for
        :return:


        Read model configurations from path_list and store them in self.model_index
          key will be generated from the model parameters and value will be the model.
        :return: models read in


:return: pandas series of model status


        Extract the parameters used in the simulations. Will include ensembleMember -- as a "fake" parameter
        :param numeric -- if True convert all parameters to numeric values using errors='coerce'.
        :param normalize -- If True normalize parameters by min and max values
        :return: pandas dataframe of parameters


        Extract the Obs used in the *individual* simulations. If simulation has no observations then it is ignored.
        :param scale If True data will be scaled.
        :param normalize If True data will be normalized -- distance in SD's from tgt
        :return: pandas dataframe of observations possibly scaled and normalized.
           None will be returned if there are no obs


        compute cost from data.
        :param: scale -- scale data.
        :return pandas series of costs.


        **copy** self.config and add parameters and obs to it. Modified config is returned
        :param filename - pathlib to file (or None). Will override filepath in new config
        :param scale -- passed to self.cost to compute cost when add_cost is True
        :param add_cost -- add cost to the returned configuration
        :return: modified config. The following methods will work:
            finalConfig.parameters() -- returns the parameters for each model simulation
            finalConfig.simObs() -- returns the simulated observations for each model simulation.


        plot cost, normalised parameter & obs values for runs.
          Could do with a clean up and make better use of pandas plotting
        :param figName: name of figure to make -- default is monitor
        :param monitorFile: name of file to save figure to if not None. Default is None
        :return: figure, (costAxis, paramAxis, obsAxis)

        Note needs matplotlib


        Archive a SubmitStudy (or anything that has an archive method)
        :param submit: SubmitStudy to be archived
        :param archive_path: name of the archive file. If None then submit.roodDir/archive.tar is used.
        :param extra_paths: Extra files to be archived. Passed to SubmitStudy.archive.
        :return: archive_path


        Extract an archive.
        :param archive_path: Path to the archive.
        :param direct: Directory where data will be extracted to. If None will be current working directory.
        :return: Archive_study (useful for querying archive info) & SubmitStudy -- ready to analyse/continue SubmitStudy.


    Dump object to file-like object fp. Uses JSON_Encoder to encode it. See json.dump() for documentation
    :param obj: Object to be dumped
    :param fp: File-like object
    *args -- args to be passed to json.dump
    **kwargs -- keyword arguments to be passed to json.dump
    :return -- result of json.dump


    Dump object to str. Uses JSON_Encoder to encode it. See json.dumps() for documentation
    :param obj: Object to be dumped
    *args -- args to be passed to json.dumps
    **kwargs -- keyword arguments to be passed to json.dumps
    :return result of json.dumps


    Load object from file-like object fp. Uses json.load() with object_hook set to JSON_Encoder.decode
    :param fp:
    :param args:
    :param kwargs:
    :return:


    Decode object from s. Uses json.loads() with object_hook set to JSON_Encoder.decode.
    See json.loads for documentation
    :param s: thing to be decoded
    :param args: args to be passed to json.loads
    :param kwargs: kwargs to be passed to json.loads
    :return: result of json.loads


        Register object method that converts an object to a list or dictionary
        :param method: method to convert
        :return:Nothing


        Register class method that generates object from list or dict.
        :param mycls: class for which class method comes from.
        :param classmethod: class method to be registered
        :return:


        Factory method to create object of type class_name using a dict
        :param class_name: Name of the class to create
        :param values: values to use to initialise object with
        :return: object


        Convert an object to something that can be serialized by json.
        :param obj: object to convert
        :return: serialized version of object. If object class has not been registered a KeyError will be raised


        Decode dct
        :param dct: dict to be decoded.

        If dct contains the key __cls__name__ then it will be decoded using value_to_obs.
        If not dct will be returned as no decoding neeed.

         Dct, for decoding,  should be of the form:
        {
            "__cls__name__" : "class_name",
            "object" : Stuff to be decoded
        }
        ValueError will be raised if the dct is not of this form.
        :return: input dct (if no decoding needed) or a decoded object.


        Called by json serialization
        :param obj: Any object

        :return: Serialised object packaged in a three element dict with the following key/value pairs:

         "__cls__name__" : name of the class of the object.
         "__module__" : name of the module where class is defined
         "object": the serialized version of the object

        Also see decode which reverses this.


 Read  value from disk file containing namelists. Files will be cached to speed up subsequent reads.
        :param dir: Directory where namelist is.
        :param clean -- If True clean the cache before reading.


        Generate namelist_var from dictionary
        :param dct: dct
        :return: return  namelist initialised from dict.


    Clean out the file cache
        :return:


        Read/cache file
        :param filepath: path to file containing namelist
        :param clean: If True,clean the cache
        :param make_copy: If True, make a deep copy of the cached data
        :return: namelists in the file. Will read in data if needed.


        Update dict indexed by files. Each containing a f90nml namelist.
        Really for internal use by this class.
        :param nl_info -- iterable of namelist, values.
        :param dirpath: path to root directory for namelists
        :param clean: clean cache if set to True
        :param update If true will update namelist from (cached) file system. If not will return nl info for changes.
        :return: dict indexed by the namelist filepath

        Example usage files=files_to_change([(VF1_nl,3.0), (RHCRIT_nl,[0.8,0.8,0.8,...0.9,0.95])],dirpath=pathlib.Path('test_dir'))


        Modifiy namelist files. Sadly f90nml.patch() is a bit flaky.
          So, for each file we read in the entire contents.
         Update using the changes, write to a temp file, remove the input file
            and move the temp file to the original location.
        :param nl_info: iterable of namelist_var, value pairs,
          will also clear cache after all modification done.
        :return: nada though all files used will be modified.

        Example usage namelist_var.nl_modify({VF1_nl:0.5,ENTCOEF_nl:[0.8,0.8,0.85,...0.9,0.95])


        Update param info by merging in other_param_info.
        Does by iterating over other_param_info.param_constructors.items() and
           then calling register() on each element in the list. This then updates everything needed!
        :param other_param_info: another param_info
        :return: nada. Modifies param info in place


        register a namelist or callable to the parameter. An error will be raised if var_to_set has already been registered unless
           parameter is being replaced.
        :param parameter: name of parameter
        :param var_to_set: namelist_vars (though could be anything that can form key to hash) or method to run
        :param duplicate: If True allow duplicate keys -- new values will be appended rather than replacing existing values
        :return: Nada
        :Examples: param_construct.register('RHCRIT',rhcrit),
                   nl= namelist_var(filepath='fred',namelist='atmos',nl_var='VF1')
                   param_construct.register(VF1=nl)


        Read parameter value from model instance.
        :param model: model instance. Only used if method first element in parameter defn,
        :param parameter: parameter wanted
        :return: value. Depends on what is in the model...


        Return parameter information for a specific value as namelist/value tuple. Later functions will actually set them
        :param model: the model instance -- only used if method used
        :param parameter: parameter name
        :param value: value to be set and passed to method

        :return:


        Generate parameter settings.
        :param model: model (needed for call to param)
        :param kwargs: parameter/values
        :return:list of things to be actually set. That actually should be done by the model


        Convert parameter info to a pandas dataframe. Any functions will be ignored as they get defined at model instance time.
        Currently, only deals with namelist_var. Extend if you need different or additional types.
        :return: df


        Add parameters from file.
        :param filepath: path to csv file (can be anything accepted by pandas.read_csv).
          parameter type  filepath namelist nl_var name -- extend for different types.
          For this implementation type **must** be namelist_var. Extend for different ways of specifing parameters.
          All four parameters needed to create a namelist_var must be present
        :param duplicate: If True allow duplicates otherwise final parameter found is used.
        :param kwargs -- all remaining arguments are passed to read_csv.
        :return: Nothing.Modifies self in place


            Convert a string to a bool, float or int. If None passed return None
            :param value: value to be converted
            :return: converted value


        Print out the parameters we have registered
        :return: Nothing


        Generate a dictionary rep using param_constructors. If callable the function name will be returned.
        Otherwise will be left alone.
        :return: dict


        Tricky part is need to ignore the functions.
        :return: object


        Return a list of all parameters that have been registered.
        :return:


    Wee test fn for trying out things.
    :param config -- configuration. Provides, parameter min, max & ranges and targets.
    :param params -- dict of parameter values
    returns  "fake" data as a pandas Series


    Expand any env vars, convert to path and then expand any user constructs.
    :param filestr: path like string
    :return:expanded path


    Function to run when error found in rmtree.
    :param func: function being called
    :param path: path to file being removed
    :param exc: failure status
    :return: None


    Recursively Delete the contents of a directory
    :param dir: path to directory to have all contents removed.
    :return: Nada


    copy a directory for test purposes. Need to do this so created directory is
    read/write so can be deleted.
    :paam inDir: Inout directory to be copied
    :param outDir: Target for copy. If it exists it will be deleted.
    :keyword trace: Defualt value False -- if True print out more info
    :keyword setDir: Default value '' -- if set copy this directory. If not set then only start will be copied
    :return: status == 0 if successful. Something else if not!


    Initialise RNG based on parameter as pandas series. So is deterministic.
    :param param: pandas series of values
    :return: seed for RNG


    Adds standard post-processing arguments to a parser and then parse the parser.
     Then read in the post_processing information
    :param parser: parser to have arguments added to.
      arguments added are:
         CONFIG -- path to config file.
         -d/-dir -- name of input directory.
         OUTPUT -- path to output file
         -v/--verbose -- increase verbosity.  -v turns logging.info on while -v -v turns logging.debug on.

    config will be loaded from CONFIG and postProcess key extracted.
    output file  taken from post_process info (if present) otherwise
    sets appropriate  logging level using basicConfig and force=True (will overwrite any existing logging)

    :return: args (after parsing) post_process dict
      args contains whatever in parser and:
        CONFIG -- path to config file
        OUTPUT -- path to OUTPUT file
        dir -- path to directory where data to be read from.
        verbose -- level of verbosity.


        Create an engine (used by submission system)
        :param engine_name: name of engine wanted.
        :param ssh_node: node to ssh to where engine can submit things
        Sets up engines which hold cmds for SGE or slurm respectively. .


        Connect command. Assumes that filesystems match on all nodes and runs commands on self.ssh_node
         If self.ssh_node is None -- just returns the cmd
        :param cmd: command to run
        :param rundir: Where to run the command.
           If not provided command will be run in current working dir when the generated command is run.
            This assumes you are running the command with  journal.run_cmd(). See doc for that method.
            If rundir is a shell var then it will be expanded when the command is run if journal.run_cmd is used.
        :return: modified cmd which includes ssh


        Submit function.
        :param mem: Memory (in Mybtes) needed for job
        :param time: Time (in seconds) needed for job
        :param cmd: list of commands to run.
        :param name: name of job
        :param outdir: Directory where output will be put.
             If None will be set to cwd/output.
        :param rundir: Directory where job will run.
           If None will run in current working dir when command is run.
        :param run_code: If provided, code to use to run the job.
        :param hold: If provided as a string or list of strings,
        this (these) jobids will need to successfully run before cmd is ran.
          If provided as a bool then job will held if hold_jid is True.
          If False no hold will be done
        :param n_cores: No of cores to use.
        :param n_tasks: no of tasks to run. Will submit an array job.
        :return: the command to be submitted.


        cmd to release_job a job
        :param jobid: jobid to release_job
        :return: cmd to release_job job


        cmd to kill a job
        :param jobid: jobid to kill
        :return: cmd to kill a job.


        Extract jobid from output of a qsub or similar command
        :param output: output from submission (or similar)
        :return: jobid as a string.


        Return the status of a job.
        :param job_id: job id for status to be checked.
        :param full_output If True will return (raw) full output
        :return: One of 'notFound','Running','Held','Error','Queuing','Suspended',f'Failed {status} or full output


        Return job id of calling process.
        :return: the job id of the process. Fails if not found


        Function to submit to SGE
        :param mem: Memory (in Mybtes) needed for job
        :param time: Time (in seconds) needed for job
        :param cmd: list of commands to run.
        :param name: name of job
        :param outdir: Directory where output will be put. If None will be set to cwd/output.
        :param rundir: Directory where job will be ran. If None will run in current working dir when command is run.
        :param run_code: If provided, code to use to run the job
        :param hold: If provided as a string or list of strings,
        this (these) jobids will need to successfully run before cmd is ran.
          If provided as a bool then job will held if hold_jid is True. If False no hold will be done
        :param n_cores: No of cores to use.
        :param n_tasks: If not None the size of the task array to be ran.
        :return: the command to be submitted.


        SGE cmd to release_job a job
        :param jobid: jobid to release_job
        :return: cmd to release_job job


        SGE cmd to kill a job
        :param jobid: jobid to kill
        :return: cmd to kill a job.


        Return the status of a job. Tuple will contain strings. Needs to actually run.
        Could be turned into a gen command then parse the output.
        :param job_id: job id for status to be checked.
        :param full_output If True will return (raw) full output
        :return: One of 'Running','Held','Error','Suspended','Queuing',"Failed"


:return: the job id of the process. Uses $JOB_ID env var to get it.


        Function to submit command to SLURM

        :param cmd: list of commands to run.
        :param name: name of job
        :param outdir: Directory where output will be put. If None then will be cwd/output
        :param rundir: Directory where job will be ran. If None will run in current working dir when command ran.
        :param run_code: If provided, code to use to run the job
        :param hold: If provided as a string, this jobid will need to successfully run before cmd is ran.
          If a list (of jobid's) then all jobs will need to be run
          If provided as a bool then job will held if hold_jid is True. If False no hold will be done
        :param mem: Memory (in Mybtes) needed by job
        :param time: Time (in seconds) needed by job,
        :param n_cores: No of cores to be requested
        :param n_tasks: If not not None the number of array tasks to run.
        :return: the command to be submitted.


        SLURM cmd to release_job a job
        :param jobid: The jobid of the job to be released
        :return: a list of things that can be ran!


        Slurm cmd to kill a job
        :param jobid: The jobid to kill
        :return: command to be ran (a list)


        Extract jobid from output of sbatch or similar command
        :param output: string of output
        :return: jobid as a string.


        Return the status of a job. Tuple will contain strings
        :param job_id: job id for status to be checked.
        :param full_output If True will return (raw) full output
        :return: One of 'Running','Held','Error','Suspended','Queuing',"Failed","NotFound"


        Return the process job id.
        :return: the job id of the process. Uses SLURM_JOB_ID env var to get it.


        staticmethod to provide now. Done so can mock it for testing.
        Times are such a pain!
        :return: utcnow datetime.datetime


        Update the _history directory. Key will be self.now()
        If self._history does not exist, then it will be created.
        Routine updates existing values so that multiple updates in a short time will preserve _history.
        Short time defined as less than precision of str(now))
        :param message:message text to be stored.
        If message is None then  self._history will be created and the control returns
        :return:


:return: last history  or None if no history.


         Print out history
         :return:


        Store output and cmd ran in self._output with key the time.
           If self._output does not exist it will be created.
        :param cmd: command that was ran
        :param result: result from the command
        If both  are None then only the creation of _output will be done
        :return: Nothing


        Print out the output
        :return: Nothing


        Run a command using subprocess.check_output and record output.
        :param cmd: command to run. Any shell variables ($VARNAME) in the cmd will be expanded at the time of running.
        :**kwargs -- kwargs to be passed to subprocess.check_output. Will update defaults which is just text=True and stderr=subprocess.DEVNULL
        :return: output from running command


    Convert flexi_path to path
    :return: if possible a path representation of path.


        Translate PurePath (coz we have moved files or system). Expect to be used in inherited class from_dict
        Probably will work if Paths too though care needed with absolute paths
        :param path - path to be translated.


        Return  class initialised from dct by copying keys to attributes in obj
        but only those that  exist after initialisation.
        Make sure your dct is sensible...This is very generic.
        This is really a factory method
        :param dct: dict containing information needed by class_name.from_dict().
         pure_paths will be converted -- see fill_attrs.


Convert pure paths (if possible to paths)
        First apply translate_path to anything thing that is a path
          then trys to convert a purePath of the right type (Windows on Windows; Posix on anything else) to a path,


        Fill in the attributes in self from dct. Used by from_dict.
        :param dct: dict of key value. self.key=value if self.key exists
        :param convert_pure_paths: If True call convert_pure_paths on the dict.
        :return: Nothing. Changes self in place


        Convert an object to a dict.
        To support portability across multiple OS. paths are converted to purePath
        :return: dct


        Load an object configuration from specified file.
        The correct type of object will be returned.
        :param check_types: types to be checked using isintance. If type not as expected error will be triggered
        :param file path to file to be read in.
           If str passed then it cls.expand will be ran on it.
        :return: Object of an appropriate type.


Return the class name.


        Write object (as json) to config_path
        :param config_path: path to write data to. Directory where this goes will be created if necessary.
        :return:whatever result of generic_json.dump is


        Expand any env vars, convert to path and then expand any user constructs.
        :param filestr: path like string or None
        :return:expanded path or, if filestr is None, None.


    Decorator to register a parameter.
    :param name: name of the parameter
    :return:


        Register all functions in the class
        :return: param_info to be merged into other information,


        Remove class from registry. By default self.
        :param name . If not None the name of a class to be removed
        :return: Class removed from registry


        Return list of known models
        :return: list of know models


        Create a model
        :param class_name: name of class to make
        :param args: positional arguments to pass to initiation
        :param kwargs: kwargs to pass to init
        :return: new model object


        Load a configuration
        :param model_path:  where the configuration  is stored
          config_path will be set to model_path
          model_dir will be set to model_path.parent.
          warnings given if these are changes.
        :return: loaded model


        Add information on parameters and functions.

        :param param_info: a dict with keys variable names and values either a namelist_var or callable.
          You probably should not use a callable here as better to register it when declared.
        :param duplicate If True allow duplicates which will add new namelist/callable info to existing.
        :return: Nothing


        Update class info on known parameters from CSV file
         Calls param_info.update_from_file(filepath) to actually do it!
         See documentation for that
        :param filepath: path to csv file
        :param duplicate -- allow duplicates.
        :return:


        Remove param info
        :return: Nada


                Compute rhcrit on multiple model levels
                :param rhcrit: meta parameter for rhcrit
                :param inverse: default False. If True invert the relationship
                :return: (value of meta parameter if inverse set otherwise
                   a tuple with namelist_var infor and  a list of rh_crit on model levels


        Reset the state for another set of evaluations
        Sets self.fns_to_eval & self.trace to empty list
        :return: nada


        Generate key from parameters.
        :param parameters -- a np array .
        :return: bytes representing the data


        return results from cache or if not in cache return None
        :param params:  Parameters
        :return: None (not in cache) or values from "model" evaluation.


        Simulate model
        :param params: Parameters being evaluated
        :return: Simulated obs,


        Evaluate all the models that needed evaluation!
        Evaluates all in  self.fns_to_eval and adds the result to the self.cache,
        :return: Nada


        Initializes using name and reference values in dct (popping them out so they don't get added twice)
        Then copies keys to attributes in obj
        but only those that  exist after initialization.
        This is really a factory method
        :param dct: dict containing information needed by class_name.from_dict()
        :return: initialised object


        Compare two objects and identify their differences by comparing their attributes.
        Based on __eq__ method. With rewriting by chatGPT
        :param other: The other object to compare against.
        :return: Set of differing attributes between the objects.


        dump a model configuration to self.model_dir/model_config_name
        :return: whatever dump does


        Get iterable  of namelist/vars  to set.
        :param parameters: If None use self.parameters augmented by self.parameters_no_key
        Calls self.param_info.gen_parameters to actually work out what namelists and values are to be set,
        then verifies  have iterable of namelist,value pairs.
        For some models you may want to override this method to deal with
          other ways of setting parameters.
           This implementation deals with namelists and functions that return None,
        :return: an iterable of  namelist, value pairs
        Example: model.gen_param_set()


        Return namelists that have been changed. Note does not actually change anything about the model.
        :return: set of namelists indexed by file.


        Create a new model by copying reference. Overwrite (and call superclass) for your own model.
        For example if you want to modify your reference model.
        :return:nothing.


        Run create_model and set_params, update status.
        Will verify that (if defined) post-processing script exists failing if not
        Also, should make any changes needed to those files.
        :return:


        Modify model. This does nothing and is designed to be overwritten in classes that inherit from it.
        Those should call this first as it checks that set_status_script exists and updates history

        :return: None


        Submit a model and its post-processing.
        Post-processing gets submitted first but held.
        Model gets cmd to release_job post-processing included before it gets submiteded.

        :param fake_function -- if provided, no submission will be done.
          Instead, this function will be used to generate fake obs.
          Designed for testing code that runs whole algorithms.
          Takes one argument -- dict of parameters. Returns pandas series.

        :return: The jobid of the post-process job submitted. (If a post-processing job submitted)
            Post processing runs after the model has completed.
             At the time of submission we don't know what the final model job id is -- because it might self continue.

        Example:
         model.submit_model()


        Generate the submission command. Over-ride this for your own model.
        If status is INSTANTIATED or PERTURBED then this runs self.engine.submit_cmd on  [self.submit_script] and
        if CONTINUE runs on  [self.continue_script]
        output should go to model_dir/'model_output' which will be created if it does not exist.


        Set status to running, store current job id & increment run_count
        :return: current job id


        Guess of a model has failed.
          STATUS = RUNNING and status of last model jobid is unKnown likely means model has failed.
        :return: True if guessed FAILED, False if not


        Set status to PERTURBED. Will need to be continued or submitted which requires submission information.
        :param parameters: dict of parameters & values to use to generate random perturbation.
        This will update parameters_no_key so key generation is unaffected and
          increase perturb_count by 1 so algorithm can adjust if multiple perturbations done,
        :return:None

        This likely needs overwriting for specific models as parameters will be set.
        For implementation in actual model class have something like:
        def perturb(self):

            parameters=dict(rand_init=random())
            super().perturb(parameters)


        Mark simulation as continuing.
        :return: Nada


        Mark simulation as restarting -- "instantiated"
        :return: nda


        Run the post-processing job by releasing self.jid
        Then set status to SUCCEEDED
        :return: output from running self.post_process_cmd


        Run the post-processing, store output and set status to COMPLETED.
        "Contract" for a post-processing script
        1) takes a json file as input (arg#1) and puts output in file (arg#2).
        2) It is being ran in the model_directory.
         arg#1 needs json.load to read the json file. Code should expect a dict and use the postProcess entry.
             This allows it ot read in and act on a StudyConfig file.
         arg#2 can be .json or .csv or .nc
        :return: output from post-processing.


        Read the post processed data.
         This default implementation reads simulated obs from netcdf, json or csv data and
         stores it in the Model as a pandas series. Tests that nothing is null.
         :param post_process_file: path to the post processed data containing the simulated observations,
        :return: a pandas series of the simulated


        Read parameter values from self.model_dir
        :param parameters: list of parameters OR parameter to read. If None all known parameters will be read.
        :param fail. If true fail if namelist file is not found.
        :return: dict of parameter/value tuples


        Return True if model is instantiable -- which means its status is CREATED
        :return:


        Return True if model is submittable -- which means its status is CONTINUE or INSTANTIATED or PERTURBED
        :return:


        Return True if model has failed
        :return:


        Return True if model is continuable.
        :return: Return True if model is continuable.


        Return True if model is running.
        :return: True if model status is RUNNING


        Return True if model.status is CREATED
        :return:


        Return True if model is processed
        :return:


        Delete all on disk stuff. Do by deleting all files in self.model_dir and self.config_path.

        :return: None


        Archive parts of model_dir & config_path to archive_path. This version will raise NotImplementedError as
          needs to be specialised for individual models
        :param archive_path:
        :return:None


        Generate key from keys and values in self.parameters.
        This should be unique (to some rounding on float parameters)
        :param fpFmt -- format to convert float to string. (Default is %.4g)
        :return: a tuple as an index. tuple is key_name, value in sorted order of key_name.


        Do nothing as perturbing initial conditions is model-specific. But needed
         for test cases.
        :param ensMember: ensemble member. The ensemble member wanted.
        :return: None (for now) as nothing done.


    :param inyold: 3 element tupple of y values
    :param inxold: 3 element tupple of x values
    :param xnew:  x value for interpolation
    :return: interpolated values


:return: The name checking length is 5


        HadCM3 modify_model. Overwrites Model version.
        Call the superclass. modify_model in case it does something!
        Then work out the path to set_status_script
        Cleans up fixClimFCG
        Modify the Submit, then script, create work dir and copy any astart ostart files there might be into it.,
        Then create the continue submit file.
        Finally create the script that tests if finished or not.
        :return:


        Perturb HadCM3 model. Default works through up to 6 parameters then gives up!
        :return: nothing


        Used by the submission system to allow the post-processing job to be submitted when the simulation has
        completed. This code also modifies the UM so that when a NRUN is finished it automatically runs the
        continuation case. This HadCM3 implementation generates a file call optclim_finished which is sourced by
        SCRIPT. SCRIPT needs to be modified to actually do this. (See modifySCRIPT).
        :param set_status_cmd. path to set_status_cmd


        Create the workdir and if self.reference/W has .astart & .ostart copy those into created workDir
        :return: nada


        modify script.
        :param set_status_script -- path to script that sets status.
         set ARCHIVE_DIR to runid/A -- not in SCRIPT??? WOnder where it comes from. Will look at modified script..
         set EXPTID to runid  --  first ^EXPTID=xhdi
         set JOBID to jobid  -- first ^JOBID

          After . submitchk insert:
         . $JOBDIR/optclim_finished ## run the check for job release_job and resubmission.
            This will be modified by the submission system
         :return:


        :param run_info -- run information.
          should include runCode and runTime.
        :param engine -- engine info. Not used  but provided with superclass method

        :return:  cmd to be run.


        Modify  Submit script for HadCM3
        Changes to SUBMIT
          set RUNID to runid in SUBMIT   == first ^export RUNID=
          set JOBDIR to dirPath   -- first ^JOBDIR=
          set CJOBN to runid in SUBMIT  -- first ^CJOBN=
          set MY_DATADIR to dirPath
          set DATAW and DATAM to $MY_DATADIR/W & $MY_DATADIR/M respectively.

        :return: nada


        Fix problems with the CLIM_FCG_* namelists so they can be parsed by f90nml.
        Generates CNTLATM.bak and modifies CNTLATM
        :return: None


        Generate continuation script for model so that it is a CRUN and STEP=4
        Must run *after* the modifyScript method has ran as the contine script needs those changes too!
        :return: script name of continue simulation.


        Get timestep info and return it in seconds. (Model internally uses steps per day (I think))
        :return: timestep in seconds


        Work out the dx and dy values for the model (in radians).
        :param expect: two element np.array of expected values. First element is dx, and second element is dy
        :param units: units to return dx and dy in. None (radians), 'degrees' or 'meters'.
        :return: dx and dy values in specified units.


        Work out the number of levels by reading  p_levels
        :param expect If not None if nlev does not equal expect an error will be raised
        :return: the number of levels.


        Compute array of eacf values for each model level. If inverse set work out what meta-parameter is from array.
        :param eacf: meta-parameter value for eacf. If None then inverse will be assumed regardless of actual value
            inverse relationship.  If namelist value not found then default value of 0.5 will be returned.


        Set/read namelists for spherical/non-spherical ice.
        :param sphIce: True if want spherical ice (which is default). If None inverse calc is done
        :return:  returns nl/tuple pair or, if inverse, the value of sphIce


        :param time_input:  start time as ISO format  string. If None inverse will be done
        :return:  namelist info and values as array. (or if inverse set return the start time)


        :param duration:  Duration as string in ISO 8061 format (PnYnMnDTnHnMn.nnS)
        :param runTarget: namelist info suitable for runTarget. (default). If False namelist info suitable for resubInterval
        :return: namelist info and values as array. (or if inverse set return the run target )
             return  Duration as 6 element array  TIODO: string in ISO 8061 format (PnYnMnDTnHnMn.nnS)


        set runTarget -- see timeDelta for documentation.
        :param target_input: target length of run as ISO string
        :return: Return list of nl,value pairs or run_target


        Set resubmit durations -- see timeDelta for documentation
        :param interval_input:


        Compute experiment and job id.
        :param name: 5-character UM name. If None inverse (read file) calculation is done
        :return: namalises and values to set. unless name is None. Then name will be returned


        Compute cw_sea from  cw_land
        :param cw_land: value of cw_land. If None then inverse calculation is done
        :return: returns dict containing cloud_cw_land and cloud_cw_sea values.


        Compute gravity wave parameters given kay parameter.
        :param kay: value of kay. If None then inverse calculation is done.
        :return: list  containing kay_gwave and kay_lee_gwave parameters. (or value of kay if inverse set)


        Compute ice albedo values (alpham & dtice) given alpham
        :param alpham: value of alpham (if forward). If None inverse calculation will be done.

        :return:


        Support Function to compute diff  coefficient
        :param dyndiff: diffusion value in hours. If None inverse calculation will be done
        :param dyndel: order of diffusion. 6 is default corresponding to 3rd order.
        :return:


        Compute arrays of diffusion co-efficients for all   levels.
        :param diff_time: time in hours for diffusion. If None inverse calculation will be done
        :return: if inverse set compute diffusion timescale otherwise compute diffusion coefficients.
             returns them as a list
             diff_coeff -- diffusion coefficient, diff_coeff_q -- diffusion coefficient for water vapour
             diff_power -- power of diffusion, diff_power_q -- power for water vapour diffusion.


        Compute rhcrit on multiple model levels
        :param rhcrit: meta parameter for rhcrit. If None inverse calculation  will be done.
        :return: namelist, value.


        Generate namelist for NH and SH ocean ice diffusion coefficients.
        :param OcnIceDiff: Value wanted for ocean ice diffusion coefficient
            Same value will be used for northern and southern hemispheres. If None inverse calculation will be done.
        :return: (by default) the namelist/value information as list


        Generate namelist for NH and SH ocean ice maximum concentration
        :param iceMaxConc: Value wanted for ocean max concentration
            Same value will be used for northern and southern hemispheres with SH maximum being 0.98.
            If None inverse condition will be done
        :return: nl/values to set the values, if inverse return the NH value.


        Generate namelist for changes to Ocean isopycnal diffusion.
        :param ocnIsoDiff: Value for ocean ice diffusion. Will set two values AM0_SI & AM1_SI
           Note these picked by examining Lettie Roach's generated files. If None then inverse calculation will be done
        :return: namelist/values to be set or the namelist value.


        :param value: value to be set -- typically the name of a file. If None then the inverse calculation will be done.
        :param nl_var: namelist variable to be set.
        :return:namlelist/value tupple to be set or whatever "value" is in the model namelist.


        :param astartV: value of ASTART
        :return: nl/tuple (if astartV is not None) to be set or whatever astart is in the models namelist.


        :param ainitialV: value of ASTART
        :return: nl/tuple (if ainitialV is not None) to be set or whatever ainitial is in the models namelist.


        :param ostartV: value of OSTART
        :return: nl/tuple (if ostartV is not None) to be set or whatever ostart is in the models namelist.


        :param oinitialV: value of OSTART
        :return: nl/tuple (if oinitialV is not None) to be set or whatever oinitial is in the models namelist.


        Do nothing as for HadCM3 random perturb is from name.
        :param ensMember: ensemble member. The ensemble member wanted.
        :return: None (for now) as nothing done.


        Do what ever is necessary. clone the reference case is the best way to proceed.
        :return:


        Make changes to run_simple_model.py
        Adds in cmds to set status.
        :return: nada


        Create the string to run a cmd to set status.
        :param status: status to set.
        :param indent: How much to indent lines
        :return: list of strings to print to model script file


        Generate the submission command.
        If status is INSTANTIATED or PERTURBED then submit to the Q run_simple_model.py


        Sets fail_probability to 0.0 so stopping and random failure.
        :return: Nada!


        Compute rhcrit on multiple model levels
        :param rhcrit: meta parameter for rhcrit. If Noen relationship will be inverted.
        :return: (value of meta parameter if inverse set otherwise
           a tuple with namelist_var infor and  a list of rh_crit on model levels


        Clean up by removing the temp directory contents
        :return:


        Simple way of checking script OK!
        :param self:
        :return:


    Decorator function to produce a clean fail if data is not present
    :param function: function to be decorated.
    :return: decorated function


    Guess the names of latitude,longitude & vertical co-ords in the dataArray
    starts with latitude/longitude, then lat/lon, then latitude_N/longitude_N then lat_N,lon_N
    and with atmosphere_hybrid_sigma_pressure_coordinate, altitude, air_pressure for vertical

    :param dataArray:
    :return: latitude, longitude & vertical co-ord names


    Compute the delta_pressures for each model level for UM hybrid-sigma grid
    For which pressure is a+b*pstar.
    :param pstar: Surface pressure
    :param a_bounds: bounds of "a" co-efficients
    :param b_bounds: bounds of "b" co-efficients
    :return: Pressure thicknesses as a function of vertical co-ord for each model grid cell


    Compute total column of substance (kg/m^2) assuming hydrostatic atmosphere.
        (Atmospheric mass in layer is \Delta P/g)
    :param data: dataArray of the variable for which column calculation is being done.
        Assumed to be a mass mixing ratio (kg/kg)
    :param delta_p: dataArray of pressure thicknesses (Pa) for each level.
    :param scale: real scale factor.If None no scaling is done
    :param vertical_coord: default is None. If None then will be guessed using guess_lat_long_vert_names()
    :return: total_column of substance


     Return dictionary of standard (or long)  names for each variable in a dataset.

    :param dataset: xarray data
    :param name: what you want to return (None, 'standard','long'). If None (default) then standard_name will be returned
    :return: dict of standard names


    Setup the processing information
    :param dataset: the dataset containing the data
    :param land_mask -- land mask as a dateArray.
    :param latitude_coord -- name of latitude co-ord (optional; default None)
      If set to None then will be guessed using
    :return: dict containing data to be processed.


        Lookup name/variable and then return datarray corresponding to it.
        If not present return None
        :param long_name:
        :param **kwargs: Remaining kw args passed to select
        :param dataset:
        :return:


    Compute means for NH extra tropics, Tropics and SH extra Tropics.
    Tropics is 30N to 30S. NH extra tropics 30N to 90N and SH extra tropics 90S to 30S
    Arguments:
        :param dataArray -- dataArray to be processed
        :param name -- name to call mean.
        :param latitude_coord: name of the latitude co-ord. Default is NOne.
            If not set then guess_lat_long_vert_names() will be used


    Convert an iso time str to 6 member vector (y,m,d,hr,min,sec)
    :param time: String
    :param ym_only: If True return only year and month -- useful for non standard calendars.
    :return: v


    Guess the names of latitude, longitude & vertical co-ords in the dataArray
    starts with latitude/longitude, then lat/lon, then latitude_N/longitude_N then lat_N,lon_N
    and with atmosphere_hybrid_sigma_pressure_coordinate, altitude, air_pressure for vertical

    :param dataArray:
    :return: latitude, longitude & vertical co-ord names


        Lookup name/variable and then return datarray corresponding to it.
        If list provided iterate over. Any None return None; and then sum values.
        If not present return None
        :param long_name:
        :param **kwargs: Remaining kw args passed to select
        :param dataset:
        :return:


    Generate a time constraint between start_time and end_time. start_time is included while end_time is not included in
    the times requested. start_time and end_time should be passed in as tuples/or lists
    TODO support 360 day calendar -- for example 2005-02-30.


    Compute means for NH extra tropics, Tropics and SH extra Tropics.
    Tropics is 30N to 30S. NH extra tropics 30N to 90N and SH extra tropics 90S to 30S


    Test fn for optimising...
    :param params: np array of parameter values.
    :*extraArgs -- extra arguments. Not used in this
    :**kwargs -- keyword arguments not actually used either.
    :return:


Run gaussnewton (or ref_doGaussNewton) and pacakge results up in a dictionary