title: Datahub Ingestion

description: Root schema of Datahub Ingestion

required: type config

description: Path to file to write to.

description: Path to server's CA certificate for verification of HTTPS communications

descritption: Path to client's CA certificate for HTTPS communications

type: number

description: Experimental: Max parallelism for REST API calls

description: Retry HTTP request also on these status codes

description: URL of DataHub GMS endpoint.

description: Per-HTTP request timeout.

description: Bearer token used for authentication.

description: Extra headers which will be added to the request.

description: Disable SSL verification for HTTPS communications.

description: Kafka bootstrap URL.

description: Passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.SerializingProducer

description: URL of schema registry being used.

description: Passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.schema_registry.SchemaRegistryClient

description: Overridden Kafka topic name for the MetadataChangeEvent

default: MetadataChangeEvent

description: Overridden Kafka topic name for the MetadataChangeProposal

default: MetadataChangeProposal

title: MySQLConfig

description: Base configuration class for stateful ingestion for source configs to inherit from.

title: Env

description: The environment that all assets produced by this connector belong to

default: PROD

title: Platform Instance

description: The instance of the platform that all assets produced by this recipe belong to

title: Options

description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.

title: Schema Pattern

description: Deprecated in favour of database_pattern.

$ref: #/definitions/AllowDenyPattern

title: Table Pattern

description: Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'

title: View Pattern

description: Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'

title: Profile Pattern

description: Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.

title: Domain

description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

description: If the source supports it, include table lineage to the underlying storage location.

title: Profiling

$ref: #/definitions/GEProfilingConfig

title: Username

description: username

title: Password

description: password

format: password

title: Host Port

description: MySQL host URL.

title: Database

description: database (catalog)

title: Database Alias

description: [Deprecated] Alias to apply to database when ingesting.

title: Scheme

default: mysql+pymysql

title: Sqlalchemy Uri

description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.

title: Database Pattern

description: Regex patterns for databases to filter in ingestion.

title: OracleConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

description: Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: If using, omit `service_name`.

title: Database Alias

title: Scheme

description: Will be set automatically to default value.

title: Sqlalchemy Uri

title: Service Name

description: Oracle service name. If using, omit `database`.

title: Add Database Name To Urn

description: Add oracle database name to urn, default urn is schema.table

title: PowerBiReportServerDashboardSourceConfig

description: Any source that produces dataset urns in a single environment should inherit this class

title: Env

title: Username

description: Windows account username

title: Password

description: Windows account password

title: Workstation Name

description: Workstation name

default: localhost

title: Host Port

description: Power BI Report Server host URL

title: Server Alias

description: Alias for Power BI Report Server host URL

title: Graphql Url

description: [deprecated] Not used

title: Report Virtual Directory Name

description: Report Virtual Directory URL name

title: Report Server Virtual Directory Name

description: Report Server Virtual Directory URL name

title: Extract Ownership

description: Whether ownership should be ingested

title: Ownership Type

description: Ownership type of owner

title: Platform Name

title: Platform Urn

title: Report Pattern

title: Chart Pattern

title: DataHubSourceConfig

title: Stateful Ingestion

description: Stateful Ingestion Config

title: Mysql Connection

description: MySQL connection config

title: Kafka Connection

description: Kafka connection config

title: Include All Versions

description: If enabled, include all versions of each aspect. Otherwise, only include the latest version of each aspect.

title: Mysql Batch Size

description: Number of records to fetch from MySQL at a time

title: Mysql Table Name

description: Name of MySQL table containing all versioned aspects

title: Kafka Topic Name

description: Name of kafka topic containing timeseries MCLs

title: Commit State Interval

description: Number of records to process before committing state

title: Commit With Parse Errors

description: Whether to update createdon timestamp and kafka offset despite parse errors. Enable if you want to ignore the errors.

title: BigQueryV2Config

title: Store Last Profiling Timestamps

description: Enable storing last profile timestamp in store.

title: Incremental Lineage

description: When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.

title: Sql Parser Use External Process

description: When enabled, sql parser will run in isolated in a separate process. This can affect processing time but can protect from sql parser's mem leak.

title: Store Last Lineage Extraction Timestamp

description: Enable checking last lineage extraction date in store.

description: Size of the time window to aggregate usage stats.

title: End Time

description: Latest date of lineage/usage to consider. Default: Current time in UTC

title: Start Time

description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.

title: Store Last Usage Extraction Timestamp

description: Enable checking last usage timestamp in store.

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Rate Limit

description: Should we rate limit requests made to API.

title: Requests Per Min

description: Used to control number of API calls made per min. Only used when `rate_limit` is set to `True`.

title: Temp Table Dataset Prefix

description: If you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use `_` because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming.

title: Sharded Table Pattern

description: The regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing,

title: Project Id Pattern

description: Regex patterns for project_id to filter in ingestion.

title: Usage

description: Usage related configs

title: Include Usage Statistics

description: Generate usage statistic

title: Capture Table Label As Tag

description: Capture BigQuery table labels as DataHub tag

title: Capture Dataset Label As Tag

description: Capture BigQuery dataset labels as DataHub tag

title: Dataset Pattern

description: Regex patterns for dataset to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'

title: Match Fully Qualified Names

description: Whether `dataset_pattern` is matched against fully qualified dataset name `<project_id>.<dataset_name>`.

title: Include External Url

description: Whether to populate BigQuery Console url to Datasets/Tables

title: Debug Include Full Payloads

description: Include full payload into events. It is only for debugging and internal use.

title: Number Of Datasets Process In Batch If Profiling Enabled

description: Number of partitioned table queried in batch when getting metadata. This is a low level config property which should be touched with care. This restriction is needed because we query partitions system view which throws error if we try to touch too many tables.

title: Column Limit

description: Maximum number of columns to process in a table. This is a low level config property which should be touched with care. This restriction is needed because excessively wide tables can result in failure to ingest the schema.

title: Project Id

description: [deprecated] Use project_id_pattern or project_ids instead.

title: Project Ids

description: Ingests specified project_ids. Use this property if you want to specify what projects to ingest or don't want to give project resourcemanager.projects.list to your service account. Overrides `project_id_pattern`.

title: Project On Behalf

description: [Advanced] The BigQuery project in which queries are executed. Will be passed when creating a job. If not passed, falls back to the project associated with the service account.

title: Lineage Use Sql Parser

description: Use sql parser to resolve view/table lineage.

title: Lineage Parse View Ddl

description: Sql parse view ddl to get lineage.

title: Lineage Sql Parser Use Raw Names

description: This parameter ignores the lowercase pattern stipulated in the SQLParser. NOTE: Ignored if lineage_use_sql_parser is False.

title: Extract Column Lineage

description: If enabled, generate column level lineage. Requires lineage_use_sql_parser to be enabled. This and `incremental_lineage` cannot both be enabled.

title: Extract Lineage From Catalog

description: This flag enables the data lineage extraction from Data Lineage API exposed by Google Data Catalog. NOTE: This extractor can't build views lineage. It's recommended to enable the view's DDL parsing. Read the docs to have more information about: https://cloud.google.com/data-catalog/docs/concepts/about-data-lineage

title: Convert Urns To Lowercase

description: Convert urns to lowercase.

title: Enable Legacy Sharded Table Support

description: Use the legacy sharded table urn suffix added.

title: Scheme

title: Log Page Size

description: The number of log item will be queried per page for lineage collection

title: Credential

description: BigQuery credential informations

title: Extra Client Options

description: Additional options to pass to google.cloud.logging_v2.client.Client.

title: Include Table Lineage

description: Option to enable/disable lineage generation. Is enabled by default.

title: Max Query Duration

description: Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time.

title: Bigquery Audit Metadata Datasets

description: A list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB.

title: Use Exported Bigquery Audit Metadata

description: When configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute lineage information.

title: Use Date Sharded Audit Log Tables

description: Whether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs.

title: Upstream Lineage In Report

description: Useful for debugging lineage information. Set to True to see the raw lineage created internally.

title: OktaConfig

title: Stateful Ingestion

description: Okta Stateful Ingestion Config.

title: Okta Domain

description: The location of your Okta Domain, without a protocol. Can be found in Okta Developer console. e.g. dev-33231928.okta.com

title: Okta Api Token

description: An API token generated for the DataHub application inside your Okta Developer Console. e.g. 00be4R_M2MzDqXawbWgfKGpKee0kuEOfX1RCQSRx00

title: Ingest Users

description: Whether users should be ingested into DataHub.

title: Ingest Groups

description: Whether groups should be ingested into DataHub.

title: Ingest Group Membership

description: Whether group membership should be ingested into DataHub. ingest_groups must be True if this is True.

title: Okta Profile To Username Attr

description: Which Okta User Profile attribute to use as input to DataHub username mapping. Common values used are - login, email.

title: Okta Profile To Username Regex

description: A regex used to parse the DataHub username from the attribute specified in `okta_profile_to_username_attr`.

title: Okta Profile To Group Name Attr

description: Which Okta Group Profile attribute to use as input to DataHub group name mapping.

title: Okta Profile To Group Name Regex

description: A regex used to parse the DataHub group name from the attribute specified in `okta_profile_to_group_name_attr`.

title: Include Deprovisioned Users

description: Whether to ingest users in the DEPROVISIONED state from Okta.

title: Include Suspended Users

description: Whether to ingest users in the SUSPENDED state from Okta.

title: Page Size

description: The number of entities requested from Okta's REST APIs in one request.

title: Delay Seconds

description: Number of seconds to wait between calls to Okta's REST APIs. (Okta rate limits). Defaults to 10ms.

title: Okta Users Filter

description: Okta filter expression (not regex) for ingesting users. Only one of `okta_users_filter` and `okta_users_search` can be set. See (https://developer.okta.com/docs/reference/api/users/#list-users-with-a-filter) for more info.

title: Okta Users Search

description: Okta search expression (not regex) for ingesting users. Only one of `okta_users_filter` and `okta_users_search` can be set. See (https://developer.okta.com/docs/reference/api/users/#list-users-with-search) for more info.

title: Okta Groups Filter

description: Okta filter expression (not regex) for ingesting groups. Only one of `okta_groups_filter` and `okta_groups_search` can be set. See (https://developer.okta.com/docs/reference/api/groups/#filters) for more info.

title: Okta Groups Search

description: Okta search expression (not regex) for ingesting groups. Only one of `okta_groups_filter` and `okta_groups_search` can be set. See (https://developer.okta.com/docs/reference/api/groups/#list-groups-with-search) for more info.

title: Mask Group Id

title: Mask User Id

title: LookMLSourceConfig

description: Any source that is a primary producer of Dataset metadata should inherit this class

title: Env

title: Platform Instance

title: Explore Naming Pattern

description: Pattern for providing dataset names to explores. Allowed variables are ['platform', 'env', 'project', 'model', 'name']

title: Explore Browse Pattern

description: Pattern for providing browse paths to explores. Allowed variables are ['platform', 'env', 'project', 'model', 'name']

title: View Naming Pattern

description: Pattern for providing dataset names to views. Allowed variables are ['platform', 'env', 'project', 'model', 'name']

title: View Browse Pattern

description: Pattern for providing browse paths to views. Allowed variables are ['platform', 'env', 'project', 'model', 'name']

title: Tag Measures And Dimensions

description: When enabled, attaches tags to measures, dimensions and dimension groups to make them more discoverable. When disabled, adds this information to the description of the column.

title: Platform Name

description: Default platform name. Don't change.

title: Extract Column Level Lineage

description: When enabled, extracts column-level lineage from Views and Explores

title: Git Info

description: Reference to your git location. If present, supplies handy links to your lookml on the dataset entity page.

title: Base Folder

description: Required if not providing github configuration and deploy keys. A pointer to a local directory (accessible to the ingestion system) where the root of the LookML repo has been checked out (typically via a git clone). This is typically the root folder where the `*.model.lkml` and `*.view.lkml` files are stored. e.g. If you have checked out your LookML repo under `/Users/jdoe/workspace/my-lookml-repo`, then set `base_folder` to `/Users/jdoe/workspace/my-lookml-repo`.

title: Project Dependencies

description: A map of project_name to local directory (accessible to the ingestion system) or Git credentials. Every local_dependencies or private remote_dependency listed in the main project's manifest.lkml file should have a corresponding entry here. If a deploy key is not provided, the ingestion system will use the same deploy key as the main project.

title: Connection To Platform Map

description: A mapping of [Looker connection names](https://docs.looker.com/reference/model-params/connection-for-model) to DataHub platform, database, and schema values.

title: Model Pattern

description: List of regex patterns for LookML models to include in the extraction.

title: View Pattern

description: List of regex patterns for LookML views to include in the extraction.

title: Parse Table Names From Sql

description: See note below.

title: Sql Parser

description: See note below.

title: Project Name

description: Required if you don't specify the `api` section. The project name within which all the model files live. See (https://docs.looker.com/data-modeling/getting-started/how-project-works) to understand what the Looker project name should be. The simplest way to see your projects is to click on `Develop` followed by `Manage LookML Projects` in the Looker application.

title: Transport Options

description: Populates the [TransportOptions](https://github.com/looker-open-source/sdk-codegen/blob/94d6047a0d52912ac082eb91616c1e7c379ab262/python/looker_sdk/rtl/transport.py#L70) struct for looker client

title: Max File Snippet Length

description: When extracting the view definition from a lookml file, the maximum number of characters to extract.

title: Emit Reachable Views Only

description: When enabled, only views that are reachable from explores defined in the model files are emitted

title: Populate Sql Logic For Missing Descriptions

description: When enabled, field descriptions will include the sql logic for computed fields if descriptions are missing

title: Process Isolation For Sql Parsing

description: When enabled, sql parsing will be executed in a separate process to prevent memory leaks.

title: Process Refinements

description: When enabled, looker refinement will be processed to adapt an existing view.

title: RedshiftConfig

title: Store Last Profiling Timestamps

title: Incremental Lineage

title: Sql Parser Use External Process

title: Store Last Lineage Extraction Timestamp

title: End Time

title: Start Time

title: Store Last Usage Extraction Timestamp

title: Top N Queries

description: Number of top queries to save to each table.

title: User Email Pattern

description: regex patterns for user emails to filter in usage.

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

description: Whether to report read operational stats. Experimental.

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Email Domain

description: Email domain of your organisation so users can be displayed on UI appropriately.

title: S3 Lineage Config

description: Common config for S3 lineage generation

title: Env

title: Platform Instance Map

description: A holder for platform -> platform_instance mappings to generate correct dataset urns

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: Default Schema

description: The default schema to use if the sql parser fails to parse the schema with `sql_based` lineage collector

title: Include Table Lineage

description: Whether table lineage should be ingested.

title: Include Copy Lineage

description: Whether lineage should be collected from copy commands

title: Include Usage Statistics

description: Generate usage statistic. email_domain config parameter needs to be set if enabled

title: Include Unload Lineage

description: Whether lineage should be collected from unload commands

title: Capture Lineage Query Parser Failures

description: Whether to capture lineage query parser errors with dataset properties for debugging

description: Which table lineage collector mode to use. Available modes are: [stl_scan_based, sql_based, mixed]

title: Extra Client Options

title: Match Fully Qualified Names

description: Whether `schema_pattern` is matched against fully qualified schema name `<database>.<schema>`.

title: RedshiftUsageConfig

title: Env

title: End Time

title: Start Time

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: S3 Lineage Config

description: Common config for S3 lineage generation

title: Platform Instance Map

title: Platform Instance

title: Options

description: Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Default Schema

title: Include Table Lineage

title: Include Copy Lineage

title: Include Unload Lineage

title: Capture Lineage Query Parser Failures

description: Whether to capture lineage query parser errors with dataset properties for debuggings

title: Email Domain

title: DemoDataConfig

title: VerticaConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: Models Pattern

description: Regex patterns for ml models to filter in ingestion.

title: Include Projections

description: Whether projections should be ingested.

title: Include Models

description: Whether Models should be ingested.

title: Include View Lineage

description: If the source supports it, include view lineage to the underlying storage location.

title: Include Projection Lineage

title: RedashConfig

title: Connect Uri

description: Redash base URL.

title: Api Key

description: Redash user API key.

title: Dashboard Patterns

description: regex patterns for dashboards to filter for ingestion.

title: Chart Patterns

description: regex patterns for charts to filter for ingestion.

title: Skip Draft

description: Only ingest published dashboards and charts.

title: Page Size

description: Limit on number of items to be queried at once.

title: Api Page Limit

description: Limit on number of pages queried for ingesting dashboards and charts API during pagination.

title: Parallelism

description: Parallelism to use while processing.

title: Parse Table Names From Sql

description: See note below.

title: Sql Parser

description: custom SQL parser. See note below for details.

title: Env

description: Environment to use in namespace when constructing URNs.

title: DBTCoreConfig

title: Incremental Lineage

title: Sql Parser Use External Process

title: Env

title: Platform Instance

title: Stateful Ingestion

description: DBT Stateful Ingestion Config.

title: Target Platform

description: The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)

title: Target Platform Instance

description: The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them.

title: Use Identifiers

description: Use model identifier instead of model name if defined (if not, default to model name).

title: Entities Enabled

description: Controls for enabling / disabling metadata emission for different dbt entities (models, test definitions, test results, etc.)

title: Tag Prefix

description: Prefix added to tags during ingestion.

title: Node Name Pattern

description: regex patterns for dbt model names to filter in ingestion.

title: Meta Mapping

description: mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings.

title: Column Meta Mapping

description: mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings.

title: Query Tag Mapping

description: mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings.

title: Write Semantics

description: Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be "PATCH" or "OVERRIDE"

title: Strip User Ids From Email

description: Whether or not to strip email id while adding owners using dbt meta actions.

title: Enable Owner Extraction

description: When enabled, ownership info will be extracted from the dbt meta

title: Owner Extraction Pattern

description: Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r"(?P<owner>(.*)): (\w+) (\w+)"` will extract `jdoe` as the owner from `"jdoe: John Doe"` (2) `r"@(?P<owner>(.*))"` will extract `alice` as the owner from `"@alice"`.

title: Include Env In Assertion Guid

description: Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True.

title: Convert Column Urns To Lowercase

description: When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True.

title: Enable Meta Mapping

description: When enabled, applies the mappings that are defined through the meta_mapping directives.

title: Enable Query Tag Mapping

description: When enabled, applies the mappings that are defined through the `query_tag_mapping` directives.

title: Manifest Path

description: Path to dbt manifest JSON. See https://docs.getdbt.com/reference/artifacts/manifest-json Note this can be a local file or a URI.

title: Catalog Path

description: Path to dbt catalog JSON. See https://docs.getdbt.com/reference/artifacts/catalog-json Note this can be a local file or a URI.

title: Sources Path

description: Path to dbt sources JSON. See https://docs.getdbt.com/reference/artifacts/sources-json. If not specified, last-modified fields will not be populated. Note this can be a local file or a URI.

title: Test Results Path

description: Path to output of dbt test run as run_results file in JSON format. See https://docs.getdbt.com/reference/artifacts/run-results-json. If not specified, test execution results will not be populated in DataHub.

title: Aws Connection

description: When fetching manifest files from s3, configuration for aws connection details

title: Git Info

description: Reference to your git location to enable easy navigation from DataHub to your dbt files.

title: PrestoOnHiveConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: Host URL and port to connect to. Example: localhost:3306

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Views Where Clause Suffix

description: Where clause to specify what Presto views should be ingested.

title: Tables Where Clause Suffix

description: Where clause to specify what Hive tables should be ingested.

title: Schemas Where Clause Suffix

description: Where clause to specify what Hive schemas should be ingested.

title: Ingestion Job Id

title: Database Pattern

description: Regex patterns for hive/presto database to filter in ingestion. Specify regex to only match the database name. e.g. to match all tables in database analytics, use the regex 'analytics'

title: Metastore Db Name

description: Name of the Hive metastore's database (usually: metastore). For backward compatibility, if this field is not provided, the database field will be used. If both the 'database' and 'metastore_db_name' fields are set then the 'database' field will be used to filter the hive/presto/trino database

description: The ingested data will be stored under this platform. Valid options: ['hive', 'presto', 'presto-on-hive', 'trino']

title: Use Catalog Subtype

description: Container Subtype name to be 'Database' or 'Catalog' Valid options: ['True', 'False']

title: Use Dataset Pascalcase Subtype

description: Dataset Subtype name to be 'Table' or 'View' Valid options: ['True', 'False']

title: Include Catalog Name In Ids

description: Add the Presto catalog name (e.g. hive) to the generated dataset urns. `urn:li:dataset:(urn:li:dataPlatform:hive,hive.user.logging_events,PROD)` versus `urn:li:dataset:(urn:li:dataPlatform:hive,user.logging_events,PROD)`

title: Enable Properties Merge

description: By default, the connector overwrites properties every time. Set this to True to enable merging of properties with what exists on the server.

title: Simplify Nested Field Paths

description: Simplify v2 field paths to v1 by default. If the schema has Union or Array types, still falls back to v2

title: PowerBiDashboardSourceConfig

title: Env

title: Platform Instance

title: Stateful Ingestion

description: PowerBI Stateful Ingestion Config.

title: Tenant Id

description: PowerBI tenant identifier

title: Workspace Id Pattern

description: Regex patterns to filter PowerBI workspaces in ingestion

title: Server To Platform Instance

description: A mapping of PowerBI datasource's server i.e host[:port] to Data platform instance. :port is optional and only needed if your datasource server is running on non-standard port.For Google BigQuery the datasource's server is google bigquery project name

title: Client Id

description: Azure app client identifier

title: Client Secret

description: Azure app client secret

title: Scan Timeout

description: timeout for PowerBI metadata scanning

title: Scan Batch Size

description: batch size for sending workspace_ids to PBI, 100 is the limit

title: Workspace Id As Urn Part

description: Highly recommend changing this to True, as you can have the same workspace nameTo maintain backward compatability, this is set to False which uses workspace name

title: Extract Ownership

description: Whether ownership should be ingested. Admin API access is required if this setting is enabled. Note that enabling this may overwrite owners that you've added inside DataHub's web application.

title: Extract Reports

description: Whether reports should be ingested

title: Ownership

description: Configure how is ownership ingested

title: Modified Since

description: Get only recently modified workspaces based on modified_since datetime '2023-02-10T00:00:00.0000000Z', excludePersonalWorkspaces and excludeInActiveWorkspaces limit to last 30 days

title: Extract Dashboards

description: Whether to ingest PBI Dashboard and Tiles as Datahub Dashboard and Chart

title: Extract Dataset Schema

description: Whether to ingest PBI Dataset Table columns and measures

title: Extract Lineage

description: Whether lineage should be ingested between X and Y. Admin API access is required if this setting is enabled

title: Extract Endorsements To Tags

description: Whether to extract endorsements to tags, note that this may overwrite existing tags. Admin API access is required is this setting is enabled

title: Filter Dataset Endorsements

description: Filter and ingest datasets which are 'Certified' or 'Promoted' endorsement. If both are added, dataset which are 'Certified' or 'Promoted' will be ingested . Default setting allows all dataset to be ingested

title: Extract Workspaces To Containers

description: Extract workspaces to DataHub containers

title: Extract Datasets To Containers

description: PBI tables will be grouped under a Datahub Container, the container reflect a PBI Dataset

title: Native Query Parsing

description: Whether PowerBI native query should be parsed to extract lineage

title: Convert Urns To Lowercase

description: Whether to convert the PowerBI assets urns to lowercase

title: Convert Lineage Urns To Lowercase

description: Whether to convert the urns of ingested lineage dataset to lowercase

title: Admin Apis Only

description: Retrieve metadata using PowerBI Admin API only. If this is enabled, then Report Pages will not be extracted. Admin API access is required if this setting is enabled

title: Extract Independent Datasets

description: Whether to extract datasets not used in any PowerBI visualization

title: ElasticsearchSourceConfig

description: Any source that connects to a platform should inherit this class

title: Env

title: Platform Instance

title: Host

description: The elastic search host URI.

title: Username

description: The username credential.

title: Password

description: The password credential.

title: Use Ssl

description: Whether to use SSL for the connection or not.

title: Verify Certs

description: Whether to verify SSL certificates.

title: Ca Certs

description: Path to a certificate authority (CA) certificate.

title: Client Cert

description: Path to the file containing the private key and the certificate, or cert only if using client_key.

title: Client Key

description: Path to the file containing the private key if using separate cert and key files.

title: Ssl Assert Hostname

description: Use hostname verification if not False.

title: Ssl Assert Fingerprint

description: Verify the supplied certificate fingerprint if not None.

title: Url Prefix

description: There are cases where an enterprise would have multiple elastic search clusters. One way for them to manage is to have a single endpoint for all the elastic search clusters and use url_prefix for routing requests to different clusters.

title: Index Pattern

description: regex patterns for indexes to filter in ingestion.

title: Ingest Index Templates

description: Ingests ES index templates if enabled.

title: Index Template Pattern

description: The regex patterns for filtering index templates to ingest.

title: SnowflakeV2Config

title: Classification

description: For details, refer [Classification](../../../../metadata-ingestion/docs/dev_guides/classification.md).

title: Store Last Profiling Timestamps

title: End Time

title: Start Time

title: Store Last Usage Extraction Timestamp

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Email Domain

title: Apply View Usage To Tables

description: Whether to apply view's usage to its base tables. If set to True, usage is applied to base tables only.

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Scheme

title: Username

description: Snowflake username.

title: Password

description: Snowflake password.

title: Private Key

description: Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' if using key pair authentication. Encrypted version of private key will be in a form of '-----BEGIN ENCRYPTED PRIVATE KEY-----\nencrypted-private-key\n-----END ECNCRYPTED PRIVATE KEY-----\n' See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html

title: Private Key Path

description: The path to the private key if using key pair authentication. Ignored if `private_key` is set. See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html

title: Private Key Password

description: Password for your private key. Required if using key pair authentication with encrypted private key.

title: Oauth Config

description: oauth configuration - https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-with-oauth

title: Authentication Type

description: The type of authenticator to use when connecting to Snowflake. Supports "DEFAULT_AUTHENTICATOR", "OAUTH_AUTHENTICATOR", "EXTERNAL_BROWSER_AUTHENTICATOR" and "KEY_PAIR_AUTHENTICATOR".

title: Account Id

description: Snowflake account identifier. e.g. xy12345,  xy12345.us-east-2.aws, xy12345.us-central1.gcp, xy12345.central-us.azure, xy12345.us-west-2.privatelink. Refer [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#format-2-legacy-account-locator-in-a-region) for more details.

title: Warehouse

description: Snowflake warehouse.

title: Role

description: Snowflake role.

title: Include Table Lineage

description: If enabled, populates the snowflake table-to-table and s3-to-snowflake table lineage. Requires appropriate grants given to the role and Snowflake Enterprise Edition or above.

title: Include View Lineage

description: If enabled, populates the snowflake view->table and table->view lineages. Requires appropriate grants given to the role, and include_table_lineage to be True. view->table lineage requires Snowflake Enterprise Edition or above.

title: Connect Args

description: Connect args to pass to Snowflake SqlAlchemy driver

title: Database Pattern

title: Ignore Start Time Lineage

title: Upstream Lineage In Report

title: Convert Urns To Lowercase

title: Include Usage Stats

description: If enabled, populates the snowflake usage statistics. Requires appropriate grants given to the role.

title: Include Technical Schema

description: If enabled, populates the snowflake technical schema and descriptions.

title: Include Column Lineage

description: Populates table->table and view->table column lineage. Requires appropriate grants given to the role and the Snowflake Enterprise Edition or above.

title: Include View Column Lineage

description: Populates view->view and table->view column lineage.

description: Optional. Allowed values are `without_lineage`, `with_lineage`, and `skip` (default). `without_lineage` only extracts tags that have been applied directly to the given entity. `with_lineage` extracts both directly applied and propagated tags, but will be significantly slower. See the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/object-tagging.html#tag-lineage) for information about tag lineage/propagation.

title: Include External Url

description: Whether to populate Snowsight url for Snowflake Objects

title: Match Fully Qualified Names

description: Whether `schema_pattern` is matched against fully qualified schema name `<catalog>.<schema>`.

title: Use Legacy Lineage Method

description: Whether to use the legacy lineage computation method. By default, uses new optimised lineage extraction method that requires less ingestion process memory. Table-to-view and view-to-view column-level lineage are not supported with the legacy method.

title: Validate Upstreams Against Patterns

description: Whether to validate upstream snowflake tables against allow-deny patterns

title: Tag Pattern

description: List of regex patterns for tags to include in ingestion. Only used if `extract_tags` is enabled.

title: Temporary Tables Pattern

description: [Advanced] Regex patterns for temporary tables to filter in lineage ingestion. Specify regex to match the entire table name in database.schema.table format. Defaults are to set in such a way to ignore the temporary staging tables created by known ETL tools. Not used if `use_legacy_lineage_method=True`

title: Email As User Identifier

description: Format user urns as an email, if the snowflake user's email is set. If `email_domain` is provided, generates email addresses for snowflake users with unset emails, based on their username.

title: SupersetConfig

title: Stateful Ingestion

description: Superset Stateful Ingestion Config.

title: Connect Uri

description: Superset host URL.

title: Display Uri

description: optional URL to use in links (if `connect_uri` is only for ingestion)

title: Username

description: Superset username.

title: Password

description: Superset password.

title: Provider

description: Superset provider.

title: Options

title: Env

description: Environment to use in namespace when constructing URNs

title: Database Alias

description: Can be used to change mapping for database names in superset to what you have in datahub

title: UnityCatalogSourceConfig

title: Store Last Profiling Timestamps

title: Env

title: Platform Instance

title: End Time

title: Start Time

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Stateful Ingestion

description: Unity Catalog Stateful Ingestion Config.

title: Token

description: Databricks personal access token

title: Workspace Url

description: Databricks workspace url. e.g. https://my-workspace.cloud.databricks.com

title: Workspace Name

description: Name of the workspace. Default to deployment name present in workspace_url

title: Catalog Pattern

description: Regex patterns for catalogs to filter in ingestion. Specify regex to match the full `metastore.catalog` name.

title: Schema Pattern

description: Regex patterns for schemas to filter in ingestion. Specify regex to the full `metastore.catalog.schema` name. e.g. to match all tables in schema analytics, use the regex `^mymetastore\.mycatalog\.analytics$`.

title: Table Pattern

description: Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in `catalog.schema.table` format. e.g. to match all tables starting with customer in Customer catalog and public schema, use the regex `Customer\.public\.customer.*`.

title: Domain

description: Attach domains to catalogs, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.

title: Include Table Lineage

description: Option to enable/disable lineage generation.

title: Include Ownership

description: Option to enable/disable ownership generation for metastores, catalogs, schemas, and tables.

title: Include Column Lineage

description: Option to enable/disable lineage generation. Currently we have to call a rest call per column to get column level lineage due to the Databrick api which can slow down ingestion.

title: Include Usage Statistics

description: Generate usage statistics.

title: Profiling

description: Data profiling configuration

title: FeastRepositorySourceConfig

title: Path

description: Path to Feast repository

title: Fs Yaml File

description: Path to the `feature_store.yaml` file used to configure the feature store

title: Environment

description: Environment to use when constructing URNs

title: ModeConfig

description:
Any non-Dataset source that produces lineage to Datasets should inherit this class.
e.g. Orchestrators, Pipelines, BI Tools etc.

title: Env

title: Platform Instance Map

title: Connect Uri

description: Mode host URL.

title: Token

description: Mode user token.

title: Password

description: Mode password for authentication.

title: Workspace

title: Default Schema

description: Default schema to use when schema is not provided in an SQL query

title: Owner Username Instead Of Email

description: Use username for owner URN instead of Email

title: Api Options

description: Retry/Wait settings for Mode API to avoid "Too many Requests" error. See Mode API Options below

title: SQLAlchemyGenericConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Platform

description: Name of platform being ingested, used in constructing URNs.

title: Connect Uri

description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls

title: HanaConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

title: Database

description: database (catalog)

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: IcebergSourceConfig

title: Env

title: Platform Instance

title: Stateful Ingestion

description: Iceberg Stateful Ingestion Config.

title: Adls

description: [Azure Data Lake Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) to crawl for Iceberg tables.  This is one filesystem type supported by this source and **only one can be configured**.

title: Localfs

description: Local path to crawl for Iceberg tables. This is one filesystem type supported by this source and **only one can be configured**.

title: Max Path Depth

description: Maximum folder depth to crawl for Iceberg tables.  Folders deeper than this value will be silently ignored.

title: Table Pattern

description: Regex patterns for tables to filter in ingestion.

title: User Ownership Property

description: Iceberg table property to look for a `CorpUser` owner.  Can only hold a single user value.  If property has no value, no owner information will be emitted.

title: Group Ownership Property

description: Iceberg table property to look for a `CorpGroup` owner.  Can only hold a single group value.  If property has no value, no owner information will be emitted.

title: Profiling

title: PulsarSourceConfig

title: Env

title: Platform Instance

title: Stateful Ingestion

description: see Stateful Ingestion

title: Web Service Url

description: The web URL for the cluster.

title: Timeout

description: Timout setting, how long to wait for the Pulsar rest api to send data before giving up

title: Issuer Url

description: The complete URL for a Custom Authorization Server. Mandatory for OAuth based authentication.

title: Client Id

description: The application's client ID

title: Client Secret

description: The application's client secret

title: Token

description: The access token for the application. Mandatory for token based authentication.

title: Verify Ssl

description: Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use.

title: Tenant Patterns

description: List of regex patterns for tenants to include/exclude from ingestion. By default all tenants are allowed.

title: Namespace Patterns

description: List of regex patterns for namespaces to include/exclude from ingestion. By default the functions namespace is denied.

title: Topic Patterns

description: List of regex patterns for topics to include/exclude from ingestion. By default the Pulsar system topics are denied.

title: Exclude Individual Partitions

description: Extract each individual partitioned topic. e.g. when turned off a topic with 100 partitions will result in 100 Datasets.

title: Tenants

description: Listing all tenants requires superUser role, alternative you can set a list of tenants you want to scrape using the tenant admin role

title: Domain

description: Domain patterns

title: Oid Config

description: Placeholder for OpenId discovery document

title: DeltaLakeSourceConfig

title: Env

title: Platform Instance

title: Base Path

description: Path to table (s3 or local file system). If path is not a delta table path then all subfolders will be scanned to detect and ingest delta tables.

title: Relative Path

description: If set, delta-tables will be searched at location '<base_path>/<relative_path>' and URNs will be created using relative_path only.

title: Platform

description: The platform that this source connects to

title: Table Pattern

description: regex patterns for tables to filter in ingestion.

title: Version History Lookback

description: Number of previous version histories to be ingested. Defaults to 1. If set to -1 all version history will be ingested.

title: Require Files

description: Whether DeltaTable should track files. Consider setting this to `False` for large delta tables, resulting in significant memory reduction for ingestion process.When set to `False`, number_of_files in delta table can not be reported.

title: AzureADConfig

description: Config to create a token and connect to Azure AD instance

title: Env

title: Platform Instance

title: Stateful Ingestion

description: Azure AD Stateful Ingestion Config.

title: Client Id

description: Application ID. Found in your app registration on Azure AD Portal

title: Tenant Id

description: Directory ID. Found in your app registration on Azure AD Portal

title: Client Secret

description: Client secret. Found in your app registration on Azure AD Portal

title: Authority

description: The authority (https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-application-configuration) is a URL that indicates a directory that MSAL can request tokens from.

title: Token Url

description: The token URL that acquires a token from Azure AD for authorizing requests.  This source will only work with v1.0 endpoint.

title: Redirect

description: Redirect URI.  Found in your app registration on Azure AD Portal.

title: Graph Url

description: [Microsoft Graph API endpoint](https://docs.microsoft.com/en-us/graph/use-the-api)

title: Azure Ad Response To Username Attr

description: Which Azure AD User Response attribute to use as input to DataHub username mapping.

title: Azure Ad Response To Username Regex

description: A regex used to parse the DataHub username from the attribute specified in `azure_ad_response_to_username_attr`.

title: Azure Ad Response To Groupname Attr

description: Which Azure AD Group Response attribute to use as input to DataHub group name mapping.

title: Azure Ad Response To Groupname Regex

description: A regex used to parse the DataHub group name from the attribute specified in `azure_ad_response_to_groupname_attr`.

title: Ingest Users

title: Ingest Groups

title: Ingest Group Membership

title: Ingest Groups Users

description: This option is useful only when `ingest_users` is set to False and `ingest_group_membership` to True. As effect, only the users which belongs to the selected groups will be ingested.

title: Users Pattern

description: regex patterns for users to filter in ingestion.

title: Groups Pattern

description: regex patterns for groups to include in ingestion.

title: Filtered Tracking

description: If enabled, report will contain names of filtered users and groups.

title: Mask Group Id

description: Whether workunit ID's for groups should be masked to avoid leaking sensitive information.

title: Mask User Id

description: Whether workunit ID's for users should be masked to avoid leaking sensitive information.

title: OpenApiConfig

title: Name

title: Url

title: Swagger File

title: Ignore Endpoints

title: Username

title: Password

title: Forced Examples

title: Token

title: Get Token

title: KafkaSourceConfig

title: Env

title: Platform Instance

title: Connection

title: Topic Patterns

title: Domain

description: A map of domain names to allow deny patterns. Domains can be urn-based (`urn:li:domain:13ae4d85-d955-49fc-8474-9004c663a810`) or bare (`13ae4d85-d955-49fc-8474-9004c663a810`).

title: Topic Subject Map

description: Provides the mapping for the `key` and the `value` schemas of a topic to the corresponding schema registry subject name. Each entry of this map has the form `<topic_name>-key`:`<schema_registry_subject_name_for_key_schema>` and `<topic_name>-value`:`<schema_registry_subject_name_for_value_schema>` for the key and the value schemas associated with the topic, respectively. This parameter is mandatory when the [RecordNameStrategy](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#how-the-naming-strategies-work) is used as the subject naming strategy in the kafka schema registry. NOTE: When provided, this overrides the default subject name resolution even when the `TopicNameStrategy` or the `TopicRecordNameStrategy` are used.

title: Schema Registry Class

description: The fully qualified implementation class(custom) that implements the KafkaSchemaRegistryBase interface.

title: Ignore Warnings On Schema Type

description: Disables warnings reported for non-AVRO/Protobuf value or key schemas if set.

title: Disable Topic Record Naming Strategy

description: Disables the utilization of the TopicRecordNameStrategy for Schema Registry subjects. For more information, visit: https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#handling-differences-between-preregistered-and-client-derived-schemas:~:text=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

title: LineageFileSourceConfig

title: File

description: File path or URL to lineage file to ingest.

title: Preserve Upstream

description: Whether we want to query datahub-gms for upstream data. False means it will hard replace upstream data for a given entity. True means it will query the backend for existing upstreams and include it in the ingestion run

title: PostgresConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog). If set to Null, all databases will be considered for ingestion.

title: Database Alias

title: Scheme

description: database scheme

title: Sqlalchemy Uri

title: Database Pattern

description: Regex patterns for databases to filter in ingestion. Note: this is not used if `database` or `sqlalchemy_uri` are provided.

title: Initial Database

description: Initial database used to query for the list of databases, when ingesting multiple databases. Note: this is not used if `database` or `sqlalchemy_uri` are provided.

title: Include View Lineage

description: Include table lineage for views

title: GCSSourceConfig

title: Path Specs

description: List of PathSpec. See [below](#path-spec) the details about PathSpec

title: Env

title: Platform Instance

title: Credential

description: Google cloud storage [HMAC keys](https://cloud.google.com/storage/docs/authentication/hmackeys)

title: Max Rows

description: Maximum number of rows to use when inferring schemas for TSV and CSV files.

title: Number Of Files To Sample

description: Number of files to list to sample for schema inference. This will be ignored if sample_files is set to False in the pathspec.

title: CSVEnricherConfig

title: Filename

description: File path or URL of CSV file to ingest.

title: Write Semantics

title: Delimiter

description: Delimiter to use when parsing CSV

title: Array Delimiter

description: Delimiter to use when parsing array fields (tags, terms and owners)

title: MetabaseConfig

title: Env

title: Platform Instance Map

title: Connect Uri

description: Metabase host URL.

title: Username

description: Metabase username.

title: Password

description: Metabase password.

title: Database Alias Map

description: Database name map to use when constructing dataset URN.

title: Engine Platform Map

description: Custom mappings between metabase database engines and DataHub platforms

title: Database Id To Instance Map

description: Custom mappings between metabase database id and DataHub platform instance

title: Default Schema

description: Default schema name to use when schema is not provided in an SQL query

title: AthenaConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Scheme

title: Username

description: Username credential. If not specified, detected with boto3 rules. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html

title: Password

description: Same detection scheme as username

title: Database

description: The athena database to ingest from. If not set it will be autodetected

title: Aws Region

description: Aws region where your Athena database is located

title: Aws Role Arn

description: AWS Role arn for Pyathena to assume in its connection

title: Aws Role Assumption Duration

description: Duration to assume the AWS Role for. Maximum of 43200 (12 hours)

title: S3 Staging Dir

description: [deprecated in favor of `query_result_location`] S3 query location

title: Work Group

description: The name of your Amazon Athena Workgroups

title: Catalog Name

description: Athena Catalog Name

title: Query Result Location

description: S3 path to the [query result bucket](https://docs.aws.amazon.com/athena/latest/ug/querying.html#query-results-specify-location) which should be used by AWS Athena to store results of thequeries executed by DataHub.

title: TrinoConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: TableauConfig

title: Connect Uri

description: Tableau host URL.

title: Username

description: Tableau username, must be set if authenticating using username/password.

title: Password

description: Tableau password, must be set if authenticating using username/password.

title: Token Name

description: Tableau token name, must be set if authenticating using a personal access token.

title: Token Value

description: Tableau token value, must be set if authenticating using a personal access token.

title: Site

description: Tableau Site. Always required for Tableau Online. Use emptystring to connect with Default site on Tableau Server.

title: Platform Instance

title: Ssl Verify

description: Whether to verify SSL certificates. If using self-signed certificates, set to false or provide the path to the .pem certificate bundle.

title: Extract Column Level Lineage

description: When enabled, extracts column-level lineage from Tableau Datasources

title: Env

title: Platform Instance Map

title: Projects

description: [deprecated] Use project_pattern instead. List of tableau projects

title: Project Pattern

description: Filter for specific Tableau projects. For example, use 'My Project' to ingest a root-level Project with name 'My Project', or 'My Project/Nested Project' to ingest a nested Project with name 'Nested Project'. By default, all Projects nested inside a matching Project will be included in ingestion. You can both allow and deny projects based on their name using their name, or a Regex pattern. Deny patterns always take precedence over allow patterns. By default, all projects will be ingested.

title: Project Path Separator

description: The separator used for the project_pattern field between project names. By default, we use a slash. You can change this if your Tableau projects contain slashes in their names, and you'd like to filter by project.

title: Default Schema Map

description: Default schema to use when schema is not found.

title: Ingest Tags

description: Ingest Tags from source. This will override Tags entered from UI

title: Ingest Owner

description: Ingest Owner from source. This will override Owner info entered from UI

title: Ingest Tables External

description: Ingest details for tables external to (not embedded in) tableau as entities.

title: Page Size

description: [advanced] Number of metadata objects (e.g. CustomSQLTable, PublishedDatasource, etc) to query at a time using the Tableau API.

title: Workbook Page Size

description: [advanced] Number of workbooks to query at a time using the Tableau API.

title: Lineage Overrides

description: Mappings to change generated dataset urns. Use only if you really know what you are doing.

title: Extract Usage Stats

description: [experimental] Extract usage statistics for dashboards and charts.

title: Ingest Embed Url

description: Ingest a URL to render an embedded Preview of assets within Tableau.

title: Ingest External Links For Dashboards

description: Ingest a URL to link out to from dashboards.

title: Ingest External Links For Charts

description: Ingest a URL to link out to from charts.

title: Extract Project Hierarchy

description: Whether to extract entire project hierarchy for nested projects.

title: Extract Lineage From Unsupported Custom Sql Queries

description: [Experimental] Whether to extract lineage from unsupported custom sql queries using SQL parsing

title: SQLServerConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: MSSQL host URL.

title: Database

title: Database Alias

title: Sqlalchemy Uri

title: Use Odbc

description: See https://docs.sqlalchemy.org/en/14/dialects/mssql.html#module-sqlalchemy.dialects.mssql.pyodbc.

title: Uri Args

description: Arguments to URL-encode when connecting. See https://docs.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver15.

title: Database Pattern

title: Convert Urns To Lowercase

description: Enable to convert the SQL Server assets urns to lowercase

title: NifiSourceConfig

title: Env

title: Site Url

description: URL for Nifi, ending with /nifi/. e.g. https://mynifi.domain/nifi/

description: Nifi authentication. must be one of : NO_AUTH, SINGLE_USER, CLIENT_CERT, KERBEROS

title: Provenance Days

description: time window to analyze provenance events for external datasets

title: Process Group Pattern

description: regex patterns for filtering process groups

title: Site Name

description: Site name to identify this site with, useful when using input and output ports receiving remote connections

title: Site Url To Site Name

description: Lookup to find site_name for site_url ending with /nifi/, required if using remote process groups in nifi flow

title: Username

description: Nifi username, must be set for auth = "SINGLE_USER"

title: Password

description: Nifi password, must be set for auth = "SINGLE_USER"

title: Client Cert File

description: Path to PEM file containing the public certificates for the user/client identity, must be set for auth = "CLIENT_CERT"

title: Client Key File

description: Path to PEM file containing the client’s secret key

title: Client Key Password

description: The password to decrypt the client_key_file

title: Ca File

description: Path to PEM file containing certs for the root CA(s) for the NiFi

title: LDAPSourceConfig

description: Config used by the LDAP Source.

title: Env

title: Platform Instance

title: Ldap Server

description: LDAP server URL.

title: Ldap User

description: LDAP user.

title: Ldap Password

description: LDAP password.

title: Base Dn

description: LDAP DN.

title: Filter

description: LDAP extractor filter.

title: Attrs List

description: Retrieved attributes list

title: Custom Props List

description: A list of custom attributes to extract from the LDAP provider.

title: Drop Missing First Last Name

description: If set to true, any users without first and last names will be dropped.

title: Page Size

description: Size of each page to fetch when extracting metadata.

title: Manager Filter Enabled

description: Use LDAP extractor filter to search managers.

title: Manager Pagination Enabled

description: [deprecated] Use pagination_enabled

title: Pagination Enabled

description: Use pagination while do search query (enabled by default).

title: User Attrs Map

title: Group Attrs Map

title: DBTCloudConfig

title: Incremental Lineage

title: Sql Parser Use External Process

title: Env

title: Platform Instance

title: Stateful Ingestion

description: DBT Stateful Ingestion Config.

title: Target Platform

title: Target Platform Instance

title: Use Identifiers

title: Entities Enabled

title: Tag Prefix

description: Prefix added to tags during ingestion.

title: Node Name Pattern

title: Meta Mapping

title: Column Meta Mapping

title: Query Tag Mapping

title: Write Semantics

title: Strip User Ids From Email

title: Enable Owner Extraction

title: Owner Extraction Pattern

title: Include Env In Assertion Guid

title: Convert Column Urns To Lowercase

title: Enable Meta Mapping

title: Enable Query Tag Mapping

title: Metadata Endpoint

description: The dbt Cloud metadata API endpoint.

title: Token

description: The API token to use to authenticate with DBT Cloud.

title: Account Id

description: The DBT Cloud account ID to use.

title: Project Id

description: The dbt Cloud project ID to use.

title: Job Id

description: The ID of the job to ingest metadata from.

title: Run Id

description: The ID of the run to ingest metadata from. If not specified, we'll default to the latest run.

title: FileSourceConfig

title: Path

description: File path to folder or file to ingest, or URL to a remote file. If pointed to a folder, all files with extension {file_extension} (default json) within that folder will be processed.

title: File Extension

description: When providing a folder to use to read files, set this field to control file extensions that you want the source to process. * is a special value that means process every file regardless of extension

title: Aspect

description: Set to an aspect to only read this aspect for ingestion.

title: Count All Before Starting

description: When enabled, counts total number of records in the file before starting. Used for accurate estimation of completion time. Turn it off if startup time is too high.

title: DataLakeSourceConfig

title: Path Specs

title: Env

title: Platform Instance

title: Platform

description: The platform that this source connects to (either 's3' or 'file'). If not specified, the platform will be inferred from the path_specs.

title: Aws Config

description: AWS configuration

title: Use S3 Bucket Tags

description: Whether or not to create tags in datahub from the s3 bucket

title: Use S3 Object Tags

description: Whether or not to create tags in datahub from the s3 object

title: Profile Patterns

description: regex patterns for tables to profile

title: Profiling

description: Data profiling configuration

title: Spark Driver Memory

description: Max amount of memory to grant Spark.

title: Spark Config

description: Spark configuration properties to set on the SparkSession. Put config property names into quotes. For example: '"spark.executor.memory": "2g"'

title: Max Rows

title: Verify Ssl

title: Number Of Files To Sample

title: LookerDashboardSourceConfig

title: Env

title: Platform Instance

title: Explore Naming Pattern

title: Explore Browse Pattern

title: View Naming Pattern

title: View Browse Pattern

title: Tag Measures And Dimensions

title: Platform Name

description: Default platform name. Don't change.

title: Extract Column Level Lineage

title: Client Id

description: Looker API client id.

title: Client Secret

description: Looker API client secret.

title: Base Url

description: Url to your Looker instance: `https://company.looker.com:19999` or `https://looker.company.com`, or similar. Used for making API calls to Looker and constructing clickable dashboard and chart urls.

title: Transport Options

title: Dashboard Pattern

description: Patterns for selecting dashboard ids that are to be included

title: Chart Pattern

description: Patterns for selecting chart ids that are to be included

title: Include Deleted

description: Whether to include deleted dashboards and looks.

title: Extract Owners

description: When enabled, extracts ownership from Looker directly. When disabled, ownership is left empty for dashboards and charts.

title: Actor

description: This config is deprecated in favor of `extract_owners`. Previously, was the actor to use in ownership properties of ingested metadata.

title: Strip User Ids From Email

description: When enabled, converts Looker user emails of the form name@domain.com to urn:li:corpuser:name when assigning ownership

title: Skip Personal Folders

description: Whether to skip ingestion of dashboards in personal folders. Setting this to True will only ingest dashboards in the Shared folder space.

title: Max Threads

description: Max parallelism for Looker API calls. Defaults to cpuCount or 40

title: External Base Url

description: Optional URL to use when constructing external URLs to Looker if the `base_url` is not the correct one to use. For example, `https://looker-public.company.com`. If not provided, the external base URL will default to `base_url`.

title: Extract Usage History

description: Whether to ingest usage statistics for dashboards. Setting this to True will query looker system activity explores to fetch historical dashboard usage.

title: Extract Usage History For Interval

description: Used only if extract_usage_history is set to True. Interval to extract looker dashboard usage history for. See https://docs.looker.com/reference/filter-expressions#date_and_time.

title: Extract Embed Urls

description: Produce URLs used to render Looker Explores as Previews inside of DataHub UI. Embeds must be enabled inside of Looker to use this feature.

title: Extract Independent Looks

description: Extract looks which are not part of any Dashboard. To enable this flag the stateful_ingestion should also be enabled.

title: SalesforceConfig

title: Env

title: Platform Instance

title: Username

description: Salesforce username

title: Password

description: Password for Salesforce user

title: Consumer Key

description: Consumer key for Salesforce JSON web token access

title: Private Key

description: Private key as a string for Salesforce JSON web token access

title: Security Token

description: Security token for Salesforce username

title: Instance Url

description: Salesforce instance url. e.g. https://MyDomainName.my.salesforce.com

title: Is Sandbox

description: Connect to Sandbox instance of your Salesforce

title: Access Token

description: Access token for instance url

title: Ingest Tags

title: Object Pattern

description: Regex patterns for Salesforce objects to filter in ingestion.

title: Domain

description: Regex patterns for tables/schemas to describe domain_key domain key (domain_key can be any string like "sales".) There can be multiple domain keys specified.

title: Profiling

title: Profile Pattern

description: Regex patterns for profiles to filter in ingestion, allowed by the `object_pattern`.

title: Platform

title: HiveConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Database Pattern

title: MySQLConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: MySQL host URL.

title: Database

description: database (catalog)

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: Database Pattern

title: ClickHouseConfig

title: Env

title: Platform Instance Map

title: End Time

title: Start Time

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: ClickHouse host URL.

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Database Pattern

title: Secure

title: Protocol

title: Uri Opts

description: The part of the URI and it's used to provide additional configuration options or parameters for the database connection.

title: Include Table Lineage

title: Include Materialized Views

title: MongoDBConfig

title: Env

title: Connect Uri

description: MongoDB connection URI.

title: Username

description: MongoDB username.

title: Password

description: MongoDB password.

title: Authmechanism

description: MongoDB authentication mechanism.

title: Options

description: Additional options to pass to `pymongo.MongoClient()`.

title: Enableschemainference

description: Whether to infer schemas.

title: Schemasamplingsize

description: Number of documents to use when inferring schema size. If set to `0`, all documents will be scanned.

title: Userandomsampling

description: If documents for schema inference should be randomly selected. If `False`, documents will be selected from start.

title: Maxschemasize

description: Maximum number of fields to include in the schema.

title: Maxdocumentsize

title: Database Pattern

description: regex patterns for databases to filter in ingestion.

title: Collection Pattern

description: regex patterns for collections to filter in ingestion.

title: JsonSchemaSourceConfig

title: Env

title: Platform Instance

title: Path

description: Set this to a single file-path or a directory-path (for recursive traversal) or a remote url. e.g. https://json.schemastore.org/petstore-v1.0.json

title: Platform

description: Set this to a platform that you want all schemas to live under. e.g. schemaregistry / schemarepo etc.

title: Use Id As Base Uri

description: When enabled, uses the `$id` field in the json schema as the base uri for following references.

title: Uri Replace Pattern

description: Use this if URI-s need to be modified during reference resolution. Simple string match - replace capabilities are supported.

title: BusinessGlossarySourceConfig

title: File

description: File path or URL to business glossary file to ingest.

title: Enable Auto Id

description: Generate guid urns instead of a plaintext path urn with the node/term's hierarchy.

title: PrestoConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: RedshiftConfig

title: S3 Lineage Config

description: Common config for S3 lineage generation

title: Env

title: Platform Instance Map

title: End Time

title: Start Time

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Default Schema

title: Include Table Lineage

title: Include Copy Lineage

title: Include Unload Lineage

title: Capture Lineage Query Parser Failures

title: KafkaConnectSourceConfig

title: Env

title: Platform Instance Map

description: Platform instance mapping to use when constructing URNs. e.g.`platform_instance_map: { "hive": "warehouse" }`

title: Platform Instance

title: Connect Uri

description: URI to connect to.

title: Username

description: Kafka Connect username.

title: Password

description: Kafka Connect password.

title: Cluster Name

description: Cluster to ingest from.

title: Convert Lineage Urns To Lowercase

title: Connector Patterns

description: regex patterns for connectors to filter for ingestion.

title: Provided Configs

description: Provided Configurations

title: Connect To Platform Map

description: Platform instance mapping when multiple instances for a platform is available. Entry for a platform should be in either `platform_instance_map` or `connect_to_platform_map`. e.g.`connect_to_platform_map: { "postgres-connector-finance-db": "postgres": "core_finance_instance" }`

title: Generic Connectors

description: Provide lineage graph for sources connectors other than Confluent JDBC Source Connector, Debezium Source Connector, and Mongo Source Connector

title: SagemakerSourceConfig

description:
Common AWS credentials config.

Currently used by:
    - Glue source
    - SageMaker source

title: Aws Access Key Id

description: AWS access key ID. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.

title: Aws Secret Access Key

description: AWS secret access key. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.

title: Aws Session Token

description: AWS session token. Can be auto-detected, see [the AWS boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for details.

title: Aws Role

description: AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role

title: Aws Profile

description: Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used

title: Aws Region

description: AWS region code.

title: Aws Endpoint Url

description: The AWS service endpoint. This is normally [constructed automatically](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html), but can be overridden here.

title: Aws Proxy

description: A set of proxy configs to use with AWS. See the [botocore.config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) docs for details.

title: Env

title: Database Pattern

title: Table Pattern

title: Extract Feature Groups

description: Whether to extract feature groups.

title: Extract Models

description: Whether to extract models.

title: Extract Jobs

description: Whether to extract AutoML jobs.

title: DruidConfig

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

description: regex patterns for schemas to filter in ingestion.

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: database (catalog)

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: GlueSourceConfig

title: Aws Access Key Id

title: Aws Secret Access Key

title: Aws Session Token

title: Aws Role

title: Aws Profile

title: Aws Region

description: AWS region code.

title: Aws Endpoint Url

title: Aws Proxy

title: Env

title: Database Pattern

title: Table Pattern

title: Platform Instance

title: Platform

description: The platform to use for the dataset URNs. Must be one of ['glue', 'athena'].

title: Extract Owners

description: When enabled, extracts ownership from Glue directly and overwrites existing owners. When disabled, ownership is left empty for datasets.

title: Extract Transforms

description: Whether to extract Glue transform jobs.

title: Ignore Unsupported Connectors

description: Whether to ignore unsupported connectors. If disabled, an error will be raised.

title: Emit S3 Lineage

description: Whether to emit S3-to-Glue lineage.

title: Glue S3 Lineage Direction

description: If `upstream`, S3 is upstream to Glue. If `downstream` S3 is downstream to Glue.

title: Domain

description: regex patterns for tables to filter to assign domain_key.

title: Catalog Id

description: The aws account id where the target glue catalog lives. If None, datahub will ingest glue in aws caller's account.

title: Ignore Resource Links

description: If set to True, ignore database resource links.

title: Use S3 Bucket Tags

description: If an S3 Buckets Tags should be created for the Tables ingested by Glue. Please Note that this will not apply tags to any folders ingested, only the files.

title: Use S3 Object Tags

description: If an S3 Objects Tags should be created for the Tables ingested by Glue.

title: Profiling

description: Configs to ingest data profiles from glue table

title: ClickHouseUsageConfig

title: Env

title: End Time

title: Start Time

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Platform Instance Map

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: ClickHouse host URL.

title: Database

description: database (catalog)

title: Database Alias

title: Sqlalchemy Uri

title: Database Pattern

title: Secure

title: Protocol

title: Uri Opts

title: Include Table Lineage

title: Include Materialized Views

title: Email Domain

title: Query Log Table

title: TrinoUsageConfig

title: End Time

title: Start Time

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Env

title: Platform Instance

title: Options

title: Schema Pattern

title: Table Pattern

title: View Pattern

title: Profile Pattern

title: Domain

title: Include Views

description: Whether views should be ingested.

title: Include Tables

description: Whether tables should be ingested.

title: Include Table Location Lineage

title: Profiling

title: Username

description: username

title: Password

description: password

title: Host Port

description: host URL

title: Database

description: The name of the catalog from getting the usage

title: Database Alias

title: Sqlalchemy Uri

title: Email Domain

description: The email domain which will be appended to the users

title: Audit Catalog

description: The catalog name where the audit table can be found

title: Audit Schema

description: The schema name where the audit table can be found

title: DynamicTypedStateProviderConfig

title: Type

description: The type of the state provider to use. For DataHub use `datahub`

title: Config

description: The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).

title: StatefulStaleMetadataRemovalConfig

description: Base specialized config for Stateful Ingestion with stale metadata removal capability.

title: Enabled

description: The type of the ingestion state provider registered with datahub.

title: Remove Stale Metadata

description: Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.

title: AllowDenyPattern

description: A class to store allow deny regexes

title: Allow

description: List of regex patterns to include in ingestion

title: Deny

description: List of regex patterns to exclude from ingestion.

title: Ignorecase

description: Whether to ignore case sensitivity during pattern matching.

title: OperationConfig

title: Lower Freq Profile Enabled

description: Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.

title: Profile Day Of Week

description: Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.

title: Profile Date Of Month

description: Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.

title: GEProfilingConfig

title: Enabled

description: Whether profiling should be done.

title: Operation Config

description: Experimental feature. To specify operation configs.

title: Limit

description: Max number of documents to profile. By default, profiles all documents.

title: Offset

description: Offset in documents to profile. By default, uses no offset.

title: Report Dropped Profiles

description: Whether to report datasets or dataset columns which were not profiled. Set to `True` for debugging purposes.

title: Turn Off Expensive Profiling Metrics

description: Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.

title: Profile Table Level Only

description: Whether to perform profiling at table-level only, or include column-level profiling as well.

title: Include Field Null Count

description: Whether to profile for the number of nulls for each column.

title: Include Field Distinct Count

description: Whether to profile for the number of distinct values for each column.

title: Include Field Min Value

description: Whether to profile for the min value of numeric columns.

title: Include Field Max Value

description: Whether to profile for the max value of numeric columns.

title: Include Field Mean Value

description: Whether to profile for the mean value of numeric columns.

title: Include Field Median Value

description: Whether to profile for the median value of numeric columns.

title: Include Field Stddev Value

description: Whether to profile for the standard deviation of numeric columns.

title: Include Field Quantiles

description: Whether to profile for the quantiles of numeric columns.

title: Include Field Distinct Value Frequencies

description: Whether to profile for distinct value frequencies.

title: Include Field Histogram

description: Whether to profile for the histogram for numeric fields.

title: Include Field Sample Values

description: Whether to profile for the sample values for all columns.

title: Field Sample Values Limit

description: Upper limit for number of sample values to collect for all columns.

title: Max Number Of Fields To Profile

description: A positive integer that specifies the maximum number of columns to profile for any table. `None` implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.

title: Profile If Updated Since Days

description: Profile table only if it has been updated since these many number of days. If set to `null`, no constraint of last modified time for tables to profile. Supported only in `snowflake` and `BigQuery`.

title: Profile Table Size Limit

description: Profile tables only if their size is less then specified GBs. If set to `null`, no limit on the size of tables to profile. Supported only in `snowflake` and `BigQuery`

title: Profile Table Row Limit

description: Profile tables only if their row count is less then specified count. If set to `null`, no limit on the row count of tables to profile. Supported only in `snowflake` and `BigQuery`

title: Profile Table Row Count Estimate Only

description: Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL.

title: Max Workers

description: Number of worker threads to use for profiling. Set to 1 to disable.

title: Query Combiner Enabled

description: *This feature is still experimental and can be disabled if it causes issues.* Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.

title: Catch Exceptions

title: Partition Profiling Enabled

title: Partition Datetime

description: For partitioned datasets profile only the partition which matches the datetime or profile the latest one if not set. Only Bigquery supports this.

title: StatefulIngestionConfig

description: Basic Stateful Ingestion Specific Configuration for any source.

title: Enabled

title: MySQLConnectionConfig

title: Username

description: username

title: Password

description: password

title: Host Port

description: MySQL host URL.

title: Database

description: database (catalog)

title: Database Alias

title: Scheme

title: Sqlalchemy Uri

title: Options

title: KafkaConsumerConnectionConfig

description: Configuration class for holding connectivity information for Kafka consumers

title: Bootstrap

title: Schema Registry Url

title: Schema Registry Config

description: Extra schema registry config serialized as JSON. These options will be passed into Kafka's SchemaRegistryClient. https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html?#schemaregistryclient

title: Client Timeout Seconds

description: The request timeout used when interacting with the Kafka APIs.

title: Consumer Config

description: Extra consumer config serialized as JSON. These options will be passed into Kafka's DeserializingConsumer. See https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#deserializingconsumer and https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md .

title: BucketDuration

description: An enumeration.

title: BigQueryUsageConfig

title: End Time

title: Start Time

title: Top N Queries

title: User Email Pattern

title: Include Operational Stats

description: Whether to display operational stats.

title: Include Read Operational Stats

title: Format Sql Queries

description: Whether to format sql queries

title: Include Top N Queries

description: Whether to ingest the top_n_queries.

title: Max Query Duration

title: Apply View Usage To Tables

description: Whether to apply view's usage to its base tables. If set to False, uses sql parser and applies usage to views / tables mentioned in the query. If set to True, usage is applied to base tables only.

title: BigQueryCredential

title: Project Id

description: Project id to set the credentials

title: Private Key Id

description: Private key id

title: Private Key

description: Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n'

title: Client Email

description: Client email

title: Client Id

description: Client Id

title: Auth Uri

description: Authentication uri

title: Token Uri

description: Token uri

title: Auth Provider X509 Cert Url

description: Auth provider x509 certificate url

title: Type

description: Authentication type

title: Client X509 Cert Url

description: If not set it will be default to https://www.googleapis.com/robot/v1/metadata/x509/client_email

title: LookerNamingPattern

title: Pattern

title: GitInfo

description: A reference to a Git repository, including a deploy key that can be used to clone it.

title: Repo

description: Name of your Git repo e.g. https://github.com/datahub-project/datahub or https://gitlab.com/gitlab-org/gitlab. If organization/repo is provided, we assume it is a GitHub repo.

title: Branch

description: Branch on which your files live by default. Typically main or master. This can also be a commit hash.

title: Url Template

description: Template for generating a URL to a file in the repo e.g. '{repo_url}/blob/{branch}/{file_path}'. We can infer this for GitHub and GitLab repos, and it is otherwise required.It supports the following variables: {repo_url}, {branch}, {file_path}

title: Deploy Key File

description: A private key file that contains an ssh key that has been configured as a deploy key for this repository. Use a file where possible, else see deploy_key for a config field that accepts a raw string.

title: Deploy Key

description: A private key that contains an ssh key that has been configured as a deploy key for this repository. See deploy_key_file if you want to use a file that contains this key.

title: Repo Ssh Locator

description: The url to call `git clone` on. We infer this for github and gitlab repos, but it is required for other hosts.

title: LookerConnectionDefinition

title: Platform

title: Default Db

title: Default Schema

title: Platform Instance

title: Platform Env

description: The environment that the platform is located in. Leaving this empty will inherit defaults from the top level Looker configuration

title: TransportOptionsConfig

title: Timeout

title: Headers

title: LookerAPIConfig

title: Client Id

description: Looker API client id.

title: Client Secret

description: Looker API client secret.

title: Base Url

title: Transport Options

title: PathSpec

title: Include

description: Path to table. Name variable `{table}` is used to mark the folder with dataset. In absence of `{table}`, file level dataset will be created. Check below examples for more details.

title: Exclude

description: list of paths in glob pattern which will be excluded while scanning for the datasets

title: File Types

description: Files with extenstions specified here (subset of default value) only will be scanned to create dataset. Other files will be omitted.

title: Default Extension

description: For files without extension it will assume the specified file type. If it is not set the files without extensions will be skipped.

title: Table Name

description: Display name of the dataset.Combination of named variables from include path and strings

title: Enable Compression

description: Enable or disable processing compressed files. Currently .gz and .bz files are supported.

title: Sample Files

description: Not listing all the files but only taking a handful amount of sample file to infer the schema. File count and file size calculation will be disabled. This can affect performance significantly if enabled

title: S3LineageProviderConfig

description: Any source that produces s3 lineage from/to Datasets should inherit this class.

title: Path Specs

description: List of PathSpec. See below the details about PathSpec

title: LineageMode

description: An enumeration.

title: EmitDirective

description: A holder for directives for emission for specific types of entities

title: DBTEntitiesEnabled

description: Controls which dbt entities are going to be emitted by this source

description: Emit metadata for dbt models when set to Yes or Only

description: Emit metadata for dbt sources when set to Yes or Only

description: Emit metadata for dbt seeds when set to Yes or Only

description: Emit metadata for dbt snapshots when set to Yes or Only

description: Emit metadata for test definitions when enabled when set to Yes or Only

description: Emit metadata for test results when set to Yes or Only

title: AwsAssumeRoleConfig

title: Rolearn

description: ARN of the role to assume.

title: Externalid

description: External ID to use when assuming the role.

title: AwsConnectionConfig

description:
Common AWS credentials config.

Currently used by:
    - Glue source
    - SageMaker source
    - dbt source

title: Aws Access Key Id

title: Aws Secret Access Key

title: Aws Session Token

title: Aws Role

title: Aws Profile

title: Aws Region

description: AWS region code.

title: Aws Endpoint Url

title: Aws Proxy

title: GitReference

description: Reference to a hosted Git repository. Used to generate "view source" links.

title: Repo

title: Branch

title: Url Template

title: PrestoOnHiveConfigMode

description: An enumeration.

title: PlatformDetail

title: Platform Instance

description: DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources.

title: Env

description: The environment that all assets produced by DataHub platform ingestion source belong to

title: OwnershipMapping

title: Create Corp User

description: Whether ingest PowerBI user as Datahub Corpuser

title: Use Powerbi Email

description: Use PowerBI User email to ingest as corpuser, default is powerbi user identifier

title: Remove Email Suffix

description: Remove PowerBI User email suffix for example, @acryl.io

title: Dataset Configured By As Owner

description: Take PBI dataset configuredBy as dataset owner if exist

title: Owner Criteria

description: Need to have certain authority to qualify as owner for example ['ReadWriteReshareExplore','Owner','Admin']

title: ElasticProfiling

title: Enabled

description: Whether to enable profiling for the elastic search source.

title: Operation Config

title: CollapseUrns

title: Urns Suffix Regex

description:
List of regex patterns to remove from the name of the URN. All of the indices before removal of URNs are considered as the same dataset. These are applied in order for each URN.
        The main case where you would want to have multiple of these if the name where you are trying to remove suffix from have different formats.
        e.g. ending with -YYYY-MM-DD as well as ending -epochtime would require you to have 2 regex patterns to remove the suffixes across all URNs.

title: DynamicTypedClassifierConfig

title: Type

description: The type of the classifier to use. For DataHub,  use `datahub`

title: Config

description: The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.

title: ClassificationConfig

title: Enabled

description: Whether classification should be used to auto-detect glossary terms

title: Sample Size

description: Number of sample values used for classification.

title: Max Workers

description: Number of worker threads to use for classification. Set to 1 to disable.

title: Table Pattern

description: Regex patterns to filter tables for classification. This is used in combination with other patterns in parent config. Specify regex to match the entire table name in `database.schema.table` format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'

title: Column Pattern

description: Regex patterns to filter columns for classification. This is used in combination with other patterns in parent config. Specify regex to match the column name in `database.schema.table.column` format.

title: Info Type To Term

description: Optional mapping to provide glossary term identifier for info type

title: Classifiers

description: Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.

title: OAuthIdentityProvider

description: An enumeration.

title: OAuthConfiguration

description: Identity provider for oauth.Supported providers are microsoft and okta.

title: Authority Url

description: Authority url of your identity provider

title: Client Id

description: client id of your registered application

title: Scopes

description: scopes required to connect to snowflake

title: Use Certificate

description: Do you want to use certificate and private key to authenticate using oauth

title: Client Secret

description: client secret of the application if use_certificate = false

title: Encoded Oauth Public Key

description: base64 encoded certificate content if use_certificate = true

title: Encoded Oauth Private Key

description: base64 encoded private key content if use_certificate = true

title: TagOption

description: An enumeration.

title: UnityCatalogProfilerConfig

title: Enabled

description: Whether profiling should be done.

title: Operation Config

title: Warehouse Id

description: SQL Warehouse id, for running profiling queries.

title: Profile Table Level Only

description: Whether to perform profiling at table-level only or include column-level profiling as well.

title: Pattern

description: Regex patterns to filter tables for profiling during ingestion. Specify regex to match the `catalog.schema.table` format. Note that only tables allowed by the `table_pattern` will be considered.

title: Call Analyze

description: Whether to call ANALYZE TABLE as part of profile ingestion.If false, will ingest the results of the most recent ANALYZE TABLE call, if any.

title: Max Wait Secs

description: Maximum time to wait for an ANALYZE TABLE query to complete.

title: Max Workers

title: ModeAPIConfig

title: Retry Backoff Multiplier

description: Multiplier for exponential backoff when waiting to retry

title: Max Retry Interval

description: Maximum interval to wait when retrying

title: Max Attempts

description: Maximum number of attempts to retry before failing

title: AdlsSourceConfig

description:
Common Azure credentials config.

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python

title: Base Path

description: Base folder in hierarchical namespaces to start from.

title: Container Name

description: Azure storage account container name.

title: Account Name

description: Name of the Azure storage account.  See [Microsoft official documentation on how to create a storage account.](https://docs.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account)

title: Account Key

description: Azure storage account access key that can be used as a credential. **An account key, a SAS token or a client secret is required for authentication.**

title: Sas Token

description: Azure storage account Shared Access Signature (SAS) token that can be used as a credential. **An account key, a SAS token or a client secret is required for authentication.**

title: Client Secret

description: Azure client secret that can be used as a credential. **An account key, a SAS token or a client secret is required for authentication.**

title: Client Id

description: Azure client (Application) ID required when a `client_secret` is used as a credential.

title: Tenant Id

description: Azure tenant (Directory) ID required when a `client_secret` is used as a credential.

title: IcebergProfilingConfig

title: Enabled

description: Whether profiling should be done.

title: Include Field Null Count

title: Include Field Min Value

title: Include Field Max Value

title: Operation Config

title: S3

title: Aws Config

description: AWS configuration

title: Use S3 Bucket Tags

title: Use S3 Object Tags

description: # Whether or not to create tags in datahub from the s3 object

title: HMACKey

title: Hmac Access Id

description: Access ID

title: Hmac Access Secret

description: Secret

title: TableauLineageOverrides

title: Platform Override Map

description: A holder for platform -> platform mappings to generate correct dataset urns

title: Database Override Map

description: A holder for database -> database mappings to generate correct dataset urns

title: NifiAuthType

description: An enumeration.

title: FileReadMode

description: An enumeration.

title: DataLakeProfilerConfig

title: Enabled

description: Whether profiling should be done.

title: Operation Config

title: Profile Table Level Only

title: Max Number Of Fields To Profile

title: Include Field Null Count

title: Include Field Min Value

title: Include Field Max Value

title: Include Field Mean Value

title: Include Field Median Value

title: Include Field Stddev Value

title: Include Field Quantiles

title: Include Field Histogram

title: Include Field Sample Values

title: SalesforceAuthType

description: An enumeration.

title: SalesforceProfilingConfig

title: Enabled

description: Whether profiling should be done. Supports only table-level profiling at this stage

title: Operation Config

title: URIReplacePattern

title: Match

description: Pattern to match on uri-s as part of reference resolution. See replace field

title: Replace

description: Pattern to replace with as part of reference resolution. See match field

title: ProvidedConfig

title: Provider

title: Path Key

title: Value

title: GenericConnectorConfig

title: Connector Name

title: Source Dataset

title: Source Platform

title: GlueProfilingConfig

title: Row Count

description: The parameter name for row count in glue table.

title: Column Count

description: The parameter name for column count in glue table.

title: Unique Count

description: The parameter name for the count of unique value in a column.

title: Unique Proportion

description: The parameter name for the proportion of unique values in a column.

title: Null Count

description: The parameter name for the count of null values in a column.

title: Null Proportion

description: The parameter name for the proportion of null values in a column.

title: Min

description: The parameter name for the min value of a column.

title: Max

description: The parameter name for the max value of a column.

title: Mean

description: The parameter name for the mean value of a column.

title: Median

description: The parameter name for the median value of a column.

title: Stdev

description: The parameter name for the standard deviation of a column.

title: Partition Patterns

description: Regex patterns for filtering partitions for profile. The pattern should be a string like: "{'key':'value'}".

title: Operation Config

description: Transformer configs see at https://datahubproject.io/docs/metadata-ingestion/docs/transformer

description: Transformer type

description: Transformer config

description: sink