Docs Home Pulumi IaC Pulumi ESC Pulumi Insights Pulumi Cloud Packages Tutorials
  1. Packages
  2. Airbyte Provider
  3. API Docs
  4. SourceGcs
airbyte 0.9.0 published on Wednesday, Apr 9, 2025 by airbytehq
airbytehq/terraform-provider-airbyte

airbyte.SourceGcs

Explore with Pulumi AI

airbyte 0.9.0 published on Wednesday, Apr 9, 2025 by airbytehq
airbytehq/terraform-provider-airbyte

SourceGcs Resource

Example Usage

Coming soon!

Coming soon!

Coming soon!

Coming soon!

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.airbyte.SourceGcs;
import com.pulumi.airbyte.SourceGcsArgs;
import com.pulumi.airbyte.inputs.SourceGcsConfigurationArgs;
import com.pulumi.airbyte.inputs.SourceGcsConfigurationCredentialsArgs;
import com.pulumi.airbyte.inputs.SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var mySourceGcs = new SourceGcs("mySourceGcs", SourceGcsArgs.builder()
            .configuration(SourceGcsConfigurationArgs.builder()
                .bucket("...my_bucket...")
                .credentials(SourceGcsConfigurationCredentialsArgs.builder()
                    .serviceAccountAuthentication(SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs.builder()
                        .serviceAccount("...my_service_account...")
                        .build())
                    .build())
                .start_date("2021-01-01T00:00:00.000000Z")
                .streams(SourceGcsConfigurationStreamArgs.builder()
                    .daysToSyncIfHistoryIsFull(6)
                    .format(SourceGcsConfigurationStreamFormatArgs.builder()
                        .excelFormat()
                        .jsonlFormat()
                        .build())
                    .globs("...")
                    .inputSchema("...my_input_schema...")
                    .name("...my_name...")
                    .recentNFilesToReadForSchemaDiscovery(6)
                    .schemaless(false)
                    .validationPolicy("Wait for Discover")
                    .build())
                .build())
            .definitionId("9b2af220-0694-476e-aec8-441217b74908")
            .secretId("...my_secret_id...")
            .workspaceId("4d3d2617-b7bf-4f1a-9c4e-25756fd184ac")
            .build());

    }
}
Copy
resources:
  mySourceGcs:
    type: airbyte:SourceGcs
    properties:
      configuration:
        bucket: '...my_bucket...'
        credentials:
          serviceAccountAuthentication:
            serviceAccount: '...my_service_account...'
        start_date: 2021-01-01T00:00:00.000000Z
        streams:
          - daysToSyncIfHistoryIsFull: 6
            format:
              excelFormat: {}
              jsonlFormat: {}
            globs:
              - '...'
            inputSchema: '...my_input_schema...'
            name: '...my_name...'
            recentNFilesToReadForSchemaDiscovery: 6
            schemaless: false
            validationPolicy: Wait for Discover
      definitionId: 9b2af220-0694-476e-aec8-441217b74908
      secretId: '...my_secret_id...'
      workspaceId: 4d3d2617-b7bf-4f1a-9c4e-25756fd184ac
Copy

Create SourceGcs Resource

Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.

Constructor syntax

new SourceGcs(name: string, args: SourceGcsArgs, opts?: CustomResourceOptions);
@overload
def SourceGcs(resource_name: str,
              args: SourceGcsArgs,
              opts: Optional[ResourceOptions] = None)

@overload
def SourceGcs(resource_name: str,
              opts: Optional[ResourceOptions] = None,
              configuration: Optional[SourceGcsConfigurationArgs] = None,
              workspace_id: Optional[str] = None,
              definition_id: Optional[str] = None,
              name: Optional[str] = None,
              secret_id: Optional[str] = None)
func NewSourceGcs(ctx *Context, name string, args SourceGcsArgs, opts ...ResourceOption) (*SourceGcs, error)
public SourceGcs(string name, SourceGcsArgs args, CustomResourceOptions? opts = null)
public SourceGcs(String name, SourceGcsArgs args)
public SourceGcs(String name, SourceGcsArgs args, CustomResourceOptions options)

type: airbyte:SourceGcs
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.

Parameters

name This property is required. string
The unique name of the resource.
args This property is required. SourceGcsArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
resource_name This property is required. str
The unique name of the resource.
args This property is required. SourceGcsArgs
The arguments to resource properties.
opts ResourceOptions
Bag of options to control resource's behavior.
ctx Context
Context object for the current deployment.
name This property is required. string
The unique name of the resource.
args This property is required. SourceGcsArgs
The arguments to resource properties.
opts ResourceOption
Bag of options to control resource's behavior.
name This property is required. string
The unique name of the resource.
args This property is required. SourceGcsArgs
The arguments to resource properties.
opts CustomResourceOptions
Bag of options to control resource's behavior.
name This property is required. String
The unique name of the resource.
args This property is required. SourceGcsArgs
The arguments to resource properties.
options CustomResourceOptions
Bag of options to control resource's behavior.

Constructor example

The following reference example uses placeholder values for all input properties.

var sourceGcsResource = new Airbyte.SourceGcs("sourceGcsResource", new()
{
    Configuration = new Airbyte.Inputs.SourceGcsConfigurationArgs
    {
        Bucket = "string",
        Credentials = new Airbyte.Inputs.SourceGcsConfigurationCredentialsArgs
        {
            AuthenticateViaGoogleOAuth = new Airbyte.Inputs.SourceGcsConfigurationCredentialsAuthenticateViaGoogleOAuthArgs
            {
                AccessToken = "string",
                ClientId = "string",
                ClientSecret = "string",
                RefreshToken = "string",
            },
            ServiceAccountAuthentication = new Airbyte.Inputs.SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs
            {
                ServiceAccount = "string",
            },
        },
        Streams = new[]
        {
            new Airbyte.Inputs.SourceGcsConfigurationStreamArgs
            {
                Format = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatArgs
                {
                    AvroFormat = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatAvroFormatArgs
                    {
                        DoubleAsString = false,
                    },
                    CsvFormat = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatCsvFormatArgs
                    {
                        Delimiter = "string",
                        DoubleQuote = false,
                        Encoding = "string",
                        EscapeChar = "string",
                        FalseValues = new[]
                        {
                            "string",
                        },
                        HeaderDefinition = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionArgs
                        {
                            Autogenerated = null,
                            FromCsv = null,
                            UserProvided = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionUserProvidedArgs
                            {
                                ColumnNames = new[]
                                {
                                    "string",
                                },
                            },
                        },
                        IgnoreErrorsOnFieldsMismatch = false,
                        NullValues = new[]
                        {
                            "string",
                        },
                        QuoteChar = "string",
                        SkipRowsAfterHeader = 0,
                        SkipRowsBeforeHeader = 0,
                        StringsCanBeNull = false,
                        TrueValues = new[]
                        {
                            "string",
                        },
                    },
                    ExcelFormat = null,
                    JsonlFormat = null,
                    ParquetFormat = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatParquetFormatArgs
                    {
                        DecimalAsFloat = false,
                    },
                    UnstructuredDocumentFormat = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatArgs
                    {
                        Processing = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingArgs
                        {
                            Local = null,
                            ViaApi = new Airbyte.Inputs.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiArgs
                            {
                                ApiKey = "string",
                                ApiUrl = "string",
                                Parameters = new[]
                                {
                                    new Airbyte.Inputs.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameterArgs
                                    {
                                        Name = "string",
                                        Value = "string",
                                    },
                                },
                            },
                        },
                        SkipUnprocessableFiles = false,
                        Strategy = "string",
                    },
                },
                Name = "string",
                DaysToSyncIfHistoryIsFull = 0,
                Globs = new[]
                {
                    "string",
                },
                InputSchema = "string",
                RecentNFilesToReadForSchemaDiscovery = 0,
                Schemaless = false,
                ValidationPolicy = "string",
            },
        },
        StartDate = "string",
    },
    WorkspaceId = "string",
    DefinitionId = "string",
    Name = "string",
    SecretId = "string",
});
Copy
example, err := airbyte.NewSourceGcs(ctx, "sourceGcsResource", &airbyte.SourceGcsArgs{
Configuration: &.SourceGcsConfigurationArgs{
Bucket: pulumi.String("string"),
Credentials: &.SourceGcsConfigurationCredentialsArgs{
AuthenticateViaGoogleOAuth: &.SourceGcsConfigurationCredentialsAuthenticateViaGoogleOAuthArgs{
AccessToken: pulumi.String("string"),
ClientId: pulumi.String("string"),
ClientSecret: pulumi.String("string"),
RefreshToken: pulumi.String("string"),
},
ServiceAccountAuthentication: &.SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs{
ServiceAccount: pulumi.String("string"),
},
},
Streams: .SourceGcsConfigurationStreamArray{
&.SourceGcsConfigurationStreamArgs{
Format: &.SourceGcsConfigurationStreamFormatArgs{
AvroFormat: &.SourceGcsConfigurationStreamFormatAvroFormatArgs{
DoubleAsString: pulumi.Bool(false),
},
CsvFormat: &.SourceGcsConfigurationStreamFormatCsvFormatArgs{
Delimiter: pulumi.String("string"),
DoubleQuote: pulumi.Bool(false),
Encoding: pulumi.String("string"),
EscapeChar: pulumi.String("string"),
FalseValues: pulumi.StringArray{
pulumi.String("string"),
},
HeaderDefinition: &.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionArgs{
Autogenerated: &.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionAutogeneratedArgs{
},
FromCsv: &.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionFromCsvArgs{
},
UserProvided: &.SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionUserProvidedArgs{
ColumnNames: pulumi.StringArray{
pulumi.String("string"),
},
},
},
IgnoreErrorsOnFieldsMismatch: pulumi.Bool(false),
NullValues: pulumi.StringArray{
pulumi.String("string"),
},
QuoteChar: pulumi.String("string"),
SkipRowsAfterHeader: pulumi.Float64(0),
SkipRowsBeforeHeader: pulumi.Float64(0),
StringsCanBeNull: pulumi.Bool(false),
TrueValues: pulumi.StringArray{
pulumi.String("string"),
},
},
ExcelFormat: &.SourceGcsConfigurationStreamFormatExcelFormatArgs{
},
JsonlFormat: &.SourceGcsConfigurationStreamFormatJsonlFormatArgs{
},
ParquetFormat: &.SourceGcsConfigurationStreamFormatParquetFormatArgs{
DecimalAsFloat: pulumi.Bool(false),
},
UnstructuredDocumentFormat: &.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatArgs{
Processing: &.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingArgs{
Local: &.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocalArgs{
},
ViaApi: &.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiArgs{
ApiKey: pulumi.String("string"),
ApiUrl: pulumi.String("string"),
Parameters: .SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameterArray{
&.SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameterArgs{
Name: pulumi.String("string"),
Value: pulumi.String("string"),
},
},
},
},
SkipUnprocessableFiles: pulumi.Bool(false),
Strategy: pulumi.String("string"),
},
},
Name: pulumi.String("string"),
DaysToSyncIfHistoryIsFull: pulumi.Float64(0),
Globs: pulumi.StringArray{
pulumi.String("string"),
},
InputSchema: pulumi.String("string"),
RecentNFilesToReadForSchemaDiscovery: pulumi.Float64(0),
Schemaless: pulumi.Bool(false),
ValidationPolicy: pulumi.String("string"),
},
},
StartDate: pulumi.String("string"),
},
WorkspaceId: pulumi.String("string"),
DefinitionId: pulumi.String("string"),
Name: pulumi.String("string"),
SecretId: pulumi.String("string"),
})
Copy
var sourceGcsResource = new SourceGcs("sourceGcsResource", SourceGcsArgs.builder()
    .configuration(SourceGcsConfigurationArgs.builder()
        .bucket("string")
        .credentials(SourceGcsConfigurationCredentialsArgs.builder()
            .authenticateViaGoogleOAuth(SourceGcsConfigurationCredentialsAuthenticateViaGoogleOAuthArgs.builder()
                .accessToken("string")
                .clientId("string")
                .clientSecret("string")
                .refreshToken("string")
                .build())
            .serviceAccountAuthentication(SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs.builder()
                .serviceAccount("string")
                .build())
            .build())
        .streams(SourceGcsConfigurationStreamArgs.builder()
            .format(SourceGcsConfigurationStreamFormatArgs.builder()
                .avroFormat(SourceGcsConfigurationStreamFormatAvroFormatArgs.builder()
                    .doubleAsString(false)
                    .build())
                .csvFormat(SourceGcsConfigurationStreamFormatCsvFormatArgs.builder()
                    .delimiter("string")
                    .doubleQuote(false)
                    .encoding("string")
                    .escapeChar("string")
                    .falseValues("string")
                    .headerDefinition(SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionArgs.builder()
                        .autogenerated()
                        .fromCsv()
                        .userProvided(SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionUserProvidedArgs.builder()
                            .columnNames("string")
                            .build())
                        .build())
                    .ignoreErrorsOnFieldsMismatch(false)
                    .nullValues("string")
                    .quoteChar("string")
                    .skipRowsAfterHeader(0)
                    .skipRowsBeforeHeader(0)
                    .stringsCanBeNull(false)
                    .trueValues("string")
                    .build())
                .excelFormat()
                .jsonlFormat()
                .parquetFormat(SourceGcsConfigurationStreamFormatParquetFormatArgs.builder()
                    .decimalAsFloat(false)
                    .build())
                .unstructuredDocumentFormat(SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatArgs.builder()
                    .processing(SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingArgs.builder()
                        .local()
                        .viaApi(SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiArgs.builder()
                            .apiKey("string")
                            .apiUrl("string")
                            .parameters(SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameterArgs.builder()
                                .name("string")
                                .value("string")
                                .build())
                            .build())
                        .build())
                    .skipUnprocessableFiles(false)
                    .strategy("string")
                    .build())
                .build())
            .name("string")
            .daysToSyncIfHistoryIsFull(0)
            .globs("string")
            .inputSchema("string")
            .recentNFilesToReadForSchemaDiscovery(0)
            .schemaless(false)
            .validationPolicy("string")
            .build())
        .startDate("string")
        .build())
    .workspaceId("string")
    .definitionId("string")
    .name("string")
    .secretId("string")
    .build());
Copy
source_gcs_resource = airbyte.SourceGcs("sourceGcsResource",
    configuration={
        "bucket": "string",
        "credentials": {
            "authenticate_via_google_o_auth": {
                "access_token": "string",
                "client_id": "string",
                "client_secret": "string",
                "refresh_token": "string",
            },
            "service_account_authentication": {
                "service_account": "string",
            },
        },
        "streams": [{
            "format": {
                "avro_format": {
                    "double_as_string": False,
                },
                "csv_format": {
                    "delimiter": "string",
                    "double_quote": False,
                    "encoding": "string",
                    "escape_char": "string",
                    "false_values": ["string"],
                    "header_definition": {
                        "autogenerated": {},
                        "from_csv": {},
                        "user_provided": {
                            "column_names": ["string"],
                        },
                    },
                    "ignore_errors_on_fields_mismatch": False,
                    "null_values": ["string"],
                    "quote_char": "string",
                    "skip_rows_after_header": 0,
                    "skip_rows_before_header": 0,
                    "strings_can_be_null": False,
                    "true_values": ["string"],
                },
                "excel_format": {},
                "jsonl_format": {},
                "parquet_format": {
                    "decimal_as_float": False,
                },
                "unstructured_document_format": {
                    "processing": {
                        "local": {},
                        "via_api": {
                            "api_key": "string",
                            "api_url": "string",
                            "parameters": [{
                                "name": "string",
                                "value": "string",
                            }],
                        },
                    },
                    "skip_unprocessable_files": False,
                    "strategy": "string",
                },
            },
            "name": "string",
            "days_to_sync_if_history_is_full": 0,
            "globs": ["string"],
            "input_schema": "string",
            "recent_n_files_to_read_for_schema_discovery": 0,
            "schemaless": False,
            "validation_policy": "string",
        }],
        "start_date": "string",
    },
    workspace_id="string",
    definition_id="string",
    name="string",
    secret_id="string")
Copy
const sourceGcsResource = new airbyte.SourceGcs("sourceGcsResource", {
    configuration: {
        bucket: "string",
        credentials: {
            authenticateViaGoogleOAuth: {
                accessToken: "string",
                clientId: "string",
                clientSecret: "string",
                refreshToken: "string",
            },
            serviceAccountAuthentication: {
                serviceAccount: "string",
            },
        },
        streams: [{
            format: {
                avroFormat: {
                    doubleAsString: false,
                },
                csvFormat: {
                    delimiter: "string",
                    doubleQuote: false,
                    encoding: "string",
                    escapeChar: "string",
                    falseValues: ["string"],
                    headerDefinition: {
                        autogenerated: {},
                        fromCsv: {},
                        userProvided: {
                            columnNames: ["string"],
                        },
                    },
                    ignoreErrorsOnFieldsMismatch: false,
                    nullValues: ["string"],
                    quoteChar: "string",
                    skipRowsAfterHeader: 0,
                    skipRowsBeforeHeader: 0,
                    stringsCanBeNull: false,
                    trueValues: ["string"],
                },
                excelFormat: {},
                jsonlFormat: {},
                parquetFormat: {
                    decimalAsFloat: false,
                },
                unstructuredDocumentFormat: {
                    processing: {
                        local: {},
                        viaApi: {
                            apiKey: "string",
                            apiUrl: "string",
                            parameters: [{
                                name: "string",
                                value: "string",
                            }],
                        },
                    },
                    skipUnprocessableFiles: false,
                    strategy: "string",
                },
            },
            name: "string",
            daysToSyncIfHistoryIsFull: 0,
            globs: ["string"],
            inputSchema: "string",
            recentNFilesToReadForSchemaDiscovery: 0,
            schemaless: false,
            validationPolicy: "string",
        }],
        startDate: "string",
    },
    workspaceId: "string",
    definitionId: "string",
    name: "string",
    secretId: "string",
});
Copy
type: airbyte:SourceGcs
properties:
    configuration:
        bucket: string
        credentials:
            authenticateViaGoogleOAuth:
                accessToken: string
                clientId: string
                clientSecret: string
                refreshToken: string
            serviceAccountAuthentication:
                serviceAccount: string
        startDate: string
        streams:
            - daysToSyncIfHistoryIsFull: 0
              format:
                avroFormat:
                    doubleAsString: false
                csvFormat:
                    delimiter: string
                    doubleQuote: false
                    encoding: string
                    escapeChar: string
                    falseValues:
                        - string
                    headerDefinition:
                        autogenerated: {}
                        fromCsv: {}
                        userProvided:
                            columnNames:
                                - string
                    ignoreErrorsOnFieldsMismatch: false
                    nullValues:
                        - string
                    quoteChar: string
                    skipRowsAfterHeader: 0
                    skipRowsBeforeHeader: 0
                    stringsCanBeNull: false
                    trueValues:
                        - string
                excelFormat: {}
                jsonlFormat: {}
                parquetFormat:
                    decimalAsFloat: false
                unstructuredDocumentFormat:
                    processing:
                        local: {}
                        viaApi:
                            apiKey: string
                            apiUrl: string
                            parameters:
                                - name: string
                                  value: string
                    skipUnprocessableFiles: false
                    strategy: string
              globs:
                - string
              inputSchema: string
              name: string
              recentNFilesToReadForSchemaDiscovery: 0
              schemaless: false
              validationPolicy: string
    definitionId: string
    name: string
    secretId: string
    workspaceId: string
Copy

SourceGcs Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

Inputs

In Python, inputs that are objects can be passed either as argument classes or as dictionary literals.

The SourceGcs resource accepts the following input properties:

Configuration This property is required. SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
WorkspaceId This property is required. string
DefinitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
Name string
Name of the source e.g. dev-mysql-instance.
SecretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
Configuration This property is required. SourceGcsConfigurationArgs
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
WorkspaceId This property is required. string
DefinitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
Name string
Name of the source e.g. dev-mysql-instance.
SecretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
configuration This property is required. SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
workspaceId This property is required. String
definitionId String
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name String
Name of the source e.g. dev-mysql-instance.
secretId String
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
configuration This property is required. SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
workspaceId This property is required. string
definitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name string
Name of the source e.g. dev-mysql-instance.
secretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
configuration This property is required. SourceGcsConfigurationArgs
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
workspace_id This property is required. str
definition_id str
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name str
Name of the source e.g. dev-mysql-instance.
secret_id str
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
configuration This property is required. Property Map
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
workspaceId This property is required. String
definitionId String
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name String
Name of the source e.g. dev-mysql-instance.
secretId String
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.

Outputs

All input properties are implicitly available as output properties. Additionally, the SourceGcs resource produces the following output properties:

CreatedAt double
Id string
The provider-assigned unique ID for this managed resource.
ResourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
SourceId string
SourceType string
CreatedAt float64
Id string
The provider-assigned unique ID for this managed resource.
ResourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
SourceId string
SourceType string
createdAt Double
id String
The provider-assigned unique ID for this managed resource.
resourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
sourceId String
sourceType String
createdAt number
id string
The provider-assigned unique ID for this managed resource.
resourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
sourceId string
sourceType string
created_at float
id str
The provider-assigned unique ID for this managed resource.
resource_allocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
source_id str
source_type str
createdAt Number
id String
The provider-assigned unique ID for this managed resource.
resourceAllocation Property Map
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
sourceId String
sourceType String

Look up Existing SourceGcs Resource

Get an existing SourceGcs resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

public static get(name: string, id: Input<ID>, state?: SourceGcsState, opts?: CustomResourceOptions): SourceGcs
@staticmethod
def get(resource_name: str,
        id: str,
        opts: Optional[ResourceOptions] = None,
        configuration: Optional[SourceGcsConfigurationArgs] = None,
        created_at: Optional[float] = None,
        definition_id: Optional[str] = None,
        name: Optional[str] = None,
        resource_allocation: Optional[SourceGcsResourceAllocationArgs] = None,
        secret_id: Optional[str] = None,
        source_id: Optional[str] = None,
        source_type: Optional[str] = None,
        workspace_id: Optional[str] = None) -> SourceGcs
func GetSourceGcs(ctx *Context, name string, id IDInput, state *SourceGcsState, opts ...ResourceOption) (*SourceGcs, error)
public static SourceGcs Get(string name, Input<string> id, SourceGcsState? state, CustomResourceOptions? opts = null)
public static SourceGcs get(String name, Output<String> id, SourceGcsState state, CustomResourceOptions options)
resources:  _:    type: airbyte:SourceGcs    get:      id: ${id}
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
resource_name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
name This property is required.
The unique name of the resulting resource.
id This property is required.
The unique provider ID of the resource to lookup.
state
Any extra arguments used during the lookup.
opts
A bag of options that control this resource's behavior.
The following state arguments are supported:
Configuration SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
CreatedAt double
DefinitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
Name string
Name of the source e.g. dev-mysql-instance.
ResourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
SecretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
SourceId string
SourceType string
WorkspaceId string
Configuration SourceGcsConfigurationArgs
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
CreatedAt float64
DefinitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
Name string
Name of the source e.g. dev-mysql-instance.
ResourceAllocation SourceGcsResourceAllocationArgs
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
SecretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
SourceId string
SourceType string
WorkspaceId string
configuration SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
createdAt Double
definitionId String
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name String
Name of the source e.g. dev-mysql-instance.
resourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
secretId String
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
sourceId String
sourceType String
workspaceId String
configuration SourceGcsConfiguration
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
createdAt number
definitionId string
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name string
Name of the source e.g. dev-mysql-instance.
resourceAllocation SourceGcsResourceAllocation
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
secretId string
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
sourceId string
sourceType string
workspaceId string
configuration SourceGcsConfigurationArgs
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
created_at float
definition_id str
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name str
Name of the source e.g. dev-mysql-instance.
resource_allocation SourceGcsResourceAllocationArgs
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
secret_id str
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
source_id str
source_type str
workspace_id str
configuration Property Map
NOTE: When this Spec is changed, legacyconfigtransformer.py must also be modified to uptake the changes because it is responsible for converting legacy GCS configs into file based configs using the File-Based CDK.
createdAt Number
definitionId String
The UUID of the connector definition. One of configuration.sourceType or definitionId must be provided. Requires replacement if changed.
name String
Name of the source e.g. dev-mysql-instance.
resourceAllocation Property Map
actor or actor definition specific resource requirements. if default is set, these are the requirements that should be set for ALL jobs run for this actor definition. it is overriden by the job type specific configurations. if not set, the platform will use defaults. these values will be overriden by configuration at the connection level.
secretId String
Optional secretID obtained through the public API OAuth redirect flow. Requires replacement if changed.
sourceId String
sourceType String
workspaceId String

Supporting Types

SourceGcsConfiguration
, SourceGcsConfigurationArgs

Bucket This property is required. string
Name of the GCS bucket where the file(s) exist.
Credentials This property is required. SourceGcsConfigurationCredentials
Credentials for connecting to the Google Cloud Storage API
Streams This property is required. List<SourceGcsConfigurationStream>
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
StartDate string
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
Bucket This property is required. string
Name of the GCS bucket where the file(s) exist.
Credentials This property is required. SourceGcsConfigurationCredentials
Credentials for connecting to the Google Cloud Storage API
Streams This property is required. []SourceGcsConfigurationStream
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
StartDate string
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
bucket This property is required. String
Name of the GCS bucket where the file(s) exist.
credentials This property is required. SourceGcsConfigurationCredentials
Credentials for connecting to the Google Cloud Storage API
streams This property is required. List<SourceGcsConfigurationStream>
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
startDate String
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
bucket This property is required. string
Name of the GCS bucket where the file(s) exist.
credentials This property is required. SourceGcsConfigurationCredentials
Credentials for connecting to the Google Cloud Storage API
streams This property is required. SourceGcsConfigurationStream[]
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
startDate string
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
bucket This property is required. str
Name of the GCS bucket where the file(s) exist.
credentials This property is required. SourceGcsConfigurationCredentials
Credentials for connecting to the Google Cloud Storage API
streams This property is required. Sequence[SourceGcsConfigurationStream]
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
start_date str
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.
bucket This property is required. String
Name of the GCS bucket where the file(s) exist.
credentials This property is required. Property Map
Credentials for connecting to the Google Cloud Storage API
streams This property is required. List<Property Map>
Each instance of this configuration defines a \n\nstream\n\n. Use this to define which files belong in the stream, their format, and how they should be parsed and validated. When sending data to warehouse destination such as Snowflake or BigQuery, each stream is a separate table.
startDate String
UTC date and time in the format 2017-01-25T00:00:00.000000Z. Any file modified before this date will not be replicated.

SourceGcsConfigurationCredentials
, SourceGcsConfigurationCredentialsArgs

SourceGcsConfigurationCredentialsAuthenticateViaGoogleOAuth
, SourceGcsConfigurationCredentialsAuthenticateViaGoogleOAuthArgs

AccessToken This property is required. string
Access Token
ClientId This property is required. string
Client ID
ClientSecret This property is required. string
Client Secret
RefreshToken This property is required. string
Access Token
AccessToken This property is required. string
Access Token
ClientId This property is required. string
Client ID
ClientSecret This property is required. string
Client Secret
RefreshToken This property is required. string
Access Token
accessToken This property is required. String
Access Token
clientId This property is required. String
Client ID
clientSecret This property is required. String
Client Secret
refreshToken This property is required. String
Access Token
accessToken This property is required. string
Access Token
clientId This property is required. string
Client ID
clientSecret This property is required. string
Client Secret
refreshToken This property is required. string
Access Token
access_token This property is required. str
Access Token
client_id This property is required. str
Client ID
client_secret This property is required. str
Client Secret
refresh_token This property is required. str
Access Token
accessToken This property is required. String
Access Token
clientId This property is required. String
Client ID
clientSecret This property is required. String
Client Secret
refreshToken This property is required. String
Access Token

SourceGcsConfigurationCredentialsServiceAccountAuthentication
, SourceGcsConfigurationCredentialsServiceAccountAuthenticationArgs

ServiceAccount This property is required. string
Enter your Google Cloud \n\nservice account key\n\n in JSON format
ServiceAccount This property is required. string
Enter your Google Cloud \n\nservice account key\n\n in JSON format
serviceAccount This property is required. String
Enter your Google Cloud \n\nservice account key\n\n in JSON format
serviceAccount This property is required. string
Enter your Google Cloud \n\nservice account key\n\n in JSON format
service_account This property is required. str
Enter your Google Cloud \n\nservice account key\n\n in JSON format
serviceAccount This property is required. String
Enter your Google Cloud \n\nservice account key\n\n in JSON format

SourceGcsConfigurationStream
, SourceGcsConfigurationStreamArgs

Format This property is required. SourceGcsConfigurationStreamFormat
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
Name This property is required. string
The name of the stream.
DaysToSyncIfHistoryIsFull double
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
Globs List<string>
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
InputSchema string
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
RecentNFilesToReadForSchemaDiscovery double
The number of resent files which will be used to discover the schema for this stream.
Schemaless bool
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
ValidationPolicy string
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]
Format This property is required. SourceGcsConfigurationStreamFormat
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
Name This property is required. string
The name of the stream.
DaysToSyncIfHistoryIsFull float64
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
Globs []string
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
InputSchema string
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
RecentNFilesToReadForSchemaDiscovery float64
The number of resent files which will be used to discover the schema for this stream.
Schemaless bool
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
ValidationPolicy string
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]
format This property is required. SourceGcsConfigurationStreamFormat
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
name This property is required. String
The name of the stream.
daysToSyncIfHistoryIsFull Double
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
globs List<String>
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
inputSchema String
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
recentNFilesToReadForSchemaDiscovery Double
The number of resent files which will be used to discover the schema for this stream.
schemaless Boolean
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
validationPolicy String
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]
format This property is required. SourceGcsConfigurationStreamFormat
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
name This property is required. string
The name of the stream.
daysToSyncIfHistoryIsFull number
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
globs string[]
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
inputSchema string
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
recentNFilesToReadForSchemaDiscovery number
The number of resent files which will be used to discover the schema for this stream.
schemaless boolean
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
validationPolicy string
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]
format This property is required. SourceGcsConfigurationStreamFormat
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
name This property is required. str
The name of the stream.
days_to_sync_if_history_is_full float
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
globs Sequence[str]
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
input_schema str
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
recent_n_files_to_read_for_schema_discovery float
The number of resent files which will be used to discover the schema for this stream.
schemaless bool
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
validation_policy str
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]
format This property is required. Property Map
The configuration options that are used to alter how to read incoming files that deviate from the standard formatting.
name This property is required. String
The name of the stream.
daysToSyncIfHistoryIsFull Number
When the state history of the file store is full, syncs will only read files that were last modified in the provided day range. Default: 3
globs List<String>
The pattern used to specify which files should be selected from the file system. For more information on glob pattern matching look \n\nhere\n\n.
inputSchema String
The schema that will be used to validate records extracted from the file. This will override the stream schema that is auto-detected from incoming files.
recentNFilesToReadForSchemaDiscovery Number
The number of resent files which will be used to discover the schema for this stream.
schemaless Boolean
When enabled, syncs will not validate or structure records against the stream's schema. Default: false
validationPolicy String
The name of the validation policy that dictates sync behavior when a record does not adhere to the stream schema. Default: "Emit Record"; must be one of ["Emit Record", "Skip Record", "Wait for Discover"]

SourceGcsConfigurationStreamFormat
, SourceGcsConfigurationStreamFormatArgs

SourceGcsConfigurationStreamFormatAvroFormat
, SourceGcsConfigurationStreamFormatAvroFormatArgs

DoubleAsString bool
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false
DoubleAsString bool
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false
doubleAsString Boolean
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false
doubleAsString boolean
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false
double_as_string bool
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false
doubleAsString Boolean
Whether to convert double fields to strings. This is recommended if you have decimal numbers with a high degree of precision because there can be a loss precision when handling floating point numbers. Default: false

SourceGcsConfigurationStreamFormatCsvFormat
, SourceGcsConfigurationStreamFormatCsvFormatArgs

Delimiter string
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
DoubleQuote bool
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
Encoding string
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
EscapeChar string
The character used for escaping special characters. To disallow escaping, leave this field blank.
FalseValues List<string>
A set of case-sensitive strings that should be interpreted as false values.
HeaderDefinition SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
IgnoreErrorsOnFieldsMismatch bool
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
NullValues List<string>
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
QuoteChar string
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
SkipRowsAfterHeader double
The number of rows to skip after the header row. Default: 0
SkipRowsBeforeHeader double
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
StringsCanBeNull bool
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
TrueValues List<string>
A set of case-sensitive strings that should be interpreted as true values.
Delimiter string
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
DoubleQuote bool
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
Encoding string
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
EscapeChar string
The character used for escaping special characters. To disallow escaping, leave this field blank.
FalseValues []string
A set of case-sensitive strings that should be interpreted as false values.
HeaderDefinition SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
IgnoreErrorsOnFieldsMismatch bool
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
NullValues []string
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
QuoteChar string
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
SkipRowsAfterHeader float64
The number of rows to skip after the header row. Default: 0
SkipRowsBeforeHeader float64
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
StringsCanBeNull bool
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
TrueValues []string
A set of case-sensitive strings that should be interpreted as true values.
delimiter String
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
doubleQuote Boolean
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
encoding String
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
escapeChar String
The character used for escaping special characters. To disallow escaping, leave this field blank.
falseValues List<String>
A set of case-sensitive strings that should be interpreted as false values.
headerDefinition SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
ignoreErrorsOnFieldsMismatch Boolean
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
nullValues List<String>
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
quoteChar String
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
skipRowsAfterHeader Double
The number of rows to skip after the header row. Default: 0
skipRowsBeforeHeader Double
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
stringsCanBeNull Boolean
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
trueValues List<String>
A set of case-sensitive strings that should be interpreted as true values.
delimiter string
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
doubleQuote boolean
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
encoding string
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
escapeChar string
The character used for escaping special characters. To disallow escaping, leave this field blank.
falseValues string[]
A set of case-sensitive strings that should be interpreted as false values.
headerDefinition SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
ignoreErrorsOnFieldsMismatch boolean
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
nullValues string[]
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
quoteChar string
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
skipRowsAfterHeader number
The number of rows to skip after the header row. Default: 0
skipRowsBeforeHeader number
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
stringsCanBeNull boolean
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
trueValues string[]
A set of case-sensitive strings that should be interpreted as true values.
delimiter str
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
double_quote bool
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
encoding str
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
escape_char str
The character used for escaping special characters. To disallow escaping, leave this field blank.
false_values Sequence[str]
A set of case-sensitive strings that should be interpreted as false values.
header_definition SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
ignore_errors_on_fields_mismatch bool
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
null_values Sequence[str]
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
quote_char str
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
skip_rows_after_header float
The number of rows to skip after the header row. Default: 0
skip_rows_before_header float
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
strings_can_be_null bool
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
true_values Sequence[str]
A set of case-sensitive strings that should be interpreted as true values.
delimiter String
The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'. Default: ","
doubleQuote Boolean
Whether two quotes in a quoted CSV value denote a single quote in the data. Default: true
encoding String
The character encoding of the CSV data. Leave blank to default to \n\nUTF8\n\n. See \n\nlist of python encodings\n\n for allowable options. Default: "utf8"
escapeChar String
The character used for escaping special characters. To disallow escaping, leave this field blank.
falseValues List<String>
A set of case-sensitive strings that should be interpreted as false values.
headerDefinition Property Map
How headers will be defined. User Provided assumes the CSV does not have a header row and uses the headers provided and Autogenerated assumes the CSV does not have a header row and the CDK will generate headers using for f{i} where i is the index starting from 0. Else, the default behavior is to use the header from the CSV file. If a user wants to autogenerate or provide column names for a CSV having headers, they can skip rows.
ignoreErrorsOnFieldsMismatch Boolean
Whether to ignore errors that occur when the number of fields in the CSV does not match the number of columns in the schema. Default: false
nullValues List<String>
A set of case-sensitive strings that should be interpreted as null values. For example, if the value 'NA' should be interpreted as null, enter 'NA' in this field.
quoteChar String
The character used for quoting CSV values. To disallow quoting, make this field blank. Default: """
skipRowsAfterHeader Number
The number of rows to skip after the header row. Default: 0
skipRowsBeforeHeader Number
The number of rows to skip before the header row. For example, if the header row is on the 3rd row, enter 2 in this field. Default: 0
stringsCanBeNull Boolean
Whether strings can be interpreted as null values. If true, strings that match the nullvalues set will be interpreted as null. If false, strings that match the nullvalues set will be interpreted as the string itself. Default: true
trueValues List<String>
A set of case-sensitive strings that should be interpreted as true values.

SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinition
, SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionArgs

SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionUserProvided
, SourceGcsConfigurationStreamFormatCsvFormatHeaderDefinitionUserProvidedArgs

ColumnNames This property is required. List<string>
The column names that will be used while emitting the CSV records
ColumnNames This property is required. []string
The column names that will be used while emitting the CSV records
columnNames This property is required. List<String>
The column names that will be used while emitting the CSV records
columnNames This property is required. string[]
The column names that will be used while emitting the CSV records
column_names This property is required. Sequence[str]
The column names that will be used while emitting the CSV records
columnNames This property is required. List<String>
The column names that will be used while emitting the CSV records

SourceGcsConfigurationStreamFormatParquetFormat
, SourceGcsConfigurationStreamFormatParquetFormatArgs

DecimalAsFloat bool
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false
DecimalAsFloat bool
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false
decimalAsFloat Boolean
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false
decimalAsFloat boolean
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false
decimal_as_float bool
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false
decimalAsFloat Boolean
Whether to convert decimal fields to floats. There is a loss of precision when converting decimals to floats, so this is not recommended. Default: false

SourceGcsConfigurationStreamFormatUnstructuredDocumentFormat
, SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatArgs

Processing SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
Processing configuration
SkipUnprocessableFiles bool
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
Strategy string
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]
Processing SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
Processing configuration
SkipUnprocessableFiles bool
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
Strategy string
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]
processing SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
Processing configuration
skipUnprocessableFiles Boolean
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
strategy String
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]
processing SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
Processing configuration
skipUnprocessableFiles boolean
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
strategy string
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]
processing SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
Processing configuration
skip_unprocessable_files bool
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
strategy str
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]
processing Property Map
Processing configuration
skipUnprocessableFiles Boolean
If true, skip files that cannot be parsed and pass the error message along as the absourcefileparse_error field. If false, fail the sync. Default: true
strategy String
The strategy used to parse documents. fast extracts text directly from the document which doesn't work for all files. ocr_only is more reliable, but slower. hi_res is the most reliable, but requires an API key and a hosted instance of unstructured and can't be used with local mode. See the unstructured.io documentation for more details: https://unstructured-io.github.io/unstructured/core/partition.html#partition-pdf. Default: "auto"; must be one of ["auto", "fast", "ocronly", "hires"]

SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessing
, SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingArgs

Local SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocal
Process files locally, supporting fast and ocr modes. This is the default option.
ViaApi SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.
Local SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocal
Process files locally, supporting fast and ocr modes. This is the default option.
ViaApi SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.
local SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocal
Process files locally, supporting fast and ocr modes. This is the default option.
viaApi SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.
local SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocal
Process files locally, supporting fast and ocr modes. This is the default option.
viaApi SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.
local SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingLocal
Process files locally, supporting fast and ocr modes. This is the default option.
via_api SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.
local Property Map
Process files locally, supporting fast and ocr modes. This is the default option.
viaApi Property Map
Process files via an API, using the hi_res mode. This option is useful for increased performance and accuracy, but requires an API key and a hosted instance of unstructured.

SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApi
, SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiArgs

ApiKey string
The API key to use matching the environment. Default: ""
ApiUrl string
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
Parameters List<SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter>
List of parameters send to the API
ApiKey string
The API key to use matching the environment. Default: ""
ApiUrl string
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
Parameters []SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter
List of parameters send to the API
apiKey String
The API key to use matching the environment. Default: ""
apiUrl String
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
parameters List<SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter>
List of parameters send to the API
apiKey string
The API key to use matching the environment. Default: ""
apiUrl string
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
parameters SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter[]
List of parameters send to the API
api_key str
The API key to use matching the environment. Default: ""
api_url str
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
parameters Sequence[SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter]
List of parameters send to the API
apiKey String
The API key to use matching the environment. Default: ""
apiUrl String
The URL of the unstructured API to use. Default: "https://api.unstructured.io"
parameters List<Property Map>
List of parameters send to the API

SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameter
, SourceGcsConfigurationStreamFormatUnstructuredDocumentFormatProcessingViaApiParameterArgs

Name This property is required. string
The name of the unstructured API parameter to use
Value This property is required. string
The value of the parameter
Name This property is required. string
The name of the unstructured API parameter to use
Value This property is required. string
The value of the parameter
name This property is required. String
The name of the unstructured API parameter to use
value This property is required. String
The value of the parameter
name This property is required. string
The name of the unstructured API parameter to use
value This property is required. string
The value of the parameter
name This property is required. str
The name of the unstructured API parameter to use
value This property is required. str
The value of the parameter
name This property is required. String
The name of the unstructured API parameter to use
value This property is required. String
The value of the parameter

SourceGcsResourceAllocation
, SourceGcsResourceAllocationArgs

Default SourceGcsResourceAllocationDefault
optional resource requirements to run workers (blank for unbounded allocations)
JobSpecifics List<SourceGcsResourceAllocationJobSpecific>
Default SourceGcsResourceAllocationDefault
optional resource requirements to run workers (blank for unbounded allocations)
JobSpecifics []SourceGcsResourceAllocationJobSpecific
default_ SourceGcsResourceAllocationDefault
optional resource requirements to run workers (blank for unbounded allocations)
jobSpecifics List<SourceGcsResourceAllocationJobSpecific>
default SourceGcsResourceAllocationDefault
optional resource requirements to run workers (blank for unbounded allocations)
jobSpecifics SourceGcsResourceAllocationJobSpecific[]
default SourceGcsResourceAllocationDefault
optional resource requirements to run workers (blank for unbounded allocations)
job_specifics Sequence[SourceGcsResourceAllocationJobSpecific]
default Property Map
optional resource requirements to run workers (blank for unbounded allocations)
jobSpecifics List<Property Map>

SourceGcsResourceAllocationDefault
, SourceGcsResourceAllocationDefaultArgs

SourceGcsResourceAllocationJobSpecific
, SourceGcsResourceAllocationJobSpecificArgs

JobType string
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
ResourceRequirements SourceGcsResourceAllocationJobSpecificResourceRequirements
optional resource requirements to run workers (blank for unbounded allocations)
JobType string
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
ResourceRequirements SourceGcsResourceAllocationJobSpecificResourceRequirements
optional resource requirements to run workers (blank for unbounded allocations)
jobType String
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
resourceRequirements SourceGcsResourceAllocationJobSpecificResourceRequirements
optional resource requirements to run workers (blank for unbounded allocations)
jobType string
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
resourceRequirements SourceGcsResourceAllocationJobSpecificResourceRequirements
optional resource requirements to run workers (blank for unbounded allocations)
job_type str
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
resource_requirements SourceGcsResourceAllocationJobSpecificResourceRequirements
optional resource requirements to run workers (blank for unbounded allocations)
jobType String
enum that describes the different types of jobs that the platform runs. must be one of ["getspec", "checkconnection", "discoverschema", "sync", "resetconnection", "connection_updater", "replicate"]
resourceRequirements Property Map
optional resource requirements to run workers (blank for unbounded allocations)

SourceGcsResourceAllocationJobSpecificResourceRequirements
, SourceGcsResourceAllocationJobSpecificResourceRequirementsArgs

Import

$ pulumi import airbyte:index/sourceGcs:SourceGcs my_airbyte_source_gcs ""
Copy

To learn more about importing existing cloud resources, see Importing resources.

Package Details

Repository
airbyte airbytehq/terraform-provider-airbyte
License
Notes
This Pulumi package is based on the airbyte Terraform Provider.
airbyte 0.9.0 published on Wednesday, Apr 9, 2025 by airbytehq
airbytehq/terraform-provider-airbyte