About Download Google Cloud Dataflow
This page describes how to use the Dataflow connector forSpanner to import, export, and modify data in SpannerGoogleSQL-dialect databases and PostgreSQL-dialect databases.Dataflow is a managed service for transforming and enrichingdata. The Dataflow connector for Spanner lets you readdata from and write data to Spanner in a Dataflowpipeline, optionally transforming or modifying the data. You can also createpipelines that transfer data between Spanner and otherGoogle Cloud products.The Dataflow connector is the recommended method for efficientlymoving data into and out of Spanner in bulk. It's also therecommended method for performing large transformations to a database which arenot supported by Partitioned DML, such as table moves and bulk deletesthat require a JOIN. When working with individual databases, there are othermethods you can use to import and export data:Use the Google Cloud console to export an individual database fromSpanner to Cloud Storage in Avroformat.Use the Google Cloud console to import a database back intoSpanner from files you exported to Cloud Storage.Use the REST API or Google Cloud CLI to run export or importjobs from Spanner to Cloud Storage and back also usingAvro format.The Dataflow connector for Spanner is part of theApache Beam Java SDK, and it provides an API for performing the previousactions. For more information about some of the concepts discussed in this page,such as PCollection objects and transforms, see the Apache Beam programmingguide.Add the connector to your Maven projectTo add the Google Cloud Dataflow connector to a Mavenproject, add the beam-sdks-java-io-google-cloud-platform Maven artifact toyour pom.xml file as a dependency.For example, assuming that your pom.xml file sets beam.version to theappropriate version number, you would add the following dependency: org.apache.beam beam-sdks-java-io-google-cloud-platform ${beam.version}Read data from SpannerTo read from Spanner, apply the SpannerIO.readtransform. Configure the read using the methods in theSpannerIO.Read class. Applying the transform returns aPCollection, where each element in the collectionrepresents an individual row
Key features of the Download Google Cloud Dataflow that are enabled through these drivers include:
- Dataflow release notes - Google Cloud
- Stream changes with Dataflow - Google Cloud
- Use Dataflow Prime - Google Cloud
- Understanding the Google Cloud Dataflow Model: An
- Using Notebooks with Google Cloud Dataflow
- DataFlow Functions in Google Cloud Functions
- What is Google Cloud Dataflow and use
Us-central1 VERSION: the version of the template that you want to useYou can use the following values: latest to use the latest version of the template, which is available in the non-dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ GCS_FILE_PATH: the Cloud Storage path that is used to store datastream events. For example: gs://bucket/path/to/data/ CLOUDSPANNER_INSTANCE: your Spanner instance. CLOUDSPANNER_DATABASE: your Spanner database. DLQ: the Cloud Storage path for the error queue directory. API To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch. POST "launch_parameter": { "jobName": "JOB_NAME", "containerSpecGcsPath": "gs://dataflow-templates-REGION_NAME/VERSION/flex/Cloud_Datastream_to_Spanner", "parameters": { "inputFilePattern": "GCS_FILE_PATH", "streamName": "STREAM_NAME" "instanceId": "CLOUDSPANNER_INSTANCE" "databaseId": "CLOUDSPANNER_DATABASE" "deadLetterQueueDirectory": "DLQ" } }} Replace the following: PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job JOB_NAME: a unique job name of your choice LOCATION: the region where you want todeploy your Dataflow job—for example, us-central1 VERSION: the version of the template that you want to useYou can use the following values: latest to use the latest version of the template, which is available in the non-dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ GCS_FILE_PATH: the Cloud Storage path that is used to store datastream events. For example: gs://bucket/path/to/data/ CLOUDSPANNER_INSTANCE: your Spanner instance. CLOUDSPANNER_DATABASE: your Spanner database. DLQ: the Cloud Storage path for the error queue directory. Template source code Java What's next Learn about Dataflow templates. See the list of Google-provided templates.
