Google Cloud Provides a ready made script to run scheduled export from datastore.
I am using that export data as input to BigQuery.
To get it done, I had created two crons,
Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.
Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.
So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.
Here is gist
I am using that export data as input to BigQuery.
To get it done, I had created two crons,
- First for calling data export operation
- Second for loading that exported data to BigQuery
Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.
Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.
So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.
Here is gist
I liked your work and, as a result, the manner you presented this content about Philippines Import Data.It is a valuable paper for us. Thank you for sharing this blog with us.
ReplyDeleteIt is truly a well-researched content and excellent wording. I got so engaged in this material that I couldn’t wait to read. Read more info about Mexico Import Data. I am impressed with your work and skill. Thanks.
ReplyDelete