Sign up and create a project in Google Console <\/a><\/h4>\n<\/div>\n\n\n\nSign up for Google Cloud <\/a>account using your Gmail account, if you would like to try then sign up for a trial account. You need to add your credit card even for trial account. Google provides $300 initial credit for your trial account. This credit is sufficient for you to try out most of the services in Google Cloud.<\/p>\n<\/div>\n\n\n <\/p>\n
![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step1\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step1\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step1\/003.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Setup Google Storage <\/a><\/h4>\n<\/div>\n\n\n\nAfter finishing the sign up for Google Cloud, the first step is to create a bucket (Google equivalent for folder) in Google Cloud Storage. This is where users will be uploading their files for processing. The bucket name should be unique across Google Cloud Storage because the bucket and files inside it can be shared via HTTP url with others. Buckets cannot be nested. Follow the instructions on the screenshots and create your first bucket.<\/p>\n<\/div>\n
\n\n <\/p>\n
![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step2\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step2\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step2\/003.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Get data for testing <\/a><\/h4>\n<\/div>\n\n\n\nThis was my biggest challenge. Where do I find a huge amount of data for my testing? Luckily Wikipedia shares its data. You can get its data from http:\/\/dumps.wikimedia.org. I have prepared two test files to help you test. One is a small file with 30 records<\/a> and another is a big file with 1 million records<\/a>. I used page view log file for my testing. A typical page view file will have over 5 million records in it.<\/p>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step3\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step3\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step3\/003.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Upload file to Google Cloud Storage <\/a><\/h4>\n<\/div>\n\n\n\nUploading a file to Google Cloud Storage is simple. You can click on UPLOAD FILES button and upload file just like how you upload files to other websites.<\/p>\n<\/div>\n
\n\n <\/p>\n
![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step4\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step4\/002.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Setup BigQuery <\/a><\/h4>\n<\/div>\n\n\n\nNavigate to BigQuery console by clicking it on the left side menu. In BigQuery the first thing to do is to create a dataset which is a container for tables. Click on the + sign next to the project and create a new dataset from the menu option.<\/p>\n<\/div>\n
\n\n\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step5\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step5\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step5\/003.png\")
<\/a><\/div>\n\u00a0![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step5\/004.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Import small size test data <\/a><\/h4>\n<\/div>\n\n\n\nWe can create a table by the import process. Click on + sign next to dataset and choose create table and go thru the steps in the screenshot to import the data from small file into the new table. Click on the Job History to see the job status about the import process.<\/p>\n<\/div>\n
<\/div>\n\n\n <\/p>\n
\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/003.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/004.png\")
<\/a><\/div>\n<\/div>\n\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/005.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step6\/006.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n Import large size test data <\/a><\/h4>\n<\/div>\n\n\n\nNow do the same kind of processing for the big file with 1 million records. Since we expect to encounter some errors while importing rows we allow up to 100,000 rows to fail in the import process. Once the data is imported run a simple sql like query to get the result you want. Always check the validator before running the query to get an estimated data volume to be used for the query. The higher the data the higher the billing so choose only the columns you need.<\/p>\n<\/div>\n
<\/div>\n\n\n <\/p>\n
\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/001.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/002.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/003.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/004.png\")
<\/a><\/div>\n<\/div>\n\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/005.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/006.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/007.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/008.png\")
<\/a><\/div>\n<\/div>\n\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/009.png\")
<\/a><\/div>\n![\"\"](\"http:\/\/old.hthirukkumaran.com\/photos\/bq\/step7\/010.png\")
<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\nCredits and References<\/h3>\n<\/div>\n\n\nCredits and References<\/h3>\n<\/div>\n\n\n- Cost based on bucket types – Pricing details<\/a><\/li>\n
- Geographic Location of Buckets – Geo locations<\/a><\/li>\n
- BigQuery SQL Reference – https:\/\/cloud.google.com\/bigquery\/query-reference<\/a><\/li>\n
- BiQuery Pricing Reference – https:\/\/cloud.google.com\/bigquery\/pricing<\/a><\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
I have been using Google BigQuery for over 18 months. It is unbelievably simple to use BigQuery for your BigData needs. In this post I will show how to load a data from file to a BigQuery table via the GUI tool provided by Google. In the next post I will show how to design […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts\/707"}],"collection":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/comments?post=707"}],"version-history":[{"count":0,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts\/707\/revisions"}],"wp:attachment":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/media?parent=707"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/categories?post=707"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/tags?post=707"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Sign up for Google Cloud <\/a>account using your Gmail account, if you would like to try then sign up for a trial account. You need to add your credit card even for trial account. Google provides $300 initial credit for your trial account. This credit is sufficient for you to try out most of the services in Google Cloud.<\/p>\n<\/div>\n <\/p>\n After finishing the sign up for Google Cloud, the first step is to create a bucket (Google equivalent for folder) in Google Cloud Storage. This is where users will be uploading their files for processing. The bucket name should be unique across Google Cloud Storage because the bucket and files inside it can be shared via HTTP url with others. Buckets cannot be nested. Follow the instructions on the screenshots and create your first bucket.<\/p>\n<\/div>\n <\/p>\n This was my biggest challenge. Where do I find a huge amount of data for my testing? Luckily Wikipedia shares its data. You can get its data from http:\/\/dumps.wikimedia.org. I have prepared two test files to help you test. One is a small file with 30 records<\/a> and another is a big file with 1 million records<\/a>. I used page view log file for my testing. A typical page view file will have over 5 million records in it.<\/p>\n Uploading a file to Google Cloud Storage is simple. You can click on UPLOAD FILES button and upload file just like how you upload files to other websites.<\/p>\n<\/div>\n <\/p>\n Navigate to BigQuery console by clicking it on the left side menu. In BigQuery the first thing to do is to create a dataset which is a container for tables. Click on the + sign next to the project and create a new dataset from the menu option.<\/p>\n<\/div>\n We can create a table by the import process. Click on + sign next to dataset and choose create table and go thru the steps in the screenshot to import the data from small file into the new table. Click on the Job History to see the job status about the import process.<\/p>\n<\/div>\n <\/p>\n Now do the same kind of processing for the big file with 1 million records. Since we expect to encounter some errors while importing rows we allow up to 100,000 rows to fail in the import process. Once the data is imported run a simple sql like query to get the result you want. Always check the validator before running the query to get an estimated data volume to be used for the query. The higher the data the higher the billing so choose only the columns you need.<\/p>\n<\/div>\n <\/p>\n I have been using Google BigQuery for over 18 months. It is unbelievably simple to use BigQuery for your BigData needs. In this post I will show how to load a data from file to a BigQuery table via the GUI tool provided by Google. In the next post I will show how to design […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts\/707"}],"collection":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/comments?post=707"}],"version-history":[{"count":0,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/posts\/707\/revisions"}],"wp:attachment":[{"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/media?parent=707"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/categories?post=707"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hthirukkumaran.com\/index.php\/wp-json\/wp\/v2\/tags?post=707"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Setup Google Storage <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Get data for testing <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n Upload file to Google Cloud Storage <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Setup BigQuery <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Import small size test data <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n Import large size test data <\/a><\/h4>\n<\/div>\n
<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/a><\/div>\n<\/div>\n<\/a><\/div>\n<\/a><\/div>\nCredits and References<\/h3>\n<\/div>\n
Credits and References<\/h3>\n<\/div>\n
\n