Feivel

Hive – Get All Tables and Column List!

Cheat Code So we have Hive, and we love Hive.There are so much good about Hive. Enough Hive! Anyway, on one beautiful morning somebody came to me and said “Hey bro, this table on hive is broken. The field name and content doesn’t seems matching?”. I said “WHAT!?” So after long drama, we found ourContinue reading “Hive – Get All Tables and Column List!”

Your Hive Query Run Slow? This cheat make it faster!

Cheat CodeANALYZE TABLE mydb.mySuperPartitionedTable PARTITION (filedate) COMPUTE STATISTICS; ANALYZE TABLE mydb.mySuperBigFlatTable COMPUTE STATISTICS; I have couple queries running bloody slow, I mean REALLY darn slow!I ran over ODBC, JDBC, Beeline,… what ever you name it.. slow!then I have a look at the execution (oh it’s LLAP + TEZ should be freaking fast, right!?).I found strangeContinue reading “Your Hive Query Run Slow? This cheat make it faster!”

Azure Data Factory working with “UCS-2 LE BOM” a.k.a “UTF-16LE with Signature” a.k.a “UTF-16” a.k.a “Windows Unicode”

Cheat CodeSet Encoding to “UTF-16” So we received this file on our blob and we need to convert it into ORC file for our hive to eat it fast n furious. Using the battle proven Azure Data Factory (ADF) we spin quickly a pipeline to read the blob data which is zipped and have tabContinue reading “Azure Data Factory working with “UCS-2 LE BOM” a.k.a “UTF-16LE with Signature” a.k.a “UTF-16” a.k.a “Windows Unicode””

SQL Server – Rename column via T-SQL

Cheat CodeEXEC sp_rename ‘[dbo].[TABLE_NAME].[OLD_COLUMN_NAME]’,’NEW_COLUMN_NAME’, ‘COLUMN’; I just received a set of tables in SQL Server which was loaded from CSV file without column header on 1st row. They provide separate EXCEL file which define the column name.You can imagine that all fields are like “[Column01]”, “[Column02]”, etc. Good luck to our Data Analyst writing “SELECTContinue reading “SQL Server – Rename column via T-SQL”

Run HIVE HQL script on Beeline with Parameter

Cheat Codebeeline -u ‘jdbc:hive2://zk0-mycluster.mydomainads.onmicrosoft.com:2181,zk2-mycluster.mydomainads.onmicrosoft.com:2181,zk3-mycluster.mydomainads.onmicrosoft.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2’ –hivevar filedate=’2019-04-10′ –hivevar dwhdatabase=’dw_reporting_global’ –hivevar stagingdatabase=’dw_reproting_local’ Reuse, reuse, reuse!I think this resonate very well with where we are today. Developer love to develop, but if you keep redeveloping e.g. rewriting same script again and again just for the sake of “clocking # of lines” then it’s a foolish!In programming world weContinue reading “Run HIVE HQL script on Beeline with Parameter”

List all partitions from Hive Table

Cheat Code> show partitions database_name.table_name I was asked to list all partitions for the transaction hive table because somehow the number are not complete, so I ran the above cheat code to check all date partitions on my transaction tables.Simple right!? Once we found the missing partitions, we can just simple reload it from ourContinue reading “List all partitions from Hive Table”

Finding SQL Server Object Dependency

Cheat CodeSELECT * FROM sys.sql_expression_dependencies This is continuation from previous blog post about my “special project” that requires us to find there the heck some object are depends on and who use them.Using the cheat code above is a fantastic breeze to find how things are wired.Give it a kick and you will find aContinue reading “Finding SQL Server Object Dependency”

Insert Overwrite Hive Partitioned Table

Cheat Code INSERT OVERWRITE TABLE apac.amazing_report partition (yyyymm=’2019-12-01′) SELECT year, month, partnerid, cost, revenue FROM apac.amazing_csvWHERE yyyymm=’2019-12-01′ I have hive partitioned table and I need to load/reload this partition from raw source which is CSV in my case. Just run the cheat code above (don’t forget to adjust the table, partition, and source to yourContinue reading “Insert Overwrite Hive Partitioned Table”

How to check Network MTU

Cheat Codeping http://www.google.com -f -l 1472 So we have issue with network connectivity that intermittently experiencing slowness (God, no one love slow network!). After checking around we found that likely it’s due to MTU settings not optimized. First is we need to check what’s the MTU set by our router (usually it’s 1500 btw). SoContinue reading “How to check Network MTU”

When Browsing is rewarding and safe!

Cheat CodeBrave Browser Long time ago (like 15 years ago) I was obsessed and thinking how can we earn money while browsing internet. It was days where people are running multiple pop-up and installing various application which promise users that they will get money by browsing internet. In reality, that never happened! Keep browsing andContinue reading “When Browsing is rewarding and safe!”