Hive Provides a strict pattern , It can prevent users from executing “ High-risk ” Query for .
By setting properties hive.mapred.mode The value is the default non-strict mode nonstrict . Turning strict mode on requires modification hive.mapred.mode The value is strict, Turning strict mode on can be disabled 3 Types of queries .
<property> <name>hive.mapred.mode</name> <value>strict</value> <description> The mode in which the Hive operations are being performed. In strict mode, some risky queries are not allowed to run. They include: Cartesian Product. No partition being picked up for a query. Comparing bigints and strings. Comparing bigints and doubles. Orderby without limit. </description> </property>
<1> For partitioned tables , Users are not allowed to scan all partitions , Unless where The statement contains a partition field filter condition to limit the scope , Otherwise, execution is not allowed . The reason for this restriction is , Partitioned tables typically have very large data sets , And the Numbers are growing fast . Queries that are not partitioned can consume unacceptably large resources to process this table .
<2> For the order by Query of statement , Be required to use limit sentence . because order by In order to perform the sorting process all the resulting data is distributed to the same Reducer Intermediate processing , Force the user to add this LIMIT Statements prevent Reducer Extra execution for a long time .
<3> Queries that restrict cartesian products . Users who know a lot about relational databases might expect to execute JOIN Not used when querying ON Statement instead USES where sentence , This allows the relational database's execution optimizer to efficiently WHERE Convert the statement to that ON sentence . Unfortunately ,Hive This optimization is not performed , therefore , If the watch is big enough , Then the query will get out of control .
That's all for this sharing , Those who benefit from it should pay attention to it ٩(๑>◡<๑)۶
Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .