jq: uniqueとcount(*) group by

Apache Hiveの監査のwikiには書いてありませんでしたが Apache Hadoop の監査ログは大部分 JSON Lines 形式のようです。このフォーマットをjqで扱うオプションについて先日書きました。

jq –slurp . <<< $jsonlinedata

jqでJSON Lines(*.jsonl)

今日はさらに uniq してみます

重複を削除してユニークな(一意の)値を取得できます。

jq コマンドを使う日常のご紹介 – Qiita

$ cat /var/log/hive/audit/*HiveExecHookContext* |
  jq --slurp '[.[].operation]|unique'
[
  "ALTERTABLE_ADDPARTS",
  "ALTERTABLE_DROPPARTS",
  "ALTERTABLE_RENAMEPART",
  "CREATETABLE",
  "CREATEVIEW",
  "DESCDATABASE",
  "DESCTABLE",
  "DROPTABLE",
  "DROPVIEW",
  "MSCK",
  "QUERY",
  "SHOWPARTITIONS",
  "SHOWTABLES",
  "SHOW_CREATETABLE",
  "SHOW_ROLES",
  "SWITCHDATABASE",
  "TRUNCATETABLE"
]

さらにSELECT OPERATION, COUNT(*) GROUP BY OPERATIONしてみました。

$ cat /var/log/hive/audit/*HiveExec* |
  jq --slurp -rc \
 'group_by(.operation)[]|{operation:.[0].operation,length:length}'
{"operation":"ALTERTABLE_ADDPARTS","length":21}
{"operation":"ALTERTABLE_DROPPARTS","length":33}
{"operation":"ALTERTABLE_RENAMEPART","length":3}
{"operation":"CREATETABLE","length":203}
{"operation":"CREATEVIEW","length":126}
{"operation":"DESCDATABASE","length":3}
{"operation":"DESCTABLE","length":146}
{"operation":"DROPTABLE","length":54}
{"operation":"DROPVIEW","length":30}
{"operation":"MSCK","length":173}
{"operation":"QUERY","length":5741}
{"operation":"SHOWPARTITIONS","length":27}
{"operation":"SHOWTABLES","length":114}
{"operation":"SHOW_CREATETABLE","length":47}
{"operation":"SHOW_ROLES","length":1}
{"operation":"SWITCHDATABASE","length":314}
{"operation":"TRUNCATETABLE","length":5}

やはり”QUERY”==select .. FROM.. が多いですね。

Cloudera のドキュメントでは監査証跡を取れる操作一覧は以下に書いてあります。

HiveServer2 /Beeline

Service Audit Events | 5.16.x | Cloudera Documentation

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s