容量削減のためインデックス自動削除に curator を導入することにした。
Amazon Elasticsearch Service のバージョンは「6.2.3」。EC2 インスタンスの AMI は「Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-07ad4b1c3af1ea214」。
$ curl -sq 'https://search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com'
{
"name" : "_XXXXXX",
"cluster_name" : "XXXXXXXXXXXX:XXXX-XXXX",
"cluster_uuid" : "XXXXXXXXXXXXXXXXXXXXXX",
"version" : {
"number" : "6.2.3",
"build_hash" : "XXXXXXX",
"build_date" : "2018-07-16T10:43:54.041989Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
$ cat /etc/ec2_version
Ubuntu 18.04.1 LTS (Bionic Beaver)
標準の APT でインストール出来る elasticsearch-curator はバージョンが 5.2.0 で、Amazon Elasticsearch Service 6.2.3 とは互換性エラーが発生するため 5.6.0 をインストールする必要があった。
DEB パッケージからインストールする場合
$ wget 'https://packages.elastic.co/curator/5/debian9/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.6.0_amd64.deb' $ sudo dpkg -i elasticsearch-curator_5.6.0_amd64.deb
リポジトリを登録してインストールする場合
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - $ echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main' | sudo tee /etc/apt/sources.list.d/curator.list deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main $ sudo apt-get update $ sudo apt-get install elasticsearch-curator
pip3 でインストールする場合
pip3 でも 5.6.0 がインストール可能だったので試してみたところ特に問題なかった。
$ pip3 install elasticsearch-curator
curator_cli でのテスト
curator_cli でひとまずインデックスが拾えるかどうかテストする。
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...
Options:
--config PATH Path to configuration file. Default:
~/.curator/curator.yml
--host TEXT Elasticsearch host.
--url_prefix TEXT Elasticsearch http url prefix.
--port TEXT Elasticsearch port.
--use_ssl Connect to Elasticsearch through SSL.
--certificate TEXT Path to certificate to use for SSL validation.
--client-cert TEXT Path to file containing SSL certificate for client auth.
--client-key TEXT Path to file containing SSL key for client auth.
--ssl-no-validate Do not validate SSL certificate
--http_auth TEXT Use Basic Authentication ex: user:pass
--timeout INTEGER Connection timeout in seconds.
--master-only Only operate on elected master node.
--dry-run Do not perform any changes.
--loglevel TEXT Log level
--logfile TEXT log file
--logformat TEXT Log output format [default|logstash|json].
--version Show the version and exit.
--help Show this message and exit.
Commands:
alias Add/Remove Indices to/from Alias
allocation Shard Routing Allocation
close Close Indices
delete_indices Delete Indices
delete_snapshots Delete Snapshots
forcemerge forceMerge Indices (reduce segment count)
open Open Indices
replicas Change Replica Count
restore Restore Indices
rollover Rollover Index associated with Alias
show_indices Show Indices
show_snapshots Show Snapshots
shrink Shrink Indices to --number_of_shards
snapshot Snapshot Indices
Amazon Elasticsearch Service は 9200 番ポートではなく 443 番ポートだが --host https:// であれば --port 443 と --use_ssl の指定が無くてもいいらしい。
$ curator_cli --host https://search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com show_indices .kibana : : $ curator_cli --host search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com --port 443 --use_ssl show_indices .kibana : :
curator でテスト
~/.curator/curator.yml を作成。テストなので項目は最低限で済ませる。
client: hosts: - search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443 use_ssl: True
~/.curator/delete_indices.yml を作成。logstash- で始まる日付ベースのインデックスを1日分残して削除する。
actions: 1: action: delete_indices description: delete old logstash indexes. options: disable_action: False ignore_empty_list: True filters: - filtertype: pattern kind: prefix value: logstash- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 1
テストなので --dry-run で実行。今日は 2018-11-21 なので 2018-11-20 のインデックスが削除される。
$ curator --dry-run ~/.curator/delete_indices.yml
2018-11-21 22:20:39,436 INFO Preparing Action ID: 1, "delete_indices"
2018-11-21 22:20:39,494 INFO Trying Action ID: 1, "delete_indices": delete old logstash indexes.
2018-11-21 22:20:39,600 INFO DRY-RUN MODE. No changes will be made.
2018-11-21 22:20:39,600 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018-11-21 22:20:39,600 INFO DRY-RUN: delete_indices: logstash-2018.11.20 with arguments: {}
2018-11-21 22:20:39,600 INFO Action ID: 1, "delete_indices" completed.
2018-11-21 22:20:39,600 INFO Job completed.
あとは保持期間を本来の運用に合わせて cron に登録しておく。
以下、elasticsearch-curator 5.2.0 での検証結果。
標準のリポジトリから elasticsearch-curator パッケージをインストール。
$ sudo apt-get install elasticsearch-curator $ curator_cli --version curator_cli, version 5.2.0
互換性エラーが発生した。
$ curator_cli show_indices
2018-11-21 20:33:05,764 ERROR Elasticsearch version 6.2.3 incompatible with this version of Curator (5.2.0)
Traceback (most recent call last):
File "/usr/bin/curator_cli", line 11, in <module>
load_entry_point('elasticsearch-curator==5.2.0', 'console_scripts', 'curator_cli')()
File "/usr/lib/python2.7/dist-packages/curator/curator_cli.py", line 5, in main
cli(obj={})
File "/usr/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/curator/singletons.py", line 531, in show_indices_singleton
client = get_client(**c_args)
File "/usr/lib/python2.7/dist-packages/curator/utils.py", line 800, in get_client
'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: Elasticsearch version 6.2.3 incompatible with this version of Curator (5.2.0)
Version Compatibility | Curator Reference [5.2] | Elastic を確認してみると Curator 5.2 は ES 5.x 系までの記載しかなかった。
2018-11-21 時点の現行バージョンは 5.6.0 なのでそっちならいけるかと思い確認してみと注釈に AWS ES (which is different from installing Elasticsearch on your own EC2 instances) version 5.3 officially supports Curator.
とは書いてあるが AWS ES 6.2 に関する記載は無かった。
