Removing Old Indices in Elasticsearch

Removing old indices in Elasticsearch is important for managing disk space and maintaining optimal performance. Here are several methods to delete old indices in Elasticsearch:

1. Using the Elasticsearch REST API

Delete a Specific Index

To delete a specific index, you can use the DELETE HTTP method with the index name.

DELETE /index_name

Example using curl:

curl -X DELETE "localhost:9200/index_name?pretty"

Replace index_name with the name of the index you want to delete.

Delete Indices with a Date Pattern

If you have indices with a date pattern (e.g., logs-2023-01-01), you can delete indices that match a specific pattern.

To delete indices that match a pattern, you can use wildcards:

DELETE /logs-2023-01-*

Example using curl:

curl -X DELETE "localhost:9200/logs-2023-01-*?pretty"

Delete Indices Based on Age or Size Using Index Lifecycle Management (ILM)

You can automate index deletion based on age or size using Index Lifecycle Management (ILM). This is useful for managing large volumes of data.

1. Define an ILM Policy: Create an ILM policy that specifies when to delete indices.

    

   Copied!

   PUT /_ilm/policy/delete-old-indices
   {
     "phases": {
       "hot": {
         "actions": {}
       },
       "delete": {
         "min_age": "30d",  // Age after which to delete indices
         "actions": {
           "delete": {}
         }
       }
     }
   }
   
   

2. Apply the ILM Policy to Indices: Set the ILM policy for your indices.

    

   Copied!

   PUT /index_name/_settings
   {
     "index.lifecycle.name": "delete-old-indices"
   }
   
   

3. Verify ILM Policy: Check if the ILM policy is applied and review its status.

    

   Copied!

   GET /_ilm/policy
   

2. Using Kibana Dev Tools

1. Open Kibana: Go to the Dev Tools section in Kibana.

2. Execute DELETE Command: Run the DELETE command for the indices you want to remove.

    

   Copied!

   DELETE /index_name
   

   To delete multiple indices matching a pattern:

    

   Copied!

   DELETE /logs-2023-01-*
   

3. Confirm Deletion: Check the status to ensure indices have been deleted.

    

   Copied!

   GET /_cat/indices?v
   

3. Using Curator (Optional)

Elasticsearch Curator is a command-line tool for managing indices and snapshots. It is useful for automating tasks like deleting old indices.

1. Install Curator: Follow the installation instructions from the Elasticsearch Curator documentation.

2. Create a Curator Configuration File: Define your Curator actions in a YAML file (e.g., delete_old_indices.yml).

    

   Copied!

   actions:
     1:
       action: delete_indices
       description: "Delete indices older than 30 days"
       options:
         ignore_empty_list: True
         allow_ilm_indices: False
       filters:
         - filtertype: age
           source: creation_date
           direction: older
           unit: days
           value: 30
   
   

3. Run Curator: Execute Curator with the configuration file.

    

   Copied!

   curator --config /path/to/curator.yml /path/to/delete_old_indices.yml
   

4. Manual Deletion (CLI or Script)

You can write scripts in languages like Python using the elasticsearch-py library to programmatically delete old indices.

Example with Python:

from elasticsearch import Elasticsearch

es = Elasticsearch(["<http://localhost:9200>"])

# Define index pattern
index_pattern = "logs-2023-01-*"

# List indices matching pattern
indices = es.cat.indices(index=index_pattern, format="json")

# Delete each index
for index in indices:
    index_name = index['index']
    es.indices.delete(index=index_name)
    print(f"Deleted index: {index_name}")

Summary

• Direct Deletion: Use the Elasticsearch REST API to delete specific or pattern-matching indices.
• ILM: Automate index management, including deletion, based on age or size.
• Kibana Dev Tools: Perform index deletion through Kibana's built-in Dev Tools console.
• Curator: Use Curator for advanced index management and automation.
• Scripted Solutions: Write scripts in programming languages for custom deletion logic.

Choose the method that best fits your needs based on your Elasticsearch setup and data management requirements.

Got an article suggestion? Let us know

Explore more

Elasticsearch Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.