Kibana Logstash Elasticsearch | Unindexed Fields Cannot Be Searched

The error "Unindexed fields cannot be searched" in Kibana typically occurs when you try to search or filter on fields that are not indexed in Elasticsearch. This happens because Elasticsearch, by default, only indexes certain fields for search and aggregation operations. Fields that are not indexed cannot be used for querying or filtering, which is why you encounter this error.

Here are the steps to understand and resolve the issue:

1. Check Index Mapping

The first step is to check whether the fields you are trying to query are indexed. Fields in Elasticsearch can have different mappings, which define how they are stored and whether they are indexed or not.

You can view the mapping for a specific index using the following API request:

curl -X GET "localhost:9200/my_index/_mapping"

In the response, check the index property for each field. If a field is set to "index": false, it means that the field is not indexed, and therefore it cannot be searched or used in filters.

Example:

"my_field": {
  "type": "text",
  "index": false
}

2. Reindex the Data with Proper Mapping

If you find that the fields you need are not indexed, you'll need to reindex your data with the correct mappings.

Step 1: Create a New Index with Updated Mapping

Create a new index with the correct mapping for the fields you want to be searchable:

curl -X PUT "localhost:9200/my_new_index" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "my_field": {
        "type": "text",
        "index": true
      }
    }
  }
}
'

In this example, my_field is explicitly set to "index": true, which ensures that it will be indexed and searchable.

Step 2: Reindex the Old Data

Once the new index is created with the updated mapping, you can reindex your old data into the new index:

curl -X POST "localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "my_old_index"
  },
  "dest": {
    "index": "my_new_index"
  }
}
'

After reindexing, the fields in the new index will be searchable.

3. Update Field Mapping Dynamically (Elasticsearch 7.x and 8.x)

Elasticsearch does not allow you to change the mapping of an existing field once the index is created. You either need to reindex the data with the correct mapping (as explained above), or in some cases, you can update dynamic field settings to allow for indexing in the future.

To make sure new fields are indexed, update the dynamic templates (if applicable):

curl -X PUT "localhost:9200/my_index/_mapping" -H 'Content-Type: application/json' -d'
{
  "dynamic": "strict"
}
'

This approach ensures that new fields are indexed automatically.

4. Use Keyword Fields for Text Searching

If you are trying to search on a text field and getting this error, Elasticsearch doesn't index the full-text fields for search by default. Instead, it provides a .keyword subfield, which you can use for exact matching.

Example query in Kibana:

{
  "query": {
    "term": {
      "my_field.keyword": "some_value"
    }
  }
}

Use .keyword if you need to perform an exact match search on text fields.

5. Check Field Capabilities in Kibana

In Kibana, you can also check whether a field is indexed by going to Management > Index Patterns, selecting the relevant index pattern, and seeing the searchable property for each field. If the field is not searchable, it means it’s not indexed.

You can also refresh the index pattern in Kibana by clicking on the refresh button, which will reload the field list and properties.

6. Set Fields to be Indexed When Ingesting via Logstash

If you're using Logstash to send data to Elasticsearch, you can control field indexing directly in Logstash by adjusting your output configuration.

Example Logstash output configuration:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "my_index"
    document_type => "_doc"
    template_overwrite => true
    template => "/path/to/template.json"
  }
}

In your template.json, ensure the fields are set to be indexed:

{
  "mappings": {
    "properties": {
      "my_field": {
        "type": "text",
        "index": true
      }
    }
  }
}

7. Use dynamic_templates for Auto-indexing

To make sure that new fields are indexed automatically when they are encountered, you can configure a dynamic template that tells Elasticsearch how to index new fields:

{
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keywords": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "index": true
          }
        }
      }
    ]
  }
}

This template ensures that any new string field is indexed as keyword and is searchable.

Summary

If you encounter the "unindexed fields cannot be searched" error:

1. Check the field mapping using the _mapping API to verify whether the field is indexed.
2. Reindex the data with updated mappings if necessary.
3. For exact text searches, use .keyword fields.
4. Use Logstash templates or dynamic templates to ensure fields are properly indexed when ingesting data.

These steps should resolve the issue and allow you to search on previously unindexed fields in Kibana.

Got an article suggestion? Let us know

Explore more

Kibana

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.