Logstash - Remove Deep Field From Json File

To remove a deep field from a JSON document in Logstash, you can use the mutate filter, specifically the remove_field directive. This allows you to specify the exact field path you want to remove from your JSON structure.

Example Scenario

Suppose you have the following JSON structure in your Logstash input:

{
  "user": {
    "id": 123,
    "name": "John Doe",
    "details": {
      "email": "john.doe@example.com",
      "address": {
        "city": "New York",
        "zip": "10001"
      }
    }
  },
  "timestamp": "2024-10-25T10:00:00Z"
}

If you want to remove the email field from the details object, you can achieve this with the following Logstash configuration.

Logstash Configuration

Here’s how you can set up your logstash.conf file:

input {
  # Your input configuration, e.g., file, beats, etc.
  stdin {}
}

filter {
  # Example JSON structure
  json {
    source => "message"  # Assuming the input is a JSON string in the message field
  }

  mutate {
    remove_field => ["user.details.email"]  # Removing the deep field
  }

  # Optionally, you can also remove the whole details object
  # remove_field => ["user.details"]
}

output {
  stdout {
    codec => rubydebug  # To see the output in a readable format
  }
}

Explanation of the Configuration

1. Input Section: This part defines how Logstash receives the data. You can modify it based on your source (e.g., a file, HTTP input, etc.). In this example, it reads from standard input.
2. Filter Section:
   • The json filter is used to parse the incoming JSON string into structured fields.
   • The mutate filter uses remove_field to specify the path of the deep field you want to remove (user.details.email).
3. Output Section: The output is directed to the console with a pretty format (rubydebug), allowing you to see the changes made to the JSON structure.

Running the Configuration

1. Save your Logstash configuration in a file (e.g., logstash.conf).
2. Run Logstash with the following command:

bin/logstash -f /path/to/logstash.conf

1. Input the JSON directly into the terminal (if using stdin) or configure the input source as needed.

Output Verification

After running Logstash, you should see the output in the console without the email field:

{
  "user": {
    "id": 123,
    "name": "John Doe",
    "details": {
      "address": {
        "city": "New York",
        "zip": "10001"
      }
    }
  },
  "timestamp": "2024-10-25T10:00:00Z"
}

Conclusion

Removing deep fields from JSON documents in Logstash can be efficiently accomplished using the mutate filter with the remove_field directive. By adjusting the path to the field you wish to remove, you can effectively clean and process your JSON data as needed.

Got an article suggestion? Let us know

Explore more

Logstash

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.