Cheatsheet Elasticsearch

GET /_cluster/health
GET /_cluster/health?level=indices
GET /_cluster/stats

// Status: green (ok), yellow (no replicas), red (missing shards)

_cluster/health shows the overall state. green = all OK, yellow = unallocated replicas, red = missing primary shards. Monitor it constantly.

POST /_reindex
{
  "source": { "index": "products-v1" },
  "dest": { "index": "products-v2" }
}

// With a filter:
POST /_reindex
{
  "source": {
    "index": "logs",
    "query": { "range": { "date": { "gte": "2024-01-01" } } }
  },
  "dest": { "index": "logs-2024" }
}

_reindex copies documents between indices. Useful to migrate mappings or filter data. Accepts a query in the source to copy only a subset. Asynchronous operation.

PUT /_component_template/base-settings
{
  "template": {
    "settings": { "number_of_shards": 1 }
  }
}

PUT /_index_template/logs
{
  "index_patterns": ["logs-*"],
  "composed_of": ["base-settings", "logs-mappings"]
}

Component templates are reusable blocks of settings/mappings. composed_of combines several into an index template. Makes maintenance and consistency across indices easier.

GET /_cat/nodes?v
GET /_cat/indices?v&s=index
GET /_cat/shards?v
GET /_cat/allocation?v

// Parameter v = show headers
// s = sort by field

The _cat API is human-readable. ?v shows headers, &s=field sorts. _cat/indices lists indices with size, docs and health.

PUT /_index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "message": { "type": "text" }
      }
    }
  }
}

index_templates automatically apply settings/mappings to new indices matching the index_patterns. Essential for time series (logs, metrics).

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

PUT /name creates an index. number_of_shards defines primary partitions (fixed after creation). number_of_replicas are copies for redundancy (adjustable).

// View shard distribution
GET /_cat/shards/products?v

// Move a shard manually
POST /_cluster/reroute
{
  "commands": [{
    "move": {
      "index": "products", "shard": 0,
      "from_node": "node1", "to_node": "node2"
    }
  }]
}

Shards are horizontal partitions of the index. Replicas are copies for high availability. Primary shards are fixed at creation; replicas are dynamically adjustable.

GET /products/_settings
GET /products/_mapping

PUT /products/_settings
{ "number_of_replicas": 2 }

DELETE /products
POST /products/_close
POST /products/_open

_settings and _mapping show the configuration. DELETE deletes the index. _close/_open disables/re-enables without deleting (saves resources).

POST /logs-000001/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_docs": 10000000,
    "max_primary_shard_size": "50gb"
  }
}

_rollover creates a new index when conditions are met (age, docs, size). Used with aliases and ILM for automatic management of time-based indices.

POST /products/_doc
{
  "name": "Laptop Pro",
  "price": 1299.99,
  "tags": ["tech", "laptop"],
  "stock": 15
}

// Response: { "_id": "abc123", "result": "created" }

POST /index/_doc indexes with an automatic ID. The document is analyzed and indexed for search. result: "created" confirms the creation. Near real time (refresh 1s).

POST /products/_update/1
{
  "script": {
    "source": "ctx._source.stock -= params.qty",
    "params": { "qty": 1 }
  }
}

// Upsert with a script:
POST /counters/_update/views
{
  "script": { "source": "ctx._source.total += 1" },
  "upsert": { "total": 1 }
}

script allows logic in Painless. ctx._source accesses the fields. params avoids hardcoding. upsert creates the doc if it does not exist.

POST /products/_update_by_query
{
  "query": {
    "term": { "category": "computers" }
  },
  "script": {
    "source": "ctx._source.discount = 0.1"
  }
}

_update_by_query applies a script to all matching documents. Useful for migrations and mass updates. Asynchronous — check progress with _tasks.

PUT /products/_doc/1
{
  "name": "Wireless Mouse",
  "price": 29.99
}

// If it already exists, it overwrites (result: "updated")
// To guarantee creation only:
PUT /products/_create/1
{ "name": "Mouse" }

PUT /index/_doc/ID sets the ID manually. If it exists, it overwrites. _create fails if the document already exists (prevents accidental overwrites).

DELETE /products/_doc/1

// Delete by query
POST /products/_delete_by_query
{
  "query": {
    "term": { "stock": 0 }
  }
}

DELETE /index/_doc/ID removes a document. _delete_by_query removes all that match the query. Asynchronous operation — use wait_for_completion=false to avoid blocking.

GET /products/_doc/1

// Only the source (no metadata)
GET /products/_source/1

// Specific fields
GET /products/_doc/1?_source=name,price

// Multiple documents
POST /products/_mget
{ "ids": ["1", "2", "3"] }

GET /index/_doc/ID returns document + metadata (_version, _seq_no). _source filters fields. _mget fetches several in one call.

POST /_bulk
{"index": {"_index": "products", "_id": "1"}}
{"name": "Keyboard", "price": 49}
{"index": {"_index": "products", "_id": "2"}}
{"name": "Monitor", "price": 299}
{"update": {"_index": "products", "_id": "1"}}
{"doc": {"price": 45}}
{"delete": {"_index": "products", "_id": "2"}}

_bulk executes multiple operations in one call. Format: action line + data line. Actions: index, create, update, delete. Recommended max: 5-15 MB per request.

POST /products/_update/1
{
  "doc": {
    "price": 999.99,
    "on_sale": true
  }
}

_update with "doc" does a partial merge — it only changes the given fields. Internally it does get + reindex. More efficient than resending the full document.

// Get the current version
GET /products/_doc/1
// → "_seq_no": 5, "_primary_term": 1

// Update with version
PUT /products/_doc/1?if_seq_no=5&if_primary_term=1
{
  "name": "Mouse v2",
  "price": 34.99
}
// Fails with 409 if the version changed

if_seq_no and if_primary_term implement optimistic concurrency control. If another process changed the document, it returns 409 Conflict. Prevents lost overwrites.

GET /products/_search
{
  "query": {
    "match": {
      "name": "laptop pro"
    }
  }
}

match analyzes the text and searches by tokens. "laptop pro" → searches "laptop" OR "pro". Uses the field analyzer. Ideal for text fields with natural search.

// Exact phrase (order matters)
{ "match_phrase": { "name": "laptop pro" } }

// With slop (distance tolerance)
{ "match_phrase": { "name": { "query": "fast laptop", "slop": 2 } } }

// Fuzzy (tolerate typos)
{ "match": { "name": { "query": "laptp", "fuzziness": "AUTO" } } }

match_phrase requires tokens in the exact order. slop allows distance between words. fuzziness: "AUTO" tolerates typos (1-2 characters). Great for search UX.

GET /products/_search
{
  "query": { "match_all": {} },
  "from": 20,
  "size": 10
}

// Default limit: from + size <= 10000
// For more, use search_after

from is the offset, size the number of results. Limit: from + size ≤ 10000 (configurable via max_result_window). For large datasets, use search_after.

GET /products/_search
{
  "query": {
    "term": {
      "status": "active"
    }
  }
}

// Multiple values:
{ "terms": { "tags": ["tech", "gaming"] } }

term searches the exact value without analysis. For keyword, boolean, number, date fields. terms accepts an array (OR). Do not use on text fields.

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "laptop",
      "fields": ["name^3", "description", "tags^2"],
      "type": "best_fields"
    }
  }
}

multi_match searches across multiple fields. ^3 gives triple weight to the field. Types: best_fields (default), most_fields, cross_fields, phrase.

{
  "query_string": {
    "query": "(laptop OR tablet) AND -used",
    "default_field": "name"
  }
}

// Safe version (no syntax errors):
{
  "simple_query_string": {
    "query": "laptop +pro -used",
    "fields": ["name", "description"]
  }
}

query_string supports Lucene syntax (AND, OR, NOT, *, ~). simple_query_string is safer — it never throws a parse error. Ideal for direct user input.

GET /products/_search
{
  "query": {
    "bool": {
      "must": [{ "match": { "name": "pro" } }],
      "must_not": [{ "term": { "status": "sold_out" } }],
      "should": [{ "term": { "tags": "sale" } }],
      "filter": [{ "range": { "price": { "lte": 1500 } } }]
    }
  }
}

bool combines queries: must (AND, with score), filter (AND, no score, cached), must_not (NOT), should (OR, optional). The most used one.

// Pattern with * and ?
{ "wildcard": { "code": "PRD-*" } }

// Prefix
{ "prefix": { "name": "lap" } }

// Field exists
{ "exists": { "field": "tags" } }

// Regex (use with care)
{ "regexp": { "code": "PRD-[0-9]+" } }

wildcard uses * (any sequence) and ? (one character). prefix searches by the beginning. exists checks field presence. regexp is powerful but slow.

{ "range": { "price": { "gte": 100, "lte": 500 } } }

{ "range": { "date": { "gte": "2024-01-01", "lt": "2025-01-01" } } }

{ "range": { "age": { "gt": 18 } } }

// Operators: gt, gte, lt, lte

range filters by numeric or time interval. Operators: gt (greater), gte (greater or equal), lt (less), lte (less or equal). Works with ISO dates.

GET /products/_search
{
  "query": { "match_all": {} },
  "sort": [
    { "price": "asc" },
    { "name.raw": "desc" },
    { "_score": "desc" }
  ]
}

sort orders the results. asc or desc. text fields need a keyword subfield to sort. When using sort, _score is disabled by default.

PUT /products
{
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "price": { "type": "float" },
      "tags": { "type": "keyword" },
      "created": { "type": "date" },
      "active": { "type": "boolean" }
    }
  }
}

mappings defines the index schema — types and behavior of each field. It must be defined before indexing. Wrong types cause irreversible search problems.

GET /products/_mapping

// Add a new field (allowed):
PUT /products/_mapping
{
  "properties": {
    "category": { "type": "keyword" }
  }
}

// ❌ You cannot change the type of an existing field
// Solution: reindex into a new index

You can add new fields to an existing mapping. But you cannot change the type of an already created field. To change types, create a new index and use _reindex.

// Disable indexing (store only):
"internal_note": {
  "type": "text",
  "index": false
}

// Disable doc_values (no sorting/aggregation):
"long_description": {
  "type": "keyword",
  "doc_values": false
}

index: false excludes the field from search (saves space). doc_values: false disables sorting/aggregation. Optimizes storage for fields that only need to be returned.

text       // full-text (analyzed, for search)
keyword    // exact (not analyzed, for filters/sorting)
integer, long, float, double, short, byte
boolean
date       // "yyyy-MM-dd" or epoch
ip         // IPv4/IPv6 addresses
geo_point  // geographic coordinates
nested     // independent objects
object     // flattened object (default)

text is analyzed (tokenized) for full-text search. keyword is exact — for filters, aggregations and sorting. nested keeps objects the separate documents.

"created": {
  "type": "date",
  "format": "yyyy-MM-dd HH:mm:ss||epoch_millis"
}

// "strict_date_optional_time" (default)
// Accepts: "2024-03-15" or "2024-03-15T10:30:00Z"
// epoch_millis: 1710500000000

date fields accept configurable formats. || separates multiple formats. epoch_millis accepts a Unix timestamp in ms. The default is strict_date_optional_time.

"name": {
  "type": "text",
  "fields": {
    "raw": { "type": "keyword" },
    "lower": {
      "type": "keyword",
      "normalizer": "lowercase"
    }
  }
}
// name → full-text search
// name.raw → sorting/exact filter
// name.lower → case-insensitive filter

fields indexes the same value in different ways. name for search, name.raw for sorting/aggregation. Essential pattern for fields that need both uses.

// object (default) - flattened, loses the relation:
"comments": { "type": "object" }

// nested - each element is independent:
"comments": {
  "type": "nested",
  "properties": {
    "author": { "type": "keyword" },
    "text": { "type": "text" }
  }
}

// Nested query:
{ "nested": { "path": "comments", "query": { ... } } }

object flattens arrays — it loses the relation between fields of the same element. nested keeps each object independent. Use nested when you need to filter by a combination of fields.

PUT /products
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": { "type": "text" }
    }
  }
}

// "true" (default): infers types automatically
// "false": ignores unmapped fields
// "strict": rejects unmapped fields (error)

dynamic controls unexpected fields. true (default) infers types — risky in production. strict rejects unknown fields. Recommended: map explicitly.

PUT /products
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercase": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "code": { "type": "keyword", "normalizer": "lowercase" }
    }
  }
}

normalizer applies transformations to keyword fields at index time (lowercase, asciifolding). Unlike an analyzer, it operates on the whole text. Useful for case-insensitive filters.

GET /products/_search
{
  "size": 0,
  "aggs": {
    "by_tag": {
      "terms": { "field": "tags", "size": 20 }
    }
  }
}
// → buckets: [{key:"tech", doc_count:45}, ...]

terms groups by distinct values of a keyword field. size: 0 in the search avoids returning documents. Returns buckets with key and count. The most used one.

"aggs": {
  "statistics": {
    "stats": { "field": "price" }
  },
  "unique_categories": {
    "cardinality": { "field": "category" }
  }
}
// stats → count, min, max, avg, sum
// cardinality → distinct count (approximate)

stats returns count, min, max, avg and sum at once. cardinality counts distinct values (like COUNT DISTINCT). It is approximate (HyperLogLog) — error ~1%.

"aggs": {
  "all_groups": {
    "composite": {
      "size": 100,
      "sources": [
        { "cat": { "terms": { "field": "category" } } },
        { "brand": { "terms": { "field": "brand" } } }
      ],
      "after": { "cat": "tech", "brand": "ASUS" }
    }
  }
}

composite paginates aggregations with after. Overcomes the 10000 bucket limit of terms. Combines multiple sources. Ideal for exporting all groups of a large dataset.

"aggs": {
  "avg_price": { "avg": { "field": "price" } },
  "total_stock": { "sum": { "field": "stock" } },
  "cheapest": { "min": { "field": "price" } },
  "most_expensive": { "max": { "field": "price" } },
  "total_value": {
    "script": { "source": "doc['price'].value * doc['stock'].value" }
  }
}

Metric aggregations: avg, sum, min, max. script allows custom calculations. Works on numeric fields. Returns a single value per aggregation.

"aggs": {
  "price_ranges": {
    "range": {
      "field": "price",
      "ranges": [
        { "to": 50 },
        { "from": 50, "to": 200 },
        { "from": 200 }
      ]
    }
  },
  "histogram": {
    "histogram": { "field": "price", "interval": 50 }
  }
}

range groups into custom intervals. histogram uses fixed intervals. Both return buckets with counts. Useful for distributions and per-range reports.

"aggs": {
  "sales_month": {
    "date_histogram": {
      "field": "created",
      "calendar_interval": "month",
      "format": "yyyy-MM",
      "min_doc_count": 0
    }
  }
}

date_histogram groups by time period. calendar_interval: minute, hour, day, week, month, quarter, year. min_doc_count: 0 includes empty periods. Essential for time series.

"aggs": {
  "active": {
    "filter": { "term": { "status": "active" } },
    "aggs": { "average": { "avg": { "field": "price" } } }
  },
  "by_status": {
    "filters": {
      "filters": {
        "available": { "term": { "stock_gt": 0 } },
        "sold_out": { "term": { "stock": 0 } }
      }
    }
  }
}

filter restricts an aggregation to documents meeting a condition. filters creates multiple named buckets with different conditions. More efficient than terms for few values.

"aggs": {
  "by_category": {
    "terms": { "field": "category" },
    "aggs": {
      "avg_price": { "avg": { "field": "price" } },
      "top_product": {
        "top_hits": { "size": 1, "sort": [{ "sales": "desc" }] }
      }
    }
  }
}

Aggregations can contain sub-aggregations (aggs inside aggs). Metrics are calculated per bucket. top_hits returns representative documents from each group.

"aggs": {
  "latency_percentiles": {
    "percentiles": {
      "field": "response_time",
      "percents": [50, 90, 95, 99]
    }
  },
  "percentile_ranks": {
    "percentile_ranks": {
      "field": "response_time",
      "values": [200, 500]
    }
  }
}

percentiles shows the distribution (p50, p90, p99). Essential for latencies. percentile_ranks tells what percentage is below a value. Approximate (t-digest somethingrithm).

POST /_analyze
{
  "analyzer": "standard",
  "text": "The Laptop Pro is very fast!"
}
// → ["the", "laptop", "pro", "is", "very", "fast"]

// Test with the index analyzer:
POST /products/_analyze
{ "field": "name", "text": "Gaming Laptop" }

_analyze shows how the text is tokenized. Essential for search debugging. Test with the real field analyzer using POST /index/_analyze with field.

lowercase       // lowercase
asciifolding    // removes accents (ã → a)
stop            // removes stop words
stemmer         // reduces to the root (english)
synonym         // synonyms
length          // filters by length
truncate        // truncates long tokens
unique          // removes duplicate tokens

Token filters transform tokens after tokenization. asciifolding normalizes accents. stemmer reduces words to their root. synonym expands with synonyms. They chain in order.

"name": {
  "type": "text",
  "analyzer": "autocomplete",
  "search_analyzer": "standard"
}

// Indexing: edge_ngram (generates prefixes)
// Search: standard (searches the whole token)
// "cat" → match on "cats", "catalog"

analyzer is used at index time, search_analyzer at search time. Separating them lets you index with ngrams but search with the whole token. Essential for efficient autocomplete.

standard    // unicode, lowercase (default)
simple      // letters only, lowercase
whitespace  // splits on spaces (no lowercase)
stop        // standard + removes stop words
keyword     // no analysis (whole text = 1 token)
english     // stemming + English stop words

standard is the default — tokenizes by Unicode and lowercases. keyword does not analyze (whole text). stop removes common words. Choose according to the use case.

"filter": {
  "en_stem": {
    "type": "stemmer",
    "language": "light_english"
  }
}
// "laptops" → "laptop"
// "computers" → "computer"
// "programming" → "program"

// Options: "english" (aggressive) or "light_english"

stemmer reduces words to their root. light_english is less aggressive (recommended). Improves recall (finds plurals/conjugations). May reduce precision — always test.

"settings": {
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "tokenizer": "standard",
        "filter": ["lowercase", "asciifolding", "en_stem"]
      }
    },
    "filter": {
      "en_stem": { "type": "stemmer", "language": "light_english" }
    }
  }
}

A custom analyzer combines tokenizer + filters. asciifolding removes accents. light_english does stemming (laptops → laptop). Filter order matters.

"filter": {
  "my_synonyms": {
    "type": "synonym",
    "synonyms": [
      "laptop, notebook, portable",
      "cell phone, mobile, smartphone",
      "screen, display, monitor"
    ]
  }
}

synonym expands tokens with equivalents. "laptop" → also searches "notebook" and "portable". Format: comma-separated values. You can use an external file with synonyms_path.

standard    // unicode word boundaries
whitespace  // splits on spaces
keyword     // no tokenization (1 token)
pattern     // custom regex: "[^\p{L}\d]+"
ngram       // subsequences: "cat" → "ca", "at"
edge_ngram  // prefixes: "cats" → "c", "ca", "cat"
path_hierarchy // "a/b/c" → "a", "a/b", "a/b/c"

tokenizer splits the text into tokens. standard is the most used. edge_ngram is ideal for autocomplete (prefixes). pattern uses a custom regex for splitting.

"settings": {
  "analysis": {
    "analyzer": {
      "autocomplete": {
        "tokenizer": "autocomplete_tokenizer"
      }
    },
    "tokenizer": {
      "autocomplete_tokenizer": {
        "type": "edge_ngram",
        "min_gram": 2,
        "max_gram": 15,
        "token_chars": ["letter", "digit"]
      }
    }
  }
}

edge_ngram generates progressive prefixes: "cats" → "ca", "cat", "cats". Ideal for autocomplete. Use a different analyzer for indexing (edge_ngram) and for search (standard).

"bool": {
  "filter": [
    { "term": { "status": "active" } },
    { "range": { "price": { "lte": 100 } } }
  ],
  "must": [
    { "match": { "name": "laptop" } }
  ]
}

filter does not compute a score and is cached — much faster. Use filter for exact conditions (status, range, term). must only for full-text relevance. Rule: filter by default.

// Force refresh (documents become searchable)
POST /products/_refresh

// Flush (write segments to disk)
POST /products/_flush

// View segments:
GET /_cat/segments/products?v

// Default refresh interval: 1s (near real-time)

refresh makes documents searchable (creates an in-memory segment). Default: 1s. flush persists to disk. Increasing refresh_interval improves write throughput.

GET /products/_search
{
  "profile": true,
  "query": {
    "match": { "name": "laptop" }
  }
}
// → shows the time of each phase:
// build_scorer, next_doc, advance, score, etc.

"profile": true details the execution time of each query component. Shows shards, collectors and time per operation. Essential tool for diagnosing slow queries.

GET /products/_search
{
  "_source": ["name", "price"],
  "query": { "match_all": {} }
}

// Exclude large fields:
{ "_source": { "excludes": ["long_description"] } }

_source limits the returned fields. Reduces bandwidth and memory. excludes removes heavy fields. Does not affect the search — only what is returned to the client.

// Consolidate segments (read-only indices only!)
POST /logs-2024/_forcemerge?max_num_segments=1

// View segments before/after:
GET /_cat/segments/logs-2024?v

// ⚠️ NEVER on active indices (heavy I/O)

_forcemerge consolidates segments into one. Improves search performance. Only use on indices that no longer receive writes (after rollover). Heavy operation — run off-peak.

// Page 1:
GET /products/_search
{
  "size": 100,
  "sort": [{ "created": "desc" }, { "_id": "asc" }],
  "query": { "match_all": {} }
}

// Page 2 (use the last sort value):
{ "search_after": ["2024-06-15T10:00:00Z", "doc_99"] }

search_after paginates beyond the 10000 limit. Uses the sort values of the last document the a cursor. More efficient than scroll. Requires a sort with a unique field (tiebreaker).

// Index with custom routing:
PUT /products/_doc/1?routing=tenant_42
{ "name": "Product", "tenant": "tenant_42" }

// Search only the tenant shard:
GET /products/_search?routing=tenant_42
{
  "query": { "match": { "name": "product" } }
}

routing directs documents to a specific shard. Searches with routing query only 1 shard instead of all. Essential for multi-tenancy. Reduces latency drastically.

// Before mass ingestion:
PUT /logs/_settings
{
  "number_of_replicas": 0,
  "refresh_interval": "30s"
}

// After ingestion, restore:
PUT /logs/_settings
{
  "number_of_replicas": 1,
  "refresh_interval": "1s"
}

For bulk indexing: disable replicas and increase refresh_interval. Reduces I/O drastically. Restore after finishing. Use 5-15 MB batches with _bulk.

// Enable result cache (default: only size=0)
GET /products/_search?request_cache=true
{
  "size": 0,
  "aggs": { "average": { "avg": { "field": "price" } } }
}

// Clear cache:
POST /products/_cache/clear

// View usage:
GET /_cat/nodes?v&h=name,request_cache_memory_size

request_cache stores aggregation results on the node. By default, it only caches queries with size: 0. Automatically invalidated on refresh. Ideal for dashboards with rarely changing data.

POST /_aliases
{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add": { "index": "products-v2", "alias": "products" } }
  ]
}

// Apps always use the "products" alias
// The index switch is transparent (zero downtime)

aliases are pointers to indices. Lets you switch indices without changing code. Atomic operations (remove + add in one call). Essential for zero-downtime deploys.

POST /_reindex
{
  "source": { "index": "old" },
  "dest": { "index": "new" },
  "script": {
    "source": """
      ctx._source.full_name = ctx._source.name;
      ctx._source.remove('name');
      ctx._source.migrated = true;
    """
  }
}

_reindex with script transforms documents during migration. Renames fields, adds flags, removes data. Useful for evolving mappings without downtime.

// Configure remote cluster:
PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.dc2.seeds": ["dc2-node1:9300"]
  }
}

// Search in both:
GET /products,dc2:products/_search
{
  "query": { "match": { "name": "laptop" } }
}

Cross-cluster search queries multiple Elasticsearch clusters in one query. Configure with cluster.remote. Reference the cluster:index. Useful for multi-datacenter.

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": { "actions": {
        "rollover": { "max_size": "50gb", "max_age": "7d" }
      }},
      "warm": { "min_age": "7d", "actions": {
        "shrink": { "number_of_shards": 1 },
        "forcemerge": { "max_num_segments": 1 }
      }},
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

ILM automates the lifecycle: hot → warm → cold → delete. rollover creates a new index when limits are reached. shrink and forcemerge optimize in warm. Essential for logs.

// Register queries:
PUT /alerts/_doc/1
{ "query": { "match": { "message": "critical error" } } }

// Check a document against registered queries:
GET /alerts/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": { "message": "A critical error occurred on the server" }
    }
  }
}

percolator inverts the search: it registers queries and checks documents against them. Ideal for alerts and notifications. "Which queries match this document?"

GET /products/_search
{
  "suggest": {
    "correction": {
      "text": "laptp gaming",
      "term": { "field": "name", "size": 3 }
    },
    "phrase_suggest": {
      "text": "laptop gamming",
      "phrase": { "field": "name" }
    }
  }
}

term suggester suggests corrections per token. phrase suggester suggests full phrases (smarter). completion suggester uses FST for ultra-fast autocomplete.

GET /products/_search
{
  "query": { "match_all": {} },
  "script_fields": {
    "price_with_vat": {
      "script": { "source": "doc['price'].value * 1.23" }
    },
    "discount_pct": {
      "script": { "source": "(doc['old_price'].value - doc['price'].value) / doc['old_price'].value * 100" }
    }
  }
}

script_fields computes fields on the fly with Painless. They are not indexed — computed on every query. Useful for derived values. Slower than precomputed fields.

// Mapping:
"suggestion": {
  "type": "completion",
  "analyzer": "simple"
}

// Query:
GET /products/_search
{
  "suggest": {
    "auto": {
      "prefix": "lap",
      "completion": { "field": "suggestion", "size": 5, "fuzzy": { "fuzziness": "AUTO" } }
    }
  }
}

completion suggester is optimized for autocomplete. Uses an in-memory FST structure — extremely fast. fuzzy tolerates typos. Index suggestions in a dedicated completion type field.

GET /products/_search
{
  "query": { "match": { "description": "fast laptop" } },
  "highlight": {
    "pre_tags": ["<mark>"],
    "post_tags": ["</mark>"],
    "fields": {
      "description": { "fragment_size": 150, "number_of_fragments": 3 }
    }
  }
}

highlight returns snippets with the matched terms highlighted. pre_tags/post_tags define the markup. fragment_size controls the excerpt size. Essential for search UX.

PUT /_ingest/pipeline/clean-products
{
  "description": "Normalize product data",
  "processors": [
    { "lowercase": { "field": "name" } },
    { "trim": { "field": "name" } },
    { "remove_null": { "field": "description" } },
    { "set": {
      "field": "created_at",
      "value": "{{_ingest.timestamp}}"
    }}
  ]
}

Ingest pipelines transform documents before indexing. Processors chain in order. Created with PUT /_ingest/pipeline/name. Executed on the ingest node.

{
  "grok": {
    "field": "message",
    "patterns": [
      "%{IP:ip} - %{WORD:user} [%{HTTPDATE:date}] \"%{WORD:method} %{URIPATHPARAM:url}\" %{INT:status} %{INT:bytes}"
    ]
  }
}
// Input: "192.168.1.1 - admin [10/Oct/2024:13:55:36] \"GET /api\" 200 1234"
// → { ip, user, date, method, url, status, bytes }

grok extracts structured fields from unstructured text using patterns. %{TYPE:field} captures values. Essential for log parsing. Can be slow — test performance.

// List all:
GET /_ingest/pipeline

// View a specific one:
GET /_ingest/pipeline/clean-products

// Delete:
DELETE /_ingest/pipeline/clean-products

// Test an individual processor:
POST /_ingest/pipeline/_simulate
{
  "pipeline": { "processors": [{ "uppercase": { "field": "name" } }] },
  "docs": [{ "_source": { "name": "test" } }]
}

GET /_ingest/pipeline lists all. DELETE removes. Inline _simulate tests processors without creating a pipeline. Useful for quick experimentation.

// Single document:
PUT /products/_doc/1?pipeline=clean-products
{ "name": "  LAPTOP PRO  " }

// Bulk:
POST /_bulk?pipeline=clean-products
{"index": {"_index": "products"}}
{"name": "KEYBOARD"}

// Default index pipeline:
PUT /products/_settings
{ "index.default_pipeline": "clean-products" }

Apply with ?pipeline=name at index time. Or set default_pipeline in the index settings to apply automatically. _simulate tests without indexing.

{
  "dissect": {
    "field": "message",
    "pattern": "%{ip} - %{user} [%{date}] \"%{method} %{url}\" %{status} %{bytes}"
  }
}

// Faster than grok (no regex)
// Uses literal delimiters

dissect is a faster alternative to grok for fixed-format logs. Uses literal delimiters instead of regex. Less flexible but much more performant.

POST /_ingest/pipeline/clean-products/_simulate
{
  "docs": [
    { "_source": { "name": "  LAPTOP  ", "price": -5 } },
    { "_source": { "name": "Mouse", "description": null } }
  ]
}
// Shows the result of each processor

_simulate tests the pipeline with sample documents without indexing. Shows the output of each processor. Essential for debugging. Reveals errors before production.

// 1. Create the enrich policy:
PUT /_enrich/policy/geo-ip
{
  "geoip": {
    "source": { "index": "ips" },
    "match_field": "ip",
    "enrich_fields": ["country", "city"]
  }
}

// 2. Execute:
POST /_enrich/policy/geo-ip/_execute

// 3. Use in the pipeline:
{ "enrich": { "policy_name": "geo-ip", "field": "ip", "target_field": "geo" } }

enrich adds data from a reference index during ingestion. Example: add geolocation from an IP. Create the policy, execute it, and use it the a processor.

set          // set a field
remove       // remove a field
rename       // rename a field
lowercase    // lowercase
uppercase    // uppercase
trim         // remove spaces
convert      // change type (string → integer)
split        // string → array
join         // array → string
gsub         // regex replace

Basic processors: set, remove, rename for restructuring. lowercase, trim, gsub for cleaning. convert changes types. split/join for arrays.

{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:ip}"],
        "on_failure": [
          { "set": { "field": "parse_error", "value": "{{_ingest.on_failure_message}}" } },
          { "set": { "field": "ip", "value": "unknown" } }
        ]
      }
    }
  ]
}

on_failure defines what to do if a processor fails. Avoids rejecting the whole document. Records the error in a dedicated field. Can have on_failure per processor or at the pipeline level.

POST /_security/api_key
{
  "name": "app-readonly",
  "expiration": "30d",
  "role_descriptors": {
    "readonly": {
      "indices": [{
        "names": ["products"],
        "privileges": ["read"]
      }]
    }
  }
}
// → { "id": "...", "api_key": "...", "encoded": "base64..." }

API keys authenticate without user/password. Define granular permissions per index. expiration limits validity. Use the header Authorization: ApiKey encoded. Ideal for apps and services.

// Most active threads:
GET /_nodes/hot_threads

// Running tasks:
GET /_tasks?actions=*search*&detailed=true

// Cancel a slow task:
POST /_tasks/task_id:12345/_cancel

// Stats per node:
GET /_nodes/stats/jvm,the,process

hot_threads shows the threads with the most CPU. _tasks lists ongoing operations. _cancel terminates slow queries. _nodes/stats gives JVM, disk and process per node.

// Red cluster:
GET /_cluster/health?level=shards
GET /_cluster/allocation/explain

// Write rejections:
GET /_nodes/stats/thread_pool
// → "write": { "rejected": 150 }

// Circuit breaker (memory):
GET /_nodes/stats/breaker

// Disk watermark:
GET /_cluster/settings?include_defaults=true&filter_path=**.disk

Red cluster → allocation/explain. Rejections → check thread_pool. OOM → breaker stats. Full disk → disk watermark (85%/90%/95%). Systematic troubleshooting approach.

POST /_security/role/app_writer
{
  "indices": [{
    "names": ["products", "products-*"],
    "privileges": ["read", "write", "create_index"],
    "field_security": { "grant": ["name", "price", "tags"] },
    "query": { "term": { "tenant": "app1" } }
  }],
  "cluster": ["monitor"]
}

roles define permissions per index and cluster. field_security limits visible fields. query restricts documents (row-level security). Combine with API keys for multi-tenancy.

// Register repository:
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": { "location": "/mnt/backups" }
}

// Create snapshot:
PUT /_snapshot/my_backup/snap-2024-01?wait_for_completion=true
{ "indices": "products,logs-*" }

// Restore:
POST /_snapshot/my_backup/snap-2024-01/_restore
{ "indices": "products" }

snapshots are incrementsl cluster backups. Register a repository (fs, S3, GCS). PUT creates a snapshot, _restore restores it. Incremental — only stores changes. Essential in production.

GET /_cluster/health?level=shards
GET /_cluster/allocation/explain

GET /_cluster/settings
GET /_cluster/pending_tasks

// Allocate an unassigned shard:
POST /_cluster/reroute
{
  "commands": [{ "allocate_replica": {
    "index": "products", "shard": 1, "node": "node2"
  }}]
}

_cluster/health?level=shards shows the state per shard. allocation/explain explains why a shard is not allocated. reroute forces manual allocation. Essential for troubleshooting.

PUT /products/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s"
}

Slow logs record slow queries and indexing. Levels: warn, info, debug, trace. Time-based thresholds. Logs in logs/index_name_search_slowlog.log. Essential for diagnostics.

GET /_cat/indices?v&s=store.size:desc
GET /_cat/count/products?v
GET /_stats
GET /products/_stats

// Top indices by size:
GET /_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc&bytes=mb

_cat/indices with s=store.size:desc sorts by size. _stats gives detailed metrics (indexing, search, merge). Monitor growth to plan capacity.

// elasticsearch.yml:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

// Set passwords:
bin/elasticsearch-setup-passwords interactive

// Basic authentication:
curl -u elastic:password https://localhost:9200

xpack.security enables authentication and TLS. transport.ssl encrypts communication between nodes. http.ssl encrypts client-server. Always enable in production. Never expose without auth.