Skip to content

[DOCS] Opster Migration: Add errors to troubleshooting pages #1116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Apr 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 199 additions & 0 deletions troubleshoot/elasticsearch/all-shards-failed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: All shards failed"
---

# Fix error: All shards failed [all-shards-failed]

```console
Error: all shards failed
```

The `all shards failed` error indicates that {{es}} couldn't get a successful response from any of the shards involved in the query. Possible causes include shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields.

## Unsupported operations on text fields

The `all shards failed` error can occur when you try to sort or aggregate on `text` fields. These fields are designed for full-text search and don't support exact-value operations like sorting and aggregation.

To fix this issue, use a `.keyword` subfield:

```console
GET my-index/_search
{
"aggs": {
"names": {
"terms": {
"field": "name.keyword"
}
}
}
}
```

If no `.keyword` subfield exists, define a [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md#types-multi-fields) mapping:

```console
PUT my-index
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
```

### Metric aggregations on text fields

The `all shards failed` error can also occur when you use a metric aggregation on a text field. [Metric aggregations](elasticsearch://reference/aggregations/metrics.md) require numeric fields.

You can use a script to convert the text value to a number at query time:

```console
GET my-index/_search
{
"aggs": {
"total_cost": {
"sum": {
"script": {
"source": "Integer.parseInt(doc.cost.value)"
}
}
}
}
}
```

Or change the field mapping to a numeric type:

```console
PUT my-index
{
"mappings": {
"properties": {
"cost": {
"type": "integer"
}
}
}
}
```

## Failed shard recovery

A shard failure during recovery can prevent successful queries.

To identify the cause, check the cluster health:

```console
GET _cluster/health
```

As a last resort, you can delete the problematic index.

## Misused global aggregation

[Global aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-global-aggregation.md) must be defined at the top level of the aggregations object. Nesting can cause errors.

To fix this issue, structure the query so that the `global` aggregation appears at the top level:

```console
GET my-index/_search
{
"size": 0,
"aggs": {
"all_products": {
"global": {},
"aggs": {
"genres": {
"terms": {
"field": "cost"
}
}
}
}
}
}
```

## Reverse_nested usage errors

Using a [`reverse_nested`](elasticsearch://reference/aggregations/search-aggregations-bucket-reverse-nested-aggregation.md) aggregation outside of a `nested` context causes errors.

To fix this issue, structure the query so that the `reverse_nested` aggregation is inside a `nested` aggregation:

```console
GET my-index/_search
{
"aggs": {
"comments": {
"nested": {
"path": "comments"
},
"aggs": {
"top_usernames": {
"terms": {
"field": "comments.username"
},
"aggs": {
"comment_issue": {
"reverse_nested": {},
"aggs": {
"top_tags": {
"terms": {
"field": "tags"
}
}
}
}
}
}
}
}
}
}
```

## Further troubleshooting

Use the `_cat/shards` API to view shard status and troubleshoot further.

```console
GET _cat/shards
```

For a specific index:

```console
GET _cat/shards/my-index
```

Example output:

```console-result
my-index 5 p STARTED 0 283b 127.0.0.1 ziap
my-index 5 r UNASSIGNED
my-index 2 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 2 r UNASSIGNED
my-index 3 p STARTED 3 7.2kb 127.0.0.1 ziap
my-index 3 r UNASSIGNED
my-index 1 p STARTED 1 3.7kb 127.0.0.1 ziap
my-index 1 r UNASSIGNED
my-index 4 p STARTED 2 3.8kb 127.0.0.1 ziap
my-index 4 r UNASSIGNED
my-index 0 p STARTED 0 283b 127.0.0.1 ziap
my-index 0 r UNASSIGNED
```
12 changes: 12 additions & 0 deletions troubleshoot/elasticsearch/errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
navigation_title: Error reference
---

# Troubleshoot common errors in {{es}}

Use the topics in this section to troubleshoot common errors in {{es}} deployments.

* [](/troubleshoot/elasticsearch/all-shards-failed.md)
* [](/troubleshoot/elasticsearch/failed-to-parse-field-of-type.md)
* [](/troubleshoot/elasticsearch/unable-to-retrieve-node-fs-stats.md)
* [](/troubleshoot/elasticsearch/unable-to-parse-response-body.md)
58 changes: 58 additions & 0 deletions troubleshoot/elasticsearch/failed-to-parse-field-of-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: Failed to parse field of type in document with id"
---

# Fix error: Failed to parse field [failed-to-parse-field-of-type]

```console
Error: failed to parse field [field] of type [type] in document with id [id]
```

This error occurs when you try to index a document, but one of the field values doesn't match the expected data type. {{es}} rejects the document when it encounters incompatible values, like a string in a numeric field or an invalid IP address.

To fix this issue, make sure each field value matches the data type defined in the mapping.

## Field types and mapping

When no explicit mapping exists, {{es}} uses [dynamic mappings](../../manage-data/data-store/mapping/dynamic-field-mapping.md) to infer a field's type based on the **first value indexed**.

For example, if you index:

```console
PUT test/_doc/1
{
"ip_address": "179.152.62.82",
"boolean_field": "off"
}
```

Without explicit mapping, {{es}} will treat `ip_address` and `boolean_field` as `text`, which might not be the intended result.

To avoid this, define the mapping explicitly:

```console
PUT test
{
"mappings": {
"properties": {
"ip_address": { "type": "ip" },
"boolean_field": { "type": "boolean" }
}
}
}
```

To check the data type of the field causing the error, first get the mapping:

```console
GET your-index-name/_mapping
```

Make sure the incoming data matches the expected type. If not, you'll need to fix the data or update the mapping. If necessary, create a new index with the correct mapping and reindex your data.
62 changes: 62 additions & 0 deletions troubleshoot/elasticsearch/security/token-invalid-expired.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
navigation_title: "Error: Token invalid or expired"
---

# Fix errors: Invalid token or token expired in {{es}} [token-invalid-expired]

```console
Error: token expired
```

```console
Error: invalid token
```

These errors occur when {{es}} receives a request containing an invalid or expired token during authentication. They're typically caused by missing, incorrect, or outdated tokens. If an invalid or expired token is used, {{es}} rejects the request.

## Invalid token

{{es}} rejects requests with invalid authentication tokens. Common causes include:

- The token is expired or revoked
- The token format is incorrect or malformed
- The Authorization header is missing or doesn’t start with Bearer
- The client or middleware failed to attach the token properly
- Security settings in {{es}} are misconfigured

To resolve this error:

- Verify the token and ensure it's correctly formatted and current.
- Check expiration and generate a new token if needed.
- Inspect your client and confirm the token is sent in the `Authorization` header.
- Review {{es}} settings and check that token auth is enabled:

```yaml
xpack.security.authc.token.enabled: true
```

- Use logs for details: {{es}} logs may provide context about the failure.


## Token expired

This error occurs when {{es}} receives a request containing an expired token during authentication.

To resolve this issue:

- Refresh the token, and obtain a new token using your token refresh workflow.
- Implement automatic token refresh and ensure your application is configured to refresh tokens before expiration.
- Avoid using expired tokens and do not reuse tokens after logout or expiration.
- Adjust token lifespan if needed and configure a longer token expiration in `elasticsearch.yml`, though this should be balanced against security needs:

```yaml
xpack.security.authc.token.timeout: 20m
```

Loading
Loading