Back to Blog
Health Checks - Infrastructure

Node Settings Check: JVM and Configuration Optimization Guide

Validate node-specific configurations, optimize JVM settings, and ensure consistent cluster behavior through proper node-level configuration management.

December 7, 2024
16 min read
ElasticDoctor Team

Configuration is King

Proper node configuration is the difference between a high-performing, stable cluster and one plagued by outages and performance issues. This check validates JVM settings, system configuration, and node-specific parameters.

While hardware provides the foundation, configuration determines how effectively that hardware is utilized. The node settings check examines JVM parameters, memory allocation, garbage collection settings, and system-level configuration to ensure optimal performance and stability.

What You'll Learn

JVM Optimization

  • • Heap sizing best practices
  • • Garbage collector selection and tuning
  • • JVM flags and performance options
  • • Memory locking and swap management

System Configuration

  • • File descriptor limits and ulimits
  • • Network and discovery settings
  • • Path and storage configuration
  • • Security and authentication setup

Critical Configuration Categories

Simple English Explanation

Think of node configuration like tuning a car engine. You need the right fuel mixture (JVM settings), proper oil levels (memory), good spark timing (GC settings), and all systems working together harmoniously.

One misconfigured setting can cause the whole "engine" to run poorly or even break down completely.

JVM Configuration

🚨 Critical: Heap Size

Best Practices
  • • Set -Xms and -Xmx to same value
  • • Use 50% of RAM maximum
  • • Never exceed 32GB (compressed OOPs)
  • • Leave RAM for file system cache
Example Configuration
-Xms16g
-Xmx16g
# For 32GB RAM server

⚡ Important: Garbage Collector

G1GC (Recommended)
  • • Best for heaps >8GB
  • • Low-latency collection
  • • Good for production
-XX:+UseG1GC
CMS (Legacy)
  • • Older ES versions
  • • Being phased out
  • • Not recommended
-XX:+UseConcMarkSweepGC
ZGC (Future)
  • • Very large heaps
  • • Ultra-low latency
  • • Experimental
-XX:+UseZGC

🔧 Essential JVM Flags

# Memory and GC
-Xms16g
-Xmx16g
-XX:+UseG1GC
-XX:G1HeapRegionSize=32m

# Performance
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
-XX:+AlwaysPreTouch

# Error handling
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch

# JIT optimization
-XX:+UseStringDeduplication
-XX:+UnlockDiagnosticVMOptions
-XX:+DebugNonSafepoints

System Configuration

🚨 Critical: Memory Locking

Configuration
bootstrap.memory_lock: true

Prevents JVM heap from being swapped to disk

System Requirements
# /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

⚠️ Important: File Descriptors

Why It Matters
  • • Each index segment uses file descriptors
  • • Network connections consume FDs
  • • Low limits cause hard failures
  • • Minimum 65536 required
Configuration
# /etc/security/limits.conf
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536

# Check current limit
ulimit -n

🔧 Additional System Settings

Virtual Memory
# Increase virtual memory areas
vm.max_map_count=262144

# Add to /etc/sysctl.conf
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
Swap Disable
# Disable swap completely
sudo swapoff -a

# Or reduce swappiness
vm.swappiness=1

Elasticsearch Configuration

📍 Essential elasticsearch.yml Settings

# Cluster identification
cluster.name: production-cluster
node.name: es-node-01

# Node roles (ES 7.x+)
node.roles: [master, data, ingest]

# Memory
bootstrap.memory_lock: true

# Network
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# Discovery
discovery.seed_hosts: ["node-01", "node-02", "node-03"]
cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]

# Paths
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# Action destructive requires name
action.destructive_requires_name: true

⚙️ Performance Tuning Settings

# Thread pools
thread_pool.write.queue_size: 1000
thread_pool.search.queue_size: 1000

# Circuit breakers
indices.breaker.total.limit: 85%
indices.breaker.fielddata.limit: 40%
indices.breaker.request.limit: 60%

# Indexing performance
indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb

# Search performance
indices.queries.cache.size: 20%
indices.requests.cache.size: 2%

Common Configuration Issues

🚨 Critical: Heap Size Misconfiguration

Common Mistakes

  • • Setting heap >32GB (loses compressed OOPs)
  • • Different -Xms and -Xmx values
  • • Using >50% of available RAM
  • • Not leaving memory for OS cache

Quick Fixes

  • • Reduce heap to 31GB maximum
  • • Set -Xms = -Xmx for consistency
  • • Calculate: (RAM * 0.5) or 31GB, whichever is smaller
  • • Monitor heap usage trends

⚠️ Warning: Memory Lock Failures

Symptoms

  • • "Unable to lock JVM Memory" warnings
  • • Degraded performance under load
  • • GC pause time increases
  • • Swap usage in monitoring

Resolution Steps

  1. 1. Check ulimit -l (should be unlimited)
  2. 2. Verify /etc/security/limits.conf
  3. 3. Restart Elasticsearch service
  4. 4. Disable swap if possible
  5. 5. Monitor memory lock status

ℹ️ Info: File Descriptor Exhaustion

Detection

  • • "Too many open files" errors
  • • Cannot create new indices
  • • Network connection failures
  • • Indexing operations fail

Prevention

  • • Set ulimit -n to 65536+
  • • Monitor current FD usage
  • • Plan for growth in indices/shards
  • • Use index lifecycle management

Configuration Best Practices

✅ JVM Optimization

  • • Use G1GC for heaps >8GB
  • • Set heap to 50% of RAM (max 31GB)
  • • Enable heap dumps for debugging
  • • Use -XX:+AlwaysPreTouch for consistent performance
  • • Enable string deduplication

💡 System Tuning

  • • Disable swap or use swappiness=1
  • • Set file descriptors to 65536+
  • • Configure virtual memory areas
  • • Enable memory locking
  • • Use dedicated elasticsearch user

❌ Configuration Pitfalls

  • • Using default JVM settings
  • • Ignoring ulimit configuration
  • • Running on systems with swap enabled
  • • Not monitoring configuration drift
  • • Inconsistent settings across nodes

⚠️ Monitoring Points

  • • Heap usage trends and patterns
  • • GC frequency and pause times
  • • File descriptor utilization
  • • Memory lock status
  • • Configuration consistency across nodes

Configuration Excellence

Key Principles

  • • Consistency: Uniform configuration across similar nodes
  • • Optimization: Tune settings for your specific workload
  • • Monitoring: Track configuration effectiveness over time
  • • Documentation: Record all configuration decisions

Action Items

  • • Audit current JVM and system settings
  • • Implement configuration management
  • • Set up monitoring for key metrics
  • • Plan configuration testing and rollback
Previous: Nodes Info CheckNext: Node Stats Check