Splunk Observability Engineer

Job Description

Job Description

For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to denis.potapenko@systemone.com - make sure to include the exact job title and job location in your email message. *** no C2C with 3rd parties, no contractors - W2 salaried candidates only *** Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer ) *** Must be able to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL

Splunk Observability Engineer : Hands on role supporting large Splunk and observability platforms in an AWS production setting.

• Run, upgrade and support Splunk Enterprise on AWS (EC2)
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
• Write and tune complex SPL queries
• Build dashboards for monitoring and incident triage
• Fix ingestion failures, missing data, volume spikes and outages
• Restore and replay data after interruptions
• Reduce noise, storage use and cost
• Connect Splunk with CloudWatch and OpenTelemetry
• Build and maintain OpenTelemetry agent and collector packages
• Automate tasks with Python
• Handle patching and security fixes
• Lead troubleshooting calls across teams
• Take part in on call rotation

Requirements

• Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
• Advanced SPL skills
• Strong AWS experience: EC2, ALB, CloudWatch
• Hands on OpenTelemetry experience, including Splunk integration
• Python for automation
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
• Strong troubleshooting and clear communication skills