Data Scientist

Description

The company needs a data scientist to work on data pipelines and reporting for a customer's cloud environment.

The role covers large-scale data processing, ETL design, data quality, and dashboards built from operating system and data logs.

Responsibilities

• Support the customer in defining requirements for new system capabilities.
• Extend and maintain a bulk data pipeline and enhancements across several applications in the customer's cloud computing environment.
• Map, extract, transform and load data, and document ETL designs with source-to-target mappings and data dictionaries.
• Turn operating system and data logs into reports, metrics and dashboards.
• Review data extensively and assess its quality.
• Work with customers and integration partners to pin down detailed objectives.
• Help Agile teams define tasks, scope and review work.

Requirements

• Hands-on experience with big data processing using Spark, PySpark and Python.
• Experience building analytic reports in CloudWatch and Kibana.
• Experience with regular expressions (RegEx).
• Experience with SQL, MySQL and PostgreSQL.
• Experience processing XML and JSON data files.
• Experience with IDEs and data modeling in notebooks and Visual Studio.
• Experience converting disparate structured and unstructured data into enriched, query-friendly structured data in indexed files.
• Experience working with customers and integration partners to clarify objectives.
• Experience supporting Agile development.
• Optional: experience deploying capabilities on the Databricks unified analytics platform.
• Optional: experience running big data workloads on Elastic MapReduce.
• Optional: experience joining multiple complex data sets with Spark.
• Optional: experience tuning Spark streaming and batch jobs for cluster utilization and speed.
• Optional: experience deploying complex, notebook-based pipelines.
• Optional: experience with Python data analysis libraries such as Pandas.
• Optional: experience with cloud services such as Lambda, SNS/SQS or EC2.
• Optional: experience with DevOps tools including CloudWatch, Lambda, SQS, DynamoDB and RDS.
• Optional: experience with the ELK stack (Elastic, Elasticsearch, Logstash and Kibana).

Conditions

Do not apply without an active Poly clearance; applications without one will not be considered.