Numerical & Compute Algorithms Engineer

Ensigncode is hiring a Numerical & Compute Algorithms Engineer to own algorithm and precision decisions: selecting and adapting algorithms for matrix and sparse operations, and validating numerical correctness, error bounds, and stability across FP32, FP16, BF16, and lower-precision paths. You'll work with kernel engineers to ensure speedups never compromise correctness.

Lower precision almost always means faster, and it almost always means riskier. As Numerical & Compute Algorithms Engineer, you’ll own that trade-off: selecting and adapting algorithms for matrix and sparse operations, and rigorously validating numerical correctness, error bounds, and stability as precision drops from FP32 toward FP16, BF16, and beyond.

You’ll build the correctness tests and reference comparisons that let the rest of the team move fast on precision changes without accidentally shipping a quietly wrong result. This role draws on real linear algebra and numerical analysis background, not just general software engineering.

Responsibilities

• Select and adapt algorithms for matrix and sparse operations, including precision, accumulation, and data-layout choices
• Validate numerical correctness, error bounds, mixed-precision behavior, and stability across representative inputs
• Evaluate FP32, FP16, BF16, and supported lower-precision paths as appropriate to the target GPU and workload
• Develop correctness tests and reference comparisons; diagnose accuracy/performance trade-offs

Core requirements

• Linear algebra; numerical analysis; matrix algorithms; C++ or Python; familiarity with GPU compute and mixed precision
• Experience with scientific computing, ML systems, or numerical libraries is relevant

Team contribution

Works with kernel engineers to ensure speedups do not compromise correctness, and guides precision decisions.

Works most closely with