The best systems engineers turn major challenges into opportunities for them to evolve their skills, both technical and nontechnical.
For instance, a senior engineer at time-series application company InfluxData said he chased down a very rare race condition in a customer use case and in doing so, acquired new field knowledge. Meanwhile, Verkada’s chief systems architect said his team of systems engineers at the cloud-based security solution provider are evolving how they collect key metrics, a project that’s introducing engineers to new levels of technical scale.
But beyond analysis, systems design and troubleshooting, successful systems engineers have to be effective communicators.
The experienced San Francisco professionals we spoke with said that systems engineers must work to stay aligned with key stakeholders so that daily tasks like problem-solving and determining requirements are done in a timely, respectful manner. And part of being communicative means documenting use cases and best practices so that the rest of the team can benefit from past experiences.
Verkada Co-Founder and Chief Systems Architect Benjamin Bercovitz said he’s been busy optimizing how metrics for the company’s security hardware and software products are collected. In doing so, he is exposing systems engineers to new levels of technical scale.
What are some key tasks on your to-do list?
The day begins with a glance at fleet metrics. If something is out of place then it will demand my undivided attention. Our myriad of charts serve as an early warning system for potential software issues and I’ve often left a load or stress test running overnight to see if something is working. For example, if I’m integrating a new smart motion search collector for our camera product, I check for errors and ensure the service output looks reasonable.
After that, I check in with the team to make sure no one is blocked, as this slows down momentum. Then I focus on a specific program or problem. I may need a new build, then I’ll continue with another part of the code. That code can be a long cycle, so I pack as many debugging pieces as possible, ranging from extra debug prints to adding debugs if it’s a kernel change. Most changes are in userland programs rather than in the kernel, so having the ability to do a thread stack trace or a memory dump is key.
Blaze your own trail and find a way to hack through all the barriers on your own.”
What’s a recent project that’s been exciting to work on?
One of our big initiatives is metrics and telemetry. One value-add of having a cloud-managed embedded device is knowing that our engineers are monitoring the health of the software and are responsive with patches for any issues.
I’ve been revamping how we collect stats on the system. Previously, we sent a JSON file to an endpoint with frequency to report, but we needed to add more diagnostic metrics due to high-traffic volumes. JSON was inefficient, so we developed a better way of collecting and pre-aggregating data. We’re working on sending a columnar table from the device for efficient data processing once on the cloud. Multiple components needed updating, from back-end APIs to device software to data workflows feeding our camera diagnostics data warehouse. Anything running on the device must run on a tight budget because there’s only so much RAM, CPU and space on the rootfs. This project is the crossroads of two challenges in our embedded systems: efficiency on hardware and monitoring at a scale beyond what most back-end engineers have seen.
What’s an important skill a systems engineer needs to be successful?
Persistence. Many systems have scant documentation, examples or support. Searching online will often give you zero hits. Even if you have access to an expert, they will not have time to show you everything. You need to blaze your own trail and find a way to hack through all the barriers on your own.
Good communication is just as important as technical ability for Andy Charlton and his engineering team at InfluxData, he said. Working on challenging projects and learning from them requires team members to always be on the same page, Charlton added.
What are some key tasks on your to-do list?
We’re responsible for not just developing features but also testing and supporting them when they’re in production, so my days vary considerably depending on what is needed at the time. I’ll often be working on new features or improvements to the system like planning, writing code or testing. If we have issues in our production clusters or a customer reports an urgent issue, then I may need to help get those challenges resolved. Code reviews are another key task on my to-do list.
The ability to communicate well is essential to be successful as an engineer.”
What’s a recent project that’s been exciting to work on?
I recently picked up an issue from a customer that reported some inconsistency in their data. The case was hard to reproduce and even harder to pin down. But eventually, we found that it was a rare race condition that only occurred when one of our Kubernetes pods restarted, and luckily no customer data was lost.
I really enjoyed working my way through the stack, testing my knowledge, reasoning skills and learning a lot in the process. After several false epiphanies, being able to finally pin down the exact cause and reproduce it on demand was an incredible feeling!
What's an important skill a systems engineer needs to be successful?
Outside of the obvious technical skills, I think the ability to communicate well is essential to be successful as an engineer. Distributed working and asynchronous communication seem to be more common these days. Communicating well both verbally and in writing helps so much in making sure that the team is on the same page and everyone is working on the right things.
