Device integration is hard. Every control system engineer knows this.
If we’re lucky, the device we’re trying to run via EPICS is a mass-produced, off-the-shelf bit of kit that’s already been integrated. In this case, it’s relatively easy - there’s a high chance that the support exists somewhere in the popular repositories. It’s then “just" a matter of installing, testing, and we can call it a day.
However, more often than not, the device we need to integrate comes from a small company that’s a leader in a very niche field of science or engineering. They excel at building specialised equipment, and since their target customers are equally focused on the physical components, they’re very successful.
What goes on behind the company’s scenes might shock you. Someone who really shouldn’t be involved in software development is attempting to cobble together something that ticks a box in the tender requirements without paying much attention to good software practices, architecture, technical debt, or any other “fancy terms” software engineers like to throw around.
Scientific systems are built with a “hardware-first” mentality, so no one kicks off engineering meetings with a deep dive into the software details. As a result, the quality of the control software attached to this shiny, cutting-edge device is… well, let’s just say it’s not exactly a priority for the company.
Whether this is okay or not is another discussion. As a control system engineer, I’m used to this world. I’ve accepted that basic control system software is a liability, not an asset, and it should be applied sparingly—like glue—just enough in the right places.
Why do I bring this up?
While we have complete control over the quality of the overarching control system, the quality of the artefacts shared by the device’s manufacturer is often, shall we say, questionable.
And the fun starts with the documentation. If you think that the docs for a device so expensive it could clear your mortgage are detailed, well-written, and factually accurate - think again.
Expect missing data, copy-paste errors, crucial details left out, and a writing style that assumes you already know exactly how the device works. This is all par for the course.
So, you’re left with a device that lacks proper documentation, doesn’t work as intended, and the manufacturer’s “expert” has no clue what’s going on.
How do you deal with this train wreck?
Adjust you expectations.
Assume that the documentation is wrong. Assume that the software is half-baked, compiled seconds before the release, and barely tested. You’re dealing with a black box. Treat the situation accordingly.
Plan.
Make sure your team knows the risks. Don’t waste time explaining technical details - they won’t care. If something is going to take longer or cost more to fix, flag it early and let them handle the fallout.
Divide and conquer.
A classic.
Test your assumptions.
Are you sure the problem is on the manufacturer’s side? Test your software thoroughly. Prove your own code is sound before pointing fingers.
…and there’s probably more I could add, but I’ve got to sort out this cutting-edge, expensive paperweight…