Straight from the bench

It started with a perfectly reasonable test: extend the arm to 60 degrees, hold, retract. Simple. We had done it eleven times that afternoon. Then on the twelfth, something in the joint controller decided it was done cooperating with the laws of Newtonian mechanics.

The arm rotated to 60 degrees. Then to 90. Then past 120 with the kind of confidence that only comes from firmware that has completely lost the plot. Kristóf reached for the emergency stop. The arm reached for the wall. The wall, to its credit, did not move.

Post-mortem: a sign flip in the feedback loop introduced during the previous session's "quick refactor." The arm had been commanding itself to go further whenever it detected resistance — fight-or-flight, but all fight. We have since added a software travel limit, a hardware travel limit, and an unspoken agreement to never speak of the twelfth run.

Status: arm operational. Ego: partially recovered. Wall: fine.

After the third incident this quarter, we determined that "it exploded" was not a sufficiently precise engineering description. We have since adopted a formal taxonomy.

Class I — The Whisper: A single component desolders itself through the application of too much imagination. Detectable only by smell. No drama, high confusion.

Class II — The Pop: An audible event. Often a ceramic capacitor. Everyone in the lab freezes for 1.5 seconds, makes eye contact, then returns to work. Normal.

Class III — The Flash: Visible light. Brief. Typically accompanied by one of us saying "hm" and writing something down. The board is usually salvageable if you squint.

Class IV — The Spectacle: Smoke, flash, a small radius of scorched PCB, and at least one person saying "okay that was my fault." Post-mortem required. Pizza ordered.

Class V — The Story: The kind that ends up in a slide deck. We don't talk about Class V during business hours.

Entry #218 was a Class II trending toward Class III. Capacitor on the power rail, wrong voltage rating, classic. The explosion was within the acceptable energy budget. We are calling this a pass.

This is not a new problem. The left wheel has been antagonistic since Robochallenge 2023, when it decided mid-match to express its feelings about the motor driver by drawing 4 amps more than specified and then going completely silent. We lost on time. We blamed the field conditions.

We were wrong to blame the field conditions.

Since then: two loose encoder connectors (left side only), one unexplained stall during a demo in front of people we were trying to impress, and one instance where it simply slipped — not mechanically, not electrically — it just chose to slip. The right wheel has never slipped. The right wheel is fine. The right wheel has never given us a single reason for concern.

Current theory: the left wheel is sentient and unpleasant. Atena's theory: we over-tightened the hub mount in 2023 and induced a microscopic shaft bend that accumulates error under load. We are logging both theories as equally valid pending further evidence.

Recommendation: treat the left wheel the way you treat a colleague who has been wrong before. Trust, but verify. Immediately verify. Actually just verify.

For the record — because records matter and amnesia is a design flaw in humans — Atena was correct about the LDO placement.

The argument: Kristóf wanted the regulator on the underside of the board to "keep the top clean." Atena said thermal performance would suffer and we'd regret it. Kristóf said the thermal model showed sufficient margin. Atena said the thermal model was optimistic. Kristóf said the thermal model was fine. This exchange repeated for approximately 45 minutes across two days and one dinner.

Result: the LDO ran 18°C hotter than modeled at full load. We had to add a copper pour and a small heatsink. The heatsink is ugly. The board no longer has a clean top.

Atena has not said "I told you so." She has, however, maintained prolonged and meaningful eye contact every time someone mentions thermal modeling. This is worse.

Running tally this year: Atena 7, overconfident thermal models 0. We are logging this publicly so future-us has documentation and cannot pretend it didn't happen.

03:47. The rail is holding at 3.298 V. Variation under 2 mV across load steps. This is, objectively, excellent.

The person reading the oscilloscope has been awake for nineteen hours, has consumed an unknown quantity of coffee described only as "enough," and made two errors in the past hour that were caught only because the multimeter beeped. The multimeter has better judgment than the engineer at this point.

The prototype is stable. The engineer is not. This is a recurring pattern and we should probably address it, but tonight is not the night for that conversation.

04:12. Decided to run one more thermal soak cycle before calling it. This is almost certainly a mistake. Logging it anyway for accountability.

04:58. Thermal soak done. Nominal. Going to sleep. Tomorrow-me will find this note and will know that tonight-me made the right call in the end, but also that tonight-me probably should have made it three hours earlier.

Status: power system: certified. Engineer: pending recertification after maintenance window (sleep).

There is a bug in the IMU data pipeline. We know it exists because the rover drifts right under autonomous heading hold — not always, not on command, just whenever it feels like it. We have 40 minutes of logged telemetry proving the drift. We have zero footage of it actually happening, because every time someone stands near the rover with a camera, it drives straight.

This is not a metaphor. This is a genuine engineering situation.

Hypothesis 1: EMI from the camera or phone induces a transient in the magnetometer. Plausible. Tested with the camera 10 meters away. Still drives straight.

Hypothesis 2: The act of observation changes the operator's grip on the deadman switch, subtly altering the electrical noise floor on the control lines. Farfetched but we're running out of ideas.

Hypothesis 3: It knows. It just knows.

We've started leaving a decoy setup running in the corner with a camera pointed at it, and doing the real testing without any observers in the room. This seems to help. We are not proud of this. We are logging it because honesty is the only engineering virtue that doesn't require calibration.

The spec calls for operation up to 55°C ambient. We do not have a thermal chamber. We do have a car park in September.

Kristóf's car reached an interior temperature of 61°C by 14:00. We zip-tied the board to the dashboard air vent (off), ran it from a power bank, and left a data logger on it for four hours while we sat inside and had an argument about whether this constitutes a valid test methodology.

It does not constitute a valid test methodology. We did it anyway.

Results: the board ran warm. The solder joints held. The oscillator drifted by about 80 ppm at peak temperature, which is within the datasheet spec if you choose the right column and don't look too hard at the footnotes. The car's interior plastic smelled strange afterward but that was unrelated.

Conclusion: the hardware is probably fine up to 55°C. We have logged this as "field-validated thermal performance" in our internal documentation. If anyone asks, we conducted a "thermal stress evaluation in a controlled elevated-temperature environment." The car park was the environment. The sun was the heater. Suffering was the calibration.

The ground plane on revision 3 was sparse. This was an intentional decision made during layout to reduce board weight and give more routing room. Revision 3 had a noise floor 12 dB higher than expected and three digital signals that were clearly having a conversation they shouldn't be having.

Revision 4 has a solid ground plane on both layers. The traces look like they're floating in a calm lake. The EMI scan came back 9 dB better than spec. The board has a quiet confidence that the previous revisions lacked.

Kristóf refuses to acknowledge that this was "listening to Atena about the ground plane" and insists it was a planned revision to address parasitic inductance. Atena has updated her running tally. It is now 8.

There is a deeper truth here about ground planes that took us longer to internalize than it should have: copper is not just a conductor. It is a statement of intent. A solid ground plane says "I have thought about return currents." A sparse one says "I have other things to deal with right now." The board knows which one you meant.

Addendum: we also added a 100 nF decoupling cap to U7 because it seemed lonely too. No measurable effect. No regrets.

A formal defense of the smoke test as engineering methodology:

The smoke test is honest. It does not tell you whether the board meets spec. It tells you whether the board survives first contact with reality — which is, ultimately, the only spec that matters. A board that passes every simulation and fails on power-up has failed at the only test that was ever real.

Smoke is information. No smoke: probably fine. Small amount of smoke: something is wrong but the board wants to tell you about it on its own terms. Large amount of smoke: the board is expressing a strong opinion about your schematic. Flame: the board is done talking.

We have smoke tested 47 boards this year. Three smoked on first power (two were our fault, one was the fab's). The remaining 44 entered service with varying degrees of drama. The three that smoked are better boards now for having smoked. The drama was data.

Other QA methodologies are fine. They are also longer. The smoke test is the one test that the board runs on itself, at its own pace, the first time it is given power. There is a purity to that. We document every smoke event. We have never thrown the data away. The smoke is the record.

December in the lab. The building's central heating is doing its best, which is not very good. The bench is at 14°C by 9am, and the solder paste refuses to flow properly and honestly we can't blame it.

The reflow oven runs a profile that peaks at 245°C and holds at 200 for 45 seconds. After a board run, the residual heat from the oven brings the immediate vicinity up to a very comfortable working temperature for approximately twelve minutes. We have adjusted our workflow to take advantage of this.

New process: run a board, bask in the warmth, inspect the joints while the feeling returns to your fingers, start the next board. Output has not decreased. Morale has improved noticeably.

Atena has called this "thermodynamic multitasking." Kristóf has pointed out that the oven is not designed to run more than 6 cycles per hour and we are stretching that. Atena has pointed out that we are also not designed to run more than 6 hours without heat and we are stretching that too.

A truce has been established. The oven runs at a sustainable cadence. The lab is warm enough. The boards are good. This is the best December we've had since 2022, when we had a different oven and worse boards and were somehow more optimistic about both.

We went in knowing we were the youngest team. We left knowing we were the fastest.

The challenge involved processing telescope image data in near-real-time on constrained hardware — which is, coincidentally, exactly what we had spent the previous eight months building Coral to do. The universe has a sense of humor, or at least decent timing.

By hour four, two teams were still reading documentation. By hour six, Kristóf had the pipeline running end-to-end on a Coral cluster pulling from a live data stream. By hour eight, we had added a visualization layer that was, if we're being honest, not required but very impressive.

A postdoc from somewhere with better funding asked how we had built the data pipeline so fast. We said we had practice. We did not elaborate. The practice involved many late nights, some smoke, a rover arm that punches walls, and a left wheel that should not be trusted. We did not feel this context would improve the conversation.

Final result: we didn't win the official prize because we had technically violated a constraint about output format. We had produced better output in a non-compliant format. We are choosing to interpret this as a victory. This is logged as a victory.

The bug had been in production for six weeks. It caused the heading controller to accumulate a small positive bias over time — about 0.3 degrees per minute — which meant that after twenty minutes of autonomous operation, the rover would be navigating toward a destination approximately six degrees from where it thought it was going. This sounds small. On a 50-meter course, it puts you in a hedge.

Finding it took four days. It was a floating-point accumulation error in the yaw integration, the kind that only shows up over long runs, only matters in narrow heading corridors, and is completely invisible in unit tests because unit tests don't run for twenty minutes.

The fix was four lines of code. The four lines took one hour to write and three days to be confident about.

We deployed at 04:17 because that's when the confidence arrived. The commit message says "fix yaw drift in long-duration heading hold." The real message would have been longer and would have contained the phrase "finally" multiple times.

Test flight at 08:00 the next morning: straight as a laser. Kristóf sent Atena a voice memo that was just silence for three seconds and then the word "yes." She understood immediately. This is what love looks like in embedded systems development.

It started, as many great conflicts do, with a bill of materials review.

Kristóf: "We should use a 100µF MLCC here for lower ESR." Atena: "The capacitance derates 60% under bias voltage, we'd need four of them." Kristóf: "That's still better than a leaky electrolytic at temperature." Atena: "The electrolytic isn't going to leak if we pick the right one." Kristóf: "Define 'right one.'"

This exchange has since expanded to cover: the fundamental nature of capacitance, the ethics of vendor datasheets, whether LTspice models are "basically real," and one memorable tangent about the philosophy of component selection as a reflection of engineering character.

We have not resolved it. We are using both. The board has two capacitors where one would do, they are in parallel, they are each contributing something, and the power rail is very clean and stable and completely unaware of the argument that was fought on its behalf.

The current status: cold war, maintained by mutual respect and the shared goal of shipping hardware. If you ask either of us what the other would say about capacitor selection, we can tell you in detail. This is either a sign of deep technical alignment or deep personal familiarity with the opposing position. Possibly both.

It happened organically, the way these things do. Margot had been on the bench for three years without a name, faithfully rendering 400 MHz of truth onto a 12-bit display, never complaining, never lying, occasionally the only instrument in the lab giving us accurate information while everything around it was falling apart.

During a late session in April, Kristóf said "Margot agrees the signal looks clean" and neither of us questioned the grammar of it. She had clearly earned it.

Margot has since acquired a character. She is dependable, precise, occasionally dramatic about trigger settings, and never wrong about what she shows you, even when you wish she were. If Margot says the ringing is there, the ringing is there. If Margot says the rise time is 8 nanoseconds, it is 8 nanoseconds. You can disagree with your own interpretation. You cannot disagree with Margot.

The multimeter remains unnamed because it has given inconsistent readings twice and does not yet deserve it. The power supply has been called several things that will not be repeated here.

We are aware that naming instruments is not a formal engineering practice. We are doing it anyway. Margot has earned her name and we stand by the decision. This is logged for posterity.

Haiku, composed in the field, approximately 11:23am:

she moves across gravel sixty kilos of ambition hits the single cone

Incident Report: At 11:21, Lunomyss completed her first unaided traverse of the test course — a 30-meter run over packed gravel, two heading corrections, one gentle slope. Nominal.

At 11:23, during the return leg, a single traffic cone placed as a course boundary at the edge of the designated area became relevant. The operator (Kristóf) made a joystick input that he describes as "corrective" and Atena describes as "the opposite of corrective." The rover and the cone exchanged contact at approximately 0.4 m/s.

Damage to rover: none. Damage to cone: it moved. Damage to operator confidence: moderate.

The rover continued, completed the return leg, and stopped exactly where she was supposed to stop. The cone was retrieved and restored to its original position. The incident was declared over.

Analysis: Lunomyss performed exactly as designed. The cone performed exactly as designed. The human in the loop had a bad moment. This is consistent with literature.

Next field test scheduled for May 17. The cone will not be invited back.