> Here’s the thing: most of our habits at work still assume that producing something means you have to understand it.
This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.
You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers.
AI systems are not the same. You can't guarantee deterministic output.
Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.
Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.
Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not.
When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.
Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned.
And if I can't say that for a coworker, well, that's performance feedback.
Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.
>This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level.
Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.
I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.
Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.
I'd argue the trust was always relative. Witness what happens when you are hired to untangle the buggy mess that was left by the previous "senior" engineer. Sure there was trust when they were in charge, but based on the code they wrote, they didn't actually understand anything they were doing.
I don't think that's the problem the source article is describing. I've seen a number of instances now where people produce work product they don't understand at any level. "Hey Claude, design this feature and send it out for review, thx!" will often seem to work if you don't know any better.
You and I have a baseline. A brain that did most of it’s developmental phase while there was still some unified accepted version of truth (even if dubious)
Future generations will have to develop in a more splintered reality. A world where there is nothing to hold on to. Not even a dubious commonly accepted narrative
I think one way to approach this is by deliberately choosing less capable tools (but still AI-assisted). You don't have to delegate huge tasks to agents wholesale, but it requires fighting against mainstream harnesses to do so because they exist to sell you more tokens. Model providers are not incentivised to offer less autonomous flows.
I wrote about this frustration a few months ago [1], then realised that if the model providers won't do it perhaps I should try fixing it myself. So, shameless plug klaxon, I wrote an open-source harness that tries to work more like pair programming and less like code review [2].
As usual, my "Show HN" post [3] sank like a stone, so I think perhaps most engineers aren't really interested in this direction. It is possible though, we don't have to just accept whatever tooling the labs throw at us.
I wonder when the people will get that intelligence is not only directed at the external, but only really starts when you look at the internal (joy, pleasure, traumas, taboos, awkwardness, abuse, trust etc.). Look up the word "interoception".
Every generation has had tech doomers. Every single one. History repeats. Yet we collectively move forward. This time it will be no different. These folks are wasting time when they could be doing something productive.
plenty of ex-civs had techdoomers who were right. When tevh doomers have no salient points to make, its fair to not listen, but this time around i think something is different to historic doomerism.
Trust is a fundamental thing, and i cannot fathom how a post-trust society could possibly host life as we know it. Life ad we know it comes from systemic efficiencies that are rooted in trust, requiring it deeply. no trust means extreme loss in system efficiency, in a system already creaking under the straign of supporting current populations.
post trust is in my reakoning a serious issue to engage with.
There are plenty of low-trust societies. They have things like armed security guards and barbed wire fences outside every apartment complex. Coughsouthafrica
No one made people like Sam Altman into a compulsive liar. No one made Musk call a service full self driving, and then make it return liability to the user the second it decides an accident is about to happen.
Stop removing agency from the people making the top level decisions and reaping all the rewards.
Would you claim that someone who took 5 shots in a row and drove home wasn’t responsible for an accident because the car was what physically did the damage?
Yeah, I think what they really mean is people, especially those with power, broke this thing called trust and now they're trying to use """ai""" as a scapegoat.
It'd be along the same lines as saying "I was drunk driving 75km/hr over the limit down a busy suburban road during the middle of the day and 5 people died because of it so we CAN NOT TRUST CARS"
This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.
AI systems are not the same. You can't guarantee deterministic output.
Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.
Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.
When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.
Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
And if I can't say that for a coworker, well, that's performance feedback.
Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.
Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.
I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.
Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.
You and I have a baseline. A brain that did most of it’s developmental phase while there was still some unified accepted version of truth (even if dubious)
Future generations will have to develop in a more splintered reality. A world where there is nothing to hold on to. Not even a dubious commonly accepted narrative
I wrote about this frustration a few months ago [1], then realised that if the model providers won't do it perhaps I should try fixing it myself. So, shameless plug klaxon, I wrote an open-source harness that tries to work more like pair programming and less like code review [2].
As usual, my "Show HN" post [3] sank like a stone, so I think perhaps most engineers aren't really interested in this direction. It is possible though, we don't have to just accept whatever tooling the labs throw at us.
[1]: https://philbooth.me/blog/agentic-coding-and-mental-models
[2]: https://www.opairdev.org/
[3]: https://news.ycombinator.com/item?id=49625009
Trust is a fundamental thing, and i cannot fathom how a post-trust society could possibly host life as we know it. Life ad we know it comes from systemic efficiencies that are rooted in trust, requiring it deeply. no trust means extreme loss in system efficiency, in a system already creaking under the straign of supporting current populations.
post trust is in my reakoning a serious issue to engage with.
No one made people like Sam Altman into a compulsive liar. No one made Musk call a service full self driving, and then make it return liability to the user the second it decides an accident is about to happen.
Stop removing agency from the people making the top level decisions and reaping all the rewards.
Would you claim that someone who took 5 shots in a row and drove home wasn’t responsible for an accident because the car was what physically did the damage?
An increasingly large percentage of people being lying POS grifters is breaking trust.
AI just happens to be a tool that the laziest of that group of people leans on a lot to supercharge their grifts.
But our massive and growing trust problem predates and is way deeper than AI.
It'd be along the same lines as saying "I was drunk driving 75km/hr over the limit down a busy suburban road during the middle of the day and 5 people died because of it so we CAN NOT TRUST CARS"
And everyone stands up and claps