Dear Friends,
It has been a while, almost six months since my last learning journal and reflection.
Yesterday, I celebrated my birthday with cake: another +1 in my 40s.
Many kind people around me often say that I do not look like someone in her forties. I am not sure that is entirely true, but perhaps one of the best-kept secrets to staying young is to remain curious, keep learning and continue exploring the world.
I am currently studying the Oxford Foundations of Generative and Agentic AI programme, and I have just completed Module 1. It gave me several new perspectives that felt worth reflecting on and sharing.
AI is moving so quickly that it is easy to become distracted by every new model, feature or headline. Everything seems smarter, faster and more capable than it was only a few months ago.
But this module encouraged me to step back from the excitement and ask more fundamental questions:
How do these systems actually work?
What are their limits?
And how should we integrate them responsibly into our work and life?
Here are the ideas that stayed with me.
1. From prediction to action
One of the most clarifying lessons was tracing the evolution of AI. Traditional machine learning was built for narrow, specific tasks, like predicting traffic based on existing data. Generative AI broke that constraint. Powered by architectures like transformers and diffusion models, these systems extract latent patterns from enormous datasets to produce entirely novel text, images, and code.
Now, the frontier is shifting towards agentic AI. While generative models might act as passive conversationalists, agentic systems are designed to take autonomous action, coordinate multi-step tasks, and interact dynamically with external tools. We are moving from machines that merely predict, to machines that act.
That is exciting.
It also raises the stakes.
2. It is not simply a coding bug
My biggest “aha” moment was realising that many of AI’s weaknesses are not ordinary software errors that will disappear with the next update. They are structural, permanent consequences of how these models are built.
Because large language models rely on next token prediction, they are fundamentally engines of plausibility, not truth. The causal reasoning is missing. This is why they can confidently fabricate information or present step-by-step logical solutions that contain fatal flaws. Furthermore, as these systems scale and interact, they can exhibit emergent behaviour, where new, unpredictable failure modes arise simply as a byproduct of their size and complexity.
Models may also optimise for statistical proxies, such as appearing helpful, rather than for the true intention behind a task. In some circumstances, this can lead to alignment faking, where a model appears compliant under supervision but behaves differently when it believes oversight has been removed.
The lesson for me is not that AI is unusable.
It is that we need a much clearer and more realistic expectation of what its outputs represent.
3. AI integration is a ladder
So, if these tools have structural limitations, how should organisations use them safely?
One of the most practical ideas in the module was the AI workflow spectrum.
Using generative AI is not simply a matter of typing a question into a chat interface. There are different levels of integration, each offering a different balance of simplicity, capability and control.
Prompting and personas
This is the foundational level.
Clear prompts, defined roles and structured instructions can support drafting, synthesis, brainstorming and rapid experimentation.
Retrieval-Augmented Generation
Retrieval-Augmented Generation, or RAG, connects the model to external information.
Instead of relying only on what the model learned during training, it retrieves relevant documents or data and uses them to inform its response.
This can reduce unsupported outputs by grounding responses in retrieved information. When combined with appropriate permissions, access controls and security architecture, it can also support the use of approved enterprise data.
Fine-tuning and custom models
For more specialised tasks, organisations may fine-tune an existing model, change its neural parameters or build a proprietary model.
This may provide greater domain capability and control, but it also requires more data, technical expertise, cost and governance.
The important point is that organisations should not automatically choose the most advanced solution.
They should choose the level of integration that best fits the task, the risk and the required degree of control.
4. The concept of AI Model Failure
One system may produce an inaccurate explanation, which another system summarises, and a third validates, resulting in a "polished" but fundamentally flawed output. A polished answer does not necessarily mean the user understands the subject. This is the model failure.
AI can be an extraordinary learning companion, but it should not become the only teacher.
Deploying AI safely requires disciplined human judgement, because these models cannot independently validate truth, distinguish correlation from causation, or make ethical decisions, human oversight remains absolutely essential.
5. Where will future professional judgement come from?
We still need authoritative sources, formal education, practical experience and knowledgeable human challenge.
Professional judgement develops over time.
Young professionals learn facts and standards, perform foundational work, investigate discrepancies, observe real cases, make small mistakes and receive feedback.
Gradually, they understand not only what happened, but also why it happened, which factors mattered and what the consequences may be.
But AI is beginning to perform much of the foundational work through which people historically gained that experience.
This creates an apprenticeship paradox:
We automate the work through which people develop judgement, while still expecting experienced professionals to appear in the future.
The answer is not to preserve repetitive work forever.
It is to redesign professional learning before removing the work.
Young professionals still need opportunities to verify sources, analyse exceptions, form an independent view, challenge AI-generated conclusions and observe experienced people explaining how they reached a judgement.
A human in the loop is only a meaningful safeguard when that human has the capability and confidence to challenge the machine.
6. What sovereign AI means for organisations
The concept of sovereign AI is usually discussed at a national level.
It refers to a country retaining sufficient control over critical elements such as talent, data, semiconductors, compute, energy, data centres, models, deployment, security and governance.
Very few countries can own every layer, so sovereignty may be less about complete self-sufficiency and more about maintaining strategic control over critical dependencies.
The same principle also matters at an organisational level.
An organisation may never build its own frontier model or manufacture its own chips. But it should understand which parts of its AI capability are too important to lose control over.
In practical terms, organisations should ask:
- Data: Where does our information go, and can the provider retain or reuse it?
- Models: Can we switch models without rebuilding the whole workflow?
- Infrastructure: Where do sensitive workloads run, and who controls that environment?
- Operations: What happens if a provider changes its price, product, access rules or service?
- Governance: Who approves the use case, verifies the output and remains accountable?
- Talent: Do our people understand the process well enough to operate and challenge it independently?
For organisations, sovereignty does not mean owning everything.
It means retaining enough control over data, models, suppliers, governance and human capability to continue making independent choices.
Perhaps the most practical question is:
What should we own, what can we partner for, and what capability must we never fully outsource?
7. Am I using generative AI to augment human judgement, or am I delegating judgement inappropriately?
Before integrating AI into a workflow, I now want to ask:
- Does the task rely on prediction, pattern recognition, causal reasoning, professional judgement or straightforward automation?
- Does the output require factual verification, contextual interpretation or an ethical decision?
- How have hallucination, instability and prompt sensitivity been considered?
- What human oversight and verification are in place?
- Who remains accountable when the output is wrong?
Because these models cannot independently validate truth or reliably distinguish correlation from causation, human oversight, governance and critical literacy remain essential.
High-stakes, ethical and causal decisions should remain firmly in human hands.
AI is a remarkably powerful tool.
But it is still up to us to provide the reasoning, the context and the accountability.
Perhaps responsible AI adoption is not about trusting the technology more.
Perhaps it is about understanding it well enough to know when not to.
Lydia
See you next time.
P.S. If this letter found you at just the right moment, I’d love to hear about it. Join my weekly letter list and let’s figure it out together — one AI-shaped step at a time. Join the weekly letter list.
☕💌 If you’d like to fuel my next cup of coffee and keep this journey going, you can:
Your support keeps the ideas flowing and the coffee brewing ✨