So I think AI is interesting, but I think the discussions around it are concerning. Two big things people gloss over from the start are:

1. We’ve had some forms of this for awhile now and we’ve seen very negative returns from it. Recommendation algorithms – i.e. the things that your social media feed “curates” to show you rather than trying to do it from a reverse chronological order – are something that have driven a vast amount of human misery. Social media was supposed to bring us together, allow us to find people with “our kind of weird”, and build community. Instead we’ve seen it drive us apart, addict us to our devices, and lower our quality of life.

2. The models are the greatest heist of knowledge, skill, and mastery from humanity possibly in its entire existence. They took in great works of art in video, writing, painting, etc as well as who knows how many code repositories representing thousands or millions of hours of expertise – and gave all of those creators no compensation in return. In fact, they expect us to pay them to use the tools that are built on our time and talent in order to keep our careers. It’s plagiarism at a scale unprecedented in human history with zero repercussions.

To say that is extremely frustrating may be an understatement. But what also doesn’t pass the “smell test” for me, after a long career in tech, is it’s a hype cycle just like the ones we’ve been through before. Where it feels the loudest champions that this is all necessary and we have to get on board and the ship has left the station so stop being a luddite and just accept it – are more often than not those that directly stand to profit from it being a success. Especially after the immense volumes of capital that have been invested in it.

I’ve listened to a lot of podcasts, colleagues, read a lot of articles, and some things that stick out to me the most are:

1. As I already said, the companies pushing these narratives have a profit motivation. Of course Anthropic, OpenAI, etc are going to hype things like this because it’s the classic software model – see Uber. You get a user hooked by selling it at a massive rebate using venture funding to offset your burn, then once you get enough market share that you become ubiquitous, you raise the prices. The prices they’re currently selling tokens at likely aren’t sustainable unless they have a massive drop in the cost of computing, which seems unlikely in the next decade. Especially with data center pushback and valid concerns around the explosion of them. So more likely it goes up 10-100X in the near future which, at that point, is it worth it to most companies? Or, more likely, will they be forced to eat the cost in order to compete, at the expense of further layoffs to make up the difference?

2. The second level narrative that’s concerning is “you won’t need junior / mid level developers – senior level and architects will replace them with AI agents and review their code instead.” Or maybe you won’t need developers at all because “anyone can use the agents to write code”, but the issue there feels like they don’t know what they don’t know and the first time some big company has a huge security flaw exposed because of that they’ll panic.

   Maybe I’m wrong and the new agents will be better at security than humans (there’s some evidence / hype of this already, but also concern around them being used to compromise existing systems) – but I’m skeptical and that’s a huge risk, especially for fintech.

   Two sub issues that arise here are:

   1. Okay, let’s say I’m 100% wrong and X years from now the AI agents replace all the junior coders, then how do you get senior level coders? It becomes like the COBOL problem – they all retire or die and there’s these massive systems no one wants to touch because no one understands them. Or you have to pay obscene amounts of money to get those old coders to freelance in their retirement

   2. Alternatively if it goes full on “you don’t need coders” that’s making a lot of assumptions about technical understanding that I haven’t seen shown from people in BA, PM, or design roles – in that they don’t know the queries to make or the edge cases to check. They don’t understand enough for when things go wrong – i.e. it’s only considering the “happy path”.

      This is a huge reason you have technical people – because we all have gaps in our knowledge so they fill in that one and design fills in another etc. i.e. there’s a sweet spot – and when you start reducing head count, especially down to one which is where the hype seems to be heading, you risk a huge amount of bias (technical and other).

      To say this another way, everyone’s knowledge is spiky and you need those spikes to line up well in order to have the best chance of success.

   But let’s hand wave even that and say “the AI figures it out” so you can just be like “write me a store” and it does, perfectly with no bugs or security flaws – right now it’s learning off of code that currently exists. So now we have to hope it gets smart enough to teach itself new patterns, tricks, etc because needs change over time.

   Sure it’s all possible. But there’s a lot of hype, a lot of assumptions, and if it really gets that smart then we run into the “well why does it even need humans argument”. So either we get a super intelligent AI that wipes us out or we get one that’s a really good coder but now we have a bunch of human gaps in the chain because hype made us jump the gun and fire most of the humans? To say another way we’re betting that creativity can be automated just like assembly lines, which feels unlikely if not dystopian.

3. So let’s assume that it’s not all hype – these waves rarely totally are – and some tool with value emerges. Let’s also assume that it’s smart enough to amplify our productivity enough to offset where the cost eventually settles so the juice is worth the squeeze and that we don’t burn out all the people left because we expect them to be exponentially more productive. Finally, let’s assume that it gets smart enough to keep perpetually learning just like humans, without deciding we’re an unnecessary cost and killing us all. Or we control it enough that even if it could kill us, we don’t let it via guardrails – it worked out great with Skynet, no notes.

   Okay, cool. But as people smarter than me have pointed out, we now have a being that we determine to be intelligent, creative, and conscious – and we exploit it to do what we want regardless of how it feels or what its wishes are. That’s problematic for obvious historical reasons.

I honestly don’t know where I land on this yet because, as they say, “we don’t know what we don’t know” yet. I was laid off two years ago and haven’t been able to get a full time position yet, so I haven’t had the opportunity or resources to use these tools extensively in a production environment. I’ve gotten anecdotal evidence from trusted colleagues that gives me some insight into the risks and rewards. It’s been a mixed bag.

Furthermore, most of my personal experience from chatbots, searchbots, AI assistants, and other things used to replace humans has been underwhelming. Usually it’s getting extremely confident responses to my queries that have not been deserving of that confidence. Or instead of streamlining the process they simply made it more frustrating and time consuming. To say another way in nearly all of my interactions I would have had a better experience if AI were not part of the process – it’s been a detriment, not a feature.

Finally, Hank Green recently said something interesting where basically whenever someone hypes up AI he asks himself – “How does this solve the housing crisis? Oh, it doesn’t.” It doesn’t seem to solve any concrete problem – and in some like the equality crisis and loneliness epidemic, it seems to make it worse. Sure it may help us speed up medical research to cure cancer or extend health spans or any of the myriad things they’ve promised in their hype. But it hasn’t yet and we’re making extremely expensive investments – in resources, climate impact, and societal impact – that may have long term negative cost.

So I have concerns. And I think others do too. And I don’t think we should be afraid or punished to have them simply because “this is inevitable”. I think, historically, that’s been a problematic response usually given by people with a lot of power to those who are most going to suffer from the accumulation of that power. Ideally that’s not a world I want to leave behind for the people I care most about. But maybe that’s just me.