Recently, I came across a post that caught my attention. It claimed extraordinary time and cost savings achieved through Amazon’s AI coding assistant, Q, for a Java upgrade project. The post described upgrading from Java 11 to 17, with results that seemed almost too good to be true:



I see a LOT of companies claiming to have “automagical” solutions and the reality is they don’t talk about the amount of work behind the scenes to get these automagical solutions to actually do what they say it will do.

The Claims

1. Upgrading a single Java application from version 11 to 17 was reduced from 50 days to just a few hours using Amazon Q.
2. The company saved 4,500 developer years of work.
3. The upgrade to Java 17 reportedly saved $260 million through improved efficiency and security.
4. They fail to publish any strategies to accomplish this and how long it took to set up training data (and what type of training data they used and the overall approach – did they upgrade from 11->12, 12->13 or was it a straight shot 11-17.)
5. They fail to discuss the testing infrastructure in place for this project. Most companies of this size and scale have fully automated testing suites.

My Analysis and Counterpoints

While these claims are attention-grabbing, I believe it’s important to approach them with a healthy dose of skepticism. Here are some key factors I think we need to consider:

1. The Hidden Costs of AI Implementation

What I find notably absent from this success story is the time and resources invested in setting up Amazon Q for this specific use case. From my perspective, it’s highly unlikely that this was a simple “plug-and-play” solution. I suspect considerable effort likely went into:

• Customizing Amazon Q for the specific codebase and upgrade requirements
• Training the AI on the company’s coding standards and practices
• Developing custom workflows and integrations

In my estimation, this preparation phase would have taken months, probably involving close collaboration with AWS teams to optimize the tool for this particular upgrade project, something that most companies don’t have access or resources to support.

2. The Reality of AI Coding Assistants

As someone who has personally tested Amazon Q and uses other AI coding assistants like Claude, I can attest that the out-of-the-box experience with these tools often falls short of the hype. In my testing, I found Amazon Q, in particular, has limitations in its current form:

• The chatbot functionality was a huge disappointment. It can’t generate code, it creates “strategies for coding”.
• The plugin to autocomplete code is similar to existing tools like GitHub Copilot
• I found the documentation and setup challenging, It took me over an hour to figure out how to set this up. The documentation was not clear and there were many setup steps that weren’t intuitive.

In my experience, successful implementation requires:

• Skilled developers who understand how to effectively prompt and guide the AI
• A solid understanding of the codebase and upgrade requirements
• Careful review and testing of AI-generated code

4. The Power of Targeted AI Assistance

Despite these caveats, I do recognize the immense potential of AI in software development. In fact, I’ve been leveraging AI to create Python automation scripts for over a year now, and my experience has been eye-opening. I’ve been using Claude.ai to build code from scratch, with remarkable results. What I’ve discovered is that the key to success lies in how you structure your instructions to the AI. I’ve developed a method where I provide:

• A clear schema for the desired output
• Code examples that illustrate the expected result
• Detailed UX guidance and rules for the script’s behavior

With this approach, I’ve been able to get Claude.ai to generate comprehensive, functional code that meets my specific requirements. It’s not just about asking the AI to write code; it’s about providing the right context and guidelines to steer its output effectively. This experience has shown me that when used correctly, AI coding assistants can be incredibly powerful tools. However, it’s crucial to understand that the quality of the output depends heavily on the quality of the input. You need to know how to “speak the language” of the AI, so to speak. In my work with Claude.ai and other AI tools, I’ve found that the key to success is to:

• Keep the scope narrow and well-defined for each coding task
• Provide clear context, requirements, and examples
• Iterate and refine the prompts based on the AI’s output
• Review and test the generated code thoroughly

This approach has allowed me to create complex automation scripts efficiently, often in a fraction of the time it would take to code them manually. However, it’s important to note that this process still requires a solid understanding of programming principles and the specific requirements of the task at hand.

3. My Experience with Amazon Q’s Limitations

While my experience with Claude.ai has been AMAZING, I’ve found that not all AI coding assistants are created equal. To illustrate some of the current limitations of Amazon Q, let me share this disappointing response I received when asking it to perform a task similar to what I’ve been successfully doing with Claude:

My Conclusion

Based on my hands-on experience with AI-assisted coding, I believe that while their claims about Amazon Q’s underestimate the amount of effort that it takes to train this solution, the potential of AI in software development is undeniable. However, the true value lies not in miraculous overnight transformations, but in the thoughtful integration of AI tools into existing development processes and the skill to effectively guide these tools. As I continue to explore and push the boundaries of what’s possible with AI in coding, I’m convinced that maintaining perspective is crucial. From my experience, AI coding assistants are indeed powerful tools, but at this time you have to not only need to have development skills to use them, but you also have to have strong prompt engineering skills. I see the future of software development as a collaboration between people and AI, rather than a complete takeover by artificial intelligence. It’s exciting to see the possiblity that one day soon, non-developers will be able to build their own software without hiring an IT team.