Why code generation platforms eventually fail

I have gotten the opportunity to work at 2 startups that centered around enabling non-technical individuals (founders, product managers, e.t.c) to create software without really open an IDE or writing code. Think of platforms like Replit.

At face value, this seems like a massive value proposition given the fact that most of the overhead costs of starting a software project can or could be “eliminated” (at least at the start).

Well, you guessed it. The users of these platforms begin typing away.

Create a social media app for pet owners.

Fix the error. Make no mistakes!!!.

You are a senior software engineer. Build me a SaaS app for invoicing.

As you can tell, the prompts are very lackluster - which leads me to the very first reason why code generation platforms do not generate production ready code.

1. Poor prompt quality and understanding

The users of these platforms are most times not software engineers. For example, they do not know what an accordions is. So explaining to the LLM that powers the code generation is a challenge. Statements such as “make the left section modern” do not cut it.

Essentially, you can not build what you can not describe. It’s just as simple as that.

What You Describe Is What You Get (WYDIWYG).

2. Lack of domain knowledge

I noticed that most of the people that use code generation platforms are not really knowledgeable about the domain they are trying to build software for.

For example, a marketing professional is trying to build a real estate platform without really taking time to understand the ins and out of the industry.

Just like in any software project, if you do not understand the domain you’re trying to build for, you will most likely fail to translate it into code (even when you have assistance from the LLM powered platform).

3. Inability to handle complex requirements

This is really a platform dependent issue. Keeping context over a long period of time is still a challenge for LLMs. Couple that with the platform dependent limitations (e.g: token limits, request limits, e.t.c) and you have a recipe for disaster.

For example, paying $20 for a limited number of platform credits in a month means that a user’s hands are starting out tied right from the beginning.

4. Project file management or platform limitations

One of the problems that you have to solve when building a code generation platform is how to manage project files.

Do you store them in a bucket somehere and retrive them for every project instance? Do you store them in a database? Or do you run a Docker image to create an initial project structure? How do you get to handle project dependencies? Or worse, how do you handle dependency conflicts?

As a developer, this seems simple on a local computer. But when you have multiple users with different project needs, things become a little spicy.

No, you can not pre-install every possible dependency that a user might need during project creation.

Few platforms have been able to solve this problem effectively.

5. Complexity of software projects

I will admit that this is a very subjjective point. Not all software projects are complex. Some are really simple.

Expecting a code generation platform to handle both simple and the most complex software is not a good idea. But in any case, anyone that wants to build complex software would hire software engineers. Right?

Wrong.

What most people do not understand is what an LLM can or can not do. What kind of projects can be delegated to an LLM powered platform and what kind of projects need human intervention right from the beginning.

Conclusion

LLM powered code generation platforms still have a long way to go before they can be relied upon to build production grade software. Until the above problems are solved (most likely they won’t), these platforms will continue to disappoint users.

In any case, you have a software project that you would like us to discuss, reach out via hello@terraconsults.co