Biggest cause is ineffective reasoning in a large context in my experience (more than 350 lines).
Im my experience, ChatGPT breaks down when it needs to consider more than 350 lines and its performance is sloppy before that.
To get solid performance out of it, I essentially need to specify the important areas and changes as well as the desired approach.
That being said, I’ve found it has cut my development time for most features by at least half, even in large codebase so I would be curious to know how you are approaching it currently.
a few examples where AI has been extremely helpful for work related things:
- writing complex sql queries for throw-away code, i.e. need to create a few charts from big query
- writing internal front end tooling
- polishing any internal messages
- helping design docs with clarity
- helping unblock highly technical areas where i haven't had any experience
- helping learn the sdk from new vendors
- helping read api documentation, i.e. does the instagram api allow for ABC
areas where it's been mildly helpful:
- helping to write verbose methods that i'm too lazy to write. e.g. maybe a custom date parser that takes a datetime and outputs some custom text according to a spec
areas where it hasn't been that helpful
- building features e2e. as you mentioned, the context window issue is really the crux here.
- building simple code like a crud api. our system, like all others, has its own idioms and spinning up a new crud api is easy enough. getting an llm to write it is do-able but by the time i've edited things to make it consistent with the rest of our system, i've lost the benefit
It might be time to retire those idioms? A realistic outcome of AI-driven development is code is no longer a long term asset. Patterns like DRY are beneficial because humans have cognitive limits, and many humans over long timeframes need to maintain complex software systems. Maybe patterns like locality of reference are better than DRY for short context window AIs, and maybe we need to start looking at how we would manage many smaller bits of code, with lots of duplication, just like we attend to developer experience today with code analysis and declarative build and deploy pipelines.
If you HAD to deal with 1,000 individual functions - let’s say due to hardware and organizational limitations - how would you manage the obvious risks? Would that be net-net cheaper than a fleet of microservices and development teams?
Comments
Biggest cause is ineffective reasoning in a large context in my experience (more than 350 lines).
Im my experience, ChatGPT breaks down when it needs to consider more than 350 lines and its performance is sloppy before that.
To get solid performance out of it, I essentially need to specify the important areas and changes as well as the desired approach.
That being said, I’ve found it has cut my development time for most features by at least half, even in large codebase so I would be curious to know how you are approaching it currently.
a few examples where AI has been extremely helpful for work related things:
- writing complex sql queries for throw-away code, i.e. need to create a few charts from big query - writing internal front end tooling - polishing any internal messages - helping design docs with clarity - helping unblock highly technical areas where i haven't had any experience - helping learn the sdk from new vendors - helping read api documentation, i.e. does the instagram api allow for ABC
areas where it's been mildly helpful:
- helping to write verbose methods that i'm too lazy to write. e.g. maybe a custom date parser that takes a datetime and outputs some custom text according to a spec
areas where it hasn't been that helpful
- building features e2e. as you mentioned, the context window issue is really the crux here. - building simple code like a crud api. our system, like all others, has its own idioms and spinning up a new crud api is easy enough. getting an llm to write it is do-able but by the time i've edited things to make it consistent with the rest of our system, i've lost the benefit
It might be time to retire those idioms? A realistic outcome of AI-driven development is code is no longer a long term asset. Patterns like DRY are beneficial because humans have cognitive limits, and many humans over long timeframes need to maintain complex software systems. Maybe patterns like locality of reference are better than DRY for short context window AIs, and maybe we need to start looking at how we would manage many smaller bits of code, with lots of duplication, just like we attend to developer experience today with code analysis and declarative build and deploy pipelines.
If you HAD to deal with 1,000 individual functions - let’s say due to hardware and organizational limitations - how would you manage the obvious risks? Would that be net-net cheaper than a fleet of microservices and development teams?
When you say ChatGPT, do you mean 4o, o1, o1 pro, or…?