Some of these frameworks like instructor use like 80% more tokens or only work with OpenAI so we aim to tackle all these problems from the ground up.
No doubt this is because open source OpenAI API implementations don't support function calling, but now that we have context free grammars in transformers (and TGI) implementation of function calling is not very far off.
Also, about the article, I chuckled, because he listed "guidance". That is (other than an api) a feature of some AI frameworks to implement negative prompting. For example in Transformers. You can send a prompt "what is a tomato", alongside a guidance prompt "you're very verbose" and a (negative) 2 guidance scale. This way the answer will be the opposite of "you're very verbose". This feature is not in the official release yet, but it exists (in git) and it is very neat.
Not sure it's related to function calling. GPT4 can do function calling without using the specific function-calling API just by injecting the schema you want into the prompt with directions and asking it to return JSON. It works like >99% of the time. Same with 3.5-turbo.
The problem is these libraries convert pydantic models into json schemas and inject them into the prompt, which uses up like 80% more tokens than just describing the schema using typescript type syntax for example. See https://microsoft.github.io/TypeChat/, where they prompt using typescript type descriptions to get json data from LLMs. It's similar to what we built but with more boilerplate.
Comments
No doubt this is because open source OpenAI API implementations don't support function calling, but now that we have context free grammars in transformers (and TGI) implementation of function calling is not very far off.
Also, about the article, I chuckled, because he listed "guidance". That is (other than an api) a feature of some AI frameworks to implement negative prompting. For example in Transformers. You can send a prompt "what is a tomato", alongside a guidance prompt "you're very verbose" and a (negative) 2 guidance scale. This way the answer will be the opposite of "you're very verbose". This feature is not in the official release yet, but it exists (in git) and it is very neat.
Not sure it's related to function calling. GPT4 can do function calling without using the specific function-calling API just by injecting the schema you want into the prompt with directions and asking it to return JSON. It works like >99% of the time. Same with 3.5-turbo.
The problem is these libraries convert pydantic models into json schemas and inject them into the prompt, which uses up like 80% more tokens than just describing the schema using typescript type syntax for example. See https://microsoft.github.io/TypeChat/, where they prompt using typescript type descriptions to get json data from LLMs. It's similar to what we built but with more boilerplate.