Hi folks - I'm the creator/teacher of this course. I'd be happy to answer any questions that you have about the course, learning deep learning in general, or the state of deep learning in 2022.
Thank you, so much, for enabling us mortals the power of ethical, modern AI. Your work, and the work of your colleagues, has brought so much good to this world.
It wasn't until the last few years I saw people start freezing the model and just fine tuning the last layers. I've watched presenters from flamingo and imagen talk about their similar approaches. I heard it here first, at fastai.
I tried the 2021 course but I didn't finish. I think the biggest friction for me was using the remote machine. I wasn't able to make steady progress like I do with my offline learning projects.
The good news is that every lesson in this course is actually run on Kaggle Notebooks, which is a free cloud environments including GPUs. So you don't need to set up anything and it runs on any computer with a modern web browser!
Mac support for all the libs used in the course will probably continue to improve in the coming months and there should be no reason you won't be able to run the stuff for the course locally on a Mac at that time. Having said that, even the M2 trains deep learning models much slower than even the free NVIDIA GPUs provided by Kaggle. So you'd only want to use local development for the smallest and simplest models. (The course shows how to train models that are fairly cutting edge and some take a while to train even on modern GPUs, so they wouldn't be a good fit for a Mac.)
I think last time I tried this, I kinda gave up as soon as it got to the point of hand-waving hardware and telling you to run notebooks on a third party's web service. How is that democratizing AI? It's the very opposite! Not good. What is the intended audience here? Uni-level students will learn most of these within their programs if they're interested in AI. So they're not really it. Is the intended audience "coders"? If so, most of these "coders" will have to somehow get their employer on board (typically a corporate entity very much NOT interested in "coding" something in a 3rd party ecosystem) or do it themeselves. Hence, I want to take an RTX 3080+, 64+ gb of ram, big ass SSD and I want to get through the training. Not learn some basics on somebody else's platform (come on, even from the pov of OS, running both training AND notebooks on some 3rd party's private platform is so against the idea of open source...) and call it a day. What use is that? That may be enough if you want to be a cog in somebody else's machine, but not if you want to do something useful by yourself (I say "may" because big tech generally isn't interested in your "mad AI skillz" unless you also have a student loan backed piece of paper proving you successfully learned that for the last couple of years).
There will always be smart individuals and talented small teams that can successfully integrate AI into their products, but it's not thanks to the courses like this.
If you're going to aim at coders, there has to be clear path demostrated from the beginning to the end. From starting up your first notebook on your local dev machine and running training on your local training machine to setting up inference in the final app (.net app or whatever)
I'm disappointed that Google shut down the Swift for Tensorflow project, because I do think Swift is a great option for deep learning.
In some ways Jax is almost "non-python deep learning" since it's treating Python more like a DSL for the XLA backend. Normal Python code doesn't work in Jax. It's a pretty reasonable compromise since you still get all the benefits of the Python ecosystem.
Julia seems like it has the best foundations for deep learning, since everything can be written directly in Julia. But it doesn't have a great ecosystem as a general programming tool.
As someone who much prefers reading over watching videos, do you think I would miss much by just going through the book in the github repo? Or are those notebooks mainly supplemental to the videos.
I'd say it's the other way around - the videos are kinda supplemental to the book. The book has a lot more content, but doesn't have the interactive explanations in the course. Also the book is a couple of years old so is missing the more recent developments (but the principles haven't changed).
Thank you so much for this course. I plan to go through it properly.
I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_.
Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?
As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary.
For the same reason, GloVe is of not much use to me.
OK, I understand. That is a different problem indeed. Although I am not sure that BERT would be of much more help there either, unless you have quite a large training corpus. The simplest/cheapest approach might be some kind of transcription normalization. As you say, what is introducing ambiguity here is the act of transcribing these sounds into the English alphabet. There’s not a single source language/alphabet? Also, I am not sure if fastText/GloVe really would not pick up the semantic similarity in spite of differing transcriptions as long as you have a large enough training corpus. I’d experiment with the settings here (bag of words/skipgram, min/max lengths etc.).
Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus.
In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.
I'd suggesting budgeting about 80 hours for the course given that background. That should get you to a place where you can work on practical projects that are reasonably well within standard applications of deep learning.
Most practical deployment is done to cloud environments rather than local notebooks. The deployment exercise we do in the course is designed to show the key components you'll need for deploying simple models in practice.
Not him nor will I talk about his course, but I’ve been in the field a reasonable amount of time (both on the academia and industry side). Honestly, applied maths will get you a long way and make it easier to digest the concepts (you might just see them as repackaged problems depending on your mileage). If you have good programming skills and discipline you practically have most of what you need.
Re the course, I just skimmed it and I think you can do most things on your own hardware but if you will actually use this for something practical (not just for you or a side project), being familiar with cloud tools is a big thing especially once you scale.
Bare minimum is basic calculus, basic linear algebra and basic statistics. By basic, I probably mean first courses for those in most undergraduate programs.
I disagree with needing none and just going along as needed. That’s how you have machine learning models that look like they work but you don’t understand why they work so there might actually be problems.
That's not quite what I said. I said to look things up if you don't understand, I made the assumption that the one asking has taken maths courses before. I interpreted bone up on as refreshing old knowledge but I could be wrong.
None, just look things up as you go along if there is something you don't understand. You're likely not going to bother understanding how the optimization functions work or how the cost functions actually work anyway. They're implementation details in most cases.
Hey Jeremy, i just want to say that I love your course and the way you teach. I refer everyone to the Fast AI in my YouTube videos on getting started with machine learning. Please keep up the great work!
Thanks for creating this fantastic content, I'm excited to give the 2022 course a look. It's an exciting time for AI. I'm curious about your thoughts on gpt3 and also the state of the art in computer vision, and object detection. All the best
Thank you for creating this course. I started out on Tensor Flow but seeing this material I am in two minds whether I should abandon my TF book and start this one or save it for later. Most likely I am going to dive in :-)
Both the Aurélien Géron and François Chollet TF books are absolutely terrific, and everything you learn from them will be extremely useful in becoming a deep learning practitioner, regardless of what framework you end up using. So if you've started with one of those books already, keep it up! :) The fast.ai course would actually be a pretty good addition to either book, since you'll get to see a whole different way of doing things, which might be useful to understanding what's going on.
My suggestion would be to learn all the stuff from this course, using fast.ai library, and then gradually move towards PyTorch.
fast.ai is a fantastic educational resource and a great way to approach solving problems. But the library itself is lacking, and if you are an experienced programmer, when building real-life projects, you will be frustrated with fast.ai library.
The goal, IMO, should be learn from Jeremy Howard, s great instructor, communicator; learn his attitude, and then move to PyTorch (keeping the attitude, the knowledge, and the lessons with you.)
I am an experienced ML Engineer of 10 years and have worked at several large flagship tech companies. I do not agree that fastai is not appropriate for real-life projects. If you know the fastai library well, you know its a layered api on top of pytorch, which allows you to customize things to your needs quite easily. For example, it is fairly straightforward to get any pytorch model out of a Learner object. Furthermore, lots of care has been taken to keep the apis very consistent with pytorch as well.
It's also the only library I know of that consistently bakes in best practices like super convergence techniques or making things like test time augmentation very seamless. Many libraries lag behind fastai 1-2 years in this regards, and frankly it can be frustrating to use other frameworks sometimes.
There is a slight learning curve, for example to learn the DataBlocks API or the callback system, but once you really understand what is happening you will understand how nice the API is and how well engineered it is.
Side note: Regarding being an experienced software engineer, I highly recommend digging into how the python language was extended for this project (fastcore) and the development workflow used (nbdev), which I think could be interesting for those software engineers you mention as well as heighten your understanding of the ecosystem of tools.
Honestly I'm not an expert on autonomous driving so I'm not sure I have great insights there. I do know quite a bit about computer vision however so feel qualified to comment on that bit -- I suspect the decision by Tesla to only use CV, and not LIDAR, may turn out to be a mistake. I don't see any reason why we couldn't achieve full autonomy with our current tech including LIDAR, although I don't know if it can be achieved at a practical latency and power budget.
The new advances in DL I'm excited about are things I show in the class: the accessibility of modern NLP thanks to the Hugging Face ecosystem; the power of ConvNeXt for even better computer vision models; the way Gradio and HF Spaces makes it trivially easy to get a working prototype application using DL online.
I'm also excited about hosted models and applications like GPT-3, DALL-E, and Codex. All the illustrations on our course website are from DALL-E, for instance!
Comments
Hi folks - I'm the creator/teacher of this course. I'd be happy to answer any questions that you have about the course, learning deep learning in general, or the state of deep learning in 2022.
Thank you, so much, for enabling us mortals the power of ethical, modern AI. Your work, and the work of your colleagues, has brought so much good to this world.
It wasn't until the last few years I saw people start freezing the model and just fine tuning the last layers. I've watched presenters from flamingo and imagen talk about their similar approaches. I heard it here first, at fastai.
Haha yes - it's been wonderful to see how (eventually) the deep learning world has taken to transfer learning!
What’s your take on meta learning?
I tried the 2021 course but I didn't finish. I think the biggest friction for me was using the remote machine. I wasn't able to make steady progress like I do with my offline learning projects.
How far away is the fast.ai from working on a Mac? PyTorch recently gained support (https://pytorch.org/blog/introducing-accelerated-pytorch-tra...) but that's only the start. Is this something that is being worked on?
The good news is that every lesson in this course is actually run on Kaggle Notebooks, which is a free cloud environments including GPUs. So you don't need to set up anything and it runs on any computer with a modern web browser!
Mac support for all the libs used in the course will probably continue to improve in the coming months and there should be no reason you won't be able to run the stuff for the course locally on a Mac at that time. Having said that, even the M2 trains deep learning models much slower than even the free NVIDIA GPUs provided by Kaggle. So you'd only want to use local development for the smallest and simplest models. (The course shows how to train models that are fairly cutting edge and some take a while to train even on modern GPUs, so they wouldn't be a good fit for a Mac.)
I think last time I tried this, I kinda gave up as soon as it got to the point of hand-waving hardware and telling you to run notebooks on a third party's web service. How is that democratizing AI? It's the very opposite! Not good. What is the intended audience here? Uni-level students will learn most of these within their programs if they're interested in AI. So they're not really it. Is the intended audience "coders"? If so, most of these "coders" will have to somehow get their employer on board (typically a corporate entity very much NOT interested in "coding" something in a 3rd party ecosystem) or do it themeselves. Hence, I want to take an RTX 3080+, 64+ gb of ram, big ass SSD and I want to get through the training. Not learn some basics on somebody else's platform (come on, even from the pov of OS, running both training AND notebooks on some 3rd party's private platform is so against the idea of open source...) and call it a day. What use is that? That may be enough if you want to be a cog in somebody else's machine, but not if you want to do something useful by yourself (I say "may" because big tech generally isn't interested in your "mad AI skillz" unless you also have a student loan backed piece of paper proving you successfully learned that for the last couple of years).
There will always be smart individuals and talented small teams that can successfully integrate AI into their products, but it's not thanks to the courses like this.
If you're going to aim at coders, there has to be clear path demostrated from the beginning to the end. From starting up your first notebook on your local dev machine and running training on your local training machine to setting up inference in the final app (.net app or whatever)
Thanks Jeremy, I'll give it another go.
I remember you were bullish about Swift a few years ago. What's your current view on non-python deep learning?
I'm disappointed that Google shut down the Swift for Tensorflow project, because I do think Swift is a great option for deep learning.
In some ways Jax is almost "non-python deep learning" since it's treating Python more like a DSL for the XLA backend. Normal Python code doesn't work in Jax. It's a pretty reasonable compromise since you still get all the benefits of the Python ecosystem.
Julia seems like it has the best foundations for deep learning, since everything can be written directly in Julia. But it doesn't have a great ecosystem as a general programming tool.
F# might turn out to be a good option.
Interested that as well, especially the old school lisp to this new Ai.
As someone who much prefers reading over watching videos, do you think I would miss much by just going through the book in the github repo? Or are those notebooks mainly supplemental to the videos.
I'd say it's the other way around - the videos are kinda supplemental to the book. The book has a lot more content, but doesn't have the interactive explanations in the course. Also the book is a couple of years old so is missing the more recent developments (but the principles haven't changed).
Thank you so much for this course. I plan to go through it properly.
I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_.
Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?
Have you tried...
a) BM25 after some preprocessing (lemmatization etc.)
b) fastText / GloVe (possibly weighted by BM25)
The results can be surprisingly good. Often no need to bother with big language models or GPUs.
As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary.
For the same reason, GloVe is of not much use to me.
OK, I understand. That is a different problem indeed. Although I am not sure that BERT would be of much more help there either, unless you have quite a large training corpus. The simplest/cheapest approach might be some kind of transcription normalization. As you say, what is introducing ambiguity here is the act of transcribing these sounds into the English alphabet. There’s not a single source language/alphabet? Also, I am not sure if fastText/GloVe really would not pick up the semantic similarity in spite of differing transcriptions as long as you have a large enough training corpus. I’d experiment with the settings here (bag of words/skipgram, min/max lengths etc.).
If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?
Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus.
In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.
How many hours do you think this course would take for an experienced developer with plenty of applied maths but ~no machine learning?
How easy is it to do the course on my own hardware rather than cloud notebooks? Would that make it closer to practical deployment?
I'd suggesting budgeting about 80 hours for the course given that background. That should get you to a place where you can work on practical projects that are reasonably well within standard applications of deep learning.
Most practical deployment is done to cloud environments rather than local notebooks. The deployment exercise we do in the course is designed to show the key components you'll need for deploying simple models in practice.
Not him nor will I talk about his course, but I’ve been in the field a reasonable amount of time (both on the academia and industry side). Honestly, applied maths will get you a long way and make it easier to digest the concepts (you might just see them as repackaged problems depending on your mileage). If you have good programming skills and discipline you practically have most of what you need.
Re the course, I just skimmed it and I think you can do most things on your own hardware but if you will actually use this for something practical (not just for you or a side project), being familiar with cloud tools is a big thing especially once you scale.
out of curiosity, how much applied math should one bone up on? (Obviously the more the better, but diminishing marginal returns and all that.)
Bare minimum is basic calculus, basic linear algebra and basic statistics. By basic, I probably mean first courses for those in most undergraduate programs.
I disagree with needing none and just going along as needed. That’s how you have machine learning models that look like they work but you don’t understand why they work so there might actually be problems.
That's not quite what I said. I said to look things up if you don't understand, I made the assumption that the one asking has taken maths courses before. I interpreted bone up on as refreshing old knowledge but I could be wrong.
None, just look things up as you go along if there is something you don't understand. You're likely not going to bother understanding how the optimization functions work or how the cost functions actually work anyway. They're implementation details in most cases.
Hey Jeremy, i just want to say that I love your course and the way you teach. I refer everyone to the Fast AI in my YouTube videos on getting started with machine learning. Please keep up the great work!
Thanks for creating this fantastic content, I'm excited to give the 2022 course a look. It's an exciting time for AI. I'm curious about your thoughts on gpt3 and also the state of the art in computer vision, and object detection. All the best
Is there a new version of the book? All the links I find lead to the 2020 edition.
No, the book is continually updated for each reprint, but there isn't a separate edition.
Thank you for creating this course. I started out on Tensor Flow but seeing this material I am in two minds whether I should abandon my TF book and start this one or save it for later. Most likely I am going to dive in :-)
Both the Aurélien Géron and François Chollet TF books are absolutely terrific, and everything you learn from them will be extremely useful in becoming a deep learning practitioner, regardless of what framework you end up using. So if you've started with one of those books already, keep it up! :) The fast.ai course would actually be a pretty good addition to either book, since you'll get to see a whole different way of doing things, which might be useful to understanding what's going on.
Thank you
My suggestion would be to learn all the stuff from this course, using fast.ai library, and then gradually move towards PyTorch.
fast.ai is a fantastic educational resource and a great way to approach solving problems. But the library itself is lacking, and if you are an experienced programmer, when building real-life projects, you will be frustrated with fast.ai library.
The goal, IMO, should be learn from Jeremy Howard, s great instructor, communicator; learn his attitude, and then move to PyTorch (keeping the attitude, the knowledge, and the lessons with you.)
I am an experienced ML Engineer of 10 years and have worked at several large flagship tech companies. I do not agree that fastai is not appropriate for real-life projects. If you know the fastai library well, you know its a layered api on top of pytorch, which allows you to customize things to your needs quite easily. For example, it is fairly straightforward to get any pytorch model out of a Learner object. Furthermore, lots of care has been taken to keep the apis very consistent with pytorch as well.
It's also the only library I know of that consistently bakes in best practices like super convergence techniques or making things like test time augmentation very seamless. Many libraries lag behind fastai 1-2 years in this regards, and frankly it can be frustrating to use other frameworks sometimes.
There is a slight learning curve, for example to learn the DataBlocks API or the callback system, but once you really understand what is happening you will understand how nice the API is and how well engineered it is.
Side note: Regarding being an experienced software engineer, I highly recommend digging into how the python language was extended for this project (fastcore) and the development workflow used (nbdev), which I think could be interesting for those software engineers you mention as well as heighten your understanding of the ecosystem of tools.
First of all, a big thanks!
What is your take on the current state of autonomous driving? Do you think we can achieve "full autonomy" with the technology we have currently?
Any new advances in DL that you are excited about?
Honestly I'm not an expert on autonomous driving so I'm not sure I have great insights there. I do know quite a bit about computer vision however so feel qualified to comment on that bit -- I suspect the decision by Tesla to only use CV, and not LIDAR, may turn out to be a mistake. I don't see any reason why we couldn't achieve full autonomy with our current tech including LIDAR, although I don't know if it can be achieved at a practical latency and power budget.
The new advances in DL I'm excited about are things I show in the class: the accessibility of modern NLP thanks to the Hugging Face ecosystem; the power of ConvNeXt for even better computer vision models; the way Gradio and HF Spaces makes it trivially easy to get a working prototype application using DL online.
I'm also excited about hosted models and applications like GPT-3, DALL-E, and Codex. All the illustrations on our course website are from DALL-E, for instance!
I hope you stay as humble as you have been. But you're my personal hero. It is just incredible what you have done for the world.
Will it help me more or achieve more out of learning this course as compared to just directly using GPT-3 or Dall-E as paid user?
Hello Jeremy do you have any specific advice on tackling ASR using fast.ai?
Jeremy, hi.
I have one question, and one only. Please answer:
Second part, when?
.m