My team and I wrote an NLP application to detect sensitive data and detect / validate schemas, etc as well as the other items provided by pandas-profiling.
That being said, we noted the same thing. It shouldn't matter what modeling you use. It's the data pipelining where 99% of the work typically is. Modeling itself always needs the same basic input -- matrix of data and outputs a matrix of data.
Some libraries are good at specific components. Others have improved speeds ups, etc. But it's all so new it's effectively going to change month-to-month. So I always tell the team to build what you can as fast as you can, with the tools you have. We can always update it later, once the pipeline is in place.
Comments
My team and I wrote an NLP application to detect sensitive data and detect / validate schemas, etc as well as the other items provided by pandas-profiling.
https://github.com/capitalone/DataProfiler
That being said, we noted the same thing. It shouldn't matter what modeling you use. It's the data pipelining where 99% of the work typically is. Modeling itself always needs the same basic input -- matrix of data and outputs a matrix of data.
Some libraries are good at specific components. Others have improved speeds ups, etc. But it's all so new it's effectively going to change month-to-month. So I always tell the team to build what you can as fast as you can, with the tools you have. We can always update it later, once the pipeline is in place.