Sweet. Thanks for the links to seaborn and plotnine. I hadn't seen those before.
I do of automation with csv, yaml, openpyxl, and jinja2. Wrapped in a simple GUI (Qt, Gtk, or Tk, pick your poison) so my non-programmer colleagues can quickly run it and get a PDF report, generated with reportlab.
fsvo 'binary', i think i agree. I've written my fair share of struct.unpack-heavy code, and it's just tedious.
I also think that __str__ and __repr__ behavior on bytes is misleading at best. I do not like getting ascii values for some bytes and not others... and iterating to get base-10 ints!?
Pillow[simd], soundfile, and imageio can do nice things for image and sound data.
Comments
The python ecosystem is really good here.
I do a lot of exploratory coding in ipython, though the threshold for "switch to a real editor and run git init and poetry new" is pretty low.
Want to munge CSV? stdlib, or pandas(https://pandas.pydata.org/pandas-docs/stable/)
Want to munge JSON? stdlib.
Want to munge a database? pandas, stdlib, anything that speaks sqlalchemy
Want to validate your json/sql/CSV/whatever and have it come in as a bunch of structured classes not std types? jsonschema https://pypi.org/project/jsonschema/, attrs+marshmallow, attrs+cattrs. http://www.attrs.org/en/stable/, https://desert.readthedocs.io/en/latest/ https://marshmallow.readthedocs.io/en/stable/ https://github.com/Tinche/cattrs
Want to transform csv into sqlite? https://github.com/simonw/csvs-to-sqlite
Want to serve sqlite as a REST api? https://datasette.readthedocs.io/en/stable/
Want to stuff simple things into a database really fast? https://dataset.readthedocs.io/en/latest/
Want to flip numerical data around? Numpy.https://docs.scipy.org/doc/numpy/reference/
Want to model it? Scipy https://www.scipy.org/docs.html, pandas
Want to plot it? seaborn https://seaborn.pydata.org/, plotnine https://plotnine.readthedocs.io/en/stable/
Want to futz around with structured data in a sensible way? glom https://glom.readthedocs.io/en/latest/, python-lenses https://github.com/ingolemo/python-lenses.
Want to spit out tabular data? Tabulate https://github.com/astanin/python-tabulate
Want to figure out where not to eat in chicago? built-ins: counter, defaultdict, comprehensions: https://www.youtube.com/watch?v=lyDLAutA88s
There's a LOT you can do pretty fast, and I can more or less hammer out a basic ETL cli script in my sleep at this point.
Sweet. Thanks for the links to seaborn and plotnine. I hadn't seen those before.
I do of automation with csv, yaml, openpyxl, and jinja2. Wrapped in a simple GUI (Qt, Gtk, or Tk, pick your poison) so my non-programmer colleagues can quickly run it and get a PDF report, generated with reportlab.
I think it is not that rich when dealing with binary data.
Sure it is. https://construct.readthedocs.io/
fsvo 'binary', i think i agree. I've written my fair share of struct.unpack-heavy code, and it's just tedious.
I also think that __str__ and __repr__ behavior on bytes is misleading at best. I do not like getting ascii values for some bytes and not others... and iterating to get base-10 ints!?
Pillow[simd], soundfile, and imageio can do nice things for image and sound data.
Thank you for the list.