SQL is as good as it gets, generally speaking. The best engine for a small project is SQLite. If you need more, the next step is Postgres. Postgres is very capable and it's unlikely that you'll need more even as a medium sized company. But it sounds like you're asking about small personal projects.

"State of the art" that goes beyond SQL/Postgres is usually about scalability and division of labor. These are concerns relevant to large companies (Facebook, Netflix, ...) and not for personal projects. By using these cutting edge DBs, you will be accepting compromises that make your life more difficult, and gaining benefits that don't apply to you.

As stated elsewhere, you can read/write from a SQL DB with Pandas, and many DB connectors support some kind of .to_pandas method. However, this won't save you form learning SQL, since you'll still have to pass the queries to fetch and upsert the data. Pandas also has a looser typing system than the typical DB, so you might often run into annoying type mismatch/conversion bugs there.

Instead of a DB, you can use pandas to read/write some simple file format, like CSV or Parquet. I would recommend CSV, it's easy to work with using other tools as well. If you want something nested, then JSON is a good start.

You should really just learn SQL. Pandas at its heart is a crude reimplementation of things SQL has perfected half a century ago.

posted 10 months ago

CC BY-SA 4.0

matthewsnyder‭

1729 reputation 43 48 198 80