I have a
Python+
SQLite app that gathers monitoring data from a number of
network devices. My app is run on a tight update cycle, so it must finish quickly. It was obvious that I had a problem when database activity took up more time than network scanning. That and my little program should not cause disk thrashing. Ouch! I now have a nice object lesson on why not to run a database commit after every insert.
Doing one commit at the end of processing, instead of one after every insert cut runtime in half. I didn't think that the difference would be
that big.
This is one of those things where I knew the right way to do it and did it the wrong way at first. Doing that goes away with experience, right?
Now I just need to find a way to do fast thread-safe inserts. I can't use
ZODB because of performance constraints, although
this post did convince me to take a look at it.