Why use my library ?
- Only needs lxml (pymongo only if you use mongodb)
- SQLite cache build time: about 2 minutes (instead of more than one day)
- SQLite cache size: about 120 mb
- Mongodb cache build time: about 3 minutes (will probably be less in the future, as it’s not optimized)
- Mongodb cache size: about 300 mb (instead of 2 Gb berkley db previous solution)
- Fast queries on both solutions
Just try it !
If you need to use gutenberg meta information and get filtered texts for your project, fast, reliable, without overloading gutenberg servers: try GutenbergPy.
Check the usage by reading the Github page
Install the pypi package:
Or if you are feeling adventurous you can install from source:
Why did you do it?
I decided to continue improving artisticmachine42 quote capabilities. As I said in a previous article, Started doing a simple reverse index for fast text searching -> wrote a library to help me scavenge for big-data from the gutenberg site.
Writing something like a reverse index, implies testing it on big data (in my case a lot of books). My go-to data source for my quote extraction was Gutenberg (great initiative, I am a big fan).
List of linked facts that made me do it :)
Tried using the sqlite interface (great sqlite fan here :) ) after about 1 day + 1 night of running -> it still didn’t populate the cache.
Thought the sqlite cache feature was added later so it must be still in development -> used the bsd db cache -> took around 5 hours to populate (on a really good pc with a really good internet connection) and 2 GB of hdd space
I then proceeded to query all the books that are from a certain category with a certain language -> it failed.
Reported bug to developer
Read “the maintainer wanted to no longer maintain the project” -> thought it will never get fixed unless I do something about it
Downloaded the repository and started debugging -> saw nasty stuff, the cache base needed to be rewritten (in my opinion)
I decided to write my own and publish for everyone to use -> help the open source community
Wrote it -> published.
Published GutenbergPy -> maintainer of old library said he fixed the query bug :) (didn’t test it yet)
What about the other promised articles?
I had some crunch time at work so I didn’t have the time to write the other (bigger) articles on my site, but rest easy:
The gabor feature generation code is ready (writing a CUDA port). I didn’t forget, the article is coming soon :) .
The reverse index code is also ready so an article about that will also be coming soon. :)