Huge Honkin Baseline

So I’ve spent most of the afternoon working on my HugeHonkinConsole project that I’ve been crowing about on and off (and lately mostly on) about for years. Seems as an incoming change in lifestyle has caused my brain to refocus on software projects as a way of settling into the new configuration.

Hell, I was going to go down to Red Phonebooth tonight and celebrate with some fantastic scotch and cigars. But frankly I’d really rather just sit here and code, even if it is lightly.

The current state of this project is both much farther along and much closer to a pile of crap thrown against the wall than the average person might presume.

What I’ve currently got running is:

– A desktop wiki server that also works seamlessly with emacs, so I can edit, view, and search pages from either platform.

– A light todo list application that I wrote in order to start getting my hands around web development even a little. I only use it for features and bugs for the whole HHC project itself. So it’s really a dogfood sub-application.

– The cigar deal downloader: This thing goes to about half a dozen cigar websites and downloads their sale pages every day, parses them, integrates the data and displays it so I don’t have to schlep all over hell’s half acre looking for Melanios or whatever at a reasonable price. It’d be a lot more slick if the damned sites didn’t change their page layouts.

– A couple little agents that sync my Todoist data back and forth using their REST api. At this stage it’s just so that I can keep a local copy of all the data.

– Various data downloaders from things like reddit, a library of rss feeds.

– Pulling, aggregating, and storing all incoming (and outgoing) emails with their attachments.

– Pulling and aggregating all of my text messages (with attachments) and call logs (from a backup file my phone drops on OneDrive)

– backing up OneDrive nightly so that I just don’t have to worry about fatfingering something and losing shittons of data.

– Many other things that I’m totally not doing because that would be bad and wrong.

All of these things sit in independent little processes, humming away on their own time, collecting, parsing and sorting data.

But now it’s time to start bringing them all together.

There are a couple major missing components. But if I wait ‘til I get to those then this will never get integrated.

I’ve already got a basic website that’s an interface to the Wiki and the Todo system. But it’s time to build a front end that I can use to view and change configuration, to watch logs, and to add and schedule subprocesses.

It’s time to actually put the console on the front of the HugeHonkinConsole, at least a simple baseline thing I can use as a starting point to build on to later.

I’m currently using a wsgi library that was in heavy use at my last job (oh irony of ironies) called bottle. It’s…fine. But development on the thing seems to have stopped a couple years ago and I’ve never heard of anyone else using it.

So I’m going to take the opportunity to adapt to a more popular (and actively maintained) one called FastApi.

One of the fun things about taking an existing code base and switching underlying frameworks like that (not that a wsgi library is thick enough to call a framework really. But it does technically fit the definition) is that it gives you an opportunity to really go over your code architecture and clean it up. You figure out very quickly what decisions you’ve made that were predicated on the specific tool set you were using, where you may have over-adapted.

Refactoring legacy code (even if not especially my own) is one of my favorite programming tasks. Problem is I get a little obsessive and over-engineery about the whole affair. Well…it’s not as though it’s a real problem on something that’s essentially an extended toy project.

So I’ll be ripping the dependency on the Bottle framework out and replacing it with something flexible and generic enough to be portable between multiple such systems (which, as a side-effect decouples things for a truly delightful level of testability as well seeing as how I could just as easily put a “framework” in place that was really just test scaffolding and the code wouldn’t know any different.)

But along with that I’ll be doing something similar (though at a lower priority and on an admittedly slower burn) to each of those little bulleted items up top. If you glance back over them you’ll see that an awful lot of them are really centered around the exact same kinds of operations: Download data, homogenize it, ingest it and keep it stored and indexed. Perhaps even generate some kind of digested output, even if it is only some kind of status report showing what happened.

Those semantics are pretty simple to pull out of each of those little project-lets. And having more than one (or two, or three) implementations of the exact same kind of thing, differing specifics notwithstanding, lets me see what the actual commonalities are rather than trying to extrapolate out to what the generic form of those little projects might be. It’ll all be splayed out in front of me, emergent pattern based design.

It’ll be nice to get my chops back, though I’m not sure how atrophied I really am. That’s pretty tough to diagnose from the inside of one’s own head. I won’t really have a good read on it for a month or so. It’s one of those things I’ve found that’s quite like chess: Taking more time off than you’d think would be useful really clears your head, as if the time away actually lets that region of your brain relax and sort all the learning and practice you’d been doing so it really integrates more thoroughly in to your thinking, becoming a more easily accessible set of conceptual tools.

Off to. I’ve been coding and fiddlyfucking with software all day. I’m going to go kill some dragons or something.