|Jekyllization - 01 November 2014|
My little blog-building script is getting more advanced. It can handle Markdown and HTML::Template.
Sooner or later I'm going to have reinvented WordPress.
|Squirrel! - 03 November 2014|
In case you were wondering, I never did get that yard work done. Now it's at that awkward stage: the grass is dormant, so there's no rush. At least this year it's relatively low - last year it had gotten high enough to go to seed (not high, but raggedy) and then it reached the point where it was kind of embarrassing to let it go (the grass showed through what little snow we had) but also embarrassing to mow it ("HI EVERYBODY THIS IS ME RUNNING THE LAWNMOWER IN DECEMBER"). Anyway, I should get the garden put to bed so I can actually plant stuff just as soon as the weather allows next spring. I didn't do it this year and ended up just letting it lie fallow in what would have been a really good year. The garden and I both needed a rest, though.
If it isn't obvious by now, what with my blog topics being all over the map, I have Adult Attention Deficit Disorder. Didn't really realize it until we had a son, and he was diagnosed ADHD (and, eventually ASD). I read a book on ADHD parenting and when I got to the part about girls and ADHD and went "holy wow, that was me." I grew up hearing "you're in your own little world" (I was) and "if you cared about people, you'd remember X" (promises, birthdays, whatever). I think my mother feels kind of guilty about it, because she is the opposite of ADHD and just could not comprehend that my brain doesn't work like hers. I can't exactly blame her, being as how I was almost 40 before I realized what was going on. Girls didn't get ADHD, after all, I was by no means hyperactive (and son is what they call "primarily inattentive") and mostly it took the form of me reading all the time and shutting out the world.
Then the book had a chapter on AADD, and I realized "holy wow, that's STILL me." You don't necessarily outgrow ADHD, you just learn coping mechanisms. And I hadn't, entirely, because I was convinced I should be able to keep everything in my head all organized and stuff. Like Mom. So what the hey, I visited a psychiatrist. Got a formal confirmation that I'm mildly crazy, which is kind of paradoxical: it changes nothing, but knowing that there's a DSM-IV label, an explanation, a name for it makes it feel more normal. Apart from confirming that I really didn't have any of the co-morbidities (depression, substance abuse, etc.) that often go along with it, I didn't learn too much from the doc except the obvious: I need to build coping mechanisms, and that's what I've done for the last several years. I refer to my tablet as my "brain," and feel a lot freer in telling people, "If you don't see me write it down, it's not going to happen." I try not to use it as an excuse, but I'm not shy about enlisting help in keeping myself accountable.
I feel kinda dumb, in hindsight, trying for all those years to be "normal" and not just acknowledging that mayyyybe I should try something different. So the moral of the story is: go see a mental health professional sooner, not later.
|Protocols - 04 November 2014|
I can't remember who pointed to this on Twitter, but I think it was a tildenizen: Liberate Apps Through Protocols: Lets Update IRC!.
Yes, let's. More importantly, let's build clients for these protocols that are as easy and slick-looking (and that have as much defense against abuse) as the silo equivalents.
|Parsing - 07 November 2014|
Yak shaving. Seriously.
Before you build a pumpkin spice client, you have to have something to read with it. I mean, technically you want to select things to read ("follow" or "friend" people) within the client, but still.
So I started fiddling with a little Perl script, takes an url on the command line and checks it out. If it's a Facebook or Twitter url, it can tell who the person/page is (and if I had bothered to register with the API, it could go off and do stuff with it). I haven't done anything with Instagram or Google Plus, but there are libraries for those on CPAN so I'm just treating them as a solved problem for now.
Tilde.club pages are really a pretty good early test for at-large web pages. Some of them, like mine, have the feed properly marked in the header, and I can just look at
Others have a feed but it's not marked, so I look for things like index.rss or feed.xml or any of the other standard-ish names in the same directory as the page I've got.
And then there's the large majority of them: pages that have no machine-readable equivalents. In my real life, I use Page2RSS to produce a changelog RSS. That works pretty well, but for two things: one, it's rather tedious in TinyTinyRSS: copy the page address, open the Page2RSS page, paste the page address into the form, copy the resulting page address, waaaaait for TinyTinyRSS to open, select Subscribe to Feed, paste the address, select the category, waaaaait for TinyTinyRSS to poll the site, waaaaaaaait for TinyTinyRSS to reload. (I'm not sure if TTR is slow, or our server is, or my browser is, or all three.)
Now, once it's set up, it works great, and generally hands me everyone's blog entry in a reasonably timely manner. So if pumpkin spice does all of those steps under the hood and asynchronously, that solves most of my problem. All that's left is the fact that I'm relying on a third-party service whose customer I am not. So PS has to have a build-in diff function that can preserve the formatting relatively well. And ideally, it should be a little smarter than Page2RSS and not hand me every update where just a timestamp (or a temperature, or whatever) just changed.
I'm really hoping somebody else has solved that problem, and that I just need to pore over CPAN (and Github, these days) to find it, because otherwise I have to go into the hell that is HTML parsing.
|git - 07 November 2014|
Yeah, like I mentioned earlier I gotta learn git. I set up a Github account, put sitemapper.pl in it via the web thingy and... that was it. I have four tabs of Github documentation and tutorials opened, I have a lovely ebook bookmarked that ~winston... uh, @suchwinston (now I can't find him on ~) pointed me to, I have a brilliant systems administrator/architect in the same house with me... but nope, I have done nothing.
So okay, I've updated sitemapper.pl to dump a JSON file as well (and commented out the private files entirely), and I need to update my repo on Github. It goes something like this:
Holy wow you guys, I did it without having to pester any live people! https://github.com/tyrosinase/tilde-tools/blob/master/sitemapper.pl
|Economies of scale - 07 November 2014|
Or scale of economies, really.
I got invited by a member of my nobility to a meeting today, the local chamber of commerce's health committee. (I didn't know they had one, but now I do.) The local Y and university have been partnering on food access studies, and this was the analysis. The presentation was very good, but it was what was wrapped around it that was boggling to me.
In Delano, we have no budget. Like, no money at all. The neighborhood association has no dues (it's not an HOA), and no budget outside the donations we take in at meetings and events. We're not in bad shape, as neighborhood associations go; got a few hundred dollars in the general fund, a few thousand in various project funds that we're working toward. The web pages we run live on surplus storage/bandwidth from my own server, and I pay for the domain registration myself. We print a newsletter at a print shop owned by the husband of one of the neighborhood pastors, paid for by ads that I scrounge from neighborhood businesses every month. (When I die, use the funeral home down the street that has faithfully run a $50 ad every month for, as far as I know, the sixteen years the newsletter has existed.) Delivery is by volunteers; high school kids that need community-service brownie points to buff up their transcripts, churches that happen to be delivering their own fliers to the neighborhood, whatever I can come up with. Some months, I can't come up with advertising or I can't come up with delivery, and we get no newsletter.
So when the chamber gal said they needed to rebrand the health committee, and they were budgeting $10-12K for the new logo and website, I may have realized I was probably in the wrong room. Wrong league. Something.
But you know? I wouldn't trade our community fabric, our huge base of social capital, for all the grant money they've got.
(Most days, anyway.)
|Identification - 08 November 2014|
If this entry is about subscribing to a particular feed (that is, a particular place-to-write stuff, be it an actual Atom/RSS feed, a blog, a social-media-network account, or just a web page), then this one's about subscribing to a particular person. Or at least identifying a particular person across multiple feeds, and letting the user decide which bits to subscribe to. I haven't written any code for this yet, but it's a little more complicated. A feed is really just a changelog, while an identity is a little fuzzier.
Let's use my tilde as an example. My hypothetical Spice user hits http://tilde.club/~silver/ and says "Whoa, this person is fascinating, I want to follow them. The index.html is easily parsed to come up with a feed, but how can we automatically identify the person behind it?
First of all, there's an atom.xml referenced in the headers and, when I haven't borked its formatting, there are
The first thing a human sees in http://tilde.club/~silver/ is "I am Karen Cravens" which links to a page where I further identify myself. Probably PS should look at me.html, about.html, bio.html and other likely candidates. The fact that that link is an exact match for an author from the feed is probably also a clue we should pay attention to. Tilde accounts aren't as ubiquitous as they used to be, but we should still consider
That said, parsing http://tilde.club/~silver/me.html nets us a link to a Twitter account, a Github account, that
But wait, phoenyx.net's whois record matches Cravens. A quick look at its homepage shows a feed, which is a stupid WordPress one that doesn't include much in the way of
We can check known social networks for me: there's a Karen Cravens on Facebook who lives in Wichita Kansas, and whose avatar matches the Twitter @kareninwichita one. That adds my maiden name and spouse's name to the list of things. And my spouse's name matches the rest of that phoenyx.net whois record, so we can add a mailing address (which matches the hometown) to the information.
An identity is starting to shape up. We have a name, some avatar to choose from, a maiden name, a spouse, a hometown, mailing address, phone number, an email address or two, a couple of blogs, a Facebook, a couple of tweet streams, and a repo. That's enough for a pretty healthy profile page. Let's go stalk another tildenizen.
Looking at http://tilde.club/~silver/sitemap2.html, and "randomly" picking the most recent update, our candidate is http://tilde.club/~john. Oh good, he has a nice cryptic page. (If you haven't figured it out, you might want to go look at it now because I'm going to SPOILER it) No feed, no email links, nothing. We have an image link to a 1x1 pixel, so that's probably not an avatar, and a link to a solitary other page. Which links to yet another page like it, and so on for what, fifteen levels? That's probably more than Spice should follow, but maybe it asks the user and the user says keep going. Eventually, it gets to one that links back to
From there we have some lovely links to places PS understands: Instagram (where we get an avatar), Tumblr, Twitter (from which we learn that John is from New York, and see the same avatar), and an email link. There's also a batch of other links there, so let's check them out.
The first one is http://attentiontoretail.co/ which, to a human who scrolls to the bottom, is obviously a blog by John. (It says "ATTENTION TO RETAIL is a blog about thinking about shopping. It is published on Mondays by John Holdun.") But can PS figure that out? There's no handy-dandy Atom or RSS, no standard CMS/blogging software obvious (and those generally build Atom or RSS in any event). The only thing we've got is that
On to the next: http://puthtml.com. There's John's twitter account at the bottom, but no other firm clues, even in whois. To make sure this isn't someone else's page that just happens to link back to John, we should probably ask the user "Is this really John Holdun's page?" User says yes, so we add it. And so on.
We can check known social networks for John as well, but even if we don't find any ~john still ends up with a pretty complete profile page too.
Now, tildenizens are probably not the "best" candidates for this (by which I really mean "worst"). Most of my family members don't have a net presence outside of Facebook, so their Pumpkin Spice profile page is going to be pretty one-dimensional. (And, interestingly, they're even more freaked out about privacy and stalkers and stuff because, I guess, it's better to give all your data to one company than to spread it all over the net? Maybe they have a point. But I once mentioned an AT&T problem on Twitter, had an AT&T rep ask me to "follow and DM my info." Having been down that road before and not particularly wanting to follow yet another AT&T tech support account, I just @-replied them the voice line the ADSL was attached to. I got an anxious reply telling me to delete the tweet because it was public. My response was something along the lines of "This, from a company that wants me to pay them extra to keep them from delivering my name, phone number and home address to the doorstep of half a million people every year?" The tech didn't have an answer for that. Nor did they fix our problem, come to that.)
|Oh, OAuth - 08 November 2014|
I had, apparently, blocked from my memory the fact that Twitter, Facebook, and probably every other place in the world requires OAuth these days.
OAuth requires, unless I'm missing something, two things: the user to sign in via the service's website (inconvenient but not impossible), and place for the service to send the oauth_token and oauth_verifier. And there it gets awkward.
Pumpkin Spice isn't supposed to be dependent on anything the user isn't a customer for (other than the already-existing networks). So who manages that token and verifier? OAuth2 gets past this but Twitter, at least, doesn't allow user-level posting from OAuth2.
Maybe screen-scraping would be more effective.
|Oops - 12 November 2014|
|Crazy wind (they say 55mph "gusts," but the baseline wind was 45mph) and temperature drop yesterday, so we weren't surprised that the ADSL was flapping all day. Complained, complained, complained... AT&T's diagnostic was uninformative, other than to tell us they had auto-updated the firmware to close the root exploit, and to tell us that we needed to schedule an "in-home tech." I was all set to grab the router-modem thingy and stomp into the AT&T store in person, but Carl went out and punched down one of the punchdowns a little harder. And hey, it hasn't dropped since. Maybe we should have done that a long time ago.|
|When in doubt, code - 12 November 2014|
It's maybe not the best policy, but like I've said elsewhere, I was raised by wolves. I can do a lot of brainstorming before settling into coding, but sometimes I just have to start laying out the Perl (or whatever, but in this case it's Perl) and seeing what happens. I may not end up with the grandiose (silo buster)[http://tilde.club/~silver/blog/silo-buster.html] I've talked about, but at the least I'll have a critter that can build Atom change feeds off Facebook and Twitter and vanilla HTML pages, which will improve my Tiny Tiny RSS life.
So that's what I did. It's been an embarrassingly long time since I've done any major coding in Perl, especially building modules from scratch, so I dinked around until I remembered how those work. I threw together a skeletal library for (Pumpkin Spice)[http://tilde.club/~silver/projects/pumpkin_spice.html] and started hanging plugins off it: Spice::HTML for those pages that just don't have any other structure, then Spice::Feed for those that have an Atom or RSS feed. My test suite used my own tilde page, and XML::Feed barfed all over my Atom feed.
I kind of knew that the evolving nature of my formatting here had maybe introduced some invalid markup here and there, but that was just comical: https://twitter.com/gamehawk/status/532395685824327680 and that's just one screenful.
So I stopped doing Atom with HTML::Template and built it with XML::Feed like I should have all along. To no one's surprise, XML::Feed then parsed it with no trouble. To those of you who've subscribed and now are suddenly getting all my back posts now that it's, you know, actually feed-able, I apologize.
|Coding for dollars - 13 November 2014|
We have pretty decent insurance through my husband's employers, but having learned that our out-of-pocket expenses on a scrip might be anywhere from $400 to $2000 a month, suddenly I'm thinking maybe I should be looking aggressively for a job or contract. Daaaang.
Which is annoying, because I'm really getting into Pumpkin Spice. I'm working on the default vanilla-HTML plugin, and remember why I like Perl. (Because of CPAN, mostly.) And be proud of me: I'm committing to a git (but not yet Github) repo every night.
(That title actually made me go look up Bowling for Dollars which I never actually saw. Because, it turns out, I didn't live in one of the markets. Huh.)
|Local cache - 13 November 2014|
Pumpkin Spice can now successfully monitor a feedless HTML page and turn changes into feed entries - retaining the full HTML formatting. It only pays attention to the text, not to the formatting, though if you bold something or otherwise break up text with HTML tags in the code, I think it'll treat that as a change.
To that end, I just subscribed it to all six-hundred-and-thirty-seven of us, with the exception of ~bw whose page redirects to a 404. PS won't (at this point) subscribe to nonexistent pages. All that means for now is that it cached a copy of everyone's front page, to compare to the next version when I tell it to poll. (Everyone except the half-dozen of us who have detectible XML feeds, that is.) Right now, it doesn't do anything with that data except dump it to STDOUT for debugging purposes.
But I could very trivially dump that to an RSS/Atom/JSON feed, either one for each user (basically a reinvention of Page2RSS) or one big huge sitewide group feed. I'll probably start building those, just because.
Eventually, of course, it will use that data internally, to present its feeds and feedlike things in the form a Facebook user might find familiar. Which strikes me as funny: I figured tilde pages made a good test set, but that also means that I'm, uh, kinda turning tilde.club into a social media network.
Probably my next step will be figuring out how to get my git repo up on Github. I've been faithfully committing my changes each night. (Or morning, really. My coder tendency to code late, sleep late does not work so well with my parent requirement to get up at 5:50am to get a kid ready for school.)
|Today in blogs - 15 November 2014|
I started a little thing on
Of course, having done it for GMT 14 November, I realized I hadn't blogged myself. So here's one for the 15th.
|Githubbed - 16 November 2014|
Well, Pumpkin Spice now has a home on Github. https://github.com/tyrosinase/pumpkin-spice
You can see by all the placeholder files the sorts of things I hope to read and write from: https://github.com/tyrosinase/pumpkin-spice/tree/master/Spice/lib/Spice
You can also see that I haven't done a darn thing with the module documentation. That's a side effect of the when in doubt, code approach. Now that I have a little more idea of the direction I'm going, I should get some of the forms properly filled out.
|Yesssss - 22 November 2014|
Pumpkin Spice can pull Twitter feeds in now.
On the downside, that made me realize: feeds->entries is not one-to-many, it's many-to-many. A tweet can be part of a user timeline AND part of a list timeline. So time to refactor the database a little.
|Be the customer? - 23 November 2014|
Hmm. Google's introducing this "Contributor" thing which on the surface looks like something I ought to approve of. I mean, I aggressively adblock (sorry, advertisers, you lost my trust with "Punch The Monkey" and never got it back) and then try to find an alternate way to send money or other rewards to content producers: Paypal tipping or making a Coors^Wmicrobrew run or whatever. (I once took a cheesecake made with duck eggs to the local paper's newsroom, which definitely qualifies as a "whatever.")
But one of the first takes on the program that I read was this (ironically, I think it came up through Google News): http://venturebeat.com/2014/11/20/google-contributor-may-be-an-elaborate-experiment-to-prove-people-prefer-ads-to-paid-content/
I don't know if it's intentional but honestly, I figure that'll be the end result anyway. Will I "Contribute"? I don't even know myself. None of the half-dozen example sites are ones I visit (but if I could pay to remove Wikihow crapola from all future Google searches, I might consider that).
What does that say for the prospects of Pumpkin Spice? Would ordinary people pay money for hosting to Be The Customer instead of using an ad- and data-supported social media site? Probably not, realistically. I expect people would be more inclined to use it because it offers them control over UX. People are way more likely to be annoyed, in my experience, that there is no way for a user to click a box and tell Facebook "I want to see every single post by X," especially if X is a Page.
|LibraryBox - 25 November 2014|
I bought myself one of the 14 Fabulous Open Source Gifts For The Holidays: a LibraryBox, or at least the hardware to DIY one.
It's the result of a confluence of things today: 3 Lessons For Building Tech for Low-Income Americans and What I Learned From Building An App For Low-Income Americans and an email from the CoW's neighborhood liaison with some essay questions for background on a "city-wide project examining the various strengths and deficiencies of each District in regards to communication at a community level" that made me articulate some of the things about my empire that I've been thinking about.
I'm not entirely sure yet what a LibraryBox can do that the empire might need, and I guess that's why I bought it. Sure, there's some value in being able to serve up hyperlocal news pages (since the cadre of volunteers that deliver the paper newsletter are getting too old to climb deteriorating porches), but what I really want is a community BBS, a mesh network, something more interactive.
Also, I'm a sucker for little magic boxes.
|Populating the box - 26 November 2014|
The flash drive will get here tomorrow, but the router hasn't shipped at all yet so the earliest I'll get to play with the LibraryBox is Friday. I'm entertaining myself in the meantime by accumulating the things that will go in the library.
The LibraryBox site has a sort of starter kit, which has some movies (about which I'm skeptical, bandwidth- and storage-wise, but if somebody wants to pull it then why not?) and only fifty-some books. Gutenberg is, um, a little too big to mirror, so maybe I'll write a utility that keeps the index on the LibraryBox and takes requests. That'll necessitate nightly (or so) updates, but that shouldn't be too difficult if each LibraryBox has a nearby librarian - someone who can hear its wifi from their own Internet-connected computer, ideally.
(Of course, having that sort of bridge then leads me to question: what are the local Internet providers' TOS's like in regards to sharing connections? Does the air gap bypass the "don't run a shared/open wifi off your home service" clauses?)
Apart from all the fiction, mirroring the neighborhood blog, perhaps in a more encapsulated grab-and-take-with-you format, is a given. I'd mirror the neighborhood Twitter account, only it is very often tweeting links that presume a live Internet connection. Maybe some of the Pumpkin Spice bits can be repurposed into a critter that caches the tweets and caches the linked-to pages. Since I'm kind of thinking "bus stop" as a good place to put them, probably a copy of the Wichita Transit website too.
Still trying to work through how (and if) some kind of BBS service would work, and whether it would immediately degenerate into drug-deals-and-hookups if it was anonymous and/or unmoderated.
|Security - 26 November 2014|
This morning I got an email notifying my that the router shipment for the LibraryBox hadn't happened because the credit card payment didn't go through, and that they'd retry it. Huh, I said, maybe a holiday overload, and went to my Amazon account to see whether it was going to delay shipment. All kinds of red flags, acting like the card was cancelled. Uh oh. And then I looked at the last four digits, and remembered.
We got issued a new card because we'd used ours in the timeframe of one of the big retailer breaches. We activated the new cards, went "hmph" at the length of time they'd given us to change over, and moved on. Changed all our happy recurring charges and didn't think much of it. Well, not quite "all" - apparently I forgot to update my Amazon account. And they left the old number active long enough for me to forget we'd even had a changeover, and just let us keep using two different active credit cards.
Really, Visa? You're making soooo much money that you can afford to let that potential for fraud hang in the breeze like that? Daaang.
|Feeding the library - 29 November 2014|
The LibraryBox is all loaded up and sitting in my front window, broadcasting its wifi name to the world. Or at least to the houses around me; I don't live near a high-traffic area. I'm figuring on putting it where it'll cover the center-of-the-neighborhood bus stop, and see what it does. I can count on at least four different places that would happily let me plug it in.
The puck is adorable, but that site would be a better candidate for the more conventional router with antennae for better range - they're cheaper, too. Three of those places are storefronts that could tuck it in a window and plug it in. The puck is good for mobile; I can plug it into a USB adaptor and power it off the car, or off a USB battery pack and carry it around. That's kinda cool: a bookmobile, and/or a master library that will update the others whenever it's brought into range.
Now I'm trying to decide exactly how to mirror the neighborhood blog (which, coincidentally, is me). I'd already been trying to figure out how to reduce the blog's footprint - WordPress is a pig, and there's no real need to run a dynamic site when most of it is never going to change. There are a couple of plugins that generate static pages, but they both have issues. I keep trying to remember what tool I used to use to slurp websites into a static file for offline use back in the dialup days, but it's not coming to me. All I can remember is Mhonarc, which I used around the same time to archive Phoenyx mailing lists. I guess I could write a Perl script to do it, but I hate to reinvent the wheel. Heck, I should probably just trawl through Wayback for old /Matt's Script Archive/ stuff. That's probably where it came from.
Whoa. Apparently I don't even have to use the Wayback Machine: Matt's Script Archive is still around. Hi, 1996!
|Derp - 30 November 2014|
After racking my brains for a way to build a static version of a website, I realized that I was planning to put the static pages in the exact same place as the dynamically generated ones, so I didn't have to deal with rewriting URLs - wget would work just fine.
Then I read the friendly manual and realized if I did need to rewrite URLs, wget does that too.
Geez louise. I don't know why I remember this being such a complicated thing back when. At first I thought maybe it was because that was before I had Unix shell access, but then I realized, no, that would have been pre-web. Mostly. Anyway, maybe I'm completely misremembering what I did back then, but at any rate the monster WordPress archive is now static-ified so I can start a new blog (WP or whatever, still haven't decided) without breaking all the old stuff.
The LibraryBox has a really primitive website, relying on straight-up webserver indexes for the download directories. The complaint people have is that those don't scale down for phone screens, so I figure I'll make a really simple directory-page builder. Been awhile since I've done anything with website design, apart from tweaking WordPress templates here and there, so I checked out the state of that. Realized webfonts are neat, but they make it hard to find any frameworks or templates that don't use them these days. Leaving aside all the issues of a dependency on a site you don't control, pay for, or have any input on the reliability of... webfonts aren't gonna be there for a LibraryBox.
Speaking of things that assume an always-there Internet connection: I was trying to think of a way to have some kind of community discussion platform. There's really nothing modern that has the ability to batch-transfer messages from one site to another, so I'm thinking something tildish: Usenet, or maybe Fidonet.