Category Archives: learning

Exploring neighbourhood libraries and other notes from the Toronto Public Library Hackathon

UPDATE 2015-11-18: I figured out how to make this entirely client-side, so you don’t have to run a separate server. First, install either Tampermonkey (Chrome) or Greasemonkey (Firefox). Then install the user script insert-visualize-link.user.js , and the Visualize link should appear next to the library branch options on Toronto Public Library search result pages. See the Github repository for more details.

Yay! My neighbourhood library visualization won at the Toronto Public Library hackathon. It added a Visualize link to the search results page which mapped the number of search results by branch. For example, here’s a visualization of a search that shows items matching “Avengers comics“.


It’s a handy way to see which branches you might want to go to so that you can browse through what’s there in person.

Librarians could also use it to help them plan their selections, since it’s easy to see the distribution across branches. For example, here’s the visualization for books in Tagalog.


The collections roughly match up with Wellbeing Toronto‘s data on Tagalog as the home language, although there are some areas that could probably use collections of their own.

tagalog census

Incidentally, I was delighted to learn that Von Totanes had done a detailed analysis of the library’s Filipino collections in the chapter he wrote in Filipinos in Canada: Disturbing Invisibility (Coloma, McElhinny, and Tungohan, 1992). Von sent me the chapter after I mentioned the hackathon on Facebook; yay people bumping into other people online!

Personally, I’m looking forward to using this visualization to see things like which branches have new videos. Videos released in the past year can only be borrowed in person – you can’t request them online – so it’s good to check branches regularly to see if they’re there. It would be even better if the library search engine had a filter for “On the shelf right now”, but in the meantime, this visualization tool gives me a good idea of our chances of picking up something new to watch while we’re folding laundry. =)


More notes will probably follow, but here are a few quick drawings:

2015-11-15b Tech - Exploring neighbourhood libraries -- index card #tpl #hackathon

2015-11-15b Tech – Exploring neighbourhood libraries – index card #tpl #hackathon

2015-11-15c Kaizen and the Toronto Public Library hackathon -- index card #tpl #hackathon #kaizen #improvement

2015-11-15c Kaizen and the Toronto Public Library hackathon – index card #tpl #hackathon #kaizen #improvement

2015-11-15d Other reflections from TPL hackathon -- index card #tpl #hackathon #introversion #prototyping #presenting

2015-11-15d Other reflections from TPL hackathon – index card #tpl #hackathon #introversion #prototyping #presenting

2015-11-15e Ideas for following up on TPL hackathon -- index card #prototyping #tpl #hackathon

2015-11-15e Ideas for following up on TPL hackathon – index card #prototyping #tpl #hackathon


The code works by extracting the branch names and totals on the left side of search pages and combining those with the locations of the branches (KML). I don’t really need the server component, so I’m thinking of rewriting the script so that it runs entirely client-side – maybe as a Chrome extension or as a user script. That way, other people can play with the idea without running their own server (and without my having to keep a server around), and we can try it out without waiting for the library to integrate it into their website. That said, it would be totally awesome to get it into the interface of the Toronto Public Library! We’ll just have to see if it can happen. =) Happy to chat with library geeks to get this sorted out.

It was fun working on this. W- decided to join me at the last minute, so it turned into a fun weekend of hanging out with my husband at the library. I wanted to keep my weekend flexible and low-key, so I decided not to go through the team matchmaking thing. W- found some comfy chairs in the corner of the area, I plugged in the long extension cord I brought, and we settled in.

I learned a lot from the hackathon mentors. In particular, I picked up some excellent search and RSS tips from Alan Harnum. You can’t search with a blank query, but he showed me how you can start with a text string, narrow the results using the facets on the left side, and then remove the text string from the query in order to end up with a search that uses only the facets. He also showed me that the RSS feed had extra information that wasn’t in the HTML source and that it could be paginated with URL parameters. Most of the RSS feeds I’d explored in the past were nonpaginated subsets of the information presented on the websites, so it was great to learn about the possibilities I had overlooked.

The faceted search was exactly what I needed to list recent videos even if I didn’t know what they were called, so I started thinking of fun tools that would make hunting for popular new videos easier. (There have been quite a few times when I’ve gone to a library at opening time so that I could snag a video that was marked as available the night before!) In addition to checking the specific item’s branch details to see where it was on the shelf and which copies were out on loan, I was also curious about whether we were checking the right library, or if other libraries were getting more new videos than our neighbourhood library was.

W- was curious about the Z39.50 protocol that lets you query a library catalogue. I showed him the little bits I’d figured out last week using yaz-client from the yaz package, and he started digging into the protocol reference. He figured out how to get it to output XML (format xml) and how to search by different attributes. I’m looking forward to reading his notes on that.

Me, I figured that there might be something interesting in the visualization of new videos and other items. I hadn’t played around a lot with geographic visualization, so it was a good excuse to pick up some skills. First, I needed to get the data into the right shape.

Step 1: Extract the data and test that I was reading it correctly

I usually find it easier to start with the data rather than visualizations. I like writing small data transformation functions and tests, since they don’t involve complex external libraries. (If you miss something important when coding a visualization, often nothing happens!)

I wrote a function to extract information from the branch CSV on the hackathon data page, using fast-csv to read it as an array of objects. I tested that with jasmine-node. Tiny, quick accomplishment.

Then I worked on extracting the branch result count from the search results page. This was just a matter of finding the right section, extracting the text, and converting the numbers. I saved a sample results page to my project and used cheerio to parse it. I decided not to hook it up to live search results until I figured out the visualization aspect. No sense in hitting the library website repeatedly or dealing with network delays.

Step 2: Make a simple map that shows library branches

I started with the Google Maps earthquake tutorial. The data I’d extracted had addresses but not coordinates. I tried using the Google geocoder, but with my rapid tests, I ran into rate limits pretty early. Then it occurred to me that with their interest in open data, the library was the sort of place that would probably have a file with branch coordinates in terms of latitude and longitude. The hackathon data page didn’t list any obvious matches, but a search for Toronto Public Library KML (an extension I remembered from W-‘s explorations with GPS and OpenStreetMap) turned up the file I wanted. I wrote a test to make sure this worked as I expected.

Step 3: Combine the data

At first I tried to combine the data on the client side, making one request for the branch information and another request for the results information. It got a bit confusing, though – I need to get the hang of using require in a from-scratch webpage. I decided the easiest way to try my idea out was to just make the server combine the data and return the GeoJSON that the tutorial showed how to visualize. That way, my client-side HTML and JS could stay simple.

Step 4: Fiddle with the visualization options

Decisions, decisions… Red was too negative. Blue and green were hard to see. W- suggested orange, and that worked out well with Google Maps’ colours. Logarithmic scale or linear scale? Based on a maximum? After experimenting with a bunch of options, I decided to go with a linear scale (calculated on the server), since it made sense for the marker for a branch with a thousand items to be significantly bigger than a branch with five hundred items. I played with this a bit until I came up with maximum and minimum sizes that made sense to me.

Step 5: Hook it up to live search data

I needed to pass the URL of the search results, and I knew I wanted to be able to call the visualization from the search results page itself. I used TamperMonkey to inject some Javascript into the Toronto Public Library webpage. The library website didn’t use JQuery, so I looked up the plain-vanilla Javascript way of selecting and modifying elements.

  .parentNode.querySelector('h3').innerHTML =
  'Library Branch <a target="_blank" style="color: white; ' +
  'text-decoration: underline" ' +
  'href="http://localhost:9000/viz.html?url=' +
  encodeURIComponent(location.href) + '">(Visualize)</a>';

Step 6: Tweak the interface

I wanted to display information on hover and filter search results on click. Most of the tutorials I saw focused on how to add event listeners to individual markers, but I eventually found an example that showed how to add a listener to and get the information from the event object. I also found out that you could add a title attribute and get a simple tooltip to display, which was great for confirming that I had the data all lined up properly.

Step 7: Cache the results

Testing with live data was a bit inconvenient because of occasional timeouts from the library website, so I decided to cache search results to the filesystem. I didn’t bother writing code for checking last modification time, since I knew it was just for demos and testing.

Step 8: Prettify the hover

The tooltip provided by title was a little bare, so I decided to spend some time figuring out how to make better information displays before taking screenshots for the presentation. I found an example that showed how to create and move an InfoWindow based on the event’s location instead of relying on marker information, so I used that to show the information with better formatting.

Step 9: Make the presentation

Here’s how I usually plan short presentations:

  1. Figure out the key message and the flow.
  2. Pick a target words-per-minute rate and come up with a word budget.
  3. Draft the script, checking it against my word budget.
  4. Read the script out loud a few times, checking for time, tone, and hard-to-say phrases.
  5. Annotate the script with notes on visual aids.
  6. Make visuals, take screenshots, etc.
  7. Record and edit short videos, splitting them up in Camtasia Studio by using markers so that I can control the pace of the video.
  8. Copy the script (or keywords) into the presenter’s notes.
  9. Test the script for time and flow, and revise as needed.

I considered two options for the flow. I could start with the personal use case (looking for new videos) and then expand from there, tying it into the library’s wider goals. That would be close to how I developed it. Or I could start with one of the hackathon challenges, establish that connection with the library’s goals, and then toss in my personal use case as a possibly amusing conclusion. After chatting about it with W- on the subway ride home from the library, I decided to start with the second approach. I figured that would make it easier for people to connect the dots in terms of relevance.

I used ~140wpm as my target, minus a bit of a buffer for demos and other things that could come up, so roughly 350 words for 3 minutes. I ran through the presentation a few times at home, clocking in at about 2:30. I tend to speak more quickly when I’m nervous, so I rehearsed with a slightly slower pace. That way, I could get a sense of what the pace should sound like. During the actual presentation, though, I was a teensy bit over time – there was a bit of unexpected applause. Also, even though I remembered to slow down, I didn’t breathe as well as I probalby should’ve; I still tend to breathe a little shallowly when I’m on stage. Maybe I should pick a lower WPM for presentations and add explicit breathing reminders. =)

I normally try to start with less material and then add details to fit the time. That way, I can easily adjust if I need to compress my talk, since I’ve added details in terms of priority. I initially had a hard time concisely expressing the problem statement and tying together the three examples I wanted to use, though. It took me a few tries to get things to fit into my word budget and flow in a way that made me happy.

Anyway, once I sorted out the script, I came up with some ideas for the visuals. I didn’t want a lot of words on the screen, since it’s hard to read and listen at the same time. Doodles work well for me. I sketched a few images and created a simple sequence. I took screenshots for the key parts I wanted to demonstrate, just in case I didn’t get around to doing a live demo or recording video. That way, I didn’t have to worry about scrambling to finish my presentation. I could start with something simple but presentable, and then I could add more frills if I had time.

Once the static slides were in place, I recorded and edited videos demonstrating the capabilities. Video is a nice way to give people a more real sense of how something works without risking as many technical issues as a live demo would.

I had started with just my regular resolution (1366×768 on my laptop) and a regular browser window, but the resulting video was not as sharp as it could have been. Since the presentation template had 4:3 aspect ratio, I redid the video with 1024×768 resolution and a full-screen browser in order to minimize the need for resizing.

I sped up boring parts of the video and added markers where I wanted to split it into slides. Camtasia Studio rendered the video into separate files based on my markers. I added the videos to individual slides, setting them to play automatically. I like the approach of splitting up videos onto separate slides because it allows me to narrate at my own pace instead of speeding up or slowing down to match the animation.

I copied the segments of my script to the presenter notes for each slide, and I used Presenter View to run through it a few more times so that I could check whether the pace worked and whether the visuals made sense. Seemed all right, yay!

Just in time, too. I had a quick lunch and headed off to the library for the conclusion of the hackathon.

There wsa a bit of time before the presentations started. I talked to Alan again to show him what I’d made, hear about what he had been working on, and pick his brain to figure out which terms might resonate with the internal jargon of the library – little things, like what they call the people who decide what kinds of books should be in which libraries, or what they call the things that libraries lend. (Items? Resources? Items.) Based on his feedback, I edited my script to change “library administrators” to “selection committees”. I don’t know if it made a difference, but it was a good excuse to learn more about the language people used.

I tested that the presentation displayed fine on the big screen, too. It turned out that the display was capable of widescreen input at a higher resolution than what I’d set, but 1024×768 was pretty safe and didn’t look too fuzzy, so I left it as it was. I used my presentation remote to flip through the slides while confirming that things looked okay from the back of the room (colours, size, important information not getting cut off by people’s heads, etc.). The hover text was a bit small, but it gave the general idea.

And then it was presentation time. I was third, which was great because once I finished, I could focus on other people’s presentations and learn from their ideas. Based on W-‘s cellphone video, it looks like I remembered to use the microphone so that the library could record, and I remembered to look up from my presenter notes and gesture from time to time (hard when you’re hidden behind the podium, but we do what we can!). I stayed pretty close to my script, but I hope I kept the script conversational enough that it sounded more like me instead of a book. I didn’t have the mental bandwidth to keep an eye on the timer in the center of the presenter view, but fortunately the time worked out reasonably well. I concluded just as the organizer was getting up to nudge me along, and I’d managed to get to all the points I wanted to along the way. Whew!

Anyway, that’s a quick braindump of the project and what it was like to hack it together. I’ll probably write some more about following up on ideas and about other people’s presentations, but I wanted to get this post out there while the experience was fresh in my head. It was fun. I hope the Toronto Public Library will take the hackathon ideas forward, and I hope they’ll get enough out of the hackathon that they’ll organize another one! =)

Thinking about problem-solving and sequencing

We’ve been helping J- with her culminating project in Grade 11 programming class, a text-based blackjack game. She was getting stuck because she wasn’t comfortable enough with Java or programming. She didn’t know where to start or what to do. I didn’t write the program for her (and that was never on the table, anyway), but I offered to guide her through the process. More experienced learners can do that kind of planning on their own, but when you’re new to a subject (and especially if you’re under time pressure), it helps to have that kind of scaffolding.

The first thing I did was to properly set up Eclipse on her computer. She had already tried setting it up, but it kept quitting with a cryptic error message. I realized that she didn’t have the Java software development kit installed. Once we added that, Eclipse started working.

Then I set up a sequence of tiny, tiny steps that she could implement with a little guidance. This was not the same sequence of events that would be in the final game: betting, shuffling, dealing, scoring, and so on. Instead, I focused on the steps she could build on bit by bit. It went something like this:

  1. Display a random number.
  2. Initialize the deck array based on the face array and suit array J- had already set up. This would contain strings like “ace of hearts”.
  3. Print the deck array.
  4. Set up a parallel array of card values, and print that as well. This would contain integers such as 1 for ace.
  5. Shuffle the deck by swapping each card with a random card, swapping the same card’s value as well.
  6. Write a function that takes an array of card values and returns the value of the hand.
  7. Modify the function to take aces into account, since aces could be either 1 or 11.
  8. Handle the initial dealing of two cards each by keeping track of the top card and updating a status array.
  9. Display the cards for a specified hand based on the status array.
  10. Update the hand value function to take the status array into account.
  11. Deal cards to the player as requested.
  12. Check if the player has lost.
  13. Follow the rules for dealing additional cards to the dealer.
  14. Check if the dealer has lost.
  15. Figure out who won that round.
  16. Read the bet.
  17. Update the bet after the player has won or lost the round.
  18. Check if the player has doubled their money or lost all their money.
  19. Add more error-checking.
  20. Add more flexibility.

2015-06-11a Blackjack implementation sequence -- index card #sequencing #problem-solving

2015-06-11a Blackjack implementation sequence – index card #sequencing #problem-solving

This sequence meant that she could write and test the less-interactive parts first (preparing the deck, shuffling the cards, calculating the score) before slowing down her compile-run-test cycle with input. It also let her work on the simplest parts first without writing a lot of redundant code and with just a little prompting from me.

Instead of this zigzag path, we could have followed the chronological flow of the program, especially if I introduced her to the practice of stubbing functions until you’re ready to work on them. This shuffled order felt a bit better in terms of demonstrable progress, and she seems to have been able to follow along with the construction process. In retrospect, the chronological flow might have been easier for her to learn and apply to other projects, though, since breaking down and shuffling the order of tasks is a skill that programming newbies probably need a while to develop.

Anyway, helping J- with her project got me thinking about how I work my way through programming challenges on my personal projects and in my consulting. I tend to try to figure things out by myself instead of asking mentors or the Internet. I also tend to write things in a bottom-up instead of top-down order: starting with small things I can write and test quickly, and then gradually building more elaborate processes around them.

2015-06-09e What do I mean by sequencing -- index card #learning #problem-solving #sequencing

2015-06-09e What do I mean by sequencing – index card #learning #problem-solving #sequencing

I think of the process of breaking down tasks and organizing them into a useful order as “sequencing”, which is part of the general domain of problem-solving. There’s probably an official term for it, but until I find it, that’s the term makes sense to me. When I was teaching undergraduate computer science, I noticed that students often struggled with the following aspects:

  • Imagining a good goal, if they were doing a self-directed project (not too big, not too small, etc.)
  • Fleshing out their goal into specifications
  • Identifying possible chunks along the path
  • Breaking those chunks down into smaller chunks as needed
  • Figuring out how to bridge those chunks together
  • Prioritizing chunks
  • Re-evaluating paths, progress, and goals as they learned more
  • Managing motivation
  • Finding resources
  • Translating or adapting those resources to what they needed to do
  • Figuring out what they could start with (what they already knew, or that first hand-hold on a chunk)

2015-06-09h Sequencing challenges -- index card #learning #problem-solving #sequencing #challenges

2015-06-09h Sequencing challenges – index card #learning #problem-solving #sequencing #challenges

I sometimes have a hard time with these aspects too, especially when I’m learning a new toolkit or language. I think I’m getting better at picking ridiculously tiny steps and celebrating that progress instead of getting frustrated by blocks. When I can’t figure tiny steps out, I know that going through tutorials and reading other people’s code will help me build my vocabulary. That way, I can get better at finding resources and planning chunks. I have a better idea of what’s out there, what things are called, and maybe even what’s probably harder and what’s probably easier.

2015-06-09f How do I develop my own sequencing skills -- index card #learning #problem-solving #sequencing

2015-06-09f How do I develop my own sequencing skills – index card #learning #problem-solving #sequencing

Programming gives me plenty of opportunities to develop my sequencing skills. By writing notes with embedded code snippets and TODOs, I can break large chunks down into smaller chunks that fit in my working memory. Sometimes I read programming documentation or source code without any particular project in mind, just collecting ideas and expanding my imagination. If I focus on writing tiny chunks that start off as “just good enough” rather than making them as elaborate as I can, that lets me move on to the rest of the program and get feedback faster. Saving snippets of unused or evolving code (either manually in my notes or automatically via a version contro system) lets me reuse them elsewhere.

I imagine that getting even better at sequencing might involve:

  • Sharing intermediate or neighbouring products to get feedback faster and create more value
  • Figuring out (and remembering!) sequences for getting into various areas such as Angular or D3
  • Becoming fluent with many styles of sequencing, like working top-down instead of just bottom-up
  • Building reliable automated tests along the way, even as my understanding of the project evolves

What about helping other people get better at sequencing?

2015-06-09g How can I help other people develop sequencing skills -- index card #learning #problem-solving #sequencing

2015-06-09g How can I help other people develop sequencing skills – index card #learning #problem-solving #sequencing

  1. If I share a pre-defined sequence, that helps people with less experience follow a possibly more efficient path.
  2. If I customize the sequence for someone I’m helping, that’s even more useful.
  3. Talking about how we’re customizing the sequence exposes sequencing as a skill.
  4. Guiding someone through the breakdown of a small chunk (from a task to pseudocode, for example) can help them get the hang of turning things into specifics.
  5. Guiding someone through the breakdown of a larger chunk into smaller chunks helps them think of larger chunks as doable.
  6. Once someone has identified a few chunks, I can help with prioritizing them based on effort, relationship between chunks, etc.
  7. In preparation for independent learning, it’s good to practice the skill of figuring out those prioritization factors: small experiments, research, and so on.
  8. And at some point (woohoo!), I can help someone reflect on and tweak a plan they developed.

I’m not yet at the point of being able to do even step 1 well, but maybe someday (with lots of practice)!

Since this is a super-useful skill and not everyone’s into programming, it might be interesting to look around for non-programming situations that can help develop sequencing skills.

2015-06-09i Non-programming examples of sequencing -- index card #learning #problem-solving #sequencing

2015-06-09i Non-programming examples of sequencing – index card #learning #problem-solving #sequencing

On the passive side, it’s interesting to learn about how things work or how they’re made. Better yet, you can actively practise sequencing with math problems, video games with quests… Writing is like this too, if you think of key points and use tools such as outlines or index cards. Making things involves imagining what you want and figuring out lots of small steps along the way. Life is filled with little opportunities for everyday problem-solving.

I’d love to learn more about this. Chunk decomposition? Problem-solving order? Problem decomposition? Step-wise refinement? I’m more comfortable with bottom-up problem-solving, but top-down solving has its merits (focus, for one), so maybe I can practice that too. Using both approaches can be helpful. (Ooh, there’s a Wikipedia article on top-down and bottom-up design…) Time to check out research into the psychology of problem-solving!

Thinking about changing interests

I’ve gone through quite a lot of interests. Sometimes they combine in useful ways, like the way coding and writing are imbued into practically all the other interests I have. Sometimes they last for years, and sometimes they’re over in months. From other people’s stories, I get the sense that this will likely continue throughout my life. =)

I want to do this better – the cultivation of interests, and the occasional letting go. Why do I want to do it better? I want to minimize the risk of being in the following situations:

  • When I overcommit to an interest:
    • Spending more money, time, or space than an interest needs
    • Making long-term promises (ex: speaking, organizing) that I might feel weird about
  • When I find myself in a lull because I haven’t cultivated my interests enough:
    • Recognizing temporary interest distance and accepting or working around it
    • Accepting the feeling of being a beginner and getting to the point of enjoyment
    • Letting go at the apppropriate point

In addition to those downsides, I want to make better use of the upsides:

  • During the initial period of fascination, I want to take unselfconscious notes
  • As my interest matures, I want to get better at seeking and organizing information, and then coming up with my own thoughts
  • I want to get better at drawing connections between interests and taking advantage of the combinations

So I’d like to learn more about how I think and learn. What kinds of things am I interested in? Why do my interests change? What stages do I go through? How can I make just the right level of commitment, feeding fledgling interests without adding too much weight to them, building on mature interests without stretching them too far, and taking breaks or letting go gracefully without flaking out? Should I focus on developing interest persistence, or get better at going with the flow?

Here’s an example of an interest I’m thinking through: I like Emacs and its community. I seem to get into Emacs cyclically. For a few months each year, I spend a lot of time looking closely at how I use Emacs and learning from what other people do. I hang out in Emacs communities, pay attention to mailing lists, tweak my config, write blog posts, sometimes create resources. (My Emacs geekery really is quite oddly rhythmic. Here’s the data by month since Nov 2011.)

Then other things take my attention, and I drift off. I haven’t tweaked my configuration or written an Emacs-related blog post in a while. I’ll get back to it at some point, I know – the oddest thing will bring me back: some idea or question – but in the meantime, I’m fine with letting it be for now.

But I’ve set up monthly Emacs Hangouts on my calendar and in Google+ events, because precommitting to those means that they happen. And there’s some kind of an Emacs Conference that I think would be an excellent idea, but I haven’t been able to muster the energy to do the kind of social outreach that I think would be needed in order to get the schedule sorted out. And there are the occasional requests for help that come in, even though I don’t feel I can contribute even a fraction of what or the relevant mailing lists could.

I feel like it would be good for me to be closer to that interest, but there are other things on my mind at the moment, so I leave things hanging. I’ll be there for the Emacs Hangouts I’ve set up, but I haven’t felt like doing anything else lately: lining up people for Emacs Chat podcasts, writing or drawing a review of Mastering Emacs, exploring the awesome new packages that are out there…

On the other hand, I know that sometimes all it takes is a little time immersing myself in it: checking out StackExchange questions or IRC conversations, reading source code, going through my long TODO list of Emacs things to learn. Likely that will kickstart my interest.

In the meantime, this lull itself is curious and interesting, because I rarely get to pay attention to feelings like this. It feels odd to be a little bit distant from Emacs and reading books, two of my long-term interests. Writing, drawing, coding, and data analysis continue to be interesting. My sewing is on hold; I think keeping myself to one type of garment a year seems like a good way to avoid burnout. Gardening has been slowed to the pace that nature keeps. I notice a fledgling interest in cognitive research and psychology.

I’m taking it easy, fanning interest when I can and relaxing when I feel like doing that instead. A mix of routine and freedom helps, I think. I like writing and coding in the morning. Sometimes it takes a little effort to get started, especially with writing, but then I get going. In the afternoon, it’s okay to relax.

I don’t think that my values and the things that tickle my brain have changed, so I’ll probably return to my long-term interests once my new interests settle down and get integrated. They’ll be richer for it too, like the way coding got better when I added writing, and writing got better when I added drawing. In the meantime, I’m curious about charting the shifts in my focus and making the most of them.

Hmm, I wonder if this is related to my hesitation around the Quantified Self talk we’re planning for November: I’m not sure if I’ll be able to give the enthusiastic performance that I imagine newcomers would find helpful…

Aha! I think that might explain it. By myself, I’m okay with the shifts in my interests. I just try to take good notes and share them along the way. Social commitments add friction to interest-changing because I don’t want to flake out and I don’t want to fake things, which is why I’m reluctant to make plans even if I miss out on opportunities because I don’t want to do so. However, it would probably be good for me to know how to work with this, because social commitments are a good way to help make things that are bigger than yourself. If I remind myself that (a) at the core, I’m still likely to enjoy the things that drew me to that interest in the first place, and (b) it’s not the end of the world even if I mess up, that might help me reduce the anxiety around essentially making a promise that future me will still have the same passions that people are drawn to in the present. So I’m likely to still avoid making big commitments (say, no convincing people to quit their jobs and start a company with me), but I can practise with the small ones I have.

At the moment, I think I’ll still want someone else to take point on organizing the Emacs Conference speaker schedule, but I can re-evaluate that in two weeks, and we can always move it further out if needed. I should be able to handle the Quantified Self talk – worst-case scenario is I don’t manage to inspire and connect with people, but I don’t expect a small 1-hour talk to change people’s lives that much anyway. So it’s okay even if I don’t feel 100% there in terms of the interests right now. I have enough good memories to know I’ll probably feel that way about those interests again soon, so I can plan accordingly.

It’s a little odd teasing apart temporary factors and long-term factors in my mind, but I’m glad I can sit and write my way through it. In the meantime, I’ll focus on keeping my experiences of those interests pleasant, tickling my brain whenever I can. There’s so much depth to each interest that I don’t really need to add more. But on the other hand, the combinations can be quite interesting, so I’ll explore away. =)

Leaning into absent-mindedness

From time to time, I notice a spike in the number of small mistakes I make due to inattention. It’s a good sign to slow things down, rejig systems and habits, and figure out how to make things better. For example, noticing that I often lost track of small things I was carrying around, I switched to a belt bag in summer and a vest in winter.

There are still quite a few slips I haven’t figured out how to work around, like the occasional times I put the oven mitts on the opposite side of the stove from where they usually are (I must have absentmindedly thought “Aha! An empty hook!”), or the time I tucked the sesame oil into the fridge. (“I’m holding a bottle; many bottles go into the shelves on the fridge door; this probably goes into the shelves on the fridge door.”)

2015-05-28b Absent-mindedness -- index card #research #fuzzy

2015-05-28b Absent-mindedness – index card #research #fuzzy

It turns out that there are lots of forms of absent-mindedness. Cheyne, Carierre, and Smilek (2005) defined a scale for attention-related cognitive errors (ARCES) that goes like this:

  1. I have absent-mindedly placed things in unintended locations (e.g., putting milk in the pantry or sugar in the fridge).
  2. When reading I find that I have read several paragraphs without being able to recall what I read.
  3. I have misplaced frequently used objects, such as keys, pens, glasses, etc.
  4. I have found myself wearing mismatched socks or other apparel.
  5. I have gone into a room to get something, got distracted, and left without what I went there for.
  6. I fail to see what I am looking for even though I am looking right at it.
  7. I begin one task and get distracted into doing something else.
  8. I have absent-mindedly mixed up targets of my action (e.g. pouring or putting something into the wrong container).
  9. I make mistakes because I am doing one thing and thinking about another.
  10. I have gone to the fridge to get one thing (e.g., milk) and taken something else (e.g., juice).
  11. I have to go back to check whether I have done something or not (e.g., turning out lights, locking doors).
  12. I go into a room to do one thing (e.g., brush my teeth) and end up doing something else (e.g., brush my hair).

I find that I tend to be okay at broad strokes (intentions), but sometimes I miss finer details. I’ve walked out of the house in inside-out or back-to-front clothing before (technical shirts feel the same either way!), although usually W- helps me catch those situations.

It’s not that bad, though. Although I sometimes don’t remember what I walked into a room for (especially if I get distracted by a conversation part way), I can almost always recall what I intended to do, and what was before that (if I hadn’t finished that yet). It also helps to have the habit of writing down quick notes and consulting my agenda for tasks to work on, but mental rehearsal is usually enough for me to “pop the stack”.

Fortunately, all this appears to be normal human experience. I might be a smidge more absent-minded than some folks, but it doesn’t get in the way of life, and even W- forgets a mug of hot water in the microwave occasionally. Besides, I enjoy working around the limitations of my brain by taking notes and tweaking the way I live.

This is probably why I enjoy reading research into the brain. It turns out that there are many possible explanations for absent-mindedness. There are different ways to measure it, and even a few ways to play around with it.

2015-05-28c Different models of absent-mindedness -- index card #fuzzy #research

2015-05-28c Different models of absent-mindedness – index card #fuzzy #research

When I read through the research, I feel oddly optimistic. Even though I know I’m likely to get more absent-minded as I grow older, I also know that experience, mindfulness, more deliberate responses, and good habits using external-memory systems can help a lot.

2015-05-24d Accept or hack fuzziness -- index card #fuzzy

2015-05-24d Accept or hack fuzziness – index card #fuzzy

I notice that I respond to the fuzziness in my brain with curiosity instead of frustration. I like this attitude, and I hope to keep it as I go through life. Instead of getting frustrated with myself, I get a good laugh out of the little mishaps (oh hey, I’ve put the plates where the saucers usually go; I can see how that happened!), and I explore it to learn more. So a bit of both, I guess: accept the fuzziness and hack around it.

Besides, the incidents aren’t that frequent. They’re just more prominent in my memory because I pay attention to them. =)

2015-05-28e Sneak previews of life -- index card #fuzzy

2015-05-28e Sneak previews of life – index card #fuzzy

Actually, it works out really nicely that I’m thinking about this at this time. I know people around me also experience absent-mindedness, so I don’t have to have a hypochrondiac’s worry about early-onset diseases. (Although if we get to the point where this does actually get in the way of an awesome life, I’ll be sure to ask for help.) Instead, since I keep my life relatively smooth (low stress, plenty of sleep), I have a baseline of feeling good. That lets me notice changes more clearly, instead of the changes getting obscured in the noise of perpetual sleep deprivation or constant background stress. It also means that I can think of fuzzy-brain moments as temporary, local, and impersonal, and I can use my non-fuzzy times to figure out how to make the fuzzy times even better.

What are some things that could make absent-minded moments better?

2015-05-28d Imagining adaptations for absent-mindedness -- index card #fuzzy

2015-05-28d Imagining adaptations for absent-mindedness – index card #fuzzy

I might need to wait for better technology for some of these ideas, but most of the ideas are ready to go. Putting things in the wrong places is a minor inconvenience, and safety hasn’t been a big issue for me yet. I’ll probably focus on fuzzy memory and observation, looking for ways to take notes on or automate the things I do. For example, I’ve added notes on how to find and deploy code to the TODO lists for my personal projects, since I might go a few months without thinking about them. Notes also help with checking and monitoring. As I gain more experience and develop those systems and habits, that will help with brain fog as well. I trust my lists to help me with task disruption, and I keep lots of buffers in my life to soften the impact of forgetting. It’s a fascinating balance between taking things slowly and keeping things interesting enough so that my brain doesn’t go into too much of an automatic mode.

It’s odd how taking this kind of perspective changes how I experience forgetfulness. Instead of thinking to myself, “Where did I put those keys? I suck!”, I find myself thinking, “Oh look! I wonder what I’ll learn from this one…” We’ll see!

Mapping knowledge

I chatted with someone about maps and personal knowledge management, so I thought I’d write an extended reflection.

2015-05-13e Mapping knowledge for yourself and others -- index card #mapping #pkm #sharing

2015-05-13e Mapping knowledge for yourself and others – index card #mapping #pkm #sharing

Mapping is useful for myself and for others. For managing my own learning:

  • Scope: What’s included, and what’s not? How does this relate to other things I’ve learned or I’m learning?
  • Landmarks and destinations: Role models, motivation, tracking progress…
  • Main path, detours: How do you get from A to B? Are there interesting places in the neighbourhood?
  • Here there be dragons, places under construction: Managing appropriate difficulty; tracking areas to explore or revisit

When helping other people learn, mapping lets me:

  • Define scope: Define a manageable chunk, and link to related maps: zooming in, zooming out, going to other places
  • Provide landmarks
  • Main path, detours: Organize a reasonable path (particularly based on someone’s interests) and nice detours
  • Here there be dragons / construction: Warn newbies, encourage intermediate/advanced exploration

So here’s my current workflow:

2015-05-13f Mapping what I know -- index card #workflow #blogging #index-cards #mapping #pkm

2015-05-13f Mapping what I know – index card #workflow #blogging #index-cards #mapping #pkm

2015-05-08c Managing my structured information -- index card #pkm #knowledge #sharing

2015-05-08c Managing my structured information – index card #pkm #knowledge #sharing

Using index cards, outlines, and chunks seems to be working well for me in terms of current thinking, although I haven’t been turning my attention to organizing, fleshing out knowledge, and filling in gaps.

Here are some notes from 2013 on mapping forwards (plans) and backwards (guides for other people). I’ve figured out ways around some of the challenges I encountered before:

  • Rough categorization of blog posts: I’ve written some Emacs Lisp code to help me update my blog post index monthly.
  • Hundreds of sketches with few links: Now I have more than a thousand sketches! But that’s okay, I have metadata in the filename, integration in my outline, and eventual chunking into blog posts.
  • Duplicate metadata entry, no synchronization: Tags in the filename and a NodeJS script that sets the same tags on Flickr upload, yay
  • No clear picture of follow-up questions, ideas, or actions: Outline still needs work; maybe also a quick way to review open sketches?
  • No clear role models: Found historical and contemporary ones, yay!

Mostly I’ve been focusing on little explorations rather than map-making. It’s like collecting nature specimens so that I can start to classify them, since you don’t see that order until later. Sometimes I look back and retrace my path. That’s when I can try to figure out where things are and how people might go a little faster or in a better order. Other times, when I’m looking forward, I’m trying to see what’s close by and how to get there. I remind myself of the landmarks in the distance, too, and what progress might look like. But I can only walk the routes until I reach a height that lets me review the paths ahead, so sometimes it’s just the accumulation of steps…

2015-05-12d What do I want to get from my blog archive, looking back twenty years from now -- index card #blogging #pkm #archive

2015-05-12d What do I want to get from my blog archive, looking back twenty years from now – index card #blogging #pkm #archive

2015-05-12e What do I want from my archive of index cards -- index card #pkm #archive #drawing #index-cards

2015-05-12e What do I want from my archive of index cards – index card #pkm #archive #drawing #index-cards

The balance between doing and improving – evaluating yak-shaving

A reader wrote:

… I came to realize that many Emacs users seem to spend a great deal of time learning about Emacs, tweaking it, and writing new extensions, rather than getting non-Emacs-related work done. Sometimes it feels as though heavy Emacs users actually get less done overall, if you consider only non-Emacs-related tasks. My question is, is it possible to get work done in Emacs, without most of that work being Emacs-related?

It got me thinking about skills or tools that can be used to improve themselves, and the balance between using and improving tools.

2015-03-15c Skills or tools that can be used to improve themselves -- index card #learning #bootstrapping

2015-03-15c Skills or tools that can be used to improve themselves – index card #learning #bootstrapping

Not all skills or tools can be used to improve themselves. I’m learning how to sew, but that doesn’t lead to making my sewing machine better (aside from fiddling with the dials).

Here are some skills that can be used reflexively:

  • Philosophy asks questions about good questions to ask
  • Learning about learning helps you learn more effectively
  • Woodworkers and machinists have a tradition of making their own tools
  • 3D printers can print parts for their own models
  • You can program tools to help you program better: testing, version control, project management, etc.

Although making your own tools takes time, here are some advantages of doing so instead of buying them off the shelf:

  • You understand the internals better, and you can appreciate the subtleties
  • You can customize it to fit the way you work
  • You can create different variants for greater flexibility. Mass customization can’t anticipate or cost-effectively provide all the different types of things people may want.
  • As your skills and needs increase, you can create better and better tools for yourself.

Many programmers spend time deliberately improving their toolkits; if they don’t, they stagnate. At the basic level, people try programs or frameworks that other people have created. The next level might be scripting things to work together. A third level might be writing customizations or extensions, and a fourth level might be creating entirely new tools or frameworks. Beginner programmers might start at the first level of reusing other people’s code, but wizardly performance often involves a mix of the other levels.

So the question is: How can we balance doing things and improving things?

No one can answer this for you.

Me, I tend to avoid hard deadlines and I do things faster than people expect them to be done, so I have plenty of leeway to improve my tools – which helps me be even more effective, so it’s a virtuous cycle.

You’ll need to find your own balance. You might get urgent stuff out of the way first, and then figure out how to balance smaller requests with investing in capabilities.

Here’s something I put together to help you figure out where you might be in terms of balance. Alternatively, if you’re thinking about whether to pick up a skill or tool that can be used to improve itself, you can use this to evaluate what you read from people sharing their experiences with the tool. Can they find a good balance for themselves, or are they frustrated by the challenges of getting something to work?

2015-03-16a The balance between using and improving tools -- index card #learning #bootstrapping

2015-03-16a The balance between using and improving tools – index card #learning #bootstrapping

  • “I have what I need in order to work.” This is the basic scenario. People focus on doing things instead of improving things.
  • I can keep pushing, but performance is dropping, so I should invest time in maintenance.” It’s like the way a knife or a saw dulls over time. When you notice diminishing returns, it might be good to invest some time in maintenance. It’s not an urgent need, but it can pay off.
  • I’d better take care of this now before it becomes a problem.” This is like maintaining a car or taking care of your health. A little time now can avoid big problems later.
  • Grr, it’s broken. I have to fix it before I can work.” If you let things go for too long, or if you’re working with something finicky, you’ll be forced into maintenance mode. For example, some 3D printers require a lot of fiddling. Watch out for this scenario.
  • It’s fine the way it is, but I know I can make it better.” The way you’re currently doing things is okay, but you know (from your experience or from what you’ve read of other people) that you can invest a little time to work more effectively. You might even know the return on investment. It’s easy to decide whether you should just go ahead with the status quo or invest the time in improving.
  • It’s fine the way it is, but I think I can make it better.” The way you’re currently doing things is okay, but you have some ideas that might make it even better. If you think those ideas might be worth it, it might be good to give yourself a time limit for exploring those ideas so that you don’t get distracted. Alternatively, you can save it for a slower time.
  • I’m waiting or stuck, so I might as well work on tools.” Maybe you’re waiting for feedback from someone else. Maybe you’re waiting for programs to compile or tests to pass. Why not spend a little time exploring how to make your tools a little better?
  • I’m doing this for fun/learning.” Tool improvement can become more enjoyable than some of the other ways you used to like spending time. For example, you might find yourself wanting to watch a screencast or try out a tweak instead of watching TV or browsing random sites on the Internet. You don’t have to completely replace other activities, you just have to shift a little time from things that have less value to you.
  • I can’t write about my actual work, but I can write about this.” If you’re wondering about yak-shaving propensity based on the blog posts you’re reading, consider: do people write about their improvements instead of the work that they’re doing because their work is confidential or hard to explain? Maybe they think blog posts about improvements are more interesting. Maybe they’re writing about improvements in the process of figuring things out (which in an excellent process, by the way). All these things can skew your perception of how much time people spend doing things versus improving things, and how much they accomplish within that time.

In terms of Emacs, these things mostly apply to me:

  • “I’m doing this for fun/learning” – Emacs tickles my brain, and the community is wonderful.
  • “I can’t write about my actual work, but I can write about this” – I suppose I could write more about the other stuff I’m interested in (sewing? cooking?), so there’s that. However, the consulting stuff is covered by agreements, and that’s a small fraction of my life anyway.

I assume other geeks are rational, especially if they have a lot of experience with it and other tools. Therefore, if people spend time tweaking (while avoiding the consequences of low performance), I assume it’s because they see the value of doing so (whether the pay-off is certain or not). On the surface, an effective person’s behaviour might resemble an ineffective person’s behaviour – six hours sharpening the saw for two hours of work, or six hours procrastinating and two hours of cramming? But if you look at:

  • if they get stuff done
  • whether other people are happy with their performance, or if they generally appear successful in their endeavours
  • how happy they are about the process

then you can get a better idea of whether it’s working for them.

As you think about your own balance or read other people’s blogs, can you identify what scenarios you and other people might resonate with? Am I missing any that I should add to the list? Please comment below!