Thinking about how I can use large language models

| ai, geek

I wanted to follow up on my post on large language models and me from January and do some more thinking/learning out loud.

Data

I have >= 8000 posts in my blog, almost 4000 sketches (mostly public, many with OCR results), and lots of text notes in my Org Mode files, many of which are in a hierarchy and many of which are just a random list of entries.

There's also the episodic memory aspect of things, trying to remember using different cues. Might be fun to figure out what I can do with >= 16,000 text journal entries (usually a sentence, sometimes longer).

Vector/approximate search

I like the idea of being able to search my notes with both exact and approximate matches. Semantic Search for Org roam | lgmoneda might be a good starting point. Khoj has a free self-hosted option that can be configured to use offline models and some support for Org Mode, but it might be too big for me to wrap my head around at this point. I'll probably write my own thing so that I can understand it and so that I can tailor it to the structure (or lack thereof) of what I've got.

I don't mind starting with just the retrieval aspect, since text generation is still a little iffy. I'd rather be able to flip through the titles and jump to the source information for now.

I'm partly looking for the modern equivalent of the Remembrance Agent, which I enjoyed using in Emacs before. It used a bag-of-words approach to look at a few hundred words around your cursor and suggest files/emails/etc. that were relevant in another window.

Episodic search

Slightly more far-off: it might be nice to be able to find something based on more retrieval cues and episodic memory, and to enrich memory.

Someday it would be pretty cool to have something help me remember where I left something. Video (SpotEM, Episodic Memory Question Answering) might be excessive, but maybe I could get into the habit of audio notes?

Transcript correction

It would be good to have more conveniences for fixing commonly-misrecognized words, although that might also be handled with simple regex-based replacements (maybe like mbork's mrr-replace-mode).

Topic identification and segmentation

It would be nice to automatically break up transcripts of braindumps or web conferences into topics. I currently do a bit of light structuring via keywords in my audio braindumps, but topic segmentation is a well-established area, so I should be able to figure this out once I set aside time for it.

Question-answering: not so much yet

I thought it was a little interesting how Dan Shipper used 10 years of journal data for retrieval-augmented text generation and question-answering (source) and how AI can make analyzing your thoughts and actions feel more like a conversation, but I don't think this is quite useful for me yet.

Question-asking might be more useful

When I'm writing, my challenge is usually resisting going down the rabbit hole of more tweaks and instead reining things in so that I can finish the post. But if I do find myself wanting to write more, I think it might potentially be interesting to ask what kinds of follow-up questions people might ask about something, and then add links or more explanations.

View org source for this post
You can comment with Disqus or you can e-mail me at sacha@sachachua.com.