Planning an Emacs-based personal wiki – Org? Muse? Hmm…

I miss my Planner wiki! I think it’s time to organize things into a personal wiki again. Blogs are great for chronological updates, but I need to be able to group ideas into more than just categories, and WordPress pages aren’t as convenient as a proper wiki. Org-mode outlines are also good, but they can get unwieldy when large. I have an 1.7MB outline right now, all plain text, and I can’t fit it into my head.

What kind of tool should I use? I thought about whether I wanted a web-based wiki editing environment. I realized that editing and publishing the wiki from Emacs is probably the way to go for me, because that gives me offline access, synchronization, and all sorts of other goodies.

Here’s what I want to do:

  1. Provide a knowledge map that links to blog posts and other resources
  2. Flesh out that knowledge map with summaries
  3. Build a coherent personal wiki

Here are other capabilities I care about:

  1. Link easily between concepts
  2. Keep tables and other forms of data
  3. Keep private and public notes, but publish only the public ones
  4. Publish parts of the tree
  5. Publish as separate files, for ease of browsing
  6. Use the same markup I use in Org Mode (or something that can be easily transformed), so that I don’t have to do anything fancy when copying entries over

I thought about using Muse because of its project-publishing support, and because of the good experience I had with Planner and Emacswiki (the predecessor to Muse). Muse supports Org-format tables, but it uses a different way to signify code blocks, examples, and other parts. For ease of implementation, then, I’ll probably see if I can get Org Mode to deal well with the case of either multiple small files, or narrowed portions of one large file. Anyway, the first step is to organize my resources, and that will be useful no matter which wiki system I end up using.

Do you have an Emacs-based personal wiki? What do you use, and what do you think about it?

  • http://www.lawnotes.com D. C. Toedt
  • BreakDS

    I think Muse is just good for me. Talking about publishing these Muse pages … really needs a nice CSS to make it pretty and clear.

  • http://www.newartisans.com John Wiegley

    Hi Sacha! Have you seen howm-mode? It is compatible with Org, since it’s a minor mode, but also lets you do Wiki-style linking and easy page creation:

    http://howm.sourceforge.jp/

  • http://sachachua.com Sacha Chua

    John: I looked at Howm before and I thought it was cool, although it might be a little harder to get it published to the web. org-post-subtree-as-page turns out to be surprisingly useful, though, and it will be even more fun once I make a few tweaks.

    D. C. Awesome! Checking it out. Seeing the XHTML export for Org nudges me to think of ways to integrate it into my Rails personal dashboard. Hmm…

    BreakDS: Glad that works for you!

  • Mohamed

    Hi Sacha,
    I’m trying to use both planner and org. The first for its simplicity and the second for its power. I have added all my muse stuff to org projects. I’m not very satisfied because I didn’t succeed to customize planner day page with org-agenda stuff*.

    Mohamed

    * The idea is to get some org-agenda commands in the Tasks section of planner pages,

  • Damon Haley

    Great post. What about a solution for Drupal 7?

    I believe or2blog has support for creating Drupal nodes.

    But I don’t know if there is support for formatting the source code annotations like ‘#+begin_src’, converting the outline structure to HTML lists or automatic conversion of org-mode tags to Drupal taxonomy.

    Perhaps a Drupal module that would convert a org-mode tree into Drupal ‘Book’ nodes would be a good solution. Then the mind map could later be manipulated through the Book module’s built-in draggable views functionality, if necessary.

    I’d be willing to collaborate creating such a Drupal module if someone else out there was interested.

    Sacha, please share what you find out or end up doing.

    Damon

  • RGalacho

    I’ve been using org-mode for just a few weeks right now.
    Don’t scold me, but – without all planning and tasks stuff -, I see org-mode like
    another Muse. I mean, you can link stuff into org files (local or foreign content), beautify your annotations with markup (a slightly different markup, but a markup after all).

    Supposing that you won’t to get out of that mode (comfort, simplicity, whatever…).
    Maybe it’s just a matter of tweaking org-mode exporting options to exclude private annotations (by using a personalized tag or something like that).
    If you want to publish into different files, you could split that big \org-wiki\ base into smaller \org-wiki\ files and fill accordingly LINK_HOME option.

    Now that I’m reading the manual… ¿¿¿¡¡¡org-mode can do latex export!!!???
    (http://orgmode.org/manual/Export-options.html)

    I had to experiment with that… o_O

    PS: Anyway, If I talk too much gibberish, please, delete and purge this post…

  • http://pubpages.unh.edu/~mlg52/Home.html Monica

    Hi Sacha,
    I have been using an Org-Mode based wiki for my research for over a year now, and LOVE it! I had tried Wiki On A Stick (WOAS), but if one thing got corrupted, it would crash and you could lose everything unless you had frequent backups saved elsewhere. PMWiki was aggravating to set-up and was limited in what it could do (in terms of what I wanted), so I turned to Org-Mode. In my .emacs file, I set up Org-mode project publishing. This allows me to publish either just one new page, or all my separate org-files that are part of the wiki at once. I keep my org files and image files separate, and then publish to a separate HTML directory. The result is an excellent interactive wiki with navigation buttons on top (home, up, down, etc.) and an smooth organization. I even have one page that is an interactive glossary where you can click a letter and see all the terms that start with that letter. I have animated gifs integrated into it as well, that help illustrate concepts I want to help keep clear. The glossary was very easy to set up, I just used some html code snippets which org-mode handles very well. A nice, simple custom CSS file makes my HTML export look nice and clean without be overly plain.
    I definitely recommend it! I tried MUSE, but found org-mode to have way more functionality, though I admit that I only tried MUSE for a day or two before getting frustrated, so if I had devoted more time maybe it would have been fine.
    The best part, of course, is that all the org files are plain text. I back then up in a subversion repository and they hardly take up any space at all. This also allows me to keep my wiki up-to-date across multiple machines, since I do not host it on the web anywhere.
    Feel free to message me with any questions. I can also send you the files if you would like to see some of what I have done with it.
    Good luck with whatever you chose!

    • http://felix11h.github.io/ Felix Hoffmann

      Monica, are you still using your setup? I want to use/build something very similar to your solution and would love to take your configuration as a basis. Thanks!

  • http://sachachua.com Sacha Chua
  • http://www.seas.upenn.edu/~nghiem Truong Nghiem

    @Monica: wow. Could you share your configuration on the net somewhere? Says github. Thanks!

  • Monica

    Sure guys! I actually haven’t blogged anything since May (I know, bad bad!), so I’ll write up a post detailing how I set it up, my config file, and sample output. I’m working on a school deadline for Monday, but I’ll try to get it up tomorrow or Sunday and will post the link here. In the meantime I’ll email you my config file tomorrow morning as I am not at my laptop right now.

  • Jazzofilus

    Mmmm! Interesting Monica, waiting to see it!

    thank u

  • Lluís

    First of all, sorry for the almost-necro-posting, but I fell behind on my feeds.

    Have you tried with ikiwiki+org-mode? I gave it a small try and works pretty well; now I just have to populate it with actual contents :)

    You can get the ikiwiki plugin here [1], but I have a slightly modified version to fix a few things I found on the way [2], plus a lisp file to set org-mode’s export settings [3].

    Let us know your experiences if you try it :)

    [1] http://www.golden-gryphon.com/blog/manoj/blog/2008/06/08/Using_org-mode_with_Ikiwiki/
    [2] http://pastebin.com/vanhkxvD
    [3] http://pastebin.com/XNmqcAMX

  • http://jasonjohnwells.com Jason John Wells

    There is also a current MediaWiki option https://launchpad.net/mediawiki-el

  • http://jasonjohnwells.com Jason John Wells

    I’m a Emacs n00b but it would be cool if mediawiki-el could make this as DONE.

    ;;; TODO:
    ;; * Optionally use org-mode formatting for editing and translate
    ;; that to mw

  • http://jasonjohnwells.com Jason John Wells

    Anyone have any thongs on using Emacs Freex?

    http://www.youtube.com/watch?v=vGqaSzD-FTE

    http://code.google.com/p/emacs-freex/

    It looks strong for adding nodes in an unstructured way Vs. Org and connecting them in a hypermedia way.

    My concern is that it doesn’t seem to have been updated since 2009 and I don’t know enough to project how long it will still be compatible with Emacs.

  • Juan G.

    For a wiki, there is also Org-mode itself. Usually, the files of wiki systems are all in the same directory, so the links between them are quick and easy.

    For example, on Wikipedia, the wikilinks are [[topic example]] or [[topic example|any text]], which link to http://en.wikipedia.org/wiki/Topic_example or similar.

    On Org-mode, it can be [[file:topic example.org][any text]] or [[./topic example.org][any text]]. These file links can be entered directly, or using C-c C-l, or C-u C-c C-l. Maybe some small lisp code could make this even easier.

  • Juan G.

    Also, when the file does not exist yet, following the link creates a buffer for the new file. Just like on Wikipedia.

  • Juan G.

    To summarize, for example, this can be a paragraph in a wiki page, with a simple Org-mode relative link (pretty similar to Wikipedia’s) to another [[file:topic.org][topic]] with its own wiki page in the same directory.

    Of course, relative links to files in other directories like [[../another folder/topic.org][topic]] or [[./subfolder/topic.org][topic]] are also possible. So, with Org-mode we can have both a file tree (useful organization if we use for example Jota Text Editor for our synced Org files on an Android phone), and an interlinked personal wiki at the same time (nice on a laptop or desktop with Emacs).

    An additional detail, with (setq org-return-follows-link t) in the .emacs file, the enter key is one of the ways to open linked pages, in addition to mouse and the C-c C-o keybinding. That is, to easily create a new wiki file, we can just write a link, and follow it with the enter key.

  • http://livinganawesomelife.com Sacha Chua

    Yup, Org is awesome. I’m getting quite spoiled by the ability to search my handwritten notes in Evernote, though, so I’m still tweaking my notetaking setup!

  • Juan G.

    Sacha, that’s a good point, how to integrate Org-mode with software for handwritten notes (tablets, etc.), or maybe even scanned paper notes. It must be possible, at least for tablets, since Org-mode integrates with so many things (see for example the Wikipedia article).

    Although inline images such as sketchnotes can be optionally displayed in Org-mode buffers, for search in .org text files of course including a copy as text would be necessary. As you said, OneNote (Windows) can convert handwritten notes to text (“Ink to Text”), so this would be a possibility. On the other hand, Evernote can search but not yet convert, so a complementary handwriting recognition software such as Ink2Text (Linux) or the Tablet PC Recognizer Pack (Windows) could possibly help (I’ve not tried them) if choosing to integrate with Evernote.

  • Juan G.

    Another detail for Org-mode personal wikis: Link abbreviations can help to have very simple wikilinks like [[wiki:Topic][topic]], that work like absolute links, the same from any directory (for easy moving of files, or copy/paste). The directory path and the extension .org are added automatically. For example:

    This is a normal wikilink to a [[wiki:Topic][topic]] in the main wiki directory.

    And this is an optional wikilink to another [[wiki:Subdirectory/Topic][topic]] in a subdirectory.

    It works adding something like the following to the .emacs file (changing /long/path/to/wiki, of course):

    ;; Link abbreviations
    (setq org-link-abbrev-alist
    ‘((“wiki” . “file:/long/path/to/wiki/%s.org”)
    (“google” . “http://www.google.com/search?q=”)))

    The additional Google example is for searches from Org-mode like [[google:anything]] .

  • Juan G.

    To integrate sketchnotes with Org-mode, for example an Org wiki, instead of the handwriting recognition (Ink2Text, etc.) that I mentioned in a comment above, there is naturally a simpler way: Just adding a line with some keywords to the image can facilitate text search with grep, or regexxer (Linux), grepWin, etc.

    In the .emacs file, we can have (customizing “/long/path/to/…”):

    ;; Link abbreviations
    (setq org-link-abbrev-alist
    ‘((“wiki” . “file:/long/path/to/wiki/%s.org”)
    (“thumb” . “file:/long/path/to/images/folder/%s_thumb.png”)
    (“image” . “file+sys:/long/path/to/images/folder/%s.png”)))

    The images folder on the computer with Emacs can be set to synchronize with another images folder on the tablet, using Dropbox, Unison, SyncToy, etc.

    So the 4-step process would be:

    1. Draw a sketchnote with our favorite tablet app.
    2. Save two copies of the image: full size (to see all the details in our default image viewer), and smaller size (to fit in an Emacs buffer).
    3. The images are synchronized to the computer.
    4. Write someting like the following on Org-mode:

    * Sketchnote title
    [[thumb:Sketchnote-title]]
    [[image:Sketchnote-title][Full size]]
    Keywords: some keywords copied from the sketchnote.

    An that’s it. The thumbnail image can be viewed directly in the Org-mode buffer, and the full size link opens the original sketchnote with our default image viewer.

    As said in the “Org-mode” Wikipedia article (http://en.wikipedia.org/wiki/Org-mode):

    “Images, such as [[/images/image.jpg]], which can be linked and exported. Images can also be shown graphically integrated with the Org text on Emacs, using org-toggle-inline-images (C-c C-x C-v), or org-startup-with-inline-images.”

  • Juan G.

    Naturally, for image search in Org wikis, apart from full-text handwriting recognition, or just some free-form keywords, and for example Emacs grep (M-x grep), etc., another possibility is to use Org :tags: and tag searches.

  • Juan G.

    I’m testing again the inline images, and it seems that probably I did some mistake during my previous testing. Sorry about that. Now it appears that link abbreviations like “image:” only work for links, but for inline images a normal “file:” link seems necessary.

    So, the example with an inline image directly displayed, and a “Full size” link, could be:

    [[file:/images/Sketchnote-title_thumb.png]]
    [[image:Sketchnote-title][Full size]]

    Probably there are some ways to improve this.

    For example, they have now in development (git) the following, that would fit images in Org buffers, with no need for thumbnails:

    (setq org-image-actual-width 300)
    => always resize inline images to 300 pixels

    etc…

  • http://twitter.com/jianlzhang leon

    sacha, that is what exactly I’m looking at. I have used org mode for years but only limited for the outline mode. Did you share your config file in the web somewhere or could you please email it to me at zhangjleon at outlook dot com? Thanks

  • http://www.techcurmudgeon.com Peter Davis

    I know it’s been a while, but I’m now facing this same decision. I’ve been using Oddmuse as a personal Wiki for years, and have been very happy with it. However, I’m now thinking I’d like a) to be able to edit in emacs, and b) to keep my files on dropbox and be able to edit and publish them from anywhere. In fact, even the publishing part is secondary, though it would be handy to get clean html I can cut and paste into my (blogger) blog.

    I have a little experience with each of muse and org mode. I like the features of org mode, but it doesn’t seem to be set up for multiple file collections as well as muse. I’m still learning though. Did anyone ever find out what configuration Monica referred to back a couple of years ago?

    • http://sachachua.com Sacha Chua

      I’ve heard from people who use Org with lots and lots of small files as part of a system for managing a static blog, so that’s probably doable. =)

      • http://www.techcurmudgeon.com Peter Davis

        Thanks! I’ll keep digging.

    • Erik Pischel

      There is an emacs package called “Yaoddmuse” for editing oddmuse pages. http://www.emacswiki.org/emacs/Yaoddmuse

      I have not tried it myself, though.

    • http://felix11h.github.io/ Felix Hoffmann

      Peter, I’m now in the same situation as you were a year ago, trying to make the jump from a MediaWiki setup. In her post below, Monica described everything I would want to have in an org-mode setup. What solution did you come up with in the end?