Planning an Emacs-based personal wiki – Org? Muse? Hmm…

Posted: - Modified: | emacs

I miss my Planner wiki! I think it’s time to organize things into a personal wiki again. Blogs are great for chronological updates, but I need to be able to group ideas into more than just categories, and WordPress pages aren’t as convenient as a proper wiki. Org-mode outlines are also good, but they can get unwieldy when large. I have an 1.7MB outline right now, all plain text, and I can’t fit it into my head.

What kind of tool should I use? I thought about whether I wanted a web-based wiki editing environment. I realized that editing and publishing the wiki from Emacs is probably the way to go for me, because that gives me offline access, synchronization, and all sorts of other goodies.

Here’s what I want to do:

  1. Provide a knowledge map that links to blog posts and other resources
  2. Flesh out that knowledge map with summaries
  3. Build a coherent personal wiki

Here are other capabilities I care about:

  1. Link easily between concepts
  2. Keep tables and other forms of data
  3. Keep private and public notes, but publish only the public ones
  4. Publish parts of the tree
  5. Publish as separate files, for ease of browsing
  6. Use the same markup I use in Org Mode (or something that can be easily transformed), so that I don’t have to do anything fancy when copying entries over

I thought about using Muse because of its project-publishing support, and because of the good experience I had with Planner and Emacswiki (the predecessor to Muse). Muse supports Org-format tables, but it uses a different way to signify code blocks, examples, and other parts. For ease of implementation, then, I’ll probably see if I can get Org Mode to deal well with the case of either multiple small files, or narrowed portions of one large file. Anyway, the first step is to organize my resources, and that will be useful no matter which wiki system I end up using.

Do you have an Emacs-based personal wiki? What do you use, and what do you think about it?

You can view 42 comments or e-mail me at sacha@sachachua.com.

42 comments

D. C. Toedt

2011-11-02T18:37:40Z

Could you adapt org-info.js? http://orgmode.org/worg/cod...

I think Muse is just good for me. Talking about publishing these Muse pages ... really needs a nice CSS to make it pretty and clear.

John Wiegley

2011-11-02T21:37:40Z

Hi Sacha! Have you seen howm-mode? It is compatible with Org, since it's a minor mode, but also lets you do Wiki-style linking and easy page creation:

http://howm.sourceforge.jp/

John: I looked at Howm before and I thought it was cool, although it might be a little harder to get it published to the web. org-post-subtree-as-page turns out to be surprisingly useful, though, and it will be even more fun once I make a few tweaks.

D. C. Awesome! Checking it out. Seeing the XHTML export for Org nudges me to think of ways to integrate it into my Rails personal dashboard. Hmm...

BreakDS: Glad that works for you!

Hi Sacha,
I'm trying to use both planner and org. The first for its simplicity and the second for its power. I have added all my muse stuff to org projects. I'm not very satisfied because I didn't succeed to customize planner day page with org-agenda stuff*.

Mohamed

* The idea is to get some org-agenda commands in the Tasks section of planner pages,

Damon Haley

2011-11-03T21:56:48Z

Great post. What about a solution for Drupal 7?

I believe or2blog has support for creating Drupal nodes.

But I don't know if there is support for formatting the source code annotations like '#+begin_src', converting the outline structure to HTML lists or automatic conversion of org-mode tags to Drupal taxonomy.

Perhaps a Drupal module that would convert a org-mode tree into Drupal 'Book' nodes would be a good solution. Then the mind map could later be manipulated through the Book module's built-in draggable views functionality, if necessary.

I'd be willing to collaborate creating such a Drupal module if someone else out there was interested.

Sacha, please share what you find out or end up doing.

Damon

I've been using org-mode for just a few weeks right now.
Don't scold me, but - without all planning and tasks stuff -, I see org-mode like
another Muse. I mean, you can link stuff into org files (local or foreign content), beautify your annotations with markup (a slightly different markup, but a markup after all).

Supposing that you won't to get out of that mode (comfort, simplicity, whatever...).
Maybe it's just a matter of tweaking org-mode exporting options to exclude private annotations (by using a personalized tag or something like that).
If you want to publish into different files, you could split that big \org-wiki\ base into smaller \org-wiki\ files and fill accordingly LINK_HOME option.

Now that I'm reading the manual... ¿¿¿¡¡¡org-mode can do latex export!!!???
(http://orgmode.org/manual/E...

I had to experiment with that... o_O

PS: Anyway, If I talk too much gibberish, please, delete and purge this post...

Hi Sacha,
I have been using an Org-Mode based wiki for my research for over a year now, and LOVE it! I had tried Wiki On A Stick (WOAS), but if one thing got corrupted, it would crash and you could lose everything unless you had frequent backups saved elsewhere. PMWiki was aggravating to set-up and was limited in what it could do (in terms of what I wanted), so I turned to Org-Mode. In my .emacs file, I set up Org-mode project publishing. This allows me to publish either just one new page, or all my separate org-files that are part of the wiki at once. I keep my org files and image files separate, and then publish to a separate HTML directory. The result is an excellent interactive wiki with navigation buttons on top (home, up, down, etc.) and an smooth organization. I even have one page that is an interactive glossary where you can click a letter and see all the terms that start with that letter. I have animated gifs integrated into it as well, that help illustrate concepts I want to help keep clear. The glossary was very easy to set up, I just used some html code snippets which org-mode handles very well. A nice, simple custom CSS file makes my HTML export look nice and clean without be overly plain.
I definitely recommend it! I tried MUSE, but found org-mode to have way more functionality, though I admit that I only tried MUSE for a day or two before getting frustrated, so if I had devoted more time maybe it would have been fine.
The best part, of course, is that all the org files are plain text. I back then up in a subversion repository and they hardly take up any space at all. This also allows me to keep my wiki up-to-date across multiple machines, since I do not host it on the web anywhere.
Feel free to message me with any questions. I can also send you the files if you would like to see some of what I have done with it.
Good luck with whatever you chose!

Felix Hoffmann

2014-12-07T17:36:47Z

Monica, are you still using your setup? I want to use/build something very similar to your solution and would love to take your configuration as a basis. Thanks!

ben kavanagh

2016-09-21T22:59:18Z

Hi Monica, do you have a github for your org-mode wiki? I am moving from a Muse-mode wiki and would love to see your setup as an example. Thanks!

Monica: Ooooooh. That sounds like the kind of setup I'd like to have. Have you shared your config files anywhere on the Net, or could you please e-mail them to me at sacha@sachachua.com? Thank you!

Truong Nghiem

2011-11-05T06:24:43Z

@Monica: wow. Could you share your configuration on the net somewhere? Says github. Thanks!

Sure guys! I actually haven't blogged anything since May (I know, bad bad!), so I'll write up a post detailing how I set it up, my config file, and sample output. I'm working on a school deadline for Monday, but I'll try to get it up tomorrow or Sunday and will post the link here. In the meantime I'll email you my config file tomorrow morning as I am not at my laptop right now.

Mmmm! Interesting Monica, waiting to see it!

thank u

First of all, sorry for the almost-necro-posting, but I fell behind on my feeds.

Have you tried with ikiwiki+org-mode? I gave it a small try and works pretty well; now I just have to populate it with actual contents :)

You can get the ikiwiki plugin here [1], but I have a slightly modified version to fix a few things I found on the way [2], plus a lisp file to set org-mode's export settings [3].

Let us know your experiences if you try it :)

[1] http://www.golden-gryphon.com/blog/manoj/blog/2008/06/08/Using_org-mode_with_Ikiwiki/
[2] http://pastebin.com/vanhkxvD
[3] http://pastebin.com/XNmqcAMX

Jason John Wells

2012-04-28T06:12:32Z

There is also a current MediaWiki option https://launchpad.net/media...

Jason John Wells

2012-04-28T06:18:01Z

I'm a Emacs n00b but it would be cool if mediawiki-el could make this as DONE.

;;; TODO:
;; * Optionally use org-mode formatting for editing and translate
;; that to mw

Jason John Wells

2012-04-28T07:37:33Z

Anyone have any thongs on using Emacs Freex?

http://www.youtube.com/watc...

http://code.google.com/p/em...

It looks strong for adding nodes in an unstructured way Vs. Org and connecting them in a hypermedia way.

My concern is that it doesn't seem to have been updated since 2009 and I don't know enough to project how long it will still be compatible with Emacs.

Hi John & co,

I'm one of the developers of Emacs Freex mode. I'm just right now updating it to make it fully compatible with Emacs 24.

More generally, I still use Freex personally dozens of times a day, so I'm highly motivated to make sure it's still running on Emacs for many years into the future!

Give me a shout if you want to hear more,

Yours,
Greg
greg@gregdetre.co.uk

Ooooh, sounds awesome. Checking out the intro video now... Interesting tagging. How does it work with thousands of nuggets and possibly ambiguous names? =)

I have about 30k nuggets, and searching is more or less instantaneous, though there's a couple hundred millisecond delay sometimes when typing - I set the fontification to be lazy, and so it's no big deal.

Ambiguous names is always going to be an issue - for instance, I noticed a link to 'sets' (used as a verb) that took me to my page on sets-as-data-structure. In that case, I can either rename it to be something more verbose, or ignore the hyperlink. In practice, I don't find that it bugs me. I'm open to better ideas...

That's great to hear. It reminds me of the memex system that Mark Carranza uses, which links things together based on the same text being in the same line. Ambiguity is natural and can lead to serendipity. =) I suppose that because he uses line content instead of words (lines lead to fewer, more intentional matches), he has more control over what gets linked... Maybe that's an option or approach that might work?

For a wiki, there is also Org-mode itself. Usually, the files of wiki systems are all in the same directory, so the links between them are quick and easy.

For example, on Wikipedia, the wikilinks are [[topic example]] or [[topic example|any text]], which link to http://en.wikipedia.org/wik... or similar.

On Org-mode, it can be [[file:topic example.org][any text]] or [[./topic example.org][any text]]. These file links can be entered directly, or using C-c C-l, or C-u C-c C-l. Maybe some small lisp code could make this even easier.

Also, when the file does not exist yet, following the link creates a buffer for the new file. Just like on Wikipedia.

To summarize, for example, this can be a paragraph in a wiki page, with a simple Org-mode relative link (pretty similar to Wikipedia's) to another [[file:topic.org][topic]] with its own wiki page in the same directory.

Of course, relative links to files in other directories like [[../another folder/topic.org][topic]] or [[./subfolder/topic.org][topic]] are also possible. So, with Org-mode we can have both a file tree (useful organization if we use for example Jota Text Editor for our synced Org files on an Android phone), and an interlinked personal wiki at the same time (nice on a laptop or desktop with Emacs).

An additional detail, with (setq org-return-follows-link t) in the .emacs file, the enter key is one of the ways to open linked pages, in addition to mouse and the C-c C-o keybinding. That is, to easily create a new wiki file, we can just write a link, and follow it with the enter key.

Yup, Org is awesome. I'm getting quite spoiled by the ability to search my handwritten notes in Evernote, though, so I'm still tweaking my notetaking setup!

Sacha, that's a good point, how to integrate Org-mode with software for handwritten notes (tablets, etc.), or maybe even scanned paper notes. It must be possible, at least for tablets, since Org-mode integrates with so many things (see for example the Wikipedia article).

Although inline images such as sketchnotes can be optionally displayed in Org-mode buffers, for search in .org text files of course including a copy as text would be necessary. As you said, OneNote (Windows) can convert handwritten notes to text ("Ink to Text"), so this would be a possibility. On the other hand, Evernote can search but not yet convert, so a complementary handwriting recognition software such as Ink2Text (Linux) or the Tablet PC Recognizer Pack (Windows) could possibly help (I've not tried them) if choosing to integrate with Evernote.

Another detail for Org-mode personal wikis: Link abbreviations can help to have very simple wikilinks like [[wiki:Topic][topic]], that work like absolute links, the same from any directory (for easy moving of files, or copy/paste). The directory path and the extension .org are added automatically. For example:

This is a normal wikilink to a [[wiki:Topic][topic]] in the main wiki directory.

And this is an optional wikilink to another [[wiki:Subdirectory/Topic][topic]] in a subdirectory.

It works adding something like the following to the .emacs file (changing /long/path/to/wiki, of course):

;; Link abbreviations
(setq org-link-abbrev-alist
'(("wiki" . "file:/long/path/to/wiki/%s.org")
("google" . "http://www.google.com/searc...")))

The additional Google example is for searches from Org-mode like [[google:anything]] .

To integrate sketchnotes with Org-mode, for example an Org wiki, instead of the handwriting recognition (Ink2Text, etc.) that I mentioned in a comment above, there is naturally a simpler way: Just adding a line with some keywords to the image can facilitate text search with grep, or regexxer (Linux), grepWin, etc.

In the .emacs file, we can have (customizing "/long/path/to/..."):

;; Link abbreviations
(setq org-link-abbrev-alist
‘((“wiki” . “file:/long/path/to/wiki/%s.org”)
(“thumb” . “file:/long/path/to/images/folder/%s_thumb.png”)
(“image” . “file+sys:/long/path/to/images/folder/%s.png”)))

The images folder on the computer with Emacs can be set to synchronize with another images folder on the tablet, using Dropbox, Unison, SyncToy, etc.

So the 4-step process would be:

1. Draw a sketchnote with our favorite tablet app.
2. Save two copies of the image: full size (to see all the details in our default image viewer), and smaller size (to fit in an Emacs buffer).
3. The images are synchronized to the computer.
4. Write someting like the following on Org-mode:

* Sketchnote title
[[thumb:Sketchnote-title]]
[[image:Sketchnote-title][Full size]]
Keywords: some keywords copied from the sketchnote.

An that's it. The thumbnail image can be viewed directly in the Org-mode buffer, and the full size link opens the original sketchnote with our default image viewer.

As said in the "Org-mode" Wikipedia article (http://en.wikipedia.org/wik...

"Images, such as [[/images/image.jpg]], which can be linked and exported. Images can also be shown graphically integrated with the Org text on Emacs, using org-toggle-inline-images (C-c C-x C-v), or org-startup-with-inline-images."

Naturally, for image search in Org wikis, apart from full-text handwriting recognition, or just some free-form keywords, and for example Emacs grep (M-x grep), etc., another possibility is to use Org :tags: and tag searches.

I'm testing again the inline images, and it seems that probably I did some mistake during my previous testing. Sorry about that. Now it appears that link abbreviations like "image:" only work for links, but for inline images a normal "file:" link seems necessary.

So, the example with an inline image directly displayed, and a "Full size" link, could be:

[[file:/images/Sketchnote-title_thumb.png]]
[[image:Sketchnote-title][Full size]]

Probably there are some ways to improve this.

For example, they have now in development (git) the following, that would fit images in Org buffers, with no need for thumbnails:

(setq org-image-actual-width 300)
=> always resize inline images to 300 pixels

etc...

sacha, that is what exactly I'm looking at. I have used org mode for years but only limited for the outline mode. Did you share your config file in the web somewhere or could you please email it to me at zhangjleon at outlook dot com? Thanks

Sure! My Emacs configuration is at http://dl.dropbox.com/u/396... , and you can see how I've used Org to keep an index of my blog posts at http://dl.dropbox.com/u/396...

That's terrific! your file is like a text book of emacs config. I will benefit from it a lot.

Peter Davis

2013-10-24T18:08:03Z

I know it's been a while, but I'm now facing this same decision. I've been using Oddmuse as a personal Wiki for years, and have been very happy with it. However, I'm now thinking I'd like a) to be able to edit in emacs, and b) to keep my files on dropbox and be able to edit and publish them from anywhere. In fact, even the publishing part is secondary, though it would be handy to get clean html I can cut and paste into my (blogger) blog.

I have a little experience with each of muse and org mode. I like the features of org mode, but it doesn't seem to be set up for multiple file collections as well as muse. I'm still learning though. Did anyone ever find out what configuration Monica referred to back a couple of years ago?

I've heard from people who use Org with lots and lots of small files as part of a system for managing a static blog, so that's probably doable. =)

Peter Davis

2013-10-24T19:57:02Z

Thanks! I'll keep digging.

Erik Pischel

2013-10-30T13:26:32Z

There is an emacs package called "Yaoddmuse" for editing oddmuse pages. http://www.emacswiki.org/em...

I have not tried it myself, though.

Felix Hoffmann

2014-12-07T17:16:27Z

Peter, I'm now in the same situation as you were a year ago, trying to make the jump from a MediaWiki setup. In her post below, Monica described everything I would want to have in an org-mode setup. What solution did you come up with in the end?

dramatictheory

2016-10-09T16:37:18Z

Sacha,

I've been reading back through your info-management posts and this is the most recent dealing with using org-mode for a nebulous collection of small files- where did you end up with these questions?

I use several large files, and I use org-refile to jump around. Still haven't gotten around to trying the deft+small files approach. Reddit.com/r/orgmode might be a good place to discuss or search for other workflows, and there's the mailing list too. Probably mostly a personal preference thing!

dramatictheory

2016-10-10T17:54:10Z

Thank you!