2015-12-10 Emacs Chat: John Wiegley on maintaining Emacs and how you can help
Posted: - Modified: | emacs, Emacs ChatJohn Wiegley shared how he uses Gnus and Org to help him with the volume of Emacs-related information, and how people can get started with Emacs development.
- 0:02 Gnus for mail and news
- 0:04 Organizing groups by topic
- 0:05 Adaptive scoring and prioritization
- 0:09 Setup for mail: Gmail, Fetchmail, Dovecot, Gnus
- 0:11 Time: 1-2 hours a day
- 0:13 Community-building
- 0:15 Using Org to keep track of initiatives
- 0:19 Reading bug reports in Gnus
- 0:22 How people can help: tests, documentation, reviewing bugs
- 0:24 Coverage
- 0:33 Efficiency, benchmarks
- 0:40 Magit, Projectile, Flycheck
- 0:45 Following up on emacs-devel topics: IDEs, APIs, lexical binding, Guile, etc.
You can e-mail John Wiegley at johnw@gnu.org. The emacs-devel mailing list is at https://lists.gnu.org/mailman/listinfo/emacs-devel.
Event page on Google+
Ogg Vorbis (audio only)
MP3 (audio only)
View the full blog post for the transcript. Thanks to Phil Hudson for volunteering to transcribe this!
s: Hello, everyone! Hi, I’m Sacha Chua, and this is another Emacs Chat. It is December 10, 2015. I’m here with John Wiegley who has, since the last time we talked to him, become the Emacs maintainer, among other things. So we’re here today to chat about how he’s dealing with the increased workload and other cool things! John, over to you.
j: Thank you, Sacha. I’m glad to have this opportunity for us to talk again. It’s been a few months.
s: By the way, if you have questions, you can submit questions by commenting on the Event page where you found this, ????? etc etc, or using the Q&A app that shows up, or you can pop into IRC on the Freenode network and look in the #emacs channel and pose questions there. I’ll try to remember to sneak questions in at the appropriate moment. But I want to kick things off with some questions about your new workflow now that you’re dealing with the increased workload of a high-volume mailing list/newsgroup, called emacs-devel, and other things you deal with as Emacs maintainer. So, what do you do?
j: Well, certainly the emacs-devel mailing list is quite high-volume. My technique for approaching that has been to customize gnus until it makes it a more manageable source of information. I’ve had to tweak the scoring setup that I use. Gnus has a feature called “adaptive scoring” where it will observe what you read and what you don’t read and it will try to change the ordering and the presentation of the things you see in the *Summary* buffer when you go in to the newsgroup, based on your reading preferences, by observing your reading habits. That has been extremely helpful. That way, if a conversation has gotten very, very long and I realize that there’s nothing more for me to contribute on there, I can hit a key and never see that conversation ever again, unless somebody uses some sort of keyword that I have a flag for that will pop that thread back up for me. Meanwhile the threads that I’m most interested in, or the ones I’ve been replying to, they will go to the top of the buffers, so they’ll be the ones that I read first upon re-entering the group.
s: So you have something that automatically prioritizes the threads for you, based on whether you’ve replied to something or whether you’ve decided this is going way off tangent until it goes back on-topic. Not that I should be asking you which keywords should bring things back to your attention…
j: Right now, it’s “John?” (with a question mark): that’s the one that will pop any thread back up.
s: Alright, you heard it here, folks, if you need to get his attention, just say “John?” and John Wiegley will appear–himself–to deal with whatever’s going on. Actually, since you can share your screen through this videocast, is that something that you want to quickly demonstrate? Like, how this works. Not sure if that works live, because you may have to read things [out].
j: As I click on the screen-sharing button, it’s not having the [?????]
s: Hah! Technology. OK, well, you keep trying that. I’ll try to fill in this space…
j: Why would it not be there…
s: Hmm. OK, we’ll demonstrate with some hand-waving… OK, actually, one thing you can try–this may not break the system: if you refresh your browser, you might get dropped back into this conversation and that thing might work again. Let’s give that a shot. … OK, so while he’s coming back, and I hope this works, because otherwise: awkward! Gnus is a mail- and news-reader for Emacs…
j: Yes, that did work!
s: Perfect! All right. I was just telling people what Gnus is. If you would like to go ahead and demonstrate how this helps you with the prioritization… fantastic!
j: Are you seeing my Emacs screen?
s: I see your screen.
j: OK, so, with the key that I have bound, I will jump into Gnus. One thing you see here is that Gnus has the ability to create topics and to place groups under those topics. Though I actually have quite a number of topics, as you can see.
s: Wow. I did not expect that.
j: Yes. Normally a topic that has no unread groups in it is displayed, and there’s a key-binding that toggles these things. So we see that emacs-devel has gotten two emails since this livecast began. So if I hit return on those… I have the display customized right now so the number on the farthest left is the score that has been assigned to that article, just because I have been trying to fine-tune my adaptive scoring rules, so I want to be aware what effect my actions are having on the scores. I have another key to reveal all the threads. If I go back out and I grab the last 100 messages for example, then I can see that there are some very lengthy threads. I have a number here in this column [highlights fifth column – how do you do that?] that shows me how many messages are hidden underneath the thread. Normally I start and I just see this view, where the thread with the aggregate highest score goes to the top.
s: Wow. So this is a thread with a 90 score and 42 messages under it?
j: Well, 90 was the score of the top message in the thread, or it might have been the aggregate score. I can hit a key to see what the score of the thread itself was. So 90 was the aggregate score, and the score of the article itself was 30. It would have gotten the score of 30 because it would have noticed me reading a message from this thread in the past. Every time I read a message in the thread, the subject of that thread gets a bump of 1. I must have read 30 messages within this thread already, to have a score of 30.
s: So folks, write good subject-line headers so John takes a look at your message.
j: Subject lines are important. Gnus is very handy for example when we have a really long thread like this and someone wants to bring up a side point. We instituted a policy of changing the subject line when you do that. When you change the subject line, Gnus has a way of making that appear as if it’s a new thread. You see this “Casting as wide a net as possible” thread? This actually was a part of the “First draft of the Emacs website” thread, but because the poster, John [Yates] here, chose a different subject line, Gnus presents that to me as if it were an independent thread.
s: And that also allows you to track your interest through adaptive scoring, for that particular subtopic.
j: Right, right. It all plugs into adaptive scoring quite nicely. As a result, even though there are days when the number of unread messages in the emacs-devel group will easily reach 200 by the end of the day, it’s not that difficult to get through them because Gnus presents them to me in roughly the order I ought to read them. If I start seeing that a thread does not really need my involvement, I can start scanning quickly, or just tell gnus I don’t want to participate in this thread from this point onward – unless, of course, someone asks for me.
s: You’re getting quite a few people more interested in Gnus, according to the #emacs channel.
j: Gnus is incredibly wonderful. Let me tell you, Gnus and Org both, they’re my mainstays in Emacs. They’re the two applications that I always have running. Gnus is somewhat like Emacs itself. There is a learning investment to be made, but I do believe it pays itself off, many times over. Further, Lars has done a brilliant job with the gnus manual. It is not only highly informative, it is very funny. It’s worth reading from front to end, more so than any other technical manual I have read. I get quite a kick out of many of the paragraphs in the Gnus manual as I go to read about different things.
s: Was it you who also pointed out that the Calc manual was also very interesting reading? So it turns out that the Info manuals for Emacs packages tend to be quite interesting reading and very useful reading too.
j: Oh yeah. I would recommend to anyone who wants to get started with Gnus to first acquire a brave heart. It’s not going to be an easy task, but… Go through the setup instructions. Get it connected to your mail source by either reading the mail from a directory or using it over IMAP. You will find that using Gnus over IMAP is very slow. This puts a lot of people off. What I do is, I have all of my emails get to delivered to GMail, because they have excellent spam-filtering services. I use fetchmail to pull the mail from GMail down to my local machine, where it gets queued into a Dovecot server that I run locally. Then Gnus talks to the Dovecot server over IMAP, and that is a very fast connection. And it lets me do local searches of emails, so that even if I don’t have Internet connectivity, I can always search all of the email I have. That’s quite a lot of email. In my dovecot, there’s probably close to a million email messages in there.
s: A million! One of the things that I want to mention is that a large part of that million is the entire emacs-devel archive, right?
j: Well, it’s the emacs-devel archive, the emacs-bugs archive, the emacs-help archive, the sources archive… Usually, if I actively participate in a mailing list, I like to go download all of its past email, so that if people make reference to previous discussions, I can see those discussions without having necessarily to be online.
s: And that takes advantage of Gnus’ ability to show you all of the messages within a given thread. It knows about them.
j: Right. Let’s say I’m on this message here, which is a response to this thread called “New update of the Emacs website”, and I want to see the context. I can hit a key, and now all of the members of this part of that thread come into my buffer. So you’ll see the lines of blue here, the ones that are now inserted… I’ll do that again, so you can see what happened. Let’s go in here [emacs-devel summary buffer] So we see that there are three lines in the summary buffer. I hit A T, and now there are 25 lines in that buffer. So I get to see the message I’m reading now in the full context of the thread in which it appeared.
s: Right. So Eric has a question, which he submitted through the Q&A thing. This seems like a lot of reading. How many hours a week do you spend on Emacs maintainer stuff?
j: I would say that in reading the emacs-devel mailing list, it’s between one to two hours a day, probably (and responding to them).
I do not spend as much time fixing bugs or writing code as other contributors to this project do. So for example, Eli Zaretskii spends a very large amount of time addressing bugs and taking care of the stability of Emacs. I rely upon him a lot for his technical expertise.
I spend more time reading the messages, trying to see which people issues there are, what things need to be changed in our organization, in our structure, so that all of these resources that we have in the community can be effective and can smoothly head us toward the next release. We have a lot of excellent people, excellent developers and volunteers and contributors. It doesn’t necessarily need me to do all this work. We have excellent people to do that. I just want to make sure they have what they need to get their jobs done.
s: That’s an interesting approach to take in terms of maintaining an open-source project. It’s less about making all the technical contributions yourself and more about unlocking the awesomeness in the community, coordinating people’s efforts, reaching out to people and pulling them in.
j: Right. And specifically I want to emphasize or focus on that at this point in time, because I feel like we do have a much better community than we’re taking advantage of for core Emacs development. There are the Emacs developers, who at the moment are fewer in number and maybe even a little bit overworked. Then outside of that core community, we have a much larger community of people doing excellent work on external projects that are not part of core Emacs or are not in ELPA. They might be, for example, in the MELPA repository. If I can bring some of those people and attract some of those outside people into the core development, then our job gets a lot easier. That’s going to be much more effective than just me becoming another programmer within the Emacs project. If I can get five other of those people to come in, that’s going to be five times what I could have contributed, for example.
s: That’s awesome. So I guess some of these people might be hiding out in emacs-devel, but where are you keeping an eye out for these people you can recruit basically and bring into core?
j: In IRC, in Twitter, there are people that I know personally who are for whatever reason on the outskirts of emacs-devel. Apparently–they’ve told me–there have been past issues with things not staying on-topic, or things getting a little out of hand, or patches dying as they get submitted and then ignored. I want to make sure that their concerns are met and we resolve those problems, which are really more human-scale problems than technical problems. If we can do that, then these people I think will come back. That will just make it easier for all of us to do our jobs.
s: I can see how like the mailing-list interface with threads and things like that… if you think about it, there’s the mailing list, and then there’s the bug tracker, where you’ve got like “This is broken”, “Maybe there’s a patch needed for this”, but there isn’t really that trackable middle ground that says “Here’s an idea that I’ve been thinking about,” and maybe that’s why some of these ideas are getting dropped. Can you share a little bit about how you’re starting to keep track of the different balls that are in the air, the different things that you have on the go, to make sure that those things aren’t falling through the cracks?
j: That’s an excellent question. I guess you might say that one of my tasks as maintainer is to keep an eye on these more inchoate or intangible things that people request or ask for that aren’t necessarily bugs, so they don’t necessarily belong in the bug database. They are ideas that get discussed on the mailing list, but as you say, when the thread gets cold, the idea might get lost.
So what I do is: when these come up, I have an Org-mode file that I try to track all of the ideas or the things people have suggested or expressed concerns about to make sure that something gets done. Either they get addressed or they get knowledgeably closed–to say “No, leave it. We considered it but we’re not going to do it.”
You’ll see that in my Org file here I have many top-level categories. There are things here, issues pertaining to the community; initiatives – things that I might want to begin but are not being done now; analysis is a little sub-project within Emacs to become more aware about potential bugs or performance problems that we might have, things that aren’t in the bug tracker.
And under each of these, I have several keywords that are labeled as “Project”. I have a project here for making improvements within our community. One of the things I did, for example, was ask you to be our ambassador and keep us all aware of things outside of core development. Making sure responsibility for more aspects of Emacs development is distributed more evenly among the contributors so we don’t get burn-out. ELPA and changing the way we use ELPA… I’d like us to rely more on ELPA, have it be an easier and more facile way for people to get code into Emacs. And of course also the release schedule.
Under these projects, then, I have various TODOs. Often–especially if they have a name here in parentheses–it means it links directly to a mailing-list article. If I hit a key to visit the link, it will pop me over to the emacs-devel mailing list and then show me that message in the context of its discussion. Or I could also then again press A T and then see it. (Although that didn’t work because I only have A T configured to work in the current group. Anyway.) But that gets me back to the original discussion about why I created that task.
s: So I can see how this gives you an overview of the different initiatives that are going on, and lets you review this to follow up with the things that might be falling between the cracks.
j: Right. The first line of activity I do is to follow the mailing list and anything that needs to be responded to there. I also like to read through the diffs log and see what changes have been made to Emacs, to see if there’s anything that needs more attention. I subscribe to the bug tracker so I see every bug that comes in, and I try to make sure that at least some response is made to every bug although I don’t always have the time for that. Then the last thing, of course, is these other things, these other non-bug-type tasks and tracking those. I’ve not been as good lately at keeping on top of this list as I have at keeping on top of the mailing list, mainly because these holidays that we’ve had with Thanksgiving in the United States and some work trips have put me a bit behind. But I do want to get back to this Org file and close these things out because people deserve to be responded to. Everything in this file represents something I felt was a legitimate concern. When people feel that they are being responded to, I think then they will be more encouraged to take part in the core Emacs development process.
s: Is this a file that you’re publicly sharing, or can you be convinced to share this to give people the same overview that you have?
j: No, but of course I would like to maybe do something semi-regularly to capture the state of Emacs development. Have not yet, but that’s something I’ve considered doing.
s: That’s cool. So you’ve got this Org file that you create as you read through emacs-devel messages, bugs and things like that, and you can jump from this Org file back to the original messages. You mentioned on IRC before you also use Gnus to read the bug reports. I think you mentioned that Gnus can show the bug reports for a specific bug or the conversation for a specific bug easily. Is that something you can quickly demonstrate?
j: Sure. So here we have for example bug #19547. I want to see the context of that bug. So there is a command called gnus-read-ephemeral-emacs-bug-group. So now I can put in here [entering text in the minibuffer] 19547. That will contact the debbugs server for the Emacs project and show me the whole conversation history of that bug as an ephemeral group, which means a group that gnus has created on the fly. That will disappear once I have left that group. This way, I can then read through the exact conversation history that has occurred in that bug.
s: I see. So it lets you see the entire context of it, and then you can add links to more information in your Org file as well.
j: Right. So if I’m in here and I hit a key, what it will do is create a capture buffer that links back to that email instantly. And then I just have to hit C-c C-c and then this will be added. Or I hit C-c C-w and then I am prompted for where in the Emacs [Org] file I want to put this issue.
s: org-refile is so powerful that way. So Gnus and Org help you deal with the volume that’s coming in and keep track of what’s going on. Are there things that you think either future maintainers of Emacs or maintainers of packages that are within Emacs, any tips that they would find useful?
j: You mean in the use of Gnus and Org?
s: Or other little scripts or tools that you’ve found helpful.
j: Nothing in particular. I think every maintainer has his own way of doing things and his own areas of interest, so everybody should pick whatever tool works for them.
s: Let’s see, just a quick check for people’s questions… You’ve let people know about Emacs-tangents, which is a fairly new mailing list. Some people didn’t know about that yet, so that’s cool.
One of the things we wanted to talk about was your workflow for dealing with Emacs, and also some of the ideas you had for improving the community and taking it forward. You’ve shared some of it in terms of making emacs-devel friendlier, bringing in people who are not core maintainers… Are there particular areas you want to encourage people to try contributing to or any resources you think might help them get started?
j: Sure. I think there are three areas in particular where I would really appreciate help.
The first would be: writing more tests. We’ve started creating headless tests that run as part of the Emacs build when you do make check using the ERT framework, which is an Elisp framework for writing tests. I know that you use it for lots of things, so you understand its value. We do not have coverage of many parts of Emacs, so people coming in who want to write new tests to address areas we’re not currently testing would be incredibly valuable.
Improving our documentation, especially with an eye to how it reads to someone who’s not familiar with a certain subject area would be very helpful.
And then going through our bug history and finding bugs that have just never been responded to, or that are no longer an issue, checking which ones are still reproducible and which aren’t, and if they are, making comments in the bug to say what you discovered in trying to reproduce the bug.
s: And maybe adding a test for that too.
j: Well that would be ideal! The ideal would be to find a bug, add a test, add something to the manual that’s appropriate for that bug. So these three areas: bugs, tests, and documentation, are the ones I would like to see addressed the most.
Emacs core is really a foundation upon which a lot of things are built that are not part of Emacs core. The thing I want most for Emacs core is not necessarily to add in more features and functionality, I would like it to be a more stable foundation, so the more complex applications like Org and Gnus that are built upon Emacs, there’s less of a moving target. It’s very efficient, it’s very well-performing, there are no crash bugs, those types of things. That’s why I’m placing an emphasis for volunteers to contribute on bugs, tests and documentation, because it will help give us that stable foundation to build other applications on.
s: I don’t suppose there’s any pretty coverage report for Emacs that people can just look at and say, OK, this file was not covered yet, so people can easily see what needs testing.
j: Phillip Lord recently rearranged our tests so that they mirror the files that they’re testing against in the various source directories. You can see from that which files in the source directory have no corresponding tests at all. If you pick a file that relates to a feature you care about, see that no test file for it exists, then you can be the first person to make that file and create the first test. Every test is valuable. I’m not asking people to commit only if they’re willing to write hundreds of tests. If you want to come into the Emacs project, get familiar with development, building the current version of the sources on your own machine, and just write one or two tests, I’d be happy with anything.
s: Maybe one way to make this process a little less intimidating would be, can you walk us through what a super-simple Emacs core test looks like, how to run it and see how that all works?
j: [????] just getting the community stuff up and running. This is not something I’ve been doing myself.
s: We’ll find somebody else to corner into doing a quick demo.
j: This is my checkout of the Emacs source tree.
s: Do you want to share your screen?
j: Oh, sorry, I forgot. OK, do you see this? So I have a clone here of the Emacs development tree, which I tend to build from day to day, just to make sure everything is going sanely. We have a ‘test’ directory in this [????], and there you’ll see we have an ‘automated’ directory, which is where Phil Lord put all of our automated testing. Inside it here, we have all kinds of automated tests. For example, here are some eshell tests. They require in ert and eshell. And then ert-deftest is the command to execute [well, define] a test. So I say that the result of evaluating this [highlighted] form should be equal to 3. And that way, when I do ‘make check’, it will run this code and ensure that [????] is 3.
s: So as we can see, the tests are really easy. They can be pretty short. They can be really simple. You don’t have to get in deep with lots of C or whatever else to write them. You can just use this to get started. And it’s all in Emacs Lisp!
j: I haven’t pulled in a little while. Let’s see…
s: Probably not a good idea to mess up your git checkout…
j: Oh, I can’t pull because my smartcard authenticates my SSH logins and I don’t have that card plugged into this machine.
s: That’s cool. So there’s a lot of code in Emacs. Much of it is not yet tested. It’s great for people to contribute tests. But Eric has another question: “Are you considering moving stuff out of core to have that smaller, simpler, better-tested core, and more things in packages?”
j: Yes. Yes, we are. So right now we have a few areas. We have core Emacs. What we would like to have is a concept within ELPA of “core ELPA”. Core ELPA are packages within ELPA that core Emacs code is able to rely upon. That will happen by a process where core ELPA packages will be copied into the Emacs source tree from time to time, so that core Emacs can directly depend upon them.
Then on top of core ELPA, there will be “tarball ELPA”. These will be packages that are within ELPA that are not copied to core Emacs, but which are made part of the release tarball before it goes out.
Then finally we will have regular ELPA, which will be packages that are installable through the M-x list-packages interface but that will not be in the tarball distribution.
Right now, core Emacs has a lot of very large things in it and a lot of packages in it that don’t really necessarily belong there, because they’re not part of the foundational API that I really think core Emacs represents. We want to take a lot of these packages and split them out into one of these ELPA categories. Very likely, everything that is currently in core Emacs will be either in core ELPA or tarball ELPA. We won’t be moving it out of the distribution entirely, but doing this will make the core be smaller. When you look at git, and see what’s been changing lately, that surface area will be smaller, but it also gives contributors from the outside ways to get things into the tarball distribution without it having to go into core Emacs.
s: I can see how that would also simplify the update process for those packages.
j: And it gives people a way to receive, to subscribe to a channel where that package will be updated frequently, even though Emacs releases might be [????]
s: Right. Cool. Simpler core, better tested, stabler core, and then this kind of split between important things that are in ELPA that core relies on, and things that are optional–used to be there, so you don’t want to break anyone’s expectations, and so they’ll be within the tarball–and of course your regular ELPA and MELPA and the other package archives as well, this universe of packages to choose from.
j: Right. I think ultimately the destination we’d like to end up at is that there’s a core set of functionality without which Emacs cannot even be Emacs, can’t even edit files, and that defines a sort of fundamental minimum. Then anything that that fundamental minimum depends on is by definition a package that has to be in the core. That network of dependencies among this minimum will be what has to be in core, and then as much as possible that does not fit that description will go into tarball ELPA.
For example, tetris. tetris is a neat module that is great to have on any machine just to demo how cool Emacs is, but the core foundational API that Emacs core represents doesn’t really need tetris to be there. So that can be moved to tarball ELPA very easily.
s: Well, I’ll come up with an objection about how essential tetris is to Emacs’ programming, but that is a very good example, yes.
j: I think it’s a good thing to always have be present in an Emacs installation. It’s just… we want the core developers to focus on the set that really matters in terms of defining the foundational API. Also, what’s in core should always be documented, should always be tested, as part of the automated tests. Not every package that we have in core Emacs today really needs that level of rigor. Tetris does not need automated tests, does not need a lot of documentation. So it’s a perfect candidate for being in tarball ELPA.
s: There is some outrage in the #emacs channel over your dissing of tetris.
j: I love the tetris module! Didn’t people hear me? I just said it always should be present in any machine that has Emacs on it.
s: I’m not sure if auto-tetris counts as automated testing for tetris, but there is a thing that automates it.
j: Cool.
s: So people can contribute by writing tests. Tests will help all that stuff happen better. Maybe at some point we can have this nice graphic that shows which files are getting how much percentage coverage. All these lovely motivating graphs, things like that. Then, of course, as you mentioned, there’s documentation to write and bugs to see if you can reproduce.
j: Then there’s one more category that I’m personally quite interested in, which is someone to focus on and sort of take ownership of efficiency and speed of Emacs. That would require writing another… not test suite, but benchmarking suite, that if I run it will give me a table of numbers to say how fast certain Lisp operations are, how fast buffer manipulation operations are, how fast starting up a graphical Emacs and creating a whole ton of frames and closing them all, how fast that is, and then to have a way of, against two builds of the benchmarking suite, [????] the delta, and then designate a build machine that somebody will volunteer time on, where they are willing to run the benchmarking suite against the current development branch every single day, and maintain the running delta as some sort of graph that shows us are we getting slower or are we getting faster in certain areas, and where are the areas that are currently the worst. Today, a thread came up on emacs-devel saying that playing back keyboard macros that involve very, very long lines–like lines that are 11000 characters long–is a very slow thing. And you think, well, that’s not entirely terrible, because how often do people do this? But it would be nice to know where these current limitations of Emacs are, and which are the ones we want to address.
s: Right. And if someone has just committed something that makes something drastically slower, or drastically faster…
j: Yes. I’ve noticed that 25.1, the release candidate, even when I build it with full optimization on, takes exactly twice as long to start up my Emacs as 24.5 does. So it’s a difference of 4.3 seconds versus 8.6 seconds. And I would like to know why is that, and when did that happen? I don’t have right now a benchmarking suite that I could just use git bisect to just take me back to the day when it got slower, so I’ll have to do a specific profiling analysis to find out what made it slower. But if we had a benchmarking suite in place, I think we’d have a lot more consciousness of the ways Emacs is evolving in terms of performance and efficiency.
s: I’m hearing that as part of a general improvement of the build tools that Emacs is using is the idea of having these continuous integration and efficiency metrics and things like that happen, so that people don’t have to keep building these little tools themselves to mess with that.
j: There are plenty of great tools out there for making this happen. I think what we need right now is we need a volunteer willing to devote themselves to seeing this happen and keeping it going.
s: It is a fair bit of heavy lifting to get started, but once it’s in place and keeps running, we hope…
j: Right. Anybody out there who loves both Emacs and performance, get in touch with me and let’s start that going.
s: Yeah. In terms of talking about Emacs startup times, I remember in our conversation about use-package, that was one of your key motivations for being able to find your own autoloads and load only the packages that you were actually using. A faster Emacs… Schools of thought are divided on this: some people do start Emacs all the time, and some people just leave it running. But everyone benefits from having things work faster.
j: Right. When I’m not working on Emacs modules, I tend to just start Emacs once in the day. But if I am working on a module, I want to make sure that I’m always testing my changes against a clean environment, I will exit and restart many, many times during the day. That’s what led to my desire for faster startup.
s: Then in terms of other build tools or development-related initiatives, there’s definitely the good news it’s already happening. I think you mentioned–you vaguely mentioned looking into Github and pull requests, something along the lines of making it easier for people to contribute?
j: This would not be something that is officially endorsed by the FSF. I have a mirror of the Emacs development tree on Github. If people want to make pull requests against it, I will copy those over into the proper channels by hand. But that is not an official portal for either receiving bug requests or pull requests.
s: OK. It’s good to play around with different ways, different channels of bringing things in, so thank you for doing that work by hand, if anyone wants to take advantage of this completely unofficial, not at all sanctioned way of submitting changes.
In terms of people getting started with Emacs development and Emacs contribution, documentation is probably the easiest for people to get started with, testing with a little bit of Emacs Lisp, digging in to bugs and making that better, and of course if you’re interested in performance and efficiency, actually building the infrastructure to make all that testing easier, some of the ways people can contribute.
What are some of the other things you would like to encourage people to look at and contribute to in terms of the community or reaching out? It’s interesting because there are a lot of people coming into Emacs but there’s also a continued perception that this is hard, this is difficult to learn, and there’s 20 weeks of configuring before you can get somewhere. Do you have any particular thoughts on making it easier at that end as well?
j: Not necessarily. I think some people out there have been doing good work towards making some starter kits and tutorials and your own videocasts, Sacha, have been helpful to a lot of people, “Emacs Rocks”… There’s different venues for creating information to attract people or interest people in Emacs. The Spacemacs project has been making Emacs a more comfortable environment for Vim users, for example. So there are things going on in the larger community. That’s something I want to see happen, but it’s not something I can have as an area of focus, since it’s really getting the core foundation stable that is the responsibility, I think. But part of me wanting you to act as our community ambassador was to help keep an eye on those needs in the greater outside community, and if there’s anything core developers can do to help support that.
s: I should remember whenever I come across interesting stories, to fold them into the EmacsWiki, which seems to be on its way back up again. People are adding to it and referring to it, which is good.
j: It’s a fantastic resource. I really recommend it to anybody. I continue to discover nuggets within it from time to time.
s: Just randomly browsing?
j: I often randomly browse, yes. After I’ve worked for an hour, I’ll sometimes just wander and stumble through the page links.
s: Cool. So, let’s see what other things people want to pick your brain on in terms of Emacs and development. Last time we talked, we were sharing Emacs development tips. You shared quite a few with redshank and elrefactor and all of that. Are there other tools you’ve found to be particularly helpful now that you’re reading a lot more code, I guess?
j: Well, I rely very very heavily on Magit. Anybody that uses git should definitely look into that. It’s also an application that I’m almost constantly in as my dashboard while working on a project. Projectile is another thing I’ve also greatly enjoyed, as a means for jumping to files in a project or grep’ing across files in an entire project. Flycheck is another excellent project, helping me keep an eye on errors I’m making in the buffer as I’m typing them.
s: Cool. Yeah, they’re all very well-reviewed, and people are very happy with those packages for sure.
j: Are there any more questions coming in to you from your various sources?
s: Hang on a second… oh, OK, so there’s a question from [????] on whether bootstrapping Emacs Lisp in Emacs Lisp is a long-term goal so that you get rid of the divide, making the C core as small as possible and doing as much as possible in Emacs Lisp, so that people can contribute more, even if they only know Emacs Lisp?
j: The answer to that is yes. When it will happen, how it will happen, I don’t have the answer to that today, but we would love to have the Emacs Lisp part of the whole Emacs equation be as large as feasibly possible.
s: And then of course every so often people are curious about having other extension languages and other capabilities.
j: Of course.
s: I haven’t been keeping track of Emacs-devel conversations about that though. Do you have any idea of where things are in terms of the big questions that people have like Guile Emacs or multithreading or this lexical-binding that’s starting to show up in the cookies in Emacs Lisp files?
j: Well, so those are several different questions. Guile Emacs is of course always progressing and I’m curious to see what will become of it. It’s not quite at the position right now where it’s something we seriously need to consider integrating but I am very curious to see where it goes and how it evolves, so I keep an eye on that.
Lexical binding is something that I hope will become more and more used and prevalent because it promises to resolve certain performance issues we have right now with dynamic binding and the need to look up a symbol within the most global possible scope every time there’s a variable reference. So you will see that being used more and more.
Otherwise, what we have today, the C core works pretty well for us as a base foundation language. Emacs Lisp is working pretty well for us. I mean, nothing that we’re doing now is preventing anyone from writing great stuff on top of Emacs. So, yes, little advantages could be made here and there by making certain changes, but we have to assess those and what value they offer as they come up. If people have particular ideas, of course, emacs-devel is the place to bring them up.
s: OK. Things are happening, and go ask on emacs-devel if you have further questions.
j: Yes. And I encourage anybody who has a deep concern about the future of Emacs to join emacs-devel and be part of the conversation.
s: You’ve started summarizing some of the longer and more involved conversations in emacs-devel proposals on EmacsWiki, right?
j: Yes, we created a Proposals page so we can distill the final result of very very long and large threads in a way that other people can review them and see what the final proposal being made is.
s: One of those long discussions happened around the time that you were also discussing your maintainership, was Emacs vs IDEs, the features that were missing, the integration of all of that together. What’s happening with this modernization of Emacs’ feature set? Additional more IDE-like features? Is that a thing?
j: Yes. We would like at the core level to have a better API and certain more general APIs for features that have become common to people in other editing environments. So the notion of what is a project, what is the membership of a project, how do you navigate within a project. We’re at the point where there are a lot of disparate features we’ve got out there in the Emacs ecosystem for getting completion lists, for auto-filling or providing automatic documentation for the thing that’s around point. Everybody is solving it in different ways because there aren’t always necessarily good APIs existing currently within core. We’d like to have those APIs in core so that people don’t have to reinvent and recreate the same back-ends for looking up symbols within projects over and over again for the various languages.
s: That makes sense. Once it’s moved into core, then it becomes something all these packages can rely on, instead of having them depend on something, but then somebody else wants to depend on a different package for very similar functionality.
j: Right. What it will look like in the end to use such an IDE-empowered Emacs, I don’t know yet. But I think right now the first question to ask is, what are those APIs? And what should they be? To enable package authors to take the best advantage of the information that’s available within the environment.
s: So I guess part of that is taking a look at the common packages that people are using for this and then trying to abstract what they’re doing in a way that then allows them build on top of that common ground.
j: Exactly. And the ultimate hope for that is that if we have, for example, an underlying API that company-mode as a completion API could rely upon, then the back-ends that you write to provide the information to company-mode could be used by all the other modes that want to glean that information about the symbols that are relevant to point within a project. So then auto-complete could use that, and helm could use it, yasnippet or dynamic-abbrev or hippie-expand or whatever, they could all use it. Now I have myself re-coded the same back-end with very subtle little changes many times just because I wanted to have that functionality available to several packages. Similar but not exactly the same.
s: Refactoring is hard. There’s always little subtleties about how you want to do something that other packages don’t quite do.
j: Right. And I think that core Emacs can better support those package authors in that way by creating a standard, a standard framework within which to capture and ask for this information.
s: That makes sense. Alright, we’ll get the core stable, and then we’ll start moving more things into core, and by that, it’s like you know, out of the C core and into Emacs Lisp, and out of packages into Emacs Lisp where it makes sense, but some things will be moved out of core and into ELPA core and ELPA tarball so that it’s easier to have this well-tested, stable core.
j: With the end goal being: make it an easier job for the core developers who are currently there, and make it easier for new people to join, because it won’t be such a huge body of code for them to approach.
s: I wonder, are there other open-source projects that are also working on expanding their core developer community, and they do that by things like hackathons or [????] this pre-reviewed list of bugs that are super-easy and simple to fix and maybe even have mentors attached to them. I can’t remember if Emacs has one of these things already. I vaguely remember browsing through debbugs and seeing some of them were tagged with “newbie” or “documentation” or whatever, but… do we have anything like that? To guide people into specific small non-intimidating areas where they can start making those contributions?
j: I seem to recall hearing about a keyword we had that was similar to this, but as far as having a hackathon with a bug bounty list, I haven’t seen that yet, and I would love to see that. That would be an excellent way for people to know right where to jump in.
s: Maybe that’s yet another way for people to contribute: where you can just triage the bugs and say, you know, this is a super-easy bug to fix, go ahead and try it out, if you get stuck, talk to me, I promise not to bite. We’ll see where it goes with hackathons and the IRC channel and all these other ways that people can coordinate.
Awesome! We’ve covered a lot of ground today.We’ve talked about the tools you are using to keep up with all the volume and keeping that overview. So, Gnus is awesome, and Org is awesome, all those things together are even awesomer; in terms of the community, how people can contribute; really, Emacs-devel… I remember dropping into it every now and then before, massive threads, hard to keep track of, and now that things are getting split up into different subject lines, that’s so much easier. So emacs-devel and various other places are becoming friendlier places, easier to keep track of. And then of course this overall movement of Emacs source code to be smaller core, more stable, more tests and all that stuff, and then all these packages to go. I’m going to wrap up here, because my throat is starting to give out. Thank you so much for sharing your thoughts and your tips for using all these tools you use to keep track of what’s going on. And thanks again for doing such a wonderful job as maintainer. I look forward to seeing what Emacs will turn into!
j: I also lastly wanted to say that if anyone out there is having difficulties contributing to Emacs, or they have found a stumbling block or a reason for them to have become disaffected or leave the community, to approach me directly about those things, because solving those problems is to me right now more important than solving specific technical issues. So my email address is johnw@gnu.org and it’s always open to people who have concerns about Emacs and where it’s going.
s: And I can vouch for you being an awesome person who definitely helped me get into Emacs development stuff when I got started in 2000-whatever so I can’t wait to see who else you bring into the community from there. As mentioned you can find John at johnw@gnu.org; are there other places people should go and find you, just in case you’re interested in finding out more?
j: On Freenode, I’m usually in the #emacs channel there.
s: And on Twitter?
j: I’m on Twitter as well, as @jwiegley. And of course the Emacs-devel mailing list. Love to see you there!
s: Alright then. I’m going to wrap up here. The recording will be available from the same place on Google+ and I’ll post it in all the usual places as well. Thanks again for joining us for the broadcast here and thanks everyon for listening.
j: Thank you, Sacha.
Thanks again to Phil Hudson for the transcript! =D
Podcast: Play in new window | Download
Subscribe: RSS