About CEDET, Completion, and compilers

[Eric M. Ludlam]

Hi all,

The long thread about "Emacs Contributions" that ended up being about 
Clang, CEDET, completion etc was interesting, and reveled problems 
people run into with CEDET, and possibly some misconceptions about what 
CEDET is with respect to the "completion" feature.  I thought I would 
try and help explain a couple things, and I'll wrap up with my thoughts 
on some TODO items that will help make CEDET better for anyone who wants 
to help out.

For those who don't know me, I wrote the first versions of CEDET and 
maintain it over in the SourceForge repository.  http://cedet.sf.net has 
lots of nifty explanations and pictures for anyone wanting to know more.

To start, CEDET is a group of very different tools, each of which covers 
a particular topic, and which work together to create a suite of 
features many associate with a modern IDE.  That includes project 
management, tagging, smart completion, and code-generation tooling.

Each of these elements is built on a framework where the raw data 
needed, such as project definitions, tag lists, and templates is 
represented in an Emacs friendly data structure.  One of CEDET's most 
important features is that data representation and the way it keeps it 
up to date.  It allows a wide range of back-ends or tools to be 
implemented in Emacs with minimal fuss.

When someone talks about using Clang or other external compiler for 
"smart completion", there is a layer in CEDET where that tool can be 
integrated in as a new smart-completion back-end, allowing any 
user-facing tool built using CEDET APIs to work with this new tool.  I 
don't really see using an external compiler as "competing" with CEDET. 
It is just a tool that fits into one corner of the big picture. 
External tools are likely to fail in many cases (ie - like when it isn't 
installed on a machine), and CEDET will simply revert back to the core 
mechanism.

Anyway, while using an external completion engine is nice for someone 
who just wants completion, it leaves out support for tagging, and the 
wide range of features you get with it.  Fortunately, it is also 
possible to integrate in a compiler (if desired) to produce tags as 
well.  In CEDET there is the tagging parser for C/C++ written in Emacs 
Lisp, plus an Exuberent CTags parser that can be used in its stead. 
Exuberent ctags is off by default simply because it isn't good enough, 
but it can be used for basic tagging for files not in buffers.  If a 
compiler such as gcc were to be extended to help support CEDET, 
preferably (from my point of view) by creating CEDET compatible tags, 
that would also allow CEDET to parse files not in buffers, but it may 
not be robust enough to handle incremental parsing as you type the way 
the Emacs parser does.  Fortunately, they do not need to be mutually 
exclusive.  Both can be used at the same time and work together because 
the underlying data structure is well defined.

This is all just basic modularization stuff applied to a particular 
problem.  Lets me discuss where CEDET tends to fall down.  My hope is 
that this could inspire someone interested in participating who just 
needs a starting point.

Detecting & Managing projects:

You can't do completion without knowing the bounds and shape of your 
sources.  One of the most common questions on the mailing list tends to 
be about setting up a "project" to get the tooling to work.  There are 
so many ways to compile code, and the Make systems are generally opaque 
making this a challenging support issue for EDE.  While I think the 
basic premise in EDE (the project management tool) is good, there is a 
diversity of support that is missing.  A good project here is making the 
task of creating a project for Emacs to describe your code a bit easier.

Quality of the C++ parser:

It was frequently noted that C++ is hard to write parsers for. 
Fortunately, the current parser that is written in Emacs Lisp covers the 
basics, and because it is by nature very forgiving, filling in the gaps 
has proven within the reach of a wide range of people.  Anyone who 
discovers a missing c++ feature should feel confident about being able 
to update the parser.

Quality of C++ completions:

The core completion engine here is also in pretty good shape, right up 
until you start playing around with templates and some other exciting 
c++ features.  David Engster has gotten quite a bit working here, but 
I'll defer to him to explain this in more detail for anyone who wants to 
help out.

Performance, and Really Large projects:

There is a core problem in the way CEDET manages tags, which is that it 
keeps them all in memory.  For large code bases, after working in them 
for a while, CEDET will make Emacs' memory use go up so much that a 
garbage collect spends more time swapping that collecting.

Fortunately, the entire system has been abstracted through some Emacs 
data structures that represent databases of tag tables.  I have done 
some work to create databases using external tools such as GNU Global 
which fit into the system transparently providing tag data.  One 
possible solution would be to develop a nice on-disk representation that 
is easy to search to move the bulk of the data out of the Emacs process.

Refactoring is non-existent:

I did start down a road with "semantic-symref" tools that can do 
primitive refactoring, but some of the key features, such as resolving 
polymorphism issues hasn't been implemented.  The data is there, the 
logic just hasn't been implemented.  At the moment it is mostly a fancy 
grep replacement.

Language X isn't supported:

The system that supports C++ has been generalized, and supports many 
languages.  C++, Java, and Emacs Lisp are perhaps the best supported so 
far.  There are a lot of opportunities to add support for new languages, 
or enhance an existing language support.


I hope this data is helpful.
Eric