DrupalCon Paris is over. Countless of badly needed discussions and brainstormings happened, and we have a couple of weeks to finalize and clean up all of the new APIs and features we added to Drupal 7. I am highly confident that Dries and Angie will continue to do an awesome job in bringing Drupal 7 home and ensuring that it will be a sane release.

But let's think a bit beyond that. I briefly talked with Dries and webchick about it. On my way home, I dedicated the useless traveling time to that fairly difficult question. Please note that this is preliminiary stuff I didn't sleep over, so it may be wrong to some extent.

In Drupal 7, we completely revamped some APIs, such as DBTNG or AJAX. And, we moved "some" functionality of contributed modules into Drupal core:

I'm not even sure whether that list is complete... But the more important point is: These modules are not only in the top 20 of contributed modules. They also belong to the list of modules having the largest issue queues on drupal.org. Which we basically also moved into core for Drupal 7.

Although there may be a few more Drupal contributors that look at core's issue queue and do an incredible job in answering support requests as well as ensuring that some bug reports are not bogus, Drupal 7 will ship with a fair amount of complexity that only a very small group of people really groks. Following up on issues for Drupal core now requires a very good understanding of the APIs and exposed/provided features. Which makes the group of people even smaller, because the functionality is covered in the API documentation and handbooks, but not necessarily in a way that allows new contributors to dive in, quickly understand it, and change it.

Yes, we don't do roadmaps. But I think that chx raised a very good point on the development list recently: We all want to see Views in core for Drupal 8. In addition to that, I would see the following points that would be worth to focus on:

  • Views in core
  • Flag in core (allowing for simple business logic in completely custom ways)
  • Rules (or something better) in core
  • Enhance the entity/object handling (further steps towards the ever wanted Data API)
  • Make arbitrary objects, variables, and strings translatable (basically, but not really, i18n in core)
  • Revamp the help system (perhaps, but not necessarily by Advanced Help in core)
  • Revamp the filter system
  • Revamp the user interface

In general - and I know that mentioning this will lead to objections - these steps will take us some major steps closer to eliminating the developer for many, many Drupal sites and use-cases - primarily regarding a lot of simple/stupid stuff. At the same time though, we not only move the largest issue queue on drupal.org to core, but we also move one of the most complex APIs into core.

By doing so, we will advance on the list of APIs a developer has to fully grok to provide something useful for Drupal core development. Currently, there are only a few developers that really grok the innards of Views. Adding Field API, DBTNG, Internationalization, and other stuff to that, the list of people who are actually able to approve or reject an approach taken in a patch vastly decreases.

Of course, we have peer-reviews. But peer-reviews only work when there are sufficient people who can validate and approve an approach. As outlined in the revised Drupal learning curve (which I want to cover in more detail in a separate post), we always have been low on people being actually able to provide a certain level of trustworthiness to Drupal core committers.

That said, becoming elevated as a Drupal core committer is most probably entirely different to what most people think of. I guess that many people think that being a Drupal core committer means to have some kind of power. But it's absolutely not. It is entirely different: Drupal core committers mainly react on all kind of stuff that is put into their "queue". It is about ensuring top quality for Drupal whenever someone thinks that something can be committed - and that something was actually reviewed and tested by the community (RTBC). And, most often, it is not ready to be committed - due to coding style issues, introducing DX WTFs, UX WTFs, theming WTFs, missing documentation, or whatnot. You don't have any time to play "king" at all (and you'd get sued for trying that anyway).

This basically means that anyone who's playing "core committer" mostly gets pulled into issues that are marked as RTBC, and mostly saying "no" to all kind of patches until they are really ready. It also means to not really touch issues/patches that are marked as "needing review", because those issues require peer-review from other community members and sub-system maintainers first, so Drupal core committers can concentrate on patches that are really considered to be ready.

In effect, that means that anyone who is elevated into core committer business is (potentially) pulled out of the already small group of people that review all kind of core patches (and are actually able to review all kind of core patches). At the same time, pulling out someone of that group of people who can provide quality reviews means that we most likely will commit patches, which this group of people is able to understand. But which are not necessarily understood by any other developers who try to contribute to Drupal core.

Which brings us to the point of documentation. While some parts of the existing documentation are pretty good, there are major parts of Drupal core, which are not documented in a way a developer could similarly lookup a topic and understand it easily (especially DBTNG, Field API, and menu system -- I'm NOT talking about the handbook pages that exist for implementors; I'm talking about the actual API/framework documentation). Take t() as an example: Very good documentation for developers who want localize strings, but not a single word about why it works that way (especially regarding get_t()).

Hence, when new contributors try to figure out how to change some code or even altering an API, they are stuck with either asking "dumb" questions in the issue or IRC to gather the reasoning for why a certain API works like it currently works. You want a really serious proof?

To summarize the macro-level challenges we need to master:

  1. Raise experience and knowledge of existing contributors
  2. Better documentation to allow people to grok and contribute at all
  3. Get more people into core development and peer-review business

Which finally brings me to my proposal for the next development cycle:

For Drupal 8, we may want to pick a novice or relatively new Drupal core contributor, who has a basic/certain understanding of sane APIs, DX, UX, patch reviewing and communication skills, and sufficient time to dedicate for maintaining Drupal core only.

What I mean is actually: Someone who does not understand the (recently added?) APIs yet - like many, many other potential core contributors, too. To get an API patch in, we would have to ensure that this maintainer (and thereby many other developers) actually understands the consequences of a patch by reading the patch. Which equates into: Decreasing the barrier for contributing to Drupal core, and therefore, allowing more people get into the peer-review business.

Lastly, as you may have noticed, this post is mainly about API/in-depth challenges. After being done with brainstorming, I actually figured that all of this does not apply to usability/interface issues at all. Most of those issues (putting IA changes aside) are actually touching "simple" code only. Which means, we already have a slight separation between the Drupal framework and Drupal CMS/application. Which also means that we could be brave and assign separate Drupal core maintainers for the former and the latter (while the outlined approach would then be kinda reversed, meaning that there would be someone highly-skilled for the UX + other stuff, and someone less-skilled for the framework area).

Just some thoughts.

I figure that an approach like this would probably nuke the entire (not really existing ;) list of potential candidates of core committers for Drupal 8, but at the same time, we all suffer from too few core contributors, peer-reviewers, and potentially even sub-system maintainers.

Kommentare

As someone who has been been using Drupal for 5 years (3+ as a full-time Drupal professional), Drupal 7 is the first version I've seen which leaves me feeling like nearly all of my previous Drupal knowledge counts for exactly squat now. Ok, that's a little facetious (not a lot though), but I am not being facetious when I say that it feels like there's as much new/changed stuff in Drupal 7 as there was in 4.7, 5, and 6 combined.

Please note, the only point of my comment is to underscore the point made in your post, which is that 'the drop' is indeed leaving a few people behind at this point. :-)

The moment that I heard that Blog API was pulled from core for Drupal 7, I felt that "the drop" was leaving me behind as an end user. "Small core" and distributions are all fine and good, but is anyone actually paying attention to make sure that the end result is a Drupal that someone (especially a single site ower/admin/writer) can use to make a site?

Drupal seems to be moving farther from that.

And I'd really like to see the whole filter situation addressed to; I really liked your battle plan a while back, and wish I had time/skill to help. I guess I'll wait and see if Drupal 7 makes image handling and general linking (inline, inside a a site, and to external sites) any better than Drupal 6, where I require a zillion contrib modules.

I couldn't agree more: Drupal 6, in my opinion, was already teetering on the edge of this problem and Drupal 7 tips over the edge. There is a lot of complexity in both core and contrib, and the number of "moving pieces" that interact with each other are nontrivial.

When things work, that may not be a problem -- but it raises the bar for contributing much, much higher. To some extent, that problem is what got me interested in (and eventually passionate about) the 'smallcore' movement for Drupal. Separating a lot of the complex 'moving pieces' and focusing more energy on streamlining and simplifying the actual 'core' of the Drupal software is one of the ways that we can work towards that.

Even that, though, is not a panacea: a small but difficult-to-understand core is still just as daunting. (Size makes comprehension easier, but there is obviously a limit: e=mc^2 is short, but unraveling its intricacies is no picnic.) Making Drupal's software architecture easy to grasp for developers is just as important as making its administration screens easy to grasp for new administrators. The two priorities are not at odds with each other, but they are not the same.

"Small core" and distributions are all fine and good, but is anyone actually paying attention to make sure that the end result is a Drupal that someone (especially a single site ower/admin/writer) can use to make a site?

Yes. The impetus behind ideas like "small core" is not to emphasize developer 'fun' over end-user experience. One of Drupal's biggest problems at the moment is that it is used for many, many use cases -- from single user blogging to community organization to ecommerce to newspapers to brochureware. Attempting to put 'what someone needs to make a site' into core will very dramatically from site to site.

BlogAPI, for example, has suffered considerably in core due to few eyes focusing on its needs. It is used by a relatively small percentage of sites. While I'm torn on whether it should have been removed or not (I'm a fan of 'enabling APIs' staying in core even if the installed base is small), it has been broken in many of the most requested use cases for some time.

Migrating it to contrib where it can be explicitly supported by those with more direct interest in the blogging use case will give it a much better chance, and a much shorter cycle time for fixes and improvements. walkah and I, for example, have wanted to work AtomPub support into it for some time, but development of the module has been tied to Drupal core's release cycle and we haven't been able to justify the work while other matters in core loomed for the 7.x release.

I particularly like the description of a core committer's job.

Also the problem of getting peer review for patches is very well presented.

I had both situations when I got the 4.7 release ready, in the end I pretty much had to commit chx' FAPI patches as they came since there were almost no other people who understood FAPI at that time.

Books like drupal pro development provide a good getting started knowledge of drupal but it does not explain any of the underlying API/frameworks.
A Drupal blible where every api/framework in the core and the api's in most popular contrib modules is explained would help students and developers dive in core more easly and start contributing.
I think not only the quality of core contributions but also contrib module would rise.
This Drupal bible would be a very boring piece of reading but after reading the "easy" books and after using drupal half a year it would be a great tool to strenghten someones drupal knowledge.
After that it could be used as reference work.

As a mostly part time Drupal developer and d.o forum answerer since the 4.3/4.4 days, I also have a growing feeling of being left behind, or at least Drupal adjusting to powerful market forces and moving on from what first attracted me to it.

Note: this is just a personal observation, not a complaint and there is certainly no expectation for the anyone in the Drupal community to change course.

I used to feel I understood Drupal and how it worked, and that anything I didn't grok yet was within my grasp if I needed to. I used to eagerly await new versions and get excited about new possibilities for customising and overriding existing functionality. I enjoyed the always moving drop.

With Drupal 7 I don't feel the same way any more, even though the new stuff sounds really cool and powerful. It is all (along with Views/CCK etc) getting too big and complex now. It makes me wonder whether the effort involved in keeping up is worth it.

Anyway don't mind me, I'll go back to polishing my buggy whips now :)

I feel like small core sounds great for development but has the potential to produce Drupal releases that satisfy the needs of only big shops/sites (who have developers).

Maybe unhinging certain modules from core will help them flourish, but maybe not. And when other modules are added in at the same time, it puts the whole effort in a different light.

I don’t use CCK, for example, on my site. What if I wanted to start a movement to remove it? While I think CCK sounds great in theory (and I want to use it sometime), it’s simply not needed for my kind of personal site. I have it installed and enabled so it gets counted in stats. How many other “big” modules are like that. I’m sure a lot are enabled by many sites and only really used by the biggest sites (again, those with developers behind them).

Granted, my fictitious effort to try to remove it would never fly, but what if?

I’m not sure how to best explain what I’m feeling as a user, other than that the Drupal community seems to be moving away from what I value. And since I’m not a developer and contributing in other ways than filing occasional issues, I don’t really have a say. I’ll just end up either having software that isn’t a solution or have to add in more/different modules to make my site work.

Also, to follow on my previous comment, I’m very happy to see Token and (hopefully good) image handling in core. No matter how left behind I’m feeling about Drupal 7.

Evolution is inevitable - take CCK in core in D7 for example.

What we need to do is IMO - improve the usability of the Drupal Core issue queue - I mean think of a way to make it feel more like the old times when CCK was separate....

I am neither a novice nor from the core maintainer gods.

Can we make maintaining core feel more like maintaining contrib modules? Something like a "democratize core movement" :)

The major problem for people with the release of Drupal 6 was that the contributed modules were not ready and indeed some of the useful ones are still in alpha or beta state.

The uptake of Drupal 6 from launch took much longer than is should have due to modules not being available.

Newcomers to Drupal were confused with D6 being promoted as the current version and then finding out modules like cck were in alpha.

I think it's a mistake to release Drupal 7 with Drupal6 still lacking stability in regard to contributed modules.

I use Drupal commercially for high profile projects. Drupal 6 needs to gain stability before diluting efforts in another direction

Looking at the Usage Stats, it's pretty clear that Drupal 6 is regarded as stable.

Nice writeup!
I said it once and i'll say it again. Drupal is similar to linux kernel in many ways. It requires a modern version control system and a feature branches. each branch to have a dedicated maintainer. core commiters will have more time to review important and massive changes because they mostly are responsible for merging branches. more people can get involved without losing quality.
For example maintainers of a huge module like views should be able to have feature branch commit access. who else understand that module better than its maintainers? when it has become stable then they can ask for merging the branch into the core.

Regards

After looking at the usage stats, its pretty clear that those numbers are extremely low especially when compared to competing CMS such as wordpress. How one can look at such statistics and declare stability is beyond me. Windows has more users than Linux yet which one would you consider more 'stable'? Questions like this can not be answered using these stats. I know I'm wasting my time replying to you but I just dislike when people are manipulative even if that wasn't there intention.

With regard to my comment about stability it as it says in the post - contributed modules. Not Drupal 6 itself.

what is difference between cms and framework?

Thanks,
Business networking

Very good post, thanks a lot.

No worries, your list is great! I find it very useful

I agree with you.There are lot of challenges for drupal but still it is managing to get to top in blog Kick Ass Costumes all development.

Indeed, there are lot of challenges for Drupal and its future seems to be not so cloudless. I've recently thought a lot about the fact that while the Drupal 7 launch events demonstrate enterprise momentum, Drupal faces a new threat, from Enterprise Resource Planning (ERP) applications, and CIOs anxious to assert control over Enterprise 2.0 features and the employees using them. I know that members of the Drupal community have been actively discussing the construction of ERP functions within Drupal, but delivery of such modules could be some time away, and many security managers may find more comfort putting communication inside their ERP systems than going the other way. I also read that integration with Customer Relationship Management (CRM) systems is also starting to emerge from third parties. It is obvious that while Enterprise 2.0 may be a handy buzzword, as a practical matter it demands integration of enterprise software systems under standards enabling interoperability across all departments. Drupal 7 is a play in that effort, but whether it's a big enough play for large enterprises has yet to be determined.