I want to be honest. I expected and feared a lot of backslash, disagreements, and even name-calling on my Drupal Crisis blog post. But to my surprise, almost everyone who provided feedback mostly agreed with most issues raised, and even more so on the overall problem space at glance. In fact, the only criticism came from "top-down" about "not being constructive".

Talk to me, darling.

It felt very good to finally get that post off my chest. Prior to writing the previous post, I merely knew that at least my fellow mates and core developers catch and chx shared these feelings.

However, an interesting part is that no one really started to raise concerns and complaints earlier. It looks like most of us swallowed pill after pill, issue after issue, announcement after announcement, in the mere hope it might be the last.

Drupal's overall (core?) development environment and "culture" of constantly criticizing, flaming, blaming, shitting, and pissing over the existing problems in APIs and code certainly is a factor here: We all do it all the way; and when another pill enters the inbox, there's a good chance it's silently trashed, since we've created plenty of flames about really annoying code-related stuff in the core queue already, so there's neither time nor energy for another one that's 1) not related to core architecture and 2) may not be so important at first sight.

Things pile up. And in retrospective, everyone obviously knows better and would have decided differently. As a matter fact, we don't have a proper public place anywhere to discuss these and related ramblings, which are crucial to the project: Does Acquia exert inappropriate influence on Drupal core?
(though this little post, given the size and amount of groups on g.d.o, looks a bit lost, compared to its importance.)

We need to discuss whether and how acquia presents a conflict of interest.

I can only applaud and cheer for effulgentsia (who happens to work for acquia): we instantly had a very long Skype call, iterating over every single issue to clarify what's ultimately meant, in order to pass that info on to others. Alex continues to amaze me with his structure, clarity, creativity, and unparalleled quality, both in terms of code and communication. If he didn't work for acquia and there wasn't that very same possible conflict of interest, I'd place all my votes for him to become a core maintainer.

rfay's core conversation about burn-out at DrupalCon London is closely related: it revealed that there are many more Drupal contributors who kept silent about their personal feelings for too long. In the end, we have several critical communication problems about fundamental topics in our community.

We need to discuss how we can resolve that lack of communication without sacrificing our contributors and putting them into a dark corner at the same time.

Chocolate and ice-cream, or: the carrot and the stick.

A detail that many didn't recognize in the original post: There have been 8,000+ new bug reports, within two years, of which every single one had to be 1) created, 2) analyzed, 3) patched, 4) peer-reviewed, 5) tested, 6) approved, and lastly 7) committed. On average, this means that 320 bugs had to go through this process per month, and 10 bugs per day.

As tsvenson pointed out, these numbers will only be increasing, and we already see a rapid pace today. The amount of bug reports itself is concerning. Somehow you'd expect the volume to go down over time - although that may be debatable, since core purposively changes things - but still, the total amount of open bugs goes up. Some core developers are arguing that we're able to cope with the increasing numbers by attracting more contributors for Drupal core development.

However, what is most concerning is the amount of work being required for the points 2 and 4-7 in above list, which has to be delivered by already experienced core developers, and, whether that is a sustainable model for the future.

So what I'm really after is this:

sun: I should have clarified what kind of stuff I'm ok with maintaining, patching, and reviewing patches against in core. And what stuff not.
catch: everyone who ever has written a post like this should've done that.

Apparently, no one did.

At least I don't think I've ever seen any kind of analysis or proposal of a core developer who stated what exact parts of Drupal core he/she actually considers as core material himself/herself. And, which he/she is willing to actively maintain in core. Instead of being forced to by current contents, present nostalgia, and current development processes.

We need to discuss, propose, agree, and define what we are able and want to maintain in Drupal core.

Another very long and heated call with chx and webchick revealed that basically all of of the "product features" in Drupal core are de facto unmaintained. In order to improve them, move them, or remove them, we need dedicated maintainers for our product features. But we're not only done with finding people, because:

We need to discuss which core features are actually product features.

Most of these feature modules are actually easy targets for newcomers. Most of them do not contain complex logic and code, so issues are on an entirely different complexity level than, say, refactoring System module.

But obviously, it's going to be hard to find maintainers for brain-dead modules like Aggregator. A product feature in line with the core architecture and the year 2012+ would rather be a total replacement with Feeds module (or whatever the current contrib lead is; sorry, I've lost track of that evolution).

The process to find people who're willing to take on maintenance for the product features and topics is going to take some time. Unless the Drupal community gets its act together. As of now, all of this based on hope only.

Many people have argued for the "traditional" open-source process on the original post already; i.e., move unmaintained ballast from core into contrib, and add new, modern features to core. Some core developers argued against this, because they originally found their way into core development through those old modules, and because a few of them basically served as unforeseen "regression testing" during Drupal 7. But yet, there's neither someone who wants to take them on, nor someone who has a vision yet.

We need to discuss a deadline for finding maintainers for product features.

And based on that:

We need to discuss what we are going to do with stuff that no one wants to maintain.

In Drupal core, we only need a certain and solid amount of consumers of core APIs. These can still make up for an awesome, basic product. But we don't need to maintain features that almost no one uses or needs. Those unnecessary features not only distract all of us during regular core development, they especially make the refactoring of core APIs harder.

Related to that, we seem to have a weird understanding of moving core features to "contrib". In the past, we only ever moved code from core to contrib that no one actually used. The proposal of moving the Standard installation profile, including the product feature modules, into a separate project (possibly everything in a single project) didn't yield much positive feedback yet.

Even though that very same project, if required, could theoretically be maintained by Dries as well. And even though all of those product features would be able to evolve in a much faster pace than Drupal core does. Which especially makes sense for pure UX modules. There are no dependencies on API changes that would have to be taken into account. In fact, all of those feature modules architecturally share the same principles as any other contributed module does.

We need to discuss whether we can or want to move product features into a separate product project in the long-term.

What may I serve you, master?

The primary objective is focus. Too many irrelevant things distract us from the important things and badly needed refactoring, and especially make the latter harder. Every single issue sucks core contributor time and energy.

And don't forget about core maintainer time. Spending less time on whatnot issues means to have more time for actual architectural discussions and reviews. Many core contributors repeatedly stated in the past that they'd really like to have "back" a technical lead. The current interactions of core maintainers are mostly about the very same topics all over again: sanity check, backwards-compatibility, API changes, coding style, documentation, yadayada, but close to zero feedback and discussion about actual architecture. The current situation is an extreme case of do-ocracy. The Drupal community expects (core and non-core) project maintainers to lead, provide and share vision, and direction. Also in core.

UPDATE:

Make sure to watch eaton's talk about Product, Framework, or Platform? held at DrupalCon London. It provides in-depth historical insight for how we arrived in the current situation. It takes one hour, but that hour is more than worth to invest. Thanks, eaton!

UPDATE #2:

Seen Jeff Eaton's presentation? Then look and see how a demo drupal.org Download page of Drupal as a platform would actually look like.

This solves pretty much all of of the identified problems (and many more). drupal.org's infrastructure is fully prepared. It's merely a matter of doing it.

But in order to do this to core, we need to have all of the discussions outlined above.

Comments

Hi Sun - Thought it worthwhile to point you at http://westendgirl.ca/burnout-way-out - we've been having similar conversations about docs this week, and are in a comparable place with a small core team trying to take care of all this work that's been heaped on us. I'll be posting probably on g.d.o with more official proposals in the near future.

Not sure where we'll go with this, but figured it was worth noting. Putting aside what's best for "Drupal", what's best for the community is becoming clear if we want to remain sustainable.

I agree with every single word ... again :-)

Very well written. We have lost the focus and aim. There are too many complex subsystems and since Dries leaved active development process, there is noone whose able to overview all features of Drupal from higher altitude. Dries has been trying to clone himself by hiring webchick and Moshe and he also created the initiatives... but I'm in doubts if this steps could be the proper replacement ... for the Leader - an Architect with the clear vision and General with the respect and power to push the things. We'll see in the forecoming months I guess.

Thank you for these thoughts. I feel like we badly need this kind of battle plan.

Let the various companies that have the budget (i.e. got investors) worry about assigning developers to maintain those features that they want to maintain. Throw everything that the core developers don't care about out... and that means everything that they are not interested in working on. The product should be maintained by those interested in delivering the product to their customers. The core should be what the developers are interested in working on. The companies can pay people to maintain the other stuff if it is important to their business success. It seems like open source developers got pulled into servicing the corporate needs of certain customers.

Thanks for posting and the original Crisis post!

I'm not a core developer. I'm making lot of projects based on Drupal as a framework and primarily not as a CMS. I've always seen Drupal as a very good and growing-capable framework. In fact, the more the code grows upon versions, the less it looks like a smart framework... and the slower it goes. In my opinion, there must be (simple to say, not so simple to do... i know) a Drupal framework and - apart - a Drupal CMS based on the Drupal framework, but the two of them must be splitted as soon as possible. Like you say, there is plenty of efforts to do "just" on core... the implementation is starting to be a real mess (part oop, part proc., etc.).

Are you suggesting to make the changes for Drupal 8?
(sorry, english is clearly not my mother-tongue too).

no one really started to raise concerns and complaints earlier

I'd take issue with that. I think the whole smallcore debate that started 2 years ago was a collection of complaints and constructive suggestions for dealing with most of these issues. The complaints and efforts to do something have rumbled on that long.

Dries keeps adopting solid proposals for evolutionary change, addressing problems in part, but it seems the pressure for more revolutionary changes continue to build. I don't blame him, his leadership style has clearly taken Drupal a long way in the right direction. Embracing radical changes like splitting Drupal dev into core and product engenders risk as well as benefits. It's a damn scary prospect for someone in Dries position (I think any potential Acquia conflict of interest pales in comparison to that).

(A point of view from my perennial position on the sidelines watching.)

That sounds like a great, positive way to try and tackle the issues our project is facing.

My main issue with all the plans that are being drawn is that they don't seem to recognize the importance of the leadership issue. I don't mean to single out Dries as an individual or even Acquia; they both have committed impressive amount of resources to make what Drupal is today, however Drupal, as an Open Source project, is in need of an available, focused, technical leadership... and it seems Dries is not anymore able to cope with that responsibility.

I think it is time we see the technical chops escape the control of a single person; the community is mature enough to handle it.

I agree that thanks to Dries criteria Drupal is what it is today. A very successful CMS (and platform, and framework). Part of this succes is due to Acquia marketing and technical resources and services. I also think that Dries, as he noted many times, is aware of the problem.

In the other hand many great community contributors have abandoned the project. I think we will regret if this continues happening. Actually, I will love to see people like Steven Wittens (a.k.a. UnConeD) or the DevSeed team back to Drupal. They are a huge part of why is Drupal so great and so successful nowadays.

I totally agreed with sun's visión and proposal of what Drupal could be. Especially with first class contributions/products.

I see that the "product" is limited by core. And that is why they are tided together. Because Drupal is focusing at the product. I agre with that, but I see that product/platform more like Eaton's London2011 presentation Apple HyperCard simile.

P.D: Jeff Eaton's presentation is a must.

Thaks,
Alessandro Mascherpa.

Thank you for starting this conversation. I think one of the things that shows the strength of our community is that we can very quickly turnaround edgy and hard conversations and start focusing on getting clear and finding achievable practical action.

I don't think anyone could disagree with making core less interdependent, and I think the vision of having a simpler, framework focused core, with separate core products is a wonderful one, whatever the actual route to that is. I think we do need to ensure we find high quality owners for "product feature" modules that will really take care of them properly, both from a strategic and code point of view (I don't think it is beneficial for Drupal as a whole to just "throw them in contrib", to borrow some folks language, although I think it demonstrates the point that some core devs are perhaps not best placed to do the job of finding good homes for code that they are already frustrated with maintaining!). As the de-facto owner of most of these, I think the direction for that process is in Dries court. I would suggest having a public process, whereby individuals lay out their vision for taking on a specific "product feature" module and Dries can select one or more initial owners, with input from the community. To allow time for the likely code untangling, perhaps prioritize a list and try and split off one every few weeks.

I think the influence of Acquia is very hard to measure, although certainly something to watch closely. I know enough Acquia-employed core devs and am pretty sure there are people who would not sit quietly if there was a corporate driven push for things that were against the interest of the community, but that is not to say that more subtle influences would go unopposed.

I think having the "product feature" modules separate from core would go a long way to balance the politics/influence (of anyone, not just Acquia). To start with, they would each be owned by people who care about them, unlike now where they are mostly just tolerated by core devs who would rather be fixing more critical framework issues. This means that there could actually be a higher bar for their development than there is now. It also means that if one owner/entity did manage to take them in a direction that was not beneficial for the main products, they would simply replace, remove or fork them - and the owner would know this, and would be less likely to take things in this direction to start with. By tying "product feature" modules up in a single core package, this bi-directional influence effect is much more complex, and pushing one piece out of balance is much more likely to affect the dynamic and success of Drupal and the community as a whole.

More philosophically, I think the drive towards simplicity is something that used to be very strong (for example, I think it was being part of Dries keynote back in Drupalcon Vancouver), but it does not feel like it has been quite as prominent recently. Obviously we are trying to do things that are much, much more powerful than we were in those days, but I think we would do well to reach more for finding simple and elegant solutions, even for these higher order functions.

Issue queue thresholds for Drupal core http://buytaert.net/issue-queue-thresholds-for-drupal-core

As a developer who uses Drupal for several of our client sites, I appreciate these two posts. Firstly I'd like to commend the core devs for producing a product in D7 that is much more enjoyable to use and set up than D6. Being able to upgrade modules from the admin section is just plain great, and the overlay features really are quite nice. Perhaps you all haven't been congratulated enough, but you deserve it. As you've explained, the high bug count is understandably dispiriting for the devs, but you all did well given the enormous challenge.

However, you are correct that these outstanding issues have led to slow catching up with many of Drupal's modules. Two of my favorites, Nodewords and Views Slideshow, have been very slow to come around on D7. But our company has pressed on with our commitment to use D7 instead of D6 for most clients this year. Your ideas for streamlining the core code of Drupal seem very logical. Drupal still lags behind Wordpress in the WYSIWYG author experience, and that is unfortunate. That really should be a matter of priority for D8.

Personally I develop our larger client projects in CakePHP because of the unrestricted freedom I have for database design and template control. Switching between Cake and Drupal provides a lot of insight on how things could improve, but the biggest issue I see is that Drupal is starting to feel the heat from its lack of an OOP development model. I can't help but think a system like Drupal developed on an OOP framework like Cake would make life much easier for core devs. It might be too late to start thinking that way, but it is something to consider as Drupal's bug count gets more and more unwieldy.

I think Drupal is still the bee's knees when it comes to a CMS that is adaptable and quick to customize for most client needs. It's got enough advantages over WordPress (which of course also lacks an OOP model) that still make it the platform of choice for sites that are not primarily blog-centered. But it now faces the same challenges that 99% of software has at some point--outdated legacy code that threatens its future viability. Hopefully the Drupal devs can agree to your basic points for a streamlined core--even if fundamental OOP changes are not feasible at this point.

Well I tried installing Aquia Drupal on Win7 64 and got a bug: the post install process never finishes. Cancelling rolls back the whole thing. Googled and it turns out the bug has been there since 2010. Aquia hasn't fixed the problem and instead posted 'hey! we're closing old bugs. you still need this open?' So I'm alarmed to hear they released Drupal 7 with so many outstanding bugs. To hell with this. I'm heading to Wordpress.