The feature freeze deadline for Drupal 8 is approaching, so it's highly important for the community to clarify goals, ensure focus, priorities, and channel contributions.

We want Drupal 8 to make a big difference and be a leading web platform. But in all seriousness, we didn't make much progress yet. More than 60% of all changes were actually bug fixes for Drupal 7. 41 weeks are remaining. In merely two weeks from now, we're down to 39.

Some core initiatives made some good pragmatic progress, but there are almost no major architectural changes yet. A few initiatives are completely stuck, due to a lack of understanding, communication, and scope resolution, and lastly, due to not being able to get everyone on the same page. We want to achieve some serious progress over the next 9 months, so we need to get our act together.

Last weekend, 14 core contributors met in Boston in order to resolve the conflicts and unblock the initiatives. This post attempts to explain

  1. some background on the particular problem child, the WSCCI initiative
  2. what happened at the sprint
  3. how Drupal 8's overall architecture will look like
  4. why we want to change Drupal's architecture

What is WSCCI?

WSCCI (pron. "whiskey") stands for the Web Services and Context Core Initiative. It is one of the six core initiatives for Drupal 8; which essentially are working groups of contributors who are focusing on a certain, special topic. To quote its mission statement:

WSCCI aims to transform Drupal from a first-class CMS to a first-class REST server with a first-class CMS on top of it. To do that, we must give Drupal a unified, powerful context system that will support smarter, context-sensitive, easily cacheable block-centric layouts and non-page responses using a robust unified plugin mechanism.

It tried to achieve a lot. The mere density of keywords and buzz-words in the mission statement makes it apparent that the initiative's goals affected pretty much every part of Drupal. These goals sound exciting, so a lot of people actively participated in the initiative's discussions, and the core initiative actually experienced an uptick of Drupal contributors who silently worked on contributed projects in the past, or who fundamentally disagreed with Drupal's architecture that grew over the years.

At the same time, the WSCCI initiative was a constant source of pain and confusions among Drupal core developers since its inception, caused by several factors:

  1. Scope

    Unlike traditional core development issues and efforts, WSCCI attempted to attack a wide range of diverse topics, and all of them at the same time. For everyone not constantly participating in the initiative's discussions, the sheer amount of change proposals affecting all kinds of Drupal core components was neither traceable nor understandable.

    Not understanding what the initiative is actually after and why it wants to perform a certain change made core developers with varying experience levels reject the initiative's patches. Even after re-thinking WSCCI and trying to approach it from a different direction, the scope was still too large, and it was still not clear what exactly WSCCI wants to change in detail when touching a certain subsystem in core, and even more importantly, why.

  2. Politics vs. Communication & Implementation

    The high-level goals of the WSCCI initiative are all reasonable and straightforward. I'll go out on a limb and actually state that almost no one in the Drupal community would disagree with its stated vision. Elevating Drupal to the next generation web is what the Drupal community has always striven for, and WSCCI's high-level goals are perfectly in line with that.

    In turn, the initiative's problems are not of a political nature. To put this into comparison, unlike some of the "usability" features that went into Drupal 7 (e.g., Toolbar, Dashboard, Overlay modules — there, I said it...), there is almost no disagreement throughout the Drupal community on whether we want to move into the outlined direction within Drupal core (vs. contributed extensions). Instead, the dissent was caused by a disagreement on the technical implementation, which to a large extent was most likely also caused by a lack of architectural communication.

    Although a range of fundamental architectural changes to Drupal core were targeted, communication about concrete change proposals resulting out of the high-level discussions mostly happened through channels that were separate from the community's main communication channels focusing on Drupal core changes. This was beneficial for the initiative's participants to concentrate on their vision, without being side-tracked and disturbed by unrelated topics. At the same time, it presented a disconnect and problem for other core contributors, since it inherently moved details about concrete change proposals out of the main channels. So without any doubt, a lot of detailed considerations probably happened, but only the minority of active initiative participants got to know of them.

  3. Specialists vs. Generalists

    Freely re-thinking Drupal's architecture on the high-level without practical constraints is definitely beneficial. But without spawning individual change proposals into individual, smaller, and targeted "efforts" that each focus on a concrete change only, we've effectively experienced 3 silos of contributors that refrain from talking to each other, since every group comes from a different background and latest state of thinking:

    • High-level initiative participants (high-level specialists), having crazy new (good) ideas that involve major architectural changes. They're trying to make sense of what they're discussing, in order to complete the big picture in everyone's brain and to make the technical prototypes and implementations actually work.

      However, most of the individual change proposals resulting of their considerations naturally depend on each other, so each change is potentially and very easily leading to a massive scope-creep. Also, things have been discussed within the group before, so it's painful to re-iterate the entire train of thought to "outsiders", who didn't participate in the group before. Most prototyped changes are very experimental, so they naturally do not explain previously discussed architectural details either. At the same time, participants may not be familiar with the entirety of core subsystems or the full architectural design of the subsystems they are touching.

    • Drupal core generalists, who know about and care for most parts of the system, often rethink and revamp specific parts of the system, and help others to contribute. They are mostly busy with a huge diversity of patches filed against Drupal core, so they don't necessarily have time to dive deep and engage in all high-level initiative discussions.

      Consequently, they're experiencing a big disconnect between what they know about current systems and existing change proposals vs. entirely different and huge changes proposed by the high-level initiative.

    • Subsystem/component specialists, who are maintaining the current subsystems and components in Drupal core, ensuring that bugs get fixed correctly, and new features or pragmatic changes are implemented properly.

      Additionally to that (mostly ungrateful) work, they try hard to adapt themselves to major change proposals, so as to be able to provide early and proper feedback on whether a particular change will work out for their subsystem or not.

      But in fact, most subsystem maintainers are not able to stay on top of both their regular duties as well as getting deeply involved in initiatives to provide technical feedback. Because of that, initiative participants were sometimes missing crucial pieces of key knowledge when architecting solutions. (that said, huge props to Yves Chedemois (yched) for being one of the few who not only provided excellent reviews, but even revised their subsystem's code.)

    (More on these roles in my Drupal core development post from 2009)

All of these individual issues slowly but certainly led to a "us" vs. "them" differentiation between WSCCI participants and other developers working on Drupal core. At the same time, the huge gap and misunderstanding of each other caused some long-term core developers to effectively step back and stop contributing to Drupal, as they weren't able to make sense of what's going on anymore. In the end, we had experienced damage to the level of trust among core developers.

Sprint anatomy

The path forward was unknown. The conflicts blocked the initiative entirely. A decision had to be made to figure out whether it's worth to pursue the initiative for Drupal 8.

Therefore, a larger group of Drupal core contributors met in Boston for three days with the following agenda:

  1. What is WSCCI trying to do, exactly?

    The entire first day was spent with outlining which individual changes WSCCI actually envisioned, which of them are important (and which not), and what the individual topics actually mean and want to do. Distilling that led to these efforts and topics:

    • Web services
      1. Structured request/response pipeline
      2. Infrastructure for building web services (menu/router system)
      3. Different response types (HTML, JSON, XML per URL; former delivery callbacks)
      4. Restful API to perform CRUD operations
    • Blocks/Layout
      1. Contextual blocks
      2. Blocks everywhere/anywhere
      3. Multiple page layouts
      4. Partial page rendering (ESI, AHAH)
      5. Better UI/UX to manage blocks
    • Framework
      1. Dependency injection
      2. Unified Plugin system
      3. Bootstrap lazy-loading (PSR-0)
      4. Entification of entities in core modules

    The goal of the day was to get consensus and a final decision on what's in and what's out for WSCCI. My personal goal for the sprint - also representing feedback from other core developers who weren't able to attend - was to attack the root cause for the problems, which meant to heavily scale down the initiative's scope.

    At the end of the day, we were able to agree on that, and the revised and newborn Web services initiative will only focus on the web service topics being outlined above. The other topics formerly encompassed in WSCCI are also important for Drupal 8, but will be dealt with in separate efforts and possibly new initiatives.

  2. How do we implement Web services?

    Fabien Potencier (fabpot), project lead of the Symfony framework, kindly and very patiently provided us a plethora of insight into how Symfony attacks the web services problem space, and how it kept him busy over the past 6 years to get this "simple" topic right.

    Like other core developers, I personally was very skeptical about incorporating and relying on an external framework at the heart of Drupal core. However, as we discussed what we actually want to do, and thinking through what that would mean in terms of actual technical implementations, and after Fabien explained the internal architecture of Symfony's design, the previously front-loaded proposal of leveraging Symfony turned into a very logical after-thought of "This totally makes sense."

    It's tough to resemble and write down the full train of thought that led to the conclusion. I guess I personally struggled most with WSCCI's original idea of introducing a "context" throughout Drupal core. It was supposed to hold all contextual information that would be required for a particular function, so nothing would require randomly created data from somewhere else anymore. It was hard to imagine what that would mean in terms of code and architecture.

    But now, "map each piece of content to an own URL" made a big difference. It implies that every piece of content can be requested separately. Instead of thinking in pages, think of an individual block. In turn, the request is the context. Thus, if you can resemble the same request context, you get the identical response back. Just as with any other web resource you're requesting.

    Symfony's core components are actually fully designed after the HTTP specification. It does not try to account for the HTTP layer as an after-thought, the concept of a HTTP request and response is the fundamental core of it's architecture. This concept, in turn, resolves a full range of other issues, which the Drupal community always struggled hard with.

    Rebasing the thinking on "URL <-> Block" clarified that we'd start to setup the contextual request information in Drupal's menu router, and from there, pass it forward to the callbacks that produce the response for the block. We'd also put all of the implicit request information on it, which is generated earlier in Drupal's bootstrap (such as the current user's session or language).

  3. How do we implement Blocks/Layouts?

    We spent the last day to discuss the other radical changes having the goal of turning all of Drupal's output into atomic pieces of content. Now that the scope of Web services got cleared up, these change proposals totally made sense on their own, too. More on them below.

  4. How do we implement Configuration?

    During the sprint, we identified a range of hard dependencies on the new configuration system for Drupal 8. Greg Dunlap (heyrocker), the initiative's lead, participated in the entire sprint. So we leveraged the day after the sprint to further discuss its architecture and make some solid progress on the already existing core patch.

The entire weekend was literally spent with in-depth discussions about the envisioned architecture and possible technical implementations. No coding involved. Each day started at 9:30 and ended late at night. I've rarely experienced such packed, intense, and very advanced level of discussions in my life - not to mention, spanning over three entire days.

Next to resolving conflicts, splitting the initiative into multiple has many other benefits:

  • Each one has an intuitive, limited, and thus understandable scope.
  • Consequently more cross-communication between individual efforts and initiatives.
  • Independent efforts mean that each working group needs to reach out to others more often to assert their assumptions, which inherently improves communication and coordination with "outsiders".
  • Dependencies between efforts are reduced, or at least more clear.
  • Different goals are attacked in parallel.
  • Each effort will end up as a new or revised subsystem in core. Each core subsystem needs dedicated maintainers to be sustainable, so individual efforts provide at least some better pointers for whom that may be (whereas initiative leaders are not necessarily supposed to take on that role).

The sprint helped to resolve misunderstandings and the disconnect between the initiative participants and other core contributors. Improving the communication about concrete change proposals for Drupal core is an issue that remains to be unresolved though.

From a generalist's perspective, the path forward includes some hope to bring back the essential discussions about wide-ranging Drupal core changes into the main communication channels, so as to get all core developers rapidly back on track and on the same page.

Many thanks to Acquia for hosting the sprint, feeding, and also flying in sprint participants!

High-level Drupal 8 Architecture

Ignoring the individual initiatives and efforts for a moment, let me provide a bigger picture of the envisioned architecture for Drupal 8:

Each piece of content gets its own URL
  1. GET /system/block/user/whos-online returns the HTML fragment for User module's Who's online block.
  2. A "page" contains many pieces of content, which will be "injected" through sub-requests within the parent request that builds and delivers the page.
  3. Each URL's response may be cached. (Like any other URL today. Forget block caching.)
HTTP at the core: A Request context and unified Response container
  1. The Request object serves as total context container for a HTTP request, holding all request-specific information:
    • loaded menu router arguments
    • superglobals ($_GET, $_POST, $_SERVER, etc.)
    • Drupal-specific request information ($_GET['q'], request_path(), URL alias)
    • other context data currently being global variables (e.g., current $user session, interface $language, and $language_content, etc)
  2. The Response object ensures that each response consists of standardized elements, i.e.:
    • Response HTTP headers (e.g., drupal_add_http_header())
    • Response body
RESTful router system
  1. Built-in content type negotiation (based on HTTP Accept header or file format extension).
  2. Native support for other response content types other than HTML (e.g., JSON, XML).
  3. HTTP method (GET, POST, PUT, DELETE) specific router information per URL (when needed).
Synchronized configuration system
  1. All configuration is stored both in an active key/value store (e.g., a database) as well as in files.
  2. Default configuration of modules and installation profiles is defined in configuration files, which are automatically imported into site configuration upon installation.
  3. All database tables containing configuration records (e.g., node types, vocabularies, etc) will be replaced by configuration objects.
  4. File-based configuration objects can be versioned and pushed between environments, which entirely replaces Features and other configuration export patterns.
Content blocks and layout blocks as plugins
  1. Each piece of content (including page callbacks) is converted into a content block plugin (rather comparable to CTools content plugins; not to be confused with current Block module blocks).
  2. A content block plugin on its own is a PHP class only. It merely defines the type of content. A block plugin does not represent or "show" a particular piece of content - if a block plugin can render a specific thing such as a certain entity, then that is configuration. (thus, block plugins are not to be confused with Block module's custom blocks)
  3. Block plugins are spawned into one or more block instances. Each block instance has its own configuration.
  4. In addition to content block plugins, there will be layout plugins (comparable to CTools layout plugins).
  5. Layout plugins consist of regions, configuration, a template, and CSS/JS asset files.
  6. Recursive layouts from the top: Layout. Region. Block. Each page uses a certain layout, and each region in a layout may contain content blocks, but also layout block plugins, of which each again contains regions that may contain...
  7. A block instance may be placed/configured for a certain layout. The configuration of the block instance is always the same though.
  8. All raw output (e.g., logo, site name, primary links, etc) in theme templates (especially page.tpl.php) will be converted into block plugins.
  9. Each plugin always gets the Request container and configuration passed into it, so it has full context.

Note: All of this is only comparable to CTools. Plugins in Drupal 8 will look different and their architectural design is going to use modern PHP5 features and patterns.

Pluggable and swappable subsystems
  1. There will be a unified system for plugins, not only used for above mentioned block plugins, but which also aims to replace many existing, custom plugin system implementations throughout Drupal core (e.g., drupal_mail_system()).
  2. All object-oriented PHP code will leverage the PSR-0 standard of using namespaces. This means that a lot of code does not have to be front-loaded, but can be lazy-loaded (on demand) instead.
  3. Potentially, entire subsystems may be replaced by contributed or custom implementations through a Dependency Injection container (often referred to as "Service container").

Why?

Drupal always changed and reinvented itself.

  1. Performance is key. A lot of Drupal's output is not cached currently. Its architecture is based around entire pages instead of individual pieces of content. By mapping each piece of content to a URL, we can heavily simplify and improve Drupal's internal caching. It also means that external caching layers can leverage dynamically generated content fragments within statically cached output (a.k.a. ESI).
  2. The mobile market is constantly expanding. We not only need to support different devices and responsive design. Drupal's challenge is also to deliver a platform that allows every site to be turned into a web service and backend/data-store for external applications (think native mobile apps). This requires a RESTful architectural design from the ground up - not as an after-thought.
  3. Simplification. By all means, the outlined architectural changes are big. But when thinking about the resulting system, we expect that it will become much easier to understand how the system works, how to develop for it, and how to implement it.

Get involved!

All of these initiatives and efforts require a lot of work, so if you haven't already, now would be the perfect time to get involved into the Drupal core project and help us to make this happen! :)

Kommentare

This all sounds really great. Thanks for the update sun.

thanks for this great summary, sun

Thanks, for the detailled write-up. Sounds like a great plan to me!

I am thrilled to see that 'not invented here' is a thing of the past. The rejection of NIH applies not only to code in the form of external libraries, but also architecture. I couldn't be more pleased. This is a sign of real maturity. Everyone in the community new and old reflect on this moment. Drupal just leveled up.

Thank you to everyone involved in these initiatives and that had the drive to push them forward.

Thank you to the symfony project and people in that community for supporting this change in Drupal.

Thank you hard-core core contributors for all you've done and continue to do for Drupal.

I can't express how exciting I think this is for Drupal. The changes outlined here have so many potential positive outcomes.

  • More elegant code.
  • More interoperability with other projects.
  • Drupal finally developing a true core 'framework'. The internals are being hot swapped and a product is still running on top. Amazing!
  • More easily testible output
  • Performance: trust me when I say, if this is done right Drupal is going to be screaming fast
  • Configuration in code (homer drool sound)

And that's only to name a few.

And all of these things have so many positive implications for all sorts of different Drupal professionals today. From testing teams, to dev ops and deploymet teams, to infrastructure teams, to developers.

If the last sprint removed the major barriers and got most people on side - then we must celebrate.

Excellent write up, thank you!

Thank you for this excellent overview. It really goes a very long way in helping those of us standing on the sideline to get a grasp of what's happening. You've also made it very clear, grokkable instead of dense! :)

I must add: holy shit that sounds awesome!

Excellent post, and thanks for helping me get a clearer understanding of what D8 will look like.

A block instance may be placed/configured for a certain layout. The configuration of the block instance is always the same though.

Perhaps I am under-informed (haven't had a ton of time to keep up with WSCCI since its inception), but does this suggest each piece of content may define itself as a block multiple times?

Thanks for the write-up, Daniel. Very helpful for a contrib subsystem specialist like me. Even sitting in on some WSCCI brainstorming sessions, I never really grasped what was coming down the pipe for Drupal 8, leaving me mostly unable to plan for Drupal Commerce in the "New World." Your high-level outline is very helpful, giving me something concrete to think about and watch for.

Thanks for the in-depth explanation, very useful. I think the goals of the initiative are excellent, but I worry about the time frame!

I`ve been following @crell and the rest of WSCCIs team members posts for some time. I was really excited, feeling that Drupal was in a huge moment, becoming from awesome to superawesome. But I really didn't see why so clear until now. Thank you for this excellent summary. And thanks to all the people working in this project. I will do my best to contribute as much as I can.

Cheers,
Alessandro Mascherpa.

It means that a single block plugin (represented by a class+config) could have multiple instances. In this case the only thing that is changing is the config object passed into the block plugin.

Rather, everything that shows up as HTML output comes from a "block" (which is more in common with Panels content type plugins of today than with the dated blocks system that's in core now). The intent is that you'd have, say, a WhoseOnline block in code, that is a class. Then you would create an *instance* of it (with configuration if appropriate), and place that instance into a layout. You can place that instance multiple times (something you cannot do today), and also create multiple instances (with different configuration) to place in different layouts.

Don't confuse a "block" (rendered fragment of HTML, aka "content") with a "node" or "entity" (a data object in the system, aka "content"). We still need to work out that terminology to make it less confusing. :-)

Who's going to lead the other two initiatives if WSCCI now = web services only?

sun, you've already done a lot of work on rallying support around the Unofficial Framework Initiative. I'd like to see that Initiative be made official, and if you're up to it, be put in charge of it.

Very clear write-up - much appreciated!

Thanks for the update from the meeting, and the summary of the WSCCI scope.

Looking forward to see Drupal level up two steps on the ass-kicking scale. This rocks.

Excellent presentation, which gives more sense to all these parts I have been trying to follow and work on.

One point of worry, though: you mention that "core components are actually fully designed after the HTTP specification. It does not try to account for the HTTP layer as an after-thought, the concept of a HTTP request and response is the fundamental core of it's architecture".

Which implies: will this heavy HTTP dependency not become a serious problem for CLI work, especially Drush, since we can not really map up a HTTP request to generic stdin/stdout/stderr content.

Very well written report. Thank you Sun for keeping us updated !

The parts of core that should be caring about being on the web should have the full HTTP spec available to them. The parts that should not but currently do need to be cleaned up so that they do not. So, eg, saving nodes needs to have no knowledge of whether it's a Drush command, Simpletest, an HTTP request from a Form, or an HTTP PUT request from a web service.

Only the last two are part of WSCCI's scope. The first two, volunteers welcome! :-)

In addition to ctools content plugins, also look at Boxes. It uses the ctools plugin architecture to reimagine the current block system (D6/7) but is implemented as you mentioned. Box types are classes and instances of these Boxes have their own configuration making it possible to have multiple instances of any box type. They are very simple to define and also provide for inline editing. In any case, there is exploratory but production worthy work already being done on some of these areas and we would love to help where appropriate.

Thanks Daniel. Absolutely brilliant writeup that I'm sure have not only sanitized the confusion for everyone that tried to keep tabs on it, but also made a much larger part of the community aware of the major benefits for Drupal this have.

Bring it on!

Thanks Sun for this overview, it's tremendously helpful and very well explained.

It is nice to see confusion blowing off!

It would be nice if this were diagramed out, with names of the key developers in charge of or working on each subsection, with sub-goals laid out for each subsection and with the specific steps to be accomplished.

This would put further organization in for the accomplishmet of D8 and progress could be tracked more easily. A little organization can go a long way.

In fairness the NIH response comes from being burned by adopting 3rd party code that was later had security issues after being abandoned upstream. I'm thinking of XTemplate and XML RPC code specifically. I think jQuery is a good counter example though. I think it's good to be hesitant when adopting technology, we'll be living with it for years.

Improving the communication about concrete change proposals for Drupal core is an issue that remains to be unresolved though.

I reference this post and look at a possible way to improve this communication in Exploring Drupal module interactions with pidgin UML (lin-clark.com/blog/exploring-drupal-module-interactions-pidgin-uml), I'd be interested to hear whether others think this could help as well.

OMG, what a great educational summary for those of us who are looking to implement websites with Drupal 8. I will re-read this article a few times and start doing some planning on how I am going to migrate my site to the new version.

  • This particular aussie slots this is the most effective that now there is for black jack and also the casino site sounds help you feel as if you are gambling inside of a actual casino.

Adaptability to changes is not only the survival rule in real world but also in virtual world. And Drupal has that criteria from its core.

As I'm late on the band wagon to D8 this was a fantastic overview, thanks!

If successfully implemented D8 will become more of the framework that I'm after. I've been lazy loading, and cutting pages into context regions for a while but on the top layer. Just sounds fantastic to be able to leverage some proper core functionality for this stuff and have a genuine API approach to building content pages! :-D

And there appears to be no trade off in having a popular CMS on top to maintain general usage and support... WIN.

Are these Symfony2 test figures done by TechEmpower valid?

http://www.techempower.com/benchmarks/#section=data-r3

How would that impact Drupal performance?

Am I reading this correct? Are they really that bad?
If that bad, should Symfony still be used then?