mattgrayisok

Craft is great, but historically, working with it long-term in a multi-environment situation required some less than optimal processes when updating a site's data architecture. Project Config attempts to address this issue and largely seems to succeed.

What's the problem?

It all boils down to the fact that Craft stores its user generated content and the configuration which governs that content in an intertwined set of database tables. It isn't possible to update one or the other independently using database exports and imports.

This poses a problem when developing locally but deploying to a live production environment: how do we put content structure changes live that we have been working on locally alongside out template changes?

With Craft there are a few options. The most popular that I've come across are:

• Designate one environment (production, staging or local) as the master environment. All content and schema changes take place here and other environments pull DB exports from here.
• Perform schema changes locally, keep notes, then reapply these changes to each environment via Craft's control panel as the updated template files are deployed to them.
• Manually create migration files for each set of database updates which can be executed at the same time as new template files are pushed up to each environment.
• Use a plugin such as Migration Manager to create migrations on your behalf.

Each option has pros and cons. The two I tend to use are keep notes for small projects and use a plugin for larger projects, however both can still succumb to human error: notes need to be kept accurately and applied correctly, the plugin requires manual intervention in order to decide which elements to export, the order of migration execution and requires deletes to be handled manually.

It's also important to understand that Craft presents users with a data schema abstraction compared to what's actually stored in the database. Craft migrations perform changes to this abstraction layer and Craft then converts those to appropriate database operations. This tends to make Craft migrations quite verbose. Compare the following Craft migration for adding one or more fields to the User object:

// Create the field group
$groupModel = new FieldGroupModel();
$groupModel->name = 'CustomGroupName';
craft()->fields->saveGroup($groupModel);

// I haven't yet found a way to get the group ID without looping through all of the groups. saveGroup() returns a boolean and doesn't appear to update the id attribute of the original model instance.
$groups = craft()->fields->getAllGroups();
foreach($groups as $group) {
    if($group->name != 'CustomGroupName') {
        continue;
    }
    $groupModel = $group;
}

// Create the tab
$tabModel = new FieldLayoutTabModel();
// Get the desired layout (in this case, the User layout) and use the tab's setLayout to link the two
$layout = craft()->fields->getLayoutByType(ElementType::User);
$tabModel->setLayout($layout);
$tabModel->name = 'CustomTabName';


// Create the fields. I have a protected class member that contains the fields in a 'fieldName' => 'FieldName' format.
foreach($this->fields as $field => $name) {
    $fieldModel = new FieldModel();
    $fieldModel->groupId = $groupModel->id;
    $fieldModel->name = Craft::t($name);
    $fieldModel->handle = $field;
    $fieldModel->translatable = false;
    $fieldModel->type = 'PlainText';

    craft()->fields->saveField($fieldModel);
}

// Get all of the group's fields. This could also be done by saving an array of the fields in the loop above, but this seems a bit cleaner to me, and since we already have the group object, saves us an object/variable declaration.
$fields = $groupModel->getFields();

// Set the tab's fields. This gives us a custom grouping in the layout editor.
$tabModel->setFields($fields);

// Get the existing tabs (and, by extension, fields), since we don't want to nuke the layout, just add to it. I couldn't find any sort of "addTab" type function, so this is how we have to do it.
$layoutTabs = $layout->getTabs();

// getTabs() returns an array, so simply add our new tab to the array.
$layoutTabs[] = $tabModel;

// Now, set the tabs back in the layout.
$layout->setTabs($layoutTabs);

Stolen from this stack overflow answer.

To what is essentially the equivalent in Laravel:

Schema::table('users', function (Blueprint $table) {
    foreach ($col in $this->columns) {
        $table->{$col['type']}($col['name']);
    }
});

Project Config

Craft CMS 3.1 attempts to address this issue by storing a representation of its data schema abstraction in a file which can be version controlled and moved between environments without destroying any user generated data. The file is created and updated by Craft whenever a change to its data schema abstraction occurs.

In order to achieve this the Craft dev team have defined a codifying format for their data structures and selected yaml as the standard to present it.

With a fresh Craft install and upon enabling the 'useProjectConfigFile' config option, Craft will generate a file named 'project.yaml' with the following contents:

dateModified: 1536946466
email:
  fromEmail: matt@mattgrayisok.com
  fromName: test
  transportType: craft\mail\transportadapters\Sendmail
fieldGroups:
  b2a1eeb4-8437-4f32-bd0d-1c6f90183164:
    name: Common
schemaVersion: 3.1.0
siteGroups:
  098acf96-e24e-476c-8a59-ef5e1ca97ecd:
    name: test
sites:
  03d7286b-b1c6-4e1b-ab90-90ecd5735dc7:
    baseUrl: '@web/'
    handle: default
    hasUrls: true
    language: en-GB
    name: test
    primary: true
    siteGroup: 098acf96-e24e-476c-8a59-ef5e1ca97ecd
    sortOrder: 1

The most important thing to note is that Craft is storing all of its different structural object types in here, along with their UUIDs.

Looking at the code we can see that whenever a control panel page is loaded Craft is going to first check if the project.yaml file has a modified date greater than the date it has cached for this file. If so it'll compare a cached copy of the entire config to the new one and create three sets of updates: 'newItems', 'removedItems' and 'changedItems'.

These sets of changes are then fired off as events for another part of the codebase to actually process and make the appropriate changes to the database.

This file also acts as a proxy for any intended changes to the schema or multi-environment state. This means that before Craft and (well-behaved) plugins make any schema changes they will need to record their intention to make a change with the ProjectConfig service. This gives the ProjectConfig service a chance to update the project.yaml file appropriately before firing off an event which will be caught by whichever class made the original change request. Upon receipt of the event the requester can then actually perform the originally intended update action.

Why all the theatrics with update requests and subsequent event catching as opposed to just telling Project Config that you already made a change?

It gives Craft a mechanism to re-play the events in each environment based on the contents of project.yaml, ensuring state can be transferred in a very generic way by Craft, modules and plugins.

Thoughts

• project.yaml contains a dateModified attribute which doesn't seem to be used for anything at the moment. Would it be safer to make use of this when figuring out the new-ness of the file rather than the filesystem modified date? Tools like rsync might set that to unexpected values.
• This should work really well for uni-directional deployment flows. Not so well if updates are taking place in multiple environments as it'll be important to get all the uuids of elements to match up everywhere. It'd be nice if there was an setting to prevent any schema changes in specific environments - maybe there is and I haven't seen it yet.
• It'll be interesting to see how well merge conflicts work. Git diffs on project.yaml might need managing manually and could potentially be HUGE for any non-trivial schema update merges.
• As well as storing schema state there's no reason why this can't be used for other application and plugin state. I think that's what the intention is. Being able to sync plugin state across environments will be useful but plugin developers might need to be careful not to store state that needs to change per-environment in this way or at least allow it to be overridden with ENV vars.
• Having a project.yaml with your common data structures pre-defined would be a good time saver when starting a new project. Currently the installation scripts for Craft don't play well with project.yaml though. (I tried it and it essentially ignores everything in there and then adds a load of duplicate entries.)
• Now that Craft schemas can be codified I'm sure it won't be long before a webapp exists which allows you to build and export your schema. Perhaps that could also have common patterns: check boxes for a standard blog, band website, real estate listing site... Just add theme... 🤔

I'm looking forward to seeing the changes in development practices around Craft in response to Project Config. It will certainly make my life easier when deploying sites to staging and production as it fits perfectly into my preferred development workflow.