How to evaluate a CMS: trial it before you commit

A trial is the only point at which you can learn what a CMS is like to use, before you have moved an archive into it and trained a team. Vendor pages describe features; a trial shows whether your editors can use them on a deadline. This guide gives a method you can run in a fortnight, with a script that makes results comparable between products.

It describes how to evaluate, not what Word & Press found. Where we mention vendor behaviour, the source is recorded on this page.

What forms can a trial take?

There are three, and most evaluations use more than one.

Some vendors make the paid editions trialable. Craft CMS’s pricing page, for example, says Team and Pro can be trialled on non-public domains before purchase. Ghost’s repository says it can be self-hosted using its CLI tool, and Drupal’s documentation points to DDEV as the community-recommended way to build a local development environment. Check each vendor’s own documentation for the current method.

How do I set up a fair test?

Give every candidate the same starting conditions.

1. Install or sign up using the vendor’s documented method, and note how long it takes and what you had to look up. Our self-hosting guide covers what a server needs.
2. Use a test domain that is not indexable by search engines, so draft content never appears publicly.
3. Apply the same theme approach to every system, or none, so you judge the editor and not the design.
4. Import or recreate realistic content: at least 30 articles in several categories, with images, captions, multiple authors, tags, an embedded video or social post, and a few very long pieces.
5. Create the roles you need: writer, editor, administrator and, if relevant, a contributor with limited access.

What should the editor task script include?

Write the tasks down and have each tester do them in the same order on every system. These are the ones that expose differences for publishers.

1. Write a new article with headline, body, two images with alt text and captions, and a category.
2. Save it as a draft, then submit it for review as a writer.
3. Approve and schedule it as an editor, for a time later in the day.
4. Edit a published article to correct a factual error, then find the earlier version.
5. Change the URL slug and confirm the old address redirects.
6. Add an author with a profile page and attach them to three articles.
7. Upload twenty images at once and note how they are resized and named.
8. Unpublish an article, then restore it.
9. Create a landing page or section front with a curated list of stories.
10. Export the content and look at what you get.

Time each task and record the number of clicks, any confusing step, and any moment a tester asked for help. Ask testers for a rating from one to five and a sentence of explanation, not a long opinion.

What should I measure?

• Editing speed and errors: time for the standard tasks, and mistakes such as lost drafts.
• Workflow fit: could the system model your real approval process without workarounds?
• Search controls: can editors set titles, descriptions and canonical URLs, and does the output contain the fields search engines read? Google lists headline, image, datePublished, author and dateModified as recommended Article properties, so inspect the page source of a test article.
• Page speed on your content: load an article page and compare, on the same connection, how heavy it is and how quickly it appears.
• Admin speed: whether the editor stays responsive with a few thousand items.
• Extensions needed: list every plugin you needed to add to pass your requirements, with its price and last update.
• Documentation and support: how quickly you found answers, and whether the vendor replied to a question sent during the trial.
• Exit route: export content, media and redirects, and check you can read the result.

Compare the findings against the feature checklist rather than your first impression.

What are the common mistakes?

Trialling with lorem ipsum hides how the system handles long text and awkward images. Letting only the developer test hides editor frustration. Skipping the export test hides lock-in. And choosing during the trial’s novelty period overweights how pleasant the interface feels in week one. Revisit your scorecard after a few days of routine work, when small irritations become obvious.

How do I involve the wider team?

Hold a short review at the end of week one and again at the end of the trial. Ask each tester to name the one task that annoyed them most and the one that impressed them, then check whether the same issues appear across several people. A problem only one person hits may be a training gap; a problem three people hit is a product gap. Also ask whoever handles legal review or corrections to run their own part of the workflow, since their needs are easy to overlook until launch.

How do I decide?

Total the scorecard, then look at the three-year cost and the licence. If two systems are close, prefer the one with clearer exit options and the smaller number of extensions you depend on. Document the decision, including what you tested and what you did not, so you can explain it to colleagues later.