How to install a drupal.org sandbox module using composer

Sandbox modules don’t have a drupal.org/project/foo URL like full contrib modules, and therefore you can’t use composer require drupal/foo to add them.

If you have a Drupal 8 site using drupal-composer/drupal-project, here’s what to edit in composer.json – using a sandbox module of mine as an example.

  1. Within the repositories section
"drupal-wturrell/ckeditor_remove_elementspath": {
    "type": "package",
    "package": {
        "name": "drupal-wturrell/ckeditor_remove_elementspath",
        "version": "0.0.1",
        "type": "drupal-module",
        "source": {
            "url": "https://git.drupal.org/sandbox/wturrell/3018599.git",
            "type": "git",
            "reference": "8.x-1.x"
        }
    }
},

2.  In require (or run composer require with the name you’ve specified)

"drupal-wturrell/ckeditor_remove_elementspath": "^0.0.1",

You can choose any name you like, but drupal-username/module makes sense to me.

Your sandbox module doesn’t need a composer.json file of it’s own.

Blocking ‘Next up’ notification on Amazon Prime (for people who have respect for movie end credits)

updated 4 Jan 2021

Regardless of your ‘auto play’ setting, Amazon Prime Video displays a “Next Up” box bottom right, with movie artwork, on reaching the credit offset point; because, after spending perhaps more than two hours watching a film, who doesn’t want immediately want to start another?

Adblock rule (last tested Nov 2018):

www.amazon.co.uk##DIV[class="nextUpWrapper"]

Custom stylesheet (last tested Dec 2020):

(in Safari, go to Preferences > Advanced and configure the CSS file to use)

.webPlayerContainer .nextUpWrapper {
  display: none;
}

As you can’t specify domain names in a custom stylesheet, you have to narrow it down.

Acquia Dev Desktop 32-bit MySQL

Update: Jan 2020 – newer versions of Dev Desktop, such as the following, are fully 64-bit on macOS:

Version: 2
Built: Sep 18 2019 03:15:11
Control panel rev: a0d1c92
Stack rev: bb3d2d8

It’s worth noting the version of MySQL that comes installed with Acquia Dev Desktop 2 (5.6.41-84.1) is still 32-bit (unlike PHP which is now 64-bit).   Recently I was testing with a particularly large database and found it kept crashing with things like:

‘MySQL server has gone away’

Lost connection to MySQL server at ‘reading authorization packet’

Intensive operations such as Drupal-to-Drupal migration tended to trigger this.

I used MySQLTuner but the changes made little difference so then it occurred to me the 32-bit version simply wasn’t up to it, so I installed the latest 64-bit mariadb from homebrew instead and moved the relevant databases over to that (I have mariadb running on port 3306 and Dev Desktop’s MySQL on it’s default 33067).

So far it’s been stable.

See also: Acquia Dev Desktop known issues page

Troubleshooting Drush with Acquia Dev Desktop

Updated Tue 8 Jan 2019

Symptom: drush updb hangs

If drush updb is hanging, first, run it with the --verbose script to see exactly what’s going on – that will show you the MySQL connections. (It turned out in my case Drush was connecting to the wrong database).

Symptom: drush connects to wrong database

Checklist:

  • Try starting the terminal connection via DevDesktop (the icon top right) and see if that makes a difference
  • Double the database connection details in the appropriate loc_ file in ~/.acquia/DevDesktop/DrupalSettings/
  • Check your drush status output
  • Check what happens if you run drush sqlc
  • Try installing the latest Drush (composer require drush/drush) rather than relying on the version that comes with DevDesktop (at time of writing that’s 8.1.7, whereas Drush is now up to 9.5.2)

My situation: I have two MySQL installations running, the 32-bit DevDesktop supplied version, and a 64-bit mariadb (for a very large site where I was experiencing connection timeouts).  They are on different ports. I had a site where ordinary drush commands and drush sqlc connection went to the correct MySQL server but drush updb used the wrong one (confirmed using the --verbose option – in fact it was connecting to a database with a different name entirely.

I couldn’t figure out what was going on, so I upgraded from Drush 8 to 9 and that fixed it immediately.

Symptom: large Drupal 7 site where cache is extremely slow or fails to clear…

drush cc all

…and eventually PHP runs out of memory (“cannot allocate”).

Check Drush is using the correct version of PHP, it may not be – use drush status or, more simply: drush php

To fix:

$ which drush
/Applications/DevDesktop/tools/drush

$ vim /Applications/DevDesktop/tools/drush

# edit the PHP version: [ -z "$PHP_ID" ] && PHP_ID=php7_1

See also: Acquia Dev Desktop known issues page

Drupal debugging: Invalid argument supplied for foreach() EntityFieldManager.php:490

You might see this error occasionally when installing/uninstall modules, running db updates etc.

The code in EntityFieldManager.php loops through an array of entities – each of which is a bundle, for example the user entity contains all the user fields you may have created.

The trick for debugging is to dump the contents of $bundle_field_map and look for something wrong – e.g. my issue was an empty (NULL) font entity, which had been left behind after attempting to remove the fontyourface module.

Backup the DB (obviously) then look in the key_value table for records where collection = entity.definitions.bundle_field_map – for me it was a case of deleting name = font, value = N; and the problem went away.

(So an alternative to writing code is to look at the DB table first.)

Add a PHP extension to Acquia Dev Desktop (example – memcached)

(Updated 16 May 2020 – after testing with latest DevDesktop 2, built May 13 2020, on macOS High Sierra 10.13.6)

These instructions now refer to PHP 7.3 (which is recommended for Drupal 8, and required for the forthcoming Drupal 9).  There is one extra requirement for 7.3 – the entry pcre.jit = 0 in your php.ini, to avoid a JIT Compilation Failed error message.  (That’s specific to the memcached extension.)

Other than that, you can change all 3 to 2 in all the instructions if you still want 7.2.

Let’s assume you have the latest Acquia Dev Desktop, which comes with PHP (switchable between 7.0, 7.1, 7.2 and 7.3), and you want to install the PHP memcached extension – so you can use Drupal’s memcache module to make your local development a bit faster (and test for any memcached issues on production).

Memcached v3.1.0 added support for PHP 7.3.

But first…


Why still use Dev Desktop in 2020? 

I still use DevDesktop, but only for quickly testing things on a clean Drupal site, and some legacy sites.  Otherwise, nowadays I recommend Laravel Valet (you can use it for loads of frameworks/content management systems besides Laravel. It takes care of almost everything, except setting up your MySQL database.)

I don’t recommend Docker. Running PHP and MySQL directly is blisteringly fast compared to Docker for Mac – usually several seconds every page load, and with ADD you can instantly PHP restarts when changing a config setting (e.g. enabling/disabling Xdebug) compared to having to wait a minute or more for a Lando container to rebuild (Docker containers are immutable; you have to rebuild the container to make a significant PHP.ini change like adding or removing an extension – you can’t just restart the service). Composer is faster (still a bit annoying, of course, but consider what it’s doing in the background). Drush is faster, including rebuilding the cache. Over a day, every day, this all really adds up.  For those of you using laptops, your battery life will be extended too. (Docker can also use quite a bit of CPU when idle).

Running PHP on the host means you’re directly accessing the file system and not having to jump through all the hoops you do with Docker and Vagrant – either slow file accesses, or having to wait for changes to sync.  This is especially true with Drupal given all the caching it does.  

It’s very unpopular to criticise Docker (which I’m not, exactly – more the hype and the rush to use it on a platform it wasn’t designed for, despite the slowness issues many people experience.)  There’s a trade off between performance and configuration – the assumption seems to be you should go for whatever requires the least configuration, but is it really that hard to install the correct version of PHP or MySQL version (how about making a note of what your site runs in your Git repository?)  I’d rather spend a bit longer up front getting the configuration right and have the performance benefit each day after.  Also, if I have a choice of what to learn, I think knowledge of PHP extensions and configuration – given I’m a PHP developer – is more useful than learning loads about containers.

Comments welcome by email. (I use Ansible heavily, by the way, so I’m not opposed to automation.)  


Install steps:

You need to repeat this once for each version of PHP you want the extension available in, e.g. 7.1 or 7.2.  The example below is for 7.3.

Download the extension from here –  https://pecl.php.net/package/memcached/3.1.5 – or you can use the quicker get latest link: pecl.php.net/get/memcached

extract the tar file into its own directory – it’s fine if it’s just in your downloads folder (tar -xvf)

Then, change to that directory and run:

/Applications/DevDesktop/php7_3_x64/bin/phpize

This next line is the important bit – to make it properly compatible with the PHP version DevDesktop is using.  Without this you may get an error like:

Symbol not found: _compiler_globals
Expected in: flat namespace

./configure --with-php-config=/Applications/DevDesktop/php7_3_x64/bin/php-config

Tip: modules that don’t successfully load don’t show up in phpinfo(), so a quick way to see what the error was when PHP attempted to load  them is to do php -v (it’ll show any warnings above the version info) or php -m to get a list of all the extensions loaded and verify yours is OK.

make
make install

Important: if you need to rerun any of these steps, wipe the directory you extracted first and start afresh.

You should see:

Installing shared extensions: /Applications/DevDesktop/php7_3_x64/ext/

There will now be a memcached.so file in that directory. If it hadn’t picked the config details up, it would have written to a different directory, e.g. your default homebrew one.

Add the memcached.so file to your php.ini – you can verify where that is via  Preferences > Config in DevDesktop, but it would normally be:

/Applications/DevDesktop/php7_3_x64/bin/php.ini

extension_dir = "/Applications/DevDesktop/php7_2_x64/ext"

extension = memcached.so
pcre.jit = 0

This doesn’t usually get overwritten when upgrading DevDesktop, but you’ll need to go through this process each time you switch to a newer PHP branch.  It will take you a few minutes at most.

NB: I rather thought PECL had an equivalent install command line switch to load the correct php-config, so you could just run pecl install command to save time, but apparently not? Let me know if I’m wrong.

Useful reading elsewhere:

Howto: Drupal 8 and memcached on Lando

Update: Only a small part of this is related to Lando, it’s mostly about configuring Drupal with Memcached, but I would just note that since I wrote it in 2018, I no longer use Lando or (or anything involving Docker / containers). Instead, if you’re a macOS user, I’d recommend Laravel Valet – which is about as fast, simple and lightweight as you can get. It handles many frameworks / content management systems, takes care of your https:// support and domain names, and makes it easy to switch between PHP versions. You need to install MySQL or MariaDB (I use the latter) yourself, which is easy using Homebrew.


    1. Make sure phpinfo(); (available at /admin/reports/status/php in Drupal for administrators) contains a memcached section – by default Lando will install the memcached PECL package.
    2. Install and enable memcache Drupal module.
    3. configure settings.php:
      $settings['memcache']['servers'] = ['cache:11211' => 'default'];
      $settings['memcache']['bins'] = ['default' => 'default'];
      $settings['memcache']['key_prefix'] = 'mysitename_';
      
      $settings['cache']['default'] = 'cache.backend.memcache';

      Note the hostname for external connections (port 11222) is localhost, but internal connections have a different hostnamecache – to verify, run lando info

      (give it the correct sitename – this is especially important if you ever use memcached for two or more sites on the same server, to avoid conflicts.

      Also, note memcached has no security so if an application knows or can guess another applications’ prefix it can read all the data.

    4. Truncate all MySQL tables beginning with cache (Drupal bootstraps from the database by default and otherwise may continue to use them)
    5. Run drush cr
    6. You can should now work.

Verify connection and performance (e.g. hits/misses and memory used) at /admin/reports/memcache

Way to test an internal connection from PHP in a container, if you can’t install what you need:

php -a
> $socket = fsockopen('cache', "11211", $errno, $errstr);

If you don’t see any errors, it has connected.

Troubleshooting PhpStorm – entire composer.json file shows as invalid

If you’re running the latest PhpStorm (2018.2.2 at time of writing) you might have a composer.json file that’s turned entirely yellow recently, even though it’s valid.

This happens when your composer.json has a “description” key and is due to this bug.

Workarounds:

  • Disable the inspection on the file
  • or remove the “description” temporarily

Drupal – troubleshooting config synchronisation broken CSS when updating theme colours/appearance

Symptom: After deploying configuration changes to production, including a change to the theme (e.g. editing colours, colour scheme or logo on /admin/appearance/settings/bartik), you get a mostly white page because the colours are all missing. Yet when you visit that admin URL everything looks correct, and as soon as you click ‘Save Configuration’ the site is normal again.

What’s going on?

This happens if you’re using a standard Drupal .gitignore file, which typically has the following:

# Ignore paths that contain user-generated content.
sites/*/files
sites/*/private

If you export your configuration and look at color.theme.bartik.yml – you’ll find this at the bottom:

stylesheets:
  - 'public://color/bartik-7a1420bf/colors.css'
files:
  - 'public://color/bartik-7a1420bf/logo.svg'
  - 'public://color/bartik-7a1420bf/colors.css'

Every time the colours are updated a new stylesheet is created in /sites/default/files/color/bartik-[hash], but .gitignore is ignoring these files, so even if you perform a configuration sync correctly, production won’t have the CSS file it needs.

The solution is to add this to .gitignore:

sites/*/files/
!sites/*/files/color/
!sites/*/files/color/*

.gitignore syntax for inverting subdirectories is pretty confusing.  You might need to use git add –force If you see references to “double-globs”, that means **, and note it doesn’t work on all platforms (e.g. Mac).

Drupal – troubleshooting missing blocks

It’s possible to hide a block from the entire site by accident, because of some counter-intuitive behaviour in the block Visibility settings.

By default, when placing a block there are settings for Content Types, Pages and Roles.  Usually if you leave these blank (i.e. don’t select any checkboxes), the relevant tab has a summary label saying ‘No restrictions’ and they are ignored.

The Pages tab behaves a bit differently.  It has Show and Hide radio buttons and a textarea to input URLs.   If you leave the button set to show (the default) and the textarea is blank, the block is visible on all pages (barring any other restrictions such as the content type).

The danger is if you change it to Hide, and specify one or more pages to exclude.  If in future you decide to remove those so the list is empty again, you must set the radio button back to ‘Show’, otherwise it’ll be hidden from all pages, even though the tab will still say ‘No restrictions’.

Specifically the combination in this screenshot hides the block everywhere, as if it were disabled completed:

This block will be hidden, even though it says ‘Not restricted’

So if you can’t figure out why a block isn’t showing, manually click the Pages tab and check the full settings.  (This can be particularly confusing if you’re testing View blocks with complex contextual filters and they’re working in the Views UI preview, but not on the page.)