Drupal debugging: Invalid argument supplied for foreach() EntityFieldManager.php:490

You might see this error occasionally when installing/uninstall modules, running db updates etc.

The code in EntityFieldManager.php loops through an array of entities – each of which is a bundle, for example the user entity contains all the user fields you may have created.

The trick for debugging is to dump the contents of $bundle_field_map and look for something wrong – e.g. my issue was an empty (NULL) font entity, which had been left behind after attempting to remove the fontyourface module.

Backup the DB (obviously) then look in the key_value table for records where collection = entity.definitions.bundle_field_map – for me it was a case of deleting name = font, value = N; and the problem went away.

(So an alternative to writing code is to look at the DB table first.)

Add a PHP extension to Acquia Dev Desktop

(quick draft, will tidy up later)

Let’s assume you have the latest Acquia Dev Desktop, which comes with various PHP versions including 7.1, and you want to install the PHP memcached extension so you can use Drupal’s memcache module to make your local development a bit faster (and test for any memcached issues on production).  But first…


Why still use Dev Desktop in 2018? Running PHP and MySQL directly is blisteringly fast compared to Docker for Mac – usually several seconds every page load, and with ADD you can instantly PHP restarts when changing a config setting (e.g. enabling/disabling Xdebug) compared to having to wait a minute or more for a Lando container to rebuild (Docker containers are immutable; you have to rebuild the container to make a significant PHP.ini change like adding or removing an extension – you can’t just restart the service). Composer is faster (still a bit annoying, of course, but consider what it’s doing in the background). Drush is faster, including rebuilding the cache. Over a day, every day, this all really adds up.  For those of you using laptops, your battery life will be extended too. (Docker can also use quite a bit of CPU when idle).

Running PHP on the host means you’re directly accessing the file system and not having to jump through all the hoops you do with Docker and Vagrant – either slow file accesses, or having to wait for changes to sync.  This is especially true with Drupal given all the caching it does.  

It’s very unpopular to criticise Docker (which I’m not, exactly – more the hype and the rush to use it on a platform it wasn’t designed for, despite the slowness issues many people experience.)  There’s a trade off between performance and configuration – the assumption seems to be you should go for whatever requires the least configuration, but is it really that hard to install the correct version of PHP or MySQL version (how about making a note of what your site runs in your Git repository?)  I’d rather spend a bit longer up front getting the configuration right and have the performance benefit each day after.  Also, if I have a choice of what to learn, I think knowledge of PHP extensions and configuration – given I’m a PHP developer – is more useful than learning loads about containers.

Comments welcome by email. (I use Ansible heavily, by the way, so I’m not opposed to automation.)  


Install steps:

Download the extension from here –  https://pecl.php.net/package/memcached/3.0.4 – or you can use the quicker get latest link: pecl.php.net/get/memcached

extract the tar file into its own directory (tar -xvf)

Then run:

/Applications/DevDesktop/php7_1/bin/phpize

This next line is the important bit – to make it properly compatible with the PHP version DevDesktop is using.  Without this you may get an error like:

Symbol not found: _compiler_globals
Expected in: flat namespace

./configure --with-php-config=/Applications/DevDesktop/php7_1/bin/php-config

Tip: modules that don’t successfully load don’t show up in phpinfo(), so a quick way to see what the error was when PHP attempted to load  them is to do php -v (it’ll show any warnings above the version info) or php -m to get a list of all the extensions loaded and verify yours is OK.

make
make install

Important: if you need to rerun any of these steps, wipe the directory you extracted first and start afresh.

You should see:

Installing shared extensions: /Applications/DevDesktop/php7_1/ext/

There will now be a memcached.so file in that directory. If it hadn’t picked the config details up, it would have written to a different directory, e.g. your default homebrew one.

Add the memcached.so file to your php.ini – you can verify where that is via  Preferences > Config in DevDesktop, but it would normally be:

/Applications/DevDesktop/php7_1/bin/php.ini

You can save a backup for when you upgrade DevDesktop and it gets overwritten, but if the PHP version has changed you should compile it again.

NB: I rather thought PECL had an equivalent install command line switch to load the correct php-config, so you could just run pecl install command to save time, but apparently not? Let me know if I’m wrong.

Useful reading elsewhere:

Howto: Drupal 8 and memcached on Lando

    1. Make sure phpinfo(); (available at /admin/reports/status/php in Drupal for administrators) contains a memcached section – by default Lando will install the memcached PECL package.
    2. Install and enable memcache Drupal module.
    3. configure settings.php:
      $settings['memcache']['servers'] = ['cache:11211' => 'default'];
      $settings['memcache']['bins'] = ['default' => 'default'];
      $settings['memcache']['key_prefix'] = 'mysitename_';
      
      $settings['cache']['default'] = 'cache.backend.memcache';

      Note the hostname for external connections (port 11222) is localhost, but internal connections have a different hostnamecache – to verify, run lando info

      (give it the correct sitename – this is especially important if you ever use memcached for two or more sites on the same server, to avoid conflicts.

      Also, note memcached has no security so if an application knows or can guess another applications’ prefix it can read all the data.

    4. Truncate all MySQL tables beginning with cache (Drupal bootstraps from the database by default and otherwise may continue to use them)
    5. Run drush cr
    6. You can should now work.

Verify connection and performance (e.g. hits/misses and memory used) at /admin/reports/memcache

Way to test an internal connection from PHP in a container, if you can’t install what you need:

php -a
> $socket = fsockopen('cache', "11211", $errno, $errstr);

If you don’t see any errors, it has connected.

Troubleshooting PhpStorm – entire composer.json file shows as invalid

If you’re running the latest PhpStorm (2018.2.2 at time of writing) you might have a composer.json file that’s turned entirely yellow recently, even though it’s valid.

This happens when your composer.json has a “description” key and is due to this bug.

Workarounds:

  • Disable the inspection on the file
  • or remove the “description” temporarily

Drupal – troubleshooting config synchronisation broken CSS when updating theme colours/appearance

Symptom: After deploying configuration changes to production, including a change to the theme (e.g. editing colours, colour scheme or logo on /admin/appearance/settings/bartik), you get a mostly white page because the colours are all missing. Yet when you visit that admin URL everything looks correct, and as soon as you click ‘Save Configuration’ the site is normal again.

What’s going on?

This happens if you’re using a standard Drupal .gitignore file, which typically has the following:

# Ignore paths that contain user-generated content.
sites/*/files
sites/*/private

If you export your configuration and look at color.theme.bartik.yml – you’ll find this at the bottom:

stylesheets:
  - 'public://color/bartik-7a1420bf/colors.css'
files:
  - 'public://color/bartik-7a1420bf/logo.svg'
  - 'public://color/bartik-7a1420bf/colors.css'

Every time the colours are updated a new stylesheet is created in /sites/default/files/color/bartik-[hash], but .gitignore is ignoring these files, so even if you perform a configuration sync correctly, production won’t have the CSS file it needs.

The solution is to add this to .gitignore:

sites/*/files/
!sites/*/files/color/
!sites/*/files/color/*

.gitignore syntax for inverting subdirectories is pretty confusing.  You might need to use git add –force If you see references to “double-globs”, that means **, and note it doesn’t work on all platforms (e.g. Mac).

Drupal – troubleshooting missing blocks

It’s possible to hide a block from the entire site by accident, because of some counter-intuitive behaviour in the block Visibility settings.

By default, when placing a block there are settings for Content Types, Pages and Roles.  Usually if you leave these blank (i.e. don’t select any checkboxes), the relevant tab has a summary label saying ‘No restrictions’ and they are ignored.

The Pages tab behaves a bit differently.  It has Show and Hide radio buttons and a textarea to input URLs.   If you leave the button set to show (the default) and the textarea is blank, the block is visible on all pages (barring any other restrictions such as the content type).

The danger is if you change it to Hide, and specify one or more pages to exclude.  If in future you decide to remove those so the list is empty again, you must set the radio button back to ‘Show’, otherwise it’ll be hidden from all pages, even though the tab will still say ‘No restrictions’.

Specifically the combination in this screenshot hides the block everywhere, as if it were disabled completed:

This block will be hidden, even though it says ‘Not restricted’

So if you can’t figure out why a block isn’t showing, manually click the Pages tab and check the full settings.  (This can be particularly confusing if you’re testing View blocks with complex contextual filters and they’re working in the Views UI preview, but not on the page.)

 

 

Ansible 2.5 change – troubleshooting include_tasks and tags

If you’re upgrading to Ansible 2.5 (released Mar 23), make sure you read this.

Specifically, until now,  if you had tagged an include_tasks item, all the tasks in the file you were including would inherit the tag (or any other attributes). But now:

“attributes applied to an include_* task will not be inherited by the tasks within.”

Symptom: when running a playbook with –tags=bar, you will see a series of lines like this, one for each item in your loop…

included: /Users/wt/Dropbox/ansible/foo/bar.yml for myserver
included: /Users/wt/Dropbox/ansible/foo/bar.yml for myserver
included: /Users/wt/Dropbox/ansible/foo/bar.yml for myserver

…but none of the tasks in bar.yml actually execute.

To fix, either you manually tag everything within (ugly) OR you can use a block (elegant).

Ansible troubleshooting tips

Last updated Wed 20 Dec 2017

Setting file/directory permissions

When using the File module, the documentation warnings about missing off the leading zero for mode, but if you’re using setuid or setgid, you also need to surround the octal number with quotes, otherwise you’ll get unpredictable results.  Write it like this:

- name: Create docroot ({{ docroot }})
  file:
    path: "{{ docroot }}"
    state: directory
    owner: www-data
    group: www-data
    # the leading 2 means set group ID
    mode: "2775"
  become: true

Debugging failing Git connections

When using the Git module, if the command hangs without an error, use the --verbosecommand line switch when running the playbook, and set a low timeout for the task using async so it finishes quickly.  It might be you are connecting the wrong way (e.g. via https when you should be using SSH) or that the server’s SSL certificate needs to be approved.

More about async and poll settings

Variable expansion syntax

You have a variable called {{ current_site }}, which matches a key in {{ websites }}.  How do you write that? i.e. what’s the YML/Ansible equivalent of PHP’s {$foo}?  AnswerL use square brackets, not extra {{ }}

 

{{ websites[current_site].git_branch }}

Passwords

Note there’s a difference between --ask-pass (for standard SSH) and --ask-become-pass(for sudo) and also that --ask-sudo-pass is deprecated.

SSH passwords in the user module need to be supplied as a hash.  MySQL DB passwords are plaintext.

Nested loops

Having everything called {{ item }} gets ambiguous/messy and you’ll get a warning.  Use loop_control and loop_var to give one list of with_items another name.  Ansible docs

Deprecation warnings

(These aren’t always very well documented.)

[defaults]hostfile option, The key is misleading as it can also be a list of hosts, a directory or a list of paths . This feature will
be removed in version 2.8.

Fix is to edit ansible.cfg and change the following (ie. hostfile -> inventory):

[default]
#hostfile = ./inventory/common/allhosts
inventory = ./inventory/common/allhosts

Then the error will go away.

See also: https://github.com/geerlingguy/drupal-vm/issues/1553

Find where you ansible.cfg file is

run "ansible --version"

ansible 2.4.2.0
 config file = /Users/foo/.ansible.cfg
 [...]

Note how on a Mac it may begin with a period (.)

AWS (Amazon Web Services) modules

Limitations:

  • note that although there is now an s3_sync module, it currently (Ansible v 2.4.2.0) only supports push mode (i.e. uploading files TO amazon, not downloading from it to a local device).   However, you can easily use the AWS CLI command in either direction – and because it’s a sync command, it’s idempotent.
aws s3 sync [--delete] s3://mybucket/path/ /path/on/local/server/

UK mobile networks factsheet

This is a FAQ I’m gradually assembling for technically minded people interested in specific network features and the basics of how mobile number portability actually works.

last updated Sun 19 November 2017

Activating new SIM on the network (Three UK):

This one is specific to Three because I haven’t done it for any other networks recently, although a common piece of advice seems to be 2 hours.

So what we’re talking about here is the time after inserting a brand new SIM (in this case PAYG) into an unlocked phone, before it changes from ‘No Service’ to ‘3’, i.e. until you get an indication the registration process has started.   We’re not talking about porting a number (more on that later).

On the first SIM I used, this was instant and I didn’t even need to reboot the phone, on the second it took almost an hour despite numerous toggling of airplane mode, checking the reception was OK, power cycling the phone and so on.  The advice I was given was to leave it in and powered on for four hours.

Until that point, dialling the new number will give “This number has not been recognised”, because it’s not actually on the network yet.

Also, I found that requesting a port before the new SIM was registered meant that it was lost/ignored/failed (despite supplying the number of the SIM) and I needed to request the port again.

Other thing worth noting -an iPhone will prompt you to install new carrier settings as soon as a new SIM is inserted in the phone, before it’s actually registered on the network.  It does this over WiFi.  You can force it to check by going to Settings > General > About.

UK mobile number porting (MNP):

Brief facts:

  • in the UK it’s still pretty manual (GUI software provided by Syniverse to handle generation/validation of PACs, and files exchanged daily over SFTP between the various MNOs to update the databases).  They pay each other a few pence to handle the porting and several thousand pounds for access to the system.
  • You can easily have three (or more) parties involved – the Original network operator, the Donor Service provider and the Recipient Service provider  (this is because the number still has to be routed via the MNO that originally owned it).  Also either the donor or recipient could be an  MNVO – a service provider that doesn’t own their own network.
  • Timings: Have a look at the chart (usual warning – could now be out of date) on page 13 of the Setup Guide for an indication of what order everything happens in.  Note how the new operator is meant to activate the number on their network before 11am, the original network operator (who owns that block of numbers) reroutes it between 11am and 3pm, and the old operator deactivates it sometime after 4pm (having received confirmation).
  • 1 day to transfer, but only Monday to Fridays, excluding bank holidays, because of the carefully choreographed sequence described above.
  • Voice and text messages are ported separately, because there are separate interconnects between operators for voice and SMS.
  • As should be apparent, records need to get updated in multiple places,  which is why, for example, you can get problems involving failing incoming calls from certain networks.

Why isn’t there just a single database?

Well there is, for issuing and checking PACs.  But the source and destination network need to enable/disable the number on their network and the original owner needs to ensure it’s routed correctly.

 

Further reading:

MNP Operator Steering Group website (start with the MNP Setup Guide)

Porting numbers to Three UK:

On the day of the transfer, you should get a text at around 8am to the Three SIM with your temporary number saying:

From Three: Your phone number 440123456789 should be transferred to us by end of the day. Just switch your phone off and on to confirm the transfer. Thanks.

Wi-Fi calling:

Three and EE support voice and SMS (i.e. you can send/receive texts when only connected to wifi).
O2 and Vodafone only support voice (also: anecdotal observation, couldn’t ever get O2 Wi-Fi calling to login, so gave up on it – Three worked instantly on same network).

Observations:

WiFi calling does add a slight amount of latency to the audio.
Conversely, call setup time (from dialling a number to first ring) is much, much faster.
Although holding the IPSec connection open does use some energy, but I’ve not seen a noticeable effect on iPhone battery life (tested from iOS 10.3.3 to 11.1.1)
WiFi calling does timeout/reconnect periodically (rather less so since iOS 11, in my experience), but I’m yet to ever lose connection during a call.

iPhone tips:

Check status. As well as Three WiFi Call in the status bar, you can check Voice/SMS support via Settings > General > About – click on Carrier, it will toggle between Three 28.4 (the Carrier settings version) and IMS Status: Voice & SMS

If you’re someone who keeps their phone in airplane mode but with wifi turned on most of the time, to save battery/reduce radiation etc., note that when you power-cycle the phone wifi calling won’t work until the Sim card has been unlocked – i.e. you’ll need to turn airplane mode off temporarily (but you can turn it straight off again, you don’t need to wait for the network connection to be established).

Technical info:

On iPhone, WiFi calling uses UDP ports 500 and 4500.

Visual Voicemail:

This isn’t supported by Three, but is by EE and O2.
List of which networks support which Apple features.

How it works on O2:

Everything, including the audio files, is sent/received over mobile data – i.e. if you have a bad connection it will be slow or you will see a message saying Visual Voicemail is not available. (At that point you can still ring the voicemail number – in o2’s case 901).  You can play back any messages already downloaded even if you have no connection.

You can set your greeting through the phone without having to dial the voicemail number.

Useful Visual Voicemail features:

  • you’re not paying a per minute rate to listen to messages (your mobile data allowance is being used, but this will be cheaper)
  • the push notification tells you who each message is from
  • you can obviously play messages in any order / scrub through the audio and see the duration of each message at a glance
  • ability to share the audio with other apps if you want to permanently save a message.
  • in Beta (US only) – auto transcription support (it downloads the message and the transcription is done on the phone).

Voicemail on Three UK:

Three doesn’t use the voicemail notification icon.  Instead you immediately get a text telling you the total number of messages in the mailbox (this uses the REPLACE_SM feature in SMS to overwrite any previous notification – so you don’t get a string of texts you have to delete).

Roaming:

Be aware that most international roaming agreements are still 2G / 3G only.  If you want 4G, you’ll typically need to buy a local SIM.  Most networks will let you look up roaming charges on a per country basis on their website.

WiFi calling usually doesn’t work abroad (e.g. O2).
Network specific apps (TuGo, ThreeinTouch etc.) may do.

Coverage maps/predictions

In addition to the network maps, Ofcom have a mobile coverage predictor.  I’ll simply observe for me it is wrong for every network (including predicting I won’t be able to get one network at all when I actually get good reception.)

Reasons for choosing the ‘big 4’ rather than an MVNO

  • MVNOs tend to get less QoS priority on their parent networks, e.g. on O2, access is tiered – o2’s PayM customers at the top, followed by o2 PAYG and Tesco Mobile, then Giffgaff.   This is usually apparent in speed tests and occasionally whether you can connect at 4G.
  • Free London underground Wi-Fi access
  • Services like tethering, WiFi calling, Visual Voicemail often aren’t available on an MVNO.

What network do I currently use?

I’ve used many (on both PayM and PAYG), but currently Three PAYG.

Reasons:

  • best local coverage (including 4G and VoLTE) – your priority when choosing a network should always be how good is the coverage in the places I will use it the most? This will usually be the deciding factor in whether you’re happy with you’re current network. (Check if the situation has changed every so often, I used not to be able to get Three at all.)
  • WiFi calling & SMS
  • best call pricing of any network (3p/min voice, 2p/text, 1p/MB data).
  • Roaming at same prices in many countries (on PAYG as well as PayM)

I would prioritise coverage, the least “hassle” and support for phone software features you wish to use (e.g. wi-fi calling) over contract prices.

PAYG or PayM?

This may be a moot point if the cost of your next phone wipes out any savings you have made.

Always try a PAYG SIM first to test coverage.

If you’re getting a contract, look at the 30-day deals, and consider if the peace of mind of not being tied into a much longer contract outweighs the extra you will be paying each month for the same data/minutes/text allocation.

The choice depends on your usage patterns, how much hassle you find topping up a PAYG phone and if any features are denied to PAYG users (in Three’s case, tethering/hotspots).   If you’re paying for addons every single month (which, for all networks, all expire after 30 days) you may be doing it wrong.

My rule of thumb: at 1p/MB data, if you’re using an average of 1GB a month (Three’s UK month average usage is 6GB apparently – how are people using that much?) PAYG makes more sense.

Note also, all the main networks currently add an annual increase to contract prices in line with inflation (RPI) – PAYG users avoid this.