Troubleshooting ‘The provided URL does not represent a valid oEmbed resource.’ with YouTube videos

If you are trying to add a YouTube video using media > remote video in Drupal 8.7 (media is now part of core) and you stumble across this error – which is not discussed anywhere in the core issue queue that I can see…

The provided URL does not represent a valid oEmbed resource.

the fix is to add this line in settings.php:

$settings['http_client_config']['force_ip_resolve'] = 'v4';

Debugging / cause:

– First verify it’s not a private video (unlisted is fine)
– Optionally you can manually test the oEmbed response in your browser (you should get some JSON data back) – the URL needs to be:
https://www.youtube.com/ombed?url=[an encoded full YouTube URL]

It’s as quick/quicker to look in Recent Log Messages in Drupal first, you may be seeing errors like this:

Client error: `GET https://www.youtube.com/oembed?url=https%3A//www.youtube.com/watch%3Fv%3Dyoutubeid` resulted in a `429 Too Many Requests` response:

and also:

Could not retrieve the oEmbed resource.

(which doesn’t tell you anything).

oEmbed requests in Drupal are simple, unauthenticated HTTP GETs – i.e. they don’t use the YouTube API.  Nothing particularly wrong about this.  However the trouble is YouTube has started blocking IPv6 blocks en-mass; apparently because it’s too easy for people to keep changing IPv6 addresses (if ISPs provide a large pool of them) and use them for spam.   Therefore all and any requests over IPv6 may simply be blocked, regardless of how few you’ve made (I can confirm this for Linode servers.)

First, verify this is true for your server by manually grabbing the URL at the command line via wget, with the  -6 and -4 switches.

Assuming IPv4 works, you now need to tell Drupal to make Drupal use it.  This is easier than expected.   Drupal uses the Guzzle HTTP client library, specifically as an HTTP client factory – but as this StackExchange answer explains  – part of the default setup automatically merges in values from $settings, i.e. anything you add under $settings[‘http_client_config’].

So make your $settings.php writeable, and configure the force_ip_resolve setting as shown at the beginning of this post.

YouTube queries should then start working immediately.

 

 

 

 

Drupal 6 Nginx config fragment

Here, for anyone needing to host a D6 LTS site, is a working Nginx fragment (tested with Nginx 1.10.3).

You can reuse your standard Drupal 8 config for everything else (e.g. images, protecting private files and so on).

# Drupal 6 LTS
index index.php;

location / {
    if (!-e $request_filename) {
       rewrite ^/(.*)$ /index.php?q=$1 last;
    }
}

Ordinarily, for modern Drupal sites, I’d use the following standard try_files statement, but I couldn’t get it serve D6 subpages correctl (it just redirects to the homepage, even with the q=… added – email me if you know why).

# Drupal 8
location / {
    try_files $uri /index.php?$query_string;
}

 

 

 

Drupal 6 – troubleshooting ‘Site off-line’ db error

A straightforward problem, but one I’ve wasted time on when setting up a D6 LTS site.

Symptom:

The site is currently not available due to technical problems. Please try again later. Thank you for your understanding.

If you are the maintainer of this site, please check your database settings in the settings.php file and ensure that your hosting provider’s database server is running. For more help, see the handbook, or contact your hosting provider.

The uncommented example line in default.settings.php is:

$db_url = ‘mysql://username:password@localhost/databasename’;

I spent some time verifying usernames/passwords and adjusting ansible scripts, what I hadn’t noticed was I need mysqli (Mysql Improved – which has been around since way with mysql v4.1.3), not mysql.

So remember to check the connection protocol as well as the credentials.

 

Drupal – troubleshooting PHP files downloading rather than executing

A fairly straightforward problem that won’t be unique to Drupal, but you may run into when migrating PHP applications from other hosts.

I was reviving an old D6 site that had been hosted on another ISP (Hostgator, as it happens) and on setting it up on Acquia DevDesktop (which is a local MAMP stack) found PHP wouldn’t execute as normal.

First, isolate the problem:

  • i.e. do other sites besides this one, running on the same computer (typically you’ll get this problem on a local dev setup) work correctly?
  • create a test PHP file (e.g. containing  <?php echo "Hello, world!";  or <?php phpinfo(); and load it

Solution in my case:

  • check the .htaccess – it had the following, which was redirecting all PHP requests to a PHP  binary that didn’t exist.  Once commented it out PHP could run correctly.
# $Id$
# Use PHP56 as default
AddHandler application/x-httpd-php56 .php
<IfModule mod_suphp.c>
 suPHP_ConfigPath /opt/php56/lib
</IfModule>

Of course any .htaccess files become irrelevant if you move your dev or production sites to Nginx, but it’s a good idea to read through it anyway.

How to install a drupal.org sandbox module using composer

Sandbox modules don’t have a drupal.org/project/foo URL like full contrib modules, and therefore you can’t use composer require drupal/foo to add them.

If you have a Drupal 8 site using drupal-composer/drupal-project, here’s what to edit in composer.json – using a sandbox module of mine as an example.

  1. Within the repositories section
"drupal-wturrell/ckeditor_remove_elementspath": {
    "type": "package",
    "package": {
        "name": "drupal-wturrell/ckeditor_remove_elementspath",
        "version": "0.0.1",
        "type": "drupal-module",
        "source": {
            "url": "https://git.drupal.org/sandbox/wturrell/3018599.git",
            "type": "git",
            "reference": "8.x-1.x"
        }
    }
},

2.  In require (or run composer require with the name you’ve specified)

"drupal-wturrell/ckeditor_remove_elementspath": "^0.0.1",

You can choose any name you like, but drupal-username/module makes sense to me.

Your sandbox module doesn’t need a composer.json file of it’s own.

Acquia Dev Desktop 32-bit MySQL

It’s worth noting the version of MySQL that comes installed with Acquia Dev Desktop 2 (5.6.41-84.1) is still 32-bit (unlike PHP which is now 64-bit).   Recently I was testing with a particularly large database and found it kept crashing with things like:

‘MySQL server has gone away’

Lost connection to MySQL server at ‘reading authorization packet’

Intensive operations such as Drupal-to-Drupal migration tended to trigger this.

I used MySQLTuner but the changes made little difference so then it occurred to me the 32-bit version simply wasn’t up to it, so I installed the latest 64-bit mariadb from homebrew instead and moved the relevant databases over to that (I have mariadb running on port 3306 and Dev Desktop’s MySQL on it’s default 33067).

So far it’s been stable.

Troubleshooting Drush with Acquia Dev Desktop

Updated Tue 8 Jan 2019

Symptom: drush updb hangs

If drush updb is hanging, first, run it with the --verbose script to see exactly what’s going on – that will show you the MySQL connections. (It turned out in my case Drush was connecting to the wrong database).

Symptom: drush connects to wrong database

Checklist:

  • Try starting the terminal connection via DevDesktop (the icon top right) and see if that makes a difference
  • Double the database connection details in the appropriate loc_ file in ~/.acquia/DevDesktop/DrupalSettings/
  • Check your drush status output
  • Check what happens if you run drush sqlc
  • Try installing the latest Drush (composer require drush/drush) rather than relying on the version that comes with DevDesktop (at time of writing that’s 8.1.7, whereas Drush is now up to 9.5.2)

My situation: I have two MySQL installations running, the 32-bit DevDesktop supplied version, and a 64-bit mariadb (for a very large site where I was experiencing connection timeouts).  They are on different ports. I had a site where ordinary drush commands and drush sqlc connection went to the correct MySQL server but drush updb used the wrong one (confirmed using the --verbose option – in fact it was connecting to a database with a different name entirely.

I couldn’t figure out what was going on, so I upgraded from Drush 8 to 9 and that fixed it immediately.

Symptom: large Drupal 7 site where cache is extremely slow or fails to clear…

drush cc all

…and eventually PHP runs out of memory (“cannot allocate”).

Check Drush is using the correct version of PHP, it may not be – use drush status or, more simply: drush php

To fix:

$ which drush
/Applications/DevDesktop/tools/drush

$ vim /Applications/DevDesktop/tools/drush

# edit the PHP version: [ -z "$PHP_ID" ] && PHP_ID=php7_1

 

Drupal debugging: Invalid argument supplied for foreach() EntityFieldManager.php:490

You might see this error occasionally when installing/uninstall modules, running db updates etc.

The code in EntityFieldManager.php loops through an array of entities – each of which is a bundle, for example the user entity contains all the user fields you may have created.

The trick for debugging is to dump the contents of $bundle_field_map and look for something wrong – e.g. my issue was an empty (NULL) font entity, which had been left behind after attempting to remove the fontyourface module.

Backup the DB (obviously) then look in the key_value table for records where collection = entity.definitions.bundle_field_map – for me it was a case of deleting name = font, value = N; and the problem went away.

(So an alternative to writing code is to look at the DB table first.)

Add a PHP extension to Acquia Dev Desktop

(Updated after testing with latest DevDesktop built 6 Jun 2019 02:34:52, on macOS High Sierra 10.13.6)

Let’s assume you have the latest Acquia Dev Desktop, which comes with PHP (switchable between 5.5 up to 7.2), and you want to install the PHP memcached extension – so you can use Drupal’s memcache module to make your local development a bit faster (and test for any memcached issues on production).

Memcached v3.1.0 added support for PHP 7.3.
Acquia DevDesktop doesn’t yet support PHP 7.3, only the latest 7.2.

But first…


Why still use Dev Desktop in 2019? Running PHP and MySQL directly is blisteringly fast compared to Docker for Mac – usually several seconds every page load, and with ADD you can instantly PHP restarts when changing a config setting (e.g. enabling/disabling Xdebug) compared to having to wait a minute or more for a Lando container to rebuild (Docker containers are immutable; you have to rebuild the container to make a significant PHP.ini change like adding or removing an extension – you can’t just restart the service). Composer is faster (still a bit annoying, of course, but consider what it’s doing in the background). Drush is faster, including rebuilding the cache. Over a day, every day, this all really adds up.  For those of you using laptops, your battery life will be extended too. (Docker can also use quite a bit of CPU when idle).

Running PHP on the host means you’re directly accessing the file system and not having to jump through all the hoops you do with Docker and Vagrant – either slow file accesses, or having to wait for changes to sync.  This is especially true with Drupal given all the caching it does.  

It’s very unpopular to criticise Docker (which I’m not, exactly – more the hype and the rush to use it on a platform it wasn’t designed for, despite the slowness issues many people experience.)  There’s a trade off between performance and configuration – the assumption seems to be you should go for whatever requires the least configuration, but is it really that hard to install the correct version of PHP or MySQL version (how about making a note of what your site runs in your Git repository?)  I’d rather spend a bit longer up front getting the configuration right and have the performance benefit each day after.  Also, if I have a choice of what to learn, I think knowledge of PHP extensions and configuration – given I’m a PHP developer – is more useful than learning loads about containers.

Comments welcome by email. (I use Ansible heavily, by the way, so I’m not opposed to automation.)  


Install steps:

You need to repeat this once for each version of PHP you want the extension available in, e.g. 7.1 or 7.2.  The example below is for 7.2.

Download the extension from here –  https://pecl.php.net/package/memcached/3.1.3 – or you can use the quicker get latest link: pecl.php.net/get/memcached

extract the tar file into its own directory – it’s fine if it’s just in your downloads folder (tar -xvf)

Then, change to that directory and run:

/Applications/DevDesktop/php7_2_x64/bin/phpize

This next line is the important bit – to make it properly compatible with the PHP version DevDesktop is using.  Without this you may get an error like:

Symbol not found: _compiler_globals
Expected in: flat namespace

./configure --with-php-config=/Applications/DevDesktop/php7_2_x64/bin/php-config

Tip: modules that don’t successfully load don’t show up in phpinfo(), so a quick way to see what the error was when PHP attempted to load  them is to do php -v (it’ll show any warnings above the version info) or php -m to get a list of all the extensions loaded and verify yours is OK.

make
make install

Important: if you need to rerun any of these steps, wipe the directory you extracted first and start afresh.

You should see:

Installing shared extensions: /Applications/DevDesktop/php7_2_x64/ext/

There will now be a memcached.so file in that directory. If it hadn’t picked the config details up, it would have written to a different directory, e.g. your default homebrew one.

Add the memcached.so file to your php.ini – you can verify where that is via  Preferences > Config in DevDesktop, but it would normally be:

/Applications/DevDesktop/php7_2_x64/bin/php.ini

extension_dir = "/Applications/DevDesktop/php7_2_x64/ext"

extension = memcached.so

You can save a backup for when you upgrade DevDesktop and it gets overwritten, but if the PHP version has changed you should compile it again – it’s a very quick process anyway (few minutes at most).

NB: I rather thought PECL had an equivalent install command line switch to load the correct php-config, so you could just run pecl install command to save time, but apparently not? Let me know if I’m wrong.

Useful reading elsewhere:

Howto: Drupal 8 and memcached on Lando

    1. Make sure phpinfo(); (available at /admin/reports/status/php in Drupal for administrators) contains a memcached section – by default Lando will install the memcached PECL package.
    2. Install and enable memcache Drupal module.
    3. configure settings.php:
      $settings['memcache']['servers'] = ['cache:11211' => 'default'];
      $settings['memcache']['bins'] = ['default' => 'default'];
      $settings['memcache']['key_prefix'] = 'mysitename_';
      
      $settings['cache']['default'] = 'cache.backend.memcache';

      Note the hostname for external connections (port 11222) is localhost, but internal connections have a different hostnamecache – to verify, run lando info

      (give it the correct sitename – this is especially important if you ever use memcached for two or more sites on the same server, to avoid conflicts.

      Also, note memcached has no security so if an application knows or can guess another applications’ prefix it can read all the data.

    4. Truncate all MySQL tables beginning with cache (Drupal bootstraps from the database by default and otherwise may continue to use them)
    5. Run drush cr
    6. You can should now work.

Verify connection and performance (e.g. hits/misses and memory used) at /admin/reports/memcache

Way to test an internal connection from PHP in a container, if you can’t install what you need:

php -a
> $socket = fsockopen('cache', "11211", $errno, $errstr);

If you don’t see any errors, it has connected.