backdrop-contrib/feeds

Import or aggregate data as nodes, users, taxonomy terms or simple database records.

Feeds

An import and aggregation framework for Backdrop. Import or aggregate data as nodes, users, taxonomy terms or simple database records.

Features

• Pluggable import configurations consisting of fetchers (get data) parsers (read and transform data) and processors (create content on Backdrop).
  • HTTP upload (with optional PubSubHubbub support).
  • File upload.
  • CSV, RSS, Atom parsing.
  • Creates nodes or terms.
  • Creates lightweight database records if Data module is installed. http://drupal.org/project/data
  • Additional fetchers/parsers or processors can be added by an object-oriented plugin system.
  • Granular mapping of parsed data to content elements.
• Import configurations can be piggybacked on nodes (thus using nodes to track subscriptions to feeds) or they can be used on a standalone form.
• Unlimited number of import configurations.
• Export feeds importer configurations.
• Optional libraries module support.

Requirements

• Job Scheduler https://github.com/backdrop-contrib/job_scheduler
• PHP safe mode is not supported, depending on your Feeds Importer configuration safe mode may cause no problems though.

Installation

• Install Feeds, Feeds Admin UI.
• To get started quick, install one or all of the following Feature modules: Feeds News, Feeds Import, Feeds Fast News (more info below).
• Make sure cron is correctly configured https://backdropcms.org/user-guide/setting-cron
• Go to import/ to import data.

SimplePie Installation

To install the SimplePie parser plugin, complete the following steps:

1. Download SimplePie from http://simplepie.org/downloads. The recommended version is: 1.5.6.
2. Decompress the downloaded zip file.
3. Rename the uncompressed folder to simplepie. For example rename simplepie-1.5 to simplepie.
4. Move the folder to /libraries. If the folder does not exist, created it. The final directory structure should be /libraries/simplepie.
5. In order for your site to recognize this library, you have two options. Either install the Libraries module, or, put this line in your settings.php file: $config['feeds.settings']['feeds_library_dir'] = 'libraries';
6. Flush the Backdrop cache.
7. The SimplePie parser should be available now in the list of parsers.

PubSubHubbub support

Feeds supports the PubSubHubbub publish/subscribe protocol http://code.google.com/p/pubsubhubbub/. Follow these steps to set it up for your site.

• Go to admin/structure/feeds and edit (override) the importer configuration you would like to use for PubSubHubbub.
• Choose the HTTP Fetcher if it is not already selected.
• On the HTTP Fetcher, click on 'settings' and check "Use PubSubHubbub".
• Optionally you can use a designated hub such as http://superfeedr.com/ or your own. If a designated hub is specified, every feed on this importer configuration will be subscribed to this hub, no matter what the feed itself specifies.

Libraries support

If you are using the Libraries module, you can place external libraries in the Libraries module's search path (for instance /libraries). The only external library used at the moment is SimplePie.

Libraries found in the libraries search path are preferred over libraries in feeds/libraries/.

API Overview

See "The developer's guide to Feeds": http://drupal.org/node/622700

Testing

See "The developer's guide to Feeds": http://drupal.org/node/622700

Debugging

Set the Backdrop config variable feeds_debug to TRUE. This will create a file /tmp/feeds_[my_site_location].log. Use tail -f on the command line to get a live view of debug output. You can either set it in feeds.settings.json in your active configuration folder or by putting $config['feeds.settings']['feeds_debug'] = TRUE; into the settings.php file.

Note: at the moment, only PubSubHubbub related actions are logged.

Performance

See "The site builder's guide to Feeds": http://drupal.org/node/622698

Hidden settings

Hidden settings are variables that you can define by either updating them in feeds.settings.json in your active configuration folder or overriding them in settings.php file like this example: $config['feeds.settings']['NAME'] = VALUE;.

Name: feeds_debug
Default: FALSE
Description: Set to TRUE for enabling debug output to /BACKDROPTMPDIR/feeds_[sitename].log

Name: feeds_importer_class
Default: 'FeedsImporter'
Description: The class to use for importing feeds.

Name: feeds_source_class
Default: 'FeedsSource'
Description: The class to use for handling feed sources.

Name: feeds_data_$importer_id
Default: feeds_data_$importer_id
Description: The table used by FeedsDataProcessor to store feed items. Usually a FeedsDataProcessor builds a table name from a prefix (feeds_data_) and the importer's id ($importer_id). This default table name can be overridden by defining a variable with the same name.

Name: feeds_process_limit
Default: 50
Description: The number of nodes feed node processor creates or deletes in one page load.

Name: http_request_timeout
Default: 15
Description: Timeout in seconds to wait for an HTTP get request to finish. Note: This setting could be overridden per importer in admin UI: admin/structure/feeds/<your_importer>/settings/<your_fetcher> page.

Name: feeds_never_use_curl
Default: FALSE
Description: Flag to stop feeds from using its cURL for http requests. See http_request_use_curl().

Name: feeds_use_mbstring
Default: TRUE
Description: The extension mbstring is used to convert encodings during parsing. The reason that this can be turned off is to be able to test Feeds behavior when the extension is not available.

Glossary

See "Feeds glossary": http://drupal.org/node/622710

License

This project is GPL v2 software. See the LICENSE.txt file in this directory for complete text.

Maintainers

• Andy Shillingford https://github.com/docwilmot/
• Herb v/d Dool https://github.com/herbdool

Credits

Drupal version currently maintained by:

• https://www.drupal.org/u/MegaChriz
• https://www.drupal.org/u/twistor
• https://www.drupal.org/u/tristanoneil
• https://www.drupal.org/u/franz
• https://www.drupal.org/u/febbraro

feeds's Issues

Update description of 'Update existing nodes' setting

In D7, they updated the description of the 'Update existing nodes' options here: https://git.drupalcode.org/project/feeds/commit/a966aba This was to better explain the differences between the 'Replace existing nodes' and 'Update existing nodes' options.

I suggest we do something similar, as this always confuses me.

Upgraded Feeds from Drupal 7 not seamless

As noted here, I had to do some troubleshooting after trying to use Feeds on a D7->Backdrop upgrade and in the end I had to end up completely uninstalling on Backdrop and then reactivating. The config file was full of NULL values, which is (I think) what was causing my problem.

{
    "_config_name": "feeds.settings",
    "feeds_reschedule": null,
    "feeds_debug": null,
    "feeds_importer_class": null,
    "feeds_library_dir": null,
    "default_feeds_importer": null,
    "menu_rebuild_needed": null,
    "feeds_source_class": null,
    "http_request_timeout": null,
    "proxy_server": null,
    "proxy_port": null,
    "proxy_username": null,
    "proxy_password": null,
    "proxy_auth_method": null,
    "feeds_never_use_curl": null,
    "feeds_process_limit": null,
    "default_nodes_main": "10"
}

The first error I got was:

Fatal error: Class name must be a valid object or a string in backdrop/modules/contrib/feeds/includes/FeedsConfigurable.inc on line 64

Debugging that with Jen's fix here gave me this:

Error: Call to a member function configFormValidate() on null in feeds_ui_create_form_validate() (line 193 of /path/modules/feeds/feeds_ui/feeds_ui.admin.inc).

And at that point after further debugging I realized the config file may be the issue and just nuked it and reinstalled.

---

#29

Convert feeds_log view

feeds_log view is still in d7 format.

Feeds does not map created date, URL or author correctly, uses current date, current logged in user and creates URL instead

Running into a big issue with Feeds importer - it is overwriting the Author and Post Date assigned in mapping and replacing with current logged in user and current date - this is a BIG issue for this client, since the announcements are sorted by date and the URL's are assigned by date - it's losing all that information on import. And there are a LOT of announcements - I can't fix them all manually. HELP!!!

Here's the field mappings. The URL, Title, Author and Date are critical: post-mapping.jpeg

Improve README

Improve the text.

PHP notice Undefined index: module in feeds_system_info_alter()

I'm seeing my dblog flodded with the following PHP notices:

Notice: Undefined index: module in feeds_system_info_alter() (line 819 of backdrop/modules/contrib/feeds/feeds.module).

The following code...

    if ($file->name === $plugin['module']) {
      $module_plugins[$plugin_id] = TRUE;
    }

...should be replaced as follows:

    if (array_key_exists('module', $plugin) && $file->name === $plugin['module']) {
      $module_plugins[$plugin_id] = TRUE;
    }

---

PR: #13

Remove EXPORT_IN_*

These are holdovers from Ctools and not needed here since everything is in config.

Update README to list maintainers, license, etc

Fix import/export of feeds importers

The import doesn't work at all. The export goes to the config page but doesn't select the right importer.

Add 'Import' to admin menu

I've always found it annoying that there doesn't seem to be a link anywhere to the 'Import' page (apart from being partially hidden on the first page of the feed configuration form). I'm constantly having to access it by typing in the URL manually.

Assuming I haven't missed something obvious, I'd like to suggest adding a link to the Import page in the Administration Menu. It seems to me that the Content section would be a good candidate:

Convert README.txt to README.md

PHP fatal on install (autoloader not loading feeds classes)

For some reason the autoloader isn't loading the feeds classes on my site. I've tried changing PHP versions to no avail. When I attempt to enable the Feeds module, at the moment I save the modules page I get the following error:

Error: Class 'FeedsConfigurable' not found in feeds_importer() (line 1066 of backdrop/modules/contrib/feeds/feeds.module).

When I add a manual call to load the file, this class is found, but the next one is not:

  module_load_include('inc', 'feeds', 'includes/FeedsConfigurable');

I then get the following error:

Fatal error: Class name must be a valid object or a string in backdrop/modules/contrib/feeds/includes/FeedsConfigurable.inc on line 64

I was able to get around the second PHP fatal by wrapping the return of the instance with a check for class:

    if (!empty($class)) {
      if (!isset($instances[$class][$id])) {
        $instances[$class][$id] = new $class($id);
      }
      return $instances[$class][$id];
    }

But I'm still not sure why any of this was necessary...

Does not recognize the Telephone field

The drop down list of mapping doesnt have the Telephone field as option

Missing DateObject class

When I attempt to import a CSV feed containing dates, the import fails with the following error:

Error: Class 'DateObject' not found in _date_feeds_get_date_object() (line 210 of [...]/modules/contrib/feeds/mappers/date.inc).

I am unable to find a definition for the DateObject class in my codebase. Is there another module that provides it?

I am running Backdrop 1.18.0 with PHP 7.2.34.

It was working for me before I updated Feeds to 1.1.0.

Allow overrides in settings file

README references setting some variables in the settings file. We'll need to explicitly add this functionality.

Bring up to date with 7.x-2.0-beta5

https://www.drupal.org/project/feeds/releases/7.x-2.0-beta5

(It's been in alpha/beta for like 10 years. Seriously just release it)

Release notes:

Major changes in this release

Imports are ran as the feed author or user 1

When running imports, Feeds now switches to a different user. This is done to reduce field validation issues and to minimize the difference in (validation) results between importing using the UI and importing using cron.

When attaching the importer to a content type, Feeds switches to the author of the feed node. If the author is anonymous, Feeds switches to user 1 instead.

When using the standalone import form, Feeds switches to user 1.

When the "Authorize" option is checked (setting for the node processor), an extra account switch happens for every item that gets imported. Feeds then switches to the author of the item.

While test coverage exists to make sure Feeds switches back after it's done, this change could in theory have regressions.

Improvements for field validation issues

Some improvements have been made to fix common field validation issues (that were introduced in 7.x-2.0-beta4). The following improvements have been made:

• Feeds now only validates mapped fields.
• Less validation issues for Organic Groups reference fields.
• A hook called hook_feeds_prevalidate() has been added to workaround validation issues: with that hook you can fix up the entity being imported right before it is validated.
• PHP 7.2+ support
• Most issues have been fixed that caused PHP notices or warnings on PHP 7.2 and above. You do may still get PHP notices/warnings when using the Date module. These issues may be fixed in the Date 7.x-2.11-beta3 release however. If not, please help on #2867810: Plan for Date 7.x-2.11 release!

Function feeds_alter() is removed

This function was deprecated 6 years ago and was removed because it did not work properly on recent PHP versions.

Fixing misspellings and grammatical errors

I have submitted a pull request (#83) to fix various misspellings and grammatical errors.

PHP errors when creating taxonomy terms

I set up a feed to import content to a content type with a taxonomy, and I made the feed settings (in the Mapping) say that if a taxonomy term didn't exist, it should be created.

The feed failed to import (the batch went to a "there was a 500 error" page). And I got the following PHP errors in the dblog in the site, which I assume were the cause of the problem:

Notice: Undefined property: TaxonomyVocabulary::$vocabulary in taxonomy_feeds_set_target() (line 106 of (my site dir)/modules/feeds/mappers/taxonomy.inc).

Recoverable fatal error: Argument 1 passed to taxonomy_term_save() must be an instance of TaxonomyTerm, instance of stdClass given, called in (my site dir)/modules/feeds/mappers/taxonomy.inc on line 175 and defined in taxonomy_term_save() (line 548 of (my site dir)/core/modules/taxonomy/taxonomy.module).

I'll see if I can fix it and supply a patch.

preventing undefined index disabled error

This is the problem described in issue #46. Is it any more than a change to the way $object->disabled is assigned in feeds_load_importer_object()?

Fatal Error when exporting feeds

Error: Cannot use object of type stdClass as array in feeds_export() (line 962 of /modules/feeds-1.x/feeds.module).

Error thrown when selecting a Feeds Importer and choosing "Export" from the available options.

Get tests working

Tests need to work.

"deprecated" notice in taxonomy import

When importing taxonomy references and mapping them by ID, there's a deprecation warning:

The function taxonomy_feeds_set_target() (taxonomy.inc:105) called deprecated function taxonomy_vocabulary_machine_name_load(). It will be removed in the next major release of Backdrop.

This should probably be fixed before the next major Backdrop release.

Notice when enabling the module

When I enabled the Feeds and Feeds UI module, the following notice was generated about 15 times:

Notice: Undefined index: module in feeds_system_info_alter() (line 742 of /path/to/backdrop-root/modules/feeds/feeds.module).

File upload field display issue

There seems to be an issue with the file upload field on the Import form:

This is because it's floated left:

.feeds-file-upload {
    float: left;
}

I'm not sure what this float is for - whether it can be safely removed or whether the container div needs to be cleared. In any case, something needed here to make it display properly.

Fatal error: Cannot use string offset as an array in readDateField

When importing I get an fatal error:

An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=58&op=do_nojs&op=do StatusText: OK ResponseText: ( ! ) Fatal error: Cannot use string offset as an array in backdrop/modules/contrib/feeds/plugins/FeedsParser.inc on line 575

It appears the problem is that the field has no value, but this part of the code assumes it does. Adding a check for a value first prevents the fatal.

New "Execute" operation

A new "Execute" operation can be useful on the page of "Feeds importers". We can open the import pages with it. (I will create a PR.)

Undefined variable $uri in function theme_feeds_upload

In the file feeds.pages.inc, the function theme_feeds_upload references the undefined variable $uri.

I think that $uri should be replaced with $file->uri.

Use dropbutton for operations

Use the dropdown form widget instead of simple links. More compact and looks more professional.

Fatal error when importing from a CSV file

I've configured a Feeds importer with no problems, but when trying to run an import of a CSV into nodes using a standalone form I get this fatal error after clicking the "Import" button:

Fatal error: Class 'JobScheduler' not found in /var/www/sitename/modules/feeds/includes/FeedsSource.inc on line 314

I've done a bit of digging but can't sort out the problem; Job Scheduler is also installed and enabled. Has anyone else run across this problem?

Uninstall should remove the view, module disable should disable the view

feeds_log sticks around even if the module is uninstalled. Can't even delete it manually.

--
PR #48

Feeds News requires Features

I assume that means it's not ported/tested?

Undefined property error when importing taxonomy terms

Just created an import feed to populate a node type with taxonomy terms and got an error:

Notice: Undefined property: stdClass::$tid a taxonomy_feeds_set_target() (línia 160 de /modules/feeds/mappers/taxonomy.inc)

This error prevents to fill in the taxonomy terms during the import.

Will fill in a PR with the solution I found.

Make 'Enabled' column the first column of the 'Feeds importers' table

It feels weird not having the 'Operations' column be the last column in the table:

I suggest moving the 'Enabled' column to the left so it's the first column. This would then match how the Modules table is arranged:

Convert some drupal.org to backdropcms.org

Change to equivalent links where it makes sense.

Enabled / Disabled setting won't stick

I can't seem to disable Feeds importers that I don't want to use on my site. Un-checking the "Enabled" checkbox and saving the listing does not disable the importer.

---

PR #27

Download CSV template has html cruft

When downloading we get the CSV fields but also the html page. Need to nip that in the bud.

Disabled color in feeds admin UI isn't accessible

See related core issue: backdrop/backdrop-issues#4519

Maybe we should wait for a color or pattern to be chosen for Seven. Then we can do the same here, or apply a class where the admin theme color will be used automatically :)

Remove feeds_importer table and convert to config if not existing yet

There's an update hook to convert settings to config file but not the feeds_importer table entries.

Remote File Import Fails With Exception

When attempting to pull in remote files via a CSV url, I get the following:
`Exception: Download of http://example.com/sites/default/files/05202018.mp3 failed with code -1. in /Drupal/localsite/modules/contrib/feeds/plugins/FeedsParser.inc:424

Stack trace:

#0 /Drupal/localsite/modules/contrib/feeds/plugins/FeedsParser.inc(483): FeedsEnclosure->getContent()

#1 /Drupal/localsite/modules/contrib/feeds/mappers/file.inc(127): FeedsEnclosure->getFile('public://audio/')

#2 /Drupal/localsite/modules/contrib/feeds/plugins/FeedsProcessor.inc(806): file_feeds_set_target(Object(FeedsSource), Object(Node), 'field_audio:uri', Array, Array)

#3 /Drupal/localsite/modules/contrib/feeds/plugins/FeedsProcessor.inc(752): FeedsProcessor->mapToTarget(Object(FeedsSource), 'field_audio:uri', Object(Node), Array, Array)

#4 /Drupal/localsite/modules/contrib/feeds/plugins/FeedsProcessor.inc(302): FeedsProcessor->map(Object(FeedsSource), Object(FeedsParserResult), Object(Node))

#5 /Drupal/localsite/modules/contrib/feeds/includes/FeedsSource.inc(399): FeedsProcessor->process(Object(FeedsSource), Object(FeedsParserResult))

#6 /Drupal/localsite/modules/contrib/feeds/feeds.module(207): FeedsSource->import()

#7 /Drupal/localsite/core/includes/batch.inc(251): feeds_batch('import', 'sermon_importer', 0, Array)

#8 /Drupal/localsite/core/includes/batch.inc(111): _batch_process()

#9 /Drupal/localsite/core/includes/batch.inc(85): _batch_do()

#10 /Drupal/localsite/core/modules/system/system.admin.inc(2198): _batch_page()

#11 /Drupal/localsite/core/includes/menu.inc(582): system_batch_page()

#12 /Drupal/localsite/core/modules/layout/layout.module(441): menu_default_route_handler(Array)

#13 /Drupal/localsite/core/includes/menu.inc(529): layout_route_handler(Array)

#14 /Drupal/localsite/index.php(21): menu_execute_active_handler()

#15 {main}
`

and:
Invalid enclosure <em class="placeholder">http://example.com/sites/default/files/05202018.mp3</em>

I confirmed the file is there and publicly readable...where were else could this problem be coming from?

Node processor mapping doesn't seem to recognize Email fields

The node processor mapping page doesn't seem to recognize Email fields. (They don't show up as an option in the Target dropdown).

Language/translation import issues

On a relatively small upgrade from Drupal 7 that used entity translation, I created an export CSV of all the content in the second language and am trying to import using Feeds. (Then I will use a little manual scripting to try to connect the old NID to the TNID column and test whether that works for converting small sites from Entity Translation to Backdrop's Content Translation.

The problem with the basic import so far is that the language selection doesn't work in Feeds Importer. The CSV/field mapping process provides configuration buttons but they don't actually open the configuration to allow a selection, so in this case I can't select the appropriate language for the fields and they are coming in as und (language neutral).

This means that when I go to edit one of these imported translated content items, the body field is blank (even though there is data in the database for it).

PHP notice: Undefined index: disabled in feeds_load_importer_object()

My error log is being flooded with thousands of entries like this one:

Notice: Undefined index: disabled in feeds_load_importer_object() (line 1421 of [...]/modules/feeds/feeds.module).

A workaround might be to replace
$object->disabled = $data['disabled'];
with something like
if (isset($data['disabled'])) { $object->disabled = $data['disabled']; }

Error when importing files by uri

Recoverable fatal error: Argument 1 passed to file_copy() must be an instance of File, instance of stdClass given, called in E:\sites\zarechny.local\www\modules\feeds\plugins\FeedsParser.inc on line 456 and defined in file_copy() (line 792 of E:\sites\zarechny.local\www\core\includes\file.inc).

Very easy to fix.

PR: #6

Better error handling in user importer

I needed more specific errors, so I made this tiny change :)

if (empty($account->name) || empty($account->mail) || !valid_email_address($account->mail)) {
  throw new FeedsValidationException(t('User name missing or email not valid.'));
}

+    if (empty($account->name) || empty($account->mail)) {
+      throw new FeedsValidationException(t('User name or email missing.'));
+    }
+    if (!valid_email_address($account->mail)) {
+      throw new FeedsValidationException(t('Email not valid.'));
     }

Too few arguments to function node_unpublish_action()

Too few arguments to function node_unpublish_action(), 1 passed in /path/modules/feeds/plugins/FeedsNodeProcessor.inc on line 412 and exactly 2 expected

This is on a cron run while testing an RSS feed import. I suspect the setting to unpublish previously imported nodes if they no longer exist was set.

Undefined property: stdClass::$tid in taxonomy_feeds_set_target

When parsing a feed and mapping values to taxonomy terms, my terms are not properly set and my log is filled wit this message:

Notice: Undefined property: stdClass::$tid in taxonomy_feeds_set_target() (line 156 of backdrop/docroot/modules/contrib/feeds/mappers/taxonomy.inc).

It looks like the stdClass item here has a property of entity_id instead of tid. Making that change gets my terms searching and saving correctly.

Want help maintaining?

@docwilmot do you want help maintaining?

Feeds UI install file needs updating

function feeds_ui_update_7000() {
// Do nothing.
}

Convert feeds_import to use config

feeds_import is a submodule that provides a couple example feeds importers. We can easily convert them to config and they'll get imported when module enabled.

Import of term with parent doesn't work

I'll add more notes here. I was testing with sample file from this module for importing terms and it fails. Most likely because of a parent term.