shephedproject/shepherd

Shepherd delivers reliable, high-quality Australian TV guide data (EPG).

Shepherd Guide Grabber

Shepherd delivers reliable, high-quality Australian TV guide data (EPG).

Background

The Shepherd project began in 2007 when Australians with Home Theatre PCs lacked a good method of obtaining an EPG. The free-to-air stations fought against the free availability of such data, apparently fearing that Home Theatre PCs / PVRs would allow people to skip ads. Instead, they filed lawsuits, sent cease & desist letters, and obfuscated their online guides Source.

Australians had two options:

1. Download an EPG from OzTivo, which was community produced by users manually describing shows
2. Use a script to access data in an online TV guide such as Yahoo!

Each method had drawbacks. The community-generated EPG could be unreliable in non-metro areas, tended to only include the most basic of show details, and was somewhat error-prone. The scripts would stop working each time the associated online guide changed format, which happened frequently as the sites tried to deny them access.

Shepherd solved this problem by:

1. Employing a flock of online guide grabbers, and switching between them on the fly as necessary to cover failures
2. Auto-updating, thus eliminating the need for users to manually intervene to fix problems

In 2013, Australians have more options. IceTV won its legal battle and sells an EPG online; many hardware HTPC/PVR devices exist with inbuilt EPG; and the TV stations themselves broadcast an EPG via EIT, which although still somewhat error-ridden is improving considerably. Nevertheless, Shepherd remains a free, robust method of obtaining high-quality guide data, particularly for MythTV.

Description / Features

Shepherd knows the capabilities of each grabber and can make intelligent decisions about how and when to deploy them to maximize data quality and coverage. It analyses the output from each grabber to determine whether any further grabbers are required to obtain a full dataset of required channels, and employs postprocessors to further supplement the data with information from sources such as IMDB and Metacritic.

When switching between data sources, Shepherd is able to keep show names consistent. For example, if you're used to recording a programme called "House" but a different data source names it "House, M.D.", Shepherd is able to identify the second name as a variation of the first, and automatically substitute the original in order to match any recording rules you have established.

Shepherd is particularly useful for MythTV users, as it can feed guide data automatically to MythTV with minimal user configuration. It can even install MythTV channel icons.

Shepherd is future-proof, requiring no manual intervention once installed. It will automatically update itself with fixes, enhancements, and additional grabbers and postprocessors as they become available.

Download & Installation

See the Installation page.

How does it work?

• The FAQ wiki page contains some questions and answers.
• The shepherd_logic wiki page describes how shepherd works. The default policies used by shepherd are documented on the Policies wiki page.
• The Security page discusses security issues associated with Shepherd.

More Information

Wiki content was migrated from original whuffy host to the Github Wiki

Contributing

Please see CONTRIBUTING

shepherd's Issues

Channel Icons

Many of the channel icons in the list are dead links and/or are outdated.

Some of Ash's can be found via web archive, but may no longer be worth it.
Freeview site has them listed in the JSON

• FTA
• Foxtel

Perhaps look at a list format change, or a source format which is then thrown into the tab-separated-format via script.

Incorrect Guide Data for SBS Viceland HD and SBS World Movies

I am using the W.A. Regional (102) region.

The program data for "SBS Viceland HD" channel contains the program data for "SBS World Movies" and there does not seem to be a choice to select "SBS World Movies" during the channel setup section of the --configure process.

So the upshot is my guide data for channel 31 "SBS Viceland HD" (SBSVicelandHD.shepherd.com) contains the data for "SBS World Movies" and channel 32 "SBS World Movies" (no XMLID) contains no data.

This may be a work in progress since the switchover happened in the 1st of July but since I didn't see an issue raised, I thought I'd raise one.

Publishing code changes

What is the strategy for releasing code changes?

It looks like the three main branches are master, release and staging. Where is development intended to occur and how / when are changes being pushed to general release / general availability.

If the intention is for this repo to be the successor to whuffy I believe we should be avoiding breaking changes for the installed user base. What checks are being completed before changes are pushed to general release?

9Gem HD not working

I've noticed about 10 days ago there's no more guide data for 9Gem HD- anyone else experiencing the same?

New channels wanted and some old ones missing

Revising my channels this week and found in my region 94 I have a few missing.

I am sure I had 10 Bold setup last week but cant locate now.
10 Shake is new but also having issues finding SBS Viceland & SBS Food

Seems ELEVEN could be renamed to 10 Peach and SBS 2 could be World Movies.

I do hope we continue with this project as a love having my own pvr (mythtv).

Region 94. 40 MythTV channels. 27 Shepherd channels.

   #  MythTV Channel                 Shepherd Guide Data
 --------------------------------------------------------
   1  10 HD                          <- TEN HD
   2  ABC                            <- ABC
   3  SBS ONE                        <- SBS
   7  7 Melbourne                    <- Seven
   9  Channel 9 Melbourne            <- Nine
  10  10                             <- TEN
  11  10 Peach                       <- ELEVEN
  12  10 BOLD                        <- -
  13  10 SHAKE                       <- -
  15  10 HD                          <- TEN HD
  16  TVSN                           <- TVSN
  17  SpreeTV                        <- SpreeTV
  20  ABC HD                         <- ABC
  21  ABC                            <- ABC
  22  ABCComedy/Kids                 <- ABC2 / KIDS
  23  ABC ME                         <- ABCME
  24  ABC NEWS                       <- ABC News 24
  30  SBS ONE HD                     <- SBS HD
  31  SBS VICELAND HD                <- -
  32  SBS World Movies               <- SBS 2
  33  SBS Food                       <- Food Network
  34  NITV                           <- NITV
  44  C31                            <- Channel 31
  70  7HD Melbourne (7 HD)           <- 7HD
  71  7 Melbourne                    <- Seven
  72  7TWO Melbourne (7 TWO)         <- 7TWO
  73  7mate Melbourne                <- 7mate
  74  7mateHD Melbourne (7 mate HD)  <- 7mate
  75  openshop                       <- -
  76  7flix Melbourne (7 flix)       <- 7flix
  78  RACING.COM                     <- RACING.COM
  90  9HD Melbourne (9 HD)           <- 9HD
  91  Channel 9 Melbourne            <- Nine
  92  9Gem Melbourne                 <- 9Gem
  93  9Go! Melbourne (9 Go!)         <- 9Go!
  94  9Life Melbourne (9 Life)       <- 9Life
  95  9GemHD Melbourne (9 Gem HD)    <- 9Gem
  96  9Rush Melbourne                <- 9Life
  97  Extra                          <- Extra
  99  9Go! Melbourne                 <- 9Go!

Perl Dependencies for Ubuntu 20.04

Hi,
Following is the start of the installation of shepherd and feedback from terminal entry.
As you can see at the bottom of the data one of the dependencies gets substituted and no substitute is found for another.
If you carry out the dependency install and then "perl shepherd" it wont install properly due to missing dependency. I am not sure if the subtitled dependency is also correct.
All install information is followed from the github installation manual.
https://github.com/ShephedProject/shepherd/wiki/Installation#steps---new-installations
Perl Dependencies are supported in Ubuntu 19.10 but not in 20.04.

:~$ wget 'https://raw.githubusercontent.com/ShephedProject/shepherd/release/applications/shepherd'
--2020-05-05 03:30:46-- https://raw.githubusercontent.com/ShephedProject/shepherd/release/applications/shepherd
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.28.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.28.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 165116 (161K) [text/plain]
Saving to: 'shepherd’

shepherd 100%[===================>] 161.25K --.-KB/s in 0.05s

2020-05-05 03:30:46 (3.37 MB/s) - 'shepherd’ saved [165116/165116]

:~$ sudo apt-get install libxml-simple-perl libalgorithm-diff-perl \libgetopt-mixed-perl libdata-dumper-simple-perl libdate-manip-perl \liblist-compare-perl libdatetime-format-strptime-perl \libhtml-parser-perl libxml-dom-perl libgd-gd2-perl libarchive-zip-perl \libio-string-perl xmltv libdbi-perl libsort-versions-perl libjson-perl libobject-tiny-perl
[sudo] password for mark:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libgd-perl' instead of 'libgd-gd2-perl'
E: Unable to locate package libgetopt-mixed-perl

Kind regards,
Mark

TTVDB and TMDB postprocessors

Xander, in case you are not aware there are working (for me) TTVDB and TMDB grabbers at my testing source:
tv_grab_au --addsource https://shep.theperkinsfamily.id.au/test/

Further detail at https://groups.google.com/forum/?hl=en#!topic/shepherd-list/1jaWNewS9pI
and
https://groups.google.com/forum/?hl=en#!topic/shepherd-list/U_59wQoEt0w

PERL5LIB overwritten - breaks on Perlbrew install

I've run Shepherd for years with system Perl, but would now like to use Perlbrew. I've been able to install all required modules under my Perlbrew-managed Perl, but I noticed a problem at line 1697 of shepherd:

my $exec = sprintf "PERL5LIB=\"%s/references\" %s 2>&1|", $CWD, $prog;

This clobbers the PERL5LIB environment variable. Unfortunately, that is exactly where all the XMLTV modules go when it is installed via make. Consequently, shepherd complains several required modules are missing from @INC (primarily XMLTV, but a couple of others as well).

My guess is this line was put there for security purposes by enforcing a known environment. If so, how critical is this? Would it be possible to do this instead:

my $exec = sprintf "PERL5LIB=\"$ENV{PERL5LIB}:%s/references\" %s 2>&1|", $CWD, $prog;

That way, the user's existing $PERL5LIB is respected and the necessary references directory is merely appended. As far as I can tell, shepherd still seems to work with this modification.

Mythfill database fails with malformed xml file

After a new install with mythtv 0.31, shepherd fails with mythfilldatabase

2022-08-20 13:55:19.554695 I Updating source #1 (VS1) with grabber tv_grab_au
2022-08-20 13:55:19.555451 I Found 24 channels for source 1 which use grabber
2022-08-20 13:55:19.555834 I Starting process manager
2022-08-20 13:55:19.555902 I Starting process signal handler
2022-08-20 13:55:19.555942 I Starting IO manager (read)
2022-08-20 13:55:19.557959 I Starting IO manager (write)
2022-08-20 13:55:20.006808 I Grabber has capabilities: baseline manualconfig preferredmethod
2022-08-20 13:55:20.457697 I Grabber prefers method: allatonce
2022-08-20 13:55:20.458905 I XMLTV config file is: /home/user/.mythtv/VS1.xmltv
2022-08-20 13:55:20.910071 E XMLTV grabber returned error code 33

channel_list updates for Newcastle, NSW

Hi,
While I'm not certain how wide this problem is, I do know that it impacts region 184 (NSW: Newcastle).

Ownership of channel Ten appears to have been transferred from WIN to Souther Cross while channel Nine has moved to WIN.

This caused MythTV to replace all my channel Ten guide data with channel Nine.

The fix was to add the new Ten related entries (10,10 Bold,10 HD,10 Peach,10 Shake) to the line starting with the region number 184:

~/.shepherd/references/channel_list/channel_list
184:10,10 Bold,10 HD,10 Peach,10 Shake,7flixPrime,7mate,7TWO,9Gem,9GO!,9HD,9Life,ABC TV,ABC Kids/ABC TV Plus,ABC TV HD,ABCME,ABCNEWS,Aspire,ishoptv,NBN,NITV,PRIME7,PRIME7 HD,RACING.COM,SBS,SBS Food,SBS HD,SB\ S World Movies,SBSVICELANDHD,Sky News,TVSN,WIN,WIN Bold,WIN HD,WIN Peach,WINGOLD

I wasn't sure what was the best way to get this change incorporated into the latest shepherd.

Cheers,

Doug

Title naming differences between grabbers

This issue arises from issue number 13.

While trying to get World Movies and SBS Viceland HD data working I have noticed that some titles are named differently between the OzTivo source and the Rex source, causing some recording rules to fail the matching process.
So far I have found 2

OzTivo: "Q&A"
Rex: "Q & A"

OzTivo: "Have You Been Paying Attention?"
Rex: "Have You Been Paying Attention"

If I discover others (some of my rules are for programs that are not currently showing) I shall post them here.

Please add 9Rush to channel_list.yml

According to Wikipedia the regions supported are:

• 73:
  state: NSW
  name: "Sydney"

• 94:
  state: VIC
  name: "Melbourne"

• 75:
  state: QLD
  name: "Brisbane"

• 255:
  state: QLD
  name: "Sunshine Coast"

• 81:
  state: SA
  name: "Adelaide"

• 101:
  state: WA
  name: "Perth"
  timezone: Australia/Perth

• 342:
  state: WA
  name: Mandurah

[grabber] acctv_website

Needs rewrite, changed from HTML to json day files.

Previous status file line

grabber         acctv_website       0.09

symlink not created during --configure-mythtv process

Hi,

Thanks for this software. I'm trying to get it running inside a Docker container using a separate system user called shepherd. When I run the configuration process, it seems like it's trying to do the right things but somehow failing to do so:

$ whoami
shepherd
$ pwd
/home/shepherd/.shepherd
$ ./tv_grab_au --configure-mythtv
shepherd v1.9.19 (linux)

Reading configuration file: /home/shepherd/.shepherd/shepherd.conf
Reading channels file: /home/shepherd/.shepherd/channels.conf

Configuring MythTV...

This will:
1. Create a symbolic link to Shepherd from tv_grab_au
2. Register Shepherd with MythTV as the default grabber
3. Turn off MythTV-driven scheduling of guide data updates
4. Turn off EIT
5. Create a cron job to periodically run Shepherd.

Step 1: Setting up symlink...
------

Shepherd would like to CREATE the following symlink:

 /usr/bin/tv_grab_au -> /home/shepherd/.shepherd/applications/shepherd/shepherd

Create symlink ? [yes,no (default=yes)] 
Symlink established:
total 64
drwxrwxr-x 3 shepherd shepherd  4096 Nov 25 10:58 applications
drwxrwxr-x 2 shepherd shepherd  4096 Nov 25 10:58 archive
-rw-rw-r-- 1 shepherd shepherd  1540 Nov 25 12:48 channels.conf
-rw-rw-r-- 1 shepherd shepherd    54 Nov 25 12:50 cron.bak
-rw-rw-r-- 1 shepherd shepherd    54 Nov 25 12:50 cron.new
drwxrwxr-x 7 shepherd shepherd  4096 Nov 25 10:58 grabbers
drwxrwxr-x 2 shepherd shepherd  4096 Nov 25 11:13 log
drwxrwxr-x 6 shepherd shepherd  4096 Nov 25 10:58 postprocessors
drwxrwxr-x 3 shepherd shepherd  4096 Nov 25 10:58 reconcilers
drwxrwxr-x 5 shepherd shepherd  4096 Nov 25 10:58 references
lrwxrwxrwx 1 shepherd shepherd    30 Nov 25 10:58 shepherd -> applications/shepherd/shepherd
-rw-rw-r-- 1 shepherd shepherd 22793 Nov 25 12:48 shepherd.conf
lrwxrwxrwx 1 shepherd shepherd    30 Nov 25 10:58 tv_grab_au -> applications/shepherd/shepherd

(further output snipped)

$ ls /usr/bin/tv_grab_au
ls: cannot access '/usr/bin/tv_grab_au': No such file or directory

This has later negative consequences for mythfilldatabase:

023-11-25 12:53:29.206827 E  tv_grab_au  --capabilities failed or we timed out waiting. You may need to upgrade your xmltv grabber

I can manually create the symlink as root:

$ whoami
root
$ ln -s /home/shepherd/.shepherd/tv_grab_au /usr/bin/tv_grab_au

After this, running mythfilldatabase works as expected.

According to the installation instructions, I can use a separate user without issue, but it seems to be causing a problem here. Am I doing something wrong, or is there an issue with the installation process?

Discrepancy between output XMLTV file path and mythfilldatabase's path

Hi,

I'm trying to understand why guide data is not updated. I can see that shepherd is successfully creating the output.xmltv file:

$ ls -al /home/shepherd/.shepherd/output.xmltv
-rw-rw-r-- 1 shepherd shepherd 6871855 Jan 20 18:23 /home/shepherd/.shepherd/output.xmltv

However, mythfilldatabase never imports it (even after waiting over a day). The cron job is configured as:

$ whoami
shepherd
$ crontab -l
21 * * * * nice /usr/bin/mythfilldatabase > /dev/null

If I execute mythfilldatabase manually I get:

$ /usr/bin/mythfilldatabase
2024-01-21 11:21:52.121888 C  mythfilldatabase version: master [v34.0~master.202401182304.3ded86daee~ubuntu23.10.1] www.mythtv.org
2024-01-21 11:21:52.121901 C  Qt version: compile: 5.15.10, runtime: 5.15.10
2024-01-21 11:21:52.121940 I  Ubuntu 23.10 (x86_64)
2024-01-21 11:21:52.121942 N  Enabled verbose msgs:  general
2024-01-21 11:21:52.121951 N  Setting Log Level to LOG_INFO
2024-01-21 11:21:52.132248 I  Added logging to the console
2024-01-21 11:21:52.132830 I  Setup Interrupt handler
2024-01-21 11:21:52.132835 I  Setup Terminated handler
2024-01-21 11:21:52.132838 I  Setup Segmentation fault handler
2024-01-21 11:21:52.132841 I  Setup Aborted handler
2024-01-21 11:21:52.132845 I  Setup Floating point exception handler
2024-01-21 11:21:52.132848 I  Setup Illegal instruction handler
2024-01-21 11:21:52.132851 I  Setup Bus error handler
2024-01-21 11:21:52.132859 I  Setup Real-time signal 0 handler
2024-01-21 11:21:52.132864 I  Setup Hangup handler
2024-01-21 11:21:52.133088 N  Using runtime prefix = /usr
2024-01-21 11:21:52.133090 N  Using configuration directory = /home/shepherd/.mythtv
2024-01-21 11:21:52.133286 I  Assumed character encoding: en_US.UTF-8
2024-01-21 11:21:52.133646 I  Empty LocalHostName. This is typical.
2024-01-21 11:21:52.133659 I  Using a profile name of: '30b1d172d44c' (Usually the same as this host's name.)
2024-01-21 11:21:52.133841 I  Start up testing connections. DB mythdatabase, BE , attempt 0, status dbAwake, Delay: 2000
2024-01-21 11:21:53.166760 N  Setting QT default locale to en_AU
2024-01-21 11:21:53.166771 I  Current locale en_AU
2024-01-21 11:21:53.166819 E  No locale defaults file for en_AU, skipping
2024-01-21 11:21:53.168341 I  Loading en_us translation for module mythfrontend
2024-01-21 11:21:53.169924 I  Current MythTV Schema Version (DBSchemaVer): 1379
2024-01-21 11:21:53.172820 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 192.168.10.211:6543 (try 1 of 1)
2024-01-21 11:21:53.173637 I  MythCoreContext::CheckProtoVersion(): Using protocol version 91 BuzzOff
2024-01-21 11:21:53.173892 I  Opening blocking connection to master backend
2024-01-21 11:21:53.174658 I  Updating source #1 (Free-to-air) with grabber tv_grab_au
2024-01-21 11:21:53.175161 I  Found 38 channels for source 1 which use grabber
2024-01-21 11:21:53.175306 I  Starting process manager
2024-01-21 11:21:53.175346 I  Starting process signal handler
2024-01-21 11:21:53.175400 I  Starting IO manager (read)
2024-01-21 11:21:53.175460 I  Starting IO manager (write)
2024-01-21 11:21:53.626223 I  Grabber has capabilities: baseline manualconfig preferredmethod
2024-01-21 11:21:54.076984 I  Grabber prefers method: allatonce
2024-01-21 11:21:54.077848 I  XMLTV config file is: /home/shepherd/.mythtv/Free-to-air.xmltv

Note the final line, which points to a completely different XMLTV file.

Here is my video source as configured in MythTV:

So I guess mythfilldatabase is somehow defaulting to using the source's name rather than a default...? In an attempt to find why mythfilldatabase is trying to import from the wrong XMLTV file, I tried:

root@30b1d172d44c:/# find / -type f -exec grep -li 'Free-to-air' {} + 2>/dev/null
/home/shepherd/.shepherd/references/Shepherd/Configure.pm

But the found shepherd script is just responsible for importing data from the Free-to-air source, and when it configures the mythfilldatabase cron job, it does so without any arguments.

So I'm confused. I could probably just manually edit the cron job to explicitly specify the correct file path, but I wanted to understand where things have gone wrong first. Any ideas?

Infinite loop with IO::Socket::SSL >= 2.076

The die override here has some undesirable interaction with a change in IO::Socket::SSL 2.076

shepherd/applications/shepherd

Lines 3110 to 3130 in 4c75eee

	sub my_die {
	my ($arg,@rest) = @_;
	my ($pack,$file,$line,$sub) = caller(0);
	# check if we are in an eval()
	if ($^S) {
	printf STDERR "* Caught a die() within eval{} from file $file line $line\n";
	if ($arg){
	print STDERR $arg;
	} else {
	print STDERR join("",@rest)
	}
	} else {
	printf STDERR "\nDIE: line %d in file %s\n",$line,$file;
	if ($arg) {
	CORE::die($arg,@rest);
	} else {
	CORE::die(join("",@rest));
	}
	}
	}

Specifically the change at noxxi/p5-io-socket-ssl@748ad39#diff-d1cdb9f512f51b16007345ca2037a640735689038dd631b9c79cd8d953334309R3479

This can be reproduced minimally with the following:

#!/usr/bin/env perl

BEGIN { *CORE::GLOBAL::die = \&my_die; }

use Net::SSLeay;

my %trace_constants = map { $_ => eval { Net::SSLeay->$_ } || -1 } qw(
    DTLS1_VERSION
);

print %trace_constants;

# ugly hack. please don't try this at home kids!
sub my_die {
    my ($arg,@rest) = @_;
    my ($pack,$file,$line,$sub) = caller(0);

    # check if we are in an eval()
    if ($^S) {
        printf STDERR "* Caught a die() within eval{} from file $file line $line\n";
            if ($arg){
                    print STDERR $arg;
            } else {
                    print STDERR join("",@rest)
            }
    } else {
                printf STDERR "\nDIE: line %d in file %s\n",$line,$file;
            if ($arg) {
                        CORE::die($arg,@rest);
            } else {
                        CORE::die(join("",@rest));
            }
    }
}

The my %trace_constants = map { $_ => eval { Net::SSLeay->$_ } || -1 } qw(DTLS1_VERSION); will infinite loop since Net::SSLeay->DTLS1_VERSION will try to die. I don't really understand why...

A temporary workaround is to add a CORE::die above (not sure what bad effects that could have).

diff --git a/applications/shepherd b/applications/shepherd
index 33d0144..df36775 100755
--- a/applications/shepherd
+++ b/applications/shepherd
@@ -3119,6 +3119,7 @@ sub my_die {
            } else {
                    print STDERR join("",@rest)
            }
+        CORE::die;
     } else {
                printf STDERR "\nDIE: line %d in file %s\n",$line,$file;
            if ($arg) {

SBS Food guide data

there appears to be a new channel, SBS Food
https://www.sbs.com.au/guide/channel/SBSFood
Channel 33. as far as I can tell Shepherd does not have data for this yet

Need to update channels for regional WIN-SCA (9-10) change

Thanks so much for Shepherd!
As of July 1, WIN and Southern Cross Austereo have swapped content providers and so channels 9 and 10 are swapped around (or something like that), see e.g. https://www.canberratimes.com.au/story/7316079/july-1s-tv-flip-what-you-need-to-know/
I've done my best guess to match what's available but can't find any way to update these properly since channels like "TEN" or "10" or "10 Shake" etc. are missing from options when I use guided channel config. Should I be able to update the available channels myself?
Please update channels for appropriate regions, or let me know how I can, thanks!

Here's the channels available when I run --configure:

Guide data sources:
( 0) (no guide)                    (16) RACING.COM                    
( 1) 7TWO                          (17) SBN                           
( 2) 7flix                         (18) SBS                           
( 3) 7mate                         (19) SBS Food                      
( 4) 9HD                           (20) SBS HD                        
( 5) 9Life                         (21) SBS World Movies              
( 6) ABC Kids/ABC TV Plus          (22) SBSVICELANDHD                 
( 7) ABC TV                        (23) Seven                         
( 8) ABC TV HD                     (24) Sky News                      
( 9) ABCME                         (25) TVSN                          
(10) ABCNEWS                       (26) WIN                           
(11) Aspire                        (27) WIN Bold                      
(12) GEM                           (28) WIN HD                        
(13) GO!                           (29) WIN Peach                     
(14) NITV                          (30) WINGOLD                       
(15) Nine                          

And here's my config:

~  ~/.shepherd/shepherd --show-config  
shepherd v1.9.14 (linux)

Reading configuration file: /home/lindsay/.shepherd/shepherd.conf
Reading channels file: /home/lindsay/.shepherd/channels.conf

Configuration
-------------
Config file: /home/lindsay/.shepherd/shepherd.conf
Debug mode : No
Output file: None
Region ID  : 257

Attempting Mysql connection to MythTV database mythconverg.
Looking for MythTV DB connection info in /home/lindsay/.mythtv/config.xml.
Using MythTV DB settings from: /home/lindsay/.mythtv/config.xml

Region 257. 22 MythTV channels. 21 Shepherd channels.

   #  MythTV Channel                 Shepherd Guide Data
 --------------------------------------------------------
   7  7 Townsville                   <- Seven
  20  ABCTV HD                       <- ABC TV HD
  22  ABCKids/TVPlus                 <- ABC Kids/ABC TV Plus
  23  ABC ME                         <- ABCME
  24  ABC NEWS                       <- ABCNEWS
  30  SBS ONE HD                     <- SBS HD
  31  SBS VICELAND HD                <- SBSVICELANDHD
  32  SBS World Movies               <- SBS World Movies
  33  SBS Food                       <- SBS Food
  50  10 HD Townsville               <- WIN
  52  10 Peach                       <- WIN Peach
  53  10 BOLD                        <- WIN Bold
  54  10 SHAKE                       <- -
  55  SBN                            <- SBN
  56  Aspire                         <- Aspire
  72  7TWO Townsville                <- 7TWO
  74  7mateHD Townsville             <- 7mate
  76  7flix Townsville               <- 7flix
  80  9HD                            <- 9HD
  81  9Gem                           <- GEM
  82  9Go!                           <- GO!
  83  Sky News on WIN                <- Sky News


                         Enabled/
 Grabber         Version Ready  Last Run  Status
 --------------- ------- ----- ---------- -------------------------------------
 oztivo             2.65 Y/Y     10:14pm  16ch/533pr/376hrs Tue06Jul-Wed07Jul
 abc_website        7.04 Y/Y     10:13pm  1ch/29pr/23hrs Tue06Jul-Tue06Jul
 rex             10.4.12 Y/Y     10:13pm  16ch/485pr/355hrs Tue06Jul-Wed07Jul
 sbsweb             5.05 Y/Y     12:30pm  3ch/695pr/576hrs Tue06Jul-Wed14Jul
 foxtel_swf         5.04 N/Y   Sat 01-May 15ch/4653pr/3888hrs Sat01May-Sun09May

 Reconciler
 --------------- ------- ----- ---------- -------------------------------------
 reconciler_mk2     0.58 Y/Y     10:14pm  19ch/609pr/447hrs Tue06Jul-Wed07Jul

 Postprocessor
 --------------- ------- ----- ---------- -------------------------------------
 the_movie_db_a..    1.0 Y/Y     10:17pm  19ch/609pr/447hrs Tue06Jul-Wed07Jul
 metacritic          1.1 Y/Y     10:16pm  19ch/609pr/447hrs Tue06Jul-Wed07Jul
 augment_timezone   0.30 Y/Y     10:14pm  19ch/609pr/447hrs Tue06Jul-Wed07Jul
 add_channel_ic..    0.4 Y/Y     10:14pm  19ch/609pr/447hrs Tue06Jul-Wed07Jul
Shepherd last ran successfully 11h1m ago and acquired 76.59% of data.
Shepherd last autorefreshed 1h27m ago.

Shepherd Integration with Plex

Now that Plex has stuffed the EPG data for Australian users I have come back to Shepherd for a look. I used it for many years with MythTV and I found it to be a great source of EPG data.

I installed it and it runs, reconciles data correctly but it doesn't work nicely with Plex. All programs are detected as "Movies" unless episode data is included. This means that the PVR functions incorrectly when you want to record multiple episodes because if it is a movie you only need to record it once.

There is another project which makes a xmltv file with the correct format here:
https://github.com/markcs/freeview_xmltv

I will start looking at how to add this function myself but I have limited time, and I imagine there are a lot of other people in the same position with Plex and I wanted to check what other people are doing.

Shepherd fails - top of the list

I'm seeing a lot of error exit codes in the log, which probably explains why I'm not seeing any EPG data in mythtv.

Top of the list is this, which is meaningless to me except for the word "FAIL".

Appear to have previously downloaded Shepherd/MythTV.pm v0.42.
Testing reference Shepherd/MythTV.pm ...
Executing: perl -e 'require "/home/pierre/.shepherd/references/Shepherd/MythTV.pm-0.42";'
Return value: 2
FAIL.

Any help on this?

Thanks.

Require grabbers to correctly handle timezones

Currently timezones are handled by the augment_timezone post processor. If the source containing augment_timezone goes down but a separate source with grabbers remains then it is possible to get output but with wrong timezones. We should be requiring grabbers to output timezone correct data and honour it through the shepherd program chain and retire augment_timezone.

UTF-8

I think that the grabbers should be modified to output utf-8, instead of iso-8859-1, as there are some characters that aren't encodable in such.

It's rare but as an example, some SBS texts contain typographical quotes (U+2018, U+2019), which don't exist directly in the iso encoding.

Cannot fork shepherd.wiki.git

Hi

I can clone this https://github.com/ShephedProject/shepherd.wiki.git fine but don't seem to be able to fork it so I can send PRs.
Is this difficult to fix?

the --set-icons option doesn't get logo_list.txt correctly

Hi,

I used the --set-icons option to get TV channel icons, and noticed that shepherd attempts to download the logo_list.txt web page from github, instead of the raw logo_list.txt file. Here is the patch that fixes this issue:

--- postprocessors/add_channel_icons/add_channel_icons.orig 2020-10-27 17:56:24.973306809 +1000
+++ postprocessors/add_channel_icons/add_channel_icons 2020-10-27 17:56:53.033496924 +1000
@@ -148,7 +148,7 @@
sub load_logo_list
{
# fetch icon styles

• $icon_styles = &Shepherd::Common::get_url('http://github.com/ShephedProject/shepherd/blob/release/logo_list.txt');

• $icon_styles = &Shepherd::Common::get_url('http://raw.githubusercontent.com/ShephedProject/shepherd/release/logo_list.txt');
  exit 1 unless ($icon_styles);

  foreach my $line (split/\n/,$icon_styles)

Cheers!

10 Shake data not scrapping for Regions 101 and 73

I have configured two copies of shepherd under two different users to grab data from regions 73 and 101. Since the WIN/Austereo changes on the 1st of July, I have not received any program data for 10 Shake. All other channels are grabbing properly.

Here is my config for region 101:

mythtv@zocalo:~$ tv_grab_au --show-config
shepherd v1.9.14 (linux)

Reading configuration file: /home/mythtv/.shepherd/shepherd.conf
Reading channels file: /home/mythtv/.shepherd/channels.conf

Configuration
-------------
Config file: /home/mythtv/.shepherd/shepherd.conf
Debug mode : No
Output file: None
Region ID  : 101

Attempting Mysql connection to MythTV database mythconverg.
Looking for MythTV DB connection info in /home/mythtv/.mythtv/config.xml.
Using MythTV DB settings from: /home/mythtv/.mythtv/config.xml

Region 101. 50 MythTV channels. 14 Shepherd channels.

   #  MythTV Channel                 Shepherd Guide Data
 --------------------------------------------------------
   1  10 Bold                        <- 10 Bold
   2  ABC                            <- ABC TV
   3  SBS                            <- -
   4  Seven                          <- Seven
   5  Channel 9                      <- Nine
   6  TEN                            <- 10
   7  10 Peach                       <- 10 Peach
   8  10 Shake Perth                 <- 10 Shake
   9  ABC KIDS                       <- -
  10  ABC TV Plus                    <- -
  11  ABC Me                         <- -
  12  ABC News                       <- -
  13  SBS VICELAND                   <- -
  14  SBS World Movies               <- -
  15  SBS Food                       <- -
  16  NITV                           <- -
  17  7TWO                           <- 7TWO
  18  7mate                          <- 7mate
  20  7flix                          <- 7flix
  21  Fuel TV                        <- -
  22  9Gem                           <- 9Gem
  23  9Go!                           <- 9Go!
  24  9Life                          <- 9Life
  25  9Rush Perth                    <- 9Rush
 201  10 Bold+2                      <- -
 202  ABC+2                          <- -
 204  Seven+2                        <- -
 205  Channel 9+2                    <- -
 206  TEN+2                          <- -
 207  10 Peach+2                     <- -
 208  10 Shake Perth+2               <- -
 217  7TWO+2                         <- -
 218  7mate+2                        <- -
 220  7flix+2                        <- -
 221  Fuel TV+2                      <- -
 222  9Gem+2                         <- -
 223  9Go!+2                         <- -
 224  9Life+2                        <- -
 225  9Rush Perth+2                  <- -
 901  ACCTV                          <- -
 902  ausbiz TV                      <- -
 903  Channel News Asia (CNA) (Channel News Asia (C) <- -
 904  Olympics Channel               <- -
 905  Sky Racing #1                  <- -
 906  Sky Racing #2                  <- -
 907  Sky Racing - Thoroughbred Central (Sky Racing - Thoroug) <- -
 908  Redbull TV                     <- -
 909  Olympics Channel+2             <- -
 910  ABC Me+2                       <- -
 911  ABC News+2                     <- -


                         Enabled/
 Grabber         Version Ready  Last Run  Status
 --------------- ------- ----- ---------- -------------------------------------
 abc_website        7.04 Y/Y      4:17am  2ch/59pr/48hrs Sun11Jul-Tue13Jul
 oztivo             2.65 Y/Y      4:17am  12ch/1472pr/1157hrs Wed07Jul-Wed14Jul
 rex             10.4.12 Y/Y      4:15am  12ch/2428pr/1987hrs Wed07Jul-Wed14Jul
 sbsweb             5.05 Y/Y   Wed 24-Mar 4ch/1021pr/770hrs Wed24Mar-Thu01Apr
 foxtel_swf         5.04 Y/Y   Wed 24-Mar 16ch/3698pr/2929hrs Tue23Mar-Mon29Mar

 Reconciler
 --------------- ------- ----- ---------- -------------------------------------
 reconciler_mk2     0.58 Y/Y      4:17am  13ch/2639pr/2165hrs Tue06Jul-Wed14Jul

 Postprocessor
 --------------- ------- ----- ---------- -------------------------------------
 the_movie_db_a..    1.0 Y/Y      4:26am  13ch/2639pr/2165hrs Tue06Jul-Wed14Jul
 metacritic          1.1 Y/Y      4:24am  13ch/2639pr/2165hrs Tue06Jul-Wed14Jul
 augment_timezone   0.30 Y/Y      4:18am  13ch/2639pr/2165hrs Tue06Jul-Wed14Jul
 add_channel_ic..    0.4 Y/Y      4:17am  13ch/2639pr/2165hrs Tue06Jul-Wed14Jul
Shepherd last ran successfully 6h23m ago and acquired 79.76% of data.

This is my config for region 73:

plustwo@zocalo:~$ tv_grab_au --show-config
shepherd v1.9.14 (linux)


*** Application/user mismatch ***
    You invoked: /home/mythtv/.shepherd/applications/shepherd/shepherd
    Instead of : /home/plustwo/.shepherd/applications/shepherd/shepherd

*** Restarting ***

shepherd v1.9.14 (linux)

Reading configuration file: /home/plustwo/.shepherd/shepherd.conf
Reading channels file: /home/plustwo/.shepherd/channels.conf

Configuration
-------------
Config file: /home/plustwo/.shepherd/shepherd.conf
Debug mode : No
Output file: None
Region ID  : 73

Attempting Mysql connection to MythTV database mythconverg.
Looking for MythTV DB connection info in /home/plustwo/.mythtv/config.xml.
Looking for MythTV DB connection info in /etc/mythtv/config.xml.
Using MythTV DB settings from: /etc/mythtv/config.xml

Region 73. 50 MythTV channels. 14 Shepherd channels.

   #  MythTV Channel                 Shepherd Guide Data
 --------------------------------------------------------
   1  10 Bold                        <- -
   2  ABC                            <- -
   3  SBS                            <- -
   4  Seven                          <- -
   5  Channel 9                      <- -
   6  TEN                            <- -
   7  10 Peach                       <- -
   8  10 Shake Perth                 <- -
   9  ABC KIDS                       <- -
  10  ABC TV Plus                    <- -
  11  ABC Me                         <- -
  12  ABC News                       <- -
  13  SBS VICELAND                   <- -
  14  SBS World Movies               <- -
  15  SBS Food                       <- -
  16  NITV                           <- -
  17  7TWO                           <- -
  18  7mate                          <- -
  20  7flix                          <- -
  21  Fuel TV                        <- -
  22  9Gem                           <- -
  23  9Go!                           <- -
  24  9Life                          <- -
  25  9Rush Perth                    <- -
 201  10 Bold+2                      <- 10 Bold
 202  ABC+2                          <- ABC TV
 204  Seven+2                        <- Seven
 205  Channel 9+2                    <- Nine
 206  TEN+2                          <- 10
 207  10 Peach+2                     <- 10 Peach
 208  10 Shake Perth+2               <- 10 Shake
 217  7TWO+2                         <- 7TWO
 218  7mate+2                        <- 7mate
 220  7flix+2                        <- 7flix
 221  Fuel TV+2                      <- -
 222  9Gem+2                         <- 9Gem
 223  9Go!+2                         <- 9Go!
 224  9Life+2                        <- 9Life
 225  9Rush Perth+2                  <- 9Rush
 901  ACCTV                          <- -
 902  ausbiz TV                      <- -
 903  Channel News Asia (CNA) (Channel News Asia (C) <- -
 904  Olympics Channel               <- -
 905  Sky Racing #1                  <- -
 906  Sky Racing #2                  <- -
 907  Sky Racing - Thoroughbred Central (Sky Racing - Thoroug) <- -
 908  Redbull TV                     <- -
 909  Olympics Channel+2             <- -
 910  ABC Me+2                       <- -
 911  ABC News+2                     <- -


                         Enabled/
 Grabber         Version Ready  Last Run  Status
 --------------- ------- ----- ---------- -------------------------------------
 abc_website        7.04 Y/Y      4:18am  1ch/20pr/25hrs Fri09Jul-Sat10Jul
 oztivo             2.65 Y/Y      4:18am  11ch/1416pr/1109hrs Wed07Jul-Wed14Jul
 rex             10.4.12 Y/Y      4:16am  12ch/2420pr/1986hrs Wed07Jul-Wed14Jul
 sbsweb             5.05 Y/Y        -
 foxtel_swf         5.04 Y/Y        -

 Reconciler
 --------------- ------- ----- ---------- -------------------------------------
 reconciler_mk2     0.58 Y/Y      4:18am  13ch/2630pr/2159hrs Tue06Jul-Wed14Jul

 Postprocessor
 --------------- ------- ----- ---------- -------------------------------------
 the_movie_db_a..    1.0 Y/Y      4:25am  13ch/2630pr/2159hrs Tue06Jul-Wed14Jul
 metacritic          1.1 Y/Y      4:23am  13ch/2630pr/2159hrs Tue06Jul-Wed14Jul
 augment_timezone   0.30 Y/Y      4:18am  13ch/2630pr/2159hrs Tue06Jul-Wed14Jul
 add_channel_ic..    0.4 Y/Y      4:18am  13ch/2630pr/2159hrs Tue06Jul-Wed14Jul
Shepherd last ran successfully 8h29m ago and acquired 79.95% of data.

Thanks.