Generating dynamic teletext pages such as weather and PC usage etc

[SonnyWalkman]

Generating dynamic teletext pages such as weather and PC usage etc

[boomlinde]

For now this only does the very basics of encoding and streaming the encoded data, but in principle you can construct a page from scratch instead of importing tti files.

Each Page is an array of Line. I have two basic types that satisfy this interface, but you can create more types that do, too.

I would look at linetypes.go. There I have the two types of lines. One way to approach this problem is to create a new type that satisfies the Line interface but encodes some kind of dynamic data via a callback that is called when Serialize is called.

Another approach is to simply dynamically generate entire pages using the existing OutputLine type.

The carousel command can be adapted so that some pages are simply static .tti files and some pages are dynamically generated.

Problem then is likely scheduling. I believe that if you have a bunch of pages, it can take a few seconds before the whole carousel is finished, so if you want something that updates every second, you can schedule to retransmit the "live" pages more often than others. You can implement this using a priority queue.

Unfortunately I don't have a good setup to test this with right now, and I will be busy for a while, but if you come up with some good idea or a tool to achieve this, I'm happy to accept pull requests once I do. A good scheduling system is definitely something I'm interested in, and some kind of abstraction over characters/control codes would be very useful.

[SonnyWalkman]

Hello @boomlinde ,
Thanks for the tip on using your code.
I'm changing the header structure to include the date time (right side) however unable to get it updating as I'd like.

I'll keep working on adding dynamic page in making it faster with a go routine.

[SonnyWalkman]

Phil, Although basic, your code is easy to read and lends itself for expansion. One of the favorable benefits of writing in GO is code written in C/ C++ and python is easy enough to translate using slices and maps and goroutines. My 6 months of coding in GO has opened doors to being able to do what I require.. I've adapted your code for a first pass teletext encoder to read in a couple of 100.tti, 898.tti pages. Simple index and an engineering page however, the page/s doesn't seem to update in a loop and add my modified header for the Date Time? Example "100 Teletext Jan 01 2023 10:00:56 " The issue is more than likely my code which reads the T42 into a buffer and I read the buffer and add a fake PTS and produce the PES before mutiplexing into the DVB transport stream. Do you have an example for generating a dynamic text page handy? Appreciate your code. it's provides a sound base. Cheers

…

On Mon, 6 Nov 2023 at 04:22, Philip Linde ***@***.***> wrote: For now this only does the very basics of encoding and streaming the encoded data, but in principle you can construct a page from scratch instead of importing tti files. Each Page is an array of Line. I have two basic types that satisfy this interface, but you can create more types that do, too. I would look at linetypes.go. There I have the two types of lines. One way to approach this problem is to create a new type that satisfies the Line interface but encodes some kind of dynamic data via a callback that is called when Serialize is called. Another approach is to simply dynamically generate entire pages using the existing OutputLine type. The carousel command can be adapted so that some pages are simply static .tti files and some pages are dynamically generated. Problem then is likely scheduling. I believe that if you have a bunch of pages, it can take a few seconds before the whole carousel is finished, so if you want something that updates every second, you can schedule to retransmit the "live" pages more often than others. You can implement this using a priority queue. Unfortunately I don't have a good setup to test this with right now, and I will be busy for a while, but if you come up with some good idea or a tool to achieve this, I'm happy to accept pull requests once I do. A good scheduling system is definitely something I'm interested in, and some kind of abstraction over characters/control codes would be very useful. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM4O2S2WSGT2UPGGU3J2TOLYC7DORAVCNFSM6AAAAAA6SHDQU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJTG44TMNJRGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[boomlinde]

Wow, that's cool! I never anticipated that this would be used for DVB.

The carousel command only does a single pass, encoding all input pages once before quitting. When I used it, I ran it repeatedly in a loop from the shell to achieve timely updates when I wanted to swap out/add new pages.

For dynamic pages, you can still use the carousel command to some extent. For example, if there are some pages page you want regenerate entirely to update refresh between other pages, you can script the command:

while true; do
    generate_page_103.sh > dynamic/103.tti
    carousel static/*.tti
    carousel dynamic/*.tti
done

If you have a lot of static pages and use this technique, you may want to interleave the dynamic pages with the static pages. At that point it might be worth it to invest in making a scheduler. I'll see if I can get my setup running again this weekend and tinker with some higher level abstraction in the library to achieve a more dynamic carousel.

As for updating something like a shared header with the current date and time, at least in principle, teletext allows you to update any line of any page in whatever order you like. I imagine a commercial broadcaster will refresh the header lines often in particular, all in one go. The carousel command now goes through each line of each page in order, but the library itself can also be used to encode single lines or single pages in arbitrary order. I don't have any code for dynamic lines, but I achieved animation playback by updating the same entire page over and over in github.com/boomlinde/televj. There's unfortunately no documentation, but IIRC the tool waits for you to either press z, at which point it plays an animation (don't remember which page) or escape, at which point it quits.

[SonnyWalkman]

Hello @boomlinde
Thanks again Phil for taking the time to answer my questions.
Wow-Yes I'm hoping to enhance you code to develop a scheduler for DVB Teletext to use with my other DVB projects.
The issue I'm facing is in producing the correct PES with header and PTS which align with the DVB standards for Teletext.

If you have a lot of static pages and use this technique, you may want to interleave the dynamic pages with the static pages. At that point it might be worth it to invest in making a scheduler. I'll see if I can get my setup running again this weekend and tinker with some higher-level abstraction in the library to achieve a more dynamic carousel.

Agree, I'm working towards a magazine and page scheduler using 1-2 goroutines with a ticker to convert pages from MGR tti format int a bit ready T42 binary buffer to squirt into a PES.

My idea is to do the time intensive tti conversion "before" then once the Teletext buffer is complete create the DVB PES complete with PES header PTS (time stamp) then the teletext buffer at a set time interval by the ticker (100ms) as a starting point.

Having the scheduler operate in this way allows for priorities what pages are sent regularly reducing the load in decoding TTI and the formation of a PES packets Multiple pages are sent within the PES. Basically cued pages and sub pages to send out in succession.

I believe the MGR TTI standard contains cycle time "CT" - Cycle time (seconds) .
Could give flexible way to sorting out magazines/pages and subpages priority?

The issue is how to create the PES in a Golangnic way generating the DVB PES header, calculate the PTS and send a complete PES with multiple text buffers?

Below is the packet formation from the service.cpp from VBIT2.. which is C++. Go lends itself well to byte Slices and Maps.
The code blow doesn't generate the PTS however is there ready waiting. the PTS is 33 bits long. Uncomment the PTS calculation section.

To use PTS the flag byte of 0x00 (no PTS) needs to be 0x80 (use PTS)

line header.push_back(0x00); // No PTS to header.push_back(0x80); //Yes PTS

for the decoder to act on the PTS (Presentation Time Stamp). It's recommended in the Spec's to use PTS in encoding and for the decoder to ignore it if it's not required.

 /* Packetized Elementary Stream for insertion into MPEG-2 transport stream */
            
            if (_lineCounter == 0)
            {
                // a new field has started - transmit data for previous field if there is any
                if (!(_PESBuffer.empty()))
                {
                    std::array<uint8_t, 46> padding;
                    padding.fill(0xff);
                    
                    std::vector<uint8_t> header = {0x00, 0x00, 0x01, 0xBD};
                    
                    int numBlocks = _PESBuffer.size() + 1; // header and N lines
                    int numTSPackets = ((numBlocks * 46) + 183) / 184; // round up
                    int packetLength = (numTSPackets * 184) - 6;
                    
                    header.push_back(packetLength >> 8);
                    header.push_back(packetLength & 0xff);
                    
                    /* bits | 7 | 6 |  5   | 4   |     3    |     2     |     1     |     0    |
                            | 1 | 0 | Scrambling | Priority | Alignment | Copyright | Original | */
                    header.push_back(0x85); // Align, Original
                    
                    /* bits |  7 | 6  |   5  |    4    |     3     |     2     |    1    |       0       |
                            | PTS DTS | ESCR | ES rate | DSM trick | copy info | PES CRC | PES extension |*/
                    header.push_back(0x00); // No PTS
                    
                    header.push_back(0x24); // PES header data length
                    
                    /* 
                    uint64_t PTS = 0; // ???
                    
                    // append PTS
                    header.push_back(0x21 | (PTS >> 30));
                    header.push_back((PTS & 0x3FC00000) >> 22);
                    header.push_back(0x01 | ((PTS & 0x3F8000) >> 14));
                    header.push_back((PTS & 0x7F80) >> 7);
                    header.push_back(0x01 | ((PTS & 0x7F) << 1));
                    */
                    
                    header.resize(0x2D, 0xff); // make PES header up to 45 bytes long with stuffing bytes.
                    
                    header.push_back(0x10); // append PES data identifier (EBU data)
                    
                    std::cout.write((char*)header.data(), header.size()); // output PES header and data_identifier
                    
                    for (unsigned int i = 0; i < _PESBuffer.size(); i++)
                    {
                        std::cout.write((char*)_PESBuffer[i].data(), 46);
                    }
                    
                    for (int i = numBlocks; i < numTSPackets * 4; i++)
                    {
                        std::cout.write((char*)padding.data(), 46); // pad out remainder of PES packet
                    }
                    
                    _PESBuffer.clear(); // empty buffer ready for next frame's packets
                }
            }
            
            std::vector<uint8_t> data = {0x02, 0x2c}; // data_unit_id and data_unit_length (EBU teletext non-subtitle, 44 bytes)
            
            if (_lineCounter > 15)
            {
                data.push_back(((_fieldCounter&1)^1) << 5); //field parity, line number undefined
            }
            else
            {
                data.push_back((((_fieldCounter&1)^1) << 5) | (_lineCounter + 7)); // field parity and line number
            }
            
            for (int i = 2; i < 45; i++)
            {
                data.push_back(ReverseByteTab[p->at(i)]); // bits are reversed in PES stream
            }
            
            _PESBuffer.push_back(data);
            
            break;
        }

Any suggestions are welcome.