Tag Archives: IPL

2000 TSI: AxN: System Design

Posted on by
11

TSI as a nexus between advertisers and newspapers. Continue reading →

The term insertion order was used in advertising to describe a list of detailed information about ads that advertisers provided to newspapers or magazines so that the publication could reserve the space. The design of AxN, TSI’s clearinghouse on the Internet for insertion orders, only really congealed when I had finished reading Bob Cancilla’s outstanding book about using the AS/400 for e-business, IBM’s word for commercial transactions that used the Internet. The process of determining what TSI wanted to do and how we learned how to do it has been described here.

The Internet servers on the AS/400 in TSI’s office were used in the project. The FTP (File Transfer Protocol) server was used to receive transmissions of data files containing insertion orders generated by the AS/400’s at our client. The SMTP (Simple Mail Transfer Protocol) server sent computer-generated emails to both advertisers and newspapers. The HTTP (Hypertext Transfer Protocol) server allowed the posting of web pages on TSI’s website, TSI-TSI.com.

TSI’s AS/400 was connected to the Internet through a T-1 line leased from AT&T. The backup line was a cable connection to Cox. It was seldom, if ever, needed. I can recall no problems with the connection.

AdDept changes: AdDept, TSI’s administrative system for the advertising departments of large retailers, included programs for producing the original insertion orders used to schedule advertising at newspapers and for producing change orders and cancellations. The users at each installation had slightly different methods of selecting the items from the media schedule to be ordered (usually once per week for ROP and once per month for inserts) and different formats for printing or faxing the orders. However, all installations used the same programs for updating the data files with the ordering information.

Newspapers were ordinarily designated on AdDept’s pub table with two “variations”—one for ROP (“run of press”—the pages printed by the newspaper) and one for inserts (the preprinted flyers and coupons inserted with the pages printed by the newspaper). Some needed additional variations for things like the Sunday supplement or the television guide. The pub table contained a field that designated how insertion orders would be delivered. The choices—as designated by a one-character field on the pub table—were faxing or just printing. A new option was added for delivery through TSI’s website on the Internet using AxN. When a paper agreed to a trial subscription for AxN, the field was changed for each of its versions. If the paper stopped using AxN, it was set back to its previous value.

The configuration on each client’s AS/400 was modified so that the FTP server was automatically started every time the system initiated the Initial Program Load (IPL). “IPL” was IBM-speak for booting or rebooting. Most of TSI’s AdDept clients IPLed once a week.

The sections of the insertion programs that updated the data files were augmented. If the pub was using the Internet for insertion orders, new code was executed for each order as a whole and for each item on the order. This code created a file for the order and wrote records on the file for the header (AxN‘s code for the pub, range of dates, type of order, whether it was a new order, revision, or cancellation, and a few additional items), detail (individual ads), and special instructions that could be for individual ads or the whole order. As records were written, they were logged on a file.

When the user closed the insertion order program, a routine was called to use FTP to send a copy of the file to a designated library1 on TSI’s server. The original files remained on the advertiser’s system for a while until the system administrator ran the cleanup program.

FTP clients can actually do a lot more than send and receive files, but the configuration of TSI’s server prohibited most of those features.

Processing the orders: The configuration for TSI’s AS/400 was also changed so that the FTP, SMTP, and HTTP servers started during each IPL. Another program that we wrote to monitor the library containing orders also was started at IPL. In the twelve years that this process was active we had zero failures.

The monitoring program examined the contents of the library designated to receive the files. If the library was empty, the program went to sleep for a minute or two. If it found anything in the library, It copied the files to a separate library. For each file that it found it started a batch job to process the file. It then deleted the files from the receiving library and went back to sleep.

The processing programs evaluated each file to make sure that it could read all the records and that none of the fields in any contained invalid information. Since all of this information had been generated by AdDept programs run on the advertising departments’ AS/400s2, problems almost never occurred. If any anomalies were discovered, an email was sent to me, Denise Bessette, and a programmer.

After validating the data the program created database files for new orders or updated the records of existing orders. Time-stamped log entries were written for all of these operations.

After the permanent database files were updated, an email was sent to the rep at the newspaper and the coordinator at the advertising department. It simply told them that the insertion order had been posted on TSI-TSI.com and provided a link to the website.

The processing programs were written in BASIC and IBM’s Control Language (CL). The former were used to validate the received file and to update the data files. The latter were used for system-oriented tasks such as sending emails and copying files.

I don’t remember the details of how it worked, but I think that there was also a program that monitored the number of FTP transactions in a period of time. Two of three times the system was the target of half-hearted attacks on the FTP server. We were easily able to repel them.

Setup for the interactive programs: The reps at the newspapers and the coordinators at the advertisers were expected to be able to access TSI’s AS/400 through the Internet using a browser. Therefore, it was necessary to change the system’s configuration on our AS/400 so that the HTTP3 server for the website was always started as part of the IPL. At the same time a second local HTTP server for development and testing by TSI programmers (primarily me) was also configured.

Not tsetse, TSI-TSI.com.

Security was provided by the AS/400’s HTTP server. The scripts for all AxN pages on TSI’s website (TSI-TSI.com) were placed in a library that was password-protected. The pages that described the company and its products and services were open to the public. Static pages were written in HTML, the language of the web. The scripts for interactive pages were written in Net.Data, the only scripting language that IBM supported on the AS/400 at the time.

The AS/400 had the ability to set up interactive sessions for users who logged in to the HTTP server. If we had implemented this feature, the system could have timed-out users who had entered their credentials and left the connection open. Implementing this would have complicated the development enormously. We decided not to do it until it became necessary. Like all other developers in that era, we were in a hurry to launch the product on the Internet. Since we never encountered any security issues, we never did.

I take it back. Actually there was one security issue. The reps and coordinators tended to store their credentials in their browsers so that they would be filled in automatically on the security screen. When there was turnover, or someone got a new computer or software update, the passwords and user ID’s were sometimes lost. The support people at TSI did not have the capability of retrieving passwords, but they could assign new ones. After each new one was assigned, it was given over the phone to the user. To insure that this solved the problem, the TSI employee stayed on the line while the user signed on. The user was then advised to change the password and, if necessary, talked through that process.

I am sure that both the newspapers and advertisers were happy that we did not automate this process. We only needed to deal with it a handful of times per month. If we had succeeded at convincing more advertisers to use AxN, we might have needed to automate it.

The interactive programs for newspapers: In the course of research for this entry I discovered the pdf file for the very comprehensive AxN: Handbook for Newspaper Users and posted it here. I had created this twenty-two-page document using PageMaker software in the early 2000’s before we had the first live implementation of the product/service.

The handbook included samples of all the screens and reports. This document was sent to every newspaper when it agreed to participate in the trial period. We provided no training, but no one had much trouble learning to use the system. I must say that I think that I did a superb job. TSI received amazingly few support calls.

The document is more or less self-explanatory. There are just a few things that are worth noting.

The AxN programs used “frames”4. Every interactive screen had three sections that consistently take up the same amount of space on the screen. The “Display” section was always the top area with the light blue background. The “Buttons” area was at the bottom and had a steel blue background. The third frame was “Empty”; that is, it had no space on the screen reserved for it. For each page there was a master script that loads the scripts for the three frames.

When a user clicked on a button, the empty frame was loaded with the appropriate script and executed. So, for example, if the “Confirm” button was clicked on after an order had been selected, the script in the empty frame would check to see if anything was in the “Confirmed by” box. If not it would highlight that field and display an error message without reloading the page. If the field contained characters, the script called a sequence of CL and BASIC programs to mark the order as confirmed and send an email to the coordinator at the advertiser.

The buttons available varied by page. However the Buttons section always contained a Home button (the one that looks like a house) and a Help button (question mark). The former returned the user to the site’s home page. The latter produced a page that explained all of the fields and buttons on the original page.

The interactive programs for advertisers: I also found the pdf file for the twenty-eight page AxN: Handbook for AdDept Users: It explained how to use both the AdDept programs to send new orders, changes, and cancellations and the AxN web pages to make sure that all newspaper reps have confirmed the orders. I posted the handbook here.

Advertisers who made changes to the schedule after an order was sent were expected to make the changes in AdDept and to send revision orders.

Daily tasks: Every morning a program searched for unconfirmed orders. Lists were prepared for both newspapers and advertisers. Each newspaper rep and advertising coordinator was emailed a list of the orders that pertained to them. A master list was also printed at TSI. An employee made a “courtesy” telephone call to warn reps each newspaper with unconfirmed orders that they were within a few days of publication.

Billing: In theory both the advertisers and the newspapers were billed for the use of the AxN service. In almost all cases the advertisers had been using the AS/400 to fax orders to the papers. They had been paying TSI a monthly fee to maintain this software. When they started using AxN we charged the same amount for support and dropped the faxing charge. So, there was a net-zero effect for the advertisers.

TSI’s computer programs required changes to accommodate the newspapers as clients. The main change was to convert the three-digit5 client number field from numeric to character. The existing clients maintained their numbers, but the newspaper clients were assigned alphabetical designations sorted by state.

A separate billing program was written to create invoices for the newspapers. Most were billed quarterly; a few preferred to be billed monthly.

AxN was extremely reliable and was very popular with both newspapers and advertisers. Support calls were rare. I vaguely recollect one instance in which a newspaper failed to run an ad or ran it incorrectly. I was able to document exactly what the system had done with timestamps, and neither side thought that AxN had contributed to the problem.

I definitely wrote all of the Net.Data scripts. Someone else at TSI may have written a few of the BASIC and CL programs used for AxN, but I did the bulk of that coding as well.

1. A “library” in the AS/400’s “native environment” was roughly equivalent to a “folder” or “directory” on a PC. However, libraries on the AS/400 could not be nested. That is, there was no such thing as a sub-library. More details about the AS/400’s design principles can be read here. The AS/400 also supported being used as a server for some other operating systems, but TSI did not use that feature.

2. The header record for the order contained a field that identified the format used for the files. Since no one except AdDept users was ever convinced to use AxN, no code for processing orders in a different format was ever written.

3. HTTP stands for Hypertext Transfer Protocol. HTTP servers were able to serve pages to a browser using the Hypertext Markup Language (HTML) and a few other formats.

4. I really liked the use of frames, but for some reason it was disparaged by supposedly sophisticated developers, and eventually frames were no longer supported by browsers. Instead a complicated use of Cascading Style Sheets (CSS) was necessary to produce a similar effect. AxN was still using frames when TSI closed in 2014. I have never been able to duplicate the effect of the empty frame with CSS.

5. The worst design mistake that I ever made was using a three-digit number for clients. None of our agency clients ever needed a larger field, and the client file on TSI’s system still had many unused client numbers when we began marketing AxN. I decided that it would require less effort to change the type of the client number field from numbers to characters than to expand the size of the field.

1983-1988 The IBM System/36

Posted on by
2

A true multi-user system. Continue reading →

A 5360. The box on the top left is the diskette magazine drive.

IBM’s introduction of the System/36 (S/36) in May of 1983 was not a monumental event for TSI. From our perspective the new system seemed more like a marginal upgrade of the System/34, which had always been much too pricey for anyone who would talk to us. The new system had only one basic model, the 5360. The starting price for one of these was still $140,000. It was also gigantic and had special electrical requirements. It was clearly designed for a small data processing department, not a small business without one.

The biggest advantage of the System/36 over any system that we had worked with was that it supported a fairly large number of terminals and printers. This was because it could run a number of jobs at the same time. It also supported batch processing, which meant that time-consuming jobs need not tie up any workstations.

We appreciated these benefits. In fact, we drooled over them. However, no prospective customer of ours ever had a six-figure budget for hardware.

The 5250 screen showed one color, green.

The peripherals were also rather expensive. IBM in those days ignored standards used in the rest of the industry. It set its own standards, and they were all proprietary. So, cheaper hardware from other vendors would not work with IBM systems.

Although the price went down through the years, a 5250 terminal cost around $2,000 when the 5360 was introduced. The cheapest printers, which used dot-matrix technology, were in the $5,000 range.

Both ends of twinax cables were male. A device with two female ends was needed to connect two together

The cabling was also not cheap. The system used twinaxial cables for direct attachment of these devices. Most competing systems used serial or parallel connections. Twinax was decidedly better, but it was also more expensive.

The local devices were connected in a serial fashion to a controller. Up to seven devices could be connected on one twinax line. Each device had two female twinax connections on the back, either in one T-shaped unit or with two short cables. One was called the “gozinda”; the other was the “gozouta”.

The T-shaped connector.

So, if the S/36 had four devices named A, B, C, and D. A would be connected to the controller by a twinax cable, B would be attached to A by a twinax cable, C would be attached to B, and D would be attached to C.

A switch on device A, B, and C needed to be set to allow pass-through to the next device on the line. On device D that switch needed to be set to denote that it was the last device on the line.

For device D to communicate with the S/36, all of the connections must be functional, and all of the switches set correctly. It reminded me of Christmas tree lights in the old days.

The S/36 also came with a serial port. Since a modem could be attached to this device, it would be possible for support people at IBM or the software vendor (or anyone else, for that matter) to sign on from a remote location. This was, of course, our dream; it would make support so much easier. The announcement brought it a little closer to reality.

The 8809 tape drive.

The hard drive capacity varied from 30 MB to 400 MB. In the twenty-first century, of course, those quantities would only hold a handful of photos. However, to most software vendors in the eighties this meant that total storage was no longer a big concern. However, the default backup device was a magazine that held only ten 8″ diskettes. Each diskette had a capacity of only 1MB each. This was a rather obvious limitation on the practical storage.

It was possible to attach an 8809 1/2″ reel-to-reel tape unit to address this. I could not find a price for these monsters. It may well have been the case that if you had to ask the price, you could not afford it.

The System/36 had two processors. The main processor (MSP) executed the code; the control processor (CSP) managed the work for the main processor. The CSP was four times as fast as the MSP, and they worked perfectly in tandem. The S/36 could perform an amazing amount of work at very fast speeds with embarrassingly puny processors. It was also extremely reliable.

Some actions, such as IPL1 and backing up files, needed to be initiated from the system console. The system console on the 5360 was built in to the top of the box. Your system operator needed a wheelchair? Life was rough all over.

I missed out on this fun stuff.

The System/36 supported five programming languages: RPG II2, COBOL, FORTRAN, BASIC, and Assembler. RPG II, a simplistic column-based language, was the most popular. Wikipedia says that this was because it was the least expensive3. The BASIC language was similar to the one used on the Datamaster. I never heard of anyone who programmed in FORTRAN or Assembler on a S/36.

BASIC programs could not be compiled. Therefore, a copy of the BASIC interpreter needed to be loaded into memory for each program that was running. Nevertheless, because the CSP was so efficient, TSI’s benchmarks showed that there was no noticeable difference in the speed of interpreted BASIC programs and compiled RPG II programs performing the same tasks. So, we never considered converting our program to anything besides S/36 BASIC.

IBM also positioned the System/36 with its DisplayWrite/36 software as a word processing server. We found it to be inferior to the Datamaster product in most ways.

The file structure of the S/36 was somewhat different from the Datamaster’s. Programs and other executable items such as BASIC procedures and system procedures were stored in “libraries”. Libraries were somewhat like directories or folders on a PC. However, there was no such thing as a sub-library.

No directory trees on the S/36.

Data files needed to have unique names. They did not reside in libraries. In order to identify which files belonged to which application, the names were preceded by a short identifier and a period. In GrandAd all the media files started with “M.”, and the other files started with “T.”.

The S/36 did not have a relational database4. However, the fields in each file were defined in the system using field names, positions, and length/type designations. These data definition specifications were called DDS. Programs could access files using the same ISAM techniques with which we were already familiar. However, IBM also offered a product named Query/36 that allowed someone who know how the files were named and structured to write queries in a manner that was similar to SQL. These queries could then be saved and executed on demand.

Query/36 was a valuable debugging tool. TSI required all of our clients to buy it.

IBM announced the 5362 in 1984. It was much smaller than the 5360—only about as large as a two-drawer file cabinets. It also had no special electricity requirements. It used the same operating system as the 5360, which was called SSP. It came with one diskette drive and up to 120MB of hard drives. A QIC (1/4″ cartridge) external tape drive was available. A 5250-type terminal plugged into the first twinax spot served as the system console. Fewer devices could be attached, but none of our clients ever reached the limit. Best of all, the starting price was only $20,000, about the same as two Datamasters and a hard drive.

Many advertising agencies in the northeast could afford this level of investment. It took some time for TSI to convert the ad agency programs. I remember spending many days at IBM’s office in Hartford working on this. After tat we needed to change the focus of our marketing efforts to larger ad agencies. At this point IBM had no system to offer to the many whose needs would be satisfied with one Datamaster with diskette drives.

We were happy with this announcement, but it was a disappointment that IBM had totally abandoned the market that had produced the majority of TSI’s sales.

The 5364 (or System/36 PC), which was announced in June of 1985, was IBM’s belated attempt to recapture that market. The starting price of $5,995 was quite attractive. For some reason the system console had to be an IBM PC, XT, or AT with a special card inside. The S/36 part was the same size and shape as an AT. It contained a 5 1/4″ diskette drive that was compatible with neither the attached PC nor any previous model of the S/36. Compatibility was not a high priority for IBM.

This is what the system console looked like with the PC on top of the 5364. Manute Bol found it very convenient.

This bizarre arrangement was very difficult to explain to a prospect. Why would a super-reliable S/36 be coupled to the least reliable hardware ever to wear the IBM logo? It did guarantee that at least one IBM PC was installed at each location, I guess. However, most installations probably did what we did—attach the oldest, slowest machine available as the system console, and then use it only when necessary.

The 5364 had two severe limitations. In the first place, only four devices (counting the system console and the printer) could be attached. So, the S/36 customer could really only attach two more displays or one display and a printer. Still, that would suffice at most small ad agencies and at TSI. We immediately ordered one.

The other limitation was, in some ways, worse. The 5364 came with only 256K of RAM. Each session could use up to 64K. However, batch jobs counted as sessions. If one person started some lengthy reports or other process, the system could possibly reach the point where jobs needed to be swapped between memory and disk. This could severely impair the system’s performance.

The announcement of the 5364 created a good deal of business for a company named Black Box. They sold a box with slots for one 8″ and one 5 1/4″ diskette and software to make image copies from one to another. We bought one, and I used it a lot.

This is a 5363 with both a diskette drive and a QIC tape drive.

In 1987 IBM finally fixed all of the problems, at least the ones that most concerned us. The 5363 was a reasonably priced machine that was suitable for almost any small business. If it had been announced two years earlier instead of the 5364, we could have sold a lot of them.

I could not find anything either on the Internet or in our basement that listed the price of one. I don’t remember that anyone balked at the cost. I also could not discover how much memory it had, but I remember distinctly that it was plenty. It could handle at least seven devices, too, and you could get one with a built-in QIC tape drive.

We enthusiastically ordered one for TSI’s office and tried to get our Datamaster customers excited about it.

By the middle of the eighties most office workers had some kind of personal computer on their desks. They did not want someone to install a terminal next to it as well. But how could they gain access to the S/36 without a terminal? There were no networks worth mentioning in the eighties, and there was nothing like USB. The only way to reach the server was through a twinax line, and IBM did not share its knowledge of how the display and printer connections worked. IBM considered anything that it produced was proprietary.

IBM and many third party companies brought 5250 terminal-emulation packages to market. The idea was to be able transform the PC into a terminal on demand and then change it back. Although the other vendors had to reverse-engineer the 5250 interface, they still were able to produce competitive products that were cheaper and had more features than what IBM offered. They generally consisted of a three pieces:

  1. A hardware card that fit into an expansion slot on the motherboard of the PC. That’s right. You had to take the cover off of the machine and add a card that had the brains needed to make the PC act like a 5250 card. You might also need to physically flip switches on the care to configure it. Then you had put the PC back together again. What could possibly go wrong?
  2. A dongle (sometimes lovingly referred to as a “pigtail”) that screwed into the interface on the part of the card that stuck out the back of the PC. It provided a gozinda and a gozouta for the twinax cable(s).
  3. Software to run on the PC.

When they first appeared, the cost of these packages was well over $1,000, nearly as much or more than a PC or terminal. However, the emulators had one very substantial advantage. The inexpensive printer attached to the PC could also be configured as a S/36 printer. This not only saved money and office space; it was also much more convenient.

For a few years the companies selling these packages did a land-office business.

1. IPL stands for Initial Program Load. It just means the starting process for the system. IBM had three-letter abbreviations of everything, including a three-letter abbreviations (TLA).

2. RPG is short for Report Program Generator. I could never understand why anyone used it.

3. IBM required separate licenses for each programming language that was used.

4. Two IBMers, Donald D. Chamberlin and Raymond F. Boyce, codified the Structured Query Language (SQL) used in relational databases in the seventies. IBM rejected it because the Indexed-Sequential ISAM structure that it used in its computers had much better performance. About three decades later the company changed its mind.