Window Management -- Overview:
F R A Hopgood, D A Duce, E V C Fielding, K Robinson, A S Williams. 29 April 1985.
This is the Proceedings of the Alvey Workshop at Cosener's House, Abingdon that took place from 29 April 1985 until 1 May 1985. It was input into the planning for the MMI part of the Alvey Programme.
The Proceedings were later published by Springer-Verlag in 1986.
James Gosling: SunDew - A Distributed and Extensible Window System.
As a result, one of the most amazing pieces of literature to come out of the X Consortium is the “Inter Client Communication Conventions Manual,” more fondly known as the “ICCCM”, “Ice Cubed,” or “I39L” (short for “I, 39 letters, L”). It describes protocols that X clients must use to communicate with each other via the X server, including diverse topics like window management, selections, keyboard and colormap focus, and session management. In short, it tries to cover everything the X designers forgot and tries to fix everything they got wrong. But it was too late — by the time ICCCM was published, people were already writing window managers and toolkits, so each new version of the ICCCM was forced to bend over backwards to be backward compatible with the mistakes of the past.
The ICCCM is unbelievably dense, it must be followed to the last letter, and it still doesn’t work. ICCCM compliance is one of the most complex ordeals of implementing X toolkits, window managers, and even simple applications. It’s so difficult, that many of the benefits just aren’t worth the hassle of compliance. And when one program doesn’t comply, it screws up other programs. This is the reason cut-and-paste never works properly with X (unless you are cutting and pasting straight ASCII text), drag-and-drop locks up the system, colormaps flash wildly and are never installed at the right time, keyboard focus lags behind the cursor, keys go to the wrong window, and deleting a popup window can quit the whole application. If you want to write an interoperable ICCCM compliant application, you have to crossbar test it with every other application, and with all possible window managers, and then plead with the vendors to fix their problems in the next release.
In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
Using these toolkits is like trying to make a bookshelf out of mashed potatoes. - Jamie Zawinski
X Myths
X is a colletion of myths that have become so widespread and so prolific in the computer industry that many of them are now accepted as “fact,” without any thought or reflection.
Myth: X Demonstrates the Power of Client/Server Computing
At the mere mention of network window systems, certain propeller heads who confuse technology with economics will start foaming at the mouth about their client/server models and how in the future palmtops will just run the X server and let the other half of the program run on some Cray down the street. They’ve become unwitting pawns in the hardware manufacturers’ conspiracy to sell newer systems each year. After all, what better way is there to force users to upgrade their hardware than to give them X, where a single application can bog down the client, the server, and the network between them, simultaneously!
The database client/server model (the server machine stores all the data, and the clients beseech it for data) makes sense. The computation client/server model (where the server is a very expensive or experimental supercomputer, and the client is a desktop workstation or portable computer) makes sense. But a graphical client/server model that slices the interface down some arbitrary middle is like Solomon following through with his child-sharing strategy. The legs, heart, and left eye end up on the server, the arms and lungs go to the client, the head is left rolling around on the floor, and blood spurts everywhere.
The fundamental problem with X’s notion of client/server is that the proper division of labor between the client and the server can only be decided on an application-by-application basis. Some applications (like a flight simulator) require that all mouse movement be sent to the application. Others need only mouse clicks. Still others need a sophisticated combination of the two, depending on the program’s state or the region of the screen where the mouse happens to be. Some programs need to update meters or widgets on the screen every second. Other programs just want to display clocks; the server could just as well do the updating, provided that there was some way to tell it to do so.
The right graphical client/server model is to have an extensible server. Application programs on remote machines can download their own special extension on demand and share libraries in the server. Downloaded code can draw windows, track input events, provide fast interactive feedback, and minimize network traffic by communicating with the application using a dynamic, high-level protocol.
As an example, imagine a CAD application built on top of such an extensible server. The application could download a program to draw an IC and associate it with a name. From then on, the client could draw the IC anywhere on the screen simply by sending the name and a pair of coordinates. Better yet, the client can download programs and data structures to draw the whole schematic, which are called automatically to refresh and scroll the window, without bothering the client. The user can drag an IC around smoothly, without any network traffic or context switching, and the server sends a single message to the client when the interaction is complete. This makes it possible to run interactive clients over low-speed (that is, slow-bandwidth) communication lines.
Sounds like science fiction? An extensible window server was precisely the strategy taken by the NeWS (Network extensible Window System) window system written by James Gosling at Sun. With such an extensible system, the user interface toolkit becomes an extensible server library of classes that clients download directly into the server (the approach taken by Sun’s TNT Toolkit). Toolkit objects in different applications share common objects in the server, saving both time and memory, and creating a look-and-feel that is both consistent across applications and customizable. With NeWS, the window manager itself was implemented inside the server, eliminating network overhead for window manipulation operations — and along with it the race conditions, context switching overhead, and interaction problems that plague X toolkits and window manager.
Ultimately, NeWS was not economically or politically viable because it solved the very problems that X was designed to create.
Cynthia Solomon has shared a treasure trove of rare classic videos of Seymour Papert, Marvin and Margaret Minsky, kids programming Logo and playing with turtles, and many other amazing things at the MIT AI Lab, MIT Media Lab, and Atari Cambridge Research:
Check out Paul Lamere's talk about playlisting that he presented at ISMIR 2010 (The International Society for Music Information Retrieval has conferences about all this stuff, and Paul founded The Echo Nest, which Spotify later bought):
ISMIR: The International Society for Music Information Retrieval
>Tutorial 4: Finding A Path Through The Jukebox -- The Playlist Tutorial. The simple playlist, in its many forms -- from the radio show, to the album, to the mixtape has long been a part of how people discover, listen to and share music. As the world of online music grows, the playlist is once again becoming a central tool to help listeners successfully experience music. Further, the playlist is increasingly a vehicle for recommendation and discovery of new or unknown music. More and more, commercial music services such as Pandora, Last.fm, iTunes and Spotify rely on the playlist to improve the listening experience. In this tutorial we look at the state of the art in playlisting. We present a brief history of the playlist, provide an overview of the different types of playlists and take an in-depth look at the state-of-the-art in automatic playlist generation including commercial and academic systems. We explore methods of evaluating playlists and ways that MIR techniques can be used to improve playlists. Our tutorial concludes with a discussion of what the future may hold for playlists and playlist generation/construction.
A lot of this discussion makes more sense if you know the history of The Echo Nest and their acquisition by Spotify.
The Echo Nest was one of the most interesting music-tech companies ever built: a music intelligence platform spun out of MIT that analyzed audio, metadata, web text, artist similarity, genre structure, and playlist construction. Spotify bought them in 2014 specifically to strengthen music discovery and recommendation. At the time, Spotify said the deal would let it use The Echo Nest's "in depth musical understanding and tools for curation", and even said the Echo Nest API would remain "free and open" for developers.
If you ever used the old Echo Nest APIs, Remix SDK, demos, Music Hack Day projects, or Paul Lamere's experiments, that was a golden era. Echo Nest had open APIs for artist similarity, track analysis, playlisting, "taste profiles", ID mapping across services, and beat/segment-level music analysis. Paul Lamere's whole ecosystem of demos came out of that world: Boil the Frog, Sort Your Music, Organize Your Music, playlistminer, and later Smarter Playlists. His GitHub still points to a lot of that lineage, and his blog is still active. In fact, he posted just this month about rebuilding Smarter Playlists after ten years of use.
The sad part is that the open developer platform mostly did not survive the acquisition. By 2016, developers were being told that the Echo Nest API would stop issuing new keys and then stop serving requests, with migration to Spotify’s API instead. Community discussions at the time also noted that some Echo Nest capabilities, especially things like Rosetta-style cross-service mapping, were not really carried over.
That's also why Spotify's current AI DJ is so frustrating. The problem is that "AI DJ" is not the same thing as a system that deeply understands musical structure, discography semantics, performance history, or classical work/movement hierarchy. It's a recommendation + narration layer, not a true MIR-native musical intelligence system.
If you're interested in the research side of this field, the conference is ISMIR: the International Society for Music Information Retrieval, which is literally dedicated to computational tools for processing, searching, organizing, and accessing music-related data. That community is still very active. The ISMIR site describes MIR exactly in those terms, and the 2010 Utrecht conference included Paul Lamere's tutorial, "Finding A Path Through The Jukebox -- The Playlist Tutorial."
>gffrd on June 26, 2023 | parent | context | favorite | on: Show HN: Mofi – Content-aware fill for audio to ch...
>Yes! It was "Infinite Jukebox," created by Paul Lamere ... it was awesome because it would analyse a track, then visualize its "components" and you could watch as the new "infinite" track looped back on itself and jumped from point-to-point in the original track to create an everlasting one.
He created some excellent products from the Rdio API, and later Spotify ... and I believe his analysis engine ended up being the foundation upon which Spotify's _play more tracks like these_ capability is based.
>Looks like he's moved over to publish on Substack -- there's a recent(ish) post reflecting on 10 years of Infinite Jukebox:
>However, that wasn't the end of the Infinite Jukebox. An enterprising developer: Izzy Dahanela made her own hack on top of mine. To make it work without using uploaded content, she matches up the Echo Nest / Spotify music analysis with the corresponding song on YouTube. She hosts this at eternalbox.dev. It runs just as well as it ever did, 10 years later.
>DonHopkins on June 28, 2023 | parent | context | favorite | on: Show HN: Mofi – Content-aware fill for audio to ch...
>I was working on some music retrieval stuff in 2010, so I joined the EchoNest developer program and played around with their web apis that let you upload music and download an analysis that you could use in all kinds of cool ways. They had an SDK with some great demos and example code. I discussed it with Eric Swenson and Paul Lamere, and had the chance to hang out with Paul Lamere and Ben Fields at ISMIR 2010 (the International Society for Music Information Retrieval conference) in Utrecht, where they gave a tutorial about playlisting:
>Tutorial 4: Finding A Path Through The Jukebox -- The Playlist Tutorial. The simple playlist, in its many forms -- from the radio show, to the album, to the mixtape has long been a part of how people discover, listen to and share music. As the world of online music grows, the playlist is once again becoming a central tool to help listeners successfully experience music. Further, the playlist is increasingly a vehicle for recommendation and discovery of new or unknown music. More and more, commercial music services such as Pandora, Last.fm, iTunes and Spotify rely on the playlist to improve the listening experience. In this tutorial we look at the state of the art in playlisting. We present a brief history of the playlist, provide an overview of the different types of playlists and take an in-depth look at the state-of-the-art in automatic playlist generation including commercial and academic systems. We explore methods of evaluating playlists and ways that MIR techniques can be used to improve playlists. Our tutorial concludes with a discussion of what the future may hold for playlists and playlist generation/construction.
>[...]
Some of the most interesting Echo Nest descendants are still around in one form or another. Paul Lamere's current/public projects include Smarter Playlists, and his GitHub still highlights SortYourMusic, OrganizeYourMusic, playlistminer, and BoilTheFrog. Glenn McDonald’s Every Noise at Once is another major descendant of that tradition: an enormous map of music genre space. Glenn's own site still describes it as an "inexorably expanding universe of music-processing experiments", and the public genre pages now explicitly say they're a long-running snapshot based on Spotify data through 2023-11-19. After Spotify's layoffs in 2023, TechCrunch reported that Glenn lost access to the internal data needed to keep Every Noise fully updated, which is why it now feels more archival than alive.
Back in 1998 when I was working on The Sims 1, I proposed in my review of the design document something I called "Moody Music": essentially a soundtrack plus a synchronized semantic/emotional control track that could affect gameplay over time. The idea was that music wouldn't just decorate the simulation; it would change it: influencing mood, motives, relationships, skills, timing, and even triggering events at specific musical moments. I wrote that up in my review of the 1998-08-07 Sims design document, along with the broader idea of letting the game recognize a player's own CDs and fetch associated "moody tracks" from the network.
Don’s review of
The Sims Design Document,
Draft 3 – 8/7/98:
>I have some ideas about how the music could effect the game, that
I will write up more completely later. In a nutshell, the people in
the house could have a cd or record collection to choose from, each
record an object that has the sound (audio wave and/or midi) and a
“moody” track synchronized with the music. Playing the music
also plays the moods into the environment that the people pick up
on. Music can subtly effect how people react to the environment,
objects, and each other. It can effect their motives and even their
skills temporarily. For example, you might be able to clean the
house better and faster if you put on some up tempo bouncy music.
The player should be able to assume the role of disc jockey on the
radio, and play from another larger library of music and
commercials, that effect the peoples moods and buying habits. The
TV of course is another source of mood altering temporal media,
with commercials and shows that should effect different people
differently. But the most important part of this idea is instead of
the game effecting the music that’s played, the music effects how
the game plays! The ultimate way for the user to effect the game
via music, is to insert one of their own CD’s into their real
computer’s CDROM drive, and the game would recognize it, and
start playing it (maybe with a simple cd player interface to select
the song). There could be a database associating the unique ID
number of the CD with a table of contents and “moody” tracks that
tell how the song effects the peoples emotions over time, with
"percussion" events at dramatic moments of the music that can
trigger arbitrary events in the game (like provoking a fight that was
brewing, or triggering an orgasm at just the right place in the
song). We hire monkeys to listen to well known CD’s, and enter
time synchronized tracks with semantic meanings in Max (like
note tracks, and user defined numeric tracks) or some other
timeline editing tool). Put the database up on the web for instant
retrieval, so when somebody sticks in a new CD, it downloads our
“moody” tracks that go with it, and it starts playing and effecting
their game! Streaming emotions over the net! Eventually there
should be an end-user tool so people can record their own
responses to music as moody tracks they can use in our games.
This mechanism could be used in all kinds of games, to varying
degrees of effect. I’m not saying that music should be the only way
to control the game – it’s more like a subtle background effect, but
there certainly could be a scenario where you try to accomplish
some task (like taming a wild beast) by using only your musical
taste and timing. The real bottom line benefit is that you get to
listen to your OWN cd collection of music you want to hear,
instead of being driven crazy by the repetitive music bundled with
the game.
In hindsight it was quite adjacent to MIR, affective computing, adaptive soundtrack systems, and some of the ambitions that Echo Nest represented. That's why I was so excited about The Echo Nest in 2010 when I was working with Will Wright at the Stupid Fun Club on a music spatial organization and navigation system called MediaGraph.
MediaGraph Music Navigation with Pie Menus Prototype developed for Will Wright's Stupid Fun Club
>This is a demo of a user interface research prototype that I developed for Will Wright at the Stupid Fun Club. It includes pie menus, an editable map of music interconnected with roads, and cellular automata.
>It uses one kind of nested hierarchical pie menu to build and edit another kind of geographic networked pie menu.
>What Apple put inside the Neo is the complete behavioral contract of the Mac.
I remember seeing and using my first powerbook 160 and being blown away that it had the "complete behavior contract of the Mac" that I knew at the time. Even the 16 shades of gray screen made it more luxurious than a classic black-and-white Mac.
And the "What's on Your Powerbook" ads, with Todd Rundgren and Rev. Don Doll, SJ.
>Todd also co-developed graphic tablet software with a music theme for Apple in a technology venture in the late eighties. With Dave Levine, he designed and developed a screensaver product called Flowfazer (see example of one of the screensavers below), with the strapline “Music for the Eye.” They introduced it at MacWorld thinking they would publish it themselves, but found there was already well-funded competition with Berkeley Systems Flying Toasters and were forced to abandon the project.
https://en.wikipedia.org/wiki/NeWS
Everything old is new again.
Alan Kay on “Should web browsers have stuck to being document viewers?” and a discussion of Smalltalk, HyperCard, NeWS, and HyperLook:
https://donhopkins.medium.com/alan-kay-on-should-web-browser...
Window Management -- Overview: F R A Hopgood, D A Duce, E V C Fielding, K Robinson, A S Williams. 29 April 1985.
This is the Proceedings of the Alvey Workshop at Cosener's House, Abingdon that took place from 29 April 1985 until 1 May 1985. It was input into the planning for the MMI part of the Alvey Programme.
The Proceedings were later published by Springer-Verlag in 1986.
James Gosling: SunDew - A Distributed and Extensible Window System.
https://www.chilton-computing.org.uk/inf/literature/books/wm...
The X-Windows Disaster: This is Chapter 7 of the UNIX-HATERS Handbook. The X-Windows Disaster chapter was written by Don Hopkins.
https://donhopkins.medium.com/the-x-windows-disaster-128d398...
As a result, one of the most amazing pieces of literature to come out of the X Consortium is the “Inter Client Communication Conventions Manual,” more fondly known as the “ICCCM”, “Ice Cubed,” or “I39L” (short for “I, 39 letters, L”). It describes protocols that X clients must use to communicate with each other via the X server, including diverse topics like window management, selections, keyboard and colormap focus, and session management. In short, it tries to cover everything the X designers forgot and tries to fix everything they got wrong. But it was too late — by the time ICCCM was published, people were already writing window managers and toolkits, so each new version of the ICCCM was forced to bend over backwards to be backward compatible with the mistakes of the past.
The ICCCM is unbelievably dense, it must be followed to the last letter, and it still doesn’t work. ICCCM compliance is one of the most complex ordeals of implementing X toolkits, window managers, and even simple applications. It’s so difficult, that many of the benefits just aren’t worth the hassle of compliance. And when one program doesn’t comply, it screws up other programs. This is the reason cut-and-paste never works properly with X (unless you are cutting and pasting straight ASCII text), drag-and-drop locks up the system, colormaps flash wildly and are never installed at the right time, keyboard focus lags behind the cursor, keys go to the wrong window, and deleting a popup window can quit the whole application. If you want to write an interoperable ICCCM compliant application, you have to crossbar test it with every other application, and with all possible window managers, and then plead with the vendors to fix their problems in the next release.
In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
Using these toolkits is like trying to make a bookshelf out of mashed potatoes. - Jamie Zawinski
X Myths
X is a colletion of myths that have become so widespread and so prolific in the computer industry that many of them are now accepted as “fact,” without any thought or reflection.
Myth: X Demonstrates the Power of Client/Server Computing At the mere mention of network window systems, certain propeller heads who confuse technology with economics will start foaming at the mouth about their client/server models and how in the future palmtops will just run the X server and let the other half of the program run on some Cray down the street. They’ve become unwitting pawns in the hardware manufacturers’ conspiracy to sell newer systems each year. After all, what better way is there to force users to upgrade their hardware than to give them X, where a single application can bog down the client, the server, and the network between them, simultaneously!
The database client/server model (the server machine stores all the data, and the clients beseech it for data) makes sense. The computation client/server model (where the server is a very expensive or experimental supercomputer, and the client is a desktop workstation or portable computer) makes sense. But a graphical client/server model that slices the interface down some arbitrary middle is like Solomon following through with his child-sharing strategy. The legs, heart, and left eye end up on the server, the arms and lungs go to the client, the head is left rolling around on the floor, and blood spurts everywhere.
The fundamental problem with X’s notion of client/server is that the proper division of labor between the client and the server can only be decided on an application-by-application basis. Some applications (like a flight simulator) require that all mouse movement be sent to the application. Others need only mouse clicks. Still others need a sophisticated combination of the two, depending on the program’s state or the region of the screen where the mouse happens to be. Some programs need to update meters or widgets on the screen every second. Other programs just want to display clocks; the server could just as well do the updating, provided that there was some way to tell it to do so.
The right graphical client/server model is to have an extensible server. Application programs on remote machines can download their own special extension on demand and share libraries in the server. Downloaded code can draw windows, track input events, provide fast interactive feedback, and minimize network traffic by communicating with the application using a dynamic, high-level protocol.
As an example, imagine a CAD application built on top of such an extensible server. The application could download a program to draw an IC and associate it with a name. From then on, the client could draw the IC anywhere on the screen simply by sending the name and a pair of coordinates. Better yet, the client can download programs and data structures to draw the whole schematic, which are called automatically to refresh and scroll the window, without bothering the client. The user can drag an IC around smoothly, without any network traffic or context switching, and the server sends a single message to the client when the interaction is complete. This makes it possible to run interactive clients over low-speed (that is, slow-bandwidth) communication lines.
Sounds like science fiction? An extensible window server was precisely the strategy taken by the NeWS (Network extensible Window System) window system written by James Gosling at Sun. With such an extensible system, the user interface toolkit becomes an extensible server library of classes that clients download directly into the server (the approach taken by Sun’s TNT Toolkit). Toolkit objects in different applications share common objects in the server, saving both time and memory, and creating a look-and-feel that is both consistent across applications and customizable. With NeWS, the window manager itself was implemented inside the server, eliminating network overhead for window manipulation operations — and along with it the race conditions, context switching overhead, and interaction problems that plague X toolkits and window manager.
Ultimately, NeWS was not economically or politically viable because it solved the very problems that X was designed to create.
reply