amyjko/user-interface-software-and-technology

A book surveying the literature on user interface software and technology.

user-interface-software-and-technology's Issues

Mediation: Cite "Algorithmic Fairness"

Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018, April). A Qualitative Exploration of Perceptions of Algorithmic Fairness. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 656). ACM.
 

Help: student feedback

1. In this paragraph, "breakdown" is used (my guess is) in relation to the definition discussed earlier in the chapter. Perhaps this word could be bolded again to remind the user we're discussing a newly learned term again?

"Another potentially useful signal of a need for help is when users undo or erase actions they have just performed. One project showed that these events can indicate confusion about how to use a feature, where 90% of the undo and erase episodes indicated severe breakdowns in user’s ability to progress on a task (Akers et al. 2009)."

2. This part read like it was an add-on at the end of the paragraph with the image break above. I think it could use some text expansion or a graphic since it seems like a cool service. Or, incorporated somehow into the Lemonaid portion.

I helped commercialize this system as a company called AnswerDash, which uses the technology to offer contextual help to websites experiencing high numbers of support requests.

Interactive: student critiques

This particular chapter echoed what I first learned about the internet and computers in undergrad, but in a more thoughtful and engaging voice. I thought introducing Xerox PARC in an earlier chapter helped brought context into WIMP.

When introducing all of these different interactions of scrolling, window placing, clicking on icons and menus, etc. I think it would be helpful visually if there were accompanying gifs. Not an overwhelming number, but enough to get confirmation of what was read to our mental model of it. For example, I'm not familiar with what a "floating dialog" looks like so having an image support some of the less known tools would be helpful. I know there's the All the Widgets video, but glancing and getting it in under a second helps.

I wish there was more to the copy and paste paragraph. I found this topic so interesting (and ended up reading one of the papers) that I wanted to read more. It was rather unsettling to end the chapter with that like you saved the best bit for last. Maybe placing it before the direct manipulation section which is already a topic covered to help smooth out the conclusion?

Homework 6 - how do we ensure sufficient depth of design work?

One student only submitted a superficial redesign of a UI.

Declarative interfaces: discuss learning barriers in programming systems

I usually lecture about this when I do this topic, but I forgot to discuss it in this chapter.

Hands: Cite DuetDraw

Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018, April). I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 649). ACM.

Consider adding a chapter on design ethics

Most of the book is pretty focused on technology and tech-optimism. What kind of chapter would help reflect on this?

Declarative interfaces: reading critique

Critique #1:
Early in the article, you make a powerful and easy-to-understand distinction as to what defines a programming user interface. What's more, is the clear and succinct characterization of a programming user interface. After effectively defining and explaining programming user interfaces, you proceed to condense the characterization of programming user interfaces to one term- declarative user interfaces. I think that this is a great word to encapsulate the three pre-established aspects of programming user interfaces but it does create some confusion. You mention that you will refer to programming user interfaces as declarative interfaces for the remainder of the chapter. However, after establishing this, you continue to use the term "programming interfaces." This led to some confusion as a reader and prompted some backtracking and re-reading to make sure programming interfaces are the same as declarative interfaces.

Critique #2
The analogy highlighted below was highly effective in explaining end user programming was highly effective.
"Much of this work can be described as supporting end-user programming, which is any programming that someone does as a means to accomplishing some other goal (Ko et al. 2011). For example, a teacher writing formulas in a spreadsheet to compute grades, a child using Scratch to create an animation, or a data scientist writing a Python script to wrangle some data—none of these people are writing code for the code itself (as professional software engineers do), they're writing code for the output their program will produce (the teacher wants the grades, the child wants the animation, the data scientist wants the wrangled data)."
It would have been helpful to see something like this earlier in the reading to help convey the three points of programming user interfaces (no direct manipulation, use of notation, and use of abstraction.)

Grammar
"Because the functional affordances in a programming language can be combined in infinite ways, there are an infinite number of programs can exist."

Declarative: Cite end user programming of home automation

Brich, J., Walch, M., Rietzler, M., Weber, M., & Schaub, F. (2017). Exploring end user programming needs in home automation. ACM Transactions on Computer-Human Interaction (TOCHI), 24(2), 11.

Declarative: cite Bifrost

McGrath, W., Drew, D., Warner, J., Kazemitabaar, M., Karchemsky, M., Mellis, D., & Hartmann, B. (2017, October). Bifröst: Visualizing and Checking Behavior of Embedded Systems across Hardware and Software. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 299-310). ACM.

Intellectual property: student feedback

"Whatever your argument for ownership, one cannot construct a notion of idea ownership without clarity about what an idea is. As we discussed above, that is not clear, so ownership is not clear either."

What are your thoughts on the proliferation of sites like GitHub, Instructables, and Thingiverse where people may share their source codes and ideas. Some ask for credit, some might not. Some modify codes to make it their own, but at which point is it theirs, and not whose source of inspiration it is? You touch upon the ambiguity in ownership in the section "Can ideas be owned?", but it would be interesting to note sites that promote collective idea sharing and what that has to do with the future of ownership.

"Designers strongly believed that the look and feel of user interfaces should not be given protection by copyright or patent law"

As aforementioned in the section for "Help", do these apply to only the traditional screens that we're used to? If we were to adhere to Weiser's view of ubiquitous computing, how many ways can we really design non-screen interfaces? What are the laws like for this in terms of what can actually be designed? Are there issues with the Google Home vs. Amazon's Alexa etc. Are these types of interfaces guided more so by technology and what's actually possible, usable, and feasible as opposed to more creative and unique?

Physical output: cite AirCode

Li, D., Nair, A. S., Nayar, S. K., & Zheng, C. (2017, October). AirCode: Unobtrusive Physical Tags for Digital Fabrication. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 449-460). ACM.

Hands: Student feedback

I felt that it took too long to get to the point of the chapter
(hands). The first paragraph was quite lengthy in its setup. It
also included a long quote, which distracted me from getting
through the chapter. I already felt myself skimming and had to
actually read the quote 3 times before really reading it.

This was a very long sentence that was confusing to get through. It
was distracting not only because of the length, but also because of
how much information was contained in the one sentence. I think the
3 later technologies mentioned were summarized pretty well. I
understood what they were about, but also was interested in learning more about them. However, the summary for Ono’s technology made me want to read more about it because I was more confused than clued into what it was.

“These leverage the skill of grasping a pen or pencil that many are familiar with from manual writing, which is often taught in school.” I thought the “which is often taught in school” part was redundant with the “many are familiar with.” It doesn’t matter where it was learned, because I think most people who would be reading this would grasp why many are familiar with writing with pen and pencil.

I don’t know if it was just the nature of the chapter, the Touch, Pens, and Gestures sections seemed a bit thin and made this chapter seem like it lacked a lot of substance as compared to other chapters. These sections seemed more like “here’s a list of cool stuff people did in these three things.” I found myself bored in this chapter because it was just a list of hand inputs that I would want to read more about later and not much discussion. I think my favorite part of your previous chapters is the discussion you go into and how you weave together a narrative for the history of these interfaces, and then implications for the future. However, again, this chapter just seemed almost like a categorized list of cool input technologies. I think adding more commentary into the implications, or just combining this chapter with your next few chapters discussion non traditional pointer interfaces might combat this. Alternatively, people might not be reading this chapter by chapter as we are, so maybe they will breeze through this and the next few chapters and it is alright as is.

Theory: personalizing user interfaces

Feedback from a student:

I also became curious about customization, and how computer interfaces like personal assistants now try to "learn" their human counterparts. Since human goals are "vague, shifting, unobservable," computers have to elicit clear cues from us to support customization. I learned a bit about this recently from a lecture about VUI design. I would be interested in reading about the affordances that computers need from humans for successful interface customization, and design techniques for drawing these out.

Mediation: Cite "Explanations as Mechanisms for Algorithmic Transparency"

Rader, E., Cotter, K., & Cho, J. (2018, April). Explanations as Mechanisms for Supporting Algorithmic Transparency. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 103). ACM.

3D: Student feedback

Maybe in the future iteration of this chapter, you can talk more about the challenges and constraints from the perspective of the whole eco-system of VR/AR related technologies. For example, how to create a seamless transition for users from mobile phone VR goggles to the standalone VR system, or how to engage users interacting more with the standalone VR system.
Another point that I would like to mention is that I think holography and volumetric display are other types of 3D visual output which might be the future trend as well. So maybe in a later version, you can mention those kinds of 3D visual output in addition to the mainstream technologies such as AR/VR?

Pointing: student feedback

I like your description of Fitts Law and the descriptions of what each variable means. One improvement for the chapter might be to remind the user about the a and b coefficients in the paragraph after the apple menu bar picture. I found myself having to go back up to the paragraph where you first described the coefficients to remind myself what they were.
I believe a graphic of a person trying to reach a target might have done better for the Fitts law description. While I understood the description in the reading with the current image, it took me a second to match the example of "how far someone has to move their finger" with the mouse cursor shown in the image. Hence, I believe either changing the example described to match the image or vice versa would reduce confusion.

Theory chapter should actually cite the theories it refers to

Currently its a list in the opening paragraphs with no citations.

Declarative: include cognitive dimensions of notations

It's a critical foundation that students should know about:

Green, T. R. G., & Petre, M. (1996). Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework. Journal of visual languages and computing, 7(2), 131-174.

Mediation: explain that one interface can have all three

Provide scenarios like UI mediate multiple things simultaneously, like the UI can mediate both information and communication at the same time, such as Twitter.

Declarative: Cite Turkit

https://doi.org/10.1145/1866029.1866040

Probably in the declarative interfaces section.

Physical: student feedback

I appreciated understanding some of the history and problems that you stated in the printing section. I felt like in a few sentences I understood why and how printing became so pervasive. I felt the other sections did not have as much to express why we are currently at this moment in time for the technology's maturity. I felt the "morphing" section is especially lacking since I am still a bit unsure what specifically characterizes morphing and how we arrived at wanting to utilize morphing as interaction pattern. Is Morphing a successor to physical buttons and switches that we physically change the state of.

I sometimes get lost in many of these chapters of what I should take away from these technologies. The level of content seems to be written for designers to have a survey of technologies to perhaps use in their designs. But the way it is written it is not explicit for who this is written for. I wonder if this can be helped by creating a narrative of how these technologies can be included in the design process or what aspects technically designers should know while designing.

Physical Output: Haptic output cite weight and grasp simulation

Choi, I., Culbertson, H., Miller, M. R., Olwal, A., & Follmer, S. (2017, October). Grabity: A Wearable Haptic Interface for Simulating Weight and Grasping in Virtual Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 119-130). ACM.

Theory: clarify and explain signifiers and affordances

This comes up every year. Need to address it.

Cite PowerPoint history

https://spectrum.ieee.org/tech-history/cyberspace/the-improbable-origins-of-powerpoint

Mediation: diagram idea

Under the "These three types of mediation each require different architectures, different affordances, and different feedback to achieve their goals:”, it might be helpful to construct the content below into a diagram based on types of mediation, their different architectures, different affordances, and different feedback? (I just think by this way, those key points could become a useful table for the future referencing.)

Adding one or two images to each section to support the explanation of the Mediating automation, information and communication.

History: emphasize context of historical design decisions

As it is mentioned at the beginning of the chapter, explaining the history of interface can be considered as a mundane way to start a discussion about user interfaces. However, describing the journey of thoughts throughout the history and the outcome of the paths that have been taken by great people can help us understand the impact of decisions and ideas on the world that we have today. I think the emphasize of the chapter has been mostly on the outcome of these ideas and not the context and situations in which these concepts have been formed. For instance, knowing how Bush’s vision was made and what influenced that idea can help us better understand the impact of external factors and needs of society on the creation of a concept.

Architecture: student critiques

From Brian Nguyen:

The first improvement in the chapter that I think would assist me in understanding event propagation a bit more is using mouse/keyboard as one example, but then potentially illustrating events in a more complicated system such as voice interface. I think mouse and keyboard are so integral to our interactions with a computer so events seem obvious to me with them, but they are not as clear for some of the other examples that you provided after these 2.

The second improvement in the chapter that I think would improve it would be to give a code example of MVC. Experience levels for programming definitely vary within the class, and I know that teaching code isn't the explicit goal of the class, but I do think explaining MVC with a short simple code example would provide a context that would be helpful for me and other students.

Add alt text to images

Theory: student critiques

1. It might be nice if the chapter touches a little on how sometimes these interface theories may not be necessarily followed in practice when designers are placed in an environment with stakeholders that may have competing priorities. For example, hidden close buttons on modals are really prevalent in ads and cases like this illustrate how business/marketing stakeholders may still win the battle sacrificing usability in hopes of increasing sales.

2. Another case I thought about when reading through this chapter was the case of Snapchat. The app is intentionally designed against all existing conventions but was still successful. Why was this so, and when is the proper time for designers to go against conventions?

3. A gesture that bothers me in some social media apps now is the ability to swipe left on the home screen of an app which goes to a camera (Instagram). This affordance lacks a signifier for a first time user but has become so popular. Why has a gesture like this become a convention even if it flies against the theories of this chapter?

Theory: Link affordances to Don Norman's signifiers

3D Chapter: mention Tango & ARCore

The 3D chapter mentions ARKit as the breakthrough smartphone, essentially software-based, AR technology making AR more widely available and ubiquitous. While I agree, it's only one example, and it would be worth noting the years of effort of the Google ATAP team on Tango that preceded ARKit and eventually led to ARCore, which like ARKit and in contrast to Tango does not require additional motion and depth cameras for environment understanding, though it is also more limited. As a point of reference, ARKit was released in June 2017, ARCore in August 2017.

3D: Mention Sutherland's first VR headset

I have a photo titled "First virtual reality headset" in my library.

More history videos

From Brock Craft:

History of UI is one of my soft spots. Can’t get enough…

These are hot stuff:
• Apollo DSKY: https://youtu.be/YIBhPsyYCiM?t=328
• https://www.youtube.com/watch?v=xx7Lfh5SKUQ (less UI stuff in this one)

and to the extent that the PSTN is a “computer and telephones the “interface”, this:
• https://www.youtube.com/watch?v=zE_2AbY2qnY

and of course the first mobile streaming music:
• https://www.youtube.com/watch?v=ILiLaRXHUr0&t=0s&list=PLDJoA80alTFypt5kix67d_gfkFWmcqVFx&index=2

Use UTF-8 encoding

Right now, all the documents are encoded as Windows-1252. This is a legacy format and UTF-8 is the default for HTML5.

I noticed this because my text editor tries to open the file as UTF-8 and all the special characters get corrupted on save.

Theory: contrast metaphors of cognition and computing

Feedback from a student:

This chapter gave a very clear overview of interface affordances. The idea that computing is "a world in which there is only forward motion" struck me as insightful and memorable. In fact, it stood out to me so much that I wished I had an equally clear metaphor for the motion of human cognition. The circular metaphor was implied, and came up near the end of the chapter, where human goals were described as "vague, shifting, unobservable." I still wish the chapter had drawn a more direct contrast early on.

2D Visual Output: student feedback

This chapter certainly covers a lot, most of which I suspect everyone has read a lot about, like typography or data visualization, so a paragraph or two is not super impactful. Maybe you could go a little deeper into each area, or speak only about graphics and typography and not take it beyond that? That said, it is interesting to get some of the tidbits about why we use the things we do. I'd be interested to learn more about 2d interaction and hierarchy paradigms and why we use those. Why do we scroll up and down? We tend to jump from page to page, but we could also be zooming into subscreens of main screens or something.

Add a link to the live version of the book

Could you include the hyperlink to the book somewhere in this repository? Either in the repository description or in a readme would be sufficient.

VR: Cite "The dream is collapsing"

Knibbe, J., Schjerlund, J., Petræus, M., & Hornbæk, K. (2018, April). The dream is collapsing: the experience of exiting VR. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 483). ACM.

Animation: cite live artwork swaps

Willett, N. S., Li, W., Popovic, J., & Finkelstein, A. (2017, October). Triggering Artwork Swaps for Live Animation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 85-95). ACM.

Cool technique for authoring animations.

Text Entry/Pointing: Add Jake Wobbrock acknolwedgement

He helped edit the text entry and pointing chapters.

Help: Cite Porta

Alok Mysore and Philip J. Guo. 2018. Porta: Profiling Software Tutorials Using Operating-System-Wide Activity Tracing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 201-212. DOI: https://doi.org/10.1145/3242587.3242633

Chapter on translating research into practice

Several have requested this and it was a major interest in the 2018 cohort.

Mediation: more concrete examples of each type of thing that must be learned

Give some explicit examples of each. They're too abstract right now. Screenshots would help.

Accessibility: student critiques

I think the third paragraph (starting with "However, this bias is
also unreasonable") is supposed to really drive the point home that
accessibility affects more than we think. However, the figures you
cite (1 billion are disabled) and the example you give ("take off
your glasses") actually make the paragraph less poignant. Without
reading the paper you cite, it makes me think the 1 billion
encapsulates "small" disabilities like fixable vision impairments
and things that "normal" people deal with everyday, that I deal
with everyday. And then I think about my "disabilities" and see
that I do not really have a problem with interfaces. So then I
think that you are exaggerating, and now I do not know how many of
these people are ACTUALLY affected so much by their disability that
they are unable to use interfaces well. And I think, "okay, so it's
not exactly a billion. He was including all these other populations
that aren't even THAT affected by their disability, so it must be
an insignificant amount." The number and examples make me feel like
you are citing a technicality, and thus make me skeptical of your
whole following argument in the paper.

I honestly think not trying to make it hit closer to home with that
bit about “Take off your glasses” would be better. I know that you
provide some detailed examples of how people with disabilities are
affected by lack of accessibility for a lot of interfaces later in
the chapter, but just a little preview here might drive the point
home a little better than trying to make it personal to the
reader.

I think the last paragraph is your strongest of the chapter. It has
impact in being “surprising,” but bringing some of these ideas to
the beginning might make a reader who is skeptical about
accessibility more convinced earlier about the importance of
accessibility.

Some Grammar Issues

I apologize in advance over how nitpicky I got with some of these
and hope the tone does not come off as condescending or
patronizing, as that was not my intent at all. It’s just that I
learn so much from your chapters and feel apprehensive about making
suggestions on the content. I am definitely not an expert on
interfaces and don’t feel qualified to critique those aspects.
However, I am confident in my grammar.

“We can speak, we can control over 600 different muscles, we can
convey hundreds of types of non-verbal information through our
gaze, posture, and orientation.”

These are technically comma splices/run-on sentences/independent
clauses incorrectly joined by commas. I understand you’re probably
going for impact, but this was noticeably bothersome. I suggest
periods or semicolons.

Colons technically should come after independent clauses only.
However, it is used incorrectly in so many places anyways, so this
usage might be fine colloquially. When you leave off with the colon
here, though, it makes me think that you are continuing that
phrase, “Access technologies include things like…” But when I read
it out like that, the bullets don’t make sense grammatically. For
example, if I were to read it all the way through, it would read,
“Access technologies include things like screen readers convert
text on a graphical interface to synthesized speech…” which is not
grammatically correct. I would suggest something like, “Access
technologies include things like the following:” or, “Access
technologies include things like: screen readers, which convert
text …”

Now I’m just being nitpicky, but double colons is just a little
jarring.

Commas

Mediation: Cite explainable AI research agenda

Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018, April). Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 582). ACM.

Text entry: student feedback

First, I think that a variety of predictive text entry techniques will help people type as fast as they think. Second, a variety of virtual keyboards in the future would enable users to type as fast as typing in the physical environments, and future systems might be able to track users’ hands from the keyboard. Finally, I image affective computing may be incorporated into the design of text entry techniques, predicting and indicating users’ moods and behaviors. Based on users’ hand movement and typing speed, these affective text entry devices will efficiently communicate affective states and emotions.

Body: student feedback

I would find it interesting if there were some motion graphics of the various muscles you mention in use - such as "When we look with our eyes, we use muscles to shift our gaze, to blink, to squint, and keep our eyes closed." This would be useful when comparing a person with high functioning motor ability compared to someone with motor impairment. Since muscles are under the skin, away from focus and invisible at times, it becomes more likely that disabilities are overlooked or dismissed due to unawareness.

I wonder if it's possible to have the links you provide automatically open in another window/tab. I know I can right click on a link and have it open in a new tab, but I often forget this and create an unintentional trail to open a PDF article on ACM (i.e. Setler et al. 2016).

I wonder if there is a logic behind the further reading. For example, if someone wants to learn more about "body" they might be recommended to read certain articles. I typically go through each reading and first look at the title, and if I am curious I will go read the abstract before I commit to a single article to read. If I already know the connection a paper has to content in the chapter, it would help me to find and avoid missing papers of a certain focus that interest me.

History: Video No Longer Available

The first video in A History of User Interfaces (after discussing Licklider's vision) is no longer available

History: mention Alan Turing

Furthermore, when I read the first paragraph I felt like two names were missing. Alan Turing and the Turing machine as the original idealized model of a computer. I think the story of Turing machine is another great example that demonstrates how today’s world depends heavily on the ideation that has been achieved throughout the history.

Not sure he has much to say about interfaces, but certain intelligence.