Ben Shneiderman

0:09:07 Born 1947 in New York City; grew up in a warm environment with parents and a sister who had been born in Paris in 1937; my parents came to New York in February 1940 to escape the oncoming war; I was born after all the turbulence and loss; all four of my grandparents were killed in the Warsaw Ghetto; the community of family that I had when growing up in Manhattan, surrounded by other cousins, was very important to me; I appreciated the environment my parents created as journalists involved in the Jewish community and the New York literary world; they wrote later about their homeland, Poland, and Eastern Europe; my parents used to write together, my father dictating and mother transcribing; my father would cut and paste with glue and scissors many times, and then my mother would retype; I don't have trouble writing as I know how difficult it is; it is difficult because people have a desire to make it perfect on the first try, and to do so takes much reworking; remember Winston Churchill's comment that he would practice his speeches a hundred times until they seemed spontaneous; I think it important to get down the basic ideas and then rework them, a lesson I learned from my parents

3:11:00 I have never been to Poland and it does not attract me; it seems like a country of sadness and loss; my sister has been and has encouraged me to go; I may do so but it has not drawn me; my mother's father was a famous publisher in Warsaw and their home was a cultural centre for writers, particularly Yiddish writers; Poland was the largest Jewish community in the world at that time; my mother was very much the academic and studied widely; she went to Berlin in 1928 to study; her younger brother Didek - the family name was Szymin - became the world famous photographer David Seymour and used the nickname Chim; that remains an important thread and inspiration in my life as I am responsible for his estate and interest in his work continues to grow; the path of being a photo-journalist was one that I considered; my visual orientation was very much inspired by my knowledge of him; he with Henri Cartier-Bresson and Robert Capa founded Magnum, the photo-cooperative, in 1947, and it continues in London, New York, Paris, Tokyo, as a leading and respected creative community of photographers; on my father's side, they were from more humble origins, from a small town called Kazimierz; my grandfather was a shoemaker of the upper shoe, not the sole; my grandfather married Sara (Mandelbaum), had several children, and then she died; he then married her younger sister, Chana, and had further children; my father was the second youngest of eight children by Chana, so I had many uncles and aunts and cousins; my parents left Warsaw in the early 1930s to join the intellectual movements in Paris, as did my uncle David Seymour and other family members; they travelled and wrote in Spain on the Spanish Civil War; my father wrote a book on it in Yiddish in 1938 (Krieg in Spanien - War in Spain) which included ten photos by David Seymour, so there was that collaboration between them; other uncles and cousins came to the US, to Venezuela, France and Spain, as a result of the War; the concentration in New York of family was a very rich source in my growing up; they had a country house in a small place called Lake Peekskill, an hour's drive north of Manhattan; by the time I was four the family had got together and bought a twenty-six acre farm in Flemington, New Jersey, an hour west of Manhattan; the plan was that my oldest first cousin, Paul, and his wife, Judy, who had come out of the Concentration Camps, and were unhappy working in Manhattan in the garment district, should go there; they joined other survivors who had moved to that area; they started an egg farming business where I would go as a child; it was a wonderful opportunity as there were lots of adult figures and cousins; we would work on the farm shoveling manure, collecting eggs, gardening, and was a great experience as a child up to high school years; there were many ways that it formed me positively, with the sense of community and shared experience; I think the parents were drawn together as survivors of the Holocaust and needed each other, and it was not always easy for them, but it was great for us kids; the multiple adult role models were important because my parents had a unique (?) way, working very hard with the demands of their new life; my father had a full time day job doing publicity and his writing was night time work, and there was not a lot of room for other things; my sister recalls that in a less happy way and she wanted more attention; I as a boy was possibly more independent, and had my cousin, George, in the same building; there were nice ways that I could compensate but I certainly saw that my parents were working hard, maybe that was one lesson, and I saw this process of writing and what they were after, their commitment to social causes, social justices, to issues after the Holocaust, important questions that I was inspired by; on the other hand, even from the time of being a teenager I had explicitly set the goal to be not like my father; he was absorbed with his work and his writing and had little time for vacations, hobbies, or anything else; he had little time too for me as a child; I have a few nice memories of him building a sand box for me, walking with me, and taking me to his work at the United Nations where he was a founding member of the UN Correspondents Association, and visits to the Yiddish Press in Manhattan where he worked; I was largely on my own and found my own way, but because of the farm, and early on sharing an apartment with an aunt and uncle, Collette and Chaim (Penchina), and their son Claude who became an important influence on me; he became a physicist and a professor, and was the only academic role model that I had; going through the school system in New York was very good, with the same group of kids from third to sixth grade, and then on through high school; I went to the Bronx High School of Science, a famous school in New York City, one of three where you were admitted by exam.

13:10:22 I went to local public schools - P. S. 75 Emily Dickinson School; we were the first students in that new red brick building which still exists; it was a nice experience; there were wonderful teachers who I still remember, and the constant set of students from third to sixth grade were a special group of about thirty; they all lived within walking distance so I came to know them and their families, and my parents came to know them too; the majority were Jewish as were the teachers; of the teachers I remember, for example, Miss Rosenberg in fifth grade; they were warm, caring types who motivated us to do well; at that time I was a normal boy, taking apart clocks etc., but with cousin Claude as an inspiration I was supposed to be doing science; I did science fairs and built projects while in that school, then in junior high school; in New York there was a special SP (Special Progress) course where you did the seventh to ninth grades in two years, then ten to twelve was high school; there was a competitive atmosphere but a good supportive one; in junior high school I had memorable teachers; Mr Scavone, an Italian, remains a vital figure who pushed us all; he taught social studies but was also our home room teacher; he was into theatre and would study the upcoming Broadway plays; remember Anne Bancroft starring in the Helen Keller story, 'The Miracle Worker', which we went to see soon after it opened; he would take us once or twice a year to plays; I remember the punishment he meted out when one of the students said that they really wanted to go to the play as they would miss regular classes, and he cancelled all the tickets to remind them that devotion to schoolwork was first and that plays were extra; the thirty of us in the special class were sent to another school on the edge of Harlem in a much more difficult neighbourhood, where most of the students were struggling academically; there was an attempt to build up the school and somehow it was hoped that we, bright, motivated kids, would raise the standards; I don't know whether it worked or not; after that I went to the Bronx High School of Science; there were three such schools in New York at that time; one, Stuyvesant, at that time did not take girls, and that decided my choice of Bronx Science even though it was further to travel; it was a wonderful school with inspirational teachers; I remember the physics teacher particularly; I was in that class when someone came in to announce the death of J.F. Kennedy in November 1963, and that is one of my moments frozen in time; by that time I was beginning to concentrate on physics and it was possible to take college level physics in the high school; the advanced placement group of physics students was a wonderful bunch of people; there was one girl in the class whom I took to the high school prom; since we travelled for forty-five minutes on the subway to get to school and back, there was actually little socializing after class, no community of people that were your neighbours as people came from all over the city; then I went to City College of New York

20:17:14 At school we also did plays; I remember playing Nikita Khrushchev in one play and for some reason having to sing 'Yes, we have no bananas'; I also enjoyed P.E. (Physical Education) but I was not especially strong; I liked to play baseball and was pretty good, probably helped by the hours playing on the farm; we would also sometimes play after school near home; when I was in High School I was selected to be on a city wide science radio programme and my mother would take me possibly twice a month for a one hour discussion by students on science topics; I began to develop my voice; I continued doing science projects - building a solar furnace with a Fresnel lens, a thermo-electric pile to generate electricity from heat, and projects of that sort; the construction mechanisms of building and making such devices, also telling the story on posters we would create; somewhere I have various medals given as prizes at these science fairs; science was an attraction, but so was speaking and being on radio; I had seen my father giving public lectures and also talking on radio and TV, so I had his role model and inspiration so it was not a trouble for me and I was effective at doing it; giving a lecture is a form of performance, of theatre, and one has the responsibility to make the work interesting for my classes and students; I do between forty and fifty public lectures a year; I have always been close to music but it wasn't quite my thing; my sister was a pianist and would perform as a college student; my mother would take me to performances of classical music; at one time I was given piano lessons but didn't take to it; I wish I had more skill in performing, although I enjoy singing it is not something an audience would enjoy; I listen to a wide range of music, both classical and folk; I don't work with music in the background nor go jogging with music in my ears

26:26:17 My father had more of a religious training at a traditional Heder, and he was quite knowledgeable and skilled at reading from the Torah in Hebrew; I went to a Hebrew school after regular school and had Bar Mitzvah at thirteen, and was quite absorbed by it; it fitted in with the sense of community in Upper Manhattan at that time; my father's work in Yiddish was also related to Jewish communities; but it was more secular and we never were a Kosher household though we participated in the holidays; I am a member of a rather unorthodox Jewish group in Washington that functions best as a community whose main form of ritual was the pot-luck dinner; that seems a positive thing; I am very proud of my Jewish heritage and background, which I find interesting and I stay connected to it, but it does not take the form of religious practice and I have drawn further away from the God emphasis; I see it as a cultural and community identification; on the question of whether I am an atheist or not, I try to evade it, not even thinking it important enough to worry about; I don't believe in God or an afterlife; I am sympathetic to Richard Dawkins' 'God Delusion', but I don't want to spend the time reading the book; there are important issues in the world; the Jewish notion of tikkun olam which means to mend the world, is one that I attach to most strongly; the sense that the world we have come to is flawed and each one of us has to do their part in mending it; I take that as an important value, and notions of social justice, equality etc. all influence my science and practice as well as the things I choose to work on; that seems the important obligation that each of us has; I try to inspire my students and others in a practical way to take up these causes, to make the world a better place

30:48:03 Although, as a Jew, there is sometimes the feeling of being an outsider, but it is not a dominant issue; I feel very self-confident and clear in my direction; I have a hard time making the choices of what I should and should not do; the group I belong to in Washington has holiday celebrations which I enjoy going to when I can, but it is okay to miss it; it is a very accepting group that feels there is room for diversity, and for people who are not Jewish to participate, intermarry, and so on; I am familiar with the issue but don't find it a troubling one; I am satisfied with my choices in life and what I have accomplished, though of course there are unfulfilled agendas

33:27:02 I followed an understandable and familiar New York trajectory through Bronx Science (High School) to City College; I arrived there in 1964 and spent four years there; it was quite wonderful; it is a well-established institution and it is free; now there is a modest charge, but all my education was free; it was a wonderful intellectual community; at the time there was a lengthy article in the New York Times describing it as the proletarian Harvard which fitted very well; earlier on it was a hotbed of the immigrants of the 1930s and 40s who were more socialist oriented; by the time I arrived it was much more just a strong intellectual source; it had some distinction of more Nobel prize-winners - Salk, for instance, came from there; Blacks, Jews and other minority groups could get an exceptional education, and was distinguished by admission by examination; when I graduated in 1968 they changed the policy to be by open admission and it essentially destroyed the institution; recently they have restored its exclusivity and hence its reputation; it is a troubling issue because we would like to be broad and welcoming to everyone, but there is a great satisfaction of that distinctive community which sought excellence, dealing with students who could not afford the opportunity; choosing college is maybe an interesting point; I see such a difference now with my students or my own children; they talk about all the colleges, they travel and visit them; at the time I was graduating from High School I had no one to guide me and had little choice as I was interested in physics, City College had a strong reputation in physics, so I went there; I didn't visit any other colleges, neither did my parents know much or invest much effort on my behalf; my sister had gone to Brooklyn College, another branch of City University, so it was just the natural thing to do; it was a good experience; I began to exercise my visual side with photography and for three years I was the photo editor for the Year Book and carried my camera every day and photographed sports events, lectures etc.; I built enough of a portfolio to have an exhibit of some forty to fifty photos by the time I graduated; my uncle had sadly been killed in 1956 in Suez while photographing for Newsweek; that was another memorable and tragic moment; my mother heard of the death of her brother on the news and that day still remains in my mind, I was nine years old; the incident was four days after the armistice; the story continues to unfold as just this month we received information about a detailed report on his death which we had not known of before; there are various controversies of whether it was Egyptian fire or British fire; the jeep that he was riding in, driven by Jean Roy of Paris Match who was a kind of wild adventurer, my uncle much more careful type, just drove through the lines and didn't observe the order to stop, and the Egyptian gunner shot their jeep up and it fell into the canal; it remains a story of vivid memories that keeps unfolding; there will be two upcoming exhibits about his work and I spend ten to fifteen percent of my time on his estate; yesterday I was at the V&A where I have donated some vintage prints to their collection; I also work with other museums like the International Center for Photography in New York, the National Gallery of Art in Washington, George Eastman House in Rochester etc., to promote and preserve his work; it is an interesting other side and I like to have multiple aspects to my life and work; this brings me into contact with a completely different set of people

39:19:24 So I was in City College and had a fruitful time, also pushing forward on the physics agenda where I was not doing well; that was a surprise and a challenge; I was supposed to be a smart kid, but I was not enjoying my physics, despite having wonderful teachers; I was doing fine in mathematics and other subjects, but once physics left the physical reality of pendulums and rolling balls and went to more abstract worlds of quantum and electro-dynamics, the abstractions got further away from me; I was working for my junior summer for a group of physicists and that was where I came into computer programming; I was working for this very big group, Bachman, Baumel and Lea, three City College physicists with Brookhaven Laboratories connections, aged eighteen; I was responsible for the computing side, data analysis of these bubble chamber photographs; we got a day at the Brookhaven eighty inch bubble chamber; you take 300,000 photographs and then you have three-dimensional views, and digitise the tracks, and then reconstruct in three dimensions what the path of the particle was, knowing what the magnetic field is, seeing the curvature, you know what has happened with the collisions; I started in the summer, sometime in June, and I was to take over the programming work of another student who had graduated; I had a very hard time getting into it, and learned a lot, and still in my mind is 8/8/66 was the day I finally succeeded in getting the program to work; the magnetic tape reels remained on my desk for two years, reminding me of the day I could begin to call myself a programmer; it was at that time that I met my dear buddy Charles Kreitzberg who was a young programmer working for the Computer Center and he helped me get things done; in the basement of Shephard Hall I had an office, and felt quite the man about campus as a junior and a senior, and at a certain point I became the supervisor of our computer; we got one of the early PDP8 computers, a refrigerator-sized box, and that was where I began to work, processing our own data in addition to the larger IBM mainframe; going to classes began to interfere with my learning; I had become absorbed with learning how to do all this stuff, and it was a great passion and intensity for those few years to make these programs work; there was a contrast between the physics I studied which became increasingly abstract, and the work of the physicists which I saw with Bachman, Baumel and Lea, where we had published a paper on the Y star 1616 resonance based on the data we had accumulated; I am acknowledged at the end; although I worked for very many hours on it I came to understand that the physicists had initiated the work and that I was merely carrying it out; that kind of abstract discovery of the Y star 1616 resonance was then challenged by others who said there was a mistake in our analysis; that abstract world of physics, the time frame of taking the photographs, two years processing, publishing a paper and getting challenged over a long period, was a dramatic contrast to the immediacy of submitting a deck of Fortran programs and it either ran or it didn't; if it didn't run it was my fault and if I fixed it and it did run, it was my success; there was a great sense of mastery and the cause and effect of my activity; I clearly had a capacity with this important skill using Fortran and other kinds of programming so I got quite strongly into this; I was also influenced by a teacher, Richard Hamming, a key figure in the history of computing; he was working in Bell Laboratories across the river in New Jersey and he came to City College to teach; I had two semester courses from him which I remember vividly because he filled the two semesters only with material that he had invented; he was very much in early numerical analysis programming; his book had the quote: "The purpose of computing is insight not numbers"; that has remained and I have used that phrase, and I say the purpose of visualization is insight and not pictures, so I have played on that; Hamming was certainly an inspirational figure who by his leadership in the field and his strong sense of his own contributions, provided a pretty potent example; this was before there were computer science departments or courses anywhere, and so we had a special opportunity to have him as our instructor

46:58:05 I use the word insight to mean a substantial jump in understanding which has some significance in the world; a former student of mine, Chris North, has taken that and tried to make insights his thing, and has measured the number and kinds of insights that people get using visualization tools; so it has become a notion that people find important because visualization and insight are natural partners, you see something, you understand something you didn't understand before; it might have cause and effect relationships, it might be anomalies, outliers, clusters, patterns, gaps, relationships - these are the kinds of features I would say are the components of an insight; I would say the insight needs to be more than the abstract, statistical or visual detection of an anomaly; it has to be related to the application domain and the significance of the insight; a mere insight that notices that one point is away from a cluster is not significant, but why it is there and how one might use that is important; I would say overall my perception of science and of my work is about cause and effect, and about the capacity to take insights and put them to work for future decisions; an insight is a momentary point, and there might be several of them which might lead you to a larger paradigm shift; insights are not mysterious to me, they are sometime 'ah ha' moments that I certainly have and experience with wonder and glee, but I see that the pursuit of insights is a much more systematic and methodical process; that we can organize ourselves so as to make insights, and that most of what I do, whether discoveries or innovations, are by a systematic way of proceeding and then recognising when there is something important about an insight; Poincaré moments are ones of preparation, incubation, illumination, verification; I very much invest myself in a problem, you become immersed in it and look at it in many different ways using the large set of skills developed to look, potentially understand some relationship or anomaly, and at some point you may have an 'ah ha' experience; the one that is very clear and I have written about was working on the idea of what have come to be known as tree maps; I was looking at a way to create a visual representation of the contents of your hard drive; at the time in the Lab we were running we had a Macintosh with a large hard drive of 80mb shared by fourteen users; when that filled up it was a chore to look at all the fourteen users to see how much each person was using and who should I bother about cleaning up their storage space; I wanted to have a single visual presentation that showed me the proportional use; in that case you might think of fourteen rectangles on the screen proportional to the number of bytes being used, but I was after a richer presentation which would be a recursive, with the notional fourteen rectangles broken up into the folders, and folders within folders etc; I had tried many ways of trying to carve up the screen, space filling so that it would fill the screen with no space between these rectangles; I had been working on this for months and drawn many versions but none had really worked; then there I was in the faculty coffee lounge, I wasn't thinking about it, but suddenly had this 'ah ha' moment and saw how to do it; it took me three days to convince myself it was really correct; there was only six lines of code to get this recursive algorithm but there it was; it was very much like Poincaré's description of how he stepped on the stair of the carriage and had the flash about differential equations; I have had a modest number of others, a few problems that I have solved in relaxed moments, but mostly my discoveries come by a much more conscious, systematic approach, working on the problem; I am quite flexible in the way I work; I don't carry a notebook with me; I am a list-maker and very persistent on things; my favourite way is socially working through; I work with my graduate students and we will meet for a few hours once a week and continue to work on a problem that we are dealing with; for example, we are now working on electronic health records; I have 10,000,000 patients and I think of each patient as having a series of events of different types; what I am interested in finding out is whether there are patterns in these event streams of these patients that have not been noticed before; can you specify a search that will give me all the patients that have more or less of a certain pattern, and what do I mean by more or less; this is a new problem called nearest neighbour problem in K dimensional space; it has an unique problem in that if we had a case in which all the patients had the same events in the same order and we only had to deal with the differences in the timing between them, that is easy; but we have a nastier problem because sometimes we have missing events, sometimes the orders are changed, and there is an additional event, so you get these nasty metrics for nearest neighbour which deal with two measures, one of how similar in time and also we have exact numbers of events, or are there switches, insertions, or deletions; there are different ways to solve the problem so they are implementing software and I am thinking about it, and in these weekly discussions they will propose something and I will agree or suggest something else; I am also very strong and broad in reading of background literature; I say to my students, even undergraduates, that I expect them to read every paper that has ever been written on the problem they are working on; they are quite stunned by that idea, but that forces them to choose something narrow enough that they can accomplish that; again I see it as a social thing and suggest they write to the authors of the papers and send them their draft; they are quite stunned by that, but many times they are graduate students like themselves, or young professors; even senior professors will be responsive if you approach them with specific questions about their work and show an interest in them; I am very much devoted to the idea that you should be willing to send your draft paper to all the people you reference; I have a very wide net of people I approach; I make my work much more social than my colleagues do; most computer scientists have a more introverted approach; I feel I should be willing to let them challenge my work even at an early stage; if I am afraid to show it then there is something wrong; it is fun tracking down literature on the area that I am working on; tracing back to my family background which was very much in the humanities; my sister became a professor of English, my parents were journalists with literary and political interests, so part of a very different world from that of science; I kept their world as very much a part of my life, so my struggle with physics at college was partly because I was so interested in other things; I was very influenced by Marshall McLuhan whose writings had a great impact on me; I went to see him and heard his lectures in New York; he was like his writing, a wild character; he was Professor of English at University of Toronto and had these fantastical ideas and brought together stuff, created a language of hot and cold media which was unfathomable to most; sometimes just a little weird and contradictory, sometimes brilliant; his point about the global village destroying linearity and sequentiality and privacy, was hard to see but he was right in a metaphoric and a real way; I came to accept that the specialist approach, either you are a photographer or a physicist, you have to major in something, that was really not the way I saw the world; I wanted to do it all and to have these different parts of my life all supported, and McLuhan confirmed that that was Okay, and that a sequence of specializations was a possible compromise, and that has been more the metaphor about how I see my work; on each part I focussed narrowly and in depth, produced an important result, and then consciously I would move on; I did not want to professionally stay in the area of my previous success, but wanted to be the eternal graduate student reading a new set of papers and a new literature that I didn't know before, and trying to understand it, and pick up what I would consider to be the low-hanging fruit in this new area and moving on.

Second Part

0:09:07 McLuhan’s notion was very important to me; as a student I printed up little business cards saying 'Ben Shneiderman general eclectic - Progress is not our most important product' which was a play on the General Electric slogan of the time; I came to accept the idea that I may not be the best at any specialism but I might be the best generalist; though in that competitive atmosphere that I was in to succeed you had to be the best of something, a harder thing to do was to be the best generalist; professionally it is not an ideal choice although it has been useful and satisfying in my life; those who chose a specialist path and follow it have an easier career; my own approach meant I had to go from interest to interest, whether photography or physics, computing and applications of computing into many different fields; I have collaborated with political scientists, English scholars, chemists, and biologists, and I find these enriching, and have given me a set of multiple perspectives that I find useful for almost any problem; I can make conversation in a lot of different circumstances; I came to feel that after six months of working on a problem with political scientists about Supreme Court and District Court citations over a hundred and fifty years, I was knowledgeable enough to make contributions in the field; the political scientists would publish their findings while I would publish the computer aspects with my students; so the capacity to look at a problem both as a specialist in one problem but generalizing to another problem is probably my best skill; five or six years ago I began to work with molecular biologists about gene expression data analysis; I found it fascinating but a bigger mountain to climb than anything I had done before; after three and a half years of working on it, producing results, I did not feel I had achieved that proficiency to be able to discuss and carry forward in that field; I had a weaker high school biology training and the gap between what I knew and what I needed to know was quite large; I had thought our work should get an acknowledgement but the ethic in biology is that all names go on papers, so there are papers that I am author of but I could not explain them to you adequately; it is a challenge to me, because the conflict between disciplines and their ways of working is unsettling; I felt for a while that I should put those biology papers in a separate section of my resume because I didn't feel I could claim the content of those articles; it is certainly different from computing where individual authors or small groups would be typical; however, the eclectic style of working suited me and I enjoyed the sequential relationships with professionals in different fields to solve their problems and stimulate our work in terms of new problems; the current example of electronic health records is yet that problem again; I think working in medical computing is a noble application of computing which is in harmony with this tikkun olam notion of mending the world; we also broaden computer science not only by its application but by bringing new problems that had not been attended to before; my colleagues think that when I work with English professors that I am doing them a favour; they rather disparage true interdisciplinary work but these English scholars are brilliant and the challenges they bring to us are a novel and important expansions of computer science; by having this sequence of collaborations and working in interdisciplinary teams I do have to establish the ground rules with the scholars that we are not there as their programmers, but are partners; we need to publish in the computing literature and my students need to get PhDs in computer science not merely programming for them; this they come to understand; we have been successful as a group as we have kept explicit several of those principles; the breadth of my reading is great, not just in computer science, but in the broader areas that I have worked in

9:04:04 On graduation from City College with a bachelor's degree in maths and physics I was looking to go to graduate school in computer science; by that time I was more knowledgeable about which universities were good or bad; the favourite place was Carnegie Mellon University to work with Allen Newell and Herb Simon who were then working on the early stages of artificial intelligence; I went to Pittsburgh, paid my down payment, excited that there was a teletype in the basement of Mudge Hall near my dormitory; the Head of Department was Alan Perlis, a famous computer scientist; the thrill of going to work for Newell was very much on my mind, but it was the time of the Vietnam War and the US draft board had said that my going to graduate school was not an acceptable deferred occupation; Carnegie had given me a fellowship and they then gave me an instructorship so that I would be working; I returned to my local draft board and they still said it was unacceptable; this was problematic and I wound up going to teach at a two-year college on Long Island; I was invited to be a full-time instructor there in the Department of Data Processing; Harold Highland was the department Chair and he took a liking to me; he had a son my age who was in similar circumstances so understood he was helping me; my draft board took this to be my national service; I taught fifteen hours of classes a week on Fortran and data processing equipment in a vocational environment; you teach people a very professional skill and they get a job doing that; I taught there for three years; I worked hard and that is how I came to be at the State University of New York at Stony Brook, also on Long Island near Brookhaven Labs; it was a new university and I came to take classes there; computer science was now emerging as a discipline, there was a department but as yet, no degree; I was also doing applied mathematics; I became the first PhD in computer science at Stony Brook in 1973; another colleague of mine there, John Hennessy, is now the President of Stanford University; we had a good group there among whom was Isaac Nassi; the university, being new, was not a very supportive place; my memorable moment outside my office - (I had come as a graduate student but had become an instructor) - there was an open steam vent which they were working on, and one night a student fell into it and died; it was that kind of lack of care which typified the place; however I had a couple of excellent instructors, Herbert Gelernter among them, an early pioneer in artificial intelligence, geometry theorem proving; his classes were magnificent; remember a conflict I had with him when I had got excited about doing some of the theorems; I offered to write a theorem proving program, which was pretty ambitious, if he would excuse me from the mid-term exam; he refused and really annoyed me, though I think he may have been right to force me to do the basics; my own professor there was Jack Heller who had worked on databases and mathematical theory of same; it was there also that I learnt the practical connection because he had been working on museum databases, and we worked on the Museum of Modern Art’s fine arts catalogue; my dissertation was a theory of these richer data structures called 'The Graph Theoretic Model of Data Structures' (editor: correct title is Data Structures: Description, Manipulation and Evaluation); it had the application from the Museum and also the generalization of the mathematical theory of graphs; I was able to apply graph theoretic strategies to making efficient models of how to implement these things; I had begun the dissertation writing process in the way that I now work with my students; the traditional approach would be to do your work, then write your dissertation, and then publish your papers from that; I was much more into doing the work, publishing four papers, then writing it as an integrated document to make the dissertation; the advantages there was in harmony with my social approach, to get your work out and let people read it, criticize you and find out what is wrong with it in time for you to make changes and adjustment to it; it also pushed me towards having the work done so it could have an impact on the Museum of Modern Art, and other museum projects that Heller was working on; the other point is that real work should be applied; I don't subscribe to the strong separation between basic research and applied research; I think you do your best basic research if you have a real problem; I follow this with my students, even undergraduates, that you must have somebody outside the classroom for whom you are building this; that has been a very productive approach for all students; I see so many times papers that are done in an abstract way; it is true that maybe twenty years from now somebody will find an application, but I don't have time to waste on abstract problems

18:53:24 I met Ted Codd - very British - I was a very early player in that space of relational data model; he was a wonderful inspiration; his February 1970 paper certainly was one of those important moments where he not only presented an idea but also its ramifications; I was at that time working on those problems and visited him at IBM; one of the papers written for my PhD was published in 1973 called 'Optimal Database Reorganization Points'; one of the issues at that time was when you added or deleted items from a database it would become less efficient, and each successive query would take longer; the desire is to reorganize often but that is costly, so you need an optimum strategy; I had a very lovely solution for that, a fairly mathematical piece, which stimulated thirty-five papers in the next few years on this topic, and opened up a new field; as a graduate student that is the kind of thing you want to see happen and that was one of the components of the dissertation; as a graduate student I had been a volunteer for the ACM at a professional event, and hosted the British software engineer, Michael Jackson; he came to speak in New York and was proposing these new notions of structured programming; rather than the 'go to' of Fortran it had a set of structures of 'if then else, do'; Jackson presented this idea and it seemed a natural jump; if you don't have 'go to' then the flow charts that were popular with arrows connecting to boxes were also not valuable; you had instead recursive structures of 'if then else', then looping structures and sequential execution; there was a nice mathematical proof that those three structures provided all the power that you needed in programming; I began to draw boxes nested within boxes and developed a set of box shapes that would represent this; it was a fifteen minute invention while listening to his talk; I returned to Stony Brook and showed my colleague Isaac Nassi as he was working directly on that for his dissertation; he got excited as he knew the justifications and why, and together we wrote this up and he helped expand the idea; we called them structured flow charts - known as Nassi-Shneiderman diagrams, NSD - Nassi prevailed on me to make an alphabetical order to authorship; we submitted this for publication in the 'Communications of the ACM' and it was returned in a few weeks rejected; the rejection letter said the authors should collect all copies of the paper and burn them; quite a shock for a graduate student to get something like this, made me think I might be on to something; others thought it was clever, innovative, useful, so I had sent it to a few people; one was a senior colleague whom I respected; he didn't respond, but a few months later another colleague sent me tear sheets from a popular science magazine in which he had published this idea under his name; that was really troubling as a graduate student; people said I should file a complaint to the professional society; I just left it and for many years it troubled me as many of the books mentioned his name; eventually this got sorted out and it’s widely known as Nassi-Shneiderman diagrams; we then published this article in a non-refereed newsletter of one of the professional societies for programming languages and there are thousands of citations to this paper, hundreds of software packages, patents that follow this, there is an international standard for doing these things; again it was one of those wonderful fifteen minute innovations that we did very little to promote but travelled very well; it is less used now as are most of these diagramming techniques, though it is still widely used in Germany; it is a great episode about the paths of creativity and how certain ideas travel well; maybe the first hundred papers were variations on the theme; it was a simple idea that people could understand and opened up the door to many possibilities; I finished my degree in 1973 and took a position at Indiana University as Assistant Professor; I had recently married and went off to do my service to the mid-west of the US; at that point I was quite strongly motivated to repay my debt to the state universities where I had studied, and Indiana University was a state university; it is a small university town at Bloomington with a small campus; there were five of us junior professors hired the same year, all with families and all Jewish; we became close and all had children; it was a nice experience for three years, and my daughter, Sara, was born in Indiana in 1975; after that, the desire to get back to the East Coast and its neuroses was attractive; I would have liked to have been closer to New York and to family but my wife wanted to be further away; the compromise was to come to College Park at the University of Maryland; there was a particular attraction there because there was a senior faculty member, Ed Sibley, who was head of a small department called Information Systems Management which dealt with the database issues for which I was making my reputation; Sibley was interested in my work on experimental studies of programmers initially; I had learned my trade as a psychologist at Indiana University which had a first rate psychology department with B.F. Skinner; one of the young faculty members there, Richard Mayer, taught me psychological experimentation and I taught him about computing and together we did some good work; he has gone on to a distinguished career in the University of California, Santa Barbara; that was a good partnership and I was gaining some reputation with early studies about programmers; when I was at City College I described working on Fortran and my friend Charlie Kreitzberg and I wrote a little guidebook on how to do good programming in Fortran; we eventually wrote this up in a book 'The Elements of Fortran Style' in about 1971; it had a moderate success and inspired a whole series of other books, but we certainly started that trend; it was a series of rules about how to write good programs, the choice of variable names, modular design, indentation, commenting, a variety of stylistic issues; those were really conjectures based on our experience; I became imbued with the notion of conducting empirical tests where I would make a good and bad version and give them to someone to debug or interpret and I would measure their performance; I was able to alter the independent variable and measure the dependent variable, and that is the way these experiments began; when I came to Maryland this Information Systems Management Department was in the Behavioral and Social Science School not in computing; that was a very interesting opportunity; it was close to the Psychology Department so I was attracted to that; Ed Sibley had been a collaborator and had been helpful to me; early on he had engaged me in doing short courses around the country and abroad on database technologies, before I got my PhD; we had a very good group at Maryland which did well; however it was a small group and eventually lost the political battles; that department was dissolved and most of the faculty left; two of us went to Computer Science so I went to my natural home; I was not especially welcome there as I was seen as a little different; I was into these psychological studies and not especially celebrated; when you bridge into interdisciplinary and newer areas you are going to have some trouble with some fraction of the established community; I had got tenure based on traditional computer science and had demonstrated my mathematical skills

34:37:19 I enjoy lecturing and teaching; early on when it was difficult to get funding for research on the psychological experiments of programmers, I would put my students to work and would make their semester projects to be an empirical study; I was most proud to be able to publish in professional journals with my undergraduates; colleagues disparage that by saying that if the work is so simple that an undergraduate can do it, it can't be very good; but I had done the mathematics, taught mathematical analysis, I can do that stuff, and I moved on to these newer forms which were fun for me; when Sara was in third or fourth grade computing was all the rage at the time; her teacher came and asked me to recommend a textbook to enable them to teach the children the language Basic programming; I was appalled to find that the books were either trivializations or if they taught you programming they depended on high school mathematics; I quickly wrote them about seventy-five pages of programming introduction where the examples were all graphical which kids could understand; it worked well for her class; the DC school system asked me to polish it and they printed 3,000 copies for summer computer camps; we did them for four different computer systems and Sara worked through the problems for one of the versions; I used to help my children with their science projects; I remember one of them was using the sound generator on a computer to generate sound at different pitches and volumes, and to see what was the difference that people could determine, in amplitude or frequency

38:50:18 By 1980 I had taken the different work I had been doing on the experimental studies of programmers and wrote a book called 'Software Psychology', a kind of marriage of disciplines; it was taken up by both computer science book of the month clubs and became a hot topic; by 1982 a group of us were meeting in Washington; I had been the organiser of the Software Psychology Society whose claim to fame was “no members, no officers, no dues”; we had a mailing list of 600 and forty to sixty showed up each month at George Washington University and I organized the speakers; for twenty years this was a very influential group that had both local and national impact; we decided to run a conference in 1982 where we hoped to bring together two to three hundred people; it was held at the National Bureau of Standards and Technology and we drew over 900 people; people were suddenly interested in the topic; the personal computer had appeared though it was still in the days before the Apple Macintosh, but these ideas were beginning to grow; the following year the ACM Special Interest Group on Computer Human Interaction was formed and they took over the conference, and it has continued to be the main source drawing as many as 3,000 every year; that bridge of disciplines was the natural spot for me to be in and I am pleased that it had succeeded; most computer science courses (departments) have one called Human-Computer Interaction; there is still some resistance in computer departments but it is definitely a well-accepted field; by 1986 I published the first edition of 'Designing the User Interface'; in the fourth edition I was joined as co-author with Catherine Plaisant who had been my research collaborator since 1987, and we have just released the fifth edition this Spring; that has been the main textbook in the field and has helped define it; my satisfaction has been to see that go; the early edition of the book focussed on a small group of users like programmers and medical specialists; by the fifth edition we are talking about four billion cell phone users, Wikipedia and YouTube - the emergence of these platforms of user-generated content is a remarkable transformation for which I am proud of my role; in some areas it is seen as new stuff and not quite science, but this softer science is the way that many things will go; I have written on the history of physics, chemistry, controlled laboratory science of the natural world which will continue; however, I suggest there is a need for a new way of thinking which I call Science 2.0, published in 'Science'; this suggests that the content and the methods need to be changed as well; my eclectic background gave me the capacity to see this perspective; not everyone accepts it but I think the notion is in the right direction; it shifts the methods from control of repeatable laboratory conditions to case studies and interventions in the way that you can do on the Web and have ten thousand data points in two hours which give you remarkable capabilities; shows the trajectory from traditional computer science to these newer bridging sciences of psychological applications and a larger vision about the transformation of science, always with the intention of making the world a better place; as a profession it is a difficult trajectory to do because I'll never be accepted as a leader in medical computing, but I feel I have made my contribution; by going from one discipline to another you start out again as a graduate student which I am happy to do; McLuhan taught me to be a dilettante and of being an amateur as a virtue, and starting on something new is satisfying as part of my everyday work; another contribution that I am proud of is hypertext; we worked on this in the mid eighties and our contribution was the link; when you click on highlighted words and go somewhere, that was our work; many people had commented on the link which is usually traced to Vannevar Bush's article in 1945 which talked about Memex; his notion of the link was that you type in a code number for a document and your microfilm devise with spin to get you there; we began to develop these electronic encyclopaedias and there was an 'ah ha' moment when we had a videodisc player for the images and a plain green screen for the text; we had captions and descriptions and then a numbered list of where you wanted to go next; in one screen there was a very short caption but a numbered list of Polish poets; the text in the caption and the list of names was almost identical and I asked Dan Ostroff, the programmer at the time, how we could replace the caption but then highlight the names and click on them; that was done and we did a number of studies comparing different forms of highlighting; we developed light blue underscored, Tim Berners-Lee saw that and in his 1989 manifesto for the Web he cited our work; it was a small contribution but had a broad impact and gave us great satisfaction; there is controversy about whether we can take pride in this claim - others were certainly working on this - but we made this particular way of doing it, provided an authoring tool and a rich environment to support all that, and that became the inspiration for Tim Berners-Lee; we had published this under the term 'embedded menus'; Tim was clever enough to call them hot links

49:51:22 Tim Berners-Lee deserves credit for inventing the Web; we were working on hypertext things that ran on a single computer and his mind-blowing idea that you could have such a link and it would jump to another computer across a network to retrieve it was wonderful; I met him before that at the Paris hypertext conference; we were building systems called Hyperties which had a commercial version as well, so we were an active player; we built the world’s first hypertext scientific journal - the July '87 issue of the CACM; we did the worlds first electronic book called 'Hypertext Hands On' which was in a paper form but it was the disc that mattered; it is the first book in the Library of Congress that has electronic media as part of it; we definitely made those things happen at scale and in marketable ways to reach thousands of people, including Tim Berners-Lee; I am on his scientific council for the web science research initiatives so see him a couple of times a year

51:17:14 On the future, I have already hinted at some of this in the 2.0 argument but the expansion of social media, of user generated content, the empowerment of individuals, and the restructuring of social and political norms are beginning to happen; as you take Facebook and YouTube and apply them to national priorities like energy, health, education, sustainability, environmental protection, you get a transformative effect; I think that is what is really dramatic; between the fourth and fifth edition of 'Designing User Interface', this has just flourished in a way I would never have predicted; YouTube which wasn't around five years ago is now number three on the Web after Yahoo and Google is a startling indicator; I think those things are only just beginning to unwrap in their implications to society; the new forms of media with the dissolution of newspapers in the US and the profound restructuring of societal norms is yet to be fully appreciated; when journalists ask me what is the next killer app I have a straight answer which is trust, empathy, responsibility and privacy; it is those who understand how to generate trust, support empathy, how to make responsibility for failure and success more clear, and protect privacy, also how to raise the level of motivation so that more people participate and contribute, what's the role of egoism, altruism, communalism, and how do we make these social structures in new ways; that is more or less the way I see the world; revising social structures, political structures, economic structures is a fascinating potent transformation; how medical care with change and people become more responsible for their own medical records and treatment; working on innovative social structures is as interesting to me as working on innovative algorithms or devices; I have had the satisfaction of seeing our more recent work on visualization become a commercial success with Spotfire; the idea which we published in a 1994 paper is still one of the most widely cited in the field with Christopher Ahlberg as a visiting student; he formed the company in 1997 which grew to two hundred people and was bought in 2007; it is nice to see a success story like that, and I think people appreciate our Lab, not only for lots of papers and good students, but we have had success stories on the commercial side too; Spotfire and Treemaps have a dozen or more widely used open source versions as well as others; the intellectual challenge is to know which idea will travel well; I am always trying new ones, putting out seeds, and some go well and others struggle; my efforts working on creativity support tools has produced a moderate result but has not become a large field; other ideas like the link on the Web or the notion of human-computer interaction, some are small focussed, others broader restructuring of disciplines; people think I do good demos, but finding ideas and developing them in a way that has an impact and then communicating them effectively is what our job is; sometimes it works well and sometimes I struggle, but still want to get ideas out there.