This week’s post is by one of our Student Research Assistants, Rafa Morris, who has recently graduated from our BSc Philosophy & Computer Science. Many thanks to Rafa for his excellent work on this project and blog post!
As a growing and multi-disciplinary field, the thematic landscape of AI Ethics can be hard to identify, especially if you are working through papers manually, paper by paper. Instead, we can explore how research themes cluster and connect across the field by representing thousands of publications as networks. This post details the first two months of my research assistantship at the Computational Philosophy Lab, where I worked to construct my first network model of AI ethics literature using 23,000 keyphrases extracted from JSTOR papers.
I hope you find the following as interesting as I did.
The corpus was put together by Brian Ball using Constellate, JSTOR's text-analysis platform (part of Ithaka), from every paper returned in a search for "Artificial Intelligence" AND "Ethics". Since the content is from JSTOR, the data I had to work with is broad in kind: alongside peer-reviewed journal articles it contained research reports, book chapters and magazine pieces. Each paper carries up to ten keywords, and these are the starting point for the first network.
Consider the article with this keyword list:
autonomous weapons
weapons systems
autonomous weapons systems
killer robots
robots
frank sauer
content uploads
control
lethal autonomous
lethal
It is no surprise to learn the paper is titled “Stopping Killer Robots”, published in Arms Control Today.
You may also notice that not all of those keywords are equally useful. "Content uploads" is a meaningless keyphrase - I would hope every article has content uploaded. And "frank sauer" is just the author who will already be in the paper’s author field, so as a keyword the name adds nothing new to any network we build. Immediately then, the first step of my work involved significant dataset cleaning to find and strip out this noise. Noise appeared in two forms: unhelpful keywords like "content uploads" and redundant entries that were not papers at all like bibliographies.
Initially, the corpus had 30,000 unique keyphrases. To clean this data, I iteratively built a cleaning pipeline consisting of a combination of standard word filters with manually compiled word exclusion lists. The latter repeatedly grew through every single stage of my work as unexpected and misleading keywords kept emerging. I also targeted and removed non-substantive entries like review extracts, indexes and bibliographies, by their titles and document categories, which brought the corpus from 7,016 papers down to 5,268. A large proportion of the keyword reduction was handled by lemmatisation and some standardisation, which combine the different grammatical forms and spellings of a word onto one root so the network can treat them the same. "Technologies" collapses into "technology", "ethical" into "ethics", and so on. A large portion of the removed keyphrases were author names and institutions, and once this cleaning pipeline was complete, I had 23,485 keywords to work with.
While all of these keywords can now be considered ‘useful’, they were not all equally valuable for understanding the wider structure of AI Ethics literature. A keyword that appears only in 4 papers, for example, is unlikely to reveal much about the broader relationships within the field. Instead, it risks distracting from any real and meaningful patterns in the data and risks making my graphs look uglier, strewn with unnecessary outliers!
To avoid this, the team and I agreed on a keyword threshold: a keyword had to appear in at least 10 papers for it to make the cut and contribute to the network. We realised this threshold in what would become our first 'robustness check', I compared thresholds of 3, 5, 7, 8, 10, 12, 15, 20, and 30 occurrences and their impact on the network. Keeping the threshold at 10 occurrences provided the best balance, where meaningful relationships were preserved without diluting the data. This final cut produced 541 unique keywords to use for our network.
Looking at the forty most frequent keywords in the corpus, the most frequently occurring are no surprise: "technology" and "artificial intelligence" appear in 560 and 471 papers respectively. However, further down the rankings we can begin to infer the more distinctive themes of the field. Keywords such as "military" (182), "defence" (170), "privacy" (110), "Chinese" (96), and "fiction" (75) point towards some of the major themes we can find running through the corpus, from defence and geo-politics to data privacy, and even science fiction.
I could now construct a keyword co-occurrence graph as the first network. Here, each keyword becomes a node, and two keywords are connected by an edge when they appear alongside each other in a paper. A handful shared no co-occurrence link with any other keyphrase and dropped out, leaving 534 nodes in a single connected graph, joined by 4,474 edges, with every keyword reachable from every other through some path of shared papers. Edge weights (the impact of each keyword connection) are determined using Jaccard similarity, which measures how much two keywords overlap by dividing the number of papers they share by the total number of papers in which either keyword appears. This rewards keywords that are distinctively related and reduces the tendency for a ubiquitous term like "technology" to dominate simply because it co-occurs with almost everything in AI Ethics.
The visualisations of the graphs that furnish this blog are made using the software Gephi. We used the weight-directed force layout algorithm ForceAtlas 2, which treats the network as a physical system where every node repels every other node while the edge weights act as springs that pull together connected nodes, according to how closely connected they are. This makes the spatial groupings visible in my graphs, and you will see that closely connected keywords end up clustered together.
With the keywords laid out as nodes and the connections between them weighted, we can now start to investigate how the field organises itself. Community detection algorithms look for groups more densely connected to each other than to the rest of the network, measured via modularity score. Modularity captures how much more densely connected the proposed groups are than you would expect in a comparable random network. A high score means the partition has gathered tightly connected keywords together while keeping weakly linked ones apart. This is perfect for finding themes in our network efficiently. I selected Louvain as the most suitable algorithm, the same method used in one of the posts about Margaret Cavendish on this blog.
Louvain was selected based on its performance against other conventional community detection algorithms, namely Label Propagation, Spectral Clustering and Leiden. Label Propagation collapsed the whole graph into a single community with a modularity of zero, while Spectral Clustering produced a very uneven result, gathering more than half the keywords into one cluster.
The closest competitor was Leiden, a refinement of Louvain built to guarantee that every community is internally well connected. Although the two methods broadly agreed on how the network was organised (agreement measured through metrics ARI and AMI) Leiden returned the same number of communities without producing a clearer partition.1 Its modularity came out lower, about 0.32 against Louvain's 0.37, so Louvain kept both the highest modularity and the most balanced, human-interpretable partition, and that is what the rest of the work is built on.
The first pass of Louvain produced six communities. The two largest - “technology / digital” and “philosophy / science” - each held more than a hundred keywords, and together with “artificial intelligence / defence”, which combines two clearly distinct themes into one cluster, they made up about 63% of the vocabulary between them, so I ran Louvain again within each, producing a three-split partition of 20 communities in total. Splitting these broader communities reveals smaller, more interpretable themes. Seeing the four focused groups: “pentagon”, “technology”, “autonomous weapon”, and “strategic”, is far more informative at a glance than a single, vague “artificial intelligence / defence” community.
As covered earlier, the keyword frequency threshold has a significant impact on the network so before we begin any interpretation work, it is crucial to scrutinise the way that other key variables influence our network map. First, I ran Louvain 50 times while changing only the random seed, which affects the order that keywords are processed. The more specialised groups appeared consistently, with the keywords of “health / economic” staying together 74% of the time. The broader groups were much less stable like “technology / digital” at 48%, which is what guided our decision to split the largest communities further.
I then varied the resolution parameter, which controls whether Louvain favours fewer larger communities or a greater number of smaller ones. At the default value of 1.0, the network produced the six-community split used throughout this analysis. Lowering the resolution to 0.8 or 0.9 had little effect, with the resulting partitions still matching the original structure around 70% of the time. Above 1.0, however, the larger communities began to divide further, reaching seven communities at 1.1 and ten at 1.2 as the algorithm became increasingly inclined to form smaller groupings. Here, it might seem more efficient to increase the resolution and avoid splitting communities hierarchically later on. In practice, however, this lowers the network’s overall modularity and re-splits every community indiscriminately, breaking up groups that were already faithfully representing the structure of the field.
Both tests indicated that the specialist communities were robust, while the broader themes were less sharply defined. This is a great result as it is exactly what we would hope to see in a network capturing meaningful structure!
With this sweet relief of meaningful structure, we can begin analysis, with this as our final list of communities:
Paper count here means the number of papers assigned to each community. Each paper goes to the community holding the most of its retained keywords, so the counts show where a paper’s keyword profile is the strongest.
The names of each community are applied automatically by collecting all its keywords and scoring them with TF-IDF, which calculates the terms which are common inside a specific community but rare across the others.2 This is a reliable way to make the label capture what makes a group distinctive rather than just its most frequent word. The top frequency word becomes the first word in the label and the highest ranked TF-IDF term becomes the second as shown below.
The communities line up with the publishing landscape mostly as you would expect. The philosophy and science-studies community is concentrated around journals such as Isis (43 papers), Technology and Culture (16) and Synthese (11), with the University of Chicago Press the dominant publisher across the cluster, behind nearly 14% of its papers, and Springer following at 10%. The learning community is centred on education and ed-tech journals, led by ASEE Prism (43) and Educational Technology & Society (31), with Springer the largest publisher at about 9%. The “ethics / business” community is closely tied to outlets such as the Journal of Business Ethics, the Journal of Medical Ethics and the Hastings Center Report, with Springer alone accounting for 22% of the community's papers.
However, the “defence / pentagon” community that was split from the AI-and-Defence community has a vast majority of its papers published by defence-sector outlets like Inside the Pentagon and Inside the Army, instead of the traditional journals which comprise the majority of the other communities - 78% from Inside Washington Publishers alone. This literature is not peer-reviewed research, yet it represents how a large proportion of the debate about AI ethics and defence is happening on its own platforms, away from the academic one.
From this first round of analysis, the keyword network provides one useful picture of how the AI ethics literature holds together. A stable core of large communities, organised around artificial intelligence, philosophy and learning is not entirely surprising. What stands out to me is that the hierarchical splits produce subcommunities that are immediately interpretable, revealing familiar areas of debate already present in the literature. Clusters concerning geopolitics or social media platforms like Facebook highlight how these topics emerge tangibly from the data, even when they are encoded as assigned keyphrases.
At the same time, we should remember that this network is only as good as the keywords it is built from. As the “killer robots” example showed, keyphrases supplied by authors and editors can be inconsistent and sometimes capture highly specific ideas that do not reflect the wider structure of the literature. So, in the second stage of my work for the CPL I have moved beyond using keywords to explore more computationally advanced ways of identifying themes in the literature. I will explore this in a future post on the Computational Philosopher.
This was a lot of fun to work on and it has been very fulfilling sharing my results with you!
Thanks for reading, and thanks to everyone at the Computational Philosophy Lab for the ideas and support along the way.
ARI (Adjusted Rand Index) and AMI (Adjusted Mutual Information) both measure how far two groupings of the same items agree, where 1 means they are identical and 0 means they agree no more than chance would. ARI is based on pairs of items, and counts how often the two methods agree on whether a pair belongs together or apart, corrected for chance. AMI is based on information, and measures how much knowing an item's group under one method tells you about its group under the other.
TF-IDF (term frequency-inverse document frequency) is a statistical method that scores a word by how often it occurs in a particular document relative to how common it is across the whole corpus. The effect is to reward words that are distinctive rather than simply frequent everywhere, so a word that is frequent in one community but spread across all of them scores low, while one that is frequent in a single community and rare elsewhere scores high. Here each community is treated as the document, so the top-scoring terms are the ones that set it apart from the rest.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.