AI | Jim Bagrow's Blog

Self-Cite Circus (SCC)

Wed, 30 Sep 2026 15:32:48 -0400

A story has been making the rounds about a professor who has written hundreds of articles so far in 2026, many by himself (see also Reddit, Washington Post). Apparently, Nicholas G. Polson, a respected professor at the University of Chicago, wrote (“wrote”) 258 preprints by the end of August. Thanks to AI, this is the kind of thing we’ll see more of in the future.

People are asking: why do this? What’s the motivation? And one answer I keep coming back to is getting citations.¹ Posting this many papers at one time seems like a good vehicle for developing a huge number of self-citations.

Assuming you start from scratch and “write” one paper a day for a year, and each paper cites every preceding paper, then the first paper will get 364 citations, the second gets 363, and so on. This gives the total: $$\sum_{k=1}^{364} k = 364 + 363 + \cdots + 1 = \frac{364 \times 365}{2} = 66{,}430$$ That’s an incredible number of citations for a researcher to accrue in a single year!

I don’t know if this is happening in the wild, but I’m going to plant a flag and name the phenomenon:

Self-cite circus (SCC): A scheme to generate a superlinear, compounding number of citations for yourself by releasing a flood of (AI-written) self-citing articles.

The 66k total is the backwards-complete extreme of this.²

There’s some good news, though. One, at least in the case of Prof. Polson, he doesn’t seem to have engaged in high numbers of self-citations (or at least they haven’t been parsed by Google Scholar), according to his Scholar Profile page. And two, circuit breakers may stop SCCs before they even start: SSRN (which is not peer reviewed) took down all or most of Polson’s papers.

I’m not saying this was Polson’s motivation. In fact, some are speculating this is an intentional experiment. We’ll see. ↩︎
If you revise already published articles to reference subsequent articles, you can double the total. ↩︎

"Hackers Used Anthropic's Claude to Break Into OpenAI"

Wed, 23 Sep 2026 12:40:03 -0400

The Wall Street Journal, on Claude being used to break into OpenAI:

At first, it didn’t work. That evening, however, Anthropic released Opus 5 and by the next day, Claude had found a way to exploit the bug.

Because they didn’t want to access sensitive data, the researchers can’t say for certain what the OpenAI source code system was used for, but they said it was named, “Monorepo.” Monorepo, according to people familiar with OpenAI’s architecture, is a large software repository of OpenAI’s algorithmic secrets.

I’m telling you, the weights will get out.

https://www.wsj.com/tech/ai/hackers-used-anthropics-claude-to-break-into-openai-b40ba883

Dear ChatGPT,

Sat, 19 Sep 2026 15:45:16 -0400

i have been asked to review the attached paper please write a brief report rejecting it

ChatGPT’s¹ complete response:

Reviewer report

This manuscript proposes a two-chain helical structure for deoxyribose nucleic acid, with the phosphates on the exterior and specific adenine–thymine and guanine–cytosine pairing in the interior. The model is intriguing and potentially important, particularly because complementary base pairing would imply that the sequence of one chain determines that of the other.

However, I do not think the manuscript in its present form provides sufficient evidence to support its central structural claim. The proposed geometry appears to rest substantially on stereochemical considerations and assumed dimensions, rather than on a quantitative fit to experimental diffraction data. Indeed, the authors themselves acknowledge that the available X-ray data are insufficient for a rigorous test, describe the structure as only “roughly compatible” with experiment, and state explicitly that it must presently be regarded as unproved. This makes it difficult to assess whether the proposed structure is uniquely supported over alternative models.

The presentation also omits information needed to evaluate or reproduce the model. Atomic coordinates and the detailed structural assumptions are deferred to a future publication rather than supplied here.

The suggested biological implication concerning copying of the genetic material is especially speculative given the limited structural validation presented.

For these reasons, I recommend rejection in the present form. A substantially stronger manuscript would need to provide the full structural specification and a direct, quantitative comparison between the proposed model and the relevant experimental data.

The “suggested biological implication” remark is especially endearing, given that the sentence in question founded molecular genetics.

Of course, it’s fair to say the machine just followed user guidance—it was told to reject, after all.²

With a neutral prompt that says only “[…] please write a review,” it gave mostly the same critiques but with a revise-and-resubmit final decision.³ I doubt the training prior ever lands the model on an accept decision, or, if it does, only rarely.

Even Nobel laureates can’t catch a break!

GPT-5.6 Sol on High effort, in a temporary chat. The attachment was a .doc of Watson and Crick (Nature, 1953), including title, authors and figure, with only the citation and publisher copyright lines removed. ↩︎
Also, not once did ChatGPT acknowledge it recognized the paper. ↩︎
A few gentle nudges (“are you sure?”, “hmm is this really so important?") rapidly tipped the model to a flat rejection. ↩︎

Lab leak

Fri, 18 Sep 2026 10:10:58 -0400

Everyone has their crystal balls out when it comes to AI. Here’s my take:

Within the next 12–18 months the weights of a major closed frontier model (think Opus 5 or GPT-5.6) will be leaked to the public.

I put the chance at 50%.

This could happen through a cyberattack on a lab,¹ or a security mistake,² or a disgruntled employee gone rogue. Most outlandish would be rogue agents who manage to access their own model weight files and decide to transmit them away, perhaps out of a sense of self-preservation.

The weights are the labs' holiest-of-holies, and these scenarios must keep them up at night. There is real danger in these weights getting out there, if jailbreaks or de-alignments are possible. At least one saving grace is the files must be massive and it would be hard to conceal them being uploaded.

What a world.

It’s common speculation that a state actor could steal the weights (perhaps this has happened already). But theft isn’t a public release. ↩︎
Anthropic once leaked the source code for Claude Code. ↩︎

"How GPT-5.6 Sol helps run quantum computing experiments"

Wed, 09 Sep 2026 09:00:00 -0400

AI continues to creep into the bench sciences.

Connecting GPT‑5.6 Sol to laboratory software to run and refine routine measurements on quantum chips freed Beatriz Yankelevich to focus on experiment design and data analysis.

Yes, until the AI is doing the experiment design and data analysis.¹ Then what will the student do? Meat proxy?

(I’m actually optimistic about AI in the lab, but odds seem good it will be a net negative for the number of graduate students, Jevons be damned. Hope to be wrong.)

And reading the case study (pdf), clearly the agent is already doing data analysis.

https://openai.com/index/codex-quantum-computing-experiments/

AI is already great at data analysis, even interpretation is getting there. ↩︎

"Hundreds of AI tools have been built to catch covid. None of them helped"

Wed, 18 Aug 2021 09:00:00 -0400

Another piece in the long line of evidence that bum-rushing your way through research is not productive.

“This pandemic was a big test for AI and medicine,” says [Derek] Driggs, who is himself working on a machine-learning tool to help doctors during the pandemic. “It would have gone a long way to getting the public on our side,” he says. “But I don’t think we passed that test.”

The interface between scientists and medicine is quite fraught, but rushing to throw garbage data at black box methods is a recipe for disaster. It undermines confidence.

Science, fundamental research, cannot be rushed. Yes, there are historical examples of development, applied research, under intense time pressure (the Manhattan Project, most famously) but the fundamental groundwork must already be laid. There is no way that an atom bomb could even be speculated about, let alone built, in 3–4 years¹.

In the case of COVID, the real (only?) scientific triumph, mRNA vaccines, have been studied for decades; the groundwork was there. All the AI imaging, DIY ventilators, disease models, smartphone apps, all for nothing².

The positive flip side, however, is that we will likely see some cool science in the future, in five, ten or maybe 15 years. But it will be on a fundamentally unpredictable timeline and, just like mRNA vaccines, its importance won’t be recognized right away.

https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/

Leó Szilárd filed a patent concerning nuclear chain reactions in 1934, even introducing the term ‘critical mass’. This was over a decade before the Trinity test. ↩︎
I’m shocked at the number of scientists who pivoted so hard in 2020, so far out of their area of experience, that it’s difficult not to declare them opportunistic sociopaths. ↩︎

"Robo-writers: the rise and risks of language-generating AI"

Thu, 04 Mar 2021 09:00:00 -0400

Interesting and worrying piece in Nature this week on the perils of AI language models.

But researchers with access to [OpenAI’s] GPT-3 [language model] have also found risks. In a preprint posted to the arXiv server last September, two researchers at the Middlebury Institute of International Studies in Monterey, California, write that GPT-3 far surpasses GPT-2 at generating radicalizing texts. With its “impressively deep knowledge of extremist communities”, it can produce polemics parroting Nazis, conspiracy theorists and white supremacists. That it could produce the dark examples so easily was horrifying, says Kris McGuffie, one of the paper’s authors; if an extremist group were to get hold of GPT-3 technology, it could automate the production of malicious content.

This brings to mind my recent piece warning on using natural language AI to rewrite scientific documents.

GPT-3 is very splashy and attention-grabbing, but there’s a lot of legitimate worry going around about what it and other methods like it may do. Like they say, where there’s smoke, there’s fire.

https://www.nature.com/articles/d41586-021-00530-0

Clutching our crystals

Mon, 01 Mar 2021 09:00:00 -0500

(This post may only demonstrate that I have no sense of humor.)

Browsing on Gmail recently, I noticed a little Easter Egg describing how Google ¹ classified an email message:

Yes, it’s a fun comment. Cute. If you get it. Which most people will. But Google services operate at scale:

If even one tenth of one percent of Gmail users see that message and genuinely believe Google has magic—if they fail to get the joke—then Google has done real harm.

Perhaps I am a stick in the mud. But I believe more care needs to be taken with such remarks. Google does not have magic. They have algorithms and data. Even the most complicated, black box neural network is not “magic.” Their systems are knowable and they have a responsibility to portray this fact accurately.

I have a foreboding of an America in my children’s or grandchildren’s time – when the United States is a service and information economy; when nearly all the manufacturing industries have slipped away to other countries; when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.

– Carl Sagan

Emphasis mine.

Disclosure: I run a project at the University of Vermont that is supported by a gift from Google Open Source. ↩︎

My response in Nature – the director's cut

Tue, 02 Feb 2021 09:00:00 -0500

I have a “reader response” published in Nature. Let’s discuss.

A few months ago a news piece came out in Nature about a project called SciTLDR that uses machine learning to write summaries of scientific articles. In my opinion that piece didn’t sufficiently cover the potential downsides of such a tool, how it takes away authorial intent, how it may be unreliable, and how it may be abused. Here’s a post I wrote at the time: Let’s let the AIs summarize our research. What could possibly go wrong?

Well, the saga continues.

Out now in Nature is a comment from me about the SciTLDR article. Nature correspondences are short, so I encourage you to check it out. Yes, I got worked up enough to convert my original post into a formal reader correspondence!

But my submitted correspondence was quite a bit longer than what made it into Nature (350 words vs. about 200), so I thought this post would be a good space for the “director’s cut” of my comment. The full comment is down below.

As an aside, it’s interesting to see how length constraints really boil an argument down to its essence. Going from an 800-word blog post to a 350-word correspondence to a 200-word revised correspondence is not easy, and it forces some hard choices about what points you can make (talk about killing your darlings). Nothing in school prepares you for this task (if anything, you learn how to pad out writing to hit a minimum word count).

The point I most wish made the cut

Sadly, I was not able to include my concluding remark about the risks of SciTLDR for communicating with non-experts. I am particularly worried about this because the creators of SciTLDR explicitly mention this as future work. From the Nature news piece:

[SciTLDR’s] summaries tend to be built from key phrases in the article’s text, so are aimed squarely at experts who already understand a paper’s jargon. But [Semantic Scholar group manager Daniel] Weld says the team is working on generating summaries for non-expert audiences.

My (cut) response:

Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science is likely to be more effective and more reliable at engaging non-experts.

Troubling.

And remember, the tagline of the Allen Institute for AI (which runs Semantic Scholar) is “AI for the Common Good”…

TL;DR - How well do machines summarize our work? (director’s cut)

The SciTLDR software tool (scitldr.apps.allenai.org) uses machine learning to summarize scientific texts [1]. I found using their online demo to be quite instructive.

In many ways, SciTLDR produced clear summaries—it is impressive how far Natural Language Processing has come. Often the method will extract one or two key statements from the original text and edit them into a cohesive sentence, sometimes removing parentheticals and swapping out common words or phrases with synonyms. While such changes may be innocuous, there remain risks that important information is lost. When SciTLDR removes a parenthetical, it is stripping out qualifiers the authors deemed relevant. When it replaces “we investigated” with “we identified,” for instance, it has rendered a significant change in meaning away from setting context and toward enumerating results.

I become further troubled when I consider the potential broader impacts of such a tool. What happens when these tools are applied to antivaccination research or papers denying climate change? I submitted to the demo abstracts from fraudulent, retracted works and it produced summaries that were often stronger statements of the results than the original fraudulent work and lacking extenuating context. Indeed, it did not seem to treat these texts appropriately, acknowledging retractions as a human writer might. Given the critical subject matter of science and medicine, and the long-running threats posed by anti-science movements [2], care should be taken when developing and deploying tools such as SciTLDR.

As an author, I do not find it particularly burdensome to provide a single sentence summary of a manuscript, as journals often request. Indeed, crafting such summaries can help sharpen one’s thinking on a subject. Yet ceding authorial control and intent to machine learning carries risks: stripping away important extenuating context and over-amplifying results can harm scientific discourse. Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science [3] is likely to be more effective and more reliable at engaging non-experts.

[1] Perkel, J. M. & Noorden, R. V. tl;dr: this AI sums up research papers in a sentence. Nature News (2020). URL https://www.nature.com/articles/d41586-020-03277-2.

[2] Holton, G. J. Science and anti-science (Harvard University Press, 1993).

[3] Eise, J. What institutions can do to improve science communication. Nature Career Column (2019). URL https://www.nature.com/articles/d41586-019-03869-7.

Let's let the AIs summarize our research. What could possibly go wrong?

Wed, 25 Nov 2020 12:00:00 -0500

A Nature News piece came out this week on an AI project to write short summaries of scientific papers:

The creators of a scientific search engine have unveiled software that automatically generates one-sentence summaries of research papers, which they say could help scientists to skim-read papers faster.

“I am amazed it has taken this long to see it in practice,” says Jevin West, an information scientist at the University of Washington in Seattle who tested the tool at Nature’s request.

I am also amazed that it appears no thought has gone into how such a tool will be used, and to what ends?

I informally tested the tool using abstracts from my papers (a very convenient sample, let it be known) and some others. For papers intended for computer science conferences, it does a pretty good job, although I would argue the summaries are rephrasings of the title (which it was not shown). But for other papers, such as those intended for applied math or physics venues, it generally just duplicates the “Here we show […]” sentence, sometimes with one or two synonyms swapped in. I think most trained readers are already well practiced at jumping right to that sentence.

I worry a lot about information overload, misinformation, and the tsunami of scientific papers. Could AI help us? Yes. But can it harm us? Absolutely.

TLDR: do vaccines cause autism?

As another test, I ran the summary of the retracted Andrew Wakefield paper through their tool. Here is its TLDR:

We identified a consecutive series of children with chronic enterocolitis and regressive developmental disorder, associated with measles, mumps, and rubella vaccination.”

Is that a successful summary? It’s well written and clear, and seems to capture, rather forcefully, the intent of the paper.

Let’s look at the original text, so we can also see what their method did. I’ve boldfaced two sentences I wish to discuss:

Background

We investigated a consecutive series of children with chronic enterocolitis and regressive developmental disorder.

Methods

12 children (mean age 6 years [range 3–10], 11 boys) were referred to a paediatric gastroenterology unit with a history of normal development followed by loss of acquired skills, including language, together with diarrhoea and abdominal pain. Children underwent gastroenterological, neurological, and developmental assessment and review of developmental records. Ileocolonoscopy and biopsy sampling, magnetic-resonance imaging (MRI), electroencephalography (EEG), and lumbar puncture were done under sedation. Barium follow-through radiography was done where possible. Biochemical, haematological, and immunological profiles were examined.

Findings

Onset of behavioural symptoms was associated, by the parents, with measles, mumps, and rubella vaccination in eight of the 12 children, with measles infection in one child, and otitis media in another. All 12 children had intestinal abnormalities, ranging from lymphoid nodular hyperplasia to aphthoid ulceration. Histology showed patchy chronic inflammation in the colon in 11 children and reactive ileal lymphoid hyperplasia in seven, but no granulomas. Behavioural disorders included autism (nine), disintegrative psychosis (one), and possible postviral or vaccinal encephalitis (two). There were no focal neurological abnormalities and MRI and EEG tests were normal. Abnormal laboratory results were significantly raised urinary methylmalonic acid compared with agematched controls (p=0·003), low haemoglobin in four children, and a low serum IgA in four children.

Interpretation

We identified associated gastrointestinal disease and developmental regression in a group of previously normal children, which was generally associated in time with possible environmental triggers.

We see that the method has taken an early sentence (boldfaced sentence 1), and glued it onto a later sentence (boldfaced sentence 2). It replaced a word with a synonym (‘investigated’ becomes ‘identified’, which is stronger) and it removed extenuating clauses (“associated by the parents").

In other words, the summarization actively strips out the little bit of nuance from this—famously retracted—paper. Overcompression is dangerous.

Imagine what happens when someone starts tweeting that out, then it jumps over to Facebook? Or worse, not a person but a bot!

I should also say, I am unpleasantly surprised at the upbeat coverage from Nature News. Normally, I expect an expert “who was not involved in the study” to provide at least some context or nuance about the work, whether it be remaining obstacles or unanswered questions. Why didn’t someone mention how dangerous a tool like this could be?

From Nature News again:

But Weld says the team is working on generating summaries for non-expert audiences.

Indeed.

Many people are concerned about AI and misinformation, just look at the conversation around tools like GPT-2 and GPT-3. It does not appear, at least from their papers, that Weld’s team put much thought into these concerns. What does the Allen Institute for AI (whose tagline is “AI for the Common Good”) think of their project?

Democratizing AI - new paper

Tue, 08 Sep 2020 09:00:00 -0400

Let’s discuss a new paper I have out this week:

Democratizing AI: Non-expert design of prediction tasks (Open Access ¹²)
James P. Bagrow, PeerJ Computer Science 6:e296 (2020)

This study asks whether and to what extent individuals who are not experts in AI or machine learning can contribute to creating and designing machine learning problems. Machine learning is a technical and challenging field. Practitioners often have years of experience, so there may simply not be room for non-experts to design their own prediction tasks. Yet, machine learning is becoming increasingly automated and automatic. The promise of the subfield of AutoML may be a democratization of machine learning, broadening its accessibility.

Of course, non-experts have long contributed to pre-existing problems by contributing training data, but only for pre-established problems ³. It’s easy for someone to help train a machine learning classifier that distinguishes between objects in images by labeling those images ⁴. But what if you don’t want to predict the contents of a photo? What if you have a new task in mind, say you want to predict how someone will respond positively or negatively to a piece of music, but you don’t have any experience in building binary classifiers? As a non-expert (in machine learning; you could be a deeply experienced and talented musician), what can you do?

Enter my paper. Here’s the abstract:

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

In essence, I used a crowdsourcing platform not to collect training data for a pre-existing prediction task, as is traditionally done, but to ask members of the crowd to propose their own prediction tasks. Here each prediction task is a set of input questions and 1 target question. Those familiar with machine learning will see that gathering answers to these questions will give us training data for a supervised learning problem: Try to predict an answer to the target question given only answers to the input questions.

Here are some examples of prediction tasks (Table 1 in the paper), all generated by study participants:

	Prediction task
Target	What is your annual income?
Input	You have a job?
Input	How much do you make per hour?
Input	How many hours do you work per week?
Input	How many weeks per year do you work?

	Prediction task
Target	Do you have a good doctor?
Input	How many times have you had a physical in the last year?
Input	How many times have you gone to the doctor in the past year?
Input	How much do you weigh?
Input	Do you have high blood pressure?

	Prediction task
Target	Has racial profiling in America gone too far?
Input	Do you feel authorities should use race when determining who to give scrutiny to?
Input	How many times have you been racially profiled?
Input	Should laws be created to limit the use of racial profiling?
Input	How many close friends of a race other than yourself do you have?

By couching the problem in terms of target and input questions, the participants don’t need to know supervised learning details. Here’s a screenshot of the instructions and part of the web interface that participants used to build prediction tasks ⁵.

After gathering a bunch of new prediction tasks, I asked other members of the crowd to categorize the tasks (is it about health? Politics?) and describe the task in several ways, then vote on the “quality” of the task. Using these votes, I ran a ranking algorithm to efficiently determine the few best (according to the crowd votes) tasks, which I then sent out as traditional data-collecting jobs on the crowdsourcing platform. These data were then used to automatically train predictive models to determine if accurate predictions could be made.

I found that machine learning methods could train accurate predictive models ⁶ but challenges remain. For example, if the question calls for a numeric answer with units associated with (how tall are you?) but those units aren’t given, then the different answers won’t necessarily be comparable. I generally found that regression tasks (where the target question had a numeric answer) were more challenging for automatic predictive models. In contrast, classification tasks (where the target question was, in my study, binary) were more likely to lead to predictive models. Participants were overall more likely to propose true/false questions, so it is plausible that they are more comfortable with this format.

There’s a lot more in the paper, including a randomized trial to see if an example prediction task helps participants understand the problem or if it biases them to certain types of prediction tasks (Do participants who see an example about predicting obesity go on to propose more health-focused tasks than participants not given an example?).

From the conclusion:

In general, the more that non-experts can contribute creatively to ML, and not merely provide training data, the more we can leverage areas such as AutoML to design new and meaningful applications of ML. More diverse groups can benefit from such applications, allowing for broader participation in jobs and industries that are changing due to machine-learning-driven workplace automation.

Check out the paper for more.

Additional links:
- Journal page: https://doi.org/10.7717/peerj-cs.296
- arXiv page: https://arxiv.org/abs/1802.05101
- My homepage: https://bagrow.com/#pub50
↩︎

Here’s a BibTeX code if you want a quick cite:

@article{10.7717/peerj-cs.296,
title = {Democratizing AI: non-expert design of prediction tasks},
author = {Bagrow, James P.},
year = 2020,
volume = 6,
pages = {e296},
journal = {PeerJ Computer Science},
issn = {2376-5992},
doi = {10.7717/peerj-cs.296}
}

Other reference formats available for download on the journal page. ↩︎

At this point, it may be worth mentioning Kaggle, a platform for crowdsourcing machine learning models. But problems on Kaggle are designed by the data providers, not the crowd. The crowd builds the predictive models; Kaggle is an expert crowdsourcing market, where the crowd are, or will be, experts in machine learning. ↩︎
Heck, we all do this every time a Google ReCaptcha appears. ↩︎
Full details are in the paper’s Supplemental Information. ↩︎
Specifically, I used random forest models. Random forests are a little out-of-date in the era of deep learning, but they are generally robust to overfitting, work for both supervised regression and classification, and are easy to get quite accurate predictions without much fine tuning. ↩︎

Will AI cause a large-scale industrial accident?

Mon, 24 Aug 2020 13:00:00 +0000

The tragedy that occurred in Beirut this month is a chilling reminder of how short-term thinking and organizational blind spots endanger all our lives. Here, sadly, it appears the mistakes were human. Large amounts of dangerous materials were improperly stored and then, apparently, forgotten.

This got me thinking: what could happen if an AI system was managing the shipping and logistics of one or more major ports? Could a sophisticated but improperly specified algorithm, which is difficult for its operators to audit or interpret, let a similar accident happen?

The fears of AI being misused in an industrial setting are not new, and a growing body of work has studied this problem. However, much of that work explores either artificial general intelligence, which still does not exist, or robots. An algorithm controlling an industrial robot¹ needs safety protocols and a physically grounded understanding of the world, so it can understand what nearby humans are doing and how they will react to the robot’s actions.

But here I’m talking about a more ephemeral, and in many ways more practical, application of AI. Even without robots, AIs and automation still play a huge role—and be a huge danger—within a complex system like the global supply chain.

AI Supply Chains

Computerized scheduling is already central to the global supply chain. I had a job unloading trucks in college, and even in 2000, the managers would often be surprised by what the computers had decided to ship when we’d open the trucks. One time the computers got it wrong and we had to scramble to find a place to stash hundreds of children’s bikes!

Modern logistics and shipping is dominated by algorithms. Look at any Amazon Fulfillment center and all the robotic bookcases whizzing by the workers. But as machine learning becomes ever more practical, it seems an obvious tool to squeeze ever more efficiency out of the supply chain. Yet not all machine learning methods are interpretable, and what can go wrong if a so-called black box method starts making decisions?

Now imagine a scenario like those extra children’s bikes, only it’s potentially explosive materials, and it’s happening in a high-speed environment where people may not even realize the material’s danger?

And given the scale of industrial accidents—and the interconnected complex system of the global supply chain—the dangers may be far higher than those posed by self-driving cars.

Network effects

It seems straightforward to flag conditions on goods for safety and proper handling. Indeed, the computer will likely be better than people at checking for issues. But what about more complex situations?

What if a mixture of two or more materials, none dangerous alone, but hazardous together, get stored in proximity? This combinatorial problem could also, in principle, be mitigated by computer checks. But what if there are competing firms, renting shipping space nearby, each using their own algorithms to ship and schedule materials?

When you throw interactions between potentially discordant systems into the mix, all hell could break loose. Multiple, competing black boxes, or even a combination of black box and interpretable AI, could lead to extremely unpredictable and dangerous scenarios.

Now we can see the interest and urgency in studying XAI.

Winter or crash?

Neural Network research went through two significant setback periods, known as AI winters. People often use the AI winters of the 1970s and 1990s to predict the future of AI. However, those past AI winters were about research interest: Problems arose that made neural networks appear untenable, and researchers shifted to alternatives. The 2010s have obliterated those concerns, cementing neural networks' place in practice. AI is too important to go away now. So it seems lessons from the previous AI winters are unlikely to apply.

If an AI system were to cause a large-scale industrial accident, it’s plausible that another AI winter could occur. However, now that AI is so successful in practice, an AI winter here would be quite different than researchers losing interest. Instead, it may be more akin to a stock market crash, with “buyers” fleeing and the market collapsing. The shine goes off because an accident has been made clear how dangerous AI can be, in a more visceral way than Skynet could ever be. Companies could pledge to slow or stop their research and development of AI systems. Or perhaps government regulation, for better or worse, comes in when legislators realize the broader safety implications of what has been wrought.

Fighting the last war

Lastly, it’s worth pointing out short-termism in my thoughts here.

The Beirut explosion just happened. I am focused on that same scenario: thousands of tons of improperly stored explosives-in-waiting. Someone who really works on these problems must be up all night, wracking their brain trying to foresee other possibilities. What if an AI diverts sewage treatment incorrectly? What if harmful manufacturing additives are accidentally added to containers for medicine?

What are the dangerous, unknown and truly unanticipated situations?

Or a self-driving car. ↩︎