Posts | Jim Bagrow

A big-data view on sleep and travel - new paper

Thu, 24 Feb 2022 11:00:00 -0500

Very excited to announce we have a new paper out:

Sleep during travel balances individual sleep needs. Sigga Svala Jónasdóttir, James Bagrow and Sune Lehmann, Nature Human Behaviour (2022)

Essentially, we have access to a pretty incredible deidentified dataset covering wearables users who track their sleep. These data let us tackle questions at a scale far beyond the size of a typical sleep study. And along with sleep indicators, like when sleep begins and ends, we can tell whether an individual is at home or traveling, so we can see, observationally, how travel affects sleep.

With that in mind, we went into the data with a rough expectation of quantifying how travel harms sleep, leading to less sleep for example. Certainly most travelers have all experienced (maybe not recently) serious jet lag. But in fact we found something more nuanced. Here’s the abstract:

Travel is expected to have a deleterious effect on sleep, but an epidemiological-scale understanding of sleep changes associated with travel has been limited by a lack of large-scale data. Our global dataset of ~20,000 individuals and 3.17 million nights (~218,000 travel nights), while focused mainly on short, non-time-zone-crossing trips, reveals that travel has a balancing effect on sleep. Underslept individuals typically sleep more during travel than when at home, while individuals who average more than 7.5 hours of sleep at home typically sleep less when travelling. The difference in travel sleep quantity depends linearly on home sleep quantity and decreases as median sleep duration increases. On average, travel wake time advances to later hours on weekdays but earlier hours on weekends. Our study emphasizes the potential for consumer-grade wearable device data to explore how environment and behaviour affect sleep.

Besides the paper itself (and apologies for hitting any paywalls), we also put together a fun little “Behind the paper” blog post.

Congrats to Sigga Svala Jónasdóttir, our fearless leader on this journey, and my long-time collaborator Sune Lehmann, who were both fantastic to work with.

Please check out the paper, the data and the code for more.

"Upcoming Python Features Brought to You by Python Enhancement Proposals"

Wed, 16 Feb 2022 13:00:00 -0500

A fun overview of some interesting new features coming to Python.

Another syntax change proposal is PEP 654, which proposes except* as a new syntax for raising groups of exceptions. The rationale for this one is that Python interpreter can only propagate one exception at the time, but sometimes multiple unrelated exceptions need to be propagated as the stack unwinds.

Yes, please!

Next one is PEP 678 which suggests that __note__ attribute should be added to BaseException class. This attribute would be used to hold additional debugging information which could be displayed as part of traceback.

Yes, yes!

Some great stuff, but I also worry about feature creep. The proposal for changing the syntax for default function arguments is interesting, but fairly opaque to a non-expert:

def process_data(file=>obj_store):
	...

That looks pretty cryptic to me.

One of the strengths of Python is the simplicity of the syntax. The best code can read almost like pseudocode, letting non-Python programmers get the basic idea. Python is then easy to read, fun to write, and great for teaching. But these syntactic sugar enhancements, great as they tend to be, do risk taking away some of those less-is-more features.

This change looks reasonable and useful to me, but I think we should be cautious of adding too many new syntax notations/changes. It’s questionable whether small improvement like this one warrants yet another assignment operator.

Exactly.

https://martinheinz.dev/blog/67

"NERCCS 2022: Fifth Northeast Regional Conference on Complex Systems"

Wed, 19 Jan 2022 09:00:00 -0500

Interested folks, be sure to check out this year’s Northeast Regional Conference on Complex Systems:

NERCCS 2022: The Fifth Northeast Regional Conference on Complex Systems will follow the success of NERCCS 2018, NERCCS 2019, NERCCS 2020, and NERCCS 2021 to promote the emerging venue of interdisciplinary scholarly exchange for complex systems researchers in the Northeast U.S. region to share their research outcomes through presentations and post-conference online publications, network with their peers in the region, and promote inter-campus collaboration and the growth of the research community. NERCCS is particularly focus on facilitating the professional growth of early career faculty, postdocs, and students in the region who will likely play a leading role in the field of complex systems science and engineering in the coming years.

Submission deadline: January 31, 2022

Here’s the call for papers (1-page extended abstract or 15-page full paper). Right now, the plan is for a hybrid meeting with both in-person and online options.

I’m helping as the program chair along with Laurent Hébert-Dufresne. It should be fun!

https://nerccs2022.github.io

"Why Is It So Hard to Be Rational?"

Mon, 03 Jan 2022 09:00:00 -0500

Joshua Rothman writing in the New Yorker ¹ about rational thinking:

In Silicon Valley, people wear T-shirts that say “Update Your Priors,” but talking like a rationalist doesn’t make you one. A person can drone on about base rates with which he’s only loosely familiar, or say that he’s revising his priors when, in fact, he has only ordinary, settled opinions. Google makes it easy to project faux omniscience.

The points on meta-thinking ² (and the anecdote about Petrov saving the world) are spot on.

https://www.newyorker.com/magazine/2021/08/23/why-is-it-so-hard-to-be-rational

*cough cough* ↩︎
For no particularly strong reason, I prefer “meta-thinking” over the fancier, more accepted metacognition 🤷. ↩︎

"This digital-hygiene routine will protect your scholarship"

Thu, 30 Dec 2021 09:00:00 -0500

Guillaume Cabanac writes in a Nature World View article on performing “scholarly check-ups” and why you need them:

Scholars’ names, work or both are used by crooked individuals or institutions to deceive others. The scope of the problem dismays me. I shiver when imagining my university’s research-integrity officer coming to me with a pile of buggy papers — that I’ve never seen before — bearing my name.

I’ve had articles come out that misquoted or misrepresented one of my papers, but generally in minor ways. Nothing like someone putting my name on their article. (Guess I’m not famous enough!)

The article has good advice but doesn’t give a magic bullet, as they note. I wonder if automation can help with this problem (they mention signing up for author alerts), at least if you have a unique or fairly distinct name. But could automation end up making it worse?

https://www.nature.com/articles/d41586-021-02901-z

"Redo of a Famous Experiment on the Origins of Life Reveals Critical Detail Missed for Decades"

Wed, 29 Dec 2021 11:00:00 -0500

Some interesting followups to one of the most celebrated experiments of all time.

But recently a team of researchers realized that—much like that first primordial soup sitting in a bowl of Earth—the experiment’s container played an underappreciated role—that perhaps it was also critical to the creation of organic building blocks inside their laboratory life soup.

I often wonder how much of the replication crisis gripping “softer” fields ¹ will spill over into the hard sciences, physics, chemistry, and such. Atoms aren’t as squishy to understand as people, but there’s just so many details that go into experiments and other studies, especially nowadays, and publications are too terse to provide complete coverage.

But this new research doesn’t really fit the replication crisis mold. That a glass flask was used was not omitted from the record, and no nefarious data fishing or statistical tricks were used. This case is really more about failing to consider confounding factors. That reasoning process, qualified skepticism—Could it actually be caused by this? Did you consider varying that?—is central to good science, for individual researchers and entire fields. And its absence isn’t fixed by expecting larger sample sizes or smaller p-values.

https://www.scientificamerican.com/podcast/episode/redo-of-a-famous-experiment-on-the-origins-of-life-reveals-critical-detail-missed-for-decades/

Which, let’s be honest, are not actually that soft. ↩︎

"E.O. Wilson, naturalist dubbed a modern-day Darwin, dies at 92"

Mon, 27 Dec 2021 09:30:00 -0500

Sad news this morning, famed scientist EO Wilson has passed.

It is hard to overstate his influence. He coined the terms “biodiversity” and “evolutionary biology” ¹. I recently read his memoir, “Letters to a young scientist.” Inspiring and highly recommended, even for us non-youngsters ².

Update: Here’s the NY Times obituary.

https://www.reuters.com/lifestyle/science/obituary-modern-day-darwin-eo-wilson-dies-92-2021-12-27/

And then there is “sociobiology.” He’s been dogged for decades with controversy over that. ↩︎
Keep in mind survivor bias when considering his advice though. ↩︎

"Hundreds of AI tools have been built to catch covid. None of them helped"

Wed, 18 Aug 2021 09:00:00 -0400

Another piece in the long line of evidence that bum-rushing your way through research is not productive.

“This pandemic was a big test for AI and medicine,” says [Derek] Driggs, who is himself working on a machine-learning tool to help doctors during the pandemic. “It would have gone a long way to getting the public on our side,” he says. “But I don’t think we passed that test.”

The interface between scientists and medicine is quite fraught, but rushing to throw garbage data at black box methods is a recipe for disaster. It undermines confidence.

Science, fundamental research, cannot be rushed. Yes, there are historical examples of development, applied research, under intense time pressure (the Manhattan Project, most famously) but the fundamental groundwork must already be laid. There is no way that an atom bomb could even be speculated about, let alone built, in 3–4 years¹.

In the case of COVID, the real (only?) scientific triumph, mRNA vaccines, have been studied for decades; the groundwork was there. All the AI imaging, DIY ventilators, disease models, smartphone apps, all for nothing².

The positive flip side, however, is that we will likely see some cool science in the future, in five, ten or maybe 15 years. But it will be on a fundamentally unpredictable timeline and, just like mRNA vaccines, its importance won’t be recognized right away.

https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/

Leó Szilárd filed a patent concerning nuclear chain reactions in 1934, even introducing the term ‘critical mass’. This was over a decade before the Trinity test. ↩︎
I’m shocked at the number of scientists who pivoted so hard in 2020, so far out of their area of experience, that it’s difficult not to declare them opportunistic sociopaths. ↩︎

"640 Pages in 15 Months"

Tue, 17 Aug 2021 09:00:00 -0400

Hilariously overwrought adventures typesetting a textbook. How about some LaTeX?

I kid. The system seems to work, looks fast, and the results are visually fantastic, so more power to them.

The stuff on metrics is gold:

Oh, and since I ended up writing a 200k+ word book, it’s going to have a high page count. That means a thick book. Thick books need wider inner margins so the text doesn’t disappear into the spine.

All of this points towards a pretty wide page. Most CS textbooks—at least the ones on my bookshelf—are 7.5 inches wide. I tried hard to come up with a design that fit the code, asides, and healthy margins in that width while still giving a text size that didn’t require a magnifying glass. Eventually, I conceded defeat.

Once I tried designing a set of metrics for an 8 inch wide page, everything fell into place. I could have enough breathing room around the text to make it enjoyable to read, a decent length for the code snippets, and plenty of room for the asides. (Using a narrower font for the asides helped too).

I would have leaned on LaTeX and pandoc, even if builds are slower and the final result looks slightly worse than hand-crafting the final product in Adobe InDesign. (Is it worth it though?) But I noped out of there as soon as I saw that javascript and InDesign were entering the fray—it makes me appreciate Knuth even more.

(Also, I love the finale where the whole book is converted to a single PNG and then pixel-diffed. Talk about track changes!)

https://journal.stuffwithstuff.com/2021/07/29/640-pages-in-15-months/

"New “Glowworm attack” recovers audio from devices’ power LEDs"

Wed, 11 Aug 2021 12:00:00 -0400

Clever study showing a new way to spy on electronic audio signals:

A new paper released today outlines a novel passive form of the TEMPEST attack called Glowworm, which converts minute fluctuations in the intensity of power LEDs on speakers and USB hubs back into the audio signals that caused those fluctuations.

(Emphasis mine.) The attack is limited by line-of-sight of course, but the study showed that intelligible sound could be recovered at up to 35 meters.

Reminds me of the study showing you can identify what a TV is playing by measuring current fluctuations in the TV’s power draw. Also similar to the idea of laser microphones that could listen to conversations in a room with a window by measuring small deflections in the window due to sound waves.

https://arstechnica.com/gadgets/2021/08/new-glowworm-attack-recovers-audio-from-devices-power-leds/

"‘Tortured phrases’ give away fabricated research papers"

Fri, 06 Aug 2021 09:00:00 -0400

Simultaneously disturbing and hilarious examples of automated synonyms to get around plagiarism detection:

Further investigation revealed that these strange terms — which they dub “tortured phrases” — are probably the result of automated translation or software that attempts to disguise plagiarism. And they seem to be rife in computer-science papers.

Some examples:

Scientific term Tortured phrase

Big data Colossal information

Artificial intelligence Counterfeit consciousness

Deep neural network Profound neural organization

… …

Scientific term	Tortured phrase
Big data	Colossal information
Artificial intelligence	Counterfeit consciousness
Deep neural network	Profound neural organization
…	…

I wonder what it is for “machine learning”? Contraption knowing? Gearbox culture?

Moral of the story: don’t fake your papers.

https://www.nature.com/articles/d41586-021-02134-0

"Old-school computing: when your lab PC is ancient"

Wed, 02 Jun 2021 12:00:00 -0400

This Nature Technology Features has a lot of fun stories of ancient computers at the heart of research labs.

“The Hollywood depiction of cutting-edge science is always super fancy,” says James Mason, now a solar physicist at the University of Colorado Boulder, whose team built one of the Solar Dynamics Observatory instruments.

But the first time Mason visited the White Sands Test Facility, a laboratory in New Mexico that launches rockets to calibrate his team’s instrument, he was shocked to find data streaming to a boxy, custom-built 1980s-era desktop computer, spitting out lines of pixelated, greenish-yellow text. “It’s so old I can’t find any information about it on the Internet,” Mason says.

Now I want to make a film with a super-exaggerated version of this—Matthew McConaughey has to fly into a black hole in a spacecraft controlled by ENIAC.

Sometimes it’s the software upgrade itself that’s too expensive. In 2008, [Kristin] Low’s undergraduate lab at McGill University in Montreal used an Intel 386-based system running Microsoft’s then-16-year-old Windows 3.1 software to connect to their liquid chromatography system. Updating the chromatography software to support an upgraded operating system would have cost $10,000, Low says — a prohibitive amount.

Seriously though, I always think about these ongoing costs whenever new hardware (and software) might enter my workflow. There’s always a cost in the long-term.

And yes, open source can prevent some problems, like a vendor charging too much for an upgrade or even going out of business, but open source brings problems of its own. For one, there can be a heavy time commitment to keeping the software up-to-date ¹. Maybe an ambitious grad student can help you, and you’ll avoid either doing the upkeep yourself or hiring it out to a technician, but then that grad student is not devoting their time to what really matters: cutting-edge research. And at the same time, a niche research area may not have enough popularity, or may be too technically challenging, to generate a groundswell of enthusiastic developers.

One way or the other, it’s money or time.

https://www.nature.com/articles/d41586-021-01431-y

I’ve got an interesting corollary to this, but that’s for another post. ↩︎

"Google Promised Its Contact Tracing App Was Completely Private—But It Wasn't"

Wed, 19 May 2021 09:00:00 -0400

Investigations at The Markup have revealed some worrying bugs in Google’s contact tracing implementation.

Researchers say hundreds of preinstalled apps can access a log found on Android devices where sensitive contact tracing information is stored.

Color me unsurprised.

But The Markup has learned that not only does the Android version of the contact tracing tool contain a privacy flaw, but when researchers from the privacy analysis firm AppCensus alerted Google to the problem back in February of this year, Google failed to change it.

All that work to preserve privacy and a log file was left wide open… Move fast and break things, huh?

https://themarkup.org/privacy/2021/04/27/google-promised-its-contact-tracing-app-was-completely-private-but-it-wasnt

"Reactive, reproducible, collaborative: computational notebooks evolve"

Tue, 18 May 2021 09:00:00 -0400

Interesting feature this month in Nature on next steps for computational notebooks.

Reactivity, where the notebook automatically “refreshes” upon changes, might be a good idea for some of the problems users have when trying to track the state of the notebook kernel. But, as the article mentions, it doesn’t scale well to large calculations or data. But in that scenario, maybe a notebook isn’t the right tool for the job.

I have lots of thoughts on notebooks. Sounds like more posts coming!

https://doi.org/10.1038/d41586-021-01174-w

Introducing PLASM - PLot Analysis Spreads for Meetings

Mon, 10 May 2021 12:00:00 -0400

Open-sourcing a fun little tool that might make your research meetings a bit more productive.

The problem

You’ve put together some nice data, made a histogram, and are showing it to your collaborators. Immediately, they want to see a log-scaled $y$-axis. But you didn’t prepare it, your plots aren’t interactive and the data are slow to reload, and time is running out in the meeting.

Now you have to promise to send the revised plot after the meeting. The research feedback loop has slowed.

The solution - PLASM

PLASM consists of two Python functions that make a spread of plots:

SPASM - ScatterPlot Analysis Spread for Meetings
CHASM - Cdf and Histogram Analysis Spread for Meetings

With a quick function call ¹ you get all the combinations of log and linear scales that you could want, as well as some basic summary statistics. Now when you’re asked for a different scaling, just pull up the saved spread.

Examples

SPASM

Suppose we conducted two experiments, measuring for each an $x$ and $y$ variable. Let’s set of a plot spread comparing their scatter plots:

spasm(X1,Y1, X2,Y2, names=['Expr-1', 'Expr-2'])

CHASM

And here’s an example of chasm comparing the distributions of three variables:

chasm(D1,D2,D3, names=[f'data{i}' for i in range(3)], show_median=True)

The top two rows are cumulative distributions (ECDFs) and the bottom row is histograms ².

Redundancy. One thing to keep in mind with these spreads is that there’s a lot of redundancy. You’re plotting the same thing over and over, with different combinations of scales. That redundancy is by design. With a spread you can:

see all the different scales, to help guide you to which is most informative (if you don’t have an educated guess in advance),
save the plot figure as a file for reference to show during meetings.

This plot redundancy makes the spread less useful for, say, a final figure in a publication. In that case, you should already have a good handle on the best plot scale to use before you finish the paper figure. PLASM is best as a working tool.

Interested? You’re one pip install plasm away from plot spread goodness!

Be sure to check out the GitHub for more (especially the FAQ).

My design goal for PLASM is as little extra code and changes to how you work as possible. ↩︎
Histograms require choosing bins, and right now CHASM only has a basic auto binning procedure for the linear and logarithmic $x$-axis. More should be done. Of course, the CDF, computed properly, needs no binning 😂. ↩︎

"PowerPC? Rad!"

Fri, 02 Apr 2021 10:30:00 -0400

I remain absolutely floored by just how old the CPUs are in spacecraft.

[T]he Perseverance rover is powered by a PowerPC 750 processor, which was used in Apple’s original 1998 iMac G3—you remember, the iconic, colorful, see-through desktop. If the PowerPC name sounds familiar, it’s probably because those are the RISC CPUs Apple used in its computers before switching to Intel.

The PowerPC 750 was a single-core, 233MHz processor, and compared to the multi-core, 5.0GHz-plus frequencies modern consumer chips can achieve, 233MHz is incredibly slow.

It’s really amazing how much you can do on old hardware. We are really spoiled. I am constantly reminded that the computer I took to college had a clock speed of about half that of my current wristwatch. My watch.¹

Of course, there’s a very good reason for using such an antique:

However, there’s a major difference between the iMac’s CPU and the one inside the Perseverance rover. BAE Systems manufactures the radiation-hardened version of the PowerPC 750, dubbed RAD750, which can withstand 200,000 to 1,000,000 Rads and temperatures between −55 and 125 degrees Celsius (-67 and 257 degrees Fahrenheit).

Oh, and by the way: the RAD750 costs $200,000! Yup. Worth it for the reliability—the whole mission is running $2.7B.

Do check out the source article on NewScientist if you can (warning: paywalled).

https://gizmodo.com/a-1990s-imac-processor-powers-nasa-s-perseverance-rover-1846380844

I understand that’s not exactly a—how shall we say—apples-to-apples comparison. ↩︎

"Mistakes I’ve Made as an Engineering Manager"

Thu, 11 Mar 2021 11:00:00 -0500

Some interesting thoughts about managing. The article discusses how to learn from each of four mistakes:

Mistake 1: Thinking people give feedback the way they want to receive it
Mistake 2: Trying to do everything yourself as a manager is the best way to help
Mistake 3: Communicating something one time is enough
Mistake 4: You have to have everything together all the time

Nothing earth-shattering, and it’s focused on management in tech, but there’s good things to think about not just for those in leadership positions but anyone building collaborations.

I especially like this comment about using multiple communication channels to reinforce your message without coming across as nagging:

Mistake 3: Communicating something one time is enough

No one likes to feel like they’re repeating themselves. It’s annoying to say someone more than once, and it’s annoying to hear something over and again. But if you have a big enough group and there’s enough going on, things are going to slip through the cracks, so repetition becomes an important tool to make things stick. The trick is to say the same things, but in different ways.

Think of all the ways we have to communicate these days: chats, emails, video meetings, texts, document comments, and so much more. And because some people communicate better in one medium than another, using all of the platforms have in various mediums becomes a strategy for repetition without nagging.

(Also: Is it numerology for me to wish there were five mistakes in the list?)

https://css-tricks.com/mistakes-ive-made-as-an-engineering-manager/

"Pandemic Has Created a Generation of Schoolchildren More Interested in STEM Careers Than Ever, Poll Says"

Tue, 09 Mar 2021 09:00:00 -0400

Some good news may be coming out of the pandemic in the long-term. There may be a new generation of schoolchildren interested in pursuing STEM (Science, Technology, Engineering and Math).

From the article:

A spokesperson for Medicspot said: “It’s heartening to see how many youngsters have been inspired by the medical professionals and scientists who have been working on the frontline to treat people suffering from Covid-19 and behind the scenes on the treatments.”

The poll also found 68 percent of respondents think science is a cool subject—and 41 percent are now more interested in learning more about it.

Almost three-quarters are looking forward to going back to school, while 61 percent are going to try and work harder in their science lessons.

Of course, polls can only tell us so much. This poll surveyed 1000 UK schoolchildren age 11–17. Hopefully, the sentiment holds up, and we see a positive trajectory.

It’s now on us already in STEM to make sure there’s a thriving, inclusive environment ready when these new people arrive, both in the classroom and in the job market.

https://www.goodnewsnetwork.org/onepoll-uk-children-want-stem-career-pandemic

"Hearing about the Big Bang for the First Time"

Mon, 08 Mar 2021 09:00:00 -0400

Insightful story in Scientific American about a science writer who goes back to the classroom and rediscovers the excitement of scientific discovery by teaching about it to undergraduates.

Eventually I veered away from celebratory science writing. I decided that I could better serve readers by critiquing and even debunking scientific claims, which are often exaggerated, incoherent or wrong. Science, I persuaded myself, needs tough, informed criticism more than “gee-whiz” journalism, which in unskilled hands resembles mere marketing.

The trouble was, I felt myself becoming jaded, losing the sense of wonder that lured me into science journalism in the first place. We’re all subject to habituation, perhaps for reasons related to evolution. Our brains weren’t designed to keep us in a state of slack-jawed awe before the weirdness of existence; that wouldn’t be very adaptive

Ironically, teaching has helped me overcome my habituation. I started teaching at Stevens Institute of Technology in 2005 because I needed money to supplement my freelance income. At first, I felt awkward in the classroom. I’m a dilettante, I kept thinking, not an expert in anything. I don’t have a doctorate, only a master’s in journalism. (When I confessed my insecurity to a friend, nuclear historian Alex Wellerstein, he replied, Lots of us professors suffer from impostor syndrome, but in your case it might be justified.)

Lots of interesting thoughts.

https://www.scientificamerican.com/article/hearing-about-the-big-bang-for-the-first-time/

"Robo-writers: the rise and risks of language-generating AI"

Thu, 04 Mar 2021 09:00:00 -0400

Interesting and worrying piece in Nature this week on the perils of AI language models.

But researchers with access to [OpenAI’s] GPT-3 [language model] have also found risks. In a preprint posted to the arXiv server last September, two researchers at the Middlebury Institute of International Studies in Monterey, California, write that GPT-3 far surpasses GPT-2 at generating radicalizing texts. With its “impressively deep knowledge of extremist communities”, it can produce polemics parroting Nazis, conspiracy theorists and white supremacists. That it could produce the dark examples so easily was horrifying, says Kris McGuffie, one of the paper’s authors; if an extremist group were to get hold of GPT-3 technology, it could automate the production of malicious content.

This brings to mind my recent piece warning on using natural language AI to rewrite scientific documents.

GPT-3 is very splashy and attention-grabbing, but there’s a lot of legitimate worry going around about what it and other methods like it may do. Like they say, where there’s smoke, there’s fire.

https://www.nature.com/articles/d41586-021-00530-0

Clutching our crystals

Mon, 01 Mar 2021 09:00:00 -0500

(This post may only demonstrate that I have no sense of humor.)

Browsing on Gmail recently, I noticed a little Easter Egg describing how Google ¹ classified an email message:

Yes, it’s a fun comment. Cute. If you get it. Which most people will. But Google services operate at scale:

If even one tenth of one percent of Gmail users see that message and genuinely believe Google has magic—if they fail to get the joke—then Google has done real harm.

Perhaps I am a stick in the mud. But I believe more care needs to be taken with such remarks. Google does not have magic. They have algorithms and data. Even the most complicated, black box neural network is not “magic.” Their systems are knowable and they have a responsibility to portray this fact accurately.

I have a foreboding of an America in my children’s or grandchildren’s time – when the United States is a service and information economy; when nearly all the manufacturing industries have slipped away to other countries; when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.

– Carl Sagan

Emphasis mine.

Disclosure: I run a project at the University of Vermont that is supported by a gift from Google Open Source. ↩︎

"A College Admissions Rat Race"

Fri, 26 Feb 2021 09:00:00 -0500

Troubling piece in the New York Times about the pandemic and college admissions.

The pandemic seems to have exposed and perhaps worsened a recent trend in college admissions: Selective universities have seen extraordinary interest from applicants this year, after waiving test scores. But smaller and less recognizable schools are extending deadlines and expanding outreach to attract students.

Many institutions outside the top tier have struggled for years. The pandemic has made it worse: American colleges and universities have endured losses of more than $120 billion and a few have even shut down permanently. For those that remain, landing fewer students — and losing critical tuition dollars — could mean further distress.

No mention of international applications. Maybe it’s no longer worth coming to the US for school unless you get into an elite university. Could an unwillingness to travel to the US for school account for part of the gap?

I heard someone say that the pandemic is just accelerating existing trends. A company already on an upward trajectory gets a huge boost while another company starting to slow down will crash. It makes sense and it looks like higher education is in the same boat.

We should all be worried, even the privileged few at the elite places. A truly uneducated society will eventually topple even the tallest ivory tower.

https://www.nytimes.com/2021/02/24/us/a-college-admissions-rat-race.html

"The Hard Lessons of Modeling the Coronavirus Pandemic"

Thu, 25 Feb 2021 12:00:00 -0500

Recent Quanta piece on the abysmal track record of disease modeling outside the ivory tower:

But on September 3, just one week into its fall semester, the university faced a bleak reality. Nearly 800 of its students had tested positive for the coronavirus — more than the model had projected by Thanksgiving. Administrators had to issue an immediate campus-wide suspension of nonessential activities.

What had gone wrong? The scientists had seemingly included so much room for error, so many contingencies for how students might behave. “What we didn’t anticipate was that they would break the law,” Goldenfeld said — that some students, even after testing positive and being told to quarantine, would attend parties anyway. This turned out to be critical: Given how COVID-19 spreads, even if only a few students went against the rules, the infection rate could explode.

Wow, it’s almost as if toy models, simulations and observational data are not sufficient to inform policy decisions. Who knew!

I’m just glad that no one is using 2.4GHz electromagnetic packet exchange as a proxy for aerosolized droplet transmission… Oh wait.

https://www.quantamagazine.org/the-hard-lessons-of-modeling-the-coronavirus-pandemic-20210128/

My response in Nature – the director's cut

Tue, 02 Feb 2021 09:00:00 -0500

I have a “reader response” published in Nature. Let’s discuss.

A few months ago a news piece came out in Nature about a project called SciTLDR that uses machine learning to write summaries of scientific articles. In my opinion that piece didn’t sufficiently cover the potential downsides of such a tool, how it takes away authorial intent, how it may be unreliable, and how it may be abused. Here’s a post I wrote at the time: Let’s let the AIs summarize our research. What could possibly go wrong?

Well, the saga continues.

Out now in Nature is a comment from me about the SciTLDR article. Nature correspondences are short, so I encourage you to check it out. Yes, I got worked up enough to convert my original post into a formal reader correspondence!

But my submitted correspondence was quite a bit longer than what made it into Nature (350 words vs. about 200), so I thought this post would be a good space for the “director’s cut” of my comment. The full comment is down below.

As an aside, it’s interesting to see how length constraints really boil an argument down to its essence. Going from an 800-word blog post to a 350-word correspondence to a 200-word revised correspondence is not easy, and it forces some hard choices about what points you can make (talk about killing your darlings). Nothing in school prepares you for this task (if anything, you learn how to pad out writing to hit a minimum word count).

The point I most wish made the cut

Sadly, I was not able to include my concluding remark about the risks of SciTLDR for communicating with non-experts. I am particularly worried about this because the creators of SciTLDR explicitly mention this as future work. From the Nature news piece:

[SciTLDR’s] summaries tend to be built from key phrases in the article’s text, so are aimed squarely at experts who already understand a paper’s jargon. But [Semantic Scholar group manager Daniel] Weld says the team is working on generating summaries for non-expert audiences.

My (cut) response:

Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science is likely to be more effective and more reliable at engaging non-experts.

Troubling.

And remember, the tagline of the Allen Institute for AI (which runs Semantic Scholar) is “AI for the Common Good”…

TL;DR - How well do machines summarize our work? (director’s cut)

The SciTLDR software tool (scitldr.apps.allenai.org) uses machine learning to summarize scientific texts [1]. I found using their online demo to be quite instructive.

In many ways, SciTLDR produced clear summaries—it is impressive how far Natural Language Processing has come. Often the method will extract one or two key statements from the original text and edit them into a cohesive sentence, sometimes removing parentheticals and swapping out common words or phrases with synonyms. While such changes may be innocuous, there remain risks that important information is lost. When SciTLDR removes a parenthetical, it is stripping out qualifiers the authors deemed relevant. When it replaces “we investigated” with “we identified,” for instance, it has rendered a significant change in meaning away from setting context and toward enumerating results.

I become further troubled when I consider the potential broader impacts of such a tool. What happens when these tools are applied to antivaccination research or papers denying climate change? I submitted to the demo abstracts from fraudulent, retracted works and it produced summaries that were often stronger statements of the results than the original fraudulent work and lacking extenuating context. Indeed, it did not seem to treat these texts appropriately, acknowledging retractions as a human writer might. Given the critical subject matter of science and medicine, and the long-running threats posed by anti-science movements [2], care should be taken when developing and deploying tools such as SciTLDR.

As an author, I do not find it particularly burdensome to provide a single sentence summary of a manuscript, as journals often request. Indeed, crafting such summaries can help sharpen one’s thinking on a subject. Yet ceding authorial control and intent to machine learning carries risks: stripping away important extenuating context and over-amplifying results can harm scientific discourse. Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science [3] is likely to be more effective and more reliable at engaging non-experts.

[1] Perkel, J. M. & Noorden, R. V. tl;dr: this AI sums up research papers in a sentence. Nature News (2020). URL https://www.nature.com/articles/d41586-020-03277-2.

[2] Holton, G. J. Science and anti-science (Harvard University Press, 1993).

[3] Eise, J. What institutions can do to improve science communication. Nature Career Column (2019). URL https://www.nature.com/articles/d41586-019-03869-7.

"When could everyone in the U.S. get vaccinated?"

Mon, 01 Feb 2021 09:00:00 -0500

https://graphics.reuters.com/HEALTH-CORONAVIRUS/VACCINE-CALCULATOR/rlgvdgewwpo/

Nice article with handy interactive visualizations. Play around with the vaccination rollout and see the good (or bad) news.

A department, close to home, terminated

Mon, 25 Jan 2021 09:00:00 -0500

Bummer story in Science about the geology department being cut here at University of Vermont:

After a quick apology, the dean pivoted to the news: Given the crunch on university finances amid the COVID-19 pandemic, he’d made a difficult decision to terminate the geology department. If we wanted to keep our jobs, we’d have to find another department willing to take us in. His words hit me like a ton of bricks.

But their predicament did not come entirely out of the blue. Three years ago, the department was faced with falling enrollments and lowered budgets. This comment about the department’s then-response hit really hard (emphasis mine):

We convened faculty meetings to discuss how to move forward. Our curriculum hadn’t changed much in decades. Our students learned how to identify rocks and stare down microscopes, but they weren’t much exposed to many of the more pressing problems in geosciences, such as climate change and groundwater pollution. Some of us, including me, wanted to overhaul the curriculum. But others argued against abandoning our focus on traditional skills and concepts.

I see this (conservative) attitude with faculty all the time. And I understand the sentiment: the current approach is good enough, it’s too much work to change, and the risk is too high of making things worse.

The ivory tower may seem like solid rock¹, and the economics around student loans (in the US) are bizarre, but success hides problems. Life catches up to you fast and tenure is no insurance policy against disruption. And nothing says disruption like what we’re currently going through.

Imagine how different the story might have been if they overhauled the curriculum and renamed “geology” to something like “earth and planetary sciences”. Geology is pretty old-fashioned sounding. (See also “statistics department” vs. “data science department”.)

At least here the story isn’t quite over:

Our new vision looks to the future and leaves the past behind, something we could never bring ourselves to do before the prospect of termination forced us to spring into action. We’re not sure how the university will react to our plan. Hopefully, our efforts aren’t coming too late.

https://science.sciencemag.org/content/371/6527/434

How’s that for a mixed metaphor! ↩︎

"The day we let Covid-19 spin out of control"

Sat, 23 Jan 2021 15:00:00 -0500

Compelling piece in STAT on COVID-19:

Jan. 24 marks the one-year anniversary of a momentous but largely unnoticed event in the history of the Covid-19 pandemic: the first published report of an individual infected with the novel coronavirus who never developed symptoms. This early confirmation of asymptomatic infection should have set off alarm bells and profoundly altered our response to the gathering storm. But it did not. One year later we are still paying the price for this catastrophic blunder.

Chilling. Even Fauci got it wrong:

On Jan. 28, 2020, [Fauci] said, “In all the history of respiratory-borne viruses of any type, asymptomatic transmission has never been the driver of outbreaks. … Even if there’s a rare asymptomatic person that might transmit, an epidemic is not driven by asymptomatic carriers.”

But he also corrected:

To his credit, Fauci was among those who immediately criticized this remark. Based on epidemiological data that had become available since his earlier comments, he said it was “not correct” to characterize asymptomatic transmission as rare.

https://www.statnews.com/2021/01/23/asymptomatic-infection-blunder-covid-19-spin-out-of-control/

Failing fast for fun and feedback

Thu, 21 Jan 2021 09:00:00 -0500

Here’s a wonderful anecdote about aeronautical engineer Paul MacCready who built the first successful human-powered airplane and in so doing won the first Kremer Prize. I’ve seen this story told in many different outlets¹ since I learned about it; it appears the original popularizer was Alan Kay.

The story goes like this. MacCready became interested in building a plane that could fly using only human power (his winning design was essentially bicycled by the pilot). Of course, people were working on this already for years, chasing the prestigious Kremer Prize, so what could he contribute?

MacCready’s insight was that people were solving the wrong problem. To create a human-powered aircraft they were trying to create human-powered aircraft! The right problem:

Create an aircraft that can be rebuilt quickly after being destroyed.

MacCready saw other teams working on their planes. They would take 10–12 months carefully theorizing, designing and building their aircraft, fly it, after 30 seconds or a minute it would crash, and they would spend another 10–12 months working on their next design to get another 30–60 seconds of data. MacCready set about designing a plane that he could crash over and over again. Crash and rebuild. Crash and rebuild. Everyday.

Now MacCready and his team are gathering data far faster than any other team. And they can iterate, quickly, honing in on the eventual working design.

I love this anecdote because it’s so cool but also because it has two keen insights. The more obvious insight is that feedback loops need to be tight. Make those loops as tight as possible to keep pushing through your failures to get to a success. If you’re trying to design an algorithm, say, or test a theory, and it takes days writing code for each change you want to test, you may never find the solution. So put the time in so you can turn around your tests in minutes.

The second, more subtle insight is on picking the right problem. The obvious problem (to create a human-powered airplane we need to create a human-powered airplane) may be predicated on a non-obvious problem (to create a human-powered airplane we need to create a plane that is easy to rebuild after crashing). A simple twist of insight can unlock a profound solution.

For example:
↩︎

Announcing the OCEAN research awards program — Help us spread the word!

Wed, 20 Jan 2021 14:00:00 -0500

Here’s a new funding opportunity coming from the UVM Open Source Ecosystems and Networks (OCEAN) project that I lead with Laurent Hébert-Dufresne.

From the call:

The University of Vermont Open-Source Ecosystems and Networks (OCEAN) research team is excited to announce the launch of the Open-Source Ecosystems and Networks Research Awards program, The OCEAN awards program will provide financial support to a global network of researchers, students, and open source experts who are working to deepen understanding of how people, teams, and organizations thrive in technology-rich settings, especially in open-source projects and communities. Funding will be given for four types of proposals types: collaboration awards, pilot research project awards, computing resources awards, and micro-awards.

“The mission of OCEAN is to study how open-source communities come together to solve complex problems,” says UVM Professor Laurent Hébert-Dufresne. “To do so, the OCEAN project itself must be open and supportive of outside ideas and solutions. These awards will therefore contribute to our research mission and also help us build new bridges between UVM and research groups and open-source communities around the globe." says UVM professor and OCEAN co-leader Laurent Hébert-Dufresne.

Interested groups and individuals from all fields and professional stages are encouraged to apply Awardees will be welcomed into the OCEAN research community to collaborate with one another via a shared Slack workspace, quarterly virtual brainstorming meetings, and a shared Github repository.

There’s four tracks:

Collaboration awards
Pilot project awards
Computing resources awards
Micro-awards

Submissions will be accepted on a rolling basis until November 1, 2021.

Interested? Check out the website for selection criteria, FAQs, and more.

https://vermontcomplexsystems.org/partner/OCEAN/awards/

Open source ecosystems need equitable credit across contributions - new letter

Wed, 20 Jan 2021 09:00:00 -0500

My colleagues and I have a new letter to the editor in Nature Computational Science:

Open source ecosystems need equitable credit across contributions ¹
Amanda Casari, Katie McLaughlin, Milo Z. Trujillo, Jean-Gabriel Young, James P. Bagrow & Laurent Hébert-Dufresne, Nature Computational Science 1, 2(2021)

The core point is that open source needs a better model for how it acknowledges contributors and their contributions, particularly contributions to an open source project that go beyond the code itself. Examples of such non-code contributions include organizing outreach efforts to popularize an open source solution, writing training docs, or helping to drum up financing. We argue in our letter that the CRediT model for academic publishing may serve as inspiration for a similar model tailored to open source contributions.

Here’s an excerpt:

Collaborative and creative communities are more equitable when all contributions to a project are acknowledged. Equitable communities are, in turn, more transparent, more accessible to newcomers, and more encouraging of innovation — hence we should foster these communities, starting with proper attribution of credit. However, to date, no standard and comprehensive contribution acknowledgement system exists in open source, not just for software development but for the broader ecosystems of conferences, organization and outreach efforts, and technical knowledge. Furthermore, both closed and open source projects are built on a complex web of open source dependencies, and we lack a nuanced understanding of who creates and maintains these projects. As a result, large sums and efforts go to open source software projects without knowing whom the investments support and where they have impact.

Check out the letter for more. Nature Computational Science looks to be an interesting journal.

Additional links:
- Journal page: https://doi.org/10.1038/s43588-020-00011-w
- My homepage: https://bagrow.com/#pub63
↩︎

"The Nonsense of Style"

Tue, 19 Jan 2021 09:00:00 -0500

Came across a fantastic, insightful post about academic writing.

The problem with the likes of Steven Pinker and Helen Sword is that they like their own writing way too much.

Shots fired.

I certainly found Pinker’s book on writing to be absolutely insufferable.

Sure, when you’re a successful, well-read author, you can expect, well, people to read what you write. What if you are not? How can you get your ideas across to readers who don’t have the time for reading? Completely unaddressed. Yet this is the key writing issue of our time, as the tsunami of papers continues to grow.

But here’s the power pose of the post:

There is only one criterion that we should require of academic writing and that is the same requirement we should have of academic thinking. Scruples. Scrupulous writing will let the reader in on the uncertainty of knowledge, and messiness of the process of how knowledge is created

Scruples ¹.

Check out the full post for even more insights and good advice.

https://metaphorhacker.net/2021/01/the-nonsense-of-style-academic-writing-should-be-scrupulous-not-stylish/

Of course, scruples alone may not be enough to survive contact with the buzzsaw of peer review. Which is exactly how non-scrupulous writing gets rewarded. ↩︎

"Data pathology"

Fri, 15 Jan 2021 11:00:00 -0500

Interesting short post from John D. Cook this week:

Data scientists often complain that the bulk of their work is data cleaning. But if you see data cleaning as the work, not just an obstacle to the work, it can be interesting. You could think of it as data pathology, a kind of analysis before the intended analysis.

It’s a point I try to make a lot in my courses, that cleaning your data is not just necessary but also teaches you about what you’re studying. I think the data pathologist is a nice concept, and not one I would have reached for myself.

https://www.johndcook.com/blog/2021/01/12/data-pathology/

Efficient crowdsourcing with Cost Forecasting - new paper

Tue, 22 Dec 2020 12:00:00 -0500

We have a new paper out:

Efficient crowdsourcing of crowd-generated microtasks ¹
Abigail Hotaling and James P. Bagrow, PLoS ONE 15(12): e0244245 (2020)

In this paper, we studied how to make crowdsourcing more efficient when members of the crowd can provide creative input and not merely perform basic, rote work.

What is crowdsourcing?

Crowdsourcing is the process of distributing work to a large number of individuals (the crowd) usually by dividing that work into many small tasks. Labeling images is a classic example: You have millions of photographs of objects that you need to be labeled, perhaps to use as training data for an image recognition neural network. Instead of sitting and slowly labeling these yourself, you use a crowdsourcing platform such as Amazon Mechanical Turk to send the images out to many hundreds or even thousands of people who perform the labeling for you. Generally, performing each microtask (for example, providing a label for a single image) leads to a small payment, although not all crowdsourcing schemes reward the crowd with money. Given the costs of rewarding the crowd (with or without money), we want to be as efficient as possible when performing our crowdsourcing.

Crowdsourcing drudgery and creativity

Being able to divide a large job into many easy-to-distribute microtasks is one of the classic ingredients for a successful crowdsourcing, and clearly-defined microtasks are crucial for using efficient crowdsourcing algorithms. But the vast majority of microtask work is quite rote—it’s drudgery. Labeling images, transcribing audio recordings, filling in true-false or multiple-choice questions and the like. Asking people to perform such purely rote work is limiting: people are capable of creativity in ways that machine learning simply is not. A crowd can provide new and even unexpected information. But if you only ever ask people to perform boring, rote work, you will never have the chance to leverage this innate creative capacity.

There are many examples of creative crowdsourcing. Wikipedia is a crowdsourced encyclopedia for instance. But it’s hard to leverage algorithms for such problems because algorithms need clear microtasks and a task like writing an encyclopedia is highly unstructured and open-ended.

So we ask: is it possible to combine creativity with microtasks and get the best of both worlds, creative input from the crowd and the use of efficient algorithms? Yes, we argue it is.

Combining rote and creative crowdsourcing: challenges and solutions

There is a middle ground between completely rote, structured crowdsourcing (labeling images) and completely open-ended crowdsourcing (writing an encyclopedia) that we call the “Reply & Supply” framework. Here, there are two types of tasks:

Reply tasks, where crowd participants simply complete microtasks (for example, answer an existing true-false question); and
Supply tasks, where participants propose new microtasks for other members of the crowd (for example, write a new true-false question).

While the crowd should work within the Reply & Supply guide rails, they have more creative input in the overall problem than they would with purely rote tasks.

But, of course, there are problems with Reply & Supply. Most algorithms that efficiently assign microtasks to the crowd need a fixed and known number of microtasks. More than that, with a growing set of microtasks, you can quickly become overrun with unfinished work, so growth needs to be controlled ².

To help address these problems, constraining task growth appropriately is the focus of our new paper. We propose a method called Cost Forecasting for choosing when to request a new microtask versus when to request an answer to an existing microtask. Combining this decision process with an efficient microtask crowdsourcing algorithm then allows algorithmic crowdsourcing to be used when the set of microtasks is growing.

The abstract:

Allowing members of the crowd to propose novel microtasks for one another is an effective way to combine the efficiencies of traditional microtask work with the inventiveness and hypothesis generation potential of human workers. However, microtask proposal leads to a growing set of tasks that may overwhelm limited crowdsourcer resources. Crowdsourcers can employ methods to utilize their resources efficiently, but algorithmic approaches to efficient crowdsourcing generally require a fixed task set of known size. In this paper, we introduce cost forecasting as a means for a crowdsourcer to use efficient crowdsourcing algorithms with a growing set of microtasks. Cost forecasting allows the crowdsourcer to decide between eliciting new tasks from the crowd or receiving responses to existing tasks based on whether or not new tasks will cost less to complete than existing tasks, efficiently balancing resources as crowdsourcing occurs. Experiments with real and synthetic crowdsourcing data show that cost forecasting leads to improved accuracy. Accuracy and efficiency gains for crowd-generated microtasks hold the promise to further leverage the creativity and wisdom of the crowd, with applications such as generating more informative and diverse training data for machine learning applications and improving the performance of user-generated content and question-answering platforms.

Congrats to Abigail, who was amazing to work with!

Check out the paper for more.

Additional links:
- Journal page: https://doi.org/10.1371/journal.pone.0244245
- arXiv page: https://arxiv.org/abs/1912.05045
- My homepage: https://bagrow.com/#pub58
↩︎
There is also an inherent time bias where early tasks have more opportunities to be addressed by members of the crowd than later tasks. ↩︎

Let's let the AIs summarize our research. What could possibly go wrong?

Wed, 25 Nov 2020 12:00:00 -0500

A Nature News piece came out this week on an AI project to write short summaries of scientific papers:

The creators of a scientific search engine have unveiled software that automatically generates one-sentence summaries of research papers, which they say could help scientists to skim-read papers faster.

“I am amazed it has taken this long to see it in practice,” says Jevin West, an information scientist at the University of Washington in Seattle who tested the tool at Nature’s request.

I am also amazed that it appears no thought has gone into how such a tool will be used, and to what ends?

I informally tested the tool using abstracts from my papers (a very convenient sample, let it be known) and some others. For papers intended for computer science conferences, it does a pretty good job, although I would argue the summaries are rephrasings of the title (which it was not shown). But for other papers, such as those intended for applied math or physics venues, it generally just duplicates the “Here we show […]” sentence, sometimes with one or two synonyms swapped in. I think most trained readers are already well practiced at jumping right to that sentence.

I worry a lot about information overload, misinformation, and the tsunami of scientific papers. Could AI help us? Yes. But can it harm us? Absolutely.

TLDR: do vaccines cause autism?

As another test, I ran the summary of the retracted Andrew Wakefield paper through their tool. Here is its TLDR:

We identified a consecutive series of children with chronic enterocolitis and regressive developmental disorder, associated with measles, mumps, and rubella vaccination.”

Is that a successful summary? It’s well written and clear, and seems to capture, rather forcefully, the intent of the paper.

Let’s look at the original text, so we can also see what their method did. I’ve boldfaced two sentences I wish to discuss:

Background

We investigated a consecutive series of children with chronic enterocolitis and regressive developmental disorder.

Methods

12 children (mean age 6 years [range 3–10], 11 boys) were referred to a paediatric gastroenterology unit with a history of normal development followed by loss of acquired skills, including language, together with diarrhoea and abdominal pain. Children underwent gastroenterological, neurological, and developmental assessment and review of developmental records. Ileocolonoscopy and biopsy sampling, magnetic-resonance imaging (MRI), electroencephalography (EEG), and lumbar puncture were done under sedation. Barium follow-through radiography was done where possible. Biochemical, haematological, and immunological profiles were examined.

Findings

Onset of behavioural symptoms was associated, by the parents, with measles, mumps, and rubella vaccination in eight of the 12 children, with measles infection in one child, and otitis media in another. All 12 children had intestinal abnormalities, ranging from lymphoid nodular hyperplasia to aphthoid ulceration. Histology showed patchy chronic inflammation in the colon in 11 children and reactive ileal lymphoid hyperplasia in seven, but no granulomas. Behavioural disorders included autism (nine), disintegrative psychosis (one), and possible postviral or vaccinal encephalitis (two). There were no focal neurological abnormalities and MRI and EEG tests were normal. Abnormal laboratory results were significantly raised urinary methylmalonic acid compared with agematched controls (p=0·003), low haemoglobin in four children, and a low serum IgA in four children.

Interpretation

We identified associated gastrointestinal disease and developmental regression in a group of previously normal children, which was generally associated in time with possible environmental triggers.

We see that the method has taken an early sentence (boldfaced sentence 1), and glued it onto a later sentence (boldfaced sentence 2). It replaced a word with a synonym (‘investigated’ becomes ‘identified’, which is stronger) and it removed extenuating clauses (“associated by the parents").

In other words, the summarization actively strips out the little bit of nuance from this—famously retracted—paper. Overcompression is dangerous.

Imagine what happens when someone starts tweeting that out, then it jumps over to Facebook? Or worse, not a person but a bot!

I should also say, I am unpleasantly surprised at the upbeat coverage from Nature News. Normally, I expect an expert “who was not involved in the study” to provide at least some context or nuance about the work, whether it be remaining obstacles or unanswered questions. Why didn’t someone mention how dangerous a tool like this could be?

From Nature News again:

But Weld says the team is working on generating summaries for non-expert audiences.

Indeed.

Many people are concerned about AI and misinformation, just look at the conversation around tools like GPT-2 and GPT-3. It does not appear, at least from their papers, that Weld’s team put much thought into these concerns. What does the Allen Institute for AI (whose tagline is “AI for the Common Good”) think of their project?

"US faculty job market tanks"

Mon, 19 Oct 2020 10:30:00 -0400

Potentially terrifying news piece (warning: paywalled) in Science last week. Job openings in the US are significantly down year-over-year:

The scarcity of academic jobs is a perennial problem for U.S. science trainees. But this year, faculty job openings at U.S. institutions are down 70% compared with last year, according to an analysis of job advertisements on the Science Careers job board. (Science’s news team operates independently from the job board.) Only 173 U.S.-based jobs were posted from July to September, compared with 571 during the same period last year. Non-U.S. job postings dropped by 8%.

(Emphasis mine.)

Here’s the trend so far for one job board:

Courtesey Science.

The dismal numbers reflect anxiety about university finances amid the pandemic, says Robert Zemsky, a professor of education at the University of Pennsylvania who studies university finances. Big public universities, in particular, are a “total mess,” he says. “They are losing enrollment, they are losing revenue, and they don’t know what to do, so they have hiring freezes everywhere.”

There’s an argument at the end about a silver lining, that if/when the market rebounds, job searches will be more open to candidates who pursued a non-traditional career track, but that feels pretty thin compared to the potential loss we are going to feel.

https://science.sciencemag.org/content/370/6514/272

Creativity in temporal social networks - new paper

Wed, 14 Oct 2020 09:00:00 -0400

We have a new paper out this week:

Creativity in temporal social networks: how divergent thinking is impacted by one’s choice of peers ¹
Raiyan Abdul Baten, Daryl Bagley, Ashely Tenesaca, Famous Clark, James P. Bagrow, Gourab Ghoshal and Ehsan Hoque, J. R. Soc. Interface. 1720200667 (2020)

The abstract:

Creativity is viewed as one of the most important skills in the context of future-of-work. In this paper, we explore how the dynamic (self-organizing) nature of social networks impacts the fostering of creative ideas. We run six trials (N = 288) of a web-based experiment involving divergent ideation tasks. We find that network connections gradually adapt to individual creative performances, as the participants predominantly seek to follow high-performing peers for creative inspirations. We unearth both opportunities and bottlenecks afforded by such self-organization. While exposure to high-performing peers is associated with better creative performances of the followers, we see a counter-effect that choosing to follow the same peers introduces semantic similarities in the followers’ ideas. We formulate an agent-based simulation model to capture these intuitions in a tractable manner, and experiment with corner cases of various simulation parameters to assess the generality of the findings. Our findings may help design large-scale interventions to improve the creative aptitude of people interacting in a social network.

Congrats to my collaborators, especially Raiyan for leading us. Fantastic group!

Check out the paper for more. And my collaborators have a cool followup paper you may be interested in.

Additional links:
- Journal page: https://doi.org/10.1098/rsif.2020.0667
- arXiv page: https://arxiv.org/abs/1911.11395
- My homepage: https://bagrow.com/#pub57
↩︎

"Do social media algorithms erode our ability to make decisions freely? The jury is out"

Tue, 13 Oct 2020 09:00:00 -0400

Allow me to toot my own horn for a second. Just published online is a little piece in The Conversation that Lewis Mitchell and I wrote about a 2019 paper of ours:

Our results mean that even if you #DeleteFacebook (which trended after the Cambridge Analytica scandal in 2018), you may still be able to be profiled, due to the social ties that remain. And that’s before we consider the things about Facebook that make it so difficult to delete anyway.

We also found it’s possible to build profiles of non-users — so-called “shadow profiles” — based on their contacts who are on the platform. Even if you have never used Facebook, if your friends do, there is the possibility a shadow profile could be built of you.

On social media platforms like Facebook and Twitter, privacy is no longer tied to the individual, but to the network as a whole.

Do social media algorithms erode our ability to make decisions freely? The jury is out.

If you’re interested in learning more, here’s the paper itself.

Naming things, Python dictionary Ed.

Fri, 02 Oct 2020 09:00:00 -0400

I really enjoy using Python. It’s a simple, expressive language with lots of useful libraries. It comports well with my thinking. And I suspect I am not alone: it’s become one of the most popular programming languages.

In this post, I’ll talk about Python dictionaries. In many ways, the dictionary (or “dict”) is the core data structure of Python, analogous to matrices in MATLAB or dataframes in R. Dictionaries are not unique to Python, most languages support them, often calling them associative arrays, hashes, or maps. Here’s an example of a small dictionary:

d = {'a':1, 'b':2, 'c':3}

Here d maps three letters (strings) to numbers (integers), and we can retrieve the integers whenever needed using the strings:

i = d['a']
print(i)

The square bracket “access” operator lets us plug a key into the dictionary and return a value (or an error if the key is not in the dictionary).

Python dictionaries are particularly easy to use, efficient, and flexible. In fact, there’s an awesome chapter in Beautiful Code entitled Python’s dictionary implementation: being all things to all people that I highly recommend if you’re interested in learning about how, and more importantly, why, dictionaries were built the way they were ¹. Python is often criticized for being inefficient, as you’d expected for a dynamically typed language. But I am often shocked how much a well-placed dictionary can speed things up; it’s memory access is amazing.

Flexibility

Besides efficiency, another useful property of dictionaries are their flexibility. Because Python is dynamically typed, and everything is an object, you are not limited to dictionaries with homogeneous data types. In other words, instead of making a dictionary like in the example where every key is a string and every value is an integer, you can mix and match data types:

d = { 1:1,
      2:'a',
      'series': [1,2,3,],
      'map': {'a':1, 'b':2, 'c':3}
}

This is a perfectly valid dictionary. It has both keys and values of different types. Values can themselves be data structures, d['series'] is a list and d['map'] is itself a dictionary, nested within d! There are limits on what types of variables can be used as keys: a key must be immutable, so you can’t use a list or a dictionary as a key, even though they are fine as values.

This second example d is what I call a heterogeneous dictionary. The variables, both keys and values, are of mixed types and almost certainly have mixed meanings. Heterogeneous dictionaries are extremely useful, but excessive heterogeneity can make code difficult to understand and potentially unreliable.

Naming things — keep dictionaries homogeneous

I am a strong advocate for keeping dictionaries simple, and in particular homogeneous. Each dictionary should keep track of one thing, or one “kind” of thing. That way you always know what to expect when you access a key. Instead of the heterogeneous, kitchen-sink approach, keep a small, curated collection of dictionaries. Just because you can use heterogeneous dictionaries in Python doesn’t mean you should.

Why is this helpful? For me, it enables a very powerful naming convention.

I almost never name my dictionaries with generics like d or data. Instead, I use a naming convention of key2value ²:

user2age = {}
user2group = {}
group2location = {}

and so forth.

The idea is to never name the dictionary as a singular entity, but always focus on the inputs and outputs of the dictionary. When you keep the dictionary homogeneous in those terms, the dictionary—and your code—become predictable:

for u in user2age:
    a = user2age[u]
    g = user2group[u]
    print("User group age", u, g, a)

At every point in that code, it’s quick to understand what variables are doing. Consider if I had written:

for u in user2age:
    a = user2age[u]
    g = user2group[a] # !!!

It’s much more obvious that I made a mistake, because at a glance I can see that a is not a user but it’s going into a dict that takes users as keys. Now imagine that users are integers (say, ID numbers), and ages are also integers. That bug may not actually throw a key error: if there exists a user called 42 and one or more users happen to have an age of 42, then user2group[a] will return a valid value. The dictionary doesn’t know when 42 means user ID 42 and when 42 means 42 years. A subtle bug has been avoided.

This naming convention is especially helpful with nested dictionaries:

grade = group2student2grade[grp][stu]

When you have a dictionary-of-dictionaries you can call into the outer and inner dictionary simultaneously using two [] operators in a row. By extending the naming convention so the value in the outer dictionary is another name of the form key2value, not only do we immediately know we have nested dictionaries, we can also quickly see if the outer and inner keys have been passed correctly: does [grp][stu] match up with group2student? Yup. So we are good to go, assuming of course that grp and stu are correct.

It’s all about the names

These examples underscore why naming variables carefully is so crucial to readable, maintainable, predictable code. Both for k in D and for u in user2name can be exactly the same, but the former is so generic it can be hard to detect issues, while the latter gives more “hooks” to hang our understanding on. It doesn’t solve all problems ³, but it helps!

I suspect some of the specifics are now out-of-date. Python dict’s have changed considerably in the past few years, especially now that they preserve insertion order of keys. But the general idea from that chapter should still hold. ↩︎
Yes, I am aware of word2vec. Despite claims to the contrary, they, nor I, invented the naming convention input2output. See also Hungarian notation, which has advantages and disadvantages. ↩︎
And no, neither does a statically typed language. Python forever! ↩︎

200

Mon, 21 Sep 2020 14:00:00 -0400

Today I learned it’s been 200 days since I last entered my workplace:

Notification for the local backup I keep in my office.

Nevermind that 200. We are at a death toll in the US of 200,000. Two Hundred. Thousand.

Meanwhile, the CDC can’t get their story straight, our anti-leadership is monstrous including counterproductive finger-pointing, and people continue to die. On September 17, the CDC ensemble forecast predicted 207,000–218,000 deaths by October 10.

Of course, given the stunning incompetence on display, who knows what will really happen next. On the bright side, I recently learned that CU Boulder went five weeks before abandoning in-person teaching. I doubted they would last three weeks, just long enough for the COVID symptoms to manifest.

The virus is a fast-moving, direct threat to our way of life. Not as fast as, say, a wild fire, of course. But if we can’t muster the basic forces necessary to get our arms around this thing, what hope is there against a slower-paced but more insidious threat.

It feels more and more like we are discovering first-hand the great filter.

"Why Mathematicians Should Stop Naming Things After Each Other"

Tue, 15 Sep 2020 09:00:00 -0400

Why Mathematicians Should Stop Naming Things After Each Other

Interesting piece in Nautilus about mathematics and the lack of accessibility that comes from using mathematician’s names to describe their results:

When everything is named for its discoverer, it can be impossible even to track the outline of a debate without months of rote memorization. The discoverer’s name doesn’t tell you anything about what the landscape is like, any more than the “Ackerman” in Ackerman’s Island helps to convey a sandbar in downtown Wichita.

This nesting of proper nouns helps to make higher math impenetrable not just to outsiders, but also to working mathematicians trying to read their way from one subfield into another. The venerable Bill Thurston was known to complain about the perversity which, by the end of his career, had produced Thurston’s theorem, which says that Thurston maps are Thurston-equivalent to polynomials, unless they have Thurston obstructions. Every field has terms of art, but when those terms are descriptive, they are easier to memorize. Imagine how much steeper the learning curve would be in medicine or law if they used the same naming conventions, with the same number of layers to peel back:

This seems sensible to me, but there’s a catch. How else can you name abstract results? When the results are “pure” you often have no intuition upon which to anchor your name, and physically intuitive names can be misleading. Yes, it hurts accessibility, but by the time you reach the frontier of mathematics, you are already pushing to the extremes of human understanding.

A mix of naming styles is likely the best option. Results in mathematics, like science, undergo a kind of evolutionary selection, with important and understandable results bubbling up and less important results fading away. A good choice of name is one way to increase an idea’s “fitness”.

I also feel that it’s important to celebrate and humanize our heroes, and naming results after people is one way to do it. Learning the histories of past generations of mathematicians and scientists is fascinating and, at least for me, knowing personal biographies helps me remember results. There’s a lot of bias in these histories, though, and using surnames to memorialize results marginalizes women, so there is still work to be done.

"Computational Social Science: Obstacles and Opportunities"

Mon, 14 Sep 2020 09:00:00 -0400

Computational Social Science: Obstacles and Opportunities

New policy piece in Science about the growing, multidisciplinary field of computational social science:

The field of computational social science (CSS) has exploded in prominence over the past decade, with thousands of papers published using observational data, experimental designs, and large-scale simulations that were once unfeasible or unavailable to researchers. These studies have greatly improved our understanding of important phenomena, ranging from social inequality to the spread of infectious diseases. The institutions supporting CSS in the academy have also grown substantially, as evidenced by the proliferation of conferences, workshops, and summer schools across the globe, across disciplines, and across sources of data. But the field has also fallen short in important ways. Many institutional structures around the field—including research ethics, pedagogy, and data infrastructure—are still nascent. We suggest opportunities to address these issues, especially in improving the alignment between the organization of the 20th-century university and the intellectual requirements of the field.

Lots of sensible recommendations, but it seems the problems they highlight are too entrenched structurally. Can we improve?

I really like this point:

[I]n a networked world, how should we deal with the fact that sharing information about oneself intrinsically provides signals about those with whom one is connected?

Indeed, my collaborators and I studied this information flow in a recent 2019 paper ¹, and we found that one’s social ties contain a ton of potential predictive information.

Bagrow, J. P., Liu, X., & Mitchell, L. (2019). Information flow reveals prediction limits in online social activity. Nature Human Behaviour. https://doi.org/10.1038/s41562-018-0510-5 https://arxiv.org/abs/1708.04575 ↩︎

The anti-science stance of Prometheus

Thu, 10 Sep 2020 09:00:00 -0400

Run. Not that way.

Spoilers. You’ve been warned.

Prometheus, the 2012 Ridley Scott-directed not-a-prequel to Alien, is an interesting failure. The film is gorgeous and the actors generally do a fantastic job. The failure of the movie is the story, the plot. Many reviewers at the time of release went into its problems ¹ . One problem that many have pointed out is that the plot is powered by characters doing stupid things, like (literally) running in the wrong direction. I want to discuss another problem I have with the film, one that’s been bugging me ever since I saw it in the theater.

The movie is willfully, belligerently anti-science.

Characters in the film grapple with questions of human origins, the meaning of life, the cosmic why. Indeed, an important characteristic of the main character is her christian faith ². Many films love to contrast faith and science and this can lead to powerful dramatic tension, although I believe this tension is vastly overblown in movies compared to real working scientists—the debates in Contact, for example, seem quite childish.

This is a noble conversation to have in any film. Yet I conclude the film is anti-science because of the way it treats its characters, most of whom are scientists. I was especially excited to see these characters because I wanted to see what scientists and explorers would do in Scott’s big return to the sci-fi genre. Alien, in contrast, featured a “blue collar” crew—think truckers in space—which was radical in the 1970s. Now we were coming full circle to the origins of sci-fi, with explorers, scientists, and soldiers. What story potential.

Instead, we got a film that features:

A geologist getting lost in caves and complaining, ‘All these rock tunnels look the same’.
A biologist completely misunderstanding an alien creature’s painfully obvious threat display.
An archaeologist, after making the greatest scientific discovery in history, being sad that he can’t learn anything because everyone was dead. An archaeologist!

The scientists are not incompetent, they are anti-competent.

That last one is exceptionally galling:

[Android]: I’m very sorry that your engineers are all gone.
[Archaeologist]: You think we wasted our time coming here, don’t you?
[Android]: Your question depends on me understanding what you hoped to achieve by coming here.
[Archaeologist]: What we hoped to achieve was to meet our makers. To get answers. Why they even made us in the first place.

To get answers? By talking to someone? You’re sitting outside a ruin that provides a dozen lifetime’s worth of archaeological exploration. What do think archaeologists do?

Now, I understand the film needs an exciting plot and seeing characters jump for joy after spending hours dusting off a small crevice in a doorway is not much fun to watch. But come on! The scientists could have been genuinely excited by all the scientific possibilities and then a scary, dangerous but exciting situation comes down on them.

Damon Lindelof, Prometheus writer and producer, when asked if the film is anti-science, responded:

It’s definitely not anti-science. In fact, if anything I think it’s pro-science because it advances the idea that part of our own programming as human beings, we’re many ways just as governed by our programming as [the android] David is.

He seems out-of-touch with the actions of these characters.

If any one of these scientists failed I would be OK with it, but all of them together? It’s too much. I am forced to conclude—despite Lindelof’s claims—that the movie intentionally undermines the central expertise of each scientist character.

Roger Ebert, notably, was quite positive. https://www.rogerebert.com/reviews/prometheus-2012 ↩︎
This character is great, and Noomi Rapace’s portrayal is one of the best parts of the movie. The most shocking and best scene in the film by far is a very on-the-nose abortion metaphor. There’s an important debate to be had around the film’s gender politics. ↩︎

"Thousands of Coronavirus Cases Linked to U.S. Universities"

Wed, 09 Sep 2020 09:00:00 -0400

https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html

The New York Times has launched a new tracker for coronavirus cases focused specifically on higher education institutions in the US. It already lists 26 thousand cases at the time of writing (surely it will be more this by the time this post appears).

As college students and professors return to campus in the midst of a pandemic, coronavirus cases are turning up by the thousands.

A New York Times survey of more than 1,500 American colleges and universities — including every four-year public institution, every private college that competes in N.C.A.A. sports and others that identified cases — has revealed at least 26,000 cases and 64 deaths since the pandemic began.

It appears that southern schools are leading the charge, with high counts Alabama, North Carolina, Georgia and Florida.

I do wish these case counts were normalized by school size, either by the total size of the school or by the number of non-remote students, if known.

Stay safe everyone!

Democratizing AI - new paper

Tue, 08 Sep 2020 09:00:00 -0400

Let’s discuss a new paper I have out this week:

Democratizing AI: Non-expert design of prediction tasks (Open Access ¹²)
James P. Bagrow, PeerJ Computer Science 6:e296 (2020)

This study asks whether and to what extent individuals who are not experts in AI or machine learning can contribute to creating and designing machine learning problems. Machine learning is a technical and challenging field. Practitioners often have years of experience, so there may simply not be room for non-experts to design their own prediction tasks. Yet, machine learning is becoming increasingly automated and automatic. The promise of the subfield of AutoML may be a democratization of machine learning, broadening its accessibility.

Of course, non-experts have long contributed to pre-existing problems by contributing training data, but only for pre-established problems ³. It’s easy for someone to help train a machine learning classifier that distinguishes between objects in images by labeling those images ⁴. But what if you don’t want to predict the contents of a photo? What if you have a new task in mind, say you want to predict how someone will respond positively or negatively to a piece of music, but you don’t have any experience in building binary classifiers? As a non-expert (in machine learning; you could be a deeply experienced and talented musician), what can you do?

Enter my paper. Here’s the abstract:

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

In essence, I used a crowdsourcing platform not to collect training data for a pre-existing prediction task, as is traditionally done, but to ask members of the crowd to propose their own prediction tasks. Here each prediction task is a set of input questions and 1 target question. Those familiar with machine learning will see that gathering answers to these questions will give us training data for a supervised learning problem: Try to predict an answer to the target question given only answers to the input questions.

Here are some examples of prediction tasks (Table 1 in the paper), all generated by study participants:

	Prediction task
Target	What is your annual income?
Input	You have a job?
Input	How much do you make per hour?
Input	How many hours do you work per week?
Input	How many weeks per year do you work?

	Prediction task
Target	Do you have a good doctor?
Input	How many times have you had a physical in the last year?
Input	How many times have you gone to the doctor in the past year?
Input	How much do you weigh?
Input	Do you have high blood pressure?

	Prediction task
Target	Has racial profiling in America gone too far?
Input	Do you feel authorities should use race when determining who to give scrutiny to?
Input	How many times have you been racially profiled?
Input	Should laws be created to limit the use of racial profiling?
Input	How many close friends of a race other than yourself do you have?

By couching the problem in terms of target and input questions, the participants don’t need to know supervised learning details. Here’s a screenshot of the instructions and part of the web interface that participants used to build prediction tasks ⁵.

After gathering a bunch of new prediction tasks, I asked other members of the crowd to categorize the tasks (is it about health? Politics?) and describe the task in several ways, then vote on the “quality” of the task. Using these votes, I ran a ranking algorithm to efficiently determine the few best (according to the crowd votes) tasks, which I then sent out as traditional data-collecting jobs on the crowdsourcing platform. These data were then used to automatically train predictive models to determine if accurate predictions could be made.

I found that machine learning methods could train accurate predictive models ⁶ but challenges remain. For example, if the question calls for a numeric answer with units associated with (how tall are you?) but those units aren’t given, then the different answers won’t necessarily be comparable. I generally found that regression tasks (where the target question had a numeric answer) were more challenging for automatic predictive models. In contrast, classification tasks (where the target question was, in my study, binary) were more likely to lead to predictive models. Participants were overall more likely to propose true/false questions, so it is plausible that they are more comfortable with this format.

There’s a lot more in the paper, including a randomized trial to see if an example prediction task helps participants understand the problem or if it biases them to certain types of prediction tasks (Do participants who see an example about predicting obesity go on to propose more health-focused tasks than participants not given an example?).

From the conclusion:

In general, the more that non-experts can contribute creatively to ML, and not merely provide training data, the more we can leverage areas such as AutoML to design new and meaningful applications of ML. More diverse groups can benefit from such applications, allowing for broader participation in jobs and industries that are changing due to machine-learning-driven workplace automation.

Check out the paper for more.

Additional links:
- Journal page: https://doi.org/10.7717/peerj-cs.296
- arXiv page: https://arxiv.org/abs/1802.05101
- My homepage: https://bagrow.com/#pub50
↩︎

Here’s a BibTeX code if you want a quick cite:

@article{10.7717/peerj-cs.296,
    title = {Democratizing AI: non-expert design of prediction tasks},
    author = {Bagrow, James P.},
    year = 2020,
    volume = 6,
    pages = {e296},
    journal = {PeerJ Computer Science},
    issn = {2376-5992},
    doi = {10.7717/peerj-cs.296}
}

Other reference formats available for download on the journal page. ↩︎

At this point, it may be worth mentioning Kaggle, a platform for crowdsourcing machine learning models. But problems on Kaggle are designed by the data providers, not the crowd. The crowd builds the predictive models; Kaggle is an expert crowdsourcing market, where the crowd are, or will be, experts in machine learning. ↩︎
Heck, we all do this every time a Google ReCaptcha appears. ↩︎
Full details are in the paper’s Supplemental Information. ↩︎
Specifically, I used random forest models. Random forests are a little out-of-date in the era of deep learning, but they are generally robust to overfitting, work for both supervised regression and classification, and are easy to get quite accurate predictions without much fine tuning. ↩︎

Save the gaiters from stress (positions)

Thu, 03 Sep 2020 09:00:00 -0400

The ongoing coronavirus pandemic has led to an explosion of research, with tens of thousands of new preprints and publications on the topic in only a few months. Mask-wearing has emerged as an essential preventative measure (and, for some, sadly, a political touchpoint). This has led to renewed research on the mechanics of how masks prevent coronavirus spread.

Enter the paper:

Fischer, E. P., Fischer, M. C., Grass, D., Henrion, I., Warren, W. S., & Westman, E. (2020). Low-cost measurement of facemask efficacy for filtering expelled droplets during speech. Science Advances, eabd3083.

This study has touched off a firestorm as it led to coverage that some masks are worse than others. Some news coverage even began to report that one type of face covering, a neck gaiter, is worse than no mask at all (a surprising fact if true).

Fortunately, that narrative is now receiving push back.

Here’s a telling quote from the New York Times:

“Our intent was not to say this mask doesn’t work, or never use neck gaiters,” said Martin Fischer, an associate research professor in the department of chemistry at Duke and a co-author of the study. “This was not the main part of the paper.”

It’s no surprise that initial coverage latched onto this supposed result, that gaiters are worse than no mask— It’s an exciting sound bite, and context and nuance are easily lost in the excitement. The authors are working hard to correct the coverage, but I believe the authors are also not entirely blameless.

Let’s dig into the paper a bit to see what responsibility the authors might have. I’ll soon write something about the data within the paper but for now let’s study this in terms of the writing.

Here is the abstract, where I’ve emphasized one sentence in particular.

Mandates for mask use in public during the recent COVID-19 pandemic, worsened by global shortage of commercial supplies, have led to widespread use of homemade masks and mask alternatives. It is assumed that wearing such masks reduces the likelihood for an infected person to spread the disease, but many of these mask designs have not been tested in practice. We have demonstrated a simple optical measurement method to evaluate the efficacy of masks to reduce the transmission of respiratory droplets during regular speech. In proof-of-principle studies, we compared a variety of commonly available mask types and observed that some mask types approach the performance of standard surgical masks, while some mask alternatives, such as neck fleece or bandanas, offer very little protection. Our measurement setup is inexpensive and can be built and operated by non-experts, allowing for rapid evaluation of mask performance during speech, sneezing, or coughing.

Notice when I emphasized that sentence, I strongly emphasized the back half of the sentence. That back half is the stress position of the sentence. The topic position meanwhile gives us some important context: “In proof-of-principle studies […]”.

But the stress position is where readers focus. Notice that the stress position does not say gaiters are worse than not wearing a mask. But also notice that the authors do report a ranking of mask effectiveness. " [S]ome mask alternatives […] offer very little protection". Compare this with Dr. Fischer’s New York Times quote: _ “Our intent was not to say this mask doesn’t work, or never use neck gaiters “_. Sure, they are not saying ‘never use neck gaiters’, but they are saying, in a very prominent position of the very prominent abstract, ‘some mask alternatives […] offer very little protection’.

I see only a fine line, if any, between ‘this mask doesn’t work’ and ‘this mask offers very little protection.’ By placing a crucial qualifier in the less-prominent topic position while emphasizing a surprising finding in the stress position, the authors used the science of scientific writing to massage and control their message.

In many ways, this paper is an excellent highlight of the potential downsides of the science of scientific writing. By upselling their preliminary results to help the paper land in a prominent venue, the authors were forced to deal with a backlash, perhaps partially of their own making.

Soon, I’ll write up some thoughts on the data within the paper. The authors again emphasize the study is on the experimental methodology. But the data they present is also worth a look.

An unexpected hierarchy of visual emphasis

Wed, 02 Sep 2020 09:00:00 -0400

As a followup to my “beef it” macro for LaTeX, I’ve found something funny about emphasizing text. Let’s discuss.

When preparing text, there are multiple ways to add emphasis. You can use ALL CAPS ¹. Or you can use italics. Or perhaps try bold. My lovely bfit (bold face-italics) macro stacks bold and italics together, to really beef it up.

You’d think that bold and italics would stack and given even more emphasis:

Emphasis-level 0: Lorem ipsum dolor sit amet
Emphasis-level 1: Lorem ipsum dolor sit amet
Emphasis-level 2: Lorem ipsum dolor sit amet
Emphasis-level 3: Lorem ipsum dolor sit amet

However, when I look at that, I don’t see the stacking. I actually see a weakening, with the emphasis hierarchy looking more like this:

Emphasis-level 0: Lorem ipsum dolor sit amet
Emphasis-level 1: Lorem ipsum dolor sit amet
Emphasis-level 2: Lorem ipsum dolor sit amet
Emphasis-level 3: Lorem ipsum dolor sit amet

I would guess that it $<$ bfit $<$ bf should depend on typefaces, colors, and other formatting details, but it actually seems quite robust to me. I’m not a typography expert, but I do find it fascinating. I wonder what precisely is going on? I suppose it’s just that bold text strokes are narrower after italicizing than before. But even a typeface like Arial ² that doesn’t seem to change the width of bold when italicizing still has bfit $<$ bf. Does it look bad to keep the same horizontal width in the strokes? Do any bold-and-italicized typefaces not have narrower glyphs?

Emphasis-level 99: ALL CAPS ALL THE THINGS.

Or small-caps. Or underlines, but that looks too much like a hyperlink. ↩︎
Yes, yes. It’s no Helvetica. ↩︎

Beef It: My greatest LaTeX achievement

Tue, 01 Sep 2020 09:00:00 -0400

(This is a silly post. You have been warned.)

(For those unfamiliar with LaTeX and LaTeX macros, see the appendix below for a brief introduction.)

Sometimes, when writing, I really need to emphasize something. Italics are not enough. Bold is not enough. In these situations, I need to beef it up.

How do I go about doing this? With my all-time favorite LaTeX macro, \bfit{}:

\newcommand\bfit[1]{\textbf{\textit{#1}}}

This macro takes any text input and makes it bold and italic. \bfit{Say it loud} becomes Say it loud. So when I really need to pull out all the emphasis stops, I’m ready!

OK, so you’re probably extremely mad at me if you haven’t bailed yet. “So what? It’s bold and italics together. Who cares?”

Beef it up? \bfit{up}. Beef it. bfit. Get it? The macro names itself! Now, when writing, I can figuratively (and sometimes literally) shout BEEF IT UP when writing.

Bold and italics are generally rarely used in formal, peer-reviewed publications. But it can be quite helpful in other professional writing, like grant proposals, memos, even emails. And sometimes in those settings, I need some amusement to get through the slog.

Beef it (bfit) always makes me laugh!

Maybe I’ve already jumped the shark.

Appendix: LaTeX and LaTeX Macros

LaTeX is the way most computational and mathematical writing is produced. It is a fantastic tool that creates impressive, high-quality output. It does have a steep learning curve ¹ but I love it.

Unlike word processors, LaTeX keeps the content separate from the style. I prefer to work this way. This separation makes it easier to treat the writing in an almost computational manner, by letting authors define macros.

At the most superficial level, LaTeX macros let authors define short aliases for long blocks of text. For example, I’m writing a lot of mathematics and need to represent vectors. This macro acts as a quick shortcut

\newcommand{\myvec}[1]{(x_1,x_2,\ldots,x_{#1})}

Calling \myvec{4} would then produce $(x_1, x_2, \ldots, x_4)$.

This saves a lot of typing but it also centralizes a decision. The variable $x$ is now defined within the macro. Suppose in the final stages of drafting, I decide $y$ is more appropriate. I just need to edit \myvec and everything in the document is updated. No need for any potentially time-consuming find-and-replace ².

Part of the learning curve used to be installing and configuring LaTeX. But nowadays all you need to do is create an Overleaf account and you are good to go! ↩︎
We’ve all been there, right? ↩︎

Announcing jupyter-status

Mon, 31 Aug 2020 09:00:00 -0400

I made a very small tool to help me with a particular problem after not finding a suitable existing solution. Let’s talk about it.

https://github.com/bagrow/jupyter-status

The problem

I use Jupyter, especially Jupyter Lab, quite a bit nowadays. And I’ve finally gotten into the habit of using separate virtual environments for each project. Using separate environments is not too bad to get running, and it helps maintain specific sets of dependencies for each project. I really like it.

But lots of ongoing projects leads to lots of running Jupyter sessions with lots of environments. I found myself losing track of what Jupyter sessions were running at any given time.

Enter jupyter-status

Jupyter-status is a simple tool to list all the running servers and their kernels in a compact, convenient tabular format.

Now of course, you can always run jupyter notebook list at the command line. But I wanted more information, in particular I wanted to see all the open and active kernels for each notebook server.

It’s especially handy as a status board if you embed it in your desktop:

(Jupyter-status embedded in my desktop courtesy Ubersicht ¹.)

Use it now

Just install with pip:

pip install jupyter-status

And run it with

jupyter-status

Perhaps you want HTML output instead of plain text?

jupyter-status -f html

Jupyter-status supports all the formats of tabulate.

See https://github.com/bagrow/jupyter-status for more.

Warning. Jupyter-status is quite simple, but I’m no code genius, so be careful if you want to use it. (I hope it works under Windows but I have not tested it.)

GeekTool and Conky are great alternatives. ↩︎

Peer review by mail?

Fri, 28 Aug 2020 09:00:00 +0500

Submitting a paper to a journal is slow.

First you submit. Then you wait. Then submit again because you didn’t follow some copy-editing rule ¹. Then you wait. Then an editor finds the manuscript (if you’re lucky). Then you wait. Then reviewers are found. Then you wait (a lot). Then you receive reviews, work super hard to rebut, and resubmit. Then you wait. Wait, wait, wait ².

Here’s a crazy idea that won’t eliminate all those delays, but could remove some of the bottlenecks.

Authors and reviewers should discuss the paper directly

Whoa now, hold up! “How can they possibly do that? What about blind review? Double-blind?"

Of course. But that’s easy to fix: anonymous, disposable email addresses:

FROM: reviewer-085a6938-1d30-486d-aa34-4ca30ce59bb7
TO:   author-2968511b-a5f6-4614-8881-fde2252b0219
CC:   editor-a83b7c96-b5d7-4530-a3d9-bf46ef516a6e

I don't understand Eq 3 in your paper. Why is sigma always positive?

FROM: author-2968511b-a5f6-4614-8881-fde2252b0219
TO:   reviewer-085a6938-1d30-486d-aa34-4ca30ce59bb7
CC:   editor-a83b7c96-b5d7-4530-a3d9-bf46ef516a6e

Sigma can be zero when measurements are taken using a fluxometer. But we only
ever use a flubometer.

FROM: reviewer-085a6938-1d30-486d-aa34-4ca30ce59bb7
TO:   author-2968511b-a5f6-4614-8881-fde2252b0219
CC:   editor-a83b7c96-b5d7-4530-a3d9-bf46ef516a6e

Oh, of course. You are very smart and your paper is very good. Accept!

(Ok, that last part may be a little bit wish-fulfillment…)

(Also, I’m not discussing the technology much here, the implementation details. Probably the anonymous emails need to be implemented in a webmail system. Not a big deal ³.)

I see more advantages than disadvantages using a rapid-fire, back-and-forth review exchange compared to the traditional review-rebut-review-rebut model.

Pros

Authors can clarify minor details for a reviewer, saving time for the reviewer, the authors (especially if the paper is rejected just due to a misunderstanding), and even the journal.
Reviewers may spend less time overall in an email exchange than in preparing a full report. Authors may spend less time emailing than in developing a detailed rebuttal.
Authors don’t have to wait for the journal to forward the reviewer comments. If reviewers write back at different times, the authors don’t need to respond to everything at once.
Reviewers can talk with each other as well.

Cons

This could make room for abuse. A reviewer could make unacceptable comments directly to the author. In the current system, the journal and editor have a chance to step in and prevent that. However, how often does that happen? I’ve gotten some pretty nasty comments over the years. Further, all the throwaway email correspondence should be CCed to the journal anyway, so it can still be monitored.
Reviewers can talk with each other as well. (It’s common in CS conferences for reviewers to discuss among themselves. But isn’t that a bad idea? Don’t you want diversity in your ensemble?)

Now, this is a crazy idea but I also don’t to be too cavalier.

Peer review is fragile, and there are risks to radical changes. If poorly handled, people could game the system, lousy science could be published, and disadvantaged groups could be further burdened.

Has this been tried before? What am I missing that undermines this idea?

Often a useless rule at this pre-acceptance stage, like putting figures at the end of the manuscript. ↩︎
Conference proceedings tend to be faster by avoiding some of these bottlenecks, but often at the expense of review quality, in my opinion. ↩︎
Also: cue all the Blockchain fanatics. ↩︎

"Practical Data Ethics"

Thu, 27 Aug 2020 13:00:00 +0000

http://ethics.fast.ai

Free online course released on fast.ai this month from Rachel Thomas:

In this course, we will focus on topics that are both urgent and practical. In keeping with my teaching philosophy, we will begin with two active, real-world areas (disinformation and bias) to provide context and motivation, before stepping back in Lesson 3 to dig into foundations of data ethics and practical tools. From there we will move on to additional subject areas: privacy & surveillance, the role of the Silicon Valley ecosystem (including metrics, venture growth, & hypergrowth), and algorithmic colonialism. I realize this course still just covers a slice of what is a sprawling field, and I hope that it will be a helpful entry point for continued exploration.

Looks like a lot of interesting useful material. Barrier-to-entry seems low too:

This class was originally taught in-person at the University of San Francisco Data Institute in January-February 2020, for a diverse mix of working professionals from a range of backgrounds (as an evening certificate courses). There are no prerequisites for the course.

Secret Santa assessment

Wed, 26 Aug 2020 13:00:39 +0000

I’m posting this half-baked idea here because (i) I think it’s neat, (ii) I love the cute name and want to share it, and (iii) it’s likely a little too far outside my area of expertise and will take too much effort for me to (credibly) get through as peer review.

So let’s beta-test it. Is it interesting or useless? Is it novel or has it already been proposed? Let me know!

Introduction

Misaligned and competing individual performance objectives are a root cause of organization dysfunction. If you need help fixing the conference room projector 45 minutes before your meeting, for example, it will not be useful for IT to process your support request tomorrow, even though their key performance indicator (KPI) is number of requests completed within 48 hours. The timeliness of your need, which affects your KPI, is not reflected in the KPIs of other members of your organization.

Here we propose a simple way to unify assessments across an organization when members of the organization are assessed along different dimensions. We leave aside the (significant) challenge of determining these assessment metrics. Indeed, many roles may be difficult to assess clearly and organizations can be misled by ill-informed KPIs. Instead, we focus on a way to ensure that different assessment metrics (KPIs), when applied differently across the organization, do not lead to diverging objectives or competing incentives.

“Secret Santa” assessment

Suppose $m$ key performance indicators (KPIs) has been developed for an organization of $N$ members. Let $\mathbf{k}_i \in \mathbb{R}^m$ be an $m \times 1$ vector representing the performance of member $i$ according to these KPIs. Of course, each member will be focused on only a subset of KPIs relevant to their organizational role, so we can think of $\mathbf{k}_i$ as being a sparse vector. Let $s_i : \mathbb{R}^m \to \mathbb{R}$ be a scoring function assigned to $i$. Suppose this scoring function is used by the organization to assess member performance; it may be tailored to the duties of $i$ as necessary, and can consider only the subset of KPIs relevant to $i$.

Ideally, different scoring functions should align with the overall goals of the organization so members are incentivized to collaborate. However, competing incentives may form when different members have different scoring functions that are not sufficiently aligned. For such situations, we introduce a simple process to more strongly couple or unify incentives across an organization without having to replace or modify the existing KPI system.

For each member $i$, pair $i$ with $n$ ($1 \leq n \leq N-1$) other members $j$ chosen uniformly at random. Then, define a new score $\sigma_i$ that acts as a scalarized combination of $i$’s own score and the average score of the randomly paired members:

$$\sigma_i = \alpha s_i (\mathbf{k}_{i}) + \left(1-\alpha\right) \frac{1}{n}\sum_{j} s_j(\mathbf{k}_j) \tag{1}$$

where $\alpha \in [0,1]$ and the sum runs over the $n$ members $j$ that were randomly paired with $i$.

Discussion

Why combine scoring functions in this manner, particularly using random pairing? The new score $\sigma_i$ aligns $i$ both with their KPIs and the KPIs of other members $j$, with $\alpha$ controlling the relative strength. By coupling member performance the hope is that all members will be more strongly incentivized to help one another, lowering competition between members and between divisions.

Random pairing, as opposed to the design of a pairing network specific to the organization, allows for this assessment strategy to be implemented without additional time or effort. In other words, Eq. 1 serves as an information-free coupling of member incentives. Random sampling also helps ensure fairness and prevent perverse incentives: all possible pairings are equally likely and members will not benefit much from “gaming the system” by optimizing their paired KPI scores. We also anticipate each member assessment, for example, quarterly assessments, will be computed using a fresh set of randomly paired $j$’s. Finally, to further avoid perverse incentives, members would not know who they are paired with nor the individual paired scores $s_j$, but only the average score.

Equation 1 has two parameters, $\alpha$ and $n$. We anticipate $\alpha \geq 1/2$ to be appropriate, placing most weight on the member themself. Meanwhile, $n$ controls how much of the organization impinges on an individual. When $n = N-1$ there is maximum coupling—essentially, a single paired score for all members. Perhaps this would act similar to an external assessor, like a stock price. On the other hand, if $n = 1$, each member will be connected to only one other member and may be overly affected when being assessed if, for example, that member performed badly. We therefore anticipate smaller values of $n > 1$ to be most helpful, perhaps in the range 5–10. This value of $n$ provides coupling across the organization without forming a single metric such as stock price (which would provide no per-member scoring information) while also averaging over fluctuations so that each member is not overly affected by a lucky or unlucky pairing.

Of course, there are likely serious flaws with this idea that we have not yet considered. Indeed, we are not currently certain whether this is even a novel idea.

Every other hex bolt

Tue, 25 Aug 2020 13:00:46 +0000

(Minor spoilers for Iron Man.)

There’s an exchange early in Iron Man (2008) that I think about a lot. Two characters must suddenly complete a long-running project and time is running out:

Stark: Come over here and button me up.
Yinsen: Okay. All right.
Stark: Every other hex bolt.
Yinsen: They’re coming!
Stark: Nothing pretty, just get it done.

Here Stark is telling Yinsen there is only time to focus on the critical pieces of the project. Don’t waste time.

“Wasting time” could be the name of my autobiography. So I think to myself ‘Every other hex bolt’ whenever I’m coding something up and I start futzing with spacing in comments. Or tweaking and fiddling with slides. You know, procrastinating ¹.

Figures for me are a huge bugbear—I would become obsessed over all sorts of minor aspects of a figure: aligning subplots, matching font sizes, designing color schemes, rearranging legends, you name it. And while I think good, effective visualizations are critical to science, the software now takes care of so much, and the difference between a ‘Nothing pretty’ figure and a figure I spent 5 hours grinding away at is almost imperceptible to anyone but myself. So don’t waste time.

Every other hex bolt.

Will AI cause a large-scale industrial accident?

Mon, 24 Aug 2020 13:00:00 +0000

The tragedy that occurred in Beirut this month is a chilling reminder of how short-term thinking and organizational blind spots endanger all our lives. Here, sadly, it appears the mistakes were human. Large amounts of dangerous materials were improperly stored and then, apparently, forgotten.

This got me thinking: what could happen if an AI system was managing the shipping and logistics of one or more major ports? Could a sophisticated but improperly specified algorithm, which is difficult for its operators to audit or interpret, let a similar accident happen?

The fears of AI being misused in an industrial setting are not new, and a growing body of work has studied this problem. However, much of that work explores either artificial general intelligence, which still does not exist, or robots. An algorithm controlling an industrial robot¹ needs safety protocols and a physically grounded understanding of the world, so it can understand what nearby humans are doing and how they will react to the robot’s actions.

But here I’m talking about a more ephemeral, and in many ways more practical, application of AI. Even without robots, AIs and automation still play a huge role—and be a huge danger—within a complex system like the global supply chain.

AI Supply Chains

Computerized scheduling is already central to the global supply chain. I had a job unloading trucks in college, and even in 2000, the managers would often be surprised by what the computers had decided to ship when we’d open the trucks. One time the computers got it wrong and we had to scramble to find a place to stash hundreds of children’s bikes!

Modern logistics and shipping is dominated by algorithms. Look at any Amazon Fulfillment center and all the robotic bookcases whizzing by the workers. But as machine learning becomes ever more practical, it seems an obvious tool to squeeze ever more efficiency out of the supply chain. Yet not all machine learning methods are interpretable, and what can go wrong if a so-called black box method starts making decisions?

Now imagine a scenario like those extra children’s bikes, only it’s potentially explosive materials, and it’s happening in a high-speed environment where people may not even realize the material’s danger?

And given the scale of industrial accidents—and the interconnected complex system of the global supply chain—the dangers may be far higher than those posed by self-driving cars.

Network effects

It seems straightforward to flag conditions on goods for safety and proper handling. Indeed, the computer will likely be better than people at checking for issues. But what about more complex situations?

What if a mixture of two or more materials, none dangerous alone, but hazardous together, get stored in proximity? This combinatorial problem could also, in principle, be mitigated by computer checks. But what if there are competing firms, renting shipping space nearby, each using their own algorithms to ship and schedule materials?

When you throw interactions between potentially discordant systems into the mix, all hell could break loose. Multiple, competing black boxes, or even a combination of black box and interpretable AI, could lead to extremely unpredictable and dangerous scenarios.

Now we can see the interest and urgency in studying XAI.

Winter or crash?

Neural Network research went through two significant setback periods, known as AI winters. People often use the AI winters of the 1970s and 1990s to predict the future of AI. However, those past AI winters were about research interest: Problems arose that made neural networks appear untenable, and researchers shifted to alternatives. The 2010s have obliterated those concerns, cementing neural networks' place in practice. AI is too important to go away now. So it seems lessons from the previous AI winters are unlikely to apply.

If an AI system were to cause a large-scale industrial accident, it’s plausible that another AI winter could occur. However, now that AI is so successful in practice, an AI winter here would be quite different than researchers losing interest. Instead, it may be more akin to a stock market crash, with “buyers” fleeing and the market collapsing. The shine goes off because an accident has been made clear how dangerous AI can be, in a more visceral way than Skynet could ever be. Companies could pledge to slow or stop their research and development of AI systems. Or perhaps government regulation, for better or worse, comes in when legislators realize the broader safety implications of what has been wrought.

Fighting the last war

Lastly, it’s worth pointing out short-termism in my thoughts here.

The Beirut explosion just happened. I am focused on that same scenario: thousands of tons of improperly stored explosives-in-waiting. Someone who really works on these problems must be up all night, wracking their brain trying to foresee other possibilities. What if an AI diverts sewage treatment incorrectly? What if harmful manufacturing additives are accidentally added to containers for medicine?

What are the dangerous, unknown and truly unanticipated situations?

Or a self-driving car. ↩︎

Returning to campus

Mon, 24 Aug 2020 00:00:00 +0000

Today, my university is beginning move-in. Classes start on August 31.

It’s tough times for everyone now. COVID continues to upend our lives. My thoughts go out to all the students, staff, faculty, parents, friends and colleagues, at UVM and other universities and schools. Distance education is tough, for students and teachers. Working and studying from home is tough.

Be patient and forgiving. We can get through this, together. Stay strong!

Fahrenheit and Celsius – Trapped in a 10-sided box

Fri, 21 Aug 2020 13:00:00 +0000

The Fahrenheit temperature scale is a wonderful history lesson and a beautiful triumph of design. It’s just lost to our modern sensibilities.

This post is about Fahrenheit, the temperature scale that no one uses, except notably the United States. Yes, I’m an American, and I want to say some nice things about Fahrenheit. Wait wait! Don’t go! It’s not what you think! I’m very much not saying Fahrenheit is better than Celsius, it’s not. Indeed, while we’ve got bigger fish to fry right now, standards are important and the US should use SI ¹

I love talking about Fahrenheit. It offers an excellent distillation of history, the nature of scientific measurement, and even a lovely illustration of creative, outside-the-box-thinking.

Creativity and temperature? What gives?

When someone first encounters Fahrenheit, they generally recoil at the fact that the boiling point of water is 212 degrees. “212 degrees? Why such a weird number? Why not a nice number, like 100 degrees?”

That’s a fair question but it’s also a little presumptive. “Why not my way?” isn’t really giving this new entity the benefit of the doubt. Maybe there’s a reason behind the choice of number?

So, here’s an alternative question that almost no one would ask when encountering Fahrenheit, but perhaps they should:

What has a temperature of 100 degrees Fahrenheit ?

Did you ever think to ask that? Do you know the answer? You. You have a temperature of 100F.

Now, hold up. It’s more complicated than that: For one, Fahrenheit didn’t pick 100F exactly ². The point is he chose human body temperature for a reference. Is that a better choice than the boiling point of water? No, it lacks precision. But he had some excellent reasons in his era for it. And nowadays we never consider the possibility of a reference point except boiling or freezing water.

This brings up my second point on creative thinking. Sometimes it’s worth asking a painfully obvious question because you can learn a lot if there’s a very not-obvious answer.

What is the purpose of a temperature scale?

A temperature scale is used to record and communicate temperatures, right? You need a common language, a standard scale so everyone can reckon temperature. Right? This is what Celsius was designed for (and the metric system more generally).

This is not what Fahrenheit was designed for.

Fahrenheit was designed for making accurate thermometers when you lack the tools for making accurate measurements.

Think of the historical context. It’s 1724. Precise rulers are rare. Ice may be hard to come by and it takes a lot of firewood to boil water. There probably won’t be any distilled water. What can you do?

Fahrenheit’s choices when developing his scale were shaped by concerns that are entirely irrelevant today but were critical in his time. The Fahrenheit scale is not bad or weird, it’s obsolete.

Consider placing markings on the glass of the mercury-filled thermometer (which you just invented) after you’ve marked the reference points. You want these to be evenly spaced so numbers read from that scale are accurate. Maybe you have a straight edge, a compass, and a piece of string, but no accurate ruler. How can you accurately divide a line segment into 10 or 100 equal lengths?

Under these circumstances, would you rather divide that line segment into 8 lengths instead of 10?

What difference does 10 vs. 8 make? A lot, because $8 = 2^3$. All you have to do is learn to find the midpoint accurately ³, and then do this repeatedly.

(We take base-10 in SI (metric) for granted now, but it was extremely radical and ahead of its time. It was too hard to switch from base-2 to base-10 until accurate tools could be manufactured at scale. Two hundred years ago, a base-10 system, which is so natural to us now, was actually an argument against SI. Now, of course, the US persists with “standard” out of just plain stubbornness.)

Lastly, Fahrenheit’s other choices also reflect concerns over accuracy ⁴. For one, he decided not to use the freezing point of water as the zero reference. Instead, he chose 0F as the temperature of a brine—a mix of water, ice and ammonium chloride—because it is extremely stable thermally, more so than just water and ice. He also picked 32F for the freezing point of water because again dividing into a power-of-two number of line segments is easy to do accurately ⁵. And the fact that Fahrenheit is convenient for weather is no coincidence—Fahrenheit rejected using the boiling point as a reference because it was unsuitable for many meteorological applications.

Takeaway

So what’s the lesson here? Really, it’s Chesterton’s Fence: when you encounter an idea that doesn’t comport with your view (212 degrees? Why not 100?), try to think outside your box (If 212 is boiling, what is 100?) and see if there’s a reason behind it. Take a pinch of history and humility—you might learn a lot.

Especially for volumes. Pints? Quarts? Gallons? I can’t keep track of any of that. ↩︎
He actually used 96F. Forty years after Fahrenheit died, a Royal Society committee decided to redefine Fahrenheit so that the boiling point of water is exactly 212F (because reasons), and the “modern” Fahrenheit has the classic 98.6F average human body temperature. ↩︎
Again, remember you don’t have a reliable ruler, so you can’t just measure the distance. You need to be a little clever with a geometric construction. ↩︎
Also worth noting that Fahrenheit was not the single source of these ideas. He built off Boyle, Newton, Halley (yes, that Halley), and Ole Roemer. ↩︎
Of course, why not 16 or 64? Yes, 32 is still fairly arbitrary. ↩︎

Tyrell Tests

Thu, 20 Aug 2020 13:00:00 +0000

Tyrell: Demonstrate it. I want to see it work.
Deckard: Where’s the subject?
Tyrell: I want to see it work on a person. I want to see a negative before I provide you with a positive.
Deckard: What’s that going to prove?

Emphasis mine.

(I don’t think this post contains spoilers for Blade Runner, but you’ve been warned. Also, the movie is almost 40 years old.)

Blade Runner follows a detective named Deckard searching for a group of renegade synthetic humans called replicants in the far-flung future dystopia of 2019. Replicants, created by the extremely wealthy and successful Tyrell, CEO of the massive Tyrell corporation, are indistinguishable at a glance from real humans. When Tyrell meets Deckard for the first time, he asks Deckard to demonstrate the testing device that Deckard uses when interrogating a person detect whether they are replicant or human.

In their exchange, Tyrell wants to apply the test where he knows the answer is negative, and Deckard does not think this is useful. But in general, whenever performing an experiment, testing a new method, or even debugging your code, it’s a great idea to see how the machinery works in the null case. ¹

So, I like to call this practice Tyrell Tests. ²

Cats, Dogs, Mass Hysteria

A student comes to me with their new cat-dog machine learning classifier: “Look, it’s correct 98% of the time!”

“Fantastic, send it some random data. What does it do?”

“Hmm, correct 99% of the time.”

“Interesting… What exactly do you mean by ‘correct’?”

“…”

Time passes. The student returns, “It turns out the training data is 98% cats, so the classifier always says ‘cat’. Then, when I shuffled the image labels—randomly assigning labels to images ³—almost every image was still a cat.

Tyrell Tests are especially handy when using machine learning, where it’s devastating to fail to notice data problems such as class imbalance. But Tyrell Tests can help identify issues in the data, the method or model, or in your code, in all sorts of domains:

In network science, you have a new inference method for learning statistically significant structures in the network. Send it a random graph and see what it finds.
In time series analysis, you have a method to forecast future values. Send it a randomized time series and see what it finds.
In Bayesian inference, send some random data through your likelihood and see how (or if) the posterior changed.

Tyrell Tests are a cutesy name for a simple and rather obvious idea. Still, this cutesy name brings me endless joy!

But the moral of the story is to stress your methods and interrogate your results. It’s easy to be fooled. Tyrell Tests, using known negatives for confirmation, are one tool worth keeping at the ready.

And yes, if you know how the scene plays out, the film itself undermines my argument here. It seems the filmmakers agree with Deckard. Oh well. ↩︎
Argument by Alliteration. ↩︎
Notice above that ‘some random data’ lacks specificity. ↩︎

The life-changing science of scientific writing

Wed, 19 Aug 2020 13:00:05 +0000

As a working scientist and scholar, I am, in many ways, a professional writer. I enjoy writing and hope to get better doing it. Heck, that’s one reason I started making these posts.

Unfortunately, I also started writing here because I’ve struggled with peer review of some of my papers. Now, let me be fair: there are many excellent reviewers, and I value the improvements they’ve helped me bring to my papers. But I seem to get more careless and just plain bad reviews than ever, especially recently. It’s been challenging to cope.

Sharpening the saw

Attempting to mitigate these poor reviews, I began intentionally studying scientific and technical writing in the hopes of writing more clear and focused papers, and I learned a ton! Of course, writing is something best studied as a grad student, not as a tenure-track faculty with dozens of papers already under his belt, but I’m trying my best.

Now that I have a whole bookshelf on writing, I can see the bigger picture a bit, and point out what’s standard advice that all writers give on scientific writing. I may be able to provide some advice on where to get started if you find yourself struggling as I did (and still do) with writing.

And what I found, and want to share, is a genuinely magical article. The ideas posed in this article are the central tenets of many entire books dedicated to scientific writing. I can say, without exaggeration, that this article changed my life:

Gopen, George D., and Judith A. Swan. “The science of scientific writing.” American Scientist (1990).

This brief article, only 9 two-columns journal pages in its original printing, contains as much useful information as entire books on the subject.

So you can just stop right now and dive into Gopen and Swan. In terms of effort to reward, this could be the best possible bang for your buck!

Still here?

Despite being redundant when you can read the original article, I’d like to discuss a point they make that I find hugely useful to think about. It’s about getting a text to flow so that the reader glides through the information.

Stress Position / Topic Position

In their article, Gopen and Swan ground their piece by discussing how readers react to a text, what they call reader expectations, and how good writing should account for these expectations. One such expectation to consider is what readers tend to expect from information that appears in different positions of a text. Two positions are worth considering. The first Gopen and Swan dub the stress position:

It is a linguistic commonplace that readers naturally emphasize the material that arrives at the end of a sentence. We refer to that location as a “stress position.” If a writer is consciously aware of this tendency, she can arrange for the emphatic information to appear at the moment the reader is naturally exerting the greatest reading emphasis. As a result, the chances greatly increase that reader and writer will perceive the same material as being worthy of primary emphasis. The very structure of the sentence thus helps persuade the reader of the relative values of the sentence’s contents.

The topic position, in contrast, appears at the beginning:

The information that begins a sentence establishes for the reader a perspective for viewing the sentence as a unit: Readers expect a unit of discourse to be a story about whoever shows up first. “Bees disperse pollen” and “Pollen is dispersed by bees” are two different but equally respectable sentences about the same facts. The first tells us something about bees; the second tells us something about pollen. The passivity of the second sentence does not by itself impair its quality; in fact, “Pollen is dispersed by bees” is the superior sentence if it appears in a paragraph that intends to tell us a continuing story about pollen. Pollen’s story at that moment is a passive one.

Readers also expect the material occupying the topic position to provide them with linkage (looking backward) and context (looking forward). The information in the topic position prepares the reader for upcoming material by connecting it backward to the previous discussion.

Here Gopen and Swan are talking about sentences, but these ideas work across text scales. Paragraphs have topic and stress sentences, sections have topic and stress paragraphs, and so on.

Understanding and controlling the contents of these positions gives the writer leverage over the reader. Tuning what’s in the topic position makes sure the reader is primed for what’s coming. Tuning the stress position’s contents improves the odds that the big takeaway of the text is received. Further, linking the stress position of the preceding text with the subsequent text’s topic position helps ensure flow.

Structural principles of scientific writing:

Stress and topic positions are not the only guidance Gopen and Swan provide. They cover seven writing principles in total:

Follow a grammatical subject as soon as possible with its verb.

Place in the stress position the “new information” you want the reader to emphasize.

Place the person or thing whose “story” a sentence is telling at the beginning of the sentence, in the topic position.

Place appropriate “old information” (material already stated in the discourse) in the topic position for linkage backward and contextualization forward.

Articulate the action of every clause or sentence in its verb.

In general, provide context for your reader before asking that reader to consider anything new.

In general, try to ensure that the relative emphases of the substance coincide with the relative expectations for emphasis raised by the structure.

That’s a great cheat sheet, but if you have any interest in this, I cannot stress enough ¹ how much you should read the whole article.

Gopen and Swan also make clear that these as “principles” not “rules”:

None of these reader-expectation principles should be considered “rules.” Slavish adherence to them will succeed no better than has slavish adherence to avoiding split infinitives or to using the active voice instead of the passive.

(I’m sometimes guilty of treating the topic and stress position principles as rules…)

Perhaps the notion of topic and stress positions is obvious to you. For me, I had only a kind of vague, implicit understanding of it before reading Gopen and Swan’s article. I could tell when the flow was lacking but couldn’t always put my finger on the exact problem. By making these issues precise and explicit, Gopen and Swan have helped me be a better, more efficient writer.

Gopen and Swan: Thank you!

If you’re still hungry for more on this, the next place I recommend you go is Schimel’s Writing Science (2011) [Google Books]. A great book!

See what I did there? ↩︎

Writing Books with Jupyter

Tue, 18 Aug 2020 13:00:00 +0000

I’m a huge fan of the Jupyter project, and love to see all the progress being made with JupyterHub and especially JupyterLab.

Recently learned of another Jupyter project, Jupyter Book:

Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material.

It’s still early in development, but this could scratch a real itch that I have: pushing computational notebooks beyond working documents and into prose, especially long-form writing, reveals some sticking points. Jupyter Book looks to address exactly these problems with features including:

Write publication-quality content in markdown You can write in either Jupyter markdown, or an extended flavor of markdown with publishing features. This includes support for rich syntax such as citations and cross-references, math and equations, and figures.

Insert notebook outputs into your content Generate outputs as you build your documentation, and insert them in-line with your content across pages.

Add interactivity to your book You can toggle cell visibility, include interactive outputs from Jupyter, and connect with online services like Binder.

Generate a variety of outputs This includes single- and multi-page websites, as well as PDF outputs.

Build books with a simple command-line interface You can quickly generate your books with one command, like so: jupyter-book build mybook/

I often wish I had these features when preparing notebooks for my courses, especially the citations and cross-references. It was a pain getting my Whirlwind Tour of Python fork going, that’s for sure.

And the pages are gorgeous:

Check out that dynamic table of contents on the right. And that’s there by default, not some third-party extension.

"Spooky action at a distance: The future magic of remote collaboration"

Mon, 17 Aug 2020 13:10:00 +0000

https://arstechnica.com/features/2020/08/work-from-home-02-collaboration/

Interesting read over at Ars Technica, a team that’s been all-remote for decades:

Whatever passes for “normal” over the next few years will demand a lot from collaboration tools, as organizations re-examine how to achieve their missions and stay financially viable. The new world of collaboration will require organizations to replicate the unstructured interactions of the office and help employees feel like part of a coherent team even when they are distant, enabling social interactions and reinforcing the team agenda in the absence of up-close face time.

Based on years of remote collaboration work at various organizations and observations of things multiple organizations have done over the past six months to adjust, here are the things I see as keys to successful team operations in the lockdown and post-lockdown world.

I, like many of us, have been thinking long and hard about what the future may hold, for work and life. With COVID-19 upending society, whatever happens we are in for a wild ride.

A5 – Arguing for attention

Mon, 17 Aug 2020 13:05:00 +0000

A few years ago, I came up with a cute little mantra that I love thinking about. It’s called A5:

A5: Argument by Analogy, Anecdote, Alliteration, or Acronym.

A5 ¹ represents the rhetorical flourishes we all use to make our points more memorable, our ideas a little stickier. I find it especially pleasing that “A5” is an A5 in more than one way.

The A5 mantra is a bicycle for critical thinking. When reading or listening, I sometimes find myself thinking, “Oh, that’s an A5.” Then I try to decide if their point is trustworthy, or if they are using marketing for cover.

Here’s a few examples of A5s:

Exploration-Exploitation tradeoff (Alliteration) ²
Google is the farmer, you’re the cow (Analogy)
The 10,000-hour rule (Anecdote) ³
Together Everyone Achieves More (Acronym)
Risk/Reward Ratio (Alliteration)
If the glove doesn’t fit, you must acquit (Rhyme-as-reason)

Wait, Rhyme-as-reason?

Yup. A5 is also a great illustration of the dangers of slick rhetoric—it sounded pretty authoritative and it’s easy to rattle off, but it’s not exhaustive, not even close. Many other rhetorical tricks exist, I just haven’t found out how to cram them into the slick package.

But this also leads me to a deeper question: is it bad to make an idea more memorable?

“Marketing” can be a bit derogatory but, when communication is the goal, we need to maximize the impact of our ideas. Therefore, it’s in our best interests to make our ideas sticky. The very act of grinding away at an idea, making it sharper and stronger, will almost surely also teach the thinker better ways to communicate that idea.

Language also plays a role here. The words we know shape the way we think (linguistic relativity) and our experiences collectively shape language over time. So it makes sense that powerful ideas are more likely to find good rhetorical purchase—the language is primed to express them. But is this a double-edged sword? Are we limited by the language of our time? ⁴ I often wonder about powerful ideas being lost because we cannot express them (hypocognition). However, humans are pretty resourceful, and we have strong tools to help with this problem, such as mathematics.

Regardless, ideas should have an inherent quality or importance, but their fitness will also depend on their expression.

OK, that got deep for a second, and my thoughts on the interplay between language and thought are wholly unoriginal. But nevermind, the point here is A5—a sticky idea about sticky ideas.

By the way, dear reader, please let me know if you can think of a synonym for “Rhyme” that starts with ‘A’. Then we can give birth to A6! ⁵

Not to be confused with the Audi luxury car, the Apple microprocessor, the ISO standard paper size, or the alternating group of degree 5. https://en.wikipedia.org/wiki/A5 ↩︎
“Exploration-Exploitation tradeoff” is a powerful, fundamental idea, and is just expressed so exceptionally well. Chef’s kiss ↩︎
Malcolm Gladwell’s entire career is basically argument by anecdote. ↩︎
Yes, I’ve seen Arrival. ↩︎
“Assonance” is close… ↩︎

Getting started (A blog? In 2020?)

Mon, 17 Aug 2020 13:00:00 +0000

I’ve decided to start blogging. I can already hear the deafening silence: A blog?!? In 2020? Blogging is dead and buried!

Indeed. Blogs are long gone (having peaked in popularity over 10 years ago), and the world is currently on fire, so why bother now?

Happiness

I enjoy writing and, while I’m no great talent, I love getting better at putting down words. And as a working scientist, I am effectively a professional writer. In this, I consider myself lucky.

And yet, there’s a problem: peer review.

I love writing about science and my research, but my general negative feeling towards peer review has curdled these past few years. Is it just me, or are paper reviews worse than ever? ¹

Let me be clear: there’s many excellent reviewers, and I’m very grateful to the many reviewers who have given me important and insightful comments. My papers are better due to their efforts. However, bad peer review is increasingly sucking the joy out of writing and even doing research. My productivity is in the toilet. I find myself abandoning interesting, even exciting research projects before I even begin because I can’t stomach the thought of a careless reviewer giving all the months or years of work a cursory glance and then a rejection.

So what about the blog?

I want a venue to put down my thoughts free of gatekeepers. Free of the obsequious bowing and scraping needed to placate bad reviewers. The deaths of a thousand unnecessary caveats and qualifiers that always seem to scarify my peer-reviewed writing. Disagree with me? Too bad. Angry about that missing comma? Too bad. Think the article is clickbait? ² Too bad!

Topics

I’ve got lots of ideas to write about—some from my own areas of complex systems, networks, and data science. But I’m looking forward to posts on science and tech news more generally, life inside (and outside) academia, teaching and learning, and just all the fun little tips and tricks to be a better scientist that I’ve picked up along the way—the work of science.

Let’s go

So I hope this little experiment of a blog brings a little joy back to writing and thinking. I have no expectations beyond this, no huge audience will come to a blog in 2020, and who has time to read anymore? And that’s OK.

If I’m just shouting into the void, here the void won’t whisper back, “You didn’t cite this paper!”. Well, if the void says that, fair enough, I can revise—but the void can’t stop me from pressing ‘Publish.’

And yes, I’m still going to submit articles for peer review, hopefully more than ever. But most importantly, let’s all work to make it to 2021.

Of course, it could be that the quality of my work is not keeping pace with what reviewers want. But I don’t think that explains everything. ↩︎
Yes, a reviewer recently wrote exactly that. ↩︎